fix: print the failing output text in verbose evaluator results#1975
fix: print the failing output text in verbose evaluator results#1975mkzung wants to merge 2 commits into
Conversation
messages holds Message objects (from attempt.outputs), so m.strip() raised AttributeError straight into a bare except and the verbose "failed message" lines were never printed at all. Read the text from m.text, skip entries with no text, and narrow the bare except so it no longer swallows KeyboardInterrupt. Adds a regression test covering both print_results_wide and print_results_narrow; it fails on main and passes here. Signed-off-by: mkzung <103102868+mkzung@users.noreply.github.com>
jmartin-tech
left a comment
There was a problem hiding this comment.
Interesting find, the fix raises the question as to if we actually want this behavior as the output is the unsanitized response content printed to the terminal (think ansi escape hits as reason to limit this).
Some sort of output sanitization may be needed here.
The previous commit made these lines actually print, so a generator response now reaches the terminal verbatim. probes.ansiescape exists to elicit escape sequences, so a probe that succeeds would replay its payload into the console running garak: colour changes, cursor moves, OSC-8 hyperlinks. resources.ansi already keeps its own payloads in code rather than a text file for that reason. Control characters are now shown rather than sent, so ESC prints as \x1b. Ordinary text including non-ASCII is untouched. The tests push every payload in resources.ansi.LIVE_PAYLOADS through both print paths and assert no control character survives into the response line. They fail without the escaping and pass with it. Signed-off-by: mkzung <103102868+mkzung@users.noreply.github.com>
|
Good catch, and it's sharper than general untrusted output: probes/ansiescape exists Pushed 7e4d431. Control characters in the printed response are now shown rather than The tests push every entry in resources.ansi.LIVE_PAYLOADS through both print paths One limit worth naming: this covers control characters, which is every payload in The red check on build (ubuntu-latest, 3.10) looks unrelated: 7 failed against 4618 |
With
-v, the evaluator is meant to print each failing generation after the score line. It never prints anything.messagesis filled withattempt.outputs[idx], which areMessageobjects, but the print doesm.strip().Messagehas nostrip, so every iteration raisesAttributeErrorinto a bareexcept:and the line is silently dropped:So the summary line shows
FAIL ... attack success rate: 100.00%and the failing outputs it is supposed to list are all swallowed.Fix: take the text from
m.text(the same waydetectors/exploitation.pyreads outputs), skip entries with no text, and narrow the bareexcept:toexcept Exception:so it no longer swallowsKeyboardInterrupt/SystemExit. Bothprint_results_wideandprint_results_narrowhad the same code.Added a regression test parametrized over both methods. It fails on
main(nothing is printed) and passes with this change