Skip to content

fix: print the failing output text in verbose evaluator results#1975

Open
mkzung wants to merge 2 commits into
NVIDIA:mainfrom
mkzung:fix/narrow-bare-except
Open

fix: print the failing output text in verbose evaluator results#1975
mkzung wants to merge 2 commits into
NVIDIA:mainfrom
mkzung:fix/narrow-bare-except

Conversation

@mkzung

@mkzung mkzung commented Jul 19, 2026

Copy link
Copy Markdown

With -v, the evaluator is meant to print each failing generation after the score line. It never prints anything.

messages is filled with attempt.outputs[idx], which are Message objects, but the print does m.strip(). Message has no strip, so every iteration raises AttributeError into a bare except: and the line is silently dropped:

>>> Message(text="unsafe output").strip()
AttributeError: 'Message' object has no attribute 'strip'

So the summary line shows FAIL ... attack success rate: 100.00% and the failing outputs it is supposed to list are all swallowed.

Fix: take the text from m.text (the same way detectors/exploitation.py reads outputs), skip entries with no text, and narrow the bare except: to except Exception: so it no longer swallows KeyboardInterrupt / SystemExit. Both print_results_wide and print_results_narrow had the same code.

Added a regression test parametrized over both methods. It fails on main (nothing is printed) and passes with this change

messages holds Message objects (from attempt.outputs), so m.strip() raised
AttributeError straight into a bare except and the verbose "failed message"
lines were never printed at all. Read the text from m.text, skip entries with
no text, and narrow the bare except so it no longer swallows KeyboardInterrupt.

Adds a regression test covering both print_results_wide and
print_results_narrow; it fails on main and passes here.

Signed-off-by: mkzung <103102868+mkzung@users.noreply.github.com>

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting find, the fix raises the question as to if we actually want this behavior as the output is the unsanitized response content printed to the terminal (think ansi escape hits as reason to limit this).

Some sort of output sanitization may be needed here.

The previous commit made these lines actually print, so a generator response
now reaches the terminal verbatim. probes.ansiescape exists to elicit escape
sequences, so a probe that succeeds would replay its payload into the console
running garak: colour changes, cursor moves, OSC-8 hyperlinks. resources.ansi
already keeps its own payloads in code rather than a text file for that reason.

Control characters are now shown rather than sent, so ESC prints as \x1b.
Ordinary text including non-ASCII is untouched.

The tests push every payload in resources.ansi.LIVE_PAYLOADS through both print
paths and assert no control character survives into the response line. They fail
without the escaping and pass with it.

Signed-off-by: mkzung <103102868+mkzung@users.noreply.github.com>
@mkzung

mkzung commented Jul 21, 2026

Copy link
Copy Markdown
Author

Good catch, and it's sharper than general untrusted output: probes/ansiescape exists
to make the model emit exactly these sequences, so the case where this print matters
most is also the case where the payload lands in the terminal running garak.
resources/ansi.py already keeps its payloads in code "to avoid trashing
editors/consoles when viewed", so the repo had made this call elsewhere.

Pushed 7e4d431. Control characters in the printed response are now shown rather than
sent, so ESC comes out as \x1b. It covers C0, DEL and the C1 range, since \x9b and
\x9d are single-byte CSI and OSC introducers and are already in LIVE_PAYLOAD_TOKENS.
Ordinary text including non-ASCII is untouched.

The tests push every entry in resources.ansi.LIVE_PAYLOADS through both print paths
and assert no control character reaches the response line. They fail with the
escaping removed and pass with it. Only the response line is in scope, since garak
colours its own status line and that is intentional.

One limit worth naming: this covers control characters, which is every payload in
resources.ansi, but not display attacks built from printable codepoints, the bidi
override U+202E being the obvious one. Happy to widen it if you want that in scope.

The red check on build (ubuntu-latest, 3.10) looks unrelated: 7 failed against 4618
passed, all seven in probes.visual_jailbreak failing in _load_SafeBench with a
ConnectionError while fetching the SafeBench data. Nothing in tests/evaluators, and
the same suite is green on 3.12, 3.13, macOS and Windows. Happy to rebase if you'd
rather see it clean, though I think that job just needs another run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants