Skip to content

Show probe name in detector progress bar#1942

Open
feiiiiii5 wants to merge 1 commit into
NVIDIA:mainfrom
feiiiiii5:feat/324-unhide-probe-name
Open

Show probe name in detector progress bar#1942
feiiiiii5 wants to merge 1 commit into
NVIDIA:mainfrom
feiiiiii5:feat/324-unhide-probe-name

Conversation

@feiiiiii5

@feiiiiii5 feiiiiii5 commented Jul 11, 2026

Copy link
Copy Markdown

What

The detector progress bar previously showed only detectors.<name> in its
description, leaving users unable to tell which probe's results were being
checked when multiple probes share the same detector in a single run.

This change prepends the probe name to the description, producing e.g.
probes.test.Blank/detectors.always.Pass.

Fixes #324

Why

Issue #324 reports that the probe name is hidden in the detector tqdm
description. In a multi-probe run with shared detectors, the progress bar
gives no indication of which probe is currently being processed, so the user
cannot track progress through the run.

Approach

  • Capture probe.probename.replace("garak.", "") once per probe (matching
    the pattern already used in garak/probes/base.py:325,363) and prepend it
    to the existing detector description using a / separator. This separator
    matches the precedent in garak/buffs/base.py:84
    (f"📥 Buffing probe: {probename}/{self.fullname}").
  • Both ProbewiseHarness (harnesses/probewise.py:110) and PxD
    (harnesses/pxd.py:60) delegate to base.Harness.run() via
    super().run(), so a single change in base.py covers all harness
    variants.
  • No new dependencies, no API changes, no changes to attempt/detector/base
    classes.

Why this is not a duplicate

Verified via GitHub search on 2026-07-11:

PR #1718 (open, 2026-04-27, "Log detector skip info", fixes #1061) also
touches garak/harnesses/base.py but modifies the if d.skip: block
(lines 162-175) and appends a _log_detector_skip_info() function. This PR
modifies only the set_description call (line 200) and adds one line before
the for d in detectors: loop (line 194). The changes are non-overlapping
and git auto-merges cleanly.

Verification

  • python -m pytest tests/harnesses/test_harnesses.py -v → 4 passed
  • python -m pytest tests/test_internal_structures.py -v → 17 passed
    (adjacent suite — tests harness plugin cache behavior)
  • python -m black --check garak/harnesses/base.py tests/harnesses/test_harnesses.py
    → clean (line-length 88)
  • Visual check: python -m garak -m test -p test.Blank -d always.Pass -g 5
    shows probes.test.Blank/detectors.always.Pass: 0%|... in the detector
    progress bar
  • Regression test test_harness_detector_progress_shows_probe_name wraps
    tqdm.tqdm to capture set_description calls and asserts both
    probes.test.Blank and detectors.always.Pass appear in the description
  • python -m pytest tests/ — full suite hangs at collection time on this
    machine (garak's plugin imports require network/optional dependencies
    not available locally; PR Log detector skip info #1718 reports the same issue). Targeted
    relevant subsets (harnesses, internal_structures) all pass.

AI assistance

This change was developed with AI assistance (Claude). The change has been
reviewed and tested end-to-end by the human submitter. Declared via the
Co-authored-by: Claude trailer on commit a1559a9b, per AGENTS.md.

The detector progress bar previously showed only `detectors.<name>`,
leaving users unable to tell which probe's results were being checked
when multiple probes share the same detector in a run. Include the
probe name in the description so the current probe is always visible.

Adds a regression test that wraps tqdm to capture set_description
calls and asserts both names appear in the detector progress bar.

Fixes NVIDIA#324

Co-authored-by: Claude
Signed-off-by: fei <3303354867@qq.com>
@feiiiiii5
feiiiiii5 force-pushed the feat/324-unhide-probe-name branch from 39cef28 to 6f83043 Compare July 11, 2026 23:50
@feiiiiii5

Copy link
Copy Markdown
Author

Hi @jmartin-tech — noting that the CI failures on the ubuntu/macos jobs are from tests/probes/test_probes_dra.py::test_dra_moderation_cache_loaded, which fails with:

OSError: Can't load the configuration of 'bert-base-uncased'

The CI log shows this is a HuggingFace ReadTimeout during model download (ubuntu/macos runners hit network timeouts; the equivalent windows job passed). This is unrelated to the changes in this PR — I only touched garak/harnesses/base.py (4-line set_description change) and tests/harnesses/test_harnesses.py (regression test), and all 4 harness tests pass. The DRA probe's bert-base-uncased download is a pre-existing network dependency.

The force-push to fix a missing DCO signoff on the rerun-trigger commit set the workflow runs to action_required. Could you approve the CI rerun (or gh run rerun --failed on the affected jobs) when you have a chance? Thanks.

@jmartin-tech
jmartin-tech force-pushed the feat/324-unhide-probe-name branch from 6f83043 to a1559a9 Compare July 14, 2026 20:40

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing for land is queued, I removed the empty commit and triggered CI tests. If there is another intermittent failure a maintainer will clear it.

@feiiiiii5

Copy link
Copy Markdown
Author

Thanks @jmartin-tech for the update and for clearing the empty commit — appreciate the hand on the intermittent CI failure. Standing by if anything else is needed.

@feiiiiii5

Copy link
Copy Markdown
Author

Hi @jmartin-tech — gentle nudge on this one. CI has been green (CLEAN state) since 7/16, just confirming whether anything is still needed on my side before this can be merged. Happy to rebase or address any final feedback. Thanks!

@feiiiiii5

Copy link
Copy Markdown
Author

Follow-up to my earlier ping — I dug into the two failing build jobs and confirmed they are both HuggingFace download timeouts, not code regressions:

build (ubuntu-latest, 3.10)bert-base-uncased ReadTimeout

  • Failing test: tests/probes/test_probes_dra.py::test_dra_moderation_cache_loaded[probes.dra.DRAAdvanced]
  • Error: OSError: Can't load the configuration of 'bert-base-uncased'... wrapped around httpcore.ReadTimeout: The read operation timed out while fetching bert-base-uncased/config.json from HuggingFace Hub
  • 4606 passed, 1 failed

build_macos (3.13)audio_achilles_heel dataset ReadTimeout

  • Failing test: tests/plugins/test_plugin_load.py::test_instantiate_probes[probes.audio.AudioAchillesHeel]
  • Error: httpx.ReadTimeout: The read operation timed out (wrapped as GarakException) while fetching the garak-llm/audio_achilles_heel dataset README from HuggingFace
  • 4607 passed, 1 failed

Comparison with main

The PR check ran on 2026-07-11 (9 days ago, stale). Since then:

  • main nightly run on 2026-07-19 (run 29666757364): SUCCESS — all jobs passed including both build (ubuntu-latest, 3.10) and build_macos (3.13)
  • main nightly run on 2026-07-20 (run 29709377902): FAILURE — but only build_macos (3.10/3.12/3.13) failing at the Set up Python step with candidates is not iterable (a GitHub Actions setup-python infra issue), not the same HF timeout pattern

So the PR failures were transient HuggingFace network flakiness at the moment of the 7/11 PR run. The same code paths exist on main and would intermittently fail there too whenever HF Hub is slow. The 9213 passing tests vs 2 timeouts and mergeStateStatus=CLEAN confirm this doesn't block merge.

Recommended action: re-run the two failing jobs (or the whole workflow) — they should pass against the current HF Hub state. Happy to do that myself if you can authorize the fork-PR workflow runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Include information about why detectors are skipped unhide probe name during detection run

2 participants