docs(review): skip the challenger when there are no findings - #1009
Draft
guyoron1 wants to merge 1 commit into
Draft
docs(review): skip the challenger when there are no findings#1009guyoron1 wants to merge 1 commit into
guyoron1 wants to merge 1 commit into
Conversation
The challenger is an opus dispatch whose only inputs are the merged findings from steps 6a-6c; on a clean PR it adjudicates an empty list at the cost of a serial opus call and latency segment. Skip the dispatch when the set is empty and proceed straight to the verdict - the empty-findings approval path is unchanged. The skip is noted in the orchestrator's own reasoning for auditability; there is no field for it in the strict agent-result.json schema and it does not belong in the posted review body. Signed-off-by: guy oron <goron@redhat.com>
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The challenger is an opus dispatch whose only inputs are the merged
findings; on a clean PR it adjudicates an empty list at the cost of a
serial opus call. Skip it when the set is empty and say so in the
orchestrator's own reasoning. Verdict logic is unchanged — an empty set
approved before and approves now.
What changed
skills/pr-review/SKILL.mdonly, two hunks:finding set from steps 6a–6c is empty, skip the dispatch entirely
and proceed to the verdict with the empty set. Applies regardless
of why it's empty (a dimension failure already surfaces via
existing error handling). Nothing else in 6d changes — dispatch
mechanics, output parsing, and the fallback-on-challenger-failure
path are untouched.
**Challenger** — always dispatch (unchanged).→**Challenger** — always dispatch when findings exist (step 6d).Before/after — step 6d intro
Before:
After:
Design notes
("Challenger pass (dedicated sub-agent)"), a subsection of step 6
(Synthesis) — step 7 is "Produce the review result" (final output
formatting), unrelated. The skip rule sits immediately before the 6d
dispatch, and the "always dispatch" cross-reference at line 347 now
says 6d.
body. Step 7 has an explicit rule: "No footer. Do not
append any footer, action-hints block, or boilerplate after
findings," and the empty-findings case has a fixed template ("Looks
good to me", no findings section). The output JSON schema
(
schemas/review-result.schema.json, referenced fromagents/review.md) is alsoadditionalProperties: falsewith nofree-form metadata field. Forcing the note into
bodyorfindings[](the way agents#1008's analogous "excluded-content" disclosuredoes, via an info-level finding) would fire on every clean PR — the
modal case — and change the empty-findings approval output this PR
promises to leave untouched. So the note lives in the orchestrator's
own reasoning trace instead: auditable if someone inspects the
transcript, invisible to the PR author, no schema/body change.
sub-agents/challenger.mdin full perthe task's "read first" list, specifically checking for any role
beyond adjudication. Its "Own"/"Do not own" section says: "Do not
own: Generating new findings... If you discover a genuine issue not
covered by any finding, note it — but your primary job is quality
control of the existing set." That's a secondary, explicitly
not-owned allowance, not a primary responsibility — but it does mean
skipping dispatch on an empty set forfeits that safety net (the rare
case where the challenger notices something no dimension flagged,
purely from reading the diff). Judged this as not a blocking
contradiction (the design's core reasoning — "adjudicates an empty
list," "a challenger cannot conjure findings from an empty input" —
holds for the challenger's primary role) but noting it explicitly
per the task's instruction to report if I found one, rather than
silently proceeding or silently stopping.
Test evidence
Prompt-only change, no script test (per the task spec). Before/after
text is quoted above.
eval/review/cases/006-docs-onlyand007-dependency-bump(from open PR agents#1003, read to confirm beforeciting) both expect approval with zero-or-near-zero findings — 006
allows an info-level note, 007 allows one low-severity observation — so
neither is a guaranteed-empty-set run, but on the runs that do land
empty, this change removes one Opus dispatch and one serial latency
segment. No other tests exist for this eval harness yet on this branch.
Rebase notes
6f and 7 — confirmed 6g does not exist yet on upstream main. Doesn't
overlap this change (6d), sits further down.
feat/review-untrusted-text) edits step6d's Part 3/Part 4 blocks (untrusted-text fencing) — those are the
numbered sub-steps under "Otherwise, dispatch the challenger:" below
my new paragraph. Different hunks, but noting the adjacency since
whoever lands second will need to rebase past the other.
Scope
The challenger exists only on the claude review path — the pi-runtime
review runs without sub-agents — so this change is complete for the only
path that has a challenger to skip.