feat(review): measured test results in pr_body + codecov-backed test-inadequate finding (POC) - #878
Draft
guyoron1 wants to merge 3 commits into
Draft
feat(review): measured test results in pr_body + codecov-backed test-inadequate finding (POC)#878guyoron1 wants to merge 3 commits into
guyoron1 wants to merge 3 commits into
Conversation
The PR body's testing section is written by the code agent, and
nothing constrained its form. Agents emit checked boxes ("- [x] Unit
tests for X") that assert results no one measured — see
fullsend-ai/fullsend#6285, whose test plan checks six boxes while
codecov/patch reports 28.24% of diff hit against an 80% target.
Step 10d now requires the section to name the command step 9c ran and
its outcome, report a coverage number only when the repo's own test
command prints one, and mark anything unrun as "not measured" with the
reason — the disclosure step 9c already requires in the commit message.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: guy oron <goron@redhat.com>
The correctness sub-agent owns the missing-test, test-inadequate and coverage-reduced categories but received no coverage data, and it has no Bash tool to fetch any — so a PR body claiming tests read as evidence of tests. On fullsend-ai/fullsend#6285 the review approved a test plan of six checked boxes over a 28.24% patch. The orchestrator already fetches PR context with gh api; step 2c adds one call for the codecov check run on the head SHA and passes the result to correctness as the context package's coverage field. Repos without Codecov get "no coverage signal", which anchors nothing — adequacy is then judged from the diff, as before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: guy oron <goron@redhat.com>
codecov check_run -> ready-tests label -> QualityFlow plans at the gap -> pr-review emits test-inadequate from the measured patch coverage. Maps each hop to the branch that carries it. Signed-off-by: guy oron <goron@redhat.com>
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / POC for the 2026-08-19 contributors meeting — not ready for review.
Reviews don't read coverage today (0/88 agent reviews on the last 40 fullsend PRs looked at codecov; 90% of test-plan checkboxes ticked). This makes test evidence measured instead of self-reported:
code-implementation/ pr_body: report measured test results (what actually ran) instead of a static checklist.pr-review: fetch the codecov check-run and emit a correctness findingtest-inadequatewhen patch coverage is below target.docs/POC.md: end-to-end story — codecov check_run →ready-testslabel → QualityFlow plans tests at the gap → pr-review flagstest-inadequate. Companion drafts: fullsend core (normeventcheck_completed+ label workflow),.fullsend(QF trigger on the label),redhat-community-ai-tools/qualityflow#1(coverage-gap mode).hack/lint-agent-docspasses. Housekeeping before un-drafting: the first two commits carry a Co-Authored-By trailer to squash out.