feat(daemon): add judgment-automation policy sweep actor#3229
Merged
Conversation
Problem: the MCP judge dispatcher (polylogue-800m) lets an operator judge assertion candidates in bulk, but nothing calls it automatically -- every agent-authored candidate defaults to waiting on a human, which does not scale to agent-authored annotation volume. What changed: new polylogue/daemon/judgment_automation.py adds a policy engine (evaluate_candidate) that reads per-kind confidence thresholds from a new [judgment_automation.policies] config table and decides accept/ reject/escalate for each CANDIDATE assertion, plus a periodic daemon sweep (periodic_judgment_automation_sweep) wired into daemon/cli.py's existing periodic_loops list. Auto-judgeable candidates are applied via the same judge_assertion_candidates storage chokepoint the MCP judge dispatcher calls; the escalated residue gets a deterministic handoff-kind assertion pointing at the candidate so it is an explicit, queryable review queue instead of silent limbo. New config: judgment_automation_enabled (fail-closed bool, off by default, following the mcp_judge_enabled pattern), interval_s, batch_limit, and the policies table. The sweep is gated by BOTH its own opt-in and mcp_judge_enabled -- it exercises the same judge write authority as the MCP dispatcher, so enabling automation alone must not grant that authority. Solution notes: reused the existing daemon periodic-loop pattern (embedding_backlog.py) rather than a DaemonConverger stage -- judgment candidates are not file/session-scoped, and docs/retro/2026-05-24-1498- cascade.md flags convergence_stages.py as due for a refactor before a 4th stage, so a periodic loop keeps this bead's footprint out of that hot file entirely. Verification: devtools verify --quick (ruff format/check, mypy --strict, render all --check, topology/layering/docs-coverage/degrade-loudly lints) -- all green. Ref polylogue-6qjc Co-Authored-By: Claude <noreply@anthropic.com>
Adds tests/unit/daemon/test_judgment_automation.py: pure-function coverage
for the policy engine (parse_judgment_automation_policy, evaluate_candidate
accept/reject/escalate paths including the missing-policy and
undecided-confidence-band escalation reasons), plus real-user.db fixture
tests for run_judgment_automation_sweep_once proving it judges through the
actual judge_assertion_candidates storage chokepoint (queried back via a
JUDGMENT-kind row with the automation actor_ref) and writes a queryable
handoff assertion for the escalated residue, is idempotent across repeated
sweeps, and no-ops on an empty policy or missing user.db. Also covers the
periodic loop's dual capability gate (judgment_automation_enabled AND
mcp_judge_enabled) with a parametrized single-flag-set case proving the
write coordinator is never invoked unless both are true.
Fixed along the way: the escalation/judge actor ref used an unregistered
ObjectRef kind ("automation:...") that upsert_assertion's ref normalizer
rejects -- switched to the existing "actor" kind
(actor:judgment-automation).
Verification: devtools test tests/unit/daemon/test_judgment_automation.py
(14 passed), mypy --strict on the two touched files (no issues), and
devtools verify --quick end to end (ruff format/check, mypy, render-all,
topology/layering/docs-coverage/degrade-loudly) -- all green.
Ref polylogue-6qjc
Co-Authored-By: Claude <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
Next review available in: 33 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (8)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
|
Coordinator review stand-in (CodeRabbit rate-limited — no findings produced). Read the diff. Assessment: policy engine + periodic daemon loop (deliberately NOT a DaemonConverger stage, with the 1498-cascade retro cited — candidates aren't file/session-scoped and convergence_stages.py is flagged pre-refactor); dual fail-closed gate (judgment_automation_enabled AND mcp_judge_enabled, re-read every tick, ConfigError on typo rather than truthiness coercion); judgments go through the real judge_assertion_candidates chokepoint (anti-vacuity: test asserts the JUDGMENT-kind row with the automation actor_ref that only the shared chokepoint writes); undecidable residue escalates as queryable handoff assertions instead of silent limbo; config inventory entries + docs complete. Deferred items (no live policy wiring yet, filter-only escalation discovery) are honest and inside the bead's scope boundary. 183 focused tests, mypy --strict, verify --quick, quick-gate all green. Merging. |
Sinity
added a commit
that referenced
this pull request
Jul 20, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ
Sinity
added a commit
that referenced
this pull request
Jul 21, 2026
…#3239) ## Summary Implements the two named residuals from polylogue-1xc.12 (PR #3235), plus fixes a live pre-existing `messages_fts_identity` UNIQUE(block_id) regression reported mid-flight by another lane: 1. Pairs `storage/sqlite/archive_tiers/write.py`'s non-bulk full-session-replace fast path's `delete_session_rows_sql`/`insert_session_rows_sql` calls with the matching `delete_session_identity_rows_sql`/`insert_session_identity_rows_sql` companions (3 delete + 2 insert sites, plus the sync/async twin purge helpers in `storage/session_replacement.py`). 2. Adds `polylogue/daemon/fts_identity_convergence.py`: a periodic 24h-cadence recompute pass that is the recompute authority for `identity_mismatch_rows`, correcting the documented `daemon/fts_startup.py` bounded-STALE-write hazard. 3. Fixes a `messages_fts_identity` block_id UNIQUE-constraint self-heal gap: every identity-ledger write now uses `INSERT OR REPLACE`/multi-target UPSERT so a stale orphaned ledger row can never abort a later colliding write. ## Problem **AC2 residual (write.py):** `message_identity_mismatch_sql`'s docstring documented that write.py's fast path did not call the identity companions inline, leaving sessions written through the dominant real-world path (`write_parsed_session_to_archive`, `merge_append=False`) coverage-incomplete until the next repair backfilled them. **AC4 residual (periodic stage):** identity self-heal ran on every rebuild/repair/startup but no *scheduled* pass recomputed the exact snapshot on a quiet cadence, and `fts_startup.py`'s bounded STALE-write path could reset a recorded nonzero `identity_mismatch_rows` to 0 without recomputing it. **Live regression (mid-flight, coordinator-assigned):** another lane reported 6 pre-existing failures on unmodified master tied to `messages_fts_identity`'s `block_id TEXT NOT NULL UNIQUE` constraint, in `tests/unit/storage/test_revision_replay.py` and `tests/unit/sources/test_live_batch_support.py`. The exact production write-sequence was **not reproducible**: both files pass cleanly and repeatedly (serial, `-n auto` xdist, multiple runs) against this branch and a disposable fresh `origin/master` worktree — the only failure present in either is a pre-existing, unrelated `test_bundle_replay_respects_unconvertible_single_session_head` file-cleanup assertion, confirmed present on unmodified `origin/master` too. However, a synthetic repro built directly against the real trigger DDL (orphan a ledger row via a trigger-suspended delete, then insert a new block computing the same `block_id` at a different rowid) reliably reproduces `sqlite3.IntegrityError: UNIQUE constraint failed: messages_fts_identity.block_id` on current code, confirming a genuine invariant gap independent of the unreproduced call sequence: the ad/au trigger arms and every bulk delete companion only ever clean up ledger rows by `rowid`, never by `block_id`. ## Solution - write.py: added paired companion calls at all 3 delete + 2 insert call sites (fast-path scoped delete+reinsert, bulk-fts-guard delete+reinsert, trigger-missing purge), plus the sync/async twin in `session_replacement.py`. - `fts_identity_convergence.py`: standalone module (not a new `DaemonConverger`/`ConvergenceStage` — `docs/retro/2026-05-24-1498-cascade.md`'s explicit verdict is "refactor before adding a fourth stage"), following the `periodic_judgment_automation_sweep` (PR #3229) shape: a sync run-once function + async scheduling loop wired into `daemon/cli.py`'s `periodic_loops`, fixed 24h cadence (matching `_periodic_db_optimize`'s cost-class reasoning), no new config knob. - `messages_fts_identity` writes now use `INSERT OR REPLACE` (trigger bodies, `insert_all_message_identity_rows_sql`, `insert_session_identity_rows_sql`) and a second `ON CONFLICT(block_id)` clause on `repair_message_identity_rows_range_sql` (SQLite 3.35+ multi-target UPSERT; project already requires 3.43+ for FTS5 `contentless_delete`), so a pre-existing row blocking on either the `rowid` PK or the `block_id` UNIQUE constraint is evicted before the incoming row lands. ## Verification - `devtools test tests/unit/storage/test_fts_identity_ledger.py` → 19 passed (new `TestWritePathIdentityCompanions` + `TestIdentityLedgerBlockIdCollisionRepro` classes) - `devtools test tests/unit/daemon/test_fts_identity_convergence.py` → 4 passed - `devtools test tests/unit/storage/test_revision_replay.py tests/unit/sources/test_live_batch_support.py tests/unit/storage/test_fts_bloat_invariants.py tests/property/test_fts_identity_state_machine.py tests/unit/storage/test_archive_tiers_write.py tests/unit/storage/test_bulk_fts_prefix_reextract.py tests/unit/storage/test_bulk_build_lifecycle.py tests/unit/storage/test_lineage_normalization.py tests/unit/daemon/test_fts_identity_convergence.py` → 205 passed, 2 failed (pre-existing unrelated bundle-replay-cleanup assertion, confirmed present on unmodified `origin/master`) - `mypy --strict` on all touched/new modules → Success, no issues - `devtools render topology-projection && devtools render topology-status` (new module) → committed - `devtools render all --check` → no "out of sync" surfaces - `devtools verify --quick` → exit 0 - Anti-vacuity (both fixes): manually reverted the paired companion calls / the `INSERT OR REPLACE` fix and reran the corresponding tests → both failed as expected (missing-entry assertion / `sqlite3.IntegrityError`); restored and reran → both passed ## AC matrix | AC | Status | | --- | --- | | write.py identity companions paired at all call sites | Satisfied | | Test proving zero missing ledger entries after full-session-replace | Satisfied | | Periodic drift recompute stage, following #3229's pattern (not a new ConvergenceStage) | Satisfied | | Stage corrects the fts_startup.py bounded-write hazard | Satisfied | | Test: corrupt → recompute nonzero → repair → recompute zero | Satisfied | | Live UNIQUE(block_id) regression (coordinator-assigned) | Satisfied — fixed via INSERT OR REPLACE/multi-target UPSERT; original reported call sequence not reproducible after extensive attempts, but the underlying invariant gap is real and independently confirmed | Ref polylogue-miwv --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the missing automated actor for the MCP
judgedispatcher's candidatequeue: a policy engine plus a daemon periodic-loop trigger surface that
decides which
CANDIDATE-status assertions are auto-judgeable and calls thesame storage chokepoint the MCP
judgedispatcher uses, escalating theresidue to an explicit, queryable
handoffassertion.Problem
Per the polylogue-800m design note (PR #3202), the MCP
judgedispatcherlets an operator judge assertion candidates in bulk, but nothing calls it
automatically. Every agent-authored candidate (pathology findings, ontology/
transform candidates, findings, ...) defaults to waiting on an explicit
human decision, which the operator has said does not scale — most
agent-authored annotations will never be seen by a human.
Solution
polylogue/daemon/judgment_automation.py(new): a policy engine(
evaluate_candidate) applies per-kind confidence thresholds(
auto_accept_min_confidence/auto_reject_max_confidence) decoded from anew
[judgment_automation.policies]config table(
parse_judgment_automation_policy). A candidate is judged(
run_judgment_automation_sweep_once) via the realjudge_assertion_candidatesstorage chokepoint — the same one the MCPjudgedispatcher calls — when the policy is unambiguous; every othercandidate (no policy for its kind, no
confidencevalue, or a confidenceinside the undecided band) is escalated: a deterministic
handoff-kindassertion pointing at the candidate is upserted so the residue is a
queryable review queue, not silent limbo.
polylogue/daemon/cli.py: wiresperiodic_judgment_automation_sweepinto the existing
periodic_loopslist next to_periodic_db_optimize/periodic_embedding_backlog_check, following the established periodic-maintenance-loop pattern rather than a
DaemonConvergerstage — judgmentcandidates are not file/session-scoped, and
docs/retro/2026-05-24-1498-cascade.mdflagsconvergence_stages.pyasdue for a refactor before a 4th stage, so this keeps the bead's footprint
out of that hot file entirely.
polylogue/config.py: four new layered config keys —judgment_automation_enabled(fail-closed bool, off by default, followingthe
mcp_judge_enabledpattern),judgment_automation_interval_s,judgment_automation_batch_limit, and the rawjudgment_automation_policytable (decoded by the owning module, same shape as
health_convergence_debt).judgment_automation_enabledANDmcp_judge_enabled(polylogue-800m'sindependent capability boundary) are true — it exercises the same judge
write authority as the MCP dispatcher, so its own opt-in must not, by
itself, grant that authority. Both flags are re-read every tick
(
reload_behavior="daemon-loop"), so flipping either off inpolylogue.tomltakes effect on the next tick without a daemon restart.docs/configuration.md(config key table) anddocs/daemon.md(new "Judgment Automation" section with a worked
polylogue.tomlexample).Verification
devtools test tests/unit/daemon/test_judgment_automation.py tests/unit/core/test_config_inventory.py tests/unit/daemon/test_daemon_cli.py→183 passedmypy --strictonpolylogue/config.py polylogue/daemon/cli.py polylogue/daemon/judgment_automation.py tests/unit/daemon/test_judgment_automation.py→Success: no issues found in 4 source filesdevtools verify --quick(ruff format/check, mypy, render-all --check, topology/layering/closure-matrix/manifests/ci-workflows/doc-commands/docs-coverage/test-infra-currency/test-clock-hygiene/pytest-timeout-overrides/degrade-loudly) →exit_code: 0devtools verify(testmon-affected) /--seed-testmon— per-repo policy this is the default pre-PR baseline, but it was explicitly deferred for this lane in favor of the focused set above plus--quick.Residual risk
[judgment_automation.policies]entry yet in this PR — the policy tablestarts empty by default, so the sweep is a bounded no-op until an operator
configures at least one kind. This is intentional (config-gated OFF,
fail-closed on missing policy) but means the automation has no live
exercise beyond the test fixtures until an operator opts a real detector
kind in.
handoffassertion is discoverable only via an explicitkind=handofffilter today (not inASSERTION_CLAIM_KINDS's default set);wiring a dedicated MCP/CLI "review queue" surface for it is left as
follow-up rather than expanding this PR's scope.
Ref polylogue-6qjc