feat(daemon): converge whale raw components via bounded streaming escalation pass (polylogue-t93b)#3256
Conversation
…alation pass Problem: the daemon census permanently resource-blocks any raw-authority component whose aggregate payload exceeds the 64MiB fast-path envelope (daemon/cli.py:89) -- live witness codex:019f49d8 (788 raws, 6.33GB) has zero index presence because every pass logs "resource-blocked ... exceed replay limit" and moves on forever. Operator ruling: a daemon that owns raw->index convergence cannot have a permanent manual/offline requirement for whale-scale components (automagic-invariants doctrine, Ref polylogue-t93b). What changed: an escalation tier, not a blanket limit raise. - polylogue/daemon/cli.py: once the ordinary trickle conveyor is genuinely quiescent for a tick, `_maybe_run_raw_materialization_whale_pass` looks up one resource-blocked, entirely stream-safe component and runs a dedicated, `raw_artifact_id`-scoped pass at a widened envelope (`raw_authority_whale_payload_bytes`, default 8 GiB) through the writer coordinator, emitting `raw_materialization_whale_pass_started`/ `_completed` daemon events. Gated by `daemon_whale_raw_materialization` (on by default). - polylogue/storage/repair.py: `raw_materialization_whale_pass_candidate` is a read-only (mode=ro) selector reusing the existing fairness-ordered component/stream-safety machinery; `_raw_materialization_component_stream_safe` requires every member stream-record-safe. New `raw_materialization_non_stream_safe_oversized_count` metric complements the existing stream-oversized one. - polylogue/sources/revision_backfill.py: `record_resource_blocked_revision_census` now records whether the blocked component is stream-safe, so the durable census detail text distinguishes "escalation-eligible: stream-safe, awaiting a bounded daemon whale pass" from "escalation-blocked: non-stream-safe, requires manual/offline convergence" -- without changing the admission fingerprint identity. - polylogue/product/raw_authority.py: `repair_materialization`/new `whale_pass_candidate` product-boundary wrappers thread `raw_artifact_id` through to storage. - polylogue/config.py: `raw_authority_whale_payload_bytes` and `daemon_whale_raw_materialization` config entries (inventory + TOML/env layering + docs/configuration.md). No new streaming/memory-bounding machinery was needed: stream-record parse dispatch (`is_stream_record_provider`), the `RawParsePrefetchCache` inflight budget, the whale-residency parsed-tree spill tier (`_ParsedSessionSpill`, polylogue-odm1), and commit-batched replay (`raw_authority_commit_batch_size`, PR #3248) were already productized -- the resource-block gate was purely an admission check on `max_payload_bytes`, so widening it for one scoped, gated, quiescence-triggered pass is the whole fix. Verification: - `devtools test tests/unit/storage/test_repair.py tests/unit/daemon/test_daemon_cli.py tests/unit/sources/test_revision_backfill.py tests/unit/core/test_config_inventory.py tests/unit/core/test_config.py tests/unit/core/test_config_resolution_regression.py` -- 382 passed, 1 pre-existing unrelated failure (test_bulk_rebuild_routing_resumable_transaction_drives_pass_even_below_threshold, reproduced identically on unmodified master via `git stash`). - `devtools verify --quick` -- format/lint/mypy/render-all-check/topology/ layering/closure-matrix/manifests/ci-workflows/doc-commands/docs-coverage/ test-infra-currency/test-clock-hygiene/pytest-timeout-overrides/ degrade-loudly all green. - Anti-vacuity (removed and reverted, confirmed red-then-green): deleting the daemon's quiescence-triggered whale-pass call breaks both orchestration tests; forcing `raw_materialization_whale_pass_candidate` to always return None breaks both the candidate-selection and end-to-end convergence tests. Ref polylogue-t93b
|
Warning Review limit reached
Next review available in: 43 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (8)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f8b2bef91
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| payload={"seed_raw_id": candidate, "max_payload_bytes": whale_limit}, | ||
| ) | ||
| try: | ||
| result = await daemon_write_coordinator().run_sync( |
There was a problem hiding this comment.
Avoid holding the writer gate for the entire whale pass
For an eligible multi-gigabyte component, this run_sync call retains the process-wide DaemonWriteCoordinator lease across all parsing and replay work. raw_authority_commit_batch_size only commits the SQLite transactions inside that call; it never releases the coordinator lock, so new ingestion, HTTP writes, heartbeats, and every other daemon writer remain queued until the whole whale finishes—potentially hours for the cited 6.33 GB case. Move the read-only parse outside the writer lease or divide execution into separate coordinator acquisitions so the default-on escalation does not monopolize the sole writer.
AGENTS.md reference: AGENTS.md:L157-L159
Useful? React with 👍 / 👎.
| if not _raw_materialization_component_stream_safe(candidates, component): | ||
| continue |
There was a problem hiding this comment.
Load stream metadata for every expanded component member
When a candidate shares a source path or logical key with an already-materialized or terminal raw, authority_components includes that older raw, but _raw_materialization_candidate_ids populates raw_origins and raw_source_paths only for direct candidate rows. The all-member safety check therefore sees missing metadata for the expanded member, treats its provider as unknown, and skips an otherwise entirely stream-safe oversized component forever. Fetch origin/path metadata alongside expanded_blob_bytes for all expanded IDs before applying this check.
AGENTS.md reference: AGENTS.md:L139-L142
Useful? React with 👍 / 👎.
|
Coordinator review (CodeRabbit rate-limited): read the full 782-line diff. Conservative escalation orchestration — ordinary 64MiB fast path untouched; whale pass fires only on genuine trickle-conveyor quiescence (correctly NOT on bulk-rebuild/spool-pending deferral breaks), one component per tick, read-only candidate selection (oversized-under-ordinary AND within-whale-envelope AND every-member-stream-safe), routed through the daemon write coordinator with commit batching from #3248 bounding hold length, distinct typed reason + metrics for non-stream-safe oversized components, start/complete daemon events, config off-switch (default on per operator ruling). No new parse machinery — pure reuse of stream dispatch + prefetch/spill + batched replay. Two anti-vacuity mutations verified red. 1498-cascade shape respected (no new ConvergenceStage). Merging on green quick-gate; live witness resolution lands with the next daemon deploy and closes the t93b acceptance. |
… deploy, blocker critical path) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ
Summary
Adds an escalation tier so the daemon converges whale-scale raw-authority components (aggregate payload over the ordinary 64MiB fast-path envelope) instead of leaving them permanently resource-blocked. Live witness:
codex:019f49d8(788 raws, 6.33GB) has zero index presence because every ordinary daemon pass logs "resource-blocked ... exceed replay limit" and moves on, forever.Problem
_RAW_MATERIALIZATION_DAEMON_BLOB_LIMIT_BYTES(64MiB,daemon/cli.py:89) exists to bound writer-hold transaction length and parse memory during the daemon's ordinary raw→index convergence pass. It works correctly as a fast-path limit, but a component that exceeds it has no escalation path — the daemon owns raw→index convergence end to end, so a permanent manual/offline requirement for oversized components is a policy bug (operator ruling, automagic-invariants doctrine).The two concerns the limit protects already have productized answers elsewhere in the codebase: stream-record parse dispatch (
is_stream_record_provider, already routes Codex/Claude Code/Beads/Hermes throughparse_stream_payload), theRawParsePrefetchCacheinflight-byte budget, a dedicated whale-residency parsed-tree spill tier (_ParsedSessionSpill, polylogue-odm1), and commit-batched replay (raw_authority_commit_batch_size, PR #3248). The resource-block gate itself is purely an admission check onmax_payload_bytes— nothing else in the pipeline needed to change.Solution
Escalation tier, not a blanket limit raise:
polylogue/daemon/cli.py):_periodic_raw_materialization_convergencenow tracks whether the ordinary trickle conveyor reached genuine quiescence this tick (no remaining candidates, or no progress — never on bulk-rebuild-in-flight or browser-capture-spool-pending breaks, which are deferrals for unrelated reasons). On quiescence,_maybe_run_raw_materialization_whale_passlooks up one resource-blocked, entirely stream-safe component via a new read-only selector and runs a dedicated pass for it, scoped byraw_artifact_idat a widened envelope (raw_authority_whale_payload_bytes, default 8 GiB), through the same writer coordinator. Emitsraw_materialization_whale_pass_started/_completeddaemon events. Gated bydaemon_whale_raw_materialization(on by default — the whole point is the daemon self-heals codex:019f49d8 without an operator flipping a switch).polylogue/storage/repair.pyadds_raw_materialization_component_stream_safe(all members) andraw_materialization_whale_pass_candidate(fairness-ordered, read-onlymode=ro, reuses the existing component/fairness machinery — never selects a component with any non-stream-safe member). Newraw_materialization_non_stream_safe_oversized_countmetric complements the existingraw_materialization_stream_oversized_count.polylogue/sources/revision_backfill.py'srecord_resource_blocked_revision_censusnow records whether the blocked component is stream-safe, so the durable per-raw census detail text says"escalation-eligible: stream-safe, awaiting a bounded daemon whale pass"vs"escalation-blocked: non-stream-safe, requires manual/offline convergence"— without touching the admission fingerprint identity (re-admission-on-wider-envelope invariant preserved).polylogue/product/raw_authority.py'srepair_materialization/newwhale_pass_candidatewrappers threadraw_artifact_idthrough to storage, keeping the daemon offpolylogue.storage.repairinternals for this call.raw_authority_whale_payload_bytes(escalation envelope) anddaemon_whale_raw_materialization(on/off switch) added to the config inventory, TOML/env layering, anddocs/configuration.md.Alternatives rejected
Verification
Anti-vacuity (mutated production code, confirmed red, reverted, confirmed green again):
if quiescent and not ...: await _maybe_run_raw_materialization_whale_pass()) breakstest_periodic_raw_materialization_convergence_schedules_whale_pass_on_quiescenceandtest_periodic_raw_materialization_convergence_skips_whale_pass_mid_burst.raw_materialization_whale_pass_candidateto always returnNonebreakstest_raw_materialization_whale_pass_candidate_selects_stream_safe_blocked_componentandtest_raw_materialization_whale_pass_converges_blocked_component_to_resolved_head.New regression tests build a synthetic multi-raw "whale" fixture (
tests/infra/revision_backfill_benchmark.build_revision_chain_corpus— a growing-file revision chain, the exact shape of the live codex:019f49d8 witness) exceeding a lowered test envelope, and prove: the ordinary pass blocks it; the whale pass converges it to a resolved head (sessionsrow materialized inindex.db); commit-batch size genuinely controls writer-hold granularity (test_raw_materialization_whale_pass_commit_batches_bounded); and a non-stream-safe oversized component gets the distinct typed census reason (test_raw_materialization_ordinary_pass_census_detail_distinguishes_escalation_eligibility).Not run: full
devtools verify(broad local gate) — the touched-file targeted run above plus--quickcover the changed surface; a full run was not required for this scope. Live-archive verification ofcodex:019f49d8itself is out of scope for this PR (would touch the live archive, which agent work must never do) — it is the deploy-time acceptance witness per the bead.AC matrix (polylogue-t93b)
repair_raw_materializationentrypoint,raw_artifact_id-scopedraw_authority_commit_batch_sizeunchanged, threaded through unmodified; test proves finer batch size yields strictly more commit boundariesis_stream_record_providerdispatch +RawParsePrefetchCache/_ParsedSessionSpillwhale tier; this PR adds no new parse codeescalation-eligible/escalation-blockeddurable census detail text +raw_materialization_non_stream_safe_oversized_countmetrictests/unit/storage/test_repair.py(5 new tests) +tests/unit/daemon/test_daemon_cli.py(5 new tests)daemon_whale_raw_materializationdefaults on, so no operator action is needed; not verified against the live archive in this PR by design (agent work never touches the live archive)raw_materialization_whale_pass_started/_completedevents viaemit_daemon_event, testedRef polylogue-t93b