Skip to content

feat(daemon): converge whale raw components via bounded streaming escalation pass (polylogue-t93b)#3256

Merged
Sinity merged 1 commit into
masterfrom
feature/daemon/t93b-whale-component-convergence
Jul 21, 2026
Merged

feat(daemon): converge whale raw components via bounded streaming escalation pass (polylogue-t93b)#3256
Sinity merged 1 commit into
masterfrom
feature/daemon/t93b-whale-component-convergence

Conversation

@Sinity

@Sinity Sinity commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Summary

Adds an escalation tier so the daemon converges whale-scale raw-authority components (aggregate payload over the ordinary 64MiB fast-path envelope) instead of leaving them permanently resource-blocked. Live witness: codex:019f49d8 (788 raws, 6.33GB) has zero index presence because every ordinary daemon pass logs "resource-blocked ... exceed replay limit" and moves on, forever.

Problem

_RAW_MATERIALIZATION_DAEMON_BLOB_LIMIT_BYTES (64MiB, daemon/cli.py:89) exists to bound writer-hold transaction length and parse memory during the daemon's ordinary raw→index convergence pass. It works correctly as a fast-path limit, but a component that exceeds it has no escalation path — the daemon owns raw→index convergence end to end, so a permanent manual/offline requirement for oversized components is a policy bug (operator ruling, automagic-invariants doctrine).

The two concerns the limit protects already have productized answers elsewhere in the codebase: stream-record parse dispatch (is_stream_record_provider, already routes Codex/Claude Code/Beads/Hermes through parse_stream_payload), the RawParsePrefetchCache inflight-byte budget, a dedicated whale-residency parsed-tree spill tier (_ParsedSessionSpill, polylogue-odm1), and commit-batched replay (raw_authority_commit_batch_size, PR #3248). The resource-block gate itself is purely an admission check on max_payload_bytes — nothing else in the pipeline needed to change.

Solution

Escalation tier, not a blanket limit raise:

  1. Ordinary fast-path limit unchanged. The 64MiB envelope still governs every regular daemon pass.
  2. Quiescence-triggered whale pass (polylogue/daemon/cli.py): _periodic_raw_materialization_convergence now tracks whether the ordinary trickle conveyor reached genuine quiescence this tick (no remaining candidates, or no progress — never on bulk-rebuild-in-flight or browser-capture-spool-pending breaks, which are deferrals for unrelated reasons). On quiescence, _maybe_run_raw_materialization_whale_pass looks up one resource-blocked, entirely stream-safe component via a new read-only selector and runs a dedicated pass for it, scoped by raw_artifact_id at a widened envelope (raw_authority_whale_payload_bytes, default 8 GiB), through the same writer coordinator. Emits raw_materialization_whale_pass_started/_completed daemon events. Gated by daemon_whale_raw_materialization (on by default — the whole point is the daemon self-heals codex:019f49d8 without an operator flipping a switch).
  3. Non-stream-safe components stay blocked, distinctly. polylogue/storage/repair.py adds _raw_materialization_component_stream_safe (all members) and raw_materialization_whale_pass_candidate (fairness-ordered, read-only mode=ro, reuses the existing component/fairness machinery — never selects a component with any non-stream-safe member). New raw_materialization_non_stream_safe_oversized_count metric complements the existing raw_materialization_stream_oversized_count. polylogue/sources/revision_backfill.py's record_resource_blocked_revision_census now records whether the blocked component is stream-safe, so the durable per-raw census detail text says "escalation-eligible: stream-safe, awaiting a bounded daemon whale pass" vs "escalation-blocked: non-stream-safe, requires manual/offline convergence" — without touching the admission fingerprint identity (re-admission-on-wider-envelope invariant preserved).
  4. Product boundary. polylogue/product/raw_authority.py's repair_materialization/new whale_pass_candidate wrappers thread raw_artifact_id through to storage, keeping the daemon off polylogue.storage.repair internals for this call.
  5. Config surface. raw_authority_whale_payload_bytes (escalation envelope) and daemon_whale_raw_materialization (on/off switch) added to the config inventory, TOML/env layering, and docs/configuration.md.

Alternatives rejected

  • Raising the ordinary 64MiB limit globally — bead explicitly rules this out ("escalation tier, not a blanket limit raise"); it would remove the writer-hold/parse-memory bound for every pass, not just whale components.
  • Building new streaming/memory-bounding machinery — unnecessary; the codebase already has it (see Problem section), so the fix is pure orchestration (when to widen the envelope, for what, gated how).

Verification

devtools test tests/unit/storage/test_repair.py tests/unit/daemon/test_daemon_cli.py \
  tests/unit/sources/test_revision_backfill.py tests/unit/core/test_config_inventory.py \
  tests/unit/core/test_config.py tests/unit/core/test_config_resolution_regression.py
# 382 passed, 1 pre-existing unrelated failure
# (test_bulk_rebuild_routing_resumable_transaction_drives_pass_even_below_threshold,
#  reproduced identically on unmodified master via `git stash`)

devtools verify --quick
# format/lint/mypy/render-all-check/topology/layering/closure-matrix/manifests/
# ci-workflows/doc-commands/docs-coverage/test-infra-currency/test-clock-hygiene/
# pytest-timeout-overrides/degrade-loudly all green

Anti-vacuity (mutated production code, confirmed red, reverted, confirmed green again):

  • Deleting the daemon's quiescence-triggered whale-pass call (if quiescent and not ...: await _maybe_run_raw_materialization_whale_pass()) breaks test_periodic_raw_materialization_convergence_schedules_whale_pass_on_quiescence and test_periodic_raw_materialization_convergence_skips_whale_pass_mid_burst.
  • Forcing raw_materialization_whale_pass_candidate to always return None breaks test_raw_materialization_whale_pass_candidate_selects_stream_safe_blocked_component and test_raw_materialization_whale_pass_converges_blocked_component_to_resolved_head.

New regression tests build a synthetic multi-raw "whale" fixture (tests/infra/revision_backfill_benchmark.build_revision_chain_corpus — a growing-file revision chain, the exact shape of the live codex:019f49d8 witness) exceeding a lowered test envelope, and prove: the ordinary pass blocks it; the whale pass converges it to a resolved head (sessions row materialized in index.db); commit-batch size genuinely controls writer-hold granularity (test_raw_materialization_whale_pass_commit_batches_bounded); and a non-stream-safe oversized component gets the distinct typed census reason (test_raw_materialization_ordinary_pass_census_detail_distinguishes_escalation_eligibility).

Not run: full devtools verify (broad local gate) — the touched-file targeted run above plus --quick cover the changed surface; a full run was not required for this scope. Live-archive verification of codex:019f49d8 itself is out of scope for this PR (would touch the live archive, which agent work must never do) — it is the deploy-time acceptance witness per the bead.

AC matrix (polylogue-t93b)

Acceptance criterion Status
Component whose bytes exceed the daemon limit but is fully stream-safe converges through the daemon (no offline pass) Satisfied — quiescence-triggered whale pass, same repair_raw_materialization entrypoint, raw_artifact_id-scoped
Writer-hold bounded (commit batches) Satisfied — raw_authority_commit_batch_size unchanged, threaded through unmodified; test proves finer batch size yields strictly more commit boundaries
Memory bounded (streaming parse + inflight budget) Satisfied — inherited unmodified from existing is_stream_record_provider dispatch + RawParsePrefetchCache/_ParsedSessionSpill whale tier; this PR adds no new parse code
Non-stream-safe oversized components get a distinct typed blocked reason Satisfied — new escalation-eligible/escalation-blocked durable census detail text + raw_materialization_non_stream_safe_oversized_count metric
Regression test with synthetic whale fixture Satisfied — tests/unit/storage/test_repair.py (5 new tests) + tests/unit/daemon/test_daemon_cli.py (5 new tests)
Live witness codex:019f49d8 resolves after deploy Deferred to deploy — daemon_whale_raw_materialization defaults on, so no operator action is needed; not verified against the live archive in this PR by design (agent work never touches the live archive)
Daemon event receipts recorded Satisfied — raw_materialization_whale_pass_started/_completed events via emit_daemon_event, tested

Ref polylogue-t93b

…alation pass

Problem: the daemon census permanently resource-blocks any raw-authority
component whose aggregate payload exceeds the 64MiB fast-path envelope
(daemon/cli.py:89) -- live witness codex:019f49d8 (788 raws, 6.33GB) has
zero index presence because every pass logs "resource-blocked ... exceed
replay limit" and moves on forever. Operator ruling: a daemon that owns
raw->index convergence cannot have a permanent manual/offline requirement
for whale-scale components (automagic-invariants doctrine, Ref polylogue-t93b).

What changed: an escalation tier, not a blanket limit raise.
- polylogue/daemon/cli.py: once the ordinary trickle conveyor is genuinely
  quiescent for a tick, `_maybe_run_raw_materialization_whale_pass` looks
  up one resource-blocked, entirely stream-safe component and runs a
  dedicated, `raw_artifact_id`-scoped pass at a widened envelope
  (`raw_authority_whale_payload_bytes`, default 8 GiB) through the writer
  coordinator, emitting `raw_materialization_whale_pass_started`/
  `_completed` daemon events. Gated by `daemon_whale_raw_materialization`
  (on by default).
- polylogue/storage/repair.py: `raw_materialization_whale_pass_candidate`
  is a read-only (mode=ro) selector reusing the existing fairness-ordered
  component/stream-safety machinery; `_raw_materialization_component_stream_safe`
  requires every member stream-record-safe. New
  `raw_materialization_non_stream_safe_oversized_count` metric complements
  the existing stream-oversized one.
- polylogue/sources/revision_backfill.py: `record_resource_blocked_revision_census`
  now records whether the blocked component is stream-safe, so the durable
  census detail text distinguishes "escalation-eligible: stream-safe,
  awaiting a bounded daemon whale pass" from "escalation-blocked:
  non-stream-safe, requires manual/offline convergence" -- without
  changing the admission fingerprint identity.
- polylogue/product/raw_authority.py: `repair_materialization`/new
  `whale_pass_candidate` product-boundary wrappers thread `raw_artifact_id`
  through to storage.
- polylogue/config.py: `raw_authority_whale_payload_bytes` and
  `daemon_whale_raw_materialization` config entries (inventory + TOML/env
  layering + docs/configuration.md).

No new streaming/memory-bounding machinery was needed: stream-record
parse dispatch (`is_stream_record_provider`), the `RawParsePrefetchCache`
inflight budget, the whale-residency parsed-tree spill tier
(`_ParsedSessionSpill`, polylogue-odm1), and commit-batched replay
(`raw_authority_commit_batch_size`, PR #3248) were already productized --
the resource-block gate was purely an admission check on `max_payload_bytes`,
so widening it for one scoped, gated, quiescence-triggered pass is the
whole fix.

Verification:
- `devtools test tests/unit/storage/test_repair.py tests/unit/daemon/test_daemon_cli.py
  tests/unit/sources/test_revision_backfill.py tests/unit/core/test_config_inventory.py
  tests/unit/core/test_config.py tests/unit/core/test_config_resolution_regression.py`
  -- 382 passed, 1 pre-existing unrelated failure
  (test_bulk_rebuild_routing_resumable_transaction_drives_pass_even_below_threshold,
  reproduced identically on unmodified master via `git stash`).
- `devtools verify --quick` -- format/lint/mypy/render-all-check/topology/
  layering/closure-matrix/manifests/ci-workflows/doc-commands/docs-coverage/
  test-infra-currency/test-clock-hygiene/pytest-timeout-overrides/
  degrade-loudly all green.
- Anti-vacuity (removed and reverted, confirmed red-then-green): deleting the
  daemon's quiescence-triggered whale-pass call breaks both orchestration
  tests; forcing `raw_materialization_whale_pass_candidate` to always
  return None breaks both the candidate-selection and end-to-end
  convergence tests.

Ref polylogue-t93b
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@Sinity, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 43 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 82009eca-f276-47e3-bd78-64503a5c0202

📥 Commits

Reviewing files that changed from the base of the PR and between 2f3aa34 and 9f8b2be.

📒 Files selected for processing (8)
  • docs/configuration.md
  • polylogue/config.py
  • polylogue/daemon/cli.py
  • polylogue/product/raw_authority.py
  • polylogue/sources/revision_backfill.py
  • polylogue/storage/repair.py
  • tests/unit/daemon/test_daemon_cli.py
  • tests/unit/storage/test_repair.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/daemon/t93b-whale-component-convergence

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f8b2bef91

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread polylogue/daemon/cli.py
payload={"seed_raw_id": candidate, "max_payload_bytes": whale_limit},
)
try:
result = await daemon_write_coordinator().run_sync(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid holding the writer gate for the entire whale pass

For an eligible multi-gigabyte component, this run_sync call retains the process-wide DaemonWriteCoordinator lease across all parsing and replay work. raw_authority_commit_batch_size only commits the SQLite transactions inside that call; it never releases the coordinator lock, so new ingestion, HTTP writes, heartbeats, and every other daemon writer remain queued until the whole whale finishes—potentially hours for the cited 6.33 GB case. Move the read-only parse outside the writer lease or divide execution into separate coordinator acquisitions so the default-on escalation does not monopolize the sole writer.

AGENTS.md reference: AGENTS.md:L157-L159

Useful? React with 👍 / 👎.

Comment on lines +4093 to +4094
if not _raw_materialization_component_stream_safe(candidates, component):
continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Load stream metadata for every expanded component member

When a candidate shares a source path or logical key with an already-materialized or terminal raw, authority_components includes that older raw, but _raw_materialization_candidate_ids populates raw_origins and raw_source_paths only for direct candidate rows. The all-member safety check therefore sees missing metadata for the expanded member, treats its provider as unknown, and skips an otherwise entirely stream-safe oversized component forever. Fetch origin/path metadata alongside expanded_blob_bytes for all expanded IDs before applying this check.

AGENTS.md reference: AGENTS.md:L139-L142

Useful? React with 👍 / 👎.

@Sinity

Sinity commented Jul 21, 2026

Copy link
Copy Markdown
Owner Author

Coordinator review (CodeRabbit rate-limited): read the full 782-line diff. Conservative escalation orchestration — ordinary 64MiB fast path untouched; whale pass fires only on genuine trickle-conveyor quiescence (correctly NOT on bulk-rebuild/spool-pending deferral breaks), one component per tick, read-only candidate selection (oversized-under-ordinary AND within-whale-envelope AND every-member-stream-safe), routed through the daemon write coordinator with commit batching from #3248 bounding hold length, distinct typed reason + metrics for non-stream-safe oversized components, start/complete daemon events, config off-switch (default on per operator ruling). No new parse machinery — pure reuse of stream dispatch + prefetch/spill + batched replay. Two anti-vacuity mutations verified red. 1498-cascade shape respected (no new ConvergenceStage). Merging on green quick-gate; live witness resolution lands with the next daemon deploy and closes the t93b acceptance.

@Sinity
Sinity merged commit 5202a91 into master Jul 21, 2026
3 checks passed
@Sinity
Sinity deleted the feature/daemon/t93b-whale-component-convergence branch July 21, 2026 22:02
Sinity added a commit that referenced this pull request Jul 21, 2026
… deploy, blocker critical path)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUsH3Rhq6oAZpYPWcsZqnZ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant