Skip to content

perf(daemon): run trace normalization on blocking workers - #2031

Open
svarlamov wants to merge 1 commit into
mainfrom
fix/trace-normalization-off-ingest-task
Open

perf(daemon): run trace normalization on blocking workers#2031
svarlamov wants to merge 1 commit into
mainfrom
fix/trace-normalization-off-ingest-task

Conversation

@svarlamov

@svarlamov svarlamov commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

  • Trace normalization can synchronously read repository and Git configuration state, and orphan sweeps inspect repository state. Running those operations inline can block a Tokio runtime worker needed by listener, control, and family tasks.
  • Run normalization and orphan sweeps with spawn_blocking_result.
  • Preserve exact trace ordering: the ingest task still awaits every blocking result before consuming the next payload.
  • Move the owned trace payload into the blocking closure, avoiding a new per-payload clone on the critical ingestion path.
  • Recover poisoned normalizer mutex guards so one caught per-payload panic cannot permanently disable normalization.

TDD / review validation

  • Added a real-TestRepo regression using a one-worker test daemon and a FIFO-backed repository config. While the real normalizer is blocked on a cold alias-config read, status.family must remain responsive; after release, a subsequent trace must normalize and complete.
  • Verified the test fails on the parent implementation with the control request timing out after 500 ms, and passes on this layer. It also passed five consecutive stability runs.
  • Retained the earlier Devin-requested poisoned-mutex recovery.
  • The two-TestRepo cross-family synchronization regression in perf(daemon): scope sync.family's trace fence to the requested family #2030 and all 24 trace-normalizer tests pass on the stack tip.
  • Full task test (2155 library tests, 77 daemon-mode tests, 3264 integration tests, and remaining test binaries/doc tests), task lint, task fmt, and git diff --check are green locally.

The test-only worker-count override is read once when constructing a test-support daemon runtime; it adds no production or ingestion-path work.

This re-lands the ingest-offload portion of #1981 on the current architecture; actor side-effect offloading already landed through #1996/#2019.

Depends on #2030.

svarlamov commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

devin-ai-integration[bot]

This comment was marked as resolved.

@svarlamov
svarlamov force-pushed the fix/trace-normalization-off-ingest-task branch 2 times, most recently from 2c0767f to b6f777e Compare August 2, 2026 22:00
@svarlamov
svarlamov force-pushed the fix/per-family-trace-fences branch from 4888713 to 52c908e Compare August 2, 2026 22:00
devin-ai-integration[bot]

This comment was marked as resolved.

Trace normalization can synchronously read repository and Git configuration state, while orphan sweeps inspect repository state. Running that work inline on the trace-ingest future can block a Tokio runtime worker needed by control, listener, and family tasks.

Run both operations with spawn_blocking_result. The ingest worker still awaits each result to preserve exact payload ordering, while the runtime worker remains available for async work. Move the owned trace payload into the blocking closure so offloading adds no payload clone on the critical path.

Protect the shared normalizer with a standard mutex and recover poisoned guards, preserving the ingest worker panic-recovery behavior.

Re-lands the ingest-offload portion of #1981 on the current architecture; the family/global side-effect offload already landed via #1996/#2019.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@svarlamov
svarlamov force-pushed the fix/trace-normalization-off-ingest-task branch from b6f777e to 6e9177d Compare August 2, 2026 22:18
Base automatically changed from fix/per-family-trace-fences to main August 3, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant