fix: IPC reactor delivered stale responses to recycled client slots and died of SIGPIPE

[charlielye]

The bug

ipc::IpcServer::run_reactor() keeps per-connection response-ordering state (arrival-order sequence counter + reorder stash) keyed by the transport's client slot id. SocketServer recycles slot ids (find_free_slot() returns the lowest freed slot) and nothing cleared the reactor's state when a connection ended. A connection dying with requests still in flight — routine in production: bb-avm-sim processes are killed on cancellation and teardown while their wsdb requests are outstanding — leaves "zombie" sequence entries on its slot, with two timing-dependent failure modes:

1. Response misdelivery. If a new connection is accepted onto the freed slot before the zombie completes, the new connection's first response is stashed behind the zombie's sequence number, and the zombie's late completion is released to the new connection as its first frame. TS clients correlate responses positionally (no request-id envelope — the reactor's FIFO release is the correctness contract), so a single leaked frame shifts every subsequent response onto the wrong caller: wrong-type decodes or right-type values for the wrong parameters.
2. Whole-server SIGPIPE death. If the zombie completes while the slot's fd is closed but not yet recycled, the reactor writes the response to the dead peer. send() passed no MSG_NOSIGNAL and nothing ignored SIGPIPE, so the write killed the entire server process — taking down every other client, including ones that never disconnected. Deterministically reproducible: any client that pipelines reads over UDS to aztec-wsdb and destroys its connection mid-flight killed the server within milliseconds (aztec-wsdb exited unexpectedly (code=null, signal=SIGPIPE)).

The fix

• IpcServer::drain_disconnected_clients() (new hook): transports report client ids whose connection ended since the last call. SocketServer records them in disconnect_client() (reactor-thread only, so no locking). The reactor polls the hook at the top of every iteration — before accept() can recycle a slot — and resets that slot's sequence counter and stash.
• Generation check in respond(): each request captures its connection's generation; a late respond() from a previous generation is dropped under the stash lock instead of re-creating state for the slot's next occupant. This closes the window where a zombie completes after cleanup.
• MSG_NOSIGNAL on SocketServer::send() (Linux) and SO_NOSIGPIPE on accepted fds (macOS), plus SIGPIPE → SIG_IGN in install_default_signal_handlers(): a write that races a disconnect yields EPIPE (already handled — the send loop disconnects the client), never a process-killing signal.
• UdsIpcClient retries ECONNRESET on connect (TS): under connection churn a connect can race the server's accept loop and get reset; previously only ECONNREFUSED/ENOENT/ETIMEDOUT were retried, so a transient reset surfaced as a hard connect failure.

Tests

C++ (ipc-runtime, socket.test.cpp) — both were RED before the fix:

• ReactorSlotReuseDoesNotLeakStaleResponses: scripted-completion-order repro of failure mode 1; verifies via a client-id echo that the second connection really landed on the first one's slot, so a scheduling race cannot false-pass. Was RED 20/20 (first frame carried the dead connection's payload + an extra leaked frame); now GREEN, 10× repeats.
• ReactorSlotReuseCleanHandoverControl: clean-handover control pinning the invariant.
• ReactorSurvivesResponseToDeadClient: reactor drops/fails responses to a dead client without dying. Note in-process writes to a just-closed peer can be absorbed by kernel buffering, so this alone cannot prove SIGPIPE immunity — hence the cross-process test below.

TS (yarn-project/world-state) — run against the rebuilt aztec-wsdb:

• wsdb_sigpipe_death.test.ts: cross-process guard for failure mode 2. A long-lived monitor connection must keep reading correct answers through 20 rounds of an unrelated peer pipelining ~400 reads and destroying its connection mid-flight. Deterministically RED against the unfixed binary (server dead in ~12 ms, monitor gets read ECONNRESET); GREEN with the fix.
• ipc_churn_correlation.test.ts: correlation load test where every response must prove it belongs to its own request by value, not just by type. Each connection plants a private fork with leaves derived from its own seed — connections share no observable state, so a cross-connection swap of same-type responses (invisible to the existing shared-state tests, where identical requests have identical answers) fails on wrong index/root/size. Legs: C sequential sanity; A single-connection pipelined reads + per-fork writes (reorder-stash pressure, no disconnects); D multi-connection read/write soak with no churn — readers assert recorded roots and exact leaf indices, writers pipeline append→read-after-write with exact-index asserts; B = D's workload plus a rotating mid-flight destroy + replace (slot recycling). WSDB_SOAK_MS extends D/B for grinding sessions (default ~4s for CI; 30s soak run clean).

Leg D is deliberately independent of this PR's fixes: it passes against the unfixed binary too (while Leg B fails there in ~56 ms), so it discriminates the disconnect-cleanup bug class from any other IPC/reorder/scheduler defect — a failure in D on any binary is a distinct bug.

Full ipc_runtime_tests suite passes (19/19); existing ipc_pipelined_read_correlation.test.ts passes against the rebuilt binary.

Context and follow-ups

Found while investigating a flaky noir-contracts TXE failure (Expected size in TreeStateReference deserialization — a positionally-mispaired wsdb response). These fixes remove the only proven server-side sources of stale/mispaired frames; whether that flake's exact delivery path is fully explained is still under investigation. Known follow-ups, deliberately out of scope here:

• The MPSC-SHM transport has the analogous slot-reuse gap (per-slot response rings are not reset when a client detaches and its slot is reclaimed). Slot claim there is client-driven, so it needs a generation/handshake mechanism rather than the socket transport's reactor-thread disconnect list.
• TXE server output is discarded unless the process exits non-zero (dump_fail in start_txes), which is why the original flake left no server-side trace; teeing it to a persistent log would make the next occurrence diagnosable.
• The TS clients' "response with no pending caller" path still only console.warns; upgrading it to fail loudly is a small separate change to the ipc-codegen templates.