feat: capture TXE server logs in CI and hexdump IPC response frames on decode errors - #25160
Open
charlielye wants to merge 1 commit into
Open
feat: capture TXE server logs in CI and hexdump IPC response frames on decode errors#25160charlielye wants to merge 1 commit into
charlielye wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A flaky noir-contracts TXE failure (
Expected size in TreeStateReference deserialization) proved undiagnosable after the fact, for two independent reasons:start_txesran the TXE underdump_fail, which only emits captured output when the process itself exits non-zero — and the TXE survives to a clean shutdown even when tests against it fail. Every wsdb spawn/exit line, IPC warning, and stack trace from the flaky run was silently dropped.What
TXE logs are now captured like the test engine's (
bootstrap.sh):start_txes→start_txe: starts the single TXE viasetsid color_prefix "txe" "denoise …", so its full output lives in its own CI log (live-published, persisted on shutdown —denoisefinalizes the log on SIGTERM). The oracle test resolver gets the same treatment.stop_txes→stop_txe: kills each setsid'd process group so node dies with its wrappers.NUM_TXESis removed — only one TXE is ever run. The aztec-nrtest_cmdsround-robin port assignment collapses to the fixed port.Generated TS clients hexdump the offending frame on decode errors (
ipc-codegen/src/typescript_codegen.ts, both Async and Sync APIs):msgpackCallnow returns{ decoded, raw }; msgpack decode failures, variant-name mismatches, and body-shape (to*) failures all rethrow with a bounded hexdump appended (first 4096 bytes + total length). The original error object is mutated rather than wrapped, preserving its type and stack.createError(server-reported error) branch deliberately bypasses the dump — those are well-formed frames whose message already says everything, and wrapping would break customIpcErrorFactoryerror types.Example of what a failure now looks like:
One glance distinguishes a mispaired frame (wrong variant) from corruption (arity/bytes) from truncation — the classification that took days of archaeology for the original flake.
Validation
bash -non both bootstrap scripts; repo-wide grep shows no remainingNUM_TXESreferences.AsyncApiandSyncApi, with the formats shown above.Note: the wsdb/avm TS packages consumed from npm pick up the client-side change on their next regeneration/publish from the labs pipeline; this PR changes the generator and the in-repo TXE/CI plumbing.