Skip to content

Debug Tooling #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SeanCheatham wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Debug Tooling #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c010938
Add core-dump and boost::stacktrace crash tooling across all xrpld nodes
SeanCheatham Jul 11, 2026
40a12d6
Integrate GDB indexes into xrpld image
SeanCheatham Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,3 +130,9 @@ Caveat: `sometimes(success)` + `conf_mpt_version_monotonic` only fire against an

### Randomness & specs
All randomness via `workload.randoms` (`AntithesisRandom`); generators in `params.py`, never hardcode. Tx docs: `xrpl.org/docs/references/protocol/transactions/types/<name>`; specs: `github.com/XRPLF/XRPL-Standards` `XLS-NNNN-<name>/`.

## Crash diagnostics (xrpld image)

`xrpld` is an unstripped Debug build with full DWARF (`/symbols/xrpld`). Debug a crash in an MVD session: rewind to the crash moment and attach gdb to the live replayed process (`thread apply all bt`), or `gcore` it. No on-disk core is needed: the replay is the source of truth, Antithesis captures cores externally, and in-guest `core_pattern` isn't reachable.

Gotcha: `gdb-add-index` (`Dockerfile.xrpld`, runtime stage) embeds a `.gdb_index` into the binaries at build time. Without it gdb rebuilds its symbol table by scanning ~600 MB of DWARF at every startup (minutes), overrunning the MVD command window; with it gdb loads in seconds. `binutils` provides the `objcopy` it uses.
16 changes: 15 additions & 1 deletion Dockerfile.xrpld
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,21 @@ RUN mkdir -p /symbols && \
ln -s /opt/fuzzer/bin/rippled-fuzzer /symbols/rippled-fuzzer

ENV DEBIAN_FRONTEND=noninteractive
RUN set -ex; apt-get update && apt-get install --yes curl jq procps gdb iproute2 && rm -rf /var/lib/apt/lists/*
# gdb opens a core / attaches to a live process to produce a symbolized backtrace
# in MVD; binutils supplies objcopy (used by gdb-add-index below) plus objdump/
# readelf as gdb companions. The xrpld binary ships full DWARF + symtab (unstripped
# Debug build), so no separate symbol package is needed.
RUN set -ex; apt-get update && apt-get install --yes curl jq procps gdb binutils iproute2 && rm -rf /var/lib/apt/lists/*

# Embed a .gdb_index in the binaries (gdb-add-index ships with gdb; it uses objcopy
# from binutils). Without it, gdb rebuilds its symbol table by scanning ~600 MB of
# DWARF at startup -- significant CPU and memory that, on a resource-constrained MVD

@vvysokikh1 vvysokikh1 Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says this "(should)" fix the timeouts — since the whole change is a performance one, worth landing a measured number in the comment rather than "minutes → seconds":

time gdb -batch -nx -ex 'file /opt/xrpld/bin/xrpld' -ex 'thread apply all bt' -ex quit

Also: clang 22 defaults to DWARF 5, and gdb-add-index --dwarf-5 emits the native .debug_names accelerator instead of .gdb_index. On DWARF-5 input that's often the faster of the two, so it seems worth A/B-ing before settling on the default.

# VM, can overrun the debug-session command window; the index lets gdb jump straight
# to the units it needs. One-time build cost (the scan happens here); the /symbols/*
# links resolve to these same files.
RUN set -ex; \

@vvysokikh1 vvysokikh1 Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This RUN rewrites files that arrived via COPY --from=build, so overlayfs stores a full second copy of both binaries in the new layer while the originals stay in the lower layer. That's a couple of gigabytes at this point.

Doing the indexing in an intermediate stage keeps one copy in the final image.

gdb-add-index /opt/xrpld/bin/xrpld; \
gdb-add-index /opt/fuzzer/bin/rippled-fuzzer

@vvysokikh1 vvysokikh1 Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gdb-add-index exits 0 when it produces no index, so set -e won't catch a failure.

maybe worth adding additional checks here?

RUN set -ex; \
>     gdb-add-index /opt/xrpld/bin/xrpld; \
>     gdb-add-index /opt/fuzzer/bin/rippled-fuzzer; \
>     readelf -S /opt/xrpld/bin/xrpld | grep -q '\.gdb_index'; \
>     readelf -S /opt/fuzzer/bin/rippled-fuzzer | grep -q '\.gdb_index'; \
>     /opt/xrpld/bin/xrpld --version

(If you switch to --dwarf-5, the grep target becomes .debug_names.)


# Create fuzzer config directory and xrpld directories
RUN mkdir -p /etc/fuzzer /var/lib/xrpld/db /var/log/xrpld
Expand Down