diff --git a/.clusterfuzzlite/Dockerfile b/.clusterfuzzlite/Dockerfile new file mode 100644 index 0000000000..b483bc0df8 --- /dev/null +++ b/.clusterfuzzlite/Dockerfile @@ -0,0 +1,18 @@ +# ClusterFuzzLite / OSS-Fuzz build image for keep-core's native Go fuzz targets. +# base-builder-go provides the Go toolchain plus the compile_native_go_fuzzer +# helper used by build.sh. +# +# Digest-pinned: the :latest tag floats and the image is rebuilt upstream +# continuously; an unpinned base silently changes the build environment (and +# is a supply-chain vector) on every CI run. Bump the digest deliberately — +# resolve the current one with: +# curl -s "https://gcr.io/v2/oss-fuzz-base/base-builder-go/manifests/latest" \ +# -H "Authorization: Bearer $(curl -s 'https://gcr.io/v2/token?service=gcr.io&scope=repository:oss-fuzz-base/base-builder-go:pull' | jq -r .token)" \ +# -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -I | grep -i docker-content-digest +FROM gcr.io/oss-fuzz-base/base-builder-go@sha256:cf761fd9baac42fff453259755067a7ad8ad70dbbe7db5027211e9fabc5cac40 + +# The ClusterFuzzLite build_fuzzers action supplies the checked-out repo as the +# Docker build context; copy it in and build from there. +COPY . $SRC/keep-core +WORKDIR $SRC/keep-core +COPY .clusterfuzzlite/build.sh $SRC/ diff --git a/.clusterfuzzlite/README.md b/.clusterfuzzlite/README.md new file mode 100644 index 0000000000..68d48ac93c --- /dev/null +++ b/.clusterfuzzlite/README.md @@ -0,0 +1,104 @@ +# Continuous fuzzing + +This directory wires keep-core's native Go fuzz targets (the `Fuzz*` functions +under `pkg/**/fuzz_test.go`) into **ClusterFuzzLite** — OSS-Fuzz's self-hosted +variant that runs in this repo's own GitHub Actions and **works on private +repos**. That last property is why ClusterFuzzLite, not OSS-Fuzz, is the right +tool for this fork (OSS-Fuzz only fuzzes public projects). + +## Files + +| file | purpose | +|---|---| +| `Dockerfile` | build image (`base-builder-go`) | +| `build.sh` | compiles every `Fuzz*` target into a libFuzzer binary (path-qualified output names — several `Fuzz*` funcs share a name across packages) | +| `project.yaml` | `language: go` | +| `../.github/workflows/cflite_pr.yml` | per-PR fuzzing of changed code (fast, exits on first crash) | +| `../.github/workflows/cflite_batch.yml` | scheduled longer run over all targets | + +## Adding / regenerating targets + +`build.sh` must list one `compile_native_go_fuzzer` line per `Fuzz*` target. +CI enforces this (`check_targets.sh` runs on every PR and fails on drift). +Regenerate after adding targets: + +```sh +for f in $(grep -rln "func Fuzz.*testing.F" pkg/ --include="*_test.go" | sort); do + d=$(dirname "$f"); p="github.com/keep-network/keep-core/$d" + pref=$(echo "$d" | sed 's#^pkg/##; s#/#_#g') + grep -oE "func (Fuzz[A-Za-z0-9_]+)\(" "$f" | sed -E 's/func (Fuzz[A-Za-z0-9_]+)\(/\1/' \ + | while read fn; do echo "compile_native_go_fuzzer $p $fn ${pref}_${fn}"; done +done +``` + +## Enabling corpus persistence (batch mode) + +Batch fuzzing benefits from carrying the corpus between runs — without it +every nightly run restarts from the in-tree seeds and the 1800s budget is a +smoke test, not coverage-accumulating fuzzing. To enable: + +1. Create a private storage repo, e.g. `tlabs-xyz/keep-core-security-fuzz-corpus`. +2. Add a `PERSONAL_ACCESS_TOKEN` repo secret. It MUST be a **fine-grained + PAT scoped to the storage repo only**, with `Contents: Read and write` + as its only permission. Never use a classic PAT here: the token is + interpolated into a clone URL inside a job that executes + repo-controlled build code (`build.sh`, `Dockerfile`), so an + over-scoped token would hand that code access to everything it can + reach. Set an expiry and rotate it. +3. Uncomment the `storage-repo*` lines in `cflite_batch.yml` (and + `upload-build`). Keep persistence OUT of `cflite_pr.yml`: PR jobs run + proposed code and must not see the token at all. + +Until then, each batch run starts from the in-tree seed corpus. + +## Fork-lifecycle policy (why this exists) + +This is a **private fork** of the public `github.com/keep-network/keep-core`. +Fuzzing finds bugs in code; whether a finding is fork-relevant depends on how +far the fork has diverged. Two facts drive the policy: + +- **Fixes do not flow back automatically.** A bug fixed upstream stays open in + this fork until deliberately back-merged (this engagement already hit exactly + that: upstream's OOB fix was incomplete and had to be back-merged by hand). +- **Fork-divergent code gets no upstream coverage.** OSS-Fuzz on the upstream + cannot see code that only exists here. + +Policy: + +1. **Run ClusterFuzzLite here** (this directory) so the fork's own code — + including divergent paths — is fuzzed in its own CI. +2. **Track upstream `main`**: reconcile within a bounded window (e.g. N commits + or one release) so shared-parser fixes found upstream reach the fork. +3. **Contribute the fuzz targets upstream** (below) so the shared parsers get + continuous OSS-Fuzz coverage at Google's scale, and so this fork inherits + that coverage on the shared code after each reconcile. + +## OSS-Fuzz for the public upstream + +The same `Dockerfile` / `build.sh` / targets work for OSS-Fuzz once the +`Fuzz*` targets are merged into `github.com/keep-network/keep-core`. To enroll +the upstream, open a PR to `google/oss-fuzz` adding `projects/keep-core/` with: + +- `project.yaml`: + + ```yaml + homepage: "https://github.com/keep-network/keep-core" + language: go + primary_contact: "" + main_repo: "https://github.com/keep-network/keep-core" + fuzzing_engines: + - libfuzzer + sanitizers: + - address + ``` + +- a `Dockerfile` that `git clone`s the upstream repo (instead of `COPY .`): + + ```dockerfile + FROM gcr.io/oss-fuzz-base/base-builder-go + RUN git clone --depth 1 https://github.com/keep-network/keep-core $SRC/keep-core + WORKDIR $SRC/keep-core + COPY build.sh $SRC/ + ``` + +- the same `build.sh` from this directory. diff --git a/.clusterfuzzlite/build.sh b/.clusterfuzzlite/build.sh new file mode 100755 index 0000000000..0cb25a00ad --- /dev/null +++ b/.clusterfuzzlite/build.sh @@ -0,0 +1,64 @@ +#!/bin/bash -eu +# +# ClusterFuzzLite / OSS-Fuzz build script for keep-core native (testing.F) +# fuzz targets. Compiles every Fuzz* target into a libFuzzer binary. Output +# names are path-qualified because several Fuzz funcs share a name across +# packages (e.g. FuzzEphemeralPublicKeyMessageUnmarshal in gjkr/dkg/signing). +# +# Regenerate the target list with: +# grep -rhoE "func (Fuzz[A-Za-z0-9_]+)\(f \*testing.F\)" pkg/ --include="*_test.go" + +cd "$SRC/keep-core" + +# Fuzzers don't need VCS build stamping, and stamping can fail in the build +# container (git "dubious ownership" / detached checkout). Disable it. +export GOFLAGS="-buildvcs=false ${GOFLAGS:-}" + +# compile_native_go_fuzzer rewrites each testing.F target onto the OSS-Fuzz +# libFuzzer shim; pull it into the module graph (build-container only, not +# committed to go.mod). Pinned to a commit SHA: this fetch happens outside +# go.sum protection on every CI build, so an unpinned HEAD would execute +# whatever upstream pushes. Bump deliberately. +go get github.com/AdamKorcz/go-118-fuzz-build/testing@a70c2aa677fa43583571959478decabe02a96cd6 + +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/dkg/result FuzzDKGResultHashSignatureMessageUnmarshal beacon_dkg_result_FuzzDKGResultHashSignatureMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/entry FuzzSignatureShareMessageUnmarshal beacon_entry_FuzzSignatureShareMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzEphemeralPublicKeyMessageUnmarshal beacon_gjkr_FuzzEphemeralPublicKeyMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzMemberCommitmentsMessageUnmarshal beacon_gjkr_FuzzMemberCommitmentsMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzPeerSharesMessageUnmarshal beacon_gjkr_FuzzPeerSharesMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzSecretSharesAccusationsMessageUnmarshal beacon_gjkr_FuzzSecretSharesAccusationsMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzMemberPublicKeySharePointsMessageUnmarshal beacon_gjkr_FuzzMemberPublicKeySharePointsMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzPointsAccusationsMessageUnmarshal beacon_gjkr_FuzzPointsAccusationsMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/beacon/gjkr FuzzMisbehavedEphemeralKeysMessageUnmarshal beacon_gjkr_FuzzMisbehavedEphemeralKeysMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/bitcoin FuzzNewScriptFromVarLenData bitcoin_FuzzNewScriptFromVarLenData +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/bitcoin FuzzTransactionDeserialize bitcoin_FuzzTransactionDeserialize +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/net/libp2p FuzzIdentityUnmarshal net_libp2p_FuzzIdentityUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/net/security/handshake FuzzAct1MessageUnmarshal net_security_handshake_FuzzAct1MessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/net/security/handshake FuzzAct2MessageUnmarshal net_security_handshake_FuzzAct2MessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/net/security/handshake FuzzAct3MessageUnmarshal net_security_handshake_FuzzAct3MessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/protocol/announcer FuzzAnnouncementMessageUnmarshal protocol_announcer_FuzzAnnouncementMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/protocol/inactivity FuzzClaimSignatureMessageUnmarshal protocol_inactivity_FuzzClaimSignatureMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzSigningDoneMessageUnmarshal tbtc_FuzzSigningDoneMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzCoordinationMessageUnmarshal tbtc_FuzzCoordinationMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzNoopProposalUnmarshal tbtc_FuzzNoopProposalUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzHeartbeatProposalUnmarshal tbtc_FuzzHeartbeatProposalUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzDepositSweepProposalUnmarshal tbtc_FuzzDepositSweepProposalUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzRedemptionProposalUnmarshal tbtc_FuzzRedemptionProposalUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzMovingFundsProposalUnmarshal tbtc_FuzzMovingFundsProposalUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tbtc FuzzMovedFundsSweepProposalUnmarshal tbtc_FuzzMovedFundsSweepProposalUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/dkg FuzzEphemeralPublicKeyMessageUnmarshal tecdsa_dkg_FuzzEphemeralPublicKeyMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/dkg FuzzTssRoundOneMessageUnmarshal tecdsa_dkg_FuzzTssRoundOneMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/dkg FuzzTssRoundTwoMessageUnmarshal tecdsa_dkg_FuzzTssRoundTwoMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/dkg FuzzTssRoundThreeMessageUnmarshal tecdsa_dkg_FuzzTssRoundThreeMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/dkg FuzzTssFinalizationMessageUnmarshal tecdsa_dkg_FuzzTssFinalizationMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/dkg FuzzResultSignatureMessageUnmarshal tecdsa_dkg_FuzzResultSignatureMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzEphemeralPublicKeyMessageUnmarshal tecdsa_signing_FuzzEphemeralPublicKeyMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundOneMessageUnmarshal tecdsa_signing_FuzzTssRoundOneMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundTwoMessageUnmarshal tecdsa_signing_FuzzTssRoundTwoMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundThreeMessageUnmarshal tecdsa_signing_FuzzTssRoundThreeMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundFourMessageUnmarshal tecdsa_signing_FuzzTssRoundFourMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundFiveMessageUnmarshal tecdsa_signing_FuzzTssRoundFiveMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundSixMessageUnmarshal tecdsa_signing_FuzzTssRoundSixMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundSevenMessageUnmarshal tecdsa_signing_FuzzTssRoundSevenMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundEightMessageUnmarshal tecdsa_signing_FuzzTssRoundEightMessageUnmarshal +compile_native_go_fuzzer github.com/keep-network/keep-core/pkg/tecdsa/signing FuzzTssRoundNineMessageUnmarshal tecdsa_signing_FuzzTssRoundNineMessageUnmarshal diff --git a/.clusterfuzzlite/check_targets.sh b/.clusterfuzzlite/check_targets.sh new file mode 100755 index 0000000000..39f15e461f --- /dev/null +++ b/.clusterfuzzlite/check_targets.sh @@ -0,0 +1,40 @@ +#!/bin/bash -eu +# +# Drift guard: fails when the set of native Fuzz* targets under pkg/ +# diverges from the compile_native_go_fuzzer registration list in +# build.sh. Without this, a new Fuzz* function compiles fine under +# `go test` but silently receives zero ClusterFuzzLite coverage. +# +# Compares exact (package, function) pairs — not counts — because +# several Fuzz functions share a name across packages. + +repo_root="$(cd "$(dirname "$0")/.." && pwd)" +module="github.com/keep-network/keep-core" + +# Work from the repo root so grep emits relative paths: the absolute path +# never enters the sed pattern, where regex metacharacters in a checkout +# location could otherwise misparse the target list. +cd "$repo_root" + +expected="$( + grep -rn --include='*_test.go' -E '^func Fuzz[A-Za-z0-9_]+\(f \*testing\.F\)' pkg | + sed -E "s|^(.+)/[^/]+\.go:[0-9]+:func (Fuzz[A-Za-z0-9_]+)\(.*$|$module/\1 \2|" | + sort -u +)" + +registered="$( + grep -E '^compile_native_go_fuzzer ' .clusterfuzzlite/build.sh | + awk '{print $2, $3}' | + sort -u +)" + +if ! diff <(echo "$expected") <(echo "$registered") >&2; then + echo >&2 + echo "Fuzz target drift detected:" >&2 + echo " < targets found in pkg/ but not registered in .clusterfuzzlite/build.sh" >&2 + echo " > targets registered in build.sh but missing from pkg/" >&2 + echo "Add/remove the matching compile_native_go_fuzzer line(s)." >&2 + exit 1 +fi + +echo "OK: $(echo "$expected" | wc -l) fuzz targets, build.sh registration list in sync." diff --git a/.clusterfuzzlite/project.yaml b/.clusterfuzzlite/project.yaml new file mode 100644 index 0000000000..29cd7ff60d --- /dev/null +++ b/.clusterfuzzlite/project.yaml @@ -0,0 +1,11 @@ +# ClusterFuzzLite project configuration. For CFLite only `language` is required; +# it is consumed by the build_fuzzers / run_fuzzers GitHub Actions. +# +# (The OSS-Fuzz integration for the PUBLIC upstream repo lives in the +# google/oss-fuzz repo under projects/keep-core/ and carries additional fields +# — homepage, primary_contact, main_repo, auto_ccs. See README.md.) +language: go +fuzzing_engines: + - libfuzzer +sanitizers: + - address diff --git a/.dockerignore b/.dockerignore index 5d24df6262..ef7c8cc2fc 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,5 +1,8 @@ # Hidden files and directories. .* +# ...except the ClusterFuzzLite build files, which must reach the build context. +!.clusterfuzzlite +!.clusterfuzzlite/** # Top-level directories unrelated to the build. docs*/ @@ -12,6 +15,28 @@ CODEOWNERS Dockerfile *.adoc +# ...except the files under those trees that the client's own tests open at +# run time: the release manifest, the schemas its records are validated +# against, the deployment scaffolds its termination grace is compared to, and +# the metrics reference the participation family is held to. Those tests +# execute inside this image, so a file one of them reads has to reach the +# context or the test fails there and nowhere else. Nothing else under docs/ +# or scripts/ does. +# +# Every file is named rather than the two directories holding them, so a file +# added beside one of these does not reach the build on its own. +# +# Listed after every rule above that could match them rather than beside the +# rule each excepts, because the last matching entry is the one that decides. +!docs/performance-metrics.adoc +!scripts/release/pr4109/compose.rehearsal.yaml +!scripts/release/pr4109/deploy/keep-client-termination-grace.k8s-patch.yaml +!scripts/release/pr4109/deploy/keep-client-termination-grace.systemd-dropin.conf +!scripts/release/pr4109/rehearsal-evidence.schema.json +!scripts/release/pr4109/release-manifest.json +!scripts/release/pr4109/release-manifest.schema.json +!scripts/release/pr4109/release-provenance.schema.json + # NPM stuff. **/node_modules/* @@ -27,6 +52,9 @@ token-tracker/ # Go stuff. **/gen/_contracts **/gen/**/*.go +# ...but keep the committed protobuf message code (gen/pb); the ClusterFuzzLite +# build does not run protoc, and the unmarshaler fuzz targets need it. +!**/gen/pb/*.go !**/gen/gen.go !**/gen/cmd/cmd.go diff --git a/.github/actions/docker-build-push/action.yml b/.github/actions/docker-build-push/action.yml index f4e806e0f1..fa59b5543a 100644 --- a/.github/actions/docker-build-push/action.yml +++ b/.github/actions/docker-build-push/action.yml @@ -14,6 +14,10 @@ inputs: description: True if the image should be published required: true default: "false" + load: + description: True if the image should be loaded into the local Docker daemon + required: false + default: "false" gcrJsonKey: description: JSON key for Google Container Registry service account (required if push is true) required: false @@ -69,6 +73,7 @@ runs: labels: | revision=${{ github.sha }} push: ${{ inputs.push == 'true' }} + load: ${{ inputs.load == 'true' }} cache-from: type=local,src=/tmp/.buildx-cache cache-to: type=local,dest=/tmp/.buildx-cache-new diff --git a/.github/workflows/cflite_batch.yml b/.github/workflows/cflite_batch.yml new file mode 100644 index 0000000000..ef067f99e2 --- /dev/null +++ b/.github/workflows/cflite_batch.yml @@ -0,0 +1,46 @@ +name: ClusterFuzzLite batch fuzzing + +# Scheduled longer fuzzing run over all targets to grow the corpus and reach +# deeper bugs than per-PR fuzzing can. Does not exit on first crash. +# +# Corpus/crash persistence requires a storage repo + a PERSONAL_ACCESS_TOKEN +# secret; uncomment the storage-repo lines once those exist (see +# .clusterfuzzlite/README.md). Without persistence the run still fuzzes but +# starts from the in-tree seed corpus each time. +on: + schedule: + - cron: "0 2 * * *" # daily, offset from the -race job (midnight) + workflow_dispatch: + +# No security-events permission: this repo has no GitHub Advanced +# Security, so SARIF upload to code scanning would 403. Crash artifacts +# are reported via the action's run output and artifacts instead. +permissions: + contents: read + +jobs: + Batch: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + sanitizer: [address] + steps: + - name: Build fuzzers (${{ matrix.sanitizer }}) + id: build + uses: google/clusterfuzzlite/actions/build_fuzzers@82652fb49e77bc29c35da1167bb286e93c6bcc05 # v1 + with: + language: go + sanitizer: ${{ matrix.sanitizer }} + # upload-build: true + - name: Run fuzzers (${{ matrix.sanitizer }}) + id: run + uses: google/clusterfuzzlite/actions/run_fuzzers@82652fb49e77bc29c35da1167bb286e93c6bcc05 # v1 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + fuzz-seconds: 1800 + mode: "batch" + sanitizer: ${{ matrix.sanitizer }} + # storage-repo: https://${{ secrets.PERSONAL_ACCESS_TOKEN }}@github.com/tlabs-xyz/keep-core-security-fuzz-corpus.git + # storage-repo-branch: main + # storage-repo-branch-coverage: gh-pages diff --git a/.github/workflows/cflite_pr.yml b/.github/workflows/cflite_pr.yml new file mode 100644 index 0000000000..3632b6733e --- /dev/null +++ b/.github/workflows/cflite_pr.yml @@ -0,0 +1,62 @@ +name: ClusterFuzzLite PR fuzzing + +# Builds the native Go fuzz targets and fuzzes only the code changed in a PR +# (code-change mode), giving fast per-PR feedback. Exits on the first crash. +# Complements the nightly -race job and the batch fuzzer below. +on: + pull_request: + paths: + - "pkg/**" + - ".clusterfuzzlite/**" + # Infra the fuzz build depends on: a change here can break the + # CFLite build without touching pkg/, and would otherwise only be + # caught by the nightly batch run. + - "go.mod" + - "go.sum" + - ".dockerignore" + - ".github/workflows/cflite_pr.yml" + - ".github/workflows/cflite_batch.yml" + +# No security-events permission: this repo has no GitHub Advanced +# Security, so SARIF upload to code scanning would 403. Crash artifacts +# are reported via the action's run output and artifacts instead. +permissions: + contents: read + +jobs: + target-sync: + # build.sh's registration list is a second source of truth: a new + # Fuzz* function that is not registered silently gets zero CFLite + # coverage. Fail the PR instead. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Check fuzz targets are registered in build.sh + run: ./.clusterfuzzlite/check_targets.sh + + PR: + runs-on: ubuntu-latest + concurrency: + group: ${{ github.workflow }}-${{ matrix.sanitizer }}-${{ github.ref }} + cancel-in-progress: true + strategy: + fail-fast: false + matrix: + # Go native fuzzing builds under libFuzzer + AddressSanitizer. + sanitizer: [address] + steps: + - name: Build fuzzers (${{ matrix.sanitizer }}) + id: build + uses: google/clusterfuzzlite/actions/build_fuzzers@82652fb49e77bc29c35da1167bb286e93c6bcc05 # v1 + with: + language: go + github-token: ${{ secrets.GITHUB_TOKEN }} + sanitizer: ${{ matrix.sanitizer }} + - name: Run fuzzers (${{ matrix.sanitizer }}) + id: run + uses: google/clusterfuzzlite/actions/run_fuzzers@82652fb49e77bc29c35da1167bb286e93c6bcc05 # v1 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + fuzz-seconds: 300 + mode: "code-change" + sanitizer: ${{ matrix.sanitizer }} diff --git a/.github/workflows/client.yml b/.github/workflows/client.yml index f9b82e90b2..0a81339d73 100644 --- a/.github/workflows/client.yml +++ b/.github/workflows/client.yml @@ -28,9 +28,16 @@ on: # Automatic releases are now handled by the dedicated release.yml workflow +permissions: + contents: read + pull-requests: read + jobs: client-detect-changes: runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read outputs: path-filter: ${{ steps.filter.outputs.path-filter }} steps: @@ -47,6 +54,9 @@ jobs: electrum-integration-detect-changes: runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read outputs: path-filter: ${{ steps.filter.outputs.path-filter }} steps: @@ -69,6 +79,20 @@ jobs: || needs.client-detect-changes.outputs.path-filter == 'true' runs-on: ubuntu-latest steps: + - name: Free disk space + # The multi-arch client binary build exhausts the default ~14GB free + # on ubuntu-latest. Reclaim ~30GB by removing preinstalled toolchains + # we don't use. + uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1 + with: + tool-cache: false + android: true + dotnet: true + haskell: true + large-packages: false + docker-images: false + swap-storage: false + - uses: actions/checkout@v4 with: # Fetch the whole history for the `git describe` command to work. @@ -143,6 +167,20 @@ jobs: if-no-files-found: warn + # The Tier-2 interceptor relies on a mutex to let a stateful Strategy run + # under concurrent Sends without a data race; TestStrategyConcurrentStatefulNoRace + # only has teeth under the race detector. Scoped to the fast, deterministic + # interception/byzantine packages (dkgtest is excluded: its real-DKG goroutines + # and wall-clock windows are timeout-amplified by -race). + - name: Run Go race tests (Tier-2 interceptor) + run: | + docker run \ + --workdir /go/src/github.com/keep-network/keep-core \ + go-build-env \ + go test -race -timeout 15m \ + ./pkg/internal/interception/... \ + ./pkg/internal/byzantine/... + - name: Build Docker Runtime Image if: github.event_name != 'workflow_dispatch' uses: docker/build-push-action@v5 @@ -308,6 +346,27 @@ jobs: install-go: false checks: "-SA1019" + client-golangci: + needs: client-detect-changes + if: | + github.event_name == 'push' + || needs.client-detect-changes.outputs.path-filter == 'true' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-go@v5 + with: + go-version-file: "go.mod" + # Additive: hosts only the project-specific ruleguard rule that bans raw + # indexing of a bitcoin.Transaction's Outputs/Inputs (use OutputAt/InputAt + # instead). The existing go vet / gofmt / staticcheck / gosec jobs are + # left intact and are not duplicated here. Config: .golangci.yml + + # .golangci-ruleguard.rules.go. + - name: golangci-lint + uses: golangci/golangci-lint-action@v9 + with: + version: v2.12.2 + client-integration-test: needs: [client-detect-changes, electrum-integration-detect-changes, client-build-test-publish] if: | @@ -334,3 +393,89 @@ jobs: --workdir /go/src/github.com/keep-network/keep-core \ go-build-env \ gotestsum -- -timeout 20m -tags=integration ./... + + client-race-test: + needs: client-build-test-publish + # Non-blocking by design: runs nightly (schedule) and on manual + # dispatch, but is intentionally NOT a required PR check until it has + # been green and stable for a while. The first runs on a codebase that + # has never had the race detector enabled are expected to surface + # latent races and timing-sensitive flakes; triage each before + # promoting this to a required check. + if: | + github.event_name == 'schedule' + || github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + permissions: + contents: read + issues: write + steps: + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Download Docker Build Image + uses: actions/download-artifact@v4 + with: + name: go-build-env-image + path: /tmp + + - name: Load Docker Build Image + run: | + docker load --input /tmp/go-build-env-image.tar + + - name: Run Go tests with the race detector + # The race detector requires cgo and a C toolchain, both present in + # the build image (g++/gcc). It is ~2-20x slower and uses ~5-10x + # more memory than a normal run, hence the longer timeout and why + # this is a separate job rather than a flag on the main test step. + # The default test scope (./...) includes the in-process protocol + # simulations (dkgtest / entrytest / gjkr roundtrip), which is where + # data races in concurrent protocol code actually surface. + run: | + docker run \ + --workdir /go/src/github.com/keep-network/keep-core \ + --env CGO_ENABLED=1 \ + go-build-env \ + gotestsum -- -race -timeout 60m + + - name: Report scheduled race-detector failure + # A non-blocking nightly job rots red silently without this: nobody + # watches the Actions tab. Upsert a labeled issue so failures are + # visible and triaged before this job is promoted to a required + # check. Manual (workflow_dispatch) runs are excluded — the person + # who dispatched them is already watching. + if: failure() && github.event_name == 'schedule' + uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7 + with: + script: | + const label = "race-detector-failure"; + const runUrl = + `${context.serverUrl}/${context.repo.owner}/` + + `${context.repo.repo}/actions/runs/${context.runId}`; + const body = + `The scheduled \`-race\` test job failed.\n\n` + + `Run: ${runUrl}\n\n` + + `Triage the race report before promoting the job to a ` + + `required PR check.`; + const open = await github.rest.issues.listForRepo({ + owner: context.repo.owner, + repo: context.repo.repo, + state: "open", + labels: label, + }); + if (open.data.length > 0) { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: open.data[0].number, + body, + }); + } else { + await github.rest.issues.create({ + owner: context.repo.owner, + repo: context.repo.repo, + title: "Nightly race-detector run failed", + labels: [label], + body, + }); + } diff --git a/.github/workflows/contracts-ecdsa-docs.yml b/.github/workflows/contracts-ecdsa-docs.yml index c9d77dbd56..cafd46e3d8 100644 --- a/.github/workflows/contracts-ecdsa-docs.yml +++ b/.github/workflows/contracts-ecdsa-docs.yml @@ -12,9 +12,16 @@ on: - "published" workflow_dispatch: +permissions: + contents: read + pull-requests: read + jobs: docs-detect-changes: runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read outputs: path-filter: ${{ steps.filter.outputs.path-filter }} steps: diff --git a/.github/workflows/contracts-ecdsa.yml b/.github/workflows/contracts-ecdsa.yml index 3ee91bba50..1b7e9171a2 100644 --- a/.github/workflows/contracts-ecdsa.yml +++ b/.github/workflows/contracts-ecdsa.yml @@ -164,8 +164,40 @@ jobs: with: imageName: keep-ecdsa-hardhat push: false + load: true context: ./solidity/ecdsa + - name: Smoke Docker image on native AMD64 + env: + EXPECTED_REVISION: ${{ github.sha }} + run: | + set -euo pipefail + { + runner_architecture="$(uname -m)" + image_platform="$(docker image inspect \ + --format '{{.Os}}/{{.Architecture}}' \ + keep-ecdsa-hardhat)" + image_revision="$(docker image inspect \ + --format '{{index .Config.Labels "revision"}}' \ + keep-ecdsa-hardhat)" + echo "runner architecture: ${runner_architecture}" + echo "image platform: ${image_platform}" + echo "image revision: ${image_revision}" + test "${runner_architecture}" = "x86_64" + test "${image_platform}" = "linux/amd64" + test "${image_revision}" = "${EXPECTED_REVISION}" + docker run --rm --pull=never keep-ecdsa-hardhat --version + } 2>&1 | tee docker-image-smoke.log + + - name: Preserve Docker image smoke log + if: always() + uses: actions/upload-artifact@v4 + with: + name: ecdsa-hardhat-docker-smoke-${{ github.sha }} + path: solidity/ecdsa/docker-image-smoke.log + if-no-files-found: error + retention-days: 30 + contracts-deployment-testnet: needs: [contracts-build-and-test] if: | diff --git a/.github/workflows/contracts-random-beacon-docs.yml b/.github/workflows/contracts-random-beacon-docs.yml index 100c4a00be..24fbf958e6 100644 --- a/.github/workflows/contracts-random-beacon-docs.yml +++ b/.github/workflows/contracts-random-beacon-docs.yml @@ -12,9 +12,16 @@ on: - "published" workflow_dispatch: +permissions: + contents: read + pull-requests: read + jobs: docs-detect-changes: runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read outputs: path-filter: ${{ steps.filter.outputs.path-filter }} steps: diff --git a/.github/workflows/contracts-random-beacon.yml b/.github/workflows/contracts-random-beacon.yml index 060cd09ad3..5cd3ff2662 100644 --- a/.github/workflows/contracts-random-beacon.yml +++ b/.github/workflows/contracts-random-beacon.yml @@ -163,6 +163,38 @@ jobs: imageName: keep-random-beacon-hardhat context: ./solidity/random-beacon push: false + load: true + + - name: Smoke Docker image on native AMD64 + env: + EXPECTED_REVISION: ${{ github.sha }} + run: | + set -euo pipefail + { + runner_architecture="$(uname -m)" + image_platform="$(docker image inspect \ + --format '{{.Os}}/{{.Architecture}}' \ + keep-random-beacon-hardhat)" + image_revision="$(docker image inspect \ + --format '{{index .Config.Labels "revision"}}' \ + keep-random-beacon-hardhat)" + echo "runner architecture: ${runner_architecture}" + echo "image platform: ${image_platform}" + echo "image revision: ${image_revision}" + test "${runner_architecture}" = "x86_64" + test "${image_platform}" = "linux/amd64" + test "${image_revision}" = "${EXPECTED_REVISION}" + docker run --rm --pull=never keep-random-beacon-hardhat --version + } 2>&1 | tee docker-image-smoke.log + + - name: Preserve Docker image smoke log + if: always() + uses: actions/upload-artifact@v4 + with: + name: random-beacon-hardhat-docker-smoke-${{ github.sha }} + path: solidity/random-beacon/docker-image-smoke.log + if-no-files-found: error + retention-days: 30 contracts-deployment-testnet: needs: [contracts-build-and-test] diff --git a/.github/workflows/cutover-rehearsal.yml b/.github/workflows/cutover-rehearsal.yml new file mode 100644 index 0000000000..77b9b702e6 --- /dev/null +++ b/.github/workflows/cutover-rehearsal.yml @@ -0,0 +1,1115 @@ +name: Cutover Rehearsal + +# Manually dispatched driver for the single-release cutover rehearsal +# scaffold. Every dispatch runs the repository-local Go proofs of the cutover +# gate inside the same build image the client CI uses, the immutable-version +# static analyzers, and the ECDSA contracts build/test, self-tests the +# source-binding and evidence-record validators, validates any produced +# evidence records against the evidence schema, and archives each stage's +# log for the dispatched SHA. The container rehearsal derives one native-runner +# job per published platform from the sealed release provenance. Each runner +# needs an isolated pre-C chain, emits only the platform record it can honestly +# observe, and uploads it for a final archive-wide validation over both gates +# and every platform. These stages run only when explicitly requested with the +# immutable image digests and rehearsal-chain inputs, and each one names the +# exact input it is missing and reports BLOCKED — a failed job — rather than +# running partially, because a rehearsal that cannot execute must never look +# green. +# +# Provenance is fail-closed: every proof stage receives the dispatched SHA +# in PR4109_EXPECTED_SOURCE_COMMIT and refuses to produce evidence unless +# the tree it is about to test is exactly that commit. For the build-image +# stage the checkout's .git and scripts/ are mounted read-only into the +# container (.dockerignore keeps both out of the build context), so the +# verification happens against the very bytes inside the image, not just +# the runner checkout. Stage logs are archived even when a stage fails — a +# red run's evidence is the most valuable kind. +# +# What the release was built into travels the other way. The reviewed manifest +# names the cutover; it cannot name the commit finally built or the immutable +# image digests, because those are outputs of a build over its own bytes. A +# release dispatch supplies them as a detached document in +# release_provenance_b64, which is decoded outside the checkout, mounted into +# the proof container, sealed into the receipt by the producer after the +# binary verifies it against the manifest, and required to be there — as the +# document that was supplied — before the receipt is archived. The container +# job checks the receipt it downloads for the same pairing before it pulls an +# image, so a release-ready receipt that names no artifact is refused at the +# dispatch boundary rather than after a fleet has run every mandatory step. + +on: + workflow_dispatch: + inputs: + artifact_environment: + description: "npm dist-tag or exact version for the contract artifacts baked into the build image; forensic only — the proof stage restores every regenerated tracked file byte-exact from the dispatched SHA before testing, and the stamp records every resolved tarball's name, version, and sha256" + required: false + default: "development" + run_container_stages: + description: "Run the container rehearsal stages (needs all inputs below)" + type: boolean + required: false + default: false + prior_image_digest: + description: "Immutable prior-production runtime digest (repo@sha256:...)" + required: false + r1_image_digest: + description: "Immutable R1 candidate runtime digest (repo@sha256:...)" + required: false + probe_image_digest: + description: "Immutable digest of the wget-carrying probe image every evidence reading is scraped with (repo@sha256:...); a mutable tag would leave the reading instrument outside the record's provenance" + required: false + rehearsal_chain_inputs_b64: + description: "Base64 of one JSON object keyed by every published platform, each value exactly {eth_ws_url,eth_rpc_url,cutover_block,chain_id}. The platform job validates the one-to-one mapping against detached provenance and emits the values into the native-runner matrix; each platform needs its own isolated pre-C chain" + required: false + bitcoin_network: + description: "Bitcoin network the rollback state audit reconciles against (e.g. testnet)" + required: false + prior_version: + description: "Release version the prior digest carries, as the rollback state audit must find it recorded" + required: false + prior_revision: + description: "Source revision the prior digest carries, as the rollback state audit must find it recorded" + required: false + release_provenance_b64: + description: "base64 of the detached release provenance: the commit built and the immutable per-platform image digests, generated after the release build and never committed to the tree it describes. Required for any dispatch whose reviewed manifest is release-ready — acceptance refuses records measured against a release that names no artifact — and legitimately empty on development dispatches" + required: false + +permissions: + contents: read + +jobs: + local-proofs: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + # Pin the checkout to the dispatched SHA so a branch moving + # between dispatch and run cannot change what is tested. + ref: ${{ github.sha }} + # Fetch the whole history for the `git describe` command to work. + fetch-depth: 0 + + - name: Resolve versions + run: | + echo "version=$(git describe --tags --match "v[0-9]*" HEAD)" >> "$GITHUB_ENV" + echo "revision=$(git rev-parse HEAD)" >> "$GITHUB_ENV" + + # The binding verifier gates every piece of evidence this workflow + # archives, so it proves itself before the expensive image build: the + # self-test drives it through checkout- and image-shaped trees and + # fails the dispatch if the verifier accepts anything beyond the + # image's documented construction. + - name: Self-test the source binding verifier + run: ./scripts/release/pr4109/test-source-binding.sh + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Cache Docker layers + uses: actions/cache@v4 + with: + path: /tmp/.buildx-cache + key: ${{ runner.os }}-buildx-rehearsal-${{ github.sha }} + restore-keys: | + ${{ runner.os }}-buildx- + + # ENVIRONMENT names the artifact input identity explicitly instead of + # riding the Makefile's implicit development default (client CI's + # empty build-arg resolves to the same tag). The tag can float on the + # registry, but nothing tested depends on it: the in-image + # verification restores every regenerated tracked file byte-exact + # from the dispatched commit before the proofs compile, and records + # the resolved tarballs — name, exact version, sha256 — in the + # archived stamp as the image build's own input identity. + - name: Build Docker Build Image + uses: docker/build-push-action@v5 + with: + target: build-docker + tags: go-build-env + build-args: | + VERSION=${{ env.version }} + REVISION=${{ env.revision }} + ENVIRONMENT=${{ inputs.artifact_environment }} + load: true # load image to local registry to use it in next steps + cache-from: type=local,src=/tmp/.buildx-cache + cache-to: type=local,dest=/tmp/.buildx-cache-new + context: . + + # The half of the release identity the reviewed manifest cannot state + # about itself: the commit finally built and the immutable per-platform + # image digests are outputs of a build over the manifest's own bytes, so + # recording them in the tree would require the tree to contain a hash of + # itself. They live in a detached document generated after the build, + # and this is where a release dispatch hands it to the proof stage. + # + # It lands in RUNNER_TEMP rather than the workspace for the same reason + # every other provisioned input does: the proof stage refuses to produce + # evidence from a tree that diverges from the dispatched commit, + # untracked files included, so a provenance document written into the + # checkout would fail the very check it exists to satisfy. + # + # A development dispatch supplies none and still runs everything. The + # receipt then carries no provenance, which the acceptance stage refuses + # only once the reviewed manifest is release-ready — that refusal is + # acceptance's to make, not this step's. + - name: Provision the detached release provenance + env: + RELEASE_PROVENANCE_B64: ${{ inputs.release_provenance_b64 }} + run: | + if [ -z "$RELEASE_PROVENANCE_B64" ]; then + echo "no detached release provenance supplied; the receipt will" \ + "carry none, and acceptance refuses records measured against a" \ + "release-ready manifest without it" + exit 0 + fi + path="$RUNNER_TEMP/release-provenance.json" + printf '%s' "$RELEASE_PROVENANCE_B64" | base64 -d >"$path" + # Only that a document arrived at all. What it has to say is the + # binary's to judge inside the container, but a mis-pasted input is + # named here rather than surfacing as an unreadable file in a stage + # whose log nobody reads on the way to a red run. + node -e 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"))' \ + "$path" || { + echo "the supplied release_provenance_b64 does not decode to" >&2 + echo "readable JSON; supply the base64 of the provenance" >&2 + echo "document generated after the release build" >&2 + exit 1 + } + echo "RELEASE_PROVENANCE_FILE=$path" >>"$GITHUB_ENV" + echo "provisioned the detached release provenance" + + - name: Run cutover gate local proofs + run: | + mkdir -p ${{ github.workspace }}/rehearsal-evidence + # Mounted at the container root, not under the working directory: + # the stage verifies the image's own source tree against the + # dispatched SHA before producing evidence, and a document inside + # that tree would be an untracked file failing that check. + provenance_mount=() + if [ -n "${RELEASE_PROVENANCE_FILE:-}" ]; then + provenance_mount=( + -v "$RELEASE_PROVENANCE_FILE:/release-provenance.json:ro" + -e PR4109_RELEASE_PROVENANCE=/release-provenance.json + ) + fi + # .dockerignore keeps scripts/ and .git out of the build context, + # so both are mounted read-only from the dispatched checkout: the + # stage script runs at the dispatched revision, and rehearse.sh + # verifies the image's own source tree against the dispatched SHA + # before producing any evidence. safe.directory is required + # because the mounted metadata is owned by the runner user, not + # the container's root. + docker run \ + --workdir /go/src/github.com/keep-network/keep-core \ + -v ${{ github.workspace }}/rehearsal-evidence:/rehearsal-evidence \ + -v ${{ github.workspace }}/.git:/go/src/github.com/keep-network/keep-core/.git:ro \ + -v ${{ github.workspace }}/scripts:/go/src/github.com/keep-network/keep-core/scripts:ro \ + -e EVIDENCE_DIR=/rehearsal-evidence \ + -e PR4109_EXPECTED_SOURCE_COMMIT=${{ github.sha }} \ + -e PR4109_SOURCE_BINDING_MODE=build-image \ + "${provenance_mount[@]}" \ + go-build-env \ + bash -c 'git config --global --add safe.directory \ + /go/src/github.com/keep-network/keep-core && \ + exec ./scripts/release/pr4109/rehearse.sh local-proofs' + + # The receipt is what the container job downloads and every record it + # produces is measured against, so a supplied provenance has to have + # survived into it — and be the document that was supplied, byte for + # byte. Without this a release dispatch could archive a receipt + # acceptance refuses while this job stayed green, and the only account + # of why would be a line in a log read by nobody on a passing run. + - name: Require the receipt to carry the supplied provenance + run: | + if [ -z "${RELEASE_PROVENANCE_FILE:-}" ]; then + echo "no provenance was supplied to this dispatch; nothing to" \ + "require of the receipt" + exit 0 + fi + recorded="${{ github.workspace }}/rehearsal-evidence/attestation/release-provenance.json" + if [ ! -f "$recorded" ]; then + echo "the local proofs recorded no detached release provenance" >&2 + echo "although this dispatch supplied one; the archived receipt" >&2 + echo "cannot admit container evidence for a release-ready" >&2 + echo "manifest" >&2 + exit 1 + fi + if ! cmp -s "$RELEASE_PROVENANCE_FILE" "$recorded"; then + echo "the provenance recorded in the receipt is not the document" >&2 + echo "this dispatch supplied and the binary verified" >&2 + exit 1 + fi + echo "the archived receipt carries the supplied release provenance" + + # The evidence validator's own self-test already ran unconditionally + # inside the local-proofs stage above (its verdicts are part of the + # archived local-proofs.log), so a record-free dispatch still proves + # the validator; this step validates whatever records a dispatch + # actually produced. The manifest attestation this stage requires was + # written by local-proofs into the same bind-mounted evidence + # directory, one level down, so it is here and it does not make the + # top-level record probe below see a record that does not exist. + # + # The dispatched SHA is handed to this stage like to every other proof + # stage: the manifest, schema, and comparison rules it judges records + # by come out of this checkout, and the attestation and every record it + # accepts must name that same commit. Without the binding a receipt + # produced at one commit would admit records claiming another whenever + # the manifest bytes did not change between them. + - name: Validate evidence records against the schema + env: + PR4109_EXPECTED_SOURCE_COMMIT: ${{ github.sha }} + run: | + if compgen -G "${{ github.workspace }}/rehearsal-evidence/*.json" > /dev/null; then + EVIDENCE_DIR=${{ github.workspace }}/rehearsal-evidence \ + ./scripts/release/pr4109/rehearse.sh validate-evidence + else + echo "no JSON evidence records produced by this dispatch;" \ + "the evidence-validator self-test ran inside the local-proofs stage" + fi + + - name: Upload rehearsal evidence + # A failing proof stage's log is the evidence most needed for + # diagnosis, so archive whatever was produced even on failure. + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: rehearsal-evidence-${{ github.sha }} + path: rehearsal-evidence/ + if-no-files-found: warn + + static-analysis: + # Mirrors the client CI analyzer jobs' Go setup; the stage itself pins + # every analyzer to an immutable version so the archived log is + # reproducible evidence for this exact SHA. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.sha }} + + - uses: actions/setup-go@v5 + with: + go-version-file: "go.mod" + + - name: Run the immutable-version static analyzers + env: + PR4109_EXPECTED_SOURCE_COMMIT: ${{ github.sha }} + run: | + mkdir -p ${{ github.workspace }}/rehearsal-evidence + EVIDENCE_DIR=${{ github.workspace }}/rehearsal-evidence \ + ./scripts/release/pr4109/rehearse.sh static-analysis + + - name: Upload static-analysis evidence + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: static-analysis-evidence-${{ github.sha }} + path: rehearsal-evidence/ + if-no-files-found: error + + solidity-proofs: + # Mirrors contracts-ecdsa.yml's contracts-build-and-test job: the exact + # Node release pinned by that job and the shared Corepack/immutable-install + # action, then the stage revalidates the install and runs the same build + # and test commands. + # + # The version below is not maintained by hand: shell-analysis reads the + # release that job pins and fails unless this one matches, and the stage + # itself blocks on any other interpreter. Bumping CI without bumping + # this line is caught by a lint, not by a dispatch nobody ran. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.sha }} + + - uses: actions/setup-node@v4 + with: + node-version: "22.23.1" + + - uses: ./.github/actions/install-yarn-deps + with: + working-directory: ./solidity/ecdsa + + - name: Build and test the ECDSA contracts surface + env: + PR4109_EXPECTED_SOURCE_COMMIT: ${{ github.sha }} + run: | + mkdir -p ${{ github.workspace }}/rehearsal-evidence + EVIDENCE_DIR=${{ github.workspace }}/rehearsal-evidence \ + ./scripts/release/pr4109/rehearse.sh solidity-proofs + + - name: Upload solidity-proofs evidence + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: solidity-proofs-evidence-${{ github.sha }} + path: rehearsal-evidence/ + if-no-files-found: error + + rehearsal-platforms: + # One evidence record can speak only for the native image its runner + # executed. Derive the runner matrix from the detached, reviewed provenance + # rather than from a hand-maintained architecture list, and refuse a release + # publishing a platform this workflow has no native runner or isolated chain + # input for. + if: inputs.run_container_stages + needs: local-proofs + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.matrix.outputs.matrix }} + steps: + - name: Restore the release receipt + uses: actions/download-artifact@v4 + with: + name: rehearsal-evidence-${{ github.sha }} + path: ${{ runner.temp }}/rehearsal-evidence + + - name: Build the native-runner matrix from release provenance + id: matrix + env: + RECEIPT_DIR: ${{ runner.temp }}/rehearsal-evidence/attestation + REHEARSAL_CHAIN_INPUTS_B64: ${{ inputs.rehearsal_chain_inputs_b64 }} + run: | + provenance="$RECEIPT_DIR/release-provenance.json" + if [ ! -f "$provenance" ]; then + echo "BLOCKED: the release receipt carries no detached provenance;" >&2 + echo "there is no reviewed platform set from which to build the" >&2 + echo "native rehearsal matrix" >&2 + exit 3 + fi + + node - "$provenance" "$GITHUB_OUTPUT" <<'NODE' + const fs = require("fs"); + const [provenancePath, outputPath] = process.argv.slice(2); + const provenance = JSON.parse( + fs.readFileSync(provenancePath, "utf8") + ); + const supported = { + amd64: { + runner: "ubuntu-latest", + artifact_key: "amd64", + }, + arm64: { + runner: "ubuntu-24.04-arm", + artifact_key: "arm64", + }, + "arm64/v8": { + runner: "ubuntu-24.04-arm", + artifact_key: "arm64-v8", + }, + }; + + const encoded = String( + process.env.REHEARSAL_CHAIN_INPUTS_B64 || "" + ); + if (!encoded) { + console.error( + "BLOCKED: rehearsal_chain_inputs_b64 is absent; supply one " + + "base64-encoded platform-to-chain JSON document" + ); + process.exit(3); + } + if ( + !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/ + .test(encoded) + ) { + console.error( + "BLOCKED: rehearsal_chain_inputs_b64 is not canonical base64" + ); + process.exit(3); + } + + const decoded = Buffer.from(encoded, "base64"); + if ( + decoded.toString("base64").replace(/=+$/, "") !== + encoded.replace(/=+$/, "") + ) { + console.error( + "BLOCKED: rehearsal_chain_inputs_b64 does not round-trip as " + + "canonical base64" + ); + process.exit(3); + } + + const decodedJSON = decoded.toString("utf8"); + if (!Buffer.from(decodedJSON, "utf8").equals(decoded)) { + console.error( + "BLOCKED: rehearsal_chain_inputs_b64 is not valid UTF-8" + ); + process.exit(3); + } + let chainInputs; + try { + chainInputs = JSON.parse(decodedJSON); + } catch (error) { + console.error( + "BLOCKED: rehearsal_chain_inputs_b64 does not decode to JSON: " + + error.message + ); + process.exit(3); + } + + // JSON.parse keeps only the final value of a duplicate object + // member. Scan the already-validated JSON text with one key set per + // object so neither a repeated platform nor a repeated chain field + // can silently overwrite the value a reviewer thought was supplied. + const containers = []; + const duplicateInputKeys = []; + for (let index = 0; index < decodedJSON.length; index++) { + const character = decodedJSON[index]; + if (character === "\"") { + const start = index; + for (index++; index < decodedJSON.length; index++) { + if (decodedJSON[index] === "\\") { + index++; + continue; + } + if (decodedJSON[index] === "\"") break; + } + let next = index + 1; + while (/\s/.test(decodedJSON[next] || "")) next++; + if (decodedJSON[next] === ":") { + const key = JSON.parse( + decodedJSON.slice(start, index + 1) + ); + const keys = containers[containers.length - 1]; + if (keys.has(key)) { + duplicateInputKeys.push(key); + } else { + keys.add(key); + } + } + continue; + } + if (character === "{") containers.push(new Set()); + if (character === "[") containers.push(null); + if (character === "}" || character === "]") containers.pop(); + } + if (duplicateInputKeys.length > 0) { + console.error( + "BLOCKED: the rehearsal chain-input document repeats " + + "object member(s) [" + + Array.from(new Set(duplicateInputKeys)).sort().join(", ") + + "]" + ); + process.exit(3); + } + + if ( + !chainInputs || + typeof chainInputs !== "object" || + Array.isArray(chainInputs) + ) { + console.error( + "BLOCKED: the rehearsal chain-input document must be an " + + "object keyed by published platform" + ); + process.exit(3); + } + + const requiredChainFields = [ + "chain_id", + "cutover_block", + "eth_rpc_url", + "eth_ws_url", + ]; + const validateEndpoint = (platform, field, value, protocols) => { + if ( + typeof value !== "string" || + value.length === 0 || + value.trim() !== value || + /[\r\n\0]/.test(value) + ) { + console.error( + "BLOCKED: chain input [" + platform + "]." + field + + " must be one nonempty endpoint string" + ); + process.exit(3); + } + let endpoint; + try { + endpoint = new URL(value); + } catch (_) { + console.error( + "BLOCKED: chain input [" + platform + "]." + field + + " is not a URL" + ); + process.exit(3); + } + if (!protocols.includes(endpoint.protocol)) { + console.error( + "BLOCKED: chain input [" + platform + "]." + field + + " uses protocol [" + endpoint.protocol + "], expected " + + protocols.join(" or ") + ); + process.exit(3); + } + }; + const validateChain = (platform, chain) => { + if (!chain || typeof chain !== "object" || Array.isArray(chain)) { + console.error( + "BLOCKED: chain input [" + platform + "] must be an object" + ); + process.exit(3); + } + const fields = Object.keys(chain).sort(); + if ( + fields.length !== requiredChainFields.length || + fields.some( + (field, index) => field !== requiredChainFields[index] + ) + ) { + console.error( + "BLOCKED: chain input [" + platform + "] must contain " + + "exactly {" + requiredChainFields.join(",") + "}, got {" + + fields.join(",") + "}" + ); + process.exit(3); + } + validateEndpoint( + platform, + "eth_ws_url", + chain.eth_ws_url, + ["ws:", "wss:"] + ); + validateEndpoint( + platform, + "eth_rpc_url", + chain.eth_rpc_url, + ["http:", "https:"] + ); + if ( + !Number.isSafeInteger(chain.cutover_block) || + chain.cutover_block < 1 + ) { + console.error( + "BLOCKED: chain input [" + platform + + "].cutover_block must be a positive safe integer" + ); + process.exit(3); + } + if ( + typeof chain.chain_id !== "string" || + !/^[1-9][0-9]*$/.test(chain.chain_id) + ) { + console.error( + "BLOCKED: chain input [" + platform + + "].chain_id must be a positive decimal string" + ); + process.exit(3); + } + }; + + const include = []; + const seen = new Set(); + for (const image of provenance.images || []) { + const platform = String(image.platform || ""); + if (seen.has(platform)) { + console.error( + "BLOCKED: detached provenance publishes platform [" + + platform + "] more than once" + ); + process.exit(3); + } + seen.add(platform); + const config = supported[platform]; + if (!config) { + console.error( + "BLOCKED: detached provenance publishes platform [" + + platform + "], for which this workflow declares no native " + + "runner and isolated chain inputs" + ); + process.exit(3); + } + const chain = chainInputs[platform]; + if (chain === undefined) { + console.error( + "BLOCKED: the rehearsal chain-input document has no entry " + + "for published platform [" + platform + "]" + ); + process.exit(3); + } + validateChain(platform, chain); + include.push({ + platform, + ...config, + eth_ws_url: chain.eth_ws_url, + eth_rpc_url: chain.eth_rpc_url, + cutover_block: String(chain.cutover_block), + chain_id: chain.chain_id, + }); + } + if (include.length === 0) { + console.error( + "BLOCKED: detached provenance publishes no runtime platform" + ); + process.exit(3); + } + + const unpublishedInputs = Object.keys(chainInputs) + .filter((platform) => !seen.has(platform)) + .sort(); + if (unpublishedInputs.length > 0) { + console.error( + "BLOCKED: the rehearsal chain-input document contains " + + "unpublished platform(s) [" + unpublishedInputs.join(", ") + + "]; the mapping must match detached provenance one-to-one" + ); + process.exit(3); + } + + fs.appendFileSync( + outputPath, + "matrix=" + JSON.stringify({ include }) + "\n" + ); + NODE + + container-rehearsal: + name: Container rehearsal (${{ matrix.platform }}) + # The container stages need the immutable digests, a rehearsal chain, the + # per-node keys/configs, and the chain-side programs no repository can + # derive — the driver that originates protocol work, and the generator that + # produces the reconciliation, quiescence, and prior-reader evidence the + # rollback audit binds its verdict to, for the snapshot the drain leaves. + # Everything the runner can be given is given below; anything still missing + # reports BLOCKED (exit 3) naming the exact input, which is the truthful + # status for a rehearsal that cannot execute. + if: inputs.run_container_stages + needs: [local-proofs, rehearsal-platforms] + strategy: + fail-fast: false + matrix: ${{ fromJSON(needs.rehearsal-platforms.outputs.matrix) }} + runs-on: ${{ matrix.runner }} + env: + PRIOR_IMAGE_DIGEST: ${{ inputs.prior_image_digest }} + R1_IMAGE_DIGEST: ${{ inputs.r1_image_digest }} + PROBE_IMAGE_DIGEST: ${{ inputs.probe_image_digest }} + ETH_WS_URL: ${{ matrix.eth_ws_url }} + ETH_RPC_URL: ${{ matrix.eth_rpc_url }} + CUTOVER_BLOCK: ${{ matrix.cutover_block }} + CHAIN_ID: ${{ matrix.chain_id }} + # arm64 and arm64/v8 are separate published artifacts when both appear + # in provenance. Force Docker/Compose to resolve the exact matrix + # platform so two jobs on the same native runner do not silently execute + # the same child image. + DOCKER_DEFAULT_PLATFORM: linux/${{ matrix.platform }} + KEEP_ETHEREUM_PASSWORD: ${{ secrets.REHEARSAL_KEEP_ETHEREUM_PASSWORD }} + PR4109_BITCOIN_NETWORK: ${{ inputs.bitcoin_network }} + PR4109_PRIOR_VERSION: ${{ inputs.prior_version }} + PR4109_PRIOR_REVISION: ${{ inputs.prior_revision }} + PR4109_EVIDENCE_RECORD_SUFFIX: ${{ matrix.artifact_key }} + REHEARSAL_PLATFORM: ${{ matrix.platform }} + REHEARSAL_PLATFORM_KEY: ${{ matrix.artifact_key }} + # The container stages verify their own source binding before they emit + # or judge a record, exactly like every other proof stage. + PR4109_EXPECTED_SOURCE_COMMIT: ${{ github.sha }} + EVIDENCE_DIR: ${{ github.workspace }}/rehearsal-evidence + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.sha }} + + # Every provisioned input lands outside the checkout. The container + # stages refuse to produce evidence from a tree that diverges from the + # dispatched commit — untracked files included — so a keystore or an + # input bundle unpacked into the workspace would fail the very stage it + # exists to enable. The one exception is the evidence directory, which + # the commit's own .gitignore covers. + - name: Resolve the provisioning paths + run: | + { + echo "REHEARSAL_KEYSTORE_ROOT=$RUNNER_TEMP/rehearsal-keystore" + echo "KEYSTORE_DIR=$RUNNER_TEMP/rehearsal-keystore/$REHEARSAL_PLATFORM_KEY" + echo "REHEARSAL_INPUTS_DIR=$RUNNER_TEMP/rehearsal-inputs" + echo "REHEARSAL_AUDIT_TRUST=$RUNNER_TEMP/rehearsal-audit-trust.json" + echo "PR4109_WORK_DRIVER=$RUNNER_TEMP/rehearsal-inputs/work-driver" + # An executable, not a bundle of records: every rollback evidence + # record must name the aggregate checksum of the snapshot it speaks + # for, and that snapshot does not exist until this run has drained + # the fleet and copied its state out. A bundle unpacked here could + # not know a checksum computed later, and one that carried it + # anyway would be describing a drain that had not happened. + echo "PR4109_ROLLBACK_EVIDENCE_GENERATOR=$RUNNER_TEMP/rehearsal-inputs/rollback-evidence-generator" + # Not a program at all: the archived independent cryptographic + # review of the dependency revision go.mod resolves. It gates + # acceptance of a single-release record rather than execution of + # any step, so it is optional here and the step below unsets this + # path when the bundle carries no such record — a rehearsal + # without it still runs everything and simply produces a record + # acceptance refuses. + echo "PR4109_TSSLIB_REVIEW=$RUNNER_TEMP/rehearsal-inputs/tsslib-review" + # Written to, not read from: the rollback stage captures each + # drained node's state here straight out of the container it + # stopped. It holds live protocol state — key shares included — + # so it stays on the runner and is never archived. What a reviewer + # reads is the audit manifest each capture produces, which the + # stage writes into the evidence directory. + echo "STORAGE_SNAPSHOT_DIR=$RUNNER_TEMP/rehearsal-snapshots" + } >> "$GITHUB_ENV" + + # The per-node keys and configurations come from one repository secret + # holding a base64-encoded tar.gz with one + # //config.toml plus key material per rehearsal + # node — rehearsal-only throwaway keys, never production material. The + # platform directory lets each isolated chain carry its own deployed + # contract addresses. Without it the stage reports BLOCKED with the exact + # missing input instead of failing preflight opaquely. + - name: Provision the rehearsal keystore bundle + env: + REHEARSAL_KEYSTORE_BUNDLE_B64: ${{ secrets.REHEARSAL_KEYSTORE_BUNDLE_B64 }} + run: | + if [ -z "$REHEARSAL_KEYSTORE_BUNDLE_B64" ]; then + echo "BLOCKED: the REHEARSAL_KEYSTORE_BUNDLE_B64 secret is not" >&2 + echo "provisioned; store a base64-encoded tar.gz holding one" >&2 + echo "//config.toml and rehearsal-only" >&2 + echo "key material per node, then re-dispatch" >&2 + exit 3 + fi + mkdir -p "$REHEARSAL_KEYSTORE_ROOT" + printf '%s' "$REHEARSAL_KEYSTORE_BUNDLE_B64" \ + | base64 -d \ + | tar -xz -C "$REHEARSAL_KEYSTORE_ROOT" + chmod -R go-rwx "$REHEARSAL_KEYSTORE_ROOT" + if [ ! -d "$KEYSTORE_DIR" ]; then + echo "BLOCKED: the rehearsal keystore bundle has no" >&2 + echo "$REHEARSAL_PLATFORM_KEY/ directory for published platform" >&2 + echo "$REHEARSAL_PLATFORM; each platform needs configs naming its" >&2 + echo "own isolated chain's deployed contract addresses" >&2 + exit 3 + fi + echo "provisioned $(find "$KEYSTORE_DIR" -mindepth 1 -maxdepth 1 \ + -type d | wc -l | tr -d ' ') rehearsal node directories" + + # The rollback audit's Ethereum trust roots must not come from the + # evidence generator they authenticate. Provision them independently, + # one set per isolated platform chain, from a separate secret document. + # The values are validated before being written to GITHUB_ENV so the + # document cannot inject another environment assignment. + - name: Provision the independent rollback trust bundle + env: + REHEARSAL_AUDIT_TRUST_B64: ${{ secrets.REHEARSAL_AUDIT_TRUST_B64 }} + run: | + if [ -z "$REHEARSAL_AUDIT_TRUST_B64" ]; then + echo "BLOCKED: the REHEARSAL_AUDIT_TRUST_B64 secret is not" >&2 + echo "provisioned; store a base64-encoded JSON object keyed by" >&2 + echo "platform key, independently from the evidence generator" >&2 + exit 3 + fi + printf '%s' "$REHEARSAL_AUDIT_TRUST_B64" \ + | base64 -d >"$REHEARSAL_AUDIT_TRUST" + + node - "$REHEARSAL_AUDIT_TRUST" "$REHEARSAL_PLATFORM_KEY" \ + "$GITHUB_ENV" <<'NODE' + const fs = require("fs"); + const [path, platform, outputPath] = process.argv.slice(2); + const document = JSON.parse(fs.readFileSync(path, "utf8")); + const trust = document[platform]; + if (!trust || typeof trust !== "object" || Array.isArray(trust)) { + console.error( + "BLOCKED: the independent rollback trust bundle has no object " + + "for platform key [" + platform + "]" + ); + process.exit(3); + } + + const fields = [ + [ + "PR4109_WALLET_REGISTRY_ADDRESS", + "wallet_registry_address", + /^0x[0-9a-f]{40}$/, + ], + [ + "PR4109_RANDOM_BEACON_ADDRESS", + "random_beacon_address", + /^0x[0-9a-f]{40}$/, + ], + [ + "PR4109_FINALIZED_ETHEREUM_BLOCK_NUMBER", + "finalized_ethereum_block_number", + /^[1-9][0-9]*$/, + ], + [ + "PR4109_FINALIZED_ETHEREUM_BLOCK_HASH", + "finalized_ethereum_block_hash", + /^0x[0-9a-f]{64}$/, + ], + [ + "PR4109_CHAIN_EVIDENCE_PUBLIC_KEY", + "chain_evidence_public_key", + /^[0-9a-f]{64}$/, + ], + ]; + const assignments = []; + for (const [environment, member, pattern] of fields) { + const value = String(trust[member] || ""); + if (!pattern.test(value)) { + console.error( + "BLOCKED: rollback trust member [" + platform + "]." + + member + " is absent or malformed" + ); + process.exit(3); + } + assignments.push(environment + "=" + value); + } + fs.appendFileSync(outputPath, assignments.join("\n") + "\n"); + NODE + + # The inputs that exist outside this repository entirely, and both of + # them are programs rather than data. The fleet only reacts to chain + # events, so without a driver no ceremony exists to observe. The rollback + # audit reports namespace consistency and nothing about rollback safety + # unless it is given the live-chain and Bitcoin reconciliations, the + # node's own quiescence outcome, and the prior release's + # reader-compatibility result — and each of those must name the exact + # snapshot it speaks for, which is state this run produces by draining + # the fleet. So the generator is provisioned and executed after each + # capture instead of a bundle of records being unpacked before the fleet + # starts. Both are checked here rather than at the point of use, so a + # bundle missing one blocks before the fleet is started instead of + # halfway through a rehearsal. + # + # The secret is mutable and both programs produce readings that become + # release evidence, so an executable bit is not provenance. The bytes are + # hashed against scripts/release/pr4109/chain-inputs.sha256 — a reviewed, + # checked-in control — in the rehearsal's own preflight, before any node + # is started, and the digests are recorded into the evidence document. + # This step therefore establishes only that the bundle arrived with both + # members; the binding is the rehearsal's, because it is the party that + # runs them. + - name: Provision the rehearsal chain inputs bundle + env: + REHEARSAL_CHAIN_INPUTS_BUNDLE_B64: ${{ secrets.REHEARSAL_CHAIN_INPUTS_BUNDLE_B64 }} + run: | + if [ -z "$REHEARSAL_CHAIN_INPUTS_BUNDLE_B64" ]; then + echo "BLOCKED: the REHEARSAL_CHAIN_INPUTS_BUNDLE_B64 secret is" >&2 + echo "not provisioned; store a base64-encoded tar.gz holding" >&2 + echo "work-driver (executable, called with the phase name) and" >&2 + echo "rollback-evidence-generator (executable, called with the" >&2 + echo "service name, that node's identity audit manifest, and an" >&2 + echo "output directory to write the four rollback evidence" >&2 + echo "records into), optionally tsslib-review (the archived" >&2 + echo "independent cryptographic review of the dependency" >&2 + echo "revision go.mod resolves, which gates acceptance rather" >&2 + echo "than execution), record each member's reviewed SHA-256 in" >&2 + echo "scripts/release/pr4109/chain-inputs.sha256, then" >&2 + echo "re-dispatch" >&2 + exit 3 + fi + mkdir -p "$REHEARSAL_INPUTS_DIR" + printf '%s' "$REHEARSAL_CHAIN_INPUTS_BUNDLE_B64" \ + | base64 -d \ + | tar -xz -C "$REHEARSAL_INPUTS_DIR" + chmod -R go-rwx "$REHEARSAL_INPUTS_DIR" + + missing="" + for member in \ + "$PR4109_WORK_DRIVER" \ + "$PR4109_ROLLBACK_EVIDENCE_GENERATOR"; do + [ -x "$member" ] || missing="$missing $member(executable)" + done + if [ -n "$missing" ]; then + echo "BLOCKED: the chain inputs bundle is missing:$missing" >&2 + exit 3 + fi + echo "provisioned the work driver and the rollback evidence generator" + + # The review record is an acceptance input, so its absence is not a + # provisioning failure: the rehearsal runs every step either way. + # The path is cleared rather than left dangling, because a variable + # naming a file that is not there is a supplied record the + # rehearsal cannot read, and that does block. + if [ -f "$PR4109_TSSLIB_REVIEW" ]; then + echo "provisioned the archived dependency review record" + else + echo "PR4109_TSSLIB_REVIEW=" >> "$GITHUB_ENV" + echo "no dependency review record in the bundle; every step will" \ + "run and the emitted single-release record will not be" \ + "acceptable until one is supplied and pinned" + fi + + # The records these stages emit are measured against the compiled bounds + # the local-proofs stage attested at this same commit, and that receipt + # lives in that job's evidence artifact. Without it here every rehearsal + # blocks at its own emitter, which is the one failure mode a dispatch + # cannot diagnose from the log it archives. + - name: Restore the release-manifest attestation + uses: actions/download-artifact@v4 + with: + name: rehearsal-evidence-${{ github.sha }} + path: ${{ github.workspace }}/rehearsal-evidence + + - name: Require the attestation these records are measured against + run: | + dir="$EVIDENCE_DIR/attestation" + for part in derived-manifest.json reviewed-manifest.sha256 \ + source-commit.txt release-ready.txt; do + if [ ! -f "$dir/$part" ]; then + echo "BLOCKED: the local-proofs attestation did not arrive:" >&2 + echo "$dir/$part is absent, so nothing here proves the" >&2 + echo "reviewed release manifest still matches the compiled" >&2 + echo "bounds these records would be judged by" >&2 + exit 3 + fi + done + + # A receipt recording a release-ready manifest is a + # release-acceptance receipt, and acceptance refuses one that names + # no artifact. Asked here, against the receipt this job actually + # downloaded and before an image is pulled or a node started, + # because the alternative is a rehearsal that drives a fleet through + # every mandatory step and is then refused at its own emitter for an + # input the dispatch could have been given. + if [ "$(tr -d '[:space:]' <"$dir/release-ready.txt")" = "yes" ] && + [ ! -f "$dir/release-provenance.json" ]; then + echo "BLOCKED: the restored receipt records a release-ready" >&2 + echo "manifest and carries no detached release provenance, so" >&2 + echo "every record this rehearsal could produce would be" >&2 + echo "measured against a release nothing identifies;" >&2 + echo "re-dispatch with release_provenance_b64 set to the" >&2 + echo "provenance generated after the release build" >&2 + exit 3 + fi + echo "attestation restored for source $(cat "$dir/source-commit.txt")" + + - name: Preflight the rehearsal inputs + id: preflight + run: ./scripts/release/pr4109/rehearse.sh preflight + + - name: Exact-image single-release rehearsal + id: single_release + run: ./scripts/release/pr4109/rehearse.sh single-release + + # The cutover fleet stops before the rollback gate starts, and it stops + # here as well as inside the stage that owns it: a single-release stage + # that failed halfway never reached its own closing step, and the fleet + # it left behind is a release candidate watching the same rehearsal chain + # the rollback gate is about to declare quiet. The rollback stage + # enumerates the daemon and refuses to release the prior binary while any + # candidate is still attached to a network, so this step is what keeps + # that refusal from being the normal outcome rather than what establishes + # the barrier. Volumes stay: the state the drained fleet left is evidence. + - name: Stop the cutover fleet before the rollback gate + if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} + run: | + docker compose --project-name pr4109-single_release \ + --file ./scripts/release/pr4109/compose.rehearsal.yaml \ + stop || true + + # A refused cutover rehearsal is exactly when the rollback gate's + # evidence matters most, so it runs on the cutover's verdict being + # anything at all. Only a failed preflight stops it: that means the + # inputs never validated, and a rollback rehearsal on unvalidated inputs + # would produce a record about nothing. + - name: Homogeneous rollback rehearsal + if: ${{ !cancelled() && steps.preflight.outcome == 'success' }} + run: ./scripts/release/pr4109/rehearse.sh rollback + + # Both projects, whatever happened above: a rehearsal that failed + # mid-fleet otherwise leaves nodes running and volumes holding live + # protocol state on the runner. + - name: Tear down the rehearsal fleets + if: ${{ always() }} + run: | + for gate in single_release rollback; do + docker compose --project-name "pr4109-$gate" \ + --file ./scripts/release/pr4109/compose.rehearsal.yaml \ + down --volumes --remove-orphans || true + done + rm -rf "$STORAGE_SNAPSHOT_DIR" + + # The full per-platform archive contains identically named receipt and + # audit files, so it cannot be merged safely. Copy only the two top-level + # gate records into a dedicated artifact; their platform suffix makes + # every fan-in member unique. + - name: Stage this platform's records for aggregate validation + if: ${{ always() }} + run: | + aggregate="$RUNNER_TEMP/rehearsal-records" + mkdir -p "$aggregate" + count=0 + for record in "$EVIDENCE_DIR"/*.json; do + [ -f "$record" ] || continue + case "$(basename "$record")" in + *-"$REHEARSAL_PLATFORM_KEY"-*.json) ;; + *) + echo "record $record carries no runner platform suffix" >&2 + exit 1 + ;; + esac + cp "$record" "$aggregate/" + count=$((count + 1)) + done + echo "staged $count record(s) from $REHEARSAL_PLATFORM" + + # The rehearsal records and the audit manifests, whatever the verdict: + # a refused gate's account of why it was refused is the evidence a + # release decision most needs to read. + - name: Upload container rehearsal evidence + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: container-rehearsal-evidence-${{ github.sha }}-${{ matrix.artifact_key }} + path: rehearsal-evidence/ + if-no-files-found: error + + - name: Upload this platform's aggregate records + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: container-rehearsal-records-${{ github.sha }}-${{ matrix.artifact_key }} + path: ${{ runner.temp }}/rehearsal-records/ + if-no-files-found: warn + + aggregate-rehearsal-evidence: + # Per-platform jobs can validate only the records they emitted. Restore the + # one receipt, fan in every native runner's uniquely named gate records, and + # ask archive completeness exactly once over the resulting set. Run even + # after a matrix failure: a missing artifact or record is itself a failed + # mandatory release gate, not a reason to skip the aggregate verdict. + if: ${{ always() && inputs.run_container_stages && !cancelled() }} + needs: [local-proofs, container-rehearsal] + runs-on: ubuntu-latest + env: + EVIDENCE_DIR: ${{ github.workspace }}/rehearsal-evidence + PR4109_EXPECTED_SOURCE_COMMIT: ${{ github.sha }} + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.sha }} + + - name: Restore the release receipt + uses: actions/download-artifact@v4 + with: + name: rehearsal-evidence-${{ github.sha }} + path: ${{ github.workspace }}/rehearsal-evidence + + - name: Download every platform's gate records + uses: actions/download-artifact@v4 + with: + pattern: container-rehearsal-records-${{ github.sha }}-* + path: ${{ runner.temp }}/rehearsal-records + merge-multiple: true + + - name: Assemble the release evidence archive + run: | + mkdir -p "$EVIDENCE_DIR" + count=0 + for record in "$RUNNER_TEMP"/rehearsal-records/*.json; do + [ -f "$record" ] || continue + cp "$record" "$EVIDENCE_DIR/" + count=$((count + 1)) + done + echo "assembled $count top-level rehearsal record(s)" + + - name: Validate complete gate and platform coverage + run: ./scripts/release/pr4109/rehearse.sh validate-evidence + + - name: Upload aggregate rehearsal evidence + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: aggregate-rehearsal-evidence-${{ github.sha }} + path: rehearsal-evidence/ + if-no-files-found: error diff --git a/.github/workflows/cutover-scaffold-lint.yml b/.github/workflows/cutover-scaffold-lint.yml new file mode 100644 index 0000000000..d8806e1407 --- /dev/null +++ b/.github/workflows/cutover-scaffold-lint.yml @@ -0,0 +1,154 @@ +name: Cutover Scaffold Lint + +# The unconditional gate on the cutover rehearsal scaffold itself. +# +# The scaffold's checkers — the source-binding verifier and the +# evidence-record validator — decide what may be accepted as release +# evidence, but everything that exercises them lives in a manually +# dispatched workflow. Nothing therefore proved a change to them until +# somebody remembered to dispatch a rehearsal. This job runs on every push +# and pull request that touches those files: shell syntax, ShellCheck, +# actionlint over the scaffold's own workflows, all boundary self-tests, +# and the check holding the verifier's hand-written build-context +# classification to the rules .dockerignore really carries. It is +# deliberately cheap — no Docker image build, no Go test suite — so it can be +# required without slowing anything down. +# +# The path filters cover the scaffold plus the build inputs its trust model +# is derived from, because those decide what it accepts just as directly as +# its own code does: the build's ignore rules are what the verifier's context +# classification mirrors, the gitignore rules decide which working-tree paths +# count as divergence at all, and the Dockerfile and Makefile define the +# regeneration the verifier restores over rather than trusts. A change to any +# of them can widen what an image tree is allowed to be missing without +# touching a line under scripts/. +# +# Both ignore files the build could read are listed. Adding a +# Dockerfile-specific one retires every rule in the root .dockerignore for +# this build, which rewrites the whole context classification without +# touching the file the classification used to be derived from. +# +# These lists are not maintained by hand, and are not trusted by inspection: +# shell-analysis enumerates the inputs out of the commit itself — the three +# workflows, the Dockerfile the rehearsal workflow's build step really +# compiles, the ignore file that name selects, the root .dockerignore, every +# file under the scaffold directory, and every committed .gitignore and +# Makefile — and requires both lists below to run on each one. It reads them +# the way the workflow parser does, last matching entry winning, so an entry +# listed here and negated further down is not coverage either. Moving the +# build onto another Dockerfile, or adding a gen/ Makefile, without moving +# these entries with it fails that check. +# +# contracts-ecdsa.yml is listed for the same reason: the contracts stage's +# evidence claims to reproduce one of its jobs, and shell-analysis holds both +# that stage and the rehearsal workflow's own setup-node to the Node release +# that job pins. A bump there touches no line under scripts/ and would +# otherwise leave the claim standing over a run that no longer reproduces it. +# +# The pull_request trigger carries no branches or types restriction on +# purpose, and shell-analysis refuses one: push fires only after a branch has +# already moved, so the pull_request event is the only one that can stop a +# change to these inputs from merging unchecked, and every restriction of it +# exempts some pull request from the gate. +# +# The analysis step below is likewise read rather than found. shell-analysis +# requires it in a step's run: body — the only key that runs anything — as that +# body's last command, unconditioned, unexcused, under the runner's own shell, +# and with nothing around it that could swallow what it says. Keep the step's +# shell to plain commands: a pipeline, a `|| true`, a shell `if`, a `set` line, +# a `shell:` naming another interpreter, or a step that merely prints this +# command all fail the gate closed rather than quietly retiring it. +# +# What surrounds the step is read for the same reason, because the identical +# command reaches something else entirely when the environment or the tree +# changes under it: an env: block on the step, the job or the workflow, an +# assignment other than EVIDENCE_DIR written onto the invocation, a +# working-directory:, a job container:, and any run: step ahead of this one in +# the job all fail closed. Keep the preceding steps to actions, and pass +# EVIDENCE_DIR on the command line as below. +# +# A command ahead of the invocation in this step's own body is accepted, and is +# the one shape of that kind that is: the words before it are read for the +# shell they open, not for what they write, so a cp over the entrypoint passes +# and the accepted final command then runs the copy. Keep this body to the +# invocation alone. +# +# What none of that can prove is that this workflow ran, or that the run +# reporting success ran this file — the check lives behind the invocation it is +# checking, and the job producing the check is defined by the same commit under +# test. A branch-protection rule requiring the scaffold-lint check therefore +# holds the job name, not this analysis. Closing that needs a ruleset requiring +# a workflow that is not this one: a `workflows` entry names one source +# repository and one path, so an entry naming this file names something every +# pull request here can rewrite. The entry has to name a repository no pull +# request into keep-core can write to, pin it by commit SHA rather than by a +# branch or tag ref, and that pinned source has to carry the analysis itself +# rather than call back into this commit for it. This file stays advisory +# however that is configured, and a commit deleting it deletes its own +# enforcement. See scripts/release/pr4109/README.md. + +on: + push: + branches: + - main + # Kept in step with the pull_request list below; the workflow parser has + # no anchors, so the two are written out. + paths: + - "scripts/release/pr4109/**" + - ".github/workflows/cutover-rehearsal.yml" + - ".github/workflows/cutover-scaffold-lint.yml" + - ".github/workflows/contracts-ecdsa.yml" + - ".github/workflows/release.yml" + - ".dockerignore" + - "Dockerfile.dockerignore" + - ".gitignore" + - "**/.gitignore" + - "Dockerfile" + - "Makefile" + - "**/Makefile" + pull_request: + paths: + - "scripts/release/pr4109/**" + - ".github/workflows/cutover-rehearsal.yml" + - ".github/workflows/cutover-scaffold-lint.yml" + - ".github/workflows/contracts-ecdsa.yml" + - ".github/workflows/release.yml" + - ".dockerignore" + - "Dockerfile.dockerignore" + - ".gitignore" + - "**/.gitignore" + - "Dockerfile" + - "Makefile" + - "**/Makefile" + workflow_dispatch: + +permissions: + contents: read + +jobs: + scaffold-lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + # The source-binding self-test builds throwaway repositories and reads + # this checkout's history; the evidence-validator self-test drives the + # real validation stage over fixture records. + - uses: actions/setup-go@v5 + with: + go-version-file: "go.mod" + + - name: Analyze the rehearsal scaffold + run: | + mkdir -p ${{ github.workspace }}/rehearsal-evidence + EVIDENCE_DIR=${{ github.workspace }}/rehearsal-evidence \ + ./scripts/release/pr4109/rehearse.sh shell-analysis + + - name: Upload scaffold-lint evidence + # A failing analyzer's log is the output most needed for diagnosis. + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: cutover-scaffold-lint-${{ github.sha }} + path: rehearsal-evidence/ + if-no-files-found: error diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 7df458d649..013e83fdb9 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -14,14 +14,17 @@ jobs: steps: - uses: actions/checkout@v4 with: - # Fetch the whole history for the `git describe` command to work. + # Release notes need the previous tag and its history. fetch-depth: 0 - - name: Resolve versions + - name: Resolve release identity + env: + RELEASE_TRIGGER_REF: ${{ github.ref }} + RELEASE_TRIGGER_TAG: ${{ github.ref_name }} run: | - echo "version=$(git describe --tags --match 'v[0-9]*' HEAD)" \ - >> $GITHUB_ENV - echo "revision=$(git rev-parse --short HEAD)" >> $GITHUB_ENV + version="$(./scripts/release/pr4109/release-trigger-tag.sh "${RELEASE_TRIGGER_REF}" "${RELEASE_TRIGGER_TAG}")" + echo "version=${version}" >> "${GITHUB_ENV}" + echo "revision=$(git rev-parse HEAD)" >> "${GITHUB_ENV}" - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -124,13 +127,17 @@ jobs: with: fetch-depth: 0 - - name: Resolve versions - run: | - echo "version=$(git describe --tags --match 'v[0-9]*' HEAD)" >> $GITHUB_ENV - echo "revision=$(git rev-parse --short HEAD)" >> $GITHUB_ENV - echo "dockerhub_org=${DOCKERHUB_ORG:-thresholdnetwork}" >> $GITHUB_ENV + - name: Resolve release identity env: + RELEASE_TRIGGER_REF: ${{ github.ref }} + RELEASE_TRIGGER_TAG: ${{ github.ref_name }} DOCKERHUB_ORG: ${{ secrets.DOCKERHUB_ORG }} + run: | + version="$(./scripts/release/pr4109/release-trigger-tag.sh "${RELEASE_TRIGGER_REF}" "${RELEASE_TRIGGER_TAG}")" + # shellcheck disable=SC2129 + echo "version=${version}" >> "${GITHUB_ENV}" + echo "revision=$(git rev-parse HEAD)" >> "${GITHUB_ENV}" + echo "dockerhub_org=${DOCKERHUB_ORG:-thresholdnetwork}" >> "${GITHUB_ENV}" - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 @@ -149,14 +156,20 @@ jobs: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} + - name: Resolve Docker tags + id: docker-tags + run: | + { + echo 'tags<> "${GITHUB_OUTPUT}" + - name: Build and Push Docker Images uses: docker/build-push-action@v5 with: target: runtime-docker - tags: | - ${{ env.dockerhub_org }}/keep-client:latest - ${{ env.dockerhub_org }}/keep-client:${{ env.version }} - ${{ env.dockerhub_org }}/keep-client:mainnet + tags: ${{ steps.docker-tags.outputs.tags }} labels: | version=${{ env.version }} revision=${{ env.revision }} @@ -172,4 +185,4 @@ jobs: - name: Move Docker cache run: | rm -rf /tmp/.buildx-cache - mv /tmp/.buildx-cache-docker-new /tmp/.buildx-cache \ No newline at end of file + mv /tmp/.buildx-cache-docker-new /tmp/.buildx-cache diff --git a/.gitignore b/.gitignore index 0c2c04268b..74068aaf40 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,14 @@ # MacOS *.DS_Store +# Environment files +.envrc +.envrc.* + # Executables /keep-client +/participation-state-audit +/cutover-roster # IDEs .vscode/ @@ -46,6 +52,8 @@ node_modules/ # Yarn yarn-error.log +# Yarn 4 local state (not for zero-installs) +**/.yarn/install-state.gz # Solidity /solidity*/**/artifacts/ @@ -54,6 +62,9 @@ yarn-error.log /solidity*/**/typechain/ /solidity*/**/export.json +# rapid property-test failure artifacts (persisted local seeds, not a committed corpus) +testdata/rapid/ + # Go bindings generator # Note: Some specific _address files are committed as empty placeholders # to satisfy //go:embed directives during CI builds that don't run go generate @@ -69,6 +80,12 @@ out/ data/ logs/ storage/ + +# AI model run logs +strix_runs/ + +# Claude Code session state +.claude/ .ralph/ __pycache__/ .venv/ @@ -79,6 +96,12 @@ venv/ target/ dist/ .DS_Store - -# Yarn 4 local state (not for zero-installs) -.yarn/install-state.gz +build/ + +# Locally produced cutover rehearsal evidence (rehearse.sh). The rehearsal +# workflow writes into the workspace root instead of the script's own +# default, and every proof stage refuses to run on a tree that diverges from +# the dispatched commit — untracked files included — so both locations have +# to be ignore rules the commit itself carries. +scripts/release/pr4109/rehearsal-evidence/ +/rehearsal-evidence/ diff --git a/.golangci-ruleguard.rules.go b/.golangci-ruleguard.rules.go new file mode 100644 index 0000000000..8f5dd0cd3e --- /dev/null +++ b/.golangci-ruleguard.rules.go @@ -0,0 +1,49 @@ +//go:build ruleguard + +// Package gorules holds ruleguard rules enforced via gocritic in +// .golangci.yml. These are lint rules, not compiled into the project (the +// ruleguard build tag keeps them out of normal builds). +package gorules + +import "github.com/quasilyte/go-ruleguard/dsl" + +// txBoundsCheckedIndexing forbids raw index access on a transaction's +// Outputs/Inputs slices and steers callers to the bounds-checked accessors +// Transaction.OutputAt(i) / Transaction.InputAt(i). +// +// A variable index derived from one transaction used to index a separately +// fetched (untrusted) transaction's slice with no bounds check is the +// out-of-bounds panic class that crashes the client. Matching the index +// expression specifically (not len()/range/assignment of the field) keeps this +// precise; safe call sites (guarded constant indices, the accessor bodies +// themselves) carry a //nolint:gocritic with a one-line rationale. +func txBoundsCheckedIndexing(m dsl.Matcher) { + // Only variable (non-constant) indices are flagged: a constant index + // (e.g. Outputs[0]) is paired with an explicit len() guard at its call + // site and is not the OOB class. The findings were all variable indices + // derived from one transaction applied to a separately fetched one. + // + // The receiver is type-constrained to bitcoin.Transaction (value and + // pointer) so unrelated types that happen to have an Outputs/Inputs + // slice field — including regenerated gen/ code, where a //nolint + // cannot survive — do not trip the rule. + // + // Known, accepted limitations: the rule guards against accidental + // reintroduction, not adversarial code. Aliasing the slice first + // (outs := tx.Outputs; outs[i]) bypasses the pattern, as does any + // helper that returns the slice. Review remains the backstop for + // those shapes. + m.Import(`github.com/keep-network/keep-core/pkg/bitcoin`) + + m.Match(`$tx.Outputs[$i]`). + Where(!m["i"].Const && + (m["tx"].Type.Is(`bitcoin.Transaction`) || + m["tx"].Type.Is(`*bitcoin.Transaction`))). + Report(`use Transaction.OutputAt($i) instead of raw Outputs[$i] indexing to bounds-check untrusted input (OOB crash class)`) + + m.Match(`$tx.Inputs[$i]`). + Where(!m["i"].Const && + (m["tx"].Type.Is(`bitcoin.Transaction`) || + m["tx"].Type.Is(`*bitcoin.Transaction`))). + Report(`use Transaction.InputAt($i) instead of raw Inputs[$i] indexing to bounds-check untrusted input (OOB crash class)`) +} diff --git a/.golangci.yml b/.golangci.yml new file mode 100644 index 0000000000..bf064d0363 --- /dev/null +++ b/.golangci.yml @@ -0,0 +1,39 @@ +# golangci-lint configuration (v2). +# +# Scope is intentionally minimal: this is additive infrastructure that hosts a +# single project-specific rule (the Transaction-indexing ban). The existing +# dedicated CI jobs (go vet, gofmt, staticcheck, gosec) are left as-is and are +# NOT duplicated here, so this does not flood CI with pre-existing findings. +# Consolidation, if ever wanted, is a separate decision. +version: "2" + +linters: + default: none + enable: + - gocritic + settings: + gocritic: + # Run ONLY the ruleguard bridge: disable gocritic's default checks (they + # would flood CI with pre-existing style findings) and enable just + # ruleguard. + disable-all: true + enabled-checks: + - ruleguard + settings: + ruleguard: + failOn: all + rules: "${base-path}/.golangci-ruleguard.rules.go" + + exclusions: + rules: + # Tests legitimately construct and index transactions with known shapes; + # the untrusted-input OOB class only applies to production code paths. + - path: _test\.go + linters: + - gocritic + +issues: + # Surface every occurrence; a non-zero cap could silently mask a new + # violation behind the audited, annotated exceptions. + max-issues-per-linter: 0 + max-same-issues: 0 diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000000..a8faf4edf8 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,82 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0/). + +## [Unreleased] + +### Added +- Added a `golangci-lint` (gocritic/ruleguard) rule and `client-golangci` CI job that bans variable-indexed `tx.Outputs[i]`/`tx.Inputs[i]` access in non-test production code, steering callers to the bounds-checked `OutputAt`/`InputAt` accessors (#36) +- Added native Go fuzz targets across the beacon, network/security handshake, protocol, tBTC, tECDSA (DKG and signing), and bitcoin packages, asserting panic-free unmarshaling/deserialization of arbitrary untrusted input (#36) +- Added a non-blocking `client-race-test` CI job (race detector, scheduled and manual-dispatch only) (#36) +- Added the dev-only `github.com/quasilyte/go-ruleguard/dsl v0.3.23` tooling dependency (pinned via `tools.go`) used by the new lint rule (#36) +- ClusterFuzzLite CI integration: a per-PR fuzzing workflow (`code-change` mode, 300s, address sanitizer) and a scheduled nightly batch fuzzing workflow (`batch` mode, 1800s, daily cron + manual dispatch), backed by `.clusterfuzzlite/` build infra (Dockerfile, `project.yaml`, `build.sh` compiling 42 `Fuzz*` targets, plus a `check_targets.sh` drift guard). The per-PR workflow triggers on changes to `pkg/**`, `.clusterfuzzlite/**`, `go.mod`, `go.sum`, `.dockerignore`, `.github/workflows/cflite_pr.yml`, and `.github/workflows/cflite_batch.yml` (i.e. fuzzed code, fuzz build infra, dependencies, and the workflows themselves) (#37) +- New `target-sync` PR check that runs `.clusterfuzzlite/check_targets.sh` on every qualifying PR and fails the PR when a `Fuzz*` target under `pkg/` is not registered in `build.sh`; an unregistered target would otherwise silently get zero ClusterFuzzLite coverage (#37) +- `rapid` model-based property tests for retry participant selection (F-009): sub-multiset / all-or-nothing operator inclusion, minimum-seat retention, determinism, and operator-exclusion invariants for key generation and signing (#37) +- `rapid` property test for Ethereum redemption event conversion (F-014) asserting `convertRedemptionRequestedEvent` maps `TxMaxFee` from the event's `TxMaxFee` (not `TreasuryFee`) and reproduces all scalar fields (#37) +- `FuzzIdentityUnmarshal` fuzz target asserting the libp2p `identity.Unmarshal` never panics on arbitrary input (#37) +- Bitcoin transaction fuzzing improvements: a serialize/re-parse fixed-point property in `FuzzTransactionDeserialize` plus two pinned seed-corpus entries capturing parser quirks (trailing bytes accepted; witness-encoded zero-input txs colliding with the segwit marker on re-encode) (#37) +- Test-only dependency `pgregory.net/rapid v1.3.0` for property-based tests (#37) +- `.clusterfuzzlite/README.md` documenting the fuzz build setup; `.dockerignore` adjustments so the fuzz build context includes `.clusterfuzzlite/**` and committed protobuf code (`**/gen/pb/*.go`); and a `.gitignore` entry for `rapid` failure artifacts (`testdata/rapid/`) (#37) +- DKG test interceptor `Strategy` action API (`Strategy`, `Outbound`, `PassThrough`, `FromRules`, `NewNetworkWithStrategy`) supporting drop/mutate/duplicate/inject of messages, targetable per-sender and per-message-type; the prior `Rules` modify-or-drop API is retained via a `FromRules` back-compat adapter (#34) +- `dkgtest.RunTestWithStrategy` to run full DKG tests with a `Strategy`; existing `RunTest` is unchanged and now delegates through it (#34) +- `byzantine` test-harness package with predicate-based strategy constructors `Inactive`, `Withhold`, `Flood`, `Corrupt`, and `MatchAll` (#34) +- Unit tests for the new Strategy API and the `byzantine` constructors, full-DKG integration demos (Withhold/Flood/Corrupt), and an env-gated (`DETERMINISM_PROBE`) determinism probe (#34) +- CI job "Run Go race tests (Tier-2 interceptor)" running `go test -race` over `./pkg/internal/interception/...` and `./pkg/internal/byzantine/...` (#34) +- Test-only `pkg/internal/signingtest` harness (`RunTest`/`RunTestWithTimeout`) that runs the whole tECDSA signing protocol across a group of members over a local broadcast channel, with optional Byzantine interception, plus assertion helpers (`AssertSignatureGenerated`, `AssertMemberFailuresCount`, `AssertSameSignature`, `AssertNoDivergentSignatures`, `AssertValidSignature`) (#38) +- First whole-protocol signing integration tests in `pkg/tecdsa/signing/integration_test.go`: a happy-path case (5 members agree on one valid signature) and a Byzantine withhold case (member 2 inactive yields 0 signatures and 5 member failures), verifying that a malicious participant stalls signing (0 signatures, 5 member failures), and guarding against divergent or invalid signatures should a future change ever let members complete under this scenario (#38) +- Test-only Byzantine coordination harness for the tBTC wallet-coordination layer (`pkg/tbtc/coordination_byzantine_test.go`), injecting adversarial behavior by wrapping a specific operator's outbound channel with an `interception.Strategy`; includes an honest baseline scenario proving the interception seam does not perturb the protocol (#39) +- Withholding-leader test scenario asserting the safety invariant that a silent coordination leader (one that generates a proposal but never broadcasts it) can at worst cause denial of service (followers coordinate no action) but can never make followers act on an unreceived proposal or diverge onto split outcomes (#39) +- Byzantine integration test `TestByzantine_F008_ReconstructionPathExecutes` driving an honest quorum (groupSize 5, threshold 3) down the phase-12 reconstructed-share else-branch — the F-008 crash site — to corroborate that the contested beacon-DKG reconstruction nil-deref is a false positive (the missing-share branch does not form under real adversarial execution) (#40) +- `dkgtest` log-capture harness: thread-safe `capturingLogger` (records `Errorf` output that `MockLogger` discards), `(*dkgtest.Result).LoggedErrors()` accessor, and `dkgtest.AssertNoReconstructionGap` assertion that fails the test if the guard's "missing revealed share" error ever fires, making the absence of the F-008 gap observable (#40) +- Unit test `TestCapturingLoggerAndGapDetection` verifying the capture/detection logic (positive and negative cases) so the new assertion cannot be vacuously green (#40) +- `security/` directory with white-box pentest deliverables: architecture, attack surface, critical paths, crypto review, threat model, and smart-contracts analysis, plus 17 verified findings (F-01 through F-17) each with a code reference and status (#2) +- `SECURITY-BREAKING-CHANGES.md` documenting the F-02/F-03 wire-breaking changes, the BC-1..BC-10 / OV-1..OV-3 operator reference table, and the required coordinated-upgrade path (#2) +- Domain-separation info labels for ECDH key derivation: `gjkrEcdhInfo`, `dkgEcdhInfo` (`tecdsa-dkg`), and `signingEcdhInfo` (`tecdsa-sign`), plus a compile-time assertion that `MemberIndex` is 1 byte (#2) +- Tests for ECDH domain separation, `G1HashToPoint` determinism/wire-format, deduplicator concurrency, and Solidity reentrancy + storage layout (#2) +- Per-PR breaking-change, redeploy, and risk analysis notes under `keep-core-release///.md`, covering this repo's PRs (#2, #8, #9, #10, #11, #13) and upstream Threshold repos keep-core (#3945, #3948, #3952), keep-common (#16, #17), and tss-lib (#4, #5, #6) (#14) +- `keep-core-release///.md` directory convention for tracking post-merge release analysis going forward (#14) + +### Release scope (non-security) + +The following changes are included in this PR for convenience but are **not** part of the coordinated cryptographic flag-day (BC-1..BC-10). They do not affect DKG/signing wire compatibility and should be treated as independently reviewable operational/CI scope when bisecting or rolling back: + +- ClusterFuzzLite continuous fuzzing (`.clusterfuzzlite/`, `.github/workflows/cflite_pr.yml`, `.github/workflows/cflite_batch.yml`) (#37) +- `.github/workflows/client.yml` rewrite and contract-docs workflow updates (#8, #37) +- Kubernetes dev Ropsten statefulset/service edits and `eth-tx-rpc-ws-networkpolicy.yaml` (#8) +- Deletion of `infrastructure/kube/keep-prd/monitoring/monitoring-ingress.yaml` (#8) +- Private-testnet bundle guide update under `infrastructure/eth-networks/` (#8) + +### Changed +- Re-landed the Tier 0 (lint rule, race-detector CI job, bounds-checked accessors) and Tier 1 (native fuzz targets) portions of the previously reverted #33 testing/correctness hardening work as a single consolidated changeset, corresponding to the original PRs #29 and #30; the ClusterFuzzLite continuous fuzzing (#31) and rapid property tests (#32) are NOT included in this PR (#36) +- Nightly scheduled `-race` CI job: timeout raised from 30m to 60m, and on scheduled-run failure it now upserts a labeled GitHub issue (`race-detector-failure`); behavior is CI-only and gated to scheduled runs (#37) +- Narrowed the ruleguard lint rule for raw `Outputs[$i]`/`Inputs[$i]` indexing to fire only on `bitcoin.Transaction` / `*bitcoin.Transaction`, reducing false positives on unrelated and generated types (#37) +- `dkgtest` DKG test runs now share a single `capturingLogger` across member goroutines instead of constructing a per-call `MockLogger`, adding mutex-synchronized error capture during test execution; only `Errorf` behavior changes (capture vs discard), all other log levels are unchanged and no production protocol behavior is affected (#40) +- **BREAKING (wire):** Changed DKG session-ID format to `dkg--` (typed prefix and fixed-width attempt number) and signing session-ID format to `signing---`. The fixed-width formats guarantee every session ID clears tss-lib's 16-byte minimum-length floor, but are incompatible with the pre-hardening `-` form, so un-upgraded peers compute mismatched session IDs (#8) +- **BREAKING (behavioral):** Made the signing session ID depend on the attempt start block (`announcementEndBlock`) in addition to message digest and attempt number, introducing a new cross-node agreement requirement: even same-version peers that disagree on the attempt start block compute different session IDs and fail to interoperate (#8) +- Computed the session ID once per attempt and threaded it through attempt parameters so the announcer and the protocol cannot drift apart on the GG20 session binding (#8) +- `signing.NewLocalParty(...)` is now called with an additional `fullBytesLen` argument (`(Curve.Params().N.BitLen()+7)/8`). In the hardened tss-lib this parameter is variadic, so existing 5-argument callers still compile; omitting it, however, changes signing message byte-width / leading-zero handling, so this is a behavioral (not compile-breaking) change (#8) +- Added `pull-requests: read` (and job-level `contents: read`) permissions to path-filter jobs in CI workflows so PR change detection runs under `GITHUB_TOKEN` (#8) +- Added tests covering the new session-ID formats, the minimum entropy width, and the session-nonce derivation (`SHA512_256` of the session ID) for DKG and signing (#8) +- `ephemeral.PrivateKey.Ecdh` now takes an `info []byte` parameter and derives the symmetric key with HKDF-SHA256 instead of SHA-256; this changes the exported signature (compile break for external callers) and the derived session key (wire-incompatible with older nodes) (#2) +- `altbn128.G1HashToPoint` reimplemented from try-and-increment to a bounded counter-based `SHA-256(m || ctr)` (max 64 attempts); it produces a different G1 point for the same input (consensus-incompatible) and now panics if no valid point is found within the bound (#2) +- `RandomBeacon` relay-entry gas offset `_relayEntrySubmissionGasOffset` raised from 11250 to 13450 to account for the reentrancy-guard SSTOREs (mirrored in the test fixture) (#2) +- Enabled `storageLayout` output selection in the random-beacon Hardhat config, removed `scryptsy` from `yarn.lock`, and added `.envrc*`, `strix_runs/`, and `.claude/` to `.gitignore` (#2) +- **Operator action (temporary compatibility):** the `clientInfo.port` default is retained at `9601` for this coordinated security release so the client-info HTTP server (`/metrics` and `/diagnostics`) stays reachable through the cutover — the primary evidence channel for a node's exact revision and stranded-peer state must not go dark during deployment. Explicit `clientInfo.port = 0` still disables the server; the endpoint is unauthenticated and must be reached only over a trusted network path. Operators must commit an explicit `clientInfo.port` value and migrate every scrape target onto its trusted path; the follow-up R2 release flips the default back to `0` only after the tracked monitoring-migration exit criteria are met (see the monitoring-migration tracking issue for owner and dated expiry — **TODO: file the tracking issue and link it here before merge**; proposed title/body drafted for review in `.ralph/spec/draft-migration-issue.md`) (#2) +- **Operator action required:** renamed the libp2p peer-count metric from `connected_bootstrap_count` to `connected_wellknown_peers_count` to match bootstrap removal (#3909); update dashboards and alerts that query the old name (#3909) + +### Fixed +- Test interceptor invoked the interception rule twice per `Send`; it is now invoked exactly once per send under a mutex (#34) +- Test interceptor silently dropped the `retransmissionStrategy` vararg; it is now forwarded to the underlying delegate (#34) +- Data race in `dkgtest` where member goroutines appended to `memberFailures` without synchronization; the append is now guarded by the existing mutex (#34) +- `tbtc` deduplicator notify methods (`notifyDKGStarted`, `notifyDKGResultSubmitted`, `notifyWalletClosed`) now use a single atomic `cache.Add` instead of non-atomic check-then-act, fixing a TOCTOU race (#2) + +### Security +- Hardened transaction parsing against out-of-bounds crashes on untrusted/malformed Bitcoin-node responses: the SPV redemption and moved-funds-sweep paths now use bounds-checked `OutputAt` accessors and return a wrapped error instead of panicking when a node-supplied transaction has insufficient outputs (#36) +- **BREAKING (protocol fork):** Bound tECDSA DKG and signing session IDs into the TSS layer via `SetSessionNonceBytes`, deriving a fail-closed, session-specific GG20 proof nonce (`SHA512_256` of the session ID) for every ceremony. Combined with the changed session-ID formats and the hardened tss-lib pin, mixed-version peers in the same DKG or signing ceremony now derive different session IDs and fail proof verification. Upgrade the whole network at once; do not roll out partially (#8) +- **BREAKING (runtime contract):** `signing.Execute` and the tECDSA DKG `Execute` now thread the caller-supplied session ID into tss-lib's fail-closed minimum-length check. The hardened tss-lib (`tss/params.go`) panics if a session ID is shorter than 16 bytes; keep-core's own callers clear this via the new fixed-width formats, but an external Go caller passing a short or custom session ID will now panic at runtime even though the exported function signatures are unchanged (#8) +- Pinned the `threshold-network/tss-lib` replacement to commit `86bd1a375cc0` (`v0.0.0-20260615180949-86bd1a375cc0`), integrating the upstream hardening branch (threshold-network/tss-lib#2 through #7): GG20 proof transcript tagging/session binding, fail-closed positive `SessionNonce` enforcement, a 16-byte `SetSessionNonceBytes` minimum-length floor, ECDSA `fullBytesLen` signing validation, MtA/range/Paillier proof hardening, and non-canonical EC point rejection (#8) +- Lengthened signing session IDs to include a typed prefix and the attempt start block so repeated same-digest ceremonies no longer reuse the GG20 proof context (#8) +- Added an inline reentrancy guard (`nonReentrant` modifier, `_reentrancyStatus` storage slot, `ReentrantCall` error) to both `RandomBeacon.submitRelayEntry` entrypoints (#2) diff --git a/Dockerfile b/Dockerfile index 97181a29ef..a219e4e5fa 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM golang:1.24-alpine3.21 AS build-sources +FROM golang:1.25.10-alpine3.23 AS build-sources ENV GOPATH=/go \ GOBIN=/go/bin \ @@ -6,7 +6,8 @@ ENV GOPATH=/go \ APP_DIR=/go/src/github.com/keep-network/keep-core \ TEST_RESULTS_DIR=/mnt/test-results \ BIN_PATH=/usr/local/bin \ - LD_LIBRARY_PATH=/usr/local/lib/ + LD_LIBRARY_PATH=/usr/local/lib/ \ + GOTOOLCHAIN=auto # TODO: Remove perl once go-ethereum is upgraded to 1.11. # See pkg/chain/ethereum/tbtc/gen/Makefile and after_abi_hook for details. @@ -88,7 +89,7 @@ RUN GOOS=linux make build \ version=$VERSION \ revision=$REVISION -FROM alpine:3.21 as runtime-docker +FROM alpine:3.23 as runtime-docker ENV APP_NAME=keep-client \ APP_DIR=/go/src/github.com/keep-network/keep-core \ @@ -108,9 +109,10 @@ CMD [] # # Build Binaries # -FROM golang:1.24-bullseye AS build-bins +FROM golang:1.25.10-bookworm AS build-bins -ENV APP_DIR=/go/src/github.com/keep-network/keep-core +ENV APP_DIR=/go/src/github.com/keep-network/keep-core \ + GOTOOLCHAIN=auto WORKDIR $APP_DIR diff --git a/Makefile b/Makefile index ab468ae08f..b47a8e94c2 100644 --- a/Makefile +++ b/Makefile @@ -146,4 +146,35 @@ cmd-help: build @echo '$$ $(app_name) start --help' > docs/resources/client-start-help ./$(app_name) start --help >> docs/resources/client-start-help -.PHONY: all development sepolia download_artifacts generate gen_proto build cmd-help release build_multi +# verify-cutover is the repository-owned automation-loop completion subset for +# changes to the coordinated cutover. It deliberately does not stand in for +# the separately evidenced local-proofs, static-analysis, or solidity-proofs +# release stages. verify-cutover-release-local composes all repository-local +# stages; neither target replaces the exact-image cutover and rollback gates. +EVIDENCE_DIR ?= $(CURDIR)/rehearsal-evidence + +verify-cutover-go: + go build ./... + go vet ./... + go test -timeout 15m ./... + go test -race -timeout 5m ./pkg/protocol/... + +verify-cutover: verify-cutover-go + @mkdir -p "$(EVIDENCE_DIR)" + EVIDENCE_DIR="$(EVIDENCE_DIR)" \ + ./scripts/release/pr4109/rehearse.sh shell-analysis + @echo 'PASS: cutover automation subset: build+vet+unit-suite+participation-race+scaffold-shell-analysis green' + +verify-cutover-release-local: verify-cutover-go + @mkdir -p "$(EVIDENCE_DIR)" + EVIDENCE_DIR="$(EVIDENCE_DIR)" \ + ./scripts/release/pr4109/rehearse.sh local-proofs + EVIDENCE_DIR="$(EVIDENCE_DIR)" \ + ./scripts/release/pr4109/rehearse.sh static-analysis + EVIDENCE_DIR="$(EVIDENCE_DIR)" \ + ./scripts/release/pr4109/rehearse.sh solidity-proofs + EVIDENCE_DIR="$(EVIDENCE_DIR)" \ + ./scripts/release/pr4109/rehearse.sh shell-analysis + @echo 'PASS: repository-local cutover release stages green; exact-image rehearsals not run' + +.PHONY: all development sepolia download_artifacts generate gen_proto build cmd-help release build_multi verify-cutover-go verify-cutover verify-cutover-release-local diff --git a/README.adoc b/README.adoc index f38d0c320a..6f2527f544 100644 --- a/README.adoc +++ b/README.adoc @@ -82,6 +82,7 @@ keep-core/ Dockerfile main.go, *.go docs/ + security/ <7> solidity/ <1> ecdsa/ random-beacon/ @@ -115,3 +116,6 @@ keep-core/ `gen/`. This subpackage should contain a single file, `gen.go`, with a `// go:generate` annotation to trigger appropriate code generation. All code generation is done with a single invocation of `go generate` at build time. +<7> Whitebox security analysis for external pentesters: architecture, attack + surface, critical paths, cryptographic review, smart contract security, and + threat model. See link:security/[`security/`]. diff --git a/SECURITY-BREAKING-CHANGES.md b/SECURITY-BREAKING-CHANGES.md new file mode 100644 index 0000000000..552f4e4f4a --- /dev/null +++ b/SECURITY-BREAKING-CHANGES.md @@ -0,0 +1,339 @@ +# Security Fix Breaking Changes + +This document tracks breaking cryptographic changes introduced by the security +remediation branch. Each change alters wire-level or key-derivation behavior +and requires a **coordinated network upgrade** -- all nodes must upgrade before +the new code activates. Rolling upgrades will cause protocol failures. + +--- + +## F-02 -- Hash-to-Curve: bounded counter-based approach (G1HashToPoint) + +**File:** `pkg/altbn128/altbn128.go` + +**What changed:** + +`G1HashToPoint` previously incremented a candidate x coordinate until a valid +G1 point was found (try-and-increment). The number of iterations depended on +the hash output, creating a timing side channel. + +The function now uses a fixed counter suffix appended to the input before +hashing: `SHA-256(message || counter)` for counter in `[0, 63]`. Each +iteration performs identical work, bounding (but not normalizing) timing across +inputs: the loop exits on the first valid point, so execution time still varies +with how many counters are tried. Using 64 candidate counters gives a failure +probability of `(1/2)^64 ≈ 5e-20`. + +**Why it breaks:** + +The counter-based approach produces a different x candidate for every input +than the try-and-increment approach. The same byte string will map to a +different G1 point. + +**Impact:** + +Any distributed protocol that relies on consistent G1HashToPoint output across +nodes (e.g., BLS signature aggregation in the random beacon DKG) will fail if +nodes run mismatched versions. + +**On-chain consumer:** + +The new counter-based `G1HashToPoint` also diverges permanently from the +on-chain `AltBn128.g1HashToPoint` +(`solidity/random-beacon/contracts/libraries/AltBn128.sol`), which keeps the +original try-and-increment mapping fixed in the deployed contract bytecode. A +client-side (Go node) upgrade cannot change that bytecode, so Go<->chain +agreement for the on-chain consumer path -- `BLS.verifyBytes` +(`libraries/BLS.sol`), reached from `RandomBeacon.reportUnauthorizedSigning` -- +can never be restored by upgrading the client alone. + +In practice this is not an operational concern: that path has no in-repo +production callers. The only Go code that maps a raw byte message to a G1 point +this way is the `bls.Sign` / `bls.Verify` byte-message helpers in +`pkg/bls/bls.go`, which have no callers in the repository and are effectively +deprecated, and the generated `RandomBeacon.reportUnauthorizedSigning` binding +is never invoked by node logic. The consumer that matters operationally is the +off-chain GJKR Pedersen `H` generator +(`pkg/beacon/gjkr/protocol_parameters.go`) -- a node-to-node concern in which +every group member must derive the same `H`, which the coordinated upgrade +below guarantees. + +**Mitigation / upgrade path:** + +1. Schedule a hard-fork block or protocol version bump. +2. Deploy the new binary to all nodes simultaneously at the upgrade height. +3. Verify with a coordinated test on a staging network first. + +**Follow-up (tracked in GH issue):** + +Replace with a constant-time RFC 9380 SWU implementation to eliminate the +remaining non-constant-time modular square root. See: +https://github.com/tlabs-xyz/keep-core-security/issues/4 + +--- + +## F-03 -- ECDH key derivation: SHA-256 replaced with HKDF-SHA256 + +**File:** `pkg/crypto/ephemeral/symmetric_key.go` (and all callers) + +**What changed:** + +`PrivateKey.Ecdh()` previously derived a 32-byte session key as +`SHA-256(shared_secret)` -- using the raw ECDH output as key material with no +domain separation. + +The function now uses HKDF-SHA256 (RFC 5869): + +``` +key = HKDF-Extract+Expand(ikm=shared_secret, salt=nil, info=context_label) +``` + +The `info` parameter binds the derived key to the specific protocol and +peer pair. Each callsite passes a label encoding: + +- A protocol prefix (`gjkr`, `tecdsa-sign`, `tecdsa-dkg`) +- The canonical (sorted) pair of member IDs, each encoded as a single byte + +This ensures keys derived for different protocols or peer pairs are +cryptographically independent, even if the ECDH shared secret is the same. + +**Invariant:** Member IDs are encoded as a single byte each. This relies on +`group.MemberIndex` being a `uint8` (max member index 255). A compile-time +assertion in `pkg/protocol/group/group.go` enforces this; if the type is ever +widened, the `*EcdhInfo` helpers must switch to a width-independent encoding +(e.g. `binary.BigEndian.PutUint16`) in the same coordinated upgrade as F-03, +otherwise members whose IDs collide modulo 256 will derive identical keys. + +**Callsites updated:** + +| File | Count | +|------|-------| +| `pkg/beacon/gjkr/protocol.go` | 4 | +| `pkg/tecdsa/signing/protocol.go` | 1 | +| `pkg/tecdsa/dkg/protocol.go` | 1 | + +**Why it breaks:** + +HKDF with a non-empty `info` label produces a different 32-byte key than +`SHA-256(shared_secret)` for the same ECDH shared secret. Two nodes running +mismatched versions will derive different session keys and fail to decrypt each +other's shares. + +**Impact:** + +Any phase of the GJKR DKG, tECDSA DKG, or tECDSA signing protocol that +involves peer-to-peer encrypted share exchange will fail if nodes run +mismatched versions. This covers the full distributed key generation and +signing flows. + +**Mitigation / upgrade path:** + +1. Schedule a hard-fork block or protocol version bump. +2. Deploy the new binary to all nodes simultaneously at the upgrade height. +3. Verify with a coordinated test on a staging network first. +4. No on-chain data migration is required -- the ECDH keys are ephemeral + (generated fresh each session) and not persisted. + +--- + +## Security release operator reference (BC-1..BC-10, OV-1..OV-3) + +The table below is the operator-facing index for the coordinated security +release (`security-release/candidate-1`). Items marked **breaking** require a +flag-day upgrade of every participant in the same DKG or signing ceremony. +Operator-visible (OV) items do not change wire formats but may require config or +monitoring updates. + +### Breaking changes + +| ID | Area | What breaks | Who must act | +|----|------|-------------|--------------| +| **BC-1** | tss-lib | Fiat-Shamir / proof challenges use tagged hashing + session binding; old and new proofs **do not cross-verify** | **All operators simultaneously** | +| **BC-2** | tss-lib + keep-core | `SetSessionNonce` / `SetSessionNonceBytes` **mandatory** before keygen/signing `Start()`; session ID must be ≥16 bytes | keep-core wires this; external callers with short IDs **panic** | +| **BC-3** | tss-lib + keep-core | ECDSA signing requires positive `fullBytesLen` at construction (panic if omitted/zero) | keep-core passes curve-order byte width | +| **BC-4** | keep-core | **Session ID formats changed** (wire): DKG `dkg--`; signing `signing---` | All parties in a ceremony | +| **BC-5** | keep-core | Signing session ID now includes **attempt start block** — peers disagreeing on block derive different IDs | Coordinator / announcer agreement | +| **BC-6** | keep-core | `ephemeral.PrivateKey.Ecdh(info []byte)` — **compile break** + HKDF-derived keys differ (wire-incompatible); see **F-03** above | Any external code calling the old signature | +| **BC-7** | keep-core | `G1HashToPoint` reimplemented — **different G1 point** for the same input; see **F-02** above | Beacon / crypto paths using hash-to-curve | +| **BC-8** | keep-core | `PrepareForSigning` returns `(wi, bigWs, err)` — **compile break** for callers | Go integrators (no in-tree keep-core callers found) | +| **BC-9** | keep-core | Bootstrap removal (#3909): embedded well-known peers + **AllowList decoupling** — all peers pass `IsRecognized()` | Operators with custom bootstrap config | +| **BC-10** | keep-core | RandomBeacon **new storage slot** for the reentrancy guard (append-only bytecode change). RandomBeacon is **directly deployed, not proxied**, so this activates **only** by deploying a new RandomBeacon and cutting over to its address — never by an in-place / proxy implementation swap | Only if this release **redeploys RandomBeacon**: perform the address cutover (see the BC-10 note below). If there is no beacon redeployment, BC-10 is **staged but not activated** on the existing deployment | + +### Operator-visible (non-breaking wire) + +| ID | Change | Operator action | +|----|--------|-----------------| +| **OV-1** | Metrics/diagnostics **temporary compatibility default**: `clientInfo.port` stays `9601` for this coordinated release (HTTP server on) so a node's exact revision and stranded-peer evidence stay visible through the cutover; explicit `clientInfo.port = 0` disables it. The follow-up R2 release flips the default back to `0` after the monitoring migration. | Commit an explicit `clientInfo.port` value now, expose it only over a trusted path, and migrate scrape targets before R2 | +| **OV-2** | Metric rename: `connected_bootstrap_count` → `connected_wellknown_peers_count` | Update Grafana/Prometheus dashboards and alerts | +| **OV-3** | `--network.bootstrap=true` deprecated (warning only) | Remove from config when convenient | + +**BC-10 note — RandomBeacon is directly deployed, not a proxy.** +`solidity/random-beacon/deploy/04_deploy_random_beacon.ts` calls +`deployments.deploy("RandomBeacon", …)` with constructor arguments and linked +libraries and **no `proxy` option**; there is no implementation-upgrade path. +The reentrancy-guard storage slot is therefore compiled into the RandomBeacon +bytecode and cannot be added to an already-deployed RandomBeacon by swapping a +proxy implementation. It becomes active **only** when a new RandomBeacon is +deployed and the network cuts over to the new address. Do **not** treat BC-10 as +a "beacon proxy upgrade": + +- **If this release includes a RandomBeacon redeployment:** follow a separately + reviewed migration runbook covering the new address, dependency wiring + (sortition pool, staking, DKG validator, ReimbursementPool authorization and + funding), ownership/governance, consumer references, and post-deployment + validation. This is a fresh deployment + cutover, not an in-place upgrade. +- **If RandomBeacon is not redeployed in this release:** BC-10 ships as a staged + bytecode change that is **not activated** on the existing deployment; no + operator action is required for it, and no existing reentrancy behavior + changes until a future beacon deployment. + +This distinguishes RandomBeacon from legitimately proxied components (e.g. +`LightRelayMaintainerProxy`), which this row does not cover. + +**F-09 note — RandomBeacon relay-entry reimbursement offset (design decision, +approved 2026-07-24).** Both `submitRelayEntry` overloads share a single +`_relayEntrySubmissionGasOffset = 13_450` +(`contracts/RandomBeacon.sol:475,1072,1138`; fixture +`test/fixtures/index.ts:59`). The offset was raised from `11_250` to `13_450` to +cover ~2,118 gas of reimbursement work that executes **after** the in-function +`gasStart - gasleft()` snapshot — the inline `nonReentrant` guard writes +`_reentrancyStatus` after the function body, and the reimbursement call itself is +partly unmeasured — plus headroom, tuned for the heavier +`submitRelayEntry(bytes,uint32[])` overload. + +- **Structural asymmetry (observed).** The heavier overload's `uint32[64]` + `membersIDs` argument is charged as intrinsic **calldata** gas *before* the + in-function snapshot and is therefore never measured; the lighter + `submitRelayEntry(bytes)` overload carries none of that calldata, so the shared + offset structurally **over-reimburses** the lighter overload by a fixed + ~9,563 gas. This is a property of the single-offset design, not a defect. +- **Decision (approved 2026-07-24).** Keep one shared offset rather than splitting it into two + governance-settable offsets. Rationale: (1) avoids adding a second storage slot + plus governance setter and the associated upgrade/migration surface on a + security-release contract; (2) the only harmful direction — + **under-reimbursement** — never occurs on either overload at `13_450` (the + submitter is always at least made whole); (3) over-reimbursement is bounded and + paid from the operator-funded `ReimbursementPool`, never from user funds; + (4) governance may still retune the offset post-deployment through the existing + `updateGasParameters` (`onlyGovernance`) path if measurements change. +- **Enforced invariants** (`test/RandomBeacon.Relay.test.ts`, + `test/RandomBeacon.StorageLayout.test.ts`): no under-reimbursement on either + overload at `13_450`; heavier-overload over-reimbursement ≤ **5,000** gas + (`TUNED_OVER_REIMBURSEMENT_GAS_TOLERANCE`); lighter-overload over-reimbursement + ≤ **10,000** gas (`BYTES_ONLY_OVER_REIMBURSEMENT_CEILING_GAS`, bracketing the + measured ~9,563 with headroom so it cannot silently grow); negative control — + the heavier overload **under-reimburses at the pre-fix `11_250` offset** + (proving the ~2,200-gas fix is necessary and that the test is sensitive to it), + while the lighter overload stays fully reimbursed even at `11_250` (+7,363 gas), + confirming the fix is not needed for that path; and a storage-layout regression + pinning the slot and the `13_450` value. Gas figures are measured under the + pinned Hardhat compiler/optimizer/EVM-hardfork settings and must be remeasured + if any of those change. +- **Release gate.** This shared-offset design and its 5,000 / 10,000-gas + over-reimbursement ceilings require contract/security-owner sign-off before + release: `[x]` approved (2026-07-24). + +**tss-lib pin (this release):** `github.com/threshold-network/tss-lib@v0.0.0-20260615180949-86bd1a375cc0` (`86bd1a3`). + +**tECDSA signing copylock fix (this candidate, reviewed and accepted 2026-07-24).** +Merging current `main` exposed a `go vet` copylock failure in +`pkg/tecdsa/signing/member.go`: tss-lib's `signing.NewLocalParty` requires a +value-typed result channel and delivers via `end <- *round.data` +(`ecdsa/signing/finalize.go`), so every consumer must copy the lock-bearing +`common.SignatureData` (a protobuf message with a `DoNotCopy` marker) on +receive. `finalizingMember.receiveTSSResult` performs that single unavoidable +receive through the type-safe generic helper `receiveFromChannel` (no +reflection), then re-homes only the signature-relevant fields (`Signature`, +`SignatureRecovery`, `R`, `S`, `M`) into a freshly allocated `SignatureData` +built with a composite literal. The returned value owns a brand-new, never-locked +`MessageState`, so `go vet` has nothing to flag, and the transient received +value (with its copied lock) is discarded at the function boundary rather than +propagated. This is a receive-side mechanical change only — it does not alter +session handling, message content, or any cryptographic computation. It is +reviewed and accepted separately from, and does not substitute for, the +external `tss-lib` dependency security audit tracked as a separate release +action item above; the ideal upstream fix (forking tss-lib's channel to a +pointer type) is intentionally out of scope for this release and tracked +separately. + +--- + +## Coordinated upgrade (flag-day) requirement + +These changes activate by code alone. There is no on-chain version gate and no +peer-version negotiation: an upgraded node has no runtime switch to fall back to +the old key-derivation, session-ID, or hash-to-point behavior when it meets an +un-upgraded peer. The whole set of nodes taking part in a given ceremony must +therefore be upgraded together -- a flag-day cutover, not a rolling upgrade. + +This requirement covers every breaking change in this release that feeds a +shared cryptographic computation: + +- **Key derivation (F-03)** -- HKDF-SHA256 with a domain-separation `info` label. +- **Session IDs (tECDSA DKG and signing)** -- the typed, fixed-width session-ID + formats and the signing session ID's added dependency on the attempt start + block. Tracked in `CHANGELOG.md` under `### Changed` (BREAKING). +- **Hash-to-curve (F-02)** -- the counter-based `G1HashToPoint`. + +Within a single DKG or signing ceremony, mixed-version peers derive different +keys, session IDs, or points and fail to interoperate. The failure mode is +liveness-only: the ceremony does not complete. It is not a fund-safety or +consensus-safety issue -- mismatched cryptography fails closed (shares do not +decrypt, signatures do not verify) and never yields a valid-but-wrong result. +Operators must upgrade the entire ceremony fleet atomically and must not run a +mixed-version set through a live DKG or signing session. + +### Coordinated release-model context + +The mixed-version hazard above is why the coordinated release is _designed_ +around a single required operator update and one release-baked cutover block +(`C`): under that design, before `C` participants speak the legacy wire formats +and canonically post-`C` work speaks security-v2. That block-height cutover gate, +and its per-ceremony legacy/security-v2 mode strategies, are a separate, +not-yet-landed change. **This build does not contain the gate and therefore +still requires the atomic flag-day upgrade described in the section above — there +is no runtime height switch yet.** The fail-closed property holds regardless +(mismatched cryptography does not decrypt or verify and never yields a +valid-but-wrong result), so an un-upgraded peer that meets upgraded peers in a +ceremony loses liveness rather than fund safety. + +Two supporting changes ship to keep the coordinated release observable and to +identify who has not converged: + +- **Client-info compatibility (Part B).** The `clientInfo.port` default is + retained at `9601` for the release window (see OV-1). This keeps the + unauthenticated metrics/diagnostics channel — the primary source of a node's + exact revision and stranded-peer evidence — alive through the cutover. + Expose it only over a trusted path. R2 flips the default back to `0` after the + monitoring migration is complete. +- **Stranded/legacy-peer observability.** An announcer session-ID mismatch + observer classifies each membership-valid announcement as legacy or hardened + and a node-local, deduplicated cutover peer roster records post-cutover legacy + sightings by normalized operator address. A separate `cutover-roster` + aggregator joins those sightings to the authoritative eligible-instance + inventory so readiness is measured against exact revision/epoch/digest, not + merely a quiet mismatch counter. + +**Release epoch.** The coordinated cutover artifact is identified by the release +epoch `security_v2_cutover`. Exporting that epoch (and the cutover block) as a +`client_info` label and diagnostics field is part of the not-yet-landed gate +change and is NOT present in this build; today the go/no-go evidence is a node's +exact revision plus the stranded-peer observability below, not the container tag. +Note the exact revision is carried by the `/diagnostics` `client_info` field +only; the `client_info` **Prometheus metric** carries just the `version` label in +this build (revision/epoch labels arrive with the not-yet-landed gate change). The +`cutover-roster` aggregator's `--expectedEpoch` flag carries the expected +`security_v2_cutover` value as plain operator-supplied configuration until the +gate ships. + +--- + +## Upgrade Coordination Checklist + +For each breaking change: + +- [ ] Hard-fork block / protocol version agreed and documented +- [ ] Staging network upgrade tested +- [ ] Node operators notified with sufficient lead time +- [ ] Rollback plan in place (revert binary, block range) +- [ ] Post-upgrade monitoring in place (alert on share decryption failures) diff --git a/cmd/cmd.go b/cmd/cmd.go index 0dbe4a24ee..7a5214a6f0 100644 --- a/cmd/cmd.go +++ b/cmd/cmd.go @@ -37,6 +37,7 @@ func init() { EthereumCommand, MaintainerCommand, MaintainerCliCommand, + ReleaseManifestCommand, ) } diff --git a/cmd/cutover-roster/main.go b/cmd/cutover-roster/main.go new file mode 100644 index 0000000000..4825c796ac --- /dev/null +++ b/cmd/cutover-roster/main.go @@ -0,0 +1,650 @@ +// Command cutover-roster is the authoritative fleet aggregation service for a +// coordinated protocol cutover. It periodically polls each ceremony-eligible +// instance's trusted report target for its exact revision/epoch/image-digest +// attestation, folds in post-cutover node-local legacy sightings, reconciles +// each operator to a fleet status, persists the central state in bbolt, and +// serves a deterministic GET /api/v1/cutover-readiness endpoint plus Prometheus +// metrics on a monitoring-only address. +// +// The --expectedEpoch and --cutoverBlock values are plain operator-supplied +// configuration. They become meaningful once the real cutover release ships; +// this tool does not derive them from any compiled gate constant. +package main + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "net/http" + "os" + "os/signal" + "strconv" + "strings" + "syscall" + "time" + + "github.com/ipfs/go-log/v2" + + "github.com/keep-network/keep-core/pkg/monitoring/cutoverroster" +) + +var logger = log.Logger("keep-cutover-roster") + +type options struct { + expectedRevision string + expectedEpoch string + expectedImageDigest string + cutoverBlock uint64 + chainID string + collectionInterval time.Duration + missedThreshold uint + successThreshold uint + dbPath string + apiAddr string + allowedCIDRs string + inventoryFile string + serviceDiscoveryFile string + sightingsFile string + quarantineEvidenceFile string + attestedDigestsFile string + reportFormat string + ethereumRPC string + walletRegistryAddress string +} + +func parseOptions() options { + var opts options + + flag.StringVar(&opts.expectedRevision, "expectedRevision", "", + "Exact git revision (short SHA) the cutover release must report.") + flag.StringVar(&opts.expectedEpoch, "expectedEpoch", + cutoverroster.ExpectedEpochSecurityV2Cutover, + "Expected release epoch. Meaningful once the real cutover release ships.") + flag.StringVar(&opts.expectedImageDigest, "expectedImageDigest", "", + "Exact runtime image digest the cutover release must report.") + flag.Uint64Var(&opts.cutoverBlock, "cutoverBlock", 0, + "Cutover block C (metadata). Meaningful once the real cutover release ships.") + flag.StringVar(&opts.chainID, "chainID", "", "Chain ID of the monitored network.") + flag.DurationVar(&opts.collectionInterval, "collectionInterval", time.Minute, + "Interval between collection cycles.") + flag.UintVar(&opts.missedThreshold, "missedThreshold", 2, + "Consecutive missed collections before an instance is offline_unknown.") + flag.UintVar(&opts.successThreshold, "successThreshold", 3, + "Consecutive exact reports required before an operator is resolved_current.") + flag.StringVar(&opts.dbPath, "dbPath", "/var/lib/cutover-roster/roster.db", + "bbolt database path for persisted central state.") + flag.StringVar(&opts.apiAddr, "apiAddr", "127.0.0.1:9701", + "Monitoring-only bind address for the readiness API and /metrics. Do not expose publicly.") + flag.StringVar(&opts.allowedCIDRs, "allowedCIDRs", "", + "Comma-separated CIDR allowlist for the readiness API (monitoring trust "+ + "boundary). When set, only clients in these networks are served; all "+ + "others receive 403. Loopback is always allowed.") + flag.StringVar(&opts.inventoryFile, "inventoryFile", "", + "Path to the authoritative ceremony-eligible inventory JSON file.") + flag.StringVar(&opts.serviceDiscoveryFile, "serviceDiscoveryFile", "", + "Path to the production Prometheus file_sd target file (keep-sd.json). "+ + "REQUIRED for a complete readiness determination: an eligible operator "+ + "absent from discovery is offline_unknown, and discovered per-instance "+ + "/metrics targets (keyed by network ID) are used to fetch reports. Without "+ + "it, readiness can never be certified complete.") + flag.StringVar(&opts.sightingsFile, "sightingsFile", "", + "Optional path to a JSON file of aggregated post-cutover legacy sightings.") + flag.StringVar(&opts.quarantineEvidenceFile, "quarantineEvidenceFile", "", + "Optional path to a JSON file of independently-verified quarantine/removal "+ + "evidence. Without it, no quarantine evidence is accepted (fail closed).") + flag.StringVar(&opts.attestedDigestsFile, "attestedDigestsFile", "", + "Optional path to a JSON file of independently-attested per-instance image "+ + "digests (and, until the node emits it, release epoch). The running "+ + "binary does not know its own image digest, so this is external attestation.") + flag.StringVar(&opts.reportFormat, "reportFormat", "metrics", + "How to fetch per-instance reports. Only 'metrics' is supported: it scrapes "+ + "each node's real /metrics and /diagnostics endpoints and validates the "+ + "node's self-attested chain address and network ID, so one responding node "+ + "cannot certify another instance. A raw 'json' attestation blob cannot "+ + "prove a node's own per-instance identity and is rejected.") + flag.StringVar(&opts.ethereumRPC, "ethereumRPC", "", + "Optional Ethereum JSON-RPC URL used to read the current block height and, "+ + "with --walletRegistryAddress, to verify operator→staking-provider identity.") + flag.StringVar(&opts.walletRegistryAddress, "walletRegistryAddress", "", + "WalletRegistry contract address. With --ethereumRPC, enables on-chain "+ + "operator→staking-provider identity verification (fail closed on "+ + "mismatch). REQUIRED for a complete readiness determination: without it, "+ + "identity is NOT verified on chain and readiness can never be certified "+ + "complete.") + + flag.Parse() + + return opts +} + +func main() { + opts := parseOptions() + + if err := run(opts); err != nil { + logger.Errorf("cutover-roster exited with error: %v", err) + os.Exit(1) + } +} + +func run(opts options) error { + store, err := cutoverroster.OpenStore(opts.dbPath) + if err != nil { + return fmt.Errorf("cannot open store: %w", err) + } + defer func() { _ = store.Close() }() + + metrics := cutoverroster.NewPrometheusMetrics() + + collector, err := cutoverroster.NewCollector( + cutoverroster.CollectorConfig{ + ExpectedRevision: opts.expectedRevision, + ExpectedEpoch: opts.expectedEpoch, + ExpectedImageDigest: opts.expectedImageDigest, + CutoverBlock: opts.cutoverBlock, + ChainID: opts.chainID, + CollectionInterval: opts.collectionInterval, + MissedThreshold: opts.missedThreshold, + SuccessThreshold: opts.successThreshold, + // Production readiness requires the full authoritative trust chain: + // service-discovery reconciliation and on-chain identity verification + // are mandatory for complete=true. A missing feed blocks readiness + // rather than silently degrading to trusting the inventory alone. + RequireServiceDiscovery: true, + RequireIdentityVerification: true, + }, + store, + metrics, + ) + if err != nil { + return fmt.Errorf("cannot construct collector: %w", err) + } + + // Record whether the production service-discovery feed is wired. Without it, + // completeness is blocked (RequireServiceDiscovery): an eligible operator's + // instances cannot be reconciled one-to-one against discovered targets. + collector.SetServiceDiscoveryConfigured(opts.serviceDiscoveryFile != "") + if opts.serviceDiscoveryFile == "" { + logger.Warnf( + "service-discovery reconciliation is DISABLED; readiness cannot be " + + "certified complete until --serviceDiscoveryFile is set so eligible " + + "instances reconcile one-to-one against discovered targets", + ) + } + + // Install the independent quarantine-evidence verifier. Absent one, the + // collector accepts no quarantine evidence (fail closed). + if opts.quarantineEvidenceFile != "" { + entries, verifierErr := loadQuarantineEvidence(opts.quarantineEvidenceFile) + if verifierErr != nil { + return fmt.Errorf("cannot load quarantine evidence: %w", verifierErr) + } + collector.SetQuarantineVerifier( + cutoverroster.NewAllowlistQuarantineVerifier(entries), + ) + logger.Infof( + "loaded %d independently-verified quarantine evidence entries", + len(entries), + ) + } + + // Install the on-chain operator→staking-provider identity verifier when an + // RPC endpoint and a WalletRegistry address are configured. Without it, + // inventory identity claims are NOT confirmed on chain — surface that gap + // explicitly rather than silently trusting the inventory. + if opts.ethereumRPC != "" && opts.walletRegistryAddress != "" { + verifier, verr := cutoverroster.NewEthCallIdentityVerifier( + opts.ethereumRPC, opts.walletRegistryAddress, nil, + ) + if verr != nil { + return fmt.Errorf("cannot construct identity verifier: %w", verr) + } + collector.SetIdentityVerifier(verifier) + logger.Infof( + "on-chain operator→staking-provider identity verification enabled " + + "against the configured WalletRegistry", + ) + } else { + logger.Warnf( + "on-chain operator→staking-provider identity verification is DISABLED; " + + "readiness cannot be certified complete until --ethereumRPC and " + + "--walletRegistryAddress are set to verify inventory identity claims " + + "against the WalletRegistry", + ) + } + + fetcher, err := buildReportFetcher(opts) + if err != nil { + return fmt.Errorf("cannot build report fetcher: %w", err) + } + + allowlist, err := cutoverroster.ParseCIDRAllowlist(opts.allowedCIDRs) + if err != nil { + return fmt.Errorf("cannot parse --allowedCIDRs: %w", err) + } + server, err := cutoverroster.NewServer(opts.apiAddr, collector, metrics, allowlist) + if err != nil { + return fmt.Errorf("cannot start API server: %w", err) + } + + ctx, stop := signal.NotifyContext( + context.Background(), syscall.SIGINT, syscall.SIGTERM, + ) + defer stop() + + go func() { + if serveErr := server.Serve(); serveErr != nil { + logger.Errorf("readiness API server error: %v", serveErr) + } + }() + logger.Infof( + "cutover-roster serving readiness API on %s (monitoring-only)", + server.Addr(), + ) + + runCollectionLoop(ctx, opts, collector, fetcher) + + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + return server.Close(shutdownCtx) +} + +func runCollectionLoop( + ctx context.Context, + opts options, + collector *cutoverroster.Collector, + fetcher reportFetcher, +) { + ticker := time.NewTicker(opts.collectionInterval) + defer ticker.Stop() + + collectOnce(ctx, opts, collector, fetcher) + + for { + select { + case <-ctx.Done(): + logger.Infof("shutdown requested; stopping collection loop") + return + case <-ticker.C: + collectOnce(ctx, opts, collector, fetcher) + } + } +} + +func collectOnce( + ctx context.Context, + opts options, + collector *cutoverroster.Collector, + fetcher reportFetcher, +) { + // Read the chain height first so it can stamp even a failed-closed snapshot; + // it is independent of the authoritative inputs read below. + currentBlock := readCurrentBlock(ctx, opts.ethereumRPC, opts.chainID) + + // An unreadable authoritative input must fail readiness closed for this cycle + // rather than leave a previous "complete=true" snapshot standing. Returning + // early would keep certifying readiness while the inventory or sightings + // evidence is missing. + inventory, err := loadInventory(opts.inventoryFile) + if err != nil { + logger.Errorf( + "cannot load authoritative inventory; failing readiness closed "+ + "for this cycle: %v", err, + ) + collector.RecordInputUnavailable(currentBlock) + return + } + + // Reconcile against the production service-discovery target file when one is + // configured: an eligible operator absent from discovery is offline_unknown, + // and discovered /metrics targets supply the report scrape URL. + if opts.serviceDiscoveryFile != "" { + sd, sdErr := loadServiceDiscovery(opts.serviceDiscoveryFile) + if sdErr != nil { + logger.Errorf( + "cannot load service-discovery target file; failing readiness "+ + "closed for this cycle: %v", sdErr, + ) + collector.RecordInputUnavailable(currentBlock) + return + } + inventory = cutoverroster.ReconcileWithDiscovery(inventory, sd) + applyDiscoveredTargets(inventory, sd) + } + + sightings, err := loadSightings(opts.sightingsFile) + if err != nil { + logger.Errorf( + "cannot load legacy sightings; failing readiness closed for this "+ + "cycle: %v", err, + ) + collector.RecordInputUnavailable(currentBlock) + return + } + + reports := pollReports(ctx, inventory, fetcher) + + // Collect itself fails readiness closed on any internal error (a persistence + // write failure supersedes the served snapshot with an incomplete one and a + // nonzero unreconciled gauge), so logging the error here is sufficient; the + // stale "complete=true" snapshot is already gone. CollectContext threads the + // cycle context so a degraded WalletRegistry RPC honors shutdown. + if _, err := collector.CollectContext(ctx, inventory, reports, sightings, currentBlock); err != nil { + logger.Errorf("collection cycle failed: %v", err) + } +} + +func loadInventory(path string) ([]cutoverroster.InventoryInstance, error) { + if path == "" { + return nil, nil + } + // #nosec G304 -- operator-supplied inventory path for the monitoring tool. + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + // Decode into the input form, which carries trusted_report_target under an + // explicit JSON key; InventoryInstance itself never serializes that field. + var inputs []cutoverroster.InventoryInstanceInput + if err := json.Unmarshal(data, &inputs); err != nil { + return nil, fmt.Errorf("cannot decode inventory: %w", err) + } + inventory := make([]cutoverroster.InventoryInstance, 0, len(inputs)) + for _, in := range inputs { + inventory = append(inventory, in.ToInventoryInstance()) + } + return inventory, nil +} + +func loadQuarantineEvidence( + path string, +) ([]cutoverroster.VerifiedQuarantineEntry, error) { + // #nosec G304 -- operator-supplied evidence path for the monitoring tool. + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var entries []cutoverroster.VerifiedQuarantineEntry + if err := json.Unmarshal(data, &entries); err != nil { + return nil, fmt.Errorf("cannot decode quarantine evidence: %w", err) + } + return entries, nil +} + +func loadSightings(path string) ([]cutoverroster.LegacySighting, error) { + if path == "" { + return nil, nil + } + // #nosec G304 -- operator-supplied sightings path for the monitoring tool. + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var sightings []cutoverroster.LegacySighting + if err := json.Unmarshal(data, &sightings); err != nil { + return nil, fmt.Errorf("cannot decode sightings: %w", err) + } + return sightings, nil +} + +// reportFetcher fetches one instance's attested report. Implementations must not +// leak the raw transport error (which embeds the target host/URL) to the caller. +type reportFetcher interface { + fetch( + ctx context.Context, inv cutoverroster.InventoryInstance, + ) (cutoverroster.InstanceReport, error) +} + +// buildReportFetcher constructs the configured report fetcher. The default +// 'metrics' fetcher scrapes the node's real /metrics and /diagnostics endpoints; +// 'json' fetches a dedicated JSON attestation endpoint from each trusted target. +func buildReportFetcher(opts options) (reportFetcher, error) { + var attestation cutoverroster.AttestationSource + if opts.attestedDigestsFile != "" { + att, err := loadAttestation(opts.attestedDigestsFile) + if err != nil { + return nil, err + } + attestation = att + } + + switch opts.reportFormat { + case "", "metrics": + return &metricsFetcher{ + source: cutoverroster.NewMetricsReportSource(nil, attestation), + }, nil + case "json": + // A raw JSON attestation blob is self-declared: a single endpoint can echo + // back whatever identity the collector expects, so it cannot prove the + // responding node's OWN per-instance (operator, network ID) identity the way + // the metrics path does by reading each node's /diagnostics. It therefore + // cannot provide the per-instance guarantee that keeps one node from + // certifying several instances, and is refused rather than left as a silent + // gap in a "complete" readiness determination. + return nil, fmt.Errorf( + "--reportFormat=json is not supported: a JSON attestation blob cannot " + + "prove a node's own per-instance identity and must not contribute to a " + + "complete readiness determination; use 'metrics'", + ) + default: + return nil, fmt.Errorf( + "unknown --reportFormat %q (want 'metrics')", opts.reportFormat, + ) + } +} + +// metricsFetcher scrapes the node's real /metrics and /diagnostics endpoints. +type metricsFetcher struct { + source *cutoverroster.MetricsReportSource +} + +func (m *metricsFetcher) fetch( + ctx context.Context, inv cutoverroster.InventoryInstance, +) (cutoverroster.InstanceReport, error) { + return m.source.Fetch(ctx, inv) +} + +// pollReports fetches each eligible instance's report via the configured fetcher. +// A target that is unreachable or returns a malformed body is simply omitted, +// which the collector treats as a missed collection. Only the sanitized +// (URL-free) fetch error is logged, and only at debug level. +func pollReports( + ctx context.Context, + inventory []cutoverroster.InventoryInstance, + fetcher reportFetcher, +) map[string]cutoverroster.InstanceReport { + reports := make(map[string]cutoverroster.InstanceReport) + + for _, inv := range inventory { + if !inv.CeremonyEligible || inv.TrustedReportTarget == "" { + continue + } + report, err := fetcher.fetch(ctx, inv) + if err != nil { + logger.Debugf("no report from instance %s: %v", inv.InstanceID, err) + continue + } + reports[inv.InstanceID] = report + } + + return reports +} + +// loadServiceDiscovery reads and parses the production Prometheus file_sd target +// file (keep-sd.json). +func loadServiceDiscovery(path string) (*cutoverroster.ServiceDiscovery, error) { + // #nosec G304 -- operator-supplied service-discovery path for the monitoring tool. + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + return cutoverroster.ParseServiceDiscovery(data) +} + +// applyDiscoveredTargets makes production service discovery authoritative for +// each eligible instance's report target. It sets the target to the discovered +// /metrics base URL for the exact (operator address, network ID) instance and +// otherwise clears it. Keying by network ID means multiple instances of one +// operator each resolve to their own discovered target rather than collapsing +// onto a single operator-level URL. +// +// Crucially, a discovered target OVERRIDES any inventory-supplied +// trusted_report_target, and an instance service discovery does not know is left +// untargeted (offline, fail closed) even if inventory named a target. This closes +// the bypass where an inventory-supplied target routed around the discovered +// per-instance identity: an explicit target can no longer stand in for a distinct +// instance that never appears in discovery. ReconcileWithDiscovery has already +// flagged such an instance DisappearedFromDiscovery, so it cannot resolve either +// way; clearing its target additionally stops it from being polled at all. +func applyDiscoveredTargets( + inventory []cutoverroster.InventoryInstance, + sd *cutoverroster.ServiceDiscovery, +) { + for i := range inventory { + if !inventory[i].CeremonyEligible { + continue + } + inventory[i].TrustedReportTarget = sd.MetricsURLForInstance( + inventory[i].OperatorAddress, inventory[i].NetworkID, + ) + } +} + +// loadAttestation reads the independently-attested per-instance image digests +// (and, until the node emits it, release epochs). +func loadAttestation(path string) (cutoverroster.AttestationSource, error) { + // #nosec G304 -- operator-supplied attestation path for the monitoring tool. + data, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var payload struct { + Digests map[string]string `json:"digests"` + Epochs map[string]string `json:"epochs"` + } + if err := json.Unmarshal(data, &payload); err != nil { + return nil, fmt.Errorf("cannot decode attestation file: %w", err) + } + return &cutoverroster.MapAttestationSource{ + Digests: payload.Digests, + Epochs: payload.Epochs, + }, nil +} + +// readCurrentBlock reads the current block height via eth_blockNumber. When a +// chain ID is configured it first verifies, via eth_chainId, that the RPC +// endpoint actually serves the expected chain. It returns 0 when no RPC URL is +// configured, on any error, or on a chain-ID mismatch — a block height read from +// the wrong chain must never be allowed to certify readiness. +func readCurrentBlock(ctx context.Context, rpcURL, expectedChainID string) uint64 { + if rpcURL == "" { + return 0 + } + + if expectedChainID != "" { + actual, err := ethRPCResult(ctx, rpcURL, "eth_chainId") + if err != nil { + logger.Errorf("cannot verify chain ID via RPC: %v", err) + return 0 + } + if !chainIDMatches(expectedChainID, actual) { + logger.Errorf( + "configured chain ID %q does not match RPC eth_chainId %q; "+ + "refusing to use a block height from the wrong chain", + expectedChainID, actual, + ) + return 0 + } + } + + result, err := ethRPCResult(ctx, rpcURL, "eth_blockNumber") + if err != nil { + logger.Debugf("cannot read current block: %v", err) + return 0 + } + block, err := parseHexUint64(result) + if err != nil { + logger.Debugf("cannot parse block number %q: %v", result, err) + return 0 + } + return block +} + +// ethRPCResult performs a single parameter-less JSON-RPC call and returns the +// string "result" field, surfacing any transport, HTTP, or JSON-RPC error. +func ethRPCResult(ctx context.Context, rpcURL, method string) (string, error) { + body := []byte(fmt.Sprintf( + `{"jsonrpc":"2.0","id":1,"method":%q,"params":[]}`, method, + )) + // #nosec G107 -- the RPC URL is operator-supplied configuration. + req, err := http.NewRequestWithContext(ctx, http.MethodPost, rpcURL, bytes.NewReader(body)) + if err != nil { + return "", err + } + req.Header.Set("Content-Type", "application/json") + + client := &http.Client{Timeout: 10 * time.Second} + resp, err := client.Do(req) + if err != nil { + // Sanitize: the raw transport error embeds the RPC URL (host/IP), which + // the spec forbids from appearing in logs. + return "", fmt.Errorf("ethereum RPC request failed") + } + defer func() { _ = resp.Body.Close() }() + + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("unexpected status %d", resp.StatusCode) + } + + var rpcResponse struct { + Result string `json:"result"` + Error *struct { + Message string `json:"message"` + } `json:"error"` + } + if err := json.NewDecoder(resp.Body).Decode(&rpcResponse); err != nil { + return "", fmt.Errorf("cannot decode RPC response") + } + if rpcResponse.Error != nil { + return "", fmt.Errorf("rpc error: %s", rpcResponse.Error.Message) + } + return rpcResponse.Result, nil +} + +// chainIDMatches reports whether the configured chain ID (decimal or 0x-hex) +// numerically equals the RPC-returned eth_chainId (hex). +func chainIDMatches(configured, rpcHex string) bool { + rpcVal, err := parseHexUint64(rpcHex) + if err != nil { + return false + } + cfgVal, err := parseUint64Flexible(configured) + if err != nil { + return false + } + return cfgVal == rpcVal +} + +// parseUint64Flexible parses an unsigned integer that may be decimal or +// 0x-prefixed hexadecimal. +func parseUint64Flexible(s string) (uint64, error) { + s = strings.TrimSpace(s) + if strings.HasPrefix(s, "0x") || strings.HasPrefix(s, "0X") { + return strconv.ParseUint(s[2:], 16, 64) + } + return strconv.ParseUint(s, 10, 64) +} + +func parseHexUint64(s string) (uint64, error) { + if len(s) < 2 || s[:2] != "0x" { + return 0, errors.New("missing 0x prefix") + } + // Delegate digit parsing to strconv, which rejects an empty body, an invalid + // digit, and — critically — a value that overflows uint64. A hand-rolled + // value*16+digit loop wraps silently on an oversized eth_blockNumber or + // eth_chainId result, which would then certify readiness from a bogus height; + // ParseUint fails closed instead. + return strconv.ParseUint(s[2:], 16, 64) +} diff --git a/cmd/cutover-roster/main_test.go b/cmd/cutover-roster/main_test.go new file mode 100644 index 0000000000..dfca903327 --- /dev/null +++ b/cmd/cutover-roster/main_test.go @@ -0,0 +1,207 @@ +package main + +import ( + "context" + "math" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/monitoring/cutoverroster" +) + +// TestLoadInventory_ReadsTrustedReportTarget proves the inventory loader ingests +// the trusted report target from the explicit JSON key. This is the regression +// guard for the bug where the target — being `json:"-"` on InventoryInstance — +// was silently dropped on input, leaving every instance untargeted. +func TestLoadInventory_ReadsTrustedReportTarget(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "inventory.json") + content := `[ + { + "instance_id": "i1", + "operator_address": "0xabc", + "staking_provider": "sp-1", + "ceremony_eligible": true, + "expected_revision": "abc123", + "expected_epoch": "security_v2_cutover", + "expected_image_digest": "sha256:deadbeef", + "trusted_report_target": "https://reports.example/i1" + } + ]` + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("cannot write inventory file: %v", err) + } + + inventory, err := loadInventory(path) + if err != nil { + t.Fatalf("loadInventory failed: %v", err) + } + if len(inventory) != 1 { + t.Fatalf("expected 1 instance, got %d", len(inventory)) + } + inv := inventory[0] + if inv.TrustedReportTarget != "https://reports.example/i1" { + t.Errorf("trusted report target not ingested: %q", inv.TrustedReportTarget) + } + if !inv.CeremonyEligible { + t.Errorf("ceremony_eligible not ingested") + } + if inv.ExpectedImageDigest != "sha256:deadbeef" { + t.Errorf("expected image digest not ingested: %q", inv.ExpectedImageDigest) + } +} + +// TestLoadInventory_EmptyPath returns no inventory without error. +func TestLoadInventory_EmptyPath(t *testing.T) { + inventory, err := loadInventory("") + if err != nil { + t.Fatalf("expected no error for empty path, got %v", err) + } + if inventory != nil { + t.Errorf("expected nil inventory for empty path, got %v", inventory) + } +} + +// TestLoadQuarantineEvidence_ReadsEntries proves independently-verified evidence +// is loaded from its separate trusted file. +func TestLoadQuarantineEvidence_ReadsEntries(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "evidence.json") + content := `[ + {"instance_id": "i1", "operator_address": "0xabc", "evidence_ref": "evidence://verified/1"} + ]` + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatalf("cannot write evidence file: %v", err) + } + + entries, err := loadQuarantineEvidence(path) + if err != nil { + t.Fatalf("loadQuarantineEvidence failed: %v", err) + } + if len(entries) != 1 { + t.Fatalf("expected 1 evidence entry, got %d", len(entries)) + } + if entries[0].EvidenceRef != "evidence://verified/1" { + t.Errorf("evidence ref not ingested: %q", entries[0].EvidenceRef) + } +} + +// TestChainIDMatches proves the configured chain ID (decimal or 0x-hex) is +// compared numerically against the RPC-returned hex eth_chainId, so a block +// height read from the wrong chain is rejected rather than certifying readiness. +func TestChainIDMatches(t *testing.T) { + cases := []struct { + configured, rpcHex string + want bool + }{ + {"1", "0x1", true}, + {"0x1", "0x1", true}, + {"11155111", "0xaa36a7", true}, // sepolia, decimal vs hex + {"1", "0x2", false}, + {"", "0x1", false}, + {"1", "", false}, + {"abc", "0x1", false}, + {"1", "0xzz", false}, + } + for _, c := range cases { + if got := chainIDMatches(c.configured, c.rpcHex); got != c.want { + t.Errorf( + "chainIDMatches(%q, %q) = %v, want %v", + c.configured, c.rpcHex, got, c.want, + ) + } + } +} + +// TestParseHexUint64 proves the JSON-RPC hex parser fails closed: it rejects a +// missing prefix, an empty body, an invalid digit, and — critically — a value +// that overflows uint64. An oversized eth_blockNumber/eth_chainId result must +// error rather than silently wrap and certify readiness from a bogus height. +func TestParseHexUint64(t *testing.T) { + maxHex := "0x" + strings.Repeat("f", 16) // exactly math.MaxUint64 + + cases := []struct { + in string + want uint64 + wantErr bool + }{ + {"0x0", 0, false}, + {"0x1", 1, false}, + {"0xaa36a7", 0xaa36a7, false}, + {maxHex, math.MaxUint64, false}, + {"0x" + strings.Repeat("f", 17), 0, true}, // 17 nibbles overflows uint64 + {"0x1" + strings.Repeat("0", 16), 0, true}, // 2^64 overflows uint64 + {"", 0, true}, // missing prefix + {"1", 0, true}, // missing prefix + {"0x", 0, true}, // empty body + {"0xzz", 0, true}, // invalid digit + } + for _, c := range cases { + got, err := parseHexUint64(c.in) + if c.wantErr { + if err == nil { + t.Errorf("parseHexUint64(%q) = %d, want error", c.in, got) + } + continue + } + if err != nil { + t.Errorf("parseHexUint64(%q) unexpected error: %v", c.in, err) + continue + } + if got != c.want { + t.Errorf("parseHexUint64(%q) = %d, want %d", c.in, got, c.want) + } + } +} + +// TestCollectOnce_InventoryUnavailableFailsClosed proves the collection loop +// fails readiness closed when its authoritative inventory cannot be read: rather +// than returning early and leaving a prior snapshot standing, it drives the +// collector to an incomplete snapshot carrying a nonzero unreconciled signal. +func TestCollectOnce_InventoryUnavailableFailsClosed(t *testing.T) { + store, err := cutoverroster.OpenStore(filepath.Join(t.TempDir(), "roster.db")) + if err != nil { + t.Fatalf("cannot open store: %v", err) + } + defer func() { _ = store.Close() }() + + collector, err := cutoverroster.NewCollector( + cutoverroster.CollectorConfig{ + ExpectedEpoch: cutoverroster.ExpectedEpochSecurityV2Cutover, + CollectionInterval: time.Minute, + MissedThreshold: 2, + SuccessThreshold: 3, + }, + store, + cutoverroster.NewPrometheusMetrics(), + ) + if err != nil { + t.Fatalf("cannot construct collector: %v", err) + } + + // A non-empty path to a file that does not exist forces the inventory load to + // fail (an empty path is a valid "no inventory" input and would not error). + opts := options{ + inventoryFile: filepath.Join(t.TempDir(), "does-not-exist.json"), + } + + fetcher, err := buildReportFetcher(opts) + if err != nil { + t.Fatalf("cannot build report fetcher: %v", err) + } + collectOnce(context.Background(), opts, collector, fetcher) + + snap := collector.Snapshot() + if snap.Complete { + t.Errorf("an unreadable inventory must not leave readiness certified complete") + } + if snap.Inventory.Unreconciled < 1 { + t.Errorf( + "an unreadable inventory must drive a nonzero unreconciled signal, got %d", + snap.Inventory.Unreconciled, + ) + } +} diff --git a/cmd/evidence_window_lifecycle_test.go b/cmd/evidence_window_lifecycle_test.go new file mode 100644 index 0000000000..e785b6cc20 --- /dev/null +++ b/cmd/evidence_window_lifecycle_test.go @@ -0,0 +1,88 @@ +package cmd + +import ( + "context" + "os" + "syscall" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +func TestEvidenceWindowSignalController_OpenAndClose(t *testing.T) { + tests := map[string]struct { + initialActive bool + signal os.Signal + expected bool + }{ + "open": { + initialActive: false, + signal: syscall.SIGUSR1, + expected: true, + }, + "close": { + initialActive: true, + signal: syscall.SIGUSR2, + expected: false, + }, + } + + for testName, testCase := range tests { + t.Run(testName, func(t *testing.T) { + evidenceWindow := participation.NewCutoverEvidenceWindowSignal() + evidenceWindow.SetActive(testCase.initialActive) + + signals := make(chan os.Signal, 1) + signals <- testCase.signal + close(signals) + + done := startEvidenceWindowSignalController( + context.Background(), + evidenceWindow, + signals, + ) + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("the evidence-window signal controller did not stop") + } + + if evidenceWindow.Active() != testCase.expected { + t.Errorf( + "unexpected evidence-window state: got [%t], want [%t]", + evidenceWindow.Active(), + testCase.expected, + ) + } + }) + } +} + +func TestEvidenceWindowSignalController_CannotCloseRollbackWindow( + t *testing.T, +) { + evidenceWindow := participation.NewCutoverEvidenceWindowSignal() + evidenceWindow.HoldActive() + + signals := make(chan os.Signal, 1) + signals <- syscall.SIGUSR2 + close(signals) + + done := startEvidenceWindowSignalController( + context.Background(), + evidenceWindow, + signals, + ) + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("the evidence-window signal controller did not stop") + } + + if !evidenceWindow.Active() { + t.Fatal("SIGUSR2 closed an evidence window held active for rollback") + } +} diff --git a/cmd/flags.go b/cmd/flags.go index 7a67ad5df8..b88dab0ff7 100644 --- a/cmd/flags.go +++ b/cmd/flags.go @@ -50,6 +50,8 @@ func initFlags( initTbtcFlags(cmd, cfg) case config.Maintainer: initMaintainerFlags(cmd, cfg) + case config.ProtocolParticipation: + initProtocolParticipationFlags(cmd, cfg) case config.Developer: initDeveloperFlags(cmd) } @@ -257,7 +259,7 @@ func initClientInfoFlags(cmd *cobra.Command, cfg *config.Config) { &cfg.ClientInfo.Port, "clientInfo.port", 9601, - "Client Info HTTP server listening port.", + "Client Info HTTP server listening port. Set to 0 to disable; expose only on a trusted network.", ) cmd.Flags().DurationVar( @@ -375,6 +377,19 @@ func initMaintainerFlags(command *cobra.Command, cfg *config.Config) { ) } +// Initialize flags for Protocol Participation configuration. +func initProtocolParticipationFlags(cmd *cobra.Command, cfg *config.Config) { + cmd.Flags().Uint64Var( + &cfg.ProtocolParticipation.CutoverBlock, + "protocolParticipation.cutoverBlock", + 0, + "Protocol cutover block override for non-mainnet networks. Mainnet "+ + "always uses the compiled release constant and rejects this "+ + "setting; testnet requires a nonzero value; developer mode may "+ + "use 0 to disable the cutover schedule.", + ) +} + // Initialize flags for Developer configuration. func initDeveloperFlags(command *cobra.Command) { initContractAddressFlag := func(contractName string) { diff --git a/cmd/flags_test.go b/cmd/flags_test.go index bb313cf50c..d1f2b05997 100644 --- a/cmd/flags_test.go +++ b/cmd/flags_test.go @@ -22,6 +22,7 @@ import ( ethereumEcdsa "github.com/keep-network/keep-core/pkg/chain/ethereum/ecdsa/gen" ethereumTbtc "github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen" ethereumThreshold "github.com/keep-network/keep-core/pkg/chain/ethereum/threshold/gen" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) var cmdFlagsTests = map[string]struct { @@ -363,6 +364,15 @@ var cmdFlagsTests = map[string]struct { expectedValueFromFlag: common.HexToAddress("0xE7d33d8AA55B73a93059a24b900366894684a497"), defaultValue: common.HexToAddress(ethereumTbtc.WalletProposalValidatorAddress), }, + "protocolParticipation.cutoverBlock": { + readValueFunc: func(c *config.Config) interface{} { + return c.ProtocolParticipation.CutoverBlock + }, + flagName: "--protocolParticipation.cutoverBlock", + flagValue: "124000", + expectedValueFromFlag: uint64(124000), + defaultValue: uint64(0), + }, } func TestFlags_ReadConfigFromFlags(t *testing.T) { @@ -487,6 +497,103 @@ func TestFlags_Mixed(t *testing.T) { } } +// TestFlags_ClientInfoPortExplicitZero proves that an explicit `--clientInfo.port 0` +// on the command line resolves to zero even though the bound flag default is now +// 9601. This is the CLI half of the two explicit-zero acceptance paths; it cannot +// stand in for the TOML path because flag binding and Viper unmarshalling have +// different precedence rules. +func TestFlags_ClientInfoPortExplicitZero(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + args := []string{ + cmdFlagsTests["ethereum.url"].flagName, cmdFlagsTests["ethereum.url"].flagValue, + cmdFlagsTests["ethereum.keyFile"].flagName, cmdFlagsTests["ethereum.keyFile"].flagValue, + cmdFlagsTests["bitcoin.electrum.url"].flagName, cmdFlagsTests["bitcoin.electrum.url"].flagValue, + cmdFlagsTests["storage.dir"].flagName, cmdFlagsTests["storage.dir"].flagValue, + "--clientInfo.port", "0", + } + testCommand.SetArgs(args) + + testCommand.Execute() + + if testConfig.ClientInfo.Port != 0 { + t.Errorf( + "expected clientInfo.port to be 0 when explicitly set on the CLI, got [%d]", + testConfig.ClientInfo.Port, + ) + } +} + +// TestFlags_ClientInfoPortZeroFromConfig proves that an explicit `[clientInfo] Port = 0` +// in a TOML file resolves to zero despite the bound flag default of 9601. This is the +// TOML half of the two explicit-zero acceptance paths; Viper must preserve a config-file +// zero over the CLI-bound default. +func TestFlags_ClientInfoPortZeroFromConfig(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + args := []string{ + "--config", "../test/config_clientinfo_zero.toml", + } + testCommand.SetArgs(args) + + testCommand.Execute() + + if testConfig.ClientInfo.Port != 0 { + t.Errorf( + "expected clientInfo.port to be 0 when set to 0 in the config file, got [%d]", + testConfig.ClientInfo.Port, + ) + } +} + +// TestFlags_ClientInfoPortExplicit9601 proves that an explicit +// `--clientInfo.port 9601` on the command line resolves to the 9601 compatibility +// port (i.e. a nonzero, server-enabling value). It is the explicit counterpart of +// the bound-default case: an operator may pin 9601 to make the intent explicit. +func TestFlags_ClientInfoPortExplicit9601(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + args := []string{ + cmdFlagsTests["ethereum.url"].flagName, cmdFlagsTests["ethereum.url"].flagValue, + cmdFlagsTests["ethereum.keyFile"].flagName, cmdFlagsTests["ethereum.keyFile"].flagValue, + cmdFlagsTests["bitcoin.electrum.url"].flagName, cmdFlagsTests["bitcoin.electrum.url"].flagValue, + cmdFlagsTests["storage.dir"].flagName, cmdFlagsTests["storage.dir"].flagValue, + "--clientInfo.port", "9601", + } + testCommand.SetArgs(args) + + testCommand.Execute() + + if testConfig.ClientInfo.Port != 9601 { + t.Errorf( + "expected clientInfo.port to be 9601 when explicitly set on the CLI, got [%d]", + testConfig.ClientInfo.Port, + ) + } +} + +// TestFlags_ClientInfoPort9601FromConfig proves that an explicit +// `[clientInfo] Port = 9601` in a TOML file resolves to 9601 (a nonzero, +// server-enabling value). It is the TOML counterpart of the explicit CLI 9601 +// case. +func TestFlags_ClientInfoPort9601FromConfig(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + args := []string{ + "--config", "../test/config_clientinfo_9601.toml", + } + testCommand.SetArgs(args) + + testCommand.Execute() + + if testConfig.ClientInfo.Port != 9601 { + t.Errorf( + "expected clientInfo.port to be 9601 when set to 9601 in the config file, got [%d]", + testConfig.ClientInfo.Port, + ) + } +} + func initTestCommand() (*cobra.Command, *config.Config, *string) { if err := os.Setenv(config.EthereumPasswordEnvVariable, "password from env var"); err != nil { panic(err) @@ -534,3 +641,227 @@ func readPeers(network commonEthereum.Network) []string { return result } + +// TestFlags_ProtocolParticipationAbsentByDefault proves that with no flag and +// no config key the cutover block resolves to the zero default and, more +// importantly, is detected as not explicitly supplied. Mainnet rejects the +// override by presence, so absence must be reliably distinguishable. +func TestFlags_ProtocolParticipationAbsentByDefault(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + args := []string{ + cmdFlagsTests["ethereum.url"].flagName, cmdFlagsTests["ethereum.url"].flagValue, + cmdFlagsTests["ethereum.keyFile"].flagName, cmdFlagsTests["ethereum.keyFile"].flagValue, + cmdFlagsTests["bitcoin.electrum.url"].flagName, cmdFlagsTests["bitcoin.electrum.url"].flagValue, + cmdFlagsTests["storage.dir"].flagName, cmdFlagsTests["storage.dir"].flagValue, + } + testCommand.SetArgs(args) + + testCommand.Execute() + + if testConfig.ProtocolParticipation.CutoverBlock != 0 { + t.Errorf( + "expected the cutover block default 0, got [%d]", + testConfig.ProtocolParticipation.CutoverBlock, + ) + } + if testConfig.ProtocolParticipation.CutoverBlockSet { + t.Error("expected the cutover block to be detected as not supplied") + } +} + +// TestFlags_ProtocolParticipationPresenceFromFlag proves that an explicitly +// changed flag is detected as supplied even when its value equals the bound +// default, which is what lets mainnet reject an explicit zero override. +func TestFlags_ProtocolParticipationPresenceFromFlag(t *testing.T) { + for _, flagValue := range []string{"124000", "0"} { + testCommand, testConfig, _ := initTestCommand() + + args := []string{ + cmdFlagsTests["ethereum.url"].flagName, cmdFlagsTests["ethereum.url"].flagValue, + cmdFlagsTests["ethereum.keyFile"].flagName, cmdFlagsTests["ethereum.keyFile"].flagValue, + cmdFlagsTests["bitcoin.electrum.url"].flagName, cmdFlagsTests["bitcoin.electrum.url"].flagValue, + cmdFlagsTests["storage.dir"].flagName, cmdFlagsTests["storage.dir"].flagValue, + "--protocolParticipation.cutoverBlock", flagValue, + } + testCommand.SetArgs(args) + + testCommand.Execute() + + if !testConfig.ProtocolParticipation.CutoverBlockSet { + t.Errorf( + "expected an explicit flag value [%s] to be detected as "+ + "supplied", + flagValue, + ) + } + } +} + +// TestFlags_ProtocolParticipationNetworkMatrix proves the per-network cutover +// schedule resolution rules on top of the command wiring: mainnet rejects any +// override (including an explicit zero), testnet requires a nonzero value, and +// developer mode accepts both zero (disabled) and nonzero. +func TestFlags_ProtocolParticipationNetworkMatrix(t *testing.T) { + baseArgs := func() []string { + return []string{ + cmdFlagsTests["ethereum.url"].flagName, cmdFlagsTests["ethereum.url"].flagValue, + cmdFlagsTests["ethereum.keyFile"].flagName, cmdFlagsTests["ethereum.keyFile"].flagValue, + cmdFlagsTests["bitcoin.electrum.url"].flagName, cmdFlagsTests["bitcoin.electrum.url"].flagValue, + cmdFlagsTests["storage.dir"].flagName, cmdFlagsTests["storage.dir"].flagValue, + } + } + + var tests = map[string]struct { + networkFlag string + cutoverFlagValue string + expectResolutionErr bool + expectedCutoverBlock uint64 + }{ + "mainnet rejects an override": { + networkFlag: "", + cutoverFlagValue: "124000", + expectResolutionErr: true, + }, + "mainnet rejects an explicit zero override": { + networkFlag: "", + cutoverFlagValue: "0", + expectResolutionErr: true, + }, + "testnet accepts a nonzero cutover block": { + networkFlag: "--testnet", + cutoverFlagValue: "124000", + expectedCutoverBlock: 124000, + }, + "testnet rejects a zero cutover block": { + networkFlag: "--testnet", + cutoverFlagValue: "", + expectResolutionErr: true, + }, + "developer accepts zero as disabled": { + networkFlag: "--developer", + cutoverFlagValue: "", + expectedCutoverBlock: 0, + }, + "developer accepts a nonzero cutover block": { + networkFlag: "--developer", + cutoverFlagValue: "42", + expectedCutoverBlock: 42, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + args := baseArgs() + if test.networkFlag != "" { + args = append(args, test.networkFlag) + } + if test.cutoverFlagValue != "" { + args = append( + args, + "--protocolParticipation.cutoverBlock", + test.cutoverFlagValue, + ) + } + testCommand.SetArgs(args) + + testCommand.Execute() + + schedule, err := participation.ResolveAndValidate( + testConfig.Ethereum.Network, + testConfig.ProtocolParticipation, + ) + + if test.expectResolutionErr { + if err == nil { + t.Fatal("expected a schedule resolution error") + } + return + } + + if err != nil { + t.Fatalf("unexpected schedule resolution error: [%v]", err) + } + if schedule.CutoverBlock != test.expectedCutoverBlock { + t.Errorf( + "expected cutover block [%d], got [%d]", + test.expectedCutoverBlock, + schedule.CutoverBlock, + ) + } + }) + } +} + +// TestFlags_ProtocolParticipationFromConfigFile proves that a +// `[protocolParticipation] CutoverBlock` config file key is decoded and +// detected as explicitly present, and that mainnet consequently rejects it. +func TestFlags_ProtocolParticipationFromConfigFile(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + testCommand.SetArgs([]string{ + "--config", "../test/config_participation_cutover.toml", + }) + + testCommand.Execute() + + if testConfig.ProtocolParticipation.CutoverBlock != 124000 { + t.Errorf( + "expected the config file cutover block [124000], got [%d]", + testConfig.ProtocolParticipation.CutoverBlock, + ) + } + if !testConfig.ProtocolParticipation.CutoverBlockSet { + t.Error("expected the config file key to be detected as supplied") + } + + if _, err := participation.ResolveAndValidate( + commonEthereum.Mainnet, + testConfig.ProtocolParticipation, + ); err == nil { + t.Error("expected mainnet to reject the config file override") + } +} + +// TestFlags_ProtocolParticipationZeroFromConfigFile proves that an explicit +// `[protocolParticipation] CutoverBlock = 0` config file key is detected as +// present even though its decoded value equals the flag default, so mainnet +// rejects it by presence and the rejection names the offending key. +func TestFlags_ProtocolParticipationZeroFromConfigFile(t *testing.T) { + testCommand, testConfig, _ := initTestCommand() + + testCommand.SetArgs([]string{ + "--config", "../test/config_participation_cutover_zero.toml", + }) + + testCommand.Execute() + + if testConfig.ProtocolParticipation.CutoverBlock != 0 { + t.Errorf( + "expected the config file cutover block [0], got [%d]", + testConfig.ProtocolParticipation.CutoverBlock, + ) + } + if !testConfig.ProtocolParticipation.CutoverBlockSet { + t.Error( + "expected the explicit zero config file key to be detected as " + + "supplied", + ) + } + + _, err := participation.ResolveAndValidate( + commonEthereum.Mainnet, + testConfig.ProtocolParticipation, + ) + if err == nil { + t.Fatal("expected mainnet to reject the explicit zero override") + } + if !strings.Contains(err.Error(), "protocolParticipation.cutoverBlock") { + t.Errorf( + "expected the rejection to name the offending key, got: [%v]", + err, + ) + } +} diff --git a/cmd/maintainer.go b/cmd/maintainer.go index b8d81e2aa5..8f8a8c1611 100644 --- a/cmd/maintainer.go +++ b/cmd/maintainer.go @@ -6,10 +6,15 @@ import ( "github.com/spf13/cobra" + "github.com/keep-network/keep-core/build" "github.com/keep-network/keep-core/config" + "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/bitcoin/electrum" "github.com/keep-network/keep-core/pkg/chain/ethereum" + "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/maintainer" + "github.com/keep-network/keep-core/pkg/maintainer/spv" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) // MaintainerCommand contains the definition of the maintainer command-line @@ -72,6 +77,15 @@ func maintainers(cmd *cobra.Command, args []string) error { ) } + // Wire client-info metrics when the client-info endpoint is enabled (opt-in + // via [clientInfo] Port / --clientInfo.port). This must happen before + // maintainer.Initialize so the SPV control loop records its redemption-proof + // counters through a recorder that is already in place. When the port is 0 + // the endpoint stays disabled and the recorder stays nil, so proof + // submission is unaffected. + stopMetrics := wireMaintainerMetrics(ctx, clientConfig, btcChain) + defer stopMetrics() + maintainer.Initialize( ctx, clientConfig.Maintainer, @@ -83,3 +97,69 @@ func maintainers(cmd *cobra.Command, args []string) error { <-ctx.Done() return fmt.Errorf("unexpected context cancellation") } + +// initializeMaintainerClientInfo enables the client-info metrics endpoint for +// the maintainer process when a client-info port is configured, and returns a +// PerformanceMetrics recorder wired into that endpoint. It registers the static +// client version information and Bitcoin connectivity that the maintainer has +// the dependencies for; the network/Ethereum peer sources wired by the start +// command are not applicable here. It returns nil when the endpoint is not +// configured (port 0), leaving metrics disabled and proof submission +// unaffected. +func initializeMaintainerClientInfo( + ctx context.Context, + config *config.Config, + btcChain bitcoin.Chain, +) *clientinfo.PerformanceMetrics { + registry, isConfigured := clientinfo.Initialize(ctx, config.ClientInfo.Port) + if !isConfigured { + logger.Infof("client info endpoint not configured") + return nil + } + + registry.RegisterMetricClientInfo( + build.Version, + build.Revision, + participation.CompiledEpoch.String(), + ) + + registry.ObserveBtcConnectivity( + btcChain, + config.ClientInfo.BitcoinMetricsTick, + ) + registry.RegisterBtcChainInfoSource(btcChain) + + performanceMetrics := clientinfo.NewPerformanceMetrics(ctx, registry) + + logger.Infof( + "enabled client info endpoint on port [%v]", + config.ClientInfo.Port, + ) + + return performanceMetrics +} + +// wireMaintainerMetrics enables the optional client-info metrics endpoint and +// wires its PerformanceMetrics recorder into the SPV maintainer. Callers must +// invoke it before maintainer.Initialize so the recorder is in place before the +// SPV control loop starts recording redemption-proof counters. It returns a +// cleanup function that resets the global recorder and stops the metrics +// goroutines; the cleanup is a no-op when the endpoint is disabled (port 0), +// leaving the recorder nil and proof submission unaffected. +func wireMaintainerMetrics( + ctx context.Context, + config *config.Config, + btcChain bitcoin.Chain, +) func() { + performanceMetrics := initializeMaintainerClientInfo(ctx, config, btcChain) + if performanceMetrics == nil { + return func() {} + } + + spv.SetMetricsRecorder(performanceMetrics) + + return func() { + spv.SetMetricsRecorder(nil) + performanceMetrics.Stop() + } +} diff --git a/cmd/maintainer_metrics_test.go b/cmd/maintainer_metrics_test.go new file mode 100644 index 0000000000..e67bd63241 --- /dev/null +++ b/cmd/maintainer_metrics_test.go @@ -0,0 +1,309 @@ +package cmd + +import ( + "context" + "fmt" + "io" + "net" + "net/http" + "strings" + "testing" + "time" + + "github.com/keep-network/keep-core/config" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/maintainer/spv" +) + +// TestMaintainerCommandExposesClientInfoFlags verifies that, after ClientInfo is +// added to config.MaintainerCategories, the maintainer command exposes the +// client-info opt-in flag so metrics can be enabled for the process that runs +// the SPV maintainer. +func TestMaintainerCommandExposesClientInfoFlags(t *testing.T) { + hasClientInfo := false + for _, category := range config.MaintainerCategories { + if category == config.ClientInfo { + hasClientInfo = true + break + } + } + if !hasClientInfo { + t.Fatal("expected config.MaintainerCategories to include ClientInfo") + } + + if flag := MaintainerCommand.Flags().Lookup("clientInfo.port"); flag == nil { + t.Fatal("expected maintainer command to expose the clientInfo.port flag") + } +} + +// TestWireMaintainerMetricsDisabled verifies that a client-info port of 0 leaves +// metrics disabled: no PerformanceMetrics is created and the production wiring +// helper leaves the SPV recorder unset, so proof submission is unaffected. +func TestWireMaintainerMetricsDisabled(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Start from a clean recorder so the assertion reflects this call only. + spv.SetMetricsRecorder(nil) + defer spv.SetMetricsRecorder(nil) + + cfg := &config.Config{} + cfg.ClientInfo.Port = 0 + + // The initializer creates no recorder when the endpoint is disabled... + if pm := initializeMaintainerClientInfo(ctx, cfg, nil); pm != nil { + t.Fatal("expected no performance metrics when client-info port is 0") + } + + // ...and the production wiring helper therefore leaves the SPV recorder nil. + stop := wireMaintainerMetrics(ctx, cfg, nil) + defer stop() + if spv.MetricsRecorder() != nil { + t.Fatal("expected the SPV metrics recorder to stay nil when port is 0") + } +} + +// TestWireMaintainerMetricsEnabled verifies that a configured client-info port +// drives the exact production wiring helper (wireMaintainerMetrics in +// cmd/maintainer.go, the same call the maintainer startup path uses before +// maintainer.Initialize), that the helper actually wires the recorder into the +// SPV maintainer, and that the three SPV redemption-proof series are present at +// zero when the /metrics endpoint is scraped so Prometheus sees them from +// startup. +// +// Driving wireMaintainerMetrics rather than calling spv.SetMetricsRecorder here +// means this test fails if production stops wiring the recorder - the gap the +// previous version could not catch. +// +// This is the single enabled-port test in the cmd package: keep-common's +// EnableServer registers "/metrics" on the global http.DefaultServeMux, which +// panics on a second registration, so all enabled-endpoint assertions live here. +func TestWireMaintainerMetricsEnabled(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Start from a clean recorder and guarantee it is reset even if an + // assertion below fails before the cleanup runs. + spv.SetMetricsRecorder(nil) + defer spv.SetMetricsRecorder(nil) + + port, err := freeTCPPort() + if err != nil { + t.Fatal(err) + } + + cfg := &config.Config{} + cfg.ClientInfo.Port = port + + // Drive the production wiring helper, exactly as the maintainer startup path + // does before maintainer.Initialize. + stop := wireMaintainerMetrics( + ctx, + cfg, + &stubBitcoinChain{latestBlockHeight: 100}, + ) + defer stop() + + // Production wiring must have installed the recorder into the SPV maintainer. + if spv.MetricsRecorder() == nil { + t.Fatal( + "expected wireMaintainerMetrics to wire the SPV metrics recorder", + ) + } + + // The three SPV redemption-proof series must be scrapeable at zero from + // startup so operators never see a gap before the first submission. + redemptionProofSeries := []string{ + "performance_" + clientinfo.MetricRedemptionProofSubmissionsTotal, + "performance_" + clientinfo.MetricRedemptionProofSubmissionsSuccessTotal, + "performance_" + clientinfo.MetricRedemptionProofSubmissionsFailedTotal, + } + + metrics := scrapeMetricsEndpoint(t, port) + for _, series := range redemptionProofSeries { + value, ok := metricValue(metrics, series) + if !ok { + t.Errorf("expected series [%s] to be exposed at /metrics", series) + continue + } + if value != "0" { + t.Errorf( + "expected series [%s] to be zero at startup, got [%s]", + series, + value, + ) + } + } + + // The cutover participation family must be scrapeable at zero from the + // same startup. These series are the evidence the exact-image rehearsal + // reads off a running node, and a recorder that only creates a series on + // first use publishes nothing until the event it is meant to warn about + // has already happened. Asserting them here is what proves the production + // registry exports them: the recorder's own getters answer for a name it + // was never asked to register, so only the exposition settles it. + for _, name := range participationSeriesExposedAtZero { + series := "performance_" + name + value, ok := metricValue(metrics, series) + if !ok { + t.Errorf("expected series [%s] to be exposed at /metrics", series) + continue + } + if value != "0" { + t.Errorf( + "expected series [%s] to be zero at startup, got [%s]", + series, + value, + ) + } + } +} + +// participationSeriesExposedAtZero is the cutover observability contract as an +// operator scraping a node sees it. It mirrors the rehearsal's +// PARTICIPATION_METRICS list; the quarantined-signer count is included because +// it is registered with the rest of the fixed family, so a node exposes it from +// startup whether or not it ever quarantines an output. +var participationSeriesExposedAtZero = []string{ + clientinfo.MetricParticipationGateState, + clientinfo.MetricParticipationCurrentBlock, + clientinfo.MetricParticipationCutoverBlock, + clientinfo.MetricParticipationAllowed, + clientinfo.MetricParticipationActiveCeremonies, + clientinfo.MetricParticipationActiveLegacyCeremonies, + clientinfo.MetricParticipationActiveSecurityV2Ceremonies, + clientinfo.MetricParticipationModeLegacyTotal, + clientinfo.MetricParticipationModeSecurityV2Total, + clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal, + clientinfo.MetricParticipationRefusalsTotal, + clientinfo.MetricParticipationCommitRefusalsTotal, + clientinfo.MetricParticipationClockErrorsTotal, + clientinfo.MetricParticipationClockAbortsTotal, + clientinfo.MetricParticipationQuiesceTotal, + clientinfo.MetricParticipationQuiesceForcedAbortsTotal, + clientinfo.MetricParticipationQuarantinedTBTCSigners, +} + +// scrapeMetricsEndpoint fetches the /metrics body from the client-info endpoint, +// retrying briefly while the server goroutine started by EnableServer comes up. +func scrapeMetricsEndpoint(t *testing.T, port int) string { + t.Helper() + + url := fmt.Sprintf("http://127.0.0.1:%d/metrics", port) + + var lastErr error + for attempt := 0; attempt < 50; attempt++ { + resp, err := http.Get(url) //nolint:gosec // fixed loopback test URL + if err != nil { + lastErr = err + time.Sleep(20 * time.Millisecond) + continue + } + + body, err := io.ReadAll(resp.Body) + resp.Body.Close() + if err != nil { + t.Fatalf("could not read /metrics response: %v", err) + } + return string(body) + } + + t.Fatalf("could not scrape /metrics on port %d: %v", port, lastErr) + return "" +} + +// metricValue extracts the value of a non-labelled metric series from the +// exposed text. Lines are formatted as " ". +func metricValue(metrics, series string) (string, bool) { + for _, line := range strings.Split(metrics, "\n") { + fields := strings.Fields(line) + if len(fields) >= 2 && fields[0] == series { + return fields[1], true + } + } + return "", false +} + +// freeTCPPort asks the OS for an unused TCP port. +func freeTCPPort() (int, error) { + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + return 0, err + } + defer listener.Close() + return listener.Addr().(*net.TCPAddr).Port, nil +} + +// stubBitcoinChain is a minimal bitcoin.Chain for the client-info wiring test. +// Only GetLatestBlockHeight is used by the connectivity observer; every other +// method panics to catch unexpected use. +type stubBitcoinChain struct { + latestBlockHeight uint +} + +func (s *stubBitcoinChain) GetLatestBlockHeight() (uint, error) { + return s.latestBlockHeight, nil +} + +func (s *stubBitcoinChain) GetTransaction(bitcoin.Hash) (*bitcoin.Transaction, error) { + panic("unexpected GetTransaction call") +} + +func (s *stubBitcoinChain) GetTransactionConfirmations(bitcoin.Hash) (uint, error) { + panic("unexpected GetTransactionConfirmations call") +} + +func (s *stubBitcoinChain) BroadcastTransaction(*bitcoin.Transaction) error { + panic("unexpected BroadcastTransaction call") +} + +func (s *stubBitcoinChain) GetBlockHeader(uint) (*bitcoin.BlockHeader, error) { + panic("unexpected GetBlockHeader call") +} + +func (s *stubBitcoinChain) GetTransactionMerkleProof( + bitcoin.Hash, + uint, +) (*bitcoin.TransactionMerkleProof, error) { + panic("unexpected GetTransactionMerkleProof call") +} + +func (s *stubBitcoinChain) GetTransactionsForPublicKeyHash( + [20]byte, + int, +) ([]*bitcoin.Transaction, error) { + panic("unexpected GetTransactionsForPublicKeyHash call") +} + +func (s *stubBitcoinChain) GetTxHashesForPublicKeyHash( + [20]byte, +) ([]bitcoin.Hash, error) { + panic("unexpected GetTxHashesForPublicKeyHash call") +} + +func (s *stubBitcoinChain) GetMempoolForPublicKeyHash( + [20]byte, +) ([]*bitcoin.Transaction, error) { + panic("unexpected GetMempoolForPublicKeyHash call") +} + +func (s *stubBitcoinChain) GetUtxosForPublicKeyHash( + [20]byte, +) ([]*bitcoin.UnspentTransactionOutput, error) { + panic("unexpected GetUtxosForPublicKeyHash call") +} + +func (s *stubBitcoinChain) GetMempoolUtxosForPublicKeyHash( + [20]byte, +) ([]*bitcoin.UnspentTransactionOutput, error) { + panic("unexpected GetMempoolUtxosForPublicKeyHash call") +} + +func (s *stubBitcoinChain) EstimateSatPerVByteFee(uint32) (int64, error) { + panic("unexpected EstimateSatPerVByteFee call") +} + +func (s *stubBitcoinChain) GetCoinbaseTxHash(uint) (bitcoin.Hash, error) { + panic("unexpected GetCoinbaseTxHash call") +} diff --git a/cmd/participation-state-audit/main.go b/cmd/participation-state-audit/main.go new file mode 100644 index 0000000000..3ae894586a --- /dev/null +++ b/cmd/participation-state-audit/main.go @@ -0,0 +1,7315 @@ +// Command participation-state-audit classifies a stopped node's persisted +// protocol state for the rollback barrier, without exposing private material. +// +// It records the snapshot identity (an aggregate checksum over every at-rest +// file and the root access mode), inventories the keystore and work +// namespaces, flags any entry the expected storage layout does not contain, +// and — when the storage password is supplied — interprets the beacon active, +// beacon quarantine, and tBTC active namespaces with the same decode paths +// the client's own loaders use. Every inconsistency is a finding: records +// that fail to decrypt or decode, quarantine halves missing their partner, +// quarantine metadata that contradicts its schema, epoch, mode, anchor, +// directory, or decrypted membership, groups present in both the active and +// quarantine namespaces, and records stored under a directory their content +// does not match. +// +// The tool MUST run against a snapshot copy of the node's storage, never the +// live directory: opening the standard persistence handles creates their +// bookkeeping subdirectories and probes write permission, and a rollback +// audit must not mutate the original evidence. +// +// Namespace consistency alone is deliberately insufficient for the rollback +// barrier. Chain reconciliation (wallet/group registration and DKG +// settlement, for active and quarantined state alike, plus the inactivity +// claims a heartbeat filed against the WalletRegistry and the relay entry +// timeout penalties a monitor earned from the RandomBeacon), Bitcoin +// transaction reconciliation, the quiescence outcome report, and prior-reader +// compatibility evidence are produced outside this offline tool; until a +// reference to each is supplied and recorded, the manifest reports the +// missing pieces as rollback blockers and the process exits nonzero. Every +// evidence record must additionally bind to the operator-supplied expected +// operational identities — Ethereum chain ID, Bitcoin network, the exact +// prior and current release versions and revisions, both immutable image +// digests, the compiled release epoch, and the cutover block — and fall +// within the evidence freshness bound: schema-valid evidence for the wrong +// target, the wrong artifact, the wrong cutover schedule, or from long +// before the rollback decision blocks the barrier exactly like missing +// evidence. This tool's output never authorizes activating quarantined +// material by itself. +package main + +import ( + "bytes" + "crypto/ed25519" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "flag" + "fmt" + "io/fs" + "math/big" + "os" + "path/filepath" + "slices" + "sort" + "strconv" + "strings" + "sync" + "time" + + "github.com/ethereum/go-ethereum/accounts/abi" + ethereumCrypto "github.com/ethereum/go-ethereum/crypto" + bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + + "github.com/keep-network/keep-core/config" + "github.com/keep-network/keep-core/pkg/altbn128" + "github.com/keep-network/keep-core/pkg/beacon/registry" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/bls" + beaconabi "github.com/keep-network/keep-core/pkg/chain/ethereum/beacon/gen/abi" + ecdsaabi "github.com/keep-network/keep-core/pkg/chain/ethereum/ecdsa/gen/abi" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/storage" + "github.com/keep-network/keep-core/pkg/tbtc" +) + +// manifestSchemaVersion versions the audit manifest document. +const manifestSchemaVersion = uint32(7) + +// The audited namespaces, relative to the storage root. The beacon quarantine +// namespace is a sibling of the active beacon keystore precisely so the +// active-group scan cannot read it; the audit re-verifies that separation. +const ( + beaconKeystoreNamespace = "keystore/beacon" + beaconQuarantineNamespace = "keystore/beacon-quarantine" + tbtcKeystoreNamespace = "keystore/tbtc" + tbtcQuarantineNamespace = "keystore/tbtc-quarantine" + tbtcWorkNamespace = "work/tbtc" + participationWorkNamespace = "work/participation" +) + +// The expected storage layout at each level. Any other entry is a finding: +// state this audit cannot classify must block the rollback barrier, not pass +// silently, and a namespace added by a later release must extend this audit +// in the same change. +var ( + knownRootEntries = []string{"keystore", "work"} + knownKeystoreEntries = []string{ + "beacon", + "beacon-quarantine", + "tbtc", + "tbtc-quarantine", + } + knownWorkEntries = []string{"participation", "tbtc"} +) + +// tbtcWorkPreparamsMarker classifies tECDSA pre-parameter pool records inside +// the tBTC work namespace; the pool is regenerable material, not ceremony +// state. +const tbtcWorkPreparamsMarker = "/preparams/" + +type fileRecord struct { + // Path is relative to the storage root. + Path string `json:"path"` + Bytes int64 `json:"bytes"` + // SHA256 is the checksum of the at-rest bytes. Key-holding files are + // encrypted at rest, so the checksum commits to the snapshot content + // without exposing key material. + SHA256 string `json:"sha256"` +} + +type namespaceInventory struct { + Name string `json:"name"` + Present bool `json:"present"` + Files []fileRecord `json:"files"` +} + +// snapshotIdentity commits the manifest to one exact snapshot: the aggregate +// checksum binds every inventoried file, and the root mode records the access +// controls the snapshot was audited under. +type snapshotIdentity struct { + Path string `json:"path"` + RootMode string `json:"root_mode"` + TotalFiles int `json:"total_files"` + TotalBytes int64 `json:"total_bytes"` + AggregateSHA256 string `json:"aggregate_sha256"` +} + +// evidenceRecord references one externally produced rollback-evidence input. +// The audit records the reference and its checksum, validates the record +// against its schema, and binds it to this exact snapshot; a record that +// fails validation stays a rollback blocker exactly like a missing one. +type evidenceRecord struct { + Name string `json:"name"` + Supplied bool `json:"supplied"` + Valid bool `json:"valid"` + Path string `json:"path,omitempty"` + SHA256 string `json:"sha256,omitempty"` +} + +// evidenceSchemaVersion versions the external rollback-evidence record +// schemas this audit accepts. +const evidenceSchemaVersion uint32 = 8 + +const chainReconciliationSignatureDomain = "keep-core/" + + "participation-state-audit/chain-reconciliation/v8\x00" + +// evidenceFutureSkewAllowance bounds how far in the future an evidence +// record's generation time may lie relative to this audit before it is a +// violation; it absorbs ordinary clock skew between the evidence generator +// and the audit host. +const evidenceFutureSkewAllowance = 5 * time.Minute + +// evidenceEnvelope is the common header of every external rollback-evidence +// record. The snapshot binding makes a record usable for exactly one audited +// snapshot: evidence generated for different storage cannot authorize this +// rollback. +type evidenceEnvelope struct { + SchemaVersion uint32 `json:"schema_version"` + EvidenceType string `json:"evidence_type"` + GeneratedAt time.Time `json:"generated_at"` + SnapshotAggregateSHA256 string `json:"snapshot_aggregate_sha256"` +} + +// ethereumLogEvidence identifies one canonical Ethereum log. The external +// evidence generator obtains these values from a receipt on the expected +// chain; the audit requires every event in the DKG lineage to name its exact +// transaction, block, and log position instead of accepting a free-standing +// settlement label. +type ethereumLogEvidence struct { + TransactionHash string `json:"transaction_hash"` + BlockHash string `json:"block_hash"` + BlockNumber uint64 `json:"block_number"` + LogIndex uint64 `json:"log_index"` +} + +// ethereumRawLogEvidence is the exact log projection returned in an Ethereum +// transaction receipt. Event summaries below are accepted only when the +// corresponding authenticated receipt contains byte-identical topics/data +// emitted by the expected WalletRegistry address. +type ethereumRawLogEvidence struct { + Address string `json:"address"` + Topics []string `json:"topics"` + Data string `json:"data"` + LogIndex uint64 `json:"log_index"` +} + +// ethereumReceiptEvidence carries the receipt fields needed to authenticate +// an event observation. The collector signature authenticates these fields; +// the audit still independently requires successful status, canonical block +// membership, exact transaction/block identity, and an exact raw event log. +type ethereumReceiptEvidence struct { + TransactionHash string `json:"transaction_hash"` + BlockHash string `json:"block_hash"` + BlockNumber uint64 `json:"block_number"` + TransactionIndex uint64 `json:"transaction_index"` + Status uint64 `json:"status"` + Logs []ethereumRawLogEvidence `json:"logs"` +} + +type ethereumCanonicalBlockEvidence struct { + BlockNumber uint64 `json:"block_number"` + BlockHash string `json:"block_hash"` +} + +// ethereumCollectorAttestation makes the chain record an authenticated +// collector artifact instead of a caller-authored decoded summary. Signature +// is Ed25519 over the domain-separated canonical JSON encoding of the entire +// chainReconciliationEvidence with this field empty. The trusted public key, +// expected finalized block, and expected WalletRegistry are independent audit +// inputs and therefore cannot be selected by the evidence generator. +type ethereumCollectorAttestation struct { + FinalizedBlockNumber uint64 `json:"finalized_block_number"` + FinalizedBlockHash string `json:"finalized_block_hash"` + CanonicalBlocks []ethereumCanonicalBlockEvidence `json:"canonical_blocks"` + Signature string `json:"signature"` +} + +// tbtcDKGChainResultEvidence is the complete EcdsaDkg.Result tuple emitted by +// DkgResultSubmitted. The audit ABI-encodes this tuple exactly as the +// WalletRegistry contract does and recomputes keccak256(abi.encode(result)). +// Summary-only inputs are deliberately insufficient: group size, +// misbehaviour, members hash, and wallet identity are all derived from these +// event bytes. +type tbtcDKGChainResultEvidence struct { + SubmitterMemberIndex uint16 `json:"submitter_member_index"` + GroupPublicKey string `json:"group_public_key"` + MisbehavedMemberIndexes []uint8 `json:"misbehaved_member_indexes"` + Signatures string `json:"signatures"` + SigningMemberIndexes []*big.Int `json:"signing_member_indexes"` + Members []uint32 `json:"members"` + MembersHash string `json:"members_hash"` +} + +type tbtcDKGStartedEventEvidence struct { + ethereumLogEvidence + Seed string `json:"seed"` +} + +type tbtcDKGResultSubmittedEventEvidence struct { + ethereumLogEvidence + ResultHash string `json:"result_hash"` + Seed string `json:"seed"` + Result tbtcDKGChainResultEvidence `json:"result"` +} + +type tbtcDKGResultApprovedEventEvidence struct { + ethereumLogEvidence + ResultHash string `json:"result_hash"` +} + +type tbtcWalletCreatedEventEvidence struct { + ethereumLogEvidence + WalletID string `json:"wallet_id"` + DKGResultHash string `json:"dkg_result_hash"` +} + +// tbtcDKGResultEvidence is the complete accepted-event lineage for the result +// that created a persisted tBTC wallet. DkgStarted binds the canonical anchor +// and seed, DkgResultSubmitted carries the bytes hashed by the contract, and +// DkgResultApproved plus WalletCreated prove the same result reached the +// wallet-creation transition. +type tbtcDKGResultEvidence struct { + Started tbtcDKGStartedEventEvidence `json:"started"` + Submitted tbtcDKGResultSubmittedEventEvidence `json:"submitted"` + Approved tbtcDKGResultApprovedEventEvidence `json:"approved"` + WalletCreated tbtcWalletCreatedEventEvidence `json:"wallet_created"` +} + +func (e *tbtcDKGResultEvidence) resultHash() string { + return strings.TrimPrefix(e.Submitted.ResultHash, "0x") +} + +func (e *tbtcDKGResultEvidence) seedHash() (string, error) { + seedBytes, err := decodeCanonicalEthereumBytes(e.Started.Seed, 32) + if err != nil { + return "", err + } + + seed := new(big.Int).SetBytes(seedBytes) + hash := sha256.Sum256(seed.Bytes()) + return hex.EncodeToString(hash[:]), nil +} + +func (e *tbtcDKGResultEvidence) startBlock() uint64 { + return e.Started.BlockNumber +} + +func (e *tbtcDKGResultEvidence) originalGroupSize() uint16 { + return uint16(len(e.Submitted.Result.Members)) +} + +func (e *tbtcDKGResultEvidence) misbehavedMemberIndexes() []uint8 { + return e.Submitted.Result.MisbehavedMemberIndexes +} + +type tbtcWalletChainEvidence struct { + WalletStorageKey string `json:"wallet_storage_key"` + WalletID string `json:"wallet_id"` + Registered bool `json:"registered"` + DKGSettlement string `json:"dkg_settlement"` + DKGResult *tbtcDKGResultEvidence `json:"dkg_result,omitempty"` +} + +// chainReconciliationEvidence records the on-chain wallet/group registration +// and DKG settlement state for every persisted group in the snapshot. An +// accepted tBTC result is part of the wallet identity: a bare "approved" +// assertion cannot prove which ceremony and original member produced a +// persisted final membership. +type chainReconciliationEvidence struct { + evidenceEnvelope + + EthereumChainID string `json:"ethereum_chain_id"` + WalletRegistryAddress string `json:"wallet_registry_address"` + RandomBeaconAddress string `json:"random_beacon_address"` + Receipts []ethereumReceiptEvidence `json:"receipts"` + CollectorAttestation ethereumCollectorAttestation `json:"collector_attestation"` + Wallets []tbtcWalletChainEvidence `json:"wallets"` + BeaconGroups []struct { + GroupPublicKey string `json:"group_public_key"` + Registered bool `json:"registered"` + } `json:"beacon_groups"` +} + +// bitcoinReconciliationEvidence records every pending Bitcoin transaction of +// the audited wallets and its mempool/chain state. +type bitcoinReconciliationEvidence struct { + evidenceEnvelope + + BitcoinNetwork string `json:"bitcoin_network"` + // Complete attests the generator enumerated every pending transaction; an + // explicitly incomplete reconciliation cannot authorize the barrier. + Complete bool `json:"complete"` + PendingTransactions []struct { + TransactionHash string `json:"transaction_hash"` + State string `json:"state"` + } `json:"pending_transactions"` +} + +// quiescenceReportEvidence records the permits active at process quiescence +// and each one's terminal outcome. The quiescing node also attests its own +// exact artifact identity — release version and revision — and the compiled +// epoch and armed cutover block it quiesced under, so the report cannot vouch +// for the state of a different candidate build or cutover schedule. +type quiescencePermitEvidence struct { + Ceremony string `json:"ceremony"` + Mode string `json:"mode"` + CanonicalStartBlock uint64 `json:"canonical_start_block"` + WorkID string `json:"work_id"` + PermitID string `json:"permit_id"` + Outcome string `json:"outcome"` +} + +type quiescenceReportEvidence struct { + evidenceEnvelope + + ReleaseVersion string `json:"release_version"` + ReleaseRevision string `json:"release_revision"` + ReleaseEpoch string `json:"release_epoch"` + CutoverBlock uint64 `json:"cutover_block"` + QuiesceCause string `json:"quiesce_cause"` + ActivePermitsAtQuiescence []quiescencePermitEvidence `json:"active_permits_at_quiescence"` +} + +// priorReaderCompatibilityEvidence records the tested prior release and its +// result against every schema this release writes, including loading and +// signing with a wallet created after the cutover block. Both sides of the +// test are pinned exactly: the prior artifact that performed the reads and +// the current release artifact that wrote the tested schemas, each with its +// version, revision, and immutable image digest. +type priorReaderCompatibilityEvidence struct { + evidenceEnvelope + + PriorVersion string `json:"prior_version"` + PriorRevision string `json:"prior_revision"` + PriorImageDigest string `json:"prior_image_digest"` + ReleaseVersion string `json:"release_version"` + ReleaseRevision string `json:"release_revision"` + ReleaseImageDigest string `json:"release_image_digest"` + SchemaResults []struct { + Schema string `json:"schema"` + Compatible bool `json:"compatible"` + } `json:"schema_results"` +} + +// The prior-reader compatibility evidence must cover every schema whose +// unreadability makes the prior-binary rollback an unacceptable mechanism. +var requiredPriorReaderSchemas = []string{ + "beacon_membership", + "tbtc_membership", + "post_cutover_wallet_load_and_sign", +} + +// Valid DKG settlement states of a reconciled tBTC wallet. "approved" is the +// only state that permits persisted active signers; "none" — no DKG result +// on chain references the wallet — is the only state that permits a +// quarantined-only share, because a pending or challenged result may still +// settle into an on-chain wallet whose share the prior binary cannot load. +var validDKGSettlementStates = map[string]struct{}{ + "approved": {}, + "pending": {}, + "challenged": {}, + "none": {}, +} + +// Valid terminal states of a reconciled pending Bitcoin transaction. +var validBitcoinTransactionStates = map[string]struct{}{ + "signed": {}, + "broadcast": {}, + "mined": {}, + "absent": {}, +} + +// Valid terminal outcomes of a permit active at quiescence. +var validQuiescencePermitOutcomes = map[string]struct{}{ + "completed": {}, + "quarantined": {}, + "exhausted": {}, +} + +type beaconMembershipRecord struct { + GroupPublicKey string `json:"group_public_key"` + MemberIndex uint8 `json:"member_index"` + ChannelName string `json:"channel_name"` +} + +type beaconQuarantineRecord struct { + registry.QuarantinedSignerMetadata + + // HasMembershipRecord reports whether the preserved membership bytes + // accompany the metadata; metadata without the membership means the key + // material was lost and the record is evidence only. + HasMembershipRecord bool `json:"has_membership_record"` + // HasHandoffRecord reports whether the output was preserved as the single + // combined record rather than as the pair. It says how the namespace took + // the output, not how complete it is: a handoff carries both halves, and + // the fields above are filled from it when the pair is missing one. + HasHandoffRecord bool `json:"has_handoff_record"` +} + +// tbtcWalletRecord summarizes the decoded signer records of one wallet in the +// tBTC active namespace. +type tbtcWalletRecord struct { + WalletStorageKey string `json:"wallet_storage_key"` + // WalletID is the ECDSA wallet ID derived from the decoded wallet public + // key — the identity chain reconciliation evidence must match exactly. + WalletID string `json:"wallet_id"` + // WalletPublicKeyHash is the Bitcoin public key hash derived from the same + // decoded wallet public key. It is the identity a tBTC permit names itself + // with, so it is what binds a node-authored outcome to the registry wallet + // its chain settlement is allowed to name. + WalletPublicKeyHash string `json:"wallet_public_key_hash"` + MemberIndexes []uint8 `json:"member_indexes"` + SigningGroupSize int `json:"signing_group_size"` +} + +type tbtcQuarantineRecord struct { + tbtc.QuarantinedSignerMetadata + + // WalletStorageKey is the quarantine directory the output was preserved + // under — the same public-key-derived key the active namespace uses — so + // chain reconciliation can match quarantined and active state of the + // same wallet one-to-one. + WalletStorageKey string `json:"wallet_storage_key"` + // SignerWalletID is the ECDSA wallet ID derived from the preserved + // signer's decoded public key. Unlike the metadata's wallet ID — recorded + // best-effort at preservation time — it is derived from the key material + // itself, so it stays the authoritative identity for chain + // reconciliation when the metadata half is incomplete. + SignerWalletID string `json:"signer_wallet_id,omitempty"` + // SignerWalletPublicKeyHash is the Bitcoin public key hash derived from + // the same preserved key material, and is authoritative for the same + // reason. Paired with SignerWalletID it tells the audit which registry + // wallet a permit naming this public key hash is about. + SignerWalletPublicKeyHash string `json:"signer_wallet_public_key_hash,omitempty"` + // HasMembershipRecord reports whether the preserved signer bytes + // accompany the metadata; metadata without the signer means the key + // material was lost and the record is evidence only. + HasMembershipRecord bool `json:"has_membership_record"` + // HasHandoffRecord reports whether the output was preserved as the single + // combined record rather than as the pair. It says how the namespace took + // the output, not how complete it is: a handoff carries both halves, and + // the fields above are filled from it when the pair is missing one. + HasHandoffRecord bool `json:"has_handoff_record"` +} + +type manifest struct { + SchemaVersion uint32 `json:"schema_version"` + GeneratedAt time.Time `json:"generated_at"` + Snapshot snapshotIdentity `json:"snapshot"` + // Interpreted reports whether the storage password was supplied and the + // beacon and tBTC namespaces were decoded; without it the manifest is a + // raw inventory only. + Interpreted bool `json:"interpreted"` + Namespaces []namespaceInventory `json:"namespaces"` + + BeaconActiveMemberships []beaconMembershipRecord `json:"beacon_active_memberships,omitempty"` + BeaconQuarantinedOutputs []beaconQuarantineRecord `json:"beacon_quarantined_outputs,omitempty"` + TBTCActiveWallets []tbtcWalletRecord `json:"tbtc_active_wallets,omitempty"` + TBTCQuarantinedOutputs []tbtcQuarantineRecord `json:"tbtc_quarantined_outputs,omitempty"` + // TBTCWorkClassification counts the tBTC work-namespace files by class; + // an unclassified work record is additionally a finding. + TBTCWorkClassification map[string]int `json:"tbtc_work_classification,omitempty"` + // QuiescenceSnapshot is decoded from the node-authored encrypted + // work/participation artifact. External evidence cannot replace this + // inventory; terminal outcomes reconcile against it. + QuiescenceSnapshot *participation.QuiescenceSnapshot `json:"quiescence_snapshot,omitempty"` + // ParticipationTerminalOutcomes is decoded from the node-authored terminal + // journal beside the gate snapshot. External quiescence evidence must match + // this record exactly; it cannot author a completed outcome. + ParticipationTerminalOutcomes *participation.TerminalOutcomeJournal `json:"participation_terminal_outcomes,omitempty"` + + // Findings lists every inconsistency; an empty list with Interpreted true + // means the namespaces are internally consistent. + Findings []string `json:"findings"` + // Consistent is true when interpretation ran and produced no findings. + // It classifies namespace integrity only and never means rollback-ready + // by itself. + Consistent bool `json:"consistent"` + + // ExpectedIdentity records the operator-supplied operational identities + // the external evidence was bound to; a missing input is a rollback + // blocker, never a silently skipped check. + ExpectedIdentity expectedIdentityRecord `json:"expected_identity"` + + // ExternalEvidence records the externally produced rollback inputs this + // offline tool cannot derive; RollbackBlockers names every one still + // missing, plus any finding that blocks the barrier. + ExternalEvidence []evidenceRecord `json:"external_evidence"` + RollbackBlockers []string `json:"rollback_blockers"` + RollbackBarrierReady bool `json:"rollback_barrier_ready"` +} + +// expectedIdentityRecord is the manifest's evidence trail of the expected +// operational identities the audit ran with. +type expectedIdentityRecord struct { + EthereumChainID string `json:"ethereum_chain_id,omitempty"` + WalletRegistryAddress string `json:"wallet_registry_address,omitempty"` + RandomBeaconAddress string `json:"random_beacon_address,omitempty"` + FinalizedEthereumBlockNumber uint64 `json:"finalized_ethereum_block_number,omitempty"` + FinalizedEthereumBlockHash string `json:"finalized_ethereum_block_hash,omitempty"` + ChainEvidencePublicKeySHA256 string `json:"chain_evidence_public_key_sha256,omitempty"` + BitcoinNetwork string `json:"bitcoin_network,omitempty"` + PriorVersion string `json:"prior_version,omitempty"` + PriorRevision string `json:"prior_revision,omitempty"` + PriorImageDigest string `json:"prior_image_digest,omitempty"` + ReleaseVersion string `json:"release_version,omitempty"` + ReleaseRevision string `json:"release_revision,omitempty"` + ReleaseImageDigest string `json:"release_image_digest,omitempty"` + ReleaseEpoch string `json:"release_epoch,omitempty"` + CutoverBlock uint64 `json:"cutover_block,omitempty"` + MaxEvidenceAge string `json:"max_evidence_age,omitempty"` +} + +// evidenceInputs carries the externally produced rollback-evidence references +// supplied on the command line. +type evidenceInputs struct { + chainReconciliation string + bitcoinReconciliation string + quiescenceReport string + priorReaderCompatibility string +} + +// expectedIdentityInputs carries the operator-supplied expected operational +// identities the audit binds the external evidence to. Every rollback-grade +// run must supply all of them: without an expected chain, network, exact +// prior and current artifact identities, immutable image digests, compiled +// epoch, cutover block, and freshness bound, schema-valid evidence generated +// against the wrong target — or long before the rollback decision — would +// pass. A missing input is a rollback blocker, not a skipped check. +type expectedIdentityInputs struct { + ethereumChainID string + walletRegistryAddress string + randomBeaconAddress string + finalizedEthereumBlockNumber uint64 + finalizedEthereumBlockHash string + chainEvidencePublicKey string + bitcoinNetwork string + priorVersion string + priorRevision string + priorImageDigest string + releaseVersion string + releaseRevision string + releaseImageDigest string + releaseEpoch string + cutoverBlock uint64 + maxEvidenceAge time.Duration +} + +func main() { + var storageDir string + var outputPath string + var evidence evidenceInputs + var expected expectedIdentityInputs + + flag.StringVar( + &storageDir, + "storage-snapshot", + "", + "path to a snapshot copy of the node's storage directory (required); "+ + "never point this at a live node's storage", + ) + flag.StringVar( + &outputPath, + "output", + "", + "write the manifest to this file instead of stdout", + ) + flag.StringVar( + &evidence.chainReconciliation, + "chain-reconciliation-evidence", + "", + "path to the Ethereum reconciliation record: wallet/group "+ + "registration and DKG settlement state for every persisted group, "+ + "including the complete accepted tBTC DKG event lineage", + ) + flag.StringVar( + &evidence.bitcoinReconciliation, + "bitcoin-reconciliation-evidence", + "", + "path to the Bitcoin reconciliation record: every pending "+ + "transaction and whether it is signed, broadcast, mined, or absent", + ) + flag.StringVar( + &evidence.quiescenceReport, + "quiescence-report", + "", + "path to the external quiescence reconciliation record: it must match "+ + "the node-authored gate inventory and terminal-outcome journal", + ) + flag.StringVar( + &evidence.priorReaderCompatibility, + "prior-reader-compatibility-evidence", + "", + "path to the prior-release reader compatibility record: the tested "+ + "prior version and its result against every schema this release "+ + "writes", + ) + flag.StringVar( + &expected.ethereumChainID, + "expected-ethereum-chain-id", + "", + "the Ethereum chain ID the rollback targets; the chain "+ + "reconciliation evidence must record exactly this chain", + ) + flag.StringVar( + &expected.walletRegistryAddress, + "expected-wallet-registry-address", + "", + "the exact WalletRegistry address whose authenticated receipt logs "+ + "may establish tBTC DKG settlement", + ) + flag.StringVar( + &expected.randomBeaconAddress, + "expected-random-beacon-address", + "", + "the exact RandomBeacon address whose authenticated receipt logs may "+ + "establish beacon relay entry request, delivery, and timeout "+ + "settlement", + ) + flag.Uint64Var( + &expected.finalizedEthereumBlockNumber, + "expected-finalized-ethereum-block-number", + 0, + "the independently obtained finalized Ethereum block number anchoring "+ + "the chain collector attestation", + ) + flag.StringVar( + &expected.finalizedEthereumBlockHash, + "expected-finalized-ethereum-block-hash", + "", + "the independently obtained finalized Ethereum block hash anchoring "+ + "the chain collector attestation", + ) + flag.StringVar( + &expected.chainEvidencePublicKey, + "expected-chain-evidence-public-key", + "", + "the lowercase hexadecimal Ed25519 public key of the independently "+ + "trusted finalized-chain evidence collector", + ) + flag.StringVar( + &expected.bitcoinNetwork, + "expected-bitcoin-network", + "", + "the Bitcoin network the rollback targets; the Bitcoin "+ + "reconciliation evidence must record exactly this network", + ) + flag.StringVar( + &expected.priorVersion, + "expected-prior-version", + "", + "the exact prior release version the rollback restores; the "+ + "prior-reader compatibility evidence must record exactly this "+ + "version", + ) + flag.StringVar( + &expected.priorRevision, + "expected-prior-revision", + "", + "the exact prior release revision the rollback restores; the "+ + "prior-reader compatibility evidence must record exactly this "+ + "revision", + ) + flag.StringVar( + &expected.priorImageDigest, + "expected-prior-image-digest", + "", + "the immutable sha256 image digest of the prior release artifact the "+ + "rollback restores; the prior-reader compatibility evidence must "+ + "record exactly this digest", + ) + flag.StringVar( + &expected.releaseVersion, + "expected-release-version", + "", + "the exact version of the release being rolled back; the quiescence "+ + "report and prior-reader compatibility evidence must record "+ + "exactly this version", + ) + flag.StringVar( + &expected.releaseRevision, + "expected-release-revision", + "", + "the exact revision of the release being rolled back; the quiescence "+ + "report and prior-reader compatibility evidence must record "+ + "exactly this revision", + ) + flag.StringVar( + &expected.releaseImageDigest, + "expected-release-image-digest", + "", + "the immutable sha256 image digest of the release artifact being "+ + "rolled back; the prior-reader compatibility evidence must record "+ + "exactly this digest", + ) + flag.StringVar( + &expected.releaseEpoch, + "expected-release-epoch", + "", + "the release epoch the audited state was written under; it must "+ + "match this audit build's compiled epoch and the quiescence "+ + "report's recorded epoch", + ) + flag.Uint64Var( + &expected.cutoverBlock, + "expected-cutover-block", + 0, + "the cutover block the audited deployment was armed with; the "+ + "quiescence report and every quarantined output must record "+ + "exactly this block", + ) + flag.DurationVar( + &expected.maxEvidenceAge, + "max-evidence-age", + 24*time.Hour, + "the maximum age of every supplied evidence record; older evidence "+ + "reflects a state the rollback decision cannot rely on", + ) + flag.Parse() + + if storageDir == "" { + fmt.Fprintln(os.Stderr, "the --storage-snapshot flag is required") + flag.Usage() + os.Exit(2) + } + + password := os.Getenv(config.EthereumPasswordEnvVariable) + + auditManifest, err := runAudit(storageDir, password, evidence, expected) + if err != nil { + fmt.Fprintf(os.Stderr, "audit failed: [%v]\n", err) + os.Exit(1) + } + + encoded, err := json.MarshalIndent(auditManifest, "", " ") + if err != nil { + fmt.Fprintf(os.Stderr, "cannot encode the manifest: [%v]\n", err) + os.Exit(1) + } + encoded = append(encoded, '\n') + + if outputPath != "" { + // #nosec G703 G304 (manifest destination provided as the operator's + // explicit output flag) + if err := os.WriteFile(outputPath, encoded, 0o600); err != nil { + fmt.Fprintf(os.Stderr, "cannot write the manifest: [%v]\n", err) + os.Exit(1) + } + } else { + if _, err := os.Stdout.Write(encoded); err != nil { + fmt.Fprintf(os.Stderr, "cannot write the manifest: [%v]\n", err) + os.Exit(1) + } + } + + if !auditManifest.Consistent || !auditManifest.RollbackBarrierReady { + os.Exit(3) + } +} + +// auditRun serializes all finding collection: interpretation drains +// persistence error channels concurrently with the descriptor loops, so every +// mutation of the manifest findings goes through one mutex. +type auditRun struct { + mu sync.Mutex + manifest *manifest + expected expectedIdentityInputs +} + +func (r *auditRun) finding(format string, args ...interface{}) { + r.mu.Lock() + defer r.mu.Unlock() + + r.manifest.Findings = append( + r.manifest.Findings, + fmt.Sprintf(format, args...), + ) +} + +// runAudit produces the audit manifest for the given storage snapshot. An +// empty password skips interpretation and produces a raw inventory whose +// missing interpretation is itself a rollback blocker, exactly like a +// missing expected-identity input. +func runAudit( + storageDir string, + password string, + evidence evidenceInputs, + expected expectedIdentityInputs, +) (*manifest, error) { + info, err := os.Stat(storageDir) + if err != nil { + return nil, fmt.Errorf("cannot read the storage snapshot: [%w]", err) + } + if !info.IsDir() { + return nil, fmt.Errorf( + "the storage snapshot [%s] is not a directory", + storageDir, + ) + } + + run := &auditRun{ + manifest: &manifest{ + SchemaVersion: manifestSchemaVersion, + GeneratedAt: time.Now().UTC(), + Snapshot: snapshotIdentity{ + Path: storageDir, + RootMode: info.Mode().String(), + }, + ExpectedIdentity: expectedIdentityRecord{ + EthereumChainID: expected.ethereumChainID, + WalletRegistryAddress: expected.walletRegistryAddress, + RandomBeaconAddress: expected.randomBeaconAddress, + FinalizedEthereumBlockNumber: expected.finalizedEthereumBlockNumber, + FinalizedEthereumBlockHash: expected.finalizedEthereumBlockHash, + ChainEvidencePublicKeySHA256: chainEvidencePublicKeySHA256( + expected.chainEvidencePublicKey, + ), + BitcoinNetwork: expected.bitcoinNetwork, + PriorVersion: expected.priorVersion, + PriorRevision: expected.priorRevision, + PriorImageDigest: expected.priorImageDigest, + ReleaseVersion: expected.releaseVersion, + ReleaseRevision: expected.releaseRevision, + ReleaseImageDigest: expected.releaseImageDigest, + ReleaseEpoch: expected.releaseEpoch, + CutoverBlock: expected.cutoverBlock, + MaxEvidenceAge: expected.maxEvidenceAge.String(), + }, + }, + expected: expected, + } + auditManifest := run.manifest + + if err := run.scanUnexpectedEntries(storageDir); err != nil { + return nil, err + } + + for _, namespace := range []string{ + beaconKeystoreNamespace, + beaconQuarantineNamespace, + tbtcKeystoreNamespace, + tbtcQuarantineNamespace, + tbtcWorkNamespace, + participationWorkNamespace, + } { + inventory, err := inventoryNamespace(storageDir, namespace) + if err != nil { + return nil, err + } + auditManifest.Namespaces = append(auditManifest.Namespaces, inventory) + } + sealSnapshotIdentity(auditManifest) + + classifyTBTCWork(run) + + if password != "" { + auditManifest.Interpreted = true + if err := interpretKeyStoreNamespaces( + storageDir, + password, + run, + ); err != nil { + return nil, err + } + } else { + run.finding( + "interpretation skipped: the [%s] environment variable is "+ + "not set", + config.EthereumPasswordEnvVariable, + ) + } + + auditManifest.Consistent = auditManifest.Interpreted && + len(auditManifest.Findings) == 0 + + recordMissingExpectedIdentity(run) + if err := recordExternalEvidence(run, evidence); err != nil { + return nil, err + } + if !auditManifest.Consistent { + auditManifest.RollbackBlockers = append( + auditManifest.RollbackBlockers, + "the storage snapshot is not interpreted as consistent; every "+ + "finding must be resolved or the ambiguous state quarantined", + ) + } + auditManifest.RollbackBarrierReady = + len(auditManifest.RollbackBlockers) == 0 + + return auditManifest, nil +} + +// scanUnexpectedEntries flags every directory entry the expected storage +// layout does not contain, at the snapshot root and inside the keystore and +// work roots. An absent root is not a finding — a node that never ran tBTC +// has no work directory — but an entry this audit cannot classify is. +func (r *auditRun) scanUnexpectedEntries(storageDir string) error { + levels := []struct { + relative string + known []string + }{ + {".", knownRootEntries}, + {"keystore", knownKeystoreEntries}, + {"work", knownWorkEntries}, + } + + for _, level := range levels { + entries, err := os.ReadDir(filepath.Join(storageDir, level.relative)) + if os.IsNotExist(err) { + continue + } else if err != nil { + return fmt.Errorf( + "cannot scan the [%s] level of the snapshot: [%w]", + level.relative, + err, + ) + } + + for _, entry := range entries { + known := false + for _, name := range level.known { + if entry.Name() == name { + known = true + break + } + } + if !known { + r.finding( + "unexpected entry [%s] under [%s]: this audit cannot "+ + "classify it and unclassifiable state blocks the "+ + "rollback barrier", + entry.Name(), + level.relative, + ) + } + } + } + + return nil +} + +// inventoryNamespace walks one namespace and records every regular file with +// the checksum of its at-rest bytes. A missing namespace is recorded as +// absent, not an error: a node that never quarantined anything has no +// quarantine directory. +func inventoryNamespace( + storageDir string, + namespace string, +) (namespaceInventory, error) { + inventory := namespaceInventory{Name: namespace} + + root := filepath.Join(storageDir, filepath.FromSlash(namespace)) + if _, err := os.Stat(root); os.IsNotExist(err) { + return inventory, nil + } else if err != nil { + return inventory, fmt.Errorf( + "cannot read namespace [%s]: [%w]", + namespace, + err, + ) + } + inventory.Present = true + + err := filepath.WalkDir(root, func( + path string, + entry fs.DirEntry, + err error, + ) error { + if err != nil { + return err + } + if entry.IsDir() { + return nil + } + // A symlink or other non-regular entry could point outside the + // snapshot; such a snapshot cannot be certified. + if !entry.Type().IsRegular() { + return fmt.Errorf( + "cannot inventory [%s]: non-regular entry in the storage "+ + "snapshot", + path, + ) + } + + // #nosec G304 G122 (path walked from the snapshot root and + // non-regular entries rejected above; checksumming every snapshot + // file is this audit's purpose) + content, err := os.ReadFile(path) + if err != nil { + return fmt.Errorf("cannot read [%s]: [%w]", path, err) + } + checksum := sha256.Sum256(content) + + relative, err := filepath.Rel(storageDir, path) + if err != nil { + return err + } + + inventory.Files = append(inventory.Files, fileRecord{ + Path: filepath.ToSlash(relative), + Bytes: int64(len(content)), + SHA256: hex.EncodeToString(checksum[:]), + }) + return nil + }) + if err != nil { + return inventory, fmt.Errorf( + "cannot inventory namespace [%s]: [%w]", + namespace, + err, + ) + } + + sort.Slice(inventory.Files, func(i, j int) bool { + return inventory.Files[i].Path < inventory.Files[j].Path + }) + + return inventory, nil +} + +// sealSnapshotIdentity derives the aggregate snapshot checksum from the +// sorted per-file checksums, so two audits agree on the snapshot identity +// exactly when they saw byte-identical namespace content. +func sealSnapshotIdentity(auditManifest *manifest) { + aggregate := sha256.New() + for _, namespace := range auditManifest.Namespaces { + for _, file := range namespace.Files { + fmt.Fprintf(aggregate, "%s:%s\n", file.Path, file.SHA256) + auditManifest.Snapshot.TotalFiles++ + auditManifest.Snapshot.TotalBytes += file.Bytes + } + } + auditManifest.Snapshot.AggregateSHA256 = + hex.EncodeToString(aggregate.Sum(nil)) +} + +// classifyTBTCWork classifies the tBTC work namespace from its inventory: +// tECDSA pre-parameter pool records are regenerable material, and anything +// else is unclassifiable work state and therefore a finding. +func classifyTBTCWork(r *auditRun) { + for _, namespace := range r.manifest.Namespaces { + if namespace.Name != tbtcWorkNamespace || !namespace.Present { + continue + } + + classification := make(map[string]int) + for _, file := range namespace.Files { + if strings.Contains(file.Path, tbtcWorkPreparamsMarker) { + classification["tecdsa_preparams"]++ + continue + } + classification["unclassified"]++ + r.finding( + "tbtc work record [%s] is not a recognized work class", + file.Path, + ) + } + if len(classification) > 0 { + r.manifest.TBTCWorkClassification = classification + } + } +} + +// isImmutableImageDigest reports whether the reference is an immutable +// sha256 image digest — "sha256:" followed by 64 hex characters. Tags and +// other mutable references cannot pin a rollback artifact. +func isImmutableImageDigest(reference string) bool { + digest, ok := strings.CutPrefix(reference, "sha256:") + if !ok || len(digest) != 64 { + return false + } + _, err := hex.DecodeString(digest) + return err == nil +} + +// isCanonicalSHA256Hex reports whether value is the canonical textual form of +// a SHA-256 digest: exactly 64 lowercase hexadecimal characters. DKG work +// identities and quarantined seed hashes use this form so case aliases or +// truncated values cannot make unrelated work compare equal during rollback +// reconciliation. +func isCanonicalSHA256Hex(value string) bool { + if len(value) != sha256.Size*2 { + return false + } + for _, character := range value { + if (character < '0' || character > '9') && + (character < 'a' || character > 'f') { + return false + } + } + return true +} + +// decodeCanonicalEthereumBytes decodes an exact-size, lowercase, 0x-prefixed +// Ethereum byte value. Requiring one canonical spelling keeps event identity, +// result hashes, and wallet IDs from acquiring case or prefix aliases. +func decodeCanonicalEthereumBytes(value string, size int) ([]byte, error) { + if len(value) != 2+(size*2) || !strings.HasPrefix(value, "0x") { + return nil, fmt.Errorf( + "expected a 0x-prefixed lowercase hexadecimal value of %d bytes", + size, + ) + } + for _, character := range value[2:] { + if (character < '0' || character > '9') && + (character < 'a' || character > 'f') { + return nil, fmt.Errorf( + "expected a 0x-prefixed lowercase hexadecimal value of %d bytes", + size, + ) + } + } + + decoded, err := hex.DecodeString(value[2:]) + if err != nil { + return nil, fmt.Errorf("cannot decode hexadecimal value: [%v]", err) + } + return decoded, nil +} + +func decodeCanonicalEthereumDynamicBytes(value string) ([]byte, error) { + if len(value) < 2 || len(value)%2 != 0 || !strings.HasPrefix(value, "0x") { + return nil, fmt.Errorf( + "expected an even-length 0x-prefixed lowercase hexadecimal value", + ) + } + for _, character := range value[2:] { + if (character < '0' || character > '9') && + (character < 'a' || character > 'f') { + return nil, fmt.Errorf( + "expected an even-length 0x-prefixed lowercase hexadecimal value", + ) + } + } + + decoded, err := hex.DecodeString(value[2:]) + if err != nil { + return nil, fmt.Errorf("cannot decode hexadecimal value: [%v]", err) + } + return decoded, nil +} + +func chainEvidencePublicKeySHA256(value string) string { + decoded, err := hex.DecodeString(value) + if err != nil || + len(decoded) != ed25519.PublicKeySize || + hex.EncodeToString(decoded) != value { + return "" + } + checksum := sha256.Sum256(decoded) + return hex.EncodeToString(checksum[:]) +} + +func tbtcDKGResultABIValue( + result tbtcDKGChainResultEvidence, +) (ecdsaabi.EcdsaDkgResult, error) { + groupPublicKey, err := decodeCanonicalEthereumBytes( + result.GroupPublicKey, + 64, + ) + if err != nil { + return ecdsaabi.EcdsaDkgResult{}, fmt.Errorf( + "invalid group public key: [%v]", + err, + ) + } + signatures, err := decodeCanonicalEthereumDynamicBytes(result.Signatures) + if err != nil { + return ecdsaabi.EcdsaDkgResult{}, fmt.Errorf( + "invalid signatures: [%v]", + err, + ) + } + membersHashBytes, err := decodeCanonicalEthereumBytes( + result.MembersHash, + 32, + ) + if err != nil { + return ecdsaabi.EcdsaDkgResult{}, fmt.Errorf( + "invalid members hash: [%v]", + err, + ) + } + var membersHash [32]byte + copy(membersHash[:], membersHashBytes) + + signingMemberIndexes := make( + []*big.Int, + len(result.SigningMemberIndexes), + ) + for i, memberIndex := range result.SigningMemberIndexes { + if memberIndex == nil { + return ecdsaabi.EcdsaDkgResult{}, fmt.Errorf( + "signing member index [%d] is null", + i, + ) + } + signingMemberIndexes[i] = new(big.Int).Set(memberIndex) + } + + return ecdsaabi.EcdsaDkgResult{ + SubmitterMemberIndex: new(big.Int).SetUint64( + uint64(result.SubmitterMemberIndex), + ), + GroupPubKey: groupPublicKey, + MisbehavedMembersIndices: result.MisbehavedMemberIndexes, + Signatures: signatures, + SigningMembersIndices: signingMemberIndexes, + Members: result.Members, + MembersHash: membersHash, + }, nil +} + +func computeTBTCDKGResultHash( + result tbtcDKGChainResultEvidence, +) (string, error) { + abiValue, err := tbtcDKGResultABIValue(result) + if err != nil { + return "", err + } + + resultType, err := abi.NewType("tuple", "", []abi.ArgumentMarshaling{ + {Name: "submitterMemberIndex", Type: "uint256"}, + {Name: "groupPubKey", Type: "bytes"}, + {Name: "misbehavedMembersIndices", Type: "uint8[]"}, + {Name: "signatures", Type: "bytes"}, + {Name: "signingMembersIndices", Type: "uint256[]"}, + {Name: "members", Type: "uint32[]"}, + {Name: "membersHash", Type: "bytes32"}, + }) + if err != nil { + return "", fmt.Errorf("cannot construct DKG result ABI type: [%v]", err) + } + + encoded, err := (abi.Arguments{{Type: resultType}}).Pack(abiValue) + if err != nil { + return "", fmt.Errorf("cannot ABI-encode DKG result: [%v]", err) + } + + return "0x" + hex.EncodeToString(ethereumCrypto.Keccak256(encoded)), nil +} + +func computeTBTCDKGMembersHash( + result tbtcDKGChainResultEvidence, +) (string, error) { + misbehaved := make( + map[uint8]struct{}, + len(result.MisbehavedMemberIndexes), + ) + for _, memberIndex := range result.MisbehavedMemberIndexes { + misbehaved[memberIndex] = struct{}{} + } + + operatingMembers := make([]uint32, 0, len(result.Members)) + for i, member := range result.Members { + if _, excluded := misbehaved[uint8(i+1)]; excluded { + continue + } + operatingMembers = append(operatingMembers, member) + } + + uint32SliceType, err := abi.NewType("uint32[]", "uint32[]", nil) + if err != nil { + return "", fmt.Errorf("cannot construct members ABI type: [%v]", err) + } + encoded, err := (abi.Arguments{{Type: uint32SliceType}}).Pack( + operatingMembers, + ) + if err != nil { + return "", fmt.Errorf("cannot ABI-encode operating members: [%v]", err) + } + + return "0x" + hex.EncodeToString(ethereumCrypto.Keccak256(encoded)), nil +} + +func computeTBTCWalletID(groupPublicKey string) (string, error) { + publicKeyBytes, err := decodeCanonicalEthereumBytes(groupPublicKey, 64) + if err != nil { + return "", err + } + uncompressed := append([]byte{0x04}, publicKeyBytes...) + if _, err := ethereumCrypto.UnmarshalPubkey(uncompressed); err != nil { + return "", fmt.Errorf("invalid secp256k1 group public key: [%v]", err) + } + return "0x" + hex.EncodeToString( + ethereumCrypto.Keccak256(publicKeyBytes), + ), nil +} + +// isStableEvidenceID reports whether value can be used as a stable +// chain-work or local-permit identity without colliding with the separators +// used by the rehearsal and audit evidence. The driver accepts the same +// alphabet for non-membership identities. +func isStableEvidenceID(value string) bool { + if value == "" || !isASCIIAlphaNumeric(value[0]) { + return false + } + for i := 1; i < len(value); i++ { + character := value[i] + if !isASCIIAlphaNumeric(character) && + character != '_' && + character != '.' && + character != ':' && + character != '-' { + return false + } + } + return true +} + +func isASCIIAlphaNumeric(character byte) bool { + return (character >= '0' && character <= '9') || + (character >= 'A' && character <= 'Z') || + (character >= 'a' && character <= 'z') +} + +// isCanonicalMemberIndex reports whether value is the canonical decimal +// representation of a real protocol member index. Group indexes start at one; +// leading zeroes and values beyond the one-byte MemberIndex range are aliases +// or invalid memberships and therefore cannot identify a local permit. +func isCanonicalMemberIndex(value string) bool { + memberIndex, err := strconv.ParseUint(value, 10, 8) + if err != nil || memberIndex == 0 || memberIndex > group.MaxMemberIndex { + return false + } + return strconv.FormatUint(memberIndex, 10) == value +} + +// recordMissingExpectedIdentity turns every unsupplied expected-identity +// input into a rollback blocker — evidence that is not bound to an explicit +// operational target can approve a rollback of the wrong chain, network, +// artifact, or cutover schedule — and every unusable one likewise: a mutable +// image reference cannot pin an artifact, and an expected epoch differing +// from this audit build's compiled epoch means the wrong audit tool is +// examining the state. +func recordMissingExpectedIdentity(r *auditRun) { + blocked := []struct { + when bool + blocker string + }{ + { + when: r.expected.ethereumChainID == "", + blocker: "the expected Ethereum chain ID is not supplied: the " + + "chain reconciliation evidence cannot be bound to the " + + "rollback's operational target", + }, + { + when: r.expected.walletRegistryAddress == "", + blocker: "the expected WalletRegistry address is not supplied: " + + "authenticated logs cannot be bound to the rollback's " + + "settlement contract", + }, + { + when: r.expected.walletRegistryAddress != "" && + func() bool { + _, err := decodeCanonicalEthereumBytes( + r.expected.walletRegistryAddress, + 20, + ) + return err != nil + }(), + blocker: fmt.Sprintf( + "the expected WalletRegistry address [%s] is not a "+ + "canonical Ethereum address", + r.expected.walletRegistryAddress, + ), + }, + { + when: r.expected.randomBeaconAddress == "", + blocker: "the expected RandomBeacon address is not supplied: " + + "authenticated logs cannot be bound to the rollback's beacon " + + "relay contract", + }, + { + when: r.expected.randomBeaconAddress != "" && + func() bool { + _, err := decodeCanonicalEthereumBytes( + r.expected.randomBeaconAddress, + 20, + ) + return err != nil + }(), + blocker: fmt.Sprintf( + "the expected RandomBeacon address [%s] is not a canonical "+ + "Ethereum address", + r.expected.randomBeaconAddress, + ), + }, + { + when: r.expected.finalizedEthereumBlockNumber == 0, + blocker: "the expected finalized Ethereum block number is not " + + "supplied: the chain evidence has no independent finality " + + "anchor", + }, + { + when: r.expected.finalizedEthereumBlockHash == "", + blocker: "the expected finalized Ethereum block hash is not " + + "supplied: the chain evidence has no independent canonical " + + "anchor", + }, + { + when: r.expected.finalizedEthereumBlockHash != "" && + func() bool { + _, err := decodeCanonicalEthereumBytes( + r.expected.finalizedEthereumBlockHash, + 32, + ) + return err != nil + }(), + blocker: fmt.Sprintf( + "the expected finalized Ethereum block hash [%s] is not "+ + "canonical", + r.expected.finalizedEthereumBlockHash, + ), + }, + { + when: r.expected.chainEvidencePublicKey == "", + blocker: "the expected chain-evidence public key is not supplied: " + + "caller-authored Ethereum summaries cannot authorize rollback", + }, + { + when: r.expected.chainEvidencePublicKey != "" && + chainEvidencePublicKeySHA256( + r.expected.chainEvidencePublicKey, + ) == "", + blocker: "the expected chain-evidence public key is not a " + + "lowercase hexadecimal Ed25519 public key", + }, + { + when: r.expected.bitcoinNetwork == "", + blocker: "the expected Bitcoin network is not supplied: the " + + "Bitcoin reconciliation evidence cannot be bound to the " + + "rollback's operational target", + }, + { + when: r.expected.priorVersion == "", + blocker: "the expected prior version is not supplied: the " + + "prior-reader compatibility evidence cannot be bound to the " + + "exact restored artifact", + }, + { + when: r.expected.priorRevision == "", + blocker: "the expected prior revision is not supplied: the " + + "prior-reader compatibility evidence cannot be bound to the " + + "exact restored artifact", + }, + { + when: r.expected.priorImageDigest == "", + blocker: "the expected prior image digest is not supplied: the " + + "prior-reader compatibility evidence cannot be bound to the " + + "exact restored image", + }, + { + when: r.expected.priorImageDigest != "" && + !isImmutableImageDigest(r.expected.priorImageDigest), + blocker: fmt.Sprintf( + "the expected prior image digest [%s] is not an immutable "+ + "sha256 image digest: a mutable reference cannot pin the "+ + "restored artifact", + r.expected.priorImageDigest, + ), + }, + { + when: r.expected.releaseVersion == "", + blocker: "the expected release version is not supplied: the " + + "evidence cannot be bound to the exact rolled-back artifact", + }, + { + when: r.expected.releaseRevision == "", + blocker: "the expected release revision is not supplied: the " + + "evidence cannot be bound to the exact rolled-back artifact", + }, + { + when: r.expected.releaseImageDigest == "", + blocker: "the expected release image digest is not supplied: the " + + "evidence cannot be bound to the exact rolled-back image", + }, + { + when: r.expected.releaseImageDigest != "" && + !isImmutableImageDigest(r.expected.releaseImageDigest), + blocker: fmt.Sprintf( + "the expected release image digest [%s] is not an immutable "+ + "sha256 image digest: a mutable reference cannot pin the "+ + "rolled-back artifact", + r.expected.releaseImageDigest, + ), + }, + { + when: r.expected.releaseEpoch == "", + blocker: "the expected release epoch is not supplied: the " + + "audited state cannot be bound to the release that wrote it", + }, + { + when: r.expected.releaseEpoch != "" && + r.expected.releaseEpoch != participation.CompiledEpoch.String(), + blocker: fmt.Sprintf( + "the expected release epoch [%s] does not match this audit "+ + "build's compiled epoch [%s]: the audit must be built from "+ + "the audited release", + r.expected.releaseEpoch, + participation.CompiledEpoch, + ), + }, + { + when: r.expected.cutoverBlock == 0, + blocker: "the expected cutover block is not supplied: the " + + "audited state cannot be bound to the armed cutover schedule", + }, + { + when: r.expected.maxEvidenceAge <= 0, + blocker: "no evidence freshness bound is supplied: arbitrarily " + + "old evidence cannot support a rollback decision", + }, + } + + for _, input := range blocked { + if input.when { + r.manifest.RollbackBlockers = append( + r.manifest.RollbackBlockers, + input.blocker, + ) + } + } +} + +// recordExternalEvidence records every externally produced rollback input, +// validates each supplied record against its mandatory schema and this exact +// snapshot, and turns each missing or invalid one into a rollback blocker. A +// supplied reference that cannot be read is an input error: fail fast instead +// of recording evidence that does not exist. +func recordExternalEvidence(r *auditRun, evidence evidenceInputs) error { + inputs := []struct { + name string + path string + missing string + validate func([]byte) []string + }{ + { + name: "chain_reconciliation", + path: evidence.chainReconciliation, + missing: "chain reconciliation evidence not supplied: on-chain " + + "wallet/group registration and DKG settlement state are " + + "unverified", + validate: r.validateChainReconciliationEvidence, + }, + { + name: "bitcoin_reconciliation", + path: evidence.bitcoinReconciliation, + missing: "bitcoin reconciliation evidence not supplied: pending " + + "transaction state is unverified", + validate: r.validateBitcoinReconciliationEvidence, + }, + { + name: "quiescence_report", + path: evidence.quiescenceReport, + missing: "quiescence report not supplied: the at-quiescence gate " + + "inventory and terminal outcomes are unverified", + validate: r.validateQuiescenceReportEvidence, + }, + { + name: "prior_reader_compatibility", + path: evidence.priorReaderCompatibility, + missing: "prior-reader compatibility evidence not supplied: the " + + "prior release's ability to read every persisted schema is " + + "unverified", + validate: r.validatePriorReaderCompatibilityEvidence, + }, + } + + for _, input := range inputs { + record := evidenceRecord{Name: input.name} + if input.path == "" { + r.manifest.ExternalEvidence = append( + r.manifest.ExternalEvidence, + record, + ) + r.manifest.RollbackBlockers = append( + r.manifest.RollbackBlockers, + input.missing, + ) + continue + } + + content, err := os.ReadFile(input.path) + if err != nil { + return fmt.Errorf( + "cannot read the supplied [%s] evidence: [%w]", + input.name, + err, + ) + } + checksum := sha256.Sum256(content) + + record.Supplied = true + record.Path = input.path + record.SHA256 = hex.EncodeToString(checksum[:]) + + violations := input.validate(content) + record.Valid = len(violations) == 0 + for _, violation := range violations { + r.manifest.RollbackBlockers = append( + r.manifest.RollbackBlockers, + fmt.Sprintf( + "[%s] evidence fails validation: %s", + input.name, + violation, + ), + ) + } + + r.manifest.ExternalEvidence = append( + r.manifest.ExternalEvidence, + record, + ) + } + + return nil +} + +// validateEnvelope checks the common header of one evidence record against +// the expected type and this audit's snapshot identity. +func (r *auditRun) validateEnvelope( + envelope evidenceEnvelope, + expectedType string, +) []string { + var violations []string + + if envelope.SchemaVersion != evidenceSchemaVersion { + violations = append(violations, fmt.Sprintf( + "schema version [%d], expected [%d]", + envelope.SchemaVersion, + evidenceSchemaVersion, + )) + } + if envelope.EvidenceType != expectedType { + violations = append(violations, fmt.Sprintf( + "evidence type [%s], expected [%s]", + envelope.EvidenceType, + expectedType, + )) + } + if envelope.GeneratedAt.IsZero() { + violations = append(violations, "the generation time is missing") + } else if r.expected.maxEvidenceAge > 0 { + // Freshness is measured against this audit's own generation time so + // the manifest and its verdict stay reproducible from the recorded + // inputs. Evidence from the future signals a clock problem in the + // generator and cannot be trusted either. + age := r.manifest.GeneratedAt.Sub(envelope.GeneratedAt) + if age > r.expected.maxEvidenceAge { + violations = append(violations, fmt.Sprintf( + "generated [%s] before this audit, exceeding the [%s] "+ + "evidence freshness bound", + age.Round(time.Second), + r.expected.maxEvidenceAge, + )) + } + if age < -evidenceFutureSkewAllowance { + violations = append(violations, fmt.Sprintf( + "generated [%s] after this audit, beyond the [%s] clock "+ + "skew allowance", + (-age).Round(time.Second), + evidenceFutureSkewAllowance, + )) + } + } + if envelope.SnapshotAggregateSHA256 != + r.manifest.Snapshot.AggregateSHA256 { + violations = append(violations, fmt.Sprintf( + "bound to snapshot [%s], not to this audited snapshot [%s]", + envelope.SnapshotAggregateSHA256, + r.manifest.Snapshot.AggregateSHA256, + )) + } + + return violations +} + +// exactIdentityViolations checks one identity field of an evidence record: +// it must be present, and it must be exactly the expected value when an +// expectation is supplied. +func exactIdentityViolations( + description string, + value string, + expected string, +) []string { + if value == "" { + return []string{fmt.Sprintf("the %s is missing", description)} + } + if expected != "" && value != expected { + return []string{fmt.Sprintf( + "%s [%s], expected [%s]", + description, + value, + expected, + )} + } + return nil +} + +// digestViolations checks one image-digest field of an evidence record: it +// must be present, immutable — a sha256 digest, never a tag — and exactly +// the expected digest when an expectation is supplied. +func digestViolations( + description string, + value string, + expected string, +) []string { + if value == "" { + return []string{fmt.Sprintf("the %s is missing", description)} + } + var violations []string + if !isImmutableImageDigest(value) { + violations = append(violations, fmt.Sprintf( + "the %s [%s] is not an immutable sha256 image digest", + description, + value, + )) + } + if expected != "" && value != expected { + violations = append(violations, fmt.Sprintf( + "%s [%s], expected [%s]", + description, + value, + expected, + )) + } + return violations +} + +func chainReconciliationSignaturePayload( + record *chainReconciliationEvidence, +) ([]byte, error) { + unsigned := *record + unsigned.CollectorAttestation.Signature = "" + + encoded, err := json.Marshal(&unsigned) + if err != nil { + return nil, fmt.Errorf( + "cannot encode the canonical chain record: [%v]", + err, + ) + } + + payload := make( + []byte, + 0, + len(chainReconciliationSignatureDomain)+len(encoded), + ) + payload = append(payload, chainReconciliationSignatureDomain...) + payload = append(payload, encoded...) + return payload, nil +} + +func (r *auditRun) validateAuthenticatedEthereumEvidence( + record *chainReconciliationEvidence, +) []string { + var violations []string + + if _, err := decodeCanonicalEthereumBytes( + record.WalletRegistryAddress, + 20, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "the WalletRegistry address [%s] is invalid: [%v]", + record.WalletRegistryAddress, + err, + )) + } else if r.expected.walletRegistryAddress != "" && + record.WalletRegistryAddress != r.expected.walletRegistryAddress { + violations = append(violations, fmt.Sprintf( + "the WalletRegistry address [%s], expected [%s]", + record.WalletRegistryAddress, + r.expected.walletRegistryAddress, + )) + } + + if _, err := decodeCanonicalEthereumBytes( + record.RandomBeaconAddress, + 20, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "the RandomBeacon address [%s] is invalid: [%v]", + record.RandomBeaconAddress, + err, + )) + } else if r.expected.randomBeaconAddress != "" && + record.RandomBeaconAddress != r.expected.randomBeaconAddress { + violations = append(violations, fmt.Sprintf( + "the RandomBeacon address [%s], expected [%s]", + record.RandomBeaconAddress, + r.expected.randomBeaconAddress, + )) + } + + attestation := record.CollectorAttestation + if attestation.FinalizedBlockNumber == 0 { + violations = append( + violations, + "the collector attestation has no finalized block number", + ) + } else if r.expected.finalizedEthereumBlockNumber != 0 && + attestation.FinalizedBlockNumber != + r.expected.finalizedEthereumBlockNumber { + violations = append(violations, fmt.Sprintf( + "the collector finalized block number [%d], expected [%d]", + attestation.FinalizedBlockNumber, + r.expected.finalizedEthereumBlockNumber, + )) + } + if _, err := decodeCanonicalEthereumBytes( + attestation.FinalizedBlockHash, + 32, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "the collector finalized block hash [%s] is invalid: [%v]", + attestation.FinalizedBlockHash, + err, + )) + } else if r.expected.finalizedEthereumBlockHash != "" && + attestation.FinalizedBlockHash != + r.expected.finalizedEthereumBlockHash { + violations = append(violations, fmt.Sprintf( + "the collector finalized block hash [%s], expected [%s]", + attestation.FinalizedBlockHash, + r.expected.finalizedEthereumBlockHash, + )) + } + + canonicalBlocks := make(map[uint64]string) + var previousBlock uint64 + for i, block := range attestation.CanonicalBlocks { + if block.BlockNumber == 0 { + violations = append(violations, fmt.Sprintf( + "canonical block entry [%d] has zero block number", + i, + )) + } + if _, err := decodeCanonicalEthereumBytes( + block.BlockHash, + 32, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "canonical block [%d] has invalid hash [%s]: [%v]", + block.BlockNumber, + block.BlockHash, + err, + )) + } + if _, duplicate := canonicalBlocks[block.BlockNumber]; duplicate { + violations = append(violations, fmt.Sprintf( + "canonical block [%d] is attested more than once", + block.BlockNumber, + )) + } + if i > 0 && block.BlockNumber <= previousBlock { + violations = append(violations, fmt.Sprintf( + "canonical block entries are not strictly increasing at [%d]", + block.BlockNumber, + )) + } + canonicalBlocks[block.BlockNumber] = block.BlockHash + previousBlock = block.BlockNumber + } + if hash, ok := canonicalBlocks[attestation.FinalizedBlockNumber]; !ok { + violations = append(violations, fmt.Sprintf( + "the finalized block [%d] is absent from the authenticated "+ + "canonical block set", + attestation.FinalizedBlockNumber, + )) + } else if hash != attestation.FinalizedBlockHash { + violations = append(violations, fmt.Sprintf( + "canonical block [%d] has hash [%s], not the attested finalized "+ + "hash [%s]", + attestation.FinalizedBlockNumber, + hash, + attestation.FinalizedBlockHash, + )) + } + + publicKey, publicKeyErr := hex.DecodeString( + r.expected.chainEvidencePublicKey, + ) + signature, signatureErr := hex.DecodeString(attestation.Signature) + switch { + case publicKeyErr != nil || + len(publicKey) != ed25519.PublicKeySize || + hex.EncodeToString(publicKey) != r.expected.chainEvidencePublicKey: + violations = append( + violations, + "the independently supplied chain-evidence public key is invalid", + ) + case signatureErr != nil || + len(signature) != ed25519.SignatureSize || + hex.EncodeToString(signature) != attestation.Signature: + violations = append( + violations, + "the collector attestation signature is not canonical Ed25519", + ) + default: + payload, err := chainReconciliationSignaturePayload(record) + if err != nil { + violations = append(violations, err.Error()) + } else if !ed25519.Verify(publicKey, payload, signature) { + violations = append( + violations, + "the chain reconciliation record is not signed by the "+ + "independently trusted finalized-chain collector", + ) + } + } + + receipts := make(map[string]struct{}) + for i, receipt := range record.Receipts { + if _, err := decodeCanonicalEthereumBytes( + receipt.TransactionHash, + 32, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "receipt [%d] has invalid transaction hash [%s]: [%v]", + i, + receipt.TransactionHash, + err, + )) + } + if _, duplicate := receipts[receipt.TransactionHash]; duplicate { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] is supplied more than once", + receipt.TransactionHash, + )) + } + receipts[receipt.TransactionHash] = struct{}{} + + if receipt.BlockNumber == 0 { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] has zero block number", + receipt.TransactionHash, + )) + } + if receipt.BlockNumber > attestation.FinalizedBlockNumber { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] at block [%d] is newer than the "+ + "attested finalized block [%d]", + receipt.TransactionHash, + receipt.BlockNumber, + attestation.FinalizedBlockNumber, + )) + } + if _, err := decodeCanonicalEthereumBytes( + receipt.BlockHash, + 32, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] has invalid block hash [%s]: [%v]", + receipt.TransactionHash, + receipt.BlockHash, + err, + )) + } + if canonicalHash, ok := canonicalBlocks[receipt.BlockNumber]; !ok { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] block [%d] is absent from the "+ + "authenticated canonical block set", + receipt.TransactionHash, + receipt.BlockNumber, + )) + } else if canonicalHash != receipt.BlockHash { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] names non-canonical block hash "+ + "[%s] at height [%d]; the collector attests [%s]", + receipt.TransactionHash, + receipt.BlockHash, + receipt.BlockNumber, + canonicalHash, + )) + } + if receipt.Status != 1 { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] has failed status [%d]", + receipt.TransactionHash, + receipt.Status, + )) + } + + logIndexes := make(map[uint64]struct{}) + for j, rawLog := range receipt.Logs { + if _, duplicate := logIndexes[rawLog.LogIndex]; duplicate { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] repeats log index [%d]", + receipt.TransactionHash, + rawLog.LogIndex, + )) + } + logIndexes[rawLog.LogIndex] = struct{}{} + if _, err := decodeCanonicalEthereumBytes( + rawLog.Address, + 20, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] log [%d] has invalid address "+ + "[%s]: [%v]", + receipt.TransactionHash, + j, + rawLog.Address, + err, + )) + } + for k, topic := range rawLog.Topics { + if _, err := decodeCanonicalEthereumBytes(topic, 32); err != nil { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] log [%d] topic [%d] is "+ + "invalid [%s]: [%v]", + receipt.TransactionHash, + j, + k, + topic, + err, + )) + } + } + if _, err := decodeCanonicalEthereumDynamicBytes( + rawLog.Data, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "transaction receipt [%s] log [%d] has invalid data: [%v]", + receipt.TransactionHash, + j, + err, + )) + } + } + } + + return violations +} + +// validateChainReconciliationEvidence checks the Ethereum reconciliation +// record: schema, snapshot binding, the expected chain identity, one-to-one +// coverage of every persisted tBTC wallet and beacon group — active and +// quarantined — with no duplicate entries and wallet IDs matching the +// decoded snapshot identities, a settled, registered on-chain state for each +// active wallet, and an explicit no-result settlement for each +// quarantined-only one. A quarantined output whose wallet or group is +// registered on chain is a blocker of its own: the prior binary would run +// that wallet without the preserved share. +func (r *auditRun) validateChainReconciliationEvidence( + content []byte, +) []string { + record := &chainReconciliationEvidence{} + if err := strictUnmarshal(content, record); err != nil { + return []string{fmt.Sprintf( + "cannot be decoded as a chain reconciliation record: [%v]", + err, + )} + } + + violations := r.validateEnvelope( + record.evidenceEnvelope, + "chain_reconciliation", + ) + + if record.EthereumChainID == "" { + violations = append(violations, "the Ethereum chain ID is missing") + } else if r.expected.ethereumChainID != "" && + record.EthereumChainID != r.expected.ethereumChainID { + violations = append(violations, fmt.Sprintf( + "reconciled against Ethereum chain [%s], expected [%s]", + record.EthereumChainID, + r.expected.ethereumChainID, + )) + } + violations = append( + violations, + r.validateAuthenticatedEthereumEvidence(record)..., + ) + + wallets := make(map[string]int) + walletIDs := make(map[string]string) + dkgResultHashes := make(map[string]string) + for i, wallet := range record.Wallets { + if wallet.WalletStorageKey == "" || wallet.WalletID == "" { + violations = append(violations, fmt.Sprintf( + "wallet entry [%d] is missing its identity", + i, + )) + continue + } + if _, duplicate := wallets[wallet.WalletStorageKey]; duplicate { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] is reconciled more than once; duplicate "+ + "entries cannot prove one-to-one coverage", + wallet.WalletStorageKey, + )) + continue + } + if previous, duplicate := walletIDs[wallet.WalletID]; duplicate { + violations = append(violations, fmt.Sprintf( + "wallet ID [%s] is claimed by both tbtc wallet [%s] and "+ + "tbtc wallet [%s]", + wallet.WalletID, + previous, + wallet.WalletStorageKey, + )) + continue + } + if _, known := validDKGSettlementStates[wallet.DKGSettlement]; !known { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] has unknown DKG settlement state [%s]", + wallet.WalletStorageKey, + wallet.DKGSettlement, + )) + } + if wallet.DKGSettlement == "none" { + if wallet.DKGResult != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] claims no DKG settlement but also "+ + "names result [%s]", + wallet.WalletStorageKey, + wallet.DKGResult.resultHash(), + )) + } + } else if wallet.DKGResult == nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] has DKG settlement [%s] without the "+ + "canonical DKG result that produced it", + wallet.WalletStorageKey, + wallet.DKGSettlement, + )) + } else { + violations = append( + violations, + validateTBTCDKGResultEvidence( + wallet.WalletStorageKey, + wallet.WalletID, + wallet.DKGResult, + record, + )..., + ) + resultHash := wallet.DKGResult.resultHash() + if previous, duplicate := dkgResultHashes[resultHash]; duplicate { + violations = append(violations, fmt.Sprintf( + "DKG result [%s] is claimed by both tbtc wallet [%s] "+ + "and tbtc wallet [%s]", + resultHash, + previous, + wallet.WalletStorageKey, + )) + } else { + dkgResultHashes[resultHash] = wallet.WalletStorageKey + } + } + wallets[wallet.WalletStorageKey] = i + walletIDs[wallet.WalletID] = wallet.WalletStorageKey + } + beaconGroups := make(map[string]int) + for i, beaconGroup := range record.BeaconGroups { + if beaconGroup.GroupPublicKey == "" { + violations = append(violations, fmt.Sprintf( + "beacon group entry [%d] is missing its group public key", + i, + )) + continue + } + if _, duplicate := beaconGroups[beaconGroup.GroupPublicKey]; duplicate { + violations = append(violations, fmt.Sprintf( + "beacon group [%s] is reconciled more than once; duplicate "+ + "entries cannot prove one-to-one coverage", + beaconGroup.GroupPublicKey, + )) + continue + } + beaconGroups[beaconGroup.GroupPublicKey] = i + } + + activeWalletKeys := make(map[string]struct{}) + for _, wallet := range r.manifest.TBTCActiveWallets { + activeWalletKeys[wallet.WalletStorageKey] = struct{}{} + + i, covered := wallets[wallet.WalletStorageKey] + if !covered { + violations = append(violations, fmt.Sprintf( + "persisted tbtc wallet [%s] is not reconciled", + wallet.WalletStorageKey, + )) + continue + } + if !record.Wallets[i].Registered { + violations = append(violations, fmt.Sprintf( + "persisted tbtc wallet [%s] is not registered on chain", + wallet.WalletStorageKey, + )) + } + if record.Wallets[i].DKGSettlement != "approved" { + violations = append(violations, fmt.Sprintf( + "persisted tbtc wallet [%s] has DKG settlement [%s], "+ + "expected [approved]", + wallet.WalletStorageKey, + record.Wallets[i].DKGSettlement, + )) + } + if wallet.WalletID != "" && + record.Wallets[i].WalletID != wallet.WalletID { + violations = append(violations, fmt.Sprintf( + "persisted tbtc wallet [%s] is reconciled under wallet ID "+ + "[%s], but its decoded records carry wallet ID [%s]", + wallet.WalletStorageKey, + record.Wallets[i].WalletID, + wallet.WalletID, + )) + } + if result := record.Wallets[i].DKGResult; result != nil { + originalGroupSize := result.originalGroupSize() + misbehavedMemberIndexes := result.misbehavedMemberIndexes() + finalGroupSize := + int(originalGroupSize) - len(misbehavedMemberIndexes) + if finalGroupSize != wallet.SigningGroupSize { + violations = append(violations, fmt.Sprintf( + "persisted tbtc wallet [%s] has signing group size "+ + "[%d], but canonical DKG result [%s] derives [%d] "+ + "members from original group size [%d] and [%d] "+ + "misbehaved seats", + wallet.WalletStorageKey, + wallet.SigningGroupSize, + result.resultHash(), + finalGroupSize, + originalGroupSize, + len(misbehavedMemberIndexes), + )) + } + } + } + quarantinedWalletKeys := make(map[string]struct{}) + for _, quarantined := range r.manifest.TBTCQuarantinedOutputs { + quarantinedWalletKeys[quarantined.WalletStorageKey] = struct{}{} + + i, covered := wallets[quarantined.WalletStorageKey] + if !covered { + violations = append(violations, fmt.Sprintf( + "quarantined tbtc wallet [%s] is not reconciled", + quarantined.WalletStorageKey, + )) + continue + } + if _, active := activeWalletKeys[quarantined.WalletStorageKey]; active { + // Both namespaces holding the same wallet is already an + // interpretation finding; the registered state is judged there. + continue + } + if record.Wallets[i].Registered { + violations = append(violations, fmt.Sprintf( + "quarantined tbtc wallet [%s] is registered on chain but "+ + "its share is preserved only in quarantine; the prior "+ + "binary would run it without the share", + quarantined.WalletStorageKey, + )) + } + // A quarantined-only share tolerates no DKG result on chain at all: + // a pending or challenged result may still settle into a wallet the + // prior binary would run without the share, and an approved one + // contradicts the quarantine itself. + if record.Wallets[i].DKGSettlement != "none" { + violations = append(violations, fmt.Sprintf( + "quarantined tbtc wallet [%s] has DKG settlement [%s], "+ + "expected [none]", + quarantined.WalletStorageKey, + record.Wallets[i].DKGSettlement, + )) + } + expectedWalletID := quarantined.SignerWalletID + if expectedWalletID == "" { + expectedWalletID = quarantined.WalletID + } + if expectedWalletID != "" && + record.Wallets[i].WalletID != expectedWalletID { + violations = append(violations, fmt.Sprintf( + "quarantined tbtc wallet [%s] is reconciled under wallet ID "+ + "[%s], but its preserved output carries wallet ID [%s]", + quarantined.WalletStorageKey, + record.Wallets[i].WalletID, + expectedWalletID, + )) + } + } + + activeBeaconGroups := make(map[string]struct{}) + for _, membership := range r.manifest.BeaconActiveMemberships { + activeBeaconGroups[membership.GroupPublicKey] = struct{}{} + + i, covered := beaconGroups[membership.GroupPublicKey] + if !covered { + violations = append(violations, fmt.Sprintf( + "persisted beacon group [%s] is not reconciled", + membership.GroupPublicKey, + )) + continue + } + if !record.BeaconGroups[i].Registered { + violations = append(violations, fmt.Sprintf( + "persisted beacon group [%s] is not registered on chain", + membership.GroupPublicKey, + )) + } + } + quarantinedBeaconGroups := make(map[string]struct{}) + for _, quarantined := range r.manifest.BeaconQuarantinedOutputs { + quarantinedBeaconGroups[quarantined.GroupPublicKey] = struct{}{} + + i, covered := beaconGroups[quarantined.GroupPublicKey] + if !covered { + violations = append(violations, fmt.Sprintf( + "quarantined beacon group [%s] is not reconciled", + quarantined.GroupPublicKey, + )) + continue + } + if _, active := activeBeaconGroups[quarantined.GroupPublicKey]; active { + continue + } + if record.BeaconGroups[i].Registered { + violations = append(violations, fmt.Sprintf( + "quarantined beacon group [%s] is registered on chain but "+ + "its share is preserved only in quarantine; the prior "+ + "binary would run it without the share", + quarantined.GroupPublicKey, + )) + } + } + + // One-to-one the other way: evidence reconciling state the snapshot does + // not hold audits a different node — or fabricates coverage — and cannot + // bind to this rollback. + for _, wallet := range record.Wallets { + if wallet.WalletStorageKey == "" { + continue + } + _, active := activeWalletKeys[wallet.WalletStorageKey] + _, quarantined := quarantinedWalletKeys[wallet.WalletStorageKey] + if !active && !quarantined { + violations = append(violations, fmt.Sprintf( + "reconciles tbtc wallet [%s] that the snapshot does not hold", + wallet.WalletStorageKey, + )) + } + } + for _, beaconGroup := range record.BeaconGroups { + if beaconGroup.GroupPublicKey == "" { + continue + } + _, active := activeBeaconGroups[beaconGroup.GroupPublicKey] + _, quarantined := quarantinedBeaconGroups[beaconGroup.GroupPublicKey] + if !active && !quarantined { + violations = append(violations, fmt.Sprintf( + "reconciles beacon group [%s] that the snapshot does not hold", + beaconGroup.GroupPublicKey, + )) + } + } + violations = append( + violations, + validateTBTCDKGTerminalLineage(r.manifest, record, wallets)..., + ) + violations = append( + violations, + r.reconcileInactivityClaimSettlements(record)..., + ) + violations = append( + violations, + r.reconcileRelayTimeoutSettlements(record)..., + ) + violations = append( + violations, + r.reconcileRelayEntryResults(record)..., + ) + + return violations +} + +// reconcileInactivityClaimSettlements joins every inactivity claim the node +// recorded submitting to the authenticated WalletRegistry logs, and to the +// wallet the punishing permit was actually about. +// +// A heartbeat that punishes members files that penalty on Ethereum, and the +// node authors nothing about it beyond a digest of its own signature. Three +// cases have to be told apart and none may pass on the node's word. +// +// A submission the node could not resolve is ambiguous: the claim may be on +// chain under a nonce this node's rollback would then contradict, so the +// barrier blocks. +// +// A settlement the node reports must be corroborated by a canonical +// InactivityClaimed log for the exact wallet and nonce named, emitted by the +// expected WalletRegistry inside an authenticated receipt. The enclosing +// receipt validation has already bound every receipt to an attested canonical +// block with successful status, so a matched log is chain state and not a +// report the evidence generator wrote. +// +// Corroboration alone would still let one wallet's outcome borrow another +// wallet's penalty: every real claim log on the chain is equally real, and +// asking only "does this claim exist" accepts any of them. The reported wallet +// is therefore also bound to the wallet the permit names, through the +// key-derived public-key-hash-to-wallet-ID mapping of the snapshot's own +// signer material. An outcome the snapshot cannot bind names a penalty against +// a wallet this node holds no key material for, which is not a claim its +// heartbeat could have filed. +func (r *auditRun) reconcileInactivityClaimSettlements( + record *chainReconciliationEvidence, +) []string { + if r.manifest.ParticipationTerminalOutcomes == nil { + return nil + } + + permitWallets := r.permitWalletIDs() + + var violations []string + var settled map[string]uint64 + for i, outcome := range r.manifest.ParticipationTerminalOutcomes.Outcomes { + settlement := outcome.Evidence.ChainSettlement + if settlement == nil || + settlement.Kind != + participation.ChainSettlementInactivityClaim { + continue + } + if settlement.Reference == "" { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] submitted an inactivity claim "+ + "whose settlement it could not resolve; the penalty may "+ + "or may not be on chain and cannot be reconciled", + i, + )) + continue + } + walletID, nonce, err := participation. + ParseInactivityClaimSettlementReference(settlement.Reference) + if err != nil { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] names inactivity claim settlement "+ + "[%s], which is not a canonical claim identity: [%v]", + i, + settlement.Reference, + err, + )) + continue + } + + permitWalletID, bound := permitWallets[outcome.Permit.PermitID] + if !bound { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports an inactivity claim for "+ + "permit [%s], but the snapshot holds no signer material "+ + "identifying that wallet, so the reported penalty cannot "+ + "be bound to the wallet it punished", + i, + outcome.Permit.PermitID, + )) + continue + } + if claimedWalletID := hex.EncodeToString( + walletID, + ); claimedWalletID != permitWalletID { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] runs under permit [%s], which is "+ + "wallet [%s], but reports settling inactivity claim [%s] "+ + "against wallet [%s]", + i, + outcome.Permit.PermitID, + permitWalletID, + settlement.Reference, + claimedWalletID, + )) + continue + } + + if settled == nil { + settled, err = authenticatedInactivityClaims(record) + if err != nil { + violations = append(violations, fmt.Sprintf( + "cannot decode the authenticated inactivity claim logs: "+ + "[%v]", + err, + )) + break + } + } + + claimed := hex.EncodeToString(walletID) + ":" + nonce.String() + if _, corroborated := settled[claimed]; !corroborated { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports inactivity claim [%s] "+ + "settled, but no authenticated WalletRegistry "+ + "InactivityClaimed log names that wallet and nonce", + i, + settlement.Reference, + )) + } + } + + return violations +} + +// validateRelayEntryTerminalResult verifies the relay entry a completed beacon +// relay signing outcome names. +// +// This is the one node-authored protocol result the offline audit can check +// outright rather than reconcile. A relay entry is a threshold BLS signature by +// the group over the previous entry, so verifying the pairing proves the entry +// came from the group's threshold key, which no single node — and no evidence +// generator — can produce. The group is required to be one the snapshot itself +// decoded key material for, because an entry verified under a group this node +// never belonged to says nothing about what this node did. +// +// A signature alone is still only a proof of authorship, and every entry the +// beacon ever produced keeps that property forever. Two further checks turn it +// into a proof about this permit's work. The reference must answer the relay +// request its own permit was issued for, and one entry must answer one request +// across the whole journal: a relay entry is deterministic for a given previous +// entry, so a genuine entry standing in as the result of a second request is a +// replay however well it verifies. Memberships of the same request share their +// entry legitimately and are not counted against it. +func validateRelayEntryTerminalResult( + outcomeIndex int, + workID string, + reference string, + beaconGroupKeys map[string]struct{}, + claimedRelayEntries map[string]relayEntryClaim, +) []string { + referenceStartBlock, groupPublicKeyBytes, previousEntryBytes, entryBytes, + err := participation.ParseBeaconRelayEntryReference(reference) + if err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names relay "+ + "entry [%s], which is not a canonical entry identity: [%v]", + outcomeIndex, + reference, + err, + )} + } + + permitStartBlock, err := participation.ParseBeaconRelayWorkID(workID) + if err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] belongs to "+ + "permit work [%s], which names no relay request: [%v]", + outcomeIndex, + workID, + err, + )} + } + if referenceStartBlock != permitStartBlock { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names an "+ + "entry answering request start block [%d], but its permit "+ + "was issued for request start block [%d]", + outcomeIndex, + referenceStartBlock, + permitStartBlock, + )} + } + + entryIdentity := hex.EncodeToString(groupPublicKeyBytes) + ":" + + hex.EncodeToString(previousEntryBytes) + ":" + + hex.EncodeToString(entryBytes) + if claimed, replayed := claimedRelayEntries[entryIdentity]; replayed { + if claimed.requestStartBlock != referenceStartBlock { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] answers "+ + "request start block [%d] with the entry outcome [%d] "+ + "already answered request start block [%d] with", + outcomeIndex, + referenceStartBlock, + claimed.outcomeIndex, + claimed.requestStartBlock, + )} + } + } else { + claimedRelayEntries[entryIdentity] = relayEntryClaim{ + outcomeIndex: outcomeIndex, + requestStartBlock: referenceStartBlock, + } + } + + groupPublicKey := hex.EncodeToString(groupPublicKeyBytes) + if _, held := beaconGroupKeys[groupPublicKey]; !held { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names an "+ + "entry signed by group [%s], which the snapshot holds no "+ + "membership of", + outcomeIndex, + groupPublicKey, + )} + } + + publicKey, err := altbn128.DecompressToG2(groupPublicKeyBytes) + if err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names group "+ + "[%s], which is not a compressed bn256 public key: [%v]", + outcomeIndex, + groupPublicKey, + err, + )} + } + + // The point at infinity pairs trivially, so naming it for both the + // previous entry and the entry would satisfy the verification below + // without any knowledge of the group's threshold key. It is also not a + // relay entry any beacon round ever produced. + if isInfinityG1(previousEntryBytes) || isInfinityG1(entryBytes) { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names the "+ + "point at infinity, which no relay round produces and which "+ + "verifies under any group", + outcomeIndex, + )} + } + + previousEntry := new(bn256.G1) + if _, err := previousEntry.Unmarshal(previousEntryBytes); err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names a "+ + "previous entry that is not a bn256 point: [%v]", + outcomeIndex, + err, + )} + } + + entry := new(bn256.G1) + if _, err := entry.Unmarshal(entryBytes); err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names an "+ + "entry that is not a bn256 point: [%v]", + outcomeIndex, + err, + )} + } + + // The beacon signs the previous entry as a curve point, so the pairing + // check is over the point itself rather than a hash of its bytes. + if !bls.VerifyG1(publicKey, previousEntry, entry) { + return []string{fmt.Sprintf( + "node-authored completed beacon relay outcome [%d] names an "+ + "entry that group [%s] did not sign over the previous entry "+ + "it names", + outcomeIndex, + groupPublicKey, + )} + } + + return nil +} + +// validateRelayTimeoutSettlement verifies the beacon timeout settlement a +// completed timeout report outcome names. +// +// A filed report is not a penalty and the node knows only that it handed a +// transaction to a provider, so the record has to name chain state rather than +// anything the node derived. The reference carries the two fields of a +// RelayEntryTimedOut log — the beacon's request identifier and the group it +// terminated — which is what lets an operator join this record to exactly one +// authenticated log. This audit runs offline and cannot fetch that log, so what +// it establishes are the properties that must hold before the join is even +// meaningful: the settlement answers the request its own permit was issued for, +// and one settlement answers one request across the whole journal. +// +// The second check is the replay guard. A beacon terminates a request once, so +// the same request identifier and group standing as the result of a second +// request is a claim on a penalty that request never earned, however real the +// underlying log is. +func validateRelayTimeoutSettlement( + outcomeIndex int, + workID string, + reference string, + claimedTimeoutSettlements map[string]relayTimeoutSettlementClaim, +) []string { + referenceStartBlock, requestID, terminatedGroupID, err := + participation.ParseBeaconRelayTimeoutSettlementReference(reference) + if err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon timeout report outcome [%d] names "+ + "settlement [%s], which is not a canonical settlement "+ + "identity: [%v]", + outcomeIndex, + reference, + err, + )} + } + + permitStartBlock, err := participation.ParseBeaconRelayWorkID(workID) + if err != nil { + return []string{fmt.Sprintf( + "node-authored completed beacon timeout report outcome [%d] "+ + "belongs to permit work [%s], which names no relay request: "+ + "[%v]", + outcomeIndex, + workID, + err, + )} + } + if referenceStartBlock != permitStartBlock { + return []string{fmt.Sprintf( + "node-authored completed beacon timeout report outcome [%d] names "+ + "a settlement terminating request start block [%d], but its "+ + "permit was issued for request start block [%d]", + outcomeIndex, + referenceStartBlock, + permitStartBlock, + )} + } + + settlementIdentity := requestID.String() + ":" + + strconv.FormatUint(terminatedGroupID, 10) + if claimed, replayed := + claimedTimeoutSettlements[settlementIdentity]; replayed { + if claimed.requestStartBlock != referenceStartBlock { + return []string{fmt.Sprintf( + "node-authored completed beacon timeout report outcome [%d] "+ + "answers request start block [%d] with the settlement "+ + "outcome [%d] already answered request start block [%d] "+ + "with", + outcomeIndex, + referenceStartBlock, + claimed.outcomeIndex, + claimed.requestStartBlock, + )} + } + } else { + claimedTimeoutSettlements[settlementIdentity] = + relayTimeoutSettlementClaim{ + outcomeIndex: outcomeIndex, + requestStartBlock: referenceStartBlock, + } + } + + return nil +} + +// isInfinityG1 reports whether marshaled bn256 G1 bytes are the point at +// infinity, which the curve encodes as all zeros. +func isInfinityG1(point []byte) bool { + for _, b := range point { + if b != 0 { + return false + } + } + return true +} + +// permitWalletIDs maps the wallet public key hash a tBTC permit names itself +// with to the ECDSA wallet identifier the registry knows that wallet by. +// +// Both halves are derived from the decoded key material rather than read from +// a label the node wrote, and one public key yields exactly one of each, so the +// mapping is the snapshot's own answer to which registry wallet a given permit +// is about. Quarantined material counts: a heartbeat can punish a wallet whose +// signer records the rollback later quarantined, and the wallet it punished is +// no less identified for that. +func (r *auditRun) permitWalletIDs() map[string]string { + walletIDs := make(map[string]string) + + for _, wallet := range r.manifest.TBTCActiveWallets { + if wallet.WalletPublicKeyHash == "" || wallet.WalletID == "" { + continue + } + walletIDs[wallet.WalletPublicKeyHash] = wallet.WalletID + } + for _, quarantined := range r.manifest.TBTCQuarantinedOutputs { + if quarantined.SignerWalletPublicKeyHash == "" || + quarantined.SignerWalletID == "" { + continue + } + walletIDs[quarantined.SignerWalletPublicKeyHash] = + quarantined.SignerWalletID + } + + return walletIDs +} + +// authenticatedInactivityClaims indexes every InactivityClaimed log the +// authenticated receipts carry, keyed by the canonical wallet-and-nonce +// identity of the claim it settled. Logs from any contract other than the +// expected WalletRegistry are ignored: an identically shaped event from an +// attacker-deployed contract names no penalty the registry ever applied. +func authenticatedInactivityClaims( + record *chainReconciliationEvidence, +) (map[string]uint64, error) { + parsed, err := ecdsaabi.WalletRegistryMetaData.GetAbi() + if err != nil { + return nil, fmt.Errorf( + "cannot load the generated WalletRegistry ABI: [%v]", + err, + ) + } + event, ok := parsed.Events["InactivityClaimed"] + if !ok { + return nil, fmt.Errorf( + "generated WalletRegistry ABI has no [InactivityClaimed] event", + ) + } + // The enclosing receipt validation already requires every log address and + // topic to be canonically encoded, and the registry address to be both + // canonical and the expected one, so exact comparison is what distinguishes + // the registry's own logs here. + eventID := event.ID.Hex() + + claims := make(map[string]uint64) + for _, receipt := range record.Receipts { + for _, rawLog := range receipt.Logs { + if rawLog.Address != record.WalletRegistryAddress { + continue + } + // walletID is the sole indexed input, so a claim log always has + // exactly the signature topic and the wallet topic. + if len(rawLog.Topics) != 2 || rawLog.Topics[0] != eventID { + continue + } + walletID, err := decodeCanonicalEthereumBytes(rawLog.Topics[1], 32) + if err != nil { + continue + } + data, err := decodeCanonicalEthereumDynamicBytes(rawLog.Data) + if err != nil { + continue + } + values, err := event.Inputs.NonIndexed().Unpack(data) + if err != nil || len(values) == 0 { + continue + } + nonce, ok := values[0].(*big.Int) + if !ok || nonce == nil || nonce.Sign() < 0 { + continue + } + + claims[hex.EncodeToString(walletID)+":"+nonce.String()] = + receipt.BlockNumber + } + } + + return claims, nil +} + +// relayEntryLifecycleLogs is the beacon's own account of the relay requests the +// authenticated receipts reach: when each request was made, which ones the +// selected group answered, and which ones an accepted timeout report +// terminated. +// +// They are indexed together because no one of them settles a permit's result on +// its own. A timeout log proves a request was terminated but not which request +// a permit was issued for; a request log supplies that binding and names the +// group the beacon selected; a group registration turns that group's registry +// index into the key a record can be held to; and a submission log contradicts +// a penalty outright and says which entry the beacon accepted for the request a +// signing permit answered. +type relayEntryLifecycleLogs struct { + // requestBlocks maps a beacon request identifier to the block its + // RelayEntryRequested log was mined in. + requestBlocks map[string]uint64 + // ambiguousRequests names every request identifier the receipts place at + // more than one block. Nothing in the logs says which block a caller meant, + // so such a request binds no permit rather than binding it to either. + ambiguousRequests map[string]struct{} + // requestIdentities maps the request identity a relay signing permit names + // itself by — the block the request was made in and the previous entry it + // signs over — to the beacon's request identifier. A relay entry reference + // carries no request identifier, so this is the only join from a recovered + // entry to the canonical request it answers. + requestIdentities map[string]string + // ambiguousIdentities names every block-and-previous-entry identity the + // receipts place more than one request under. + ambiguousIdentities map[string]struct{} + // identityRequests is the reverse of requestIdentities, kept so a request + // identifier the receipts hand to two different identities can be named + // ambiguous. One request identifier answering two requests would let a + // permit be closed by the evidence of the other one. + identityRequests map[string]string + // requestGroups maps a beacon request identifier to the registry index of + // the group the beacon selected to answer it, in decimal. The selection is + // the beacon's, not the node's: it is what says which group's permit an + // accepted entry is allowed to close. + requestGroups map[string]string + // ambiguousRequestGroups names every request identifier the receipts place + // under more than one selected group. + ambiguousRequestGroups map[string]struct{} + // registeredGroupKeys maps a group's registry index to the hash of the + // public key it was registered under. GroupRegistered indexes that key, so + // the log carries its hash rather than its bytes; the hash is still the + // exact binding between the index a request names and the key a permit's + // record carries. + registeredGroupKeys map[string]string + // ambiguousGroupRegistrations names every registry index the receipts + // register more than one differing public key under. + ambiguousGroupRegistrations map[string]struct{} + // submittedEntries maps a request identifier to the entry a + // RelayEntrySubmitted log records the beacon accepting for it, normalized + // to the canonical point encoding. A request whose receipts carry more than + // one differing accepted entry is absent and named ambiguous instead. + submittedEntries map[string][]byte + // ambiguousSubmissions names every request identifier the receipts record + // more than one differing accepted entry for. + ambiguousSubmissions map[string]struct{} + // submittedRequests names every request identifier a RelayEntrySubmitted + // log answers, including one whose entry could not be decoded. + submittedRequests map[string]struct{} + // timeouts maps the canonical settlement identity — request identifier and + // terminated group — to the block its RelayEntryTimedOut log was mined in. + timeouts map[string]uint64 +} + +// relayEntryIdentity renders the request identity a relay signing permit names +// itself by: the block the request was made in and the previous entry it signs +// over. +func relayEntryIdentity(blockNumber uint64, previousEntry []byte) string { + return strconv.FormatUint(blockNumber, 10) + ":" + + hex.EncodeToString(previousEntry) +} + +// canonicalRelayPoint normalizes marshaled beacon curve-point bytes to the +// encoding the node's own terminal records are rendered in, so a log carrying a +// non-canonical rendering of the same point still compares equal. Bytes that +// are not a point at all name nothing and are refused. +func canonicalRelayPoint(value []byte) ([]byte, bool) { + point := new(bn256.G1) + if _, err := point.Unmarshal(value); err != nil { + return nil, false + } + return point.Marshal(), true +} + +// authenticatedRelayEntryLogs indexes every RelayEntryRequested, +// RelayEntrySubmitted and RelayEntryTimedOut log the authenticated receipts +// carry. Logs from any contract other than the expected RandomBeacon are +// ignored: an identically shaped event from an attacker-deployed contract names +// no request the beacon ever made and no penalty it ever applied. +func authenticatedRelayEntryLogs( + record *chainReconciliationEvidence, +) (*relayEntryLifecycleLogs, error) { + parsed, err := beaconabi.RandomBeaconMetaData.GetAbi() + if err != nil { + return nil, fmt.Errorf( + "cannot load the generated RandomBeacon ABI: [%v]", + err, + ) + } + + events := make(map[string]abi.Event, 4) + for _, name := range []string{ + "RelayEntryRequested", + "RelayEntrySubmitted", + "RelayEntryTimedOut", + "GroupRegistered", + } { + event, ok := parsed.Events[name] + if !ok { + return nil, fmt.Errorf( + "generated RandomBeacon ABI has no [%s] event", + name, + ) + } + events[name] = event + } + + logs := &relayEntryLifecycleLogs{ + requestBlocks: make(map[string]uint64), + ambiguousRequests: make(map[string]struct{}), + requestIdentities: make(map[string]string), + ambiguousIdentities: make(map[string]struct{}), + identityRequests: make(map[string]string), + requestGroups: make(map[string]string), + ambiguousRequestGroups: make(map[string]struct{}), + registeredGroupKeys: make(map[string]string), + ambiguousGroupRegistrations: make(map[string]struct{}), + submittedEntries: make(map[string][]byte), + ambiguousSubmissions: make(map[string]struct{}), + submittedRequests: make(map[string]struct{}), + timeouts: make(map[string]uint64), + } + + // The enclosing receipt validation already requires every log address and + // topic to be canonically encoded, and the beacon address to be both + // canonical and the expected one, so exact comparison is what distinguishes + // the beacon's own logs here. + for _, receipt := range record.Receipts { + for _, rawLog := range receipt.Logs { + if rawLog.Address != record.RandomBeaconAddress { + continue + } + if len(rawLog.Topics) == 0 { + continue + } + + // GroupRegistered is the one event here with two indexed inputs: + // the registry index a request names a group by, and the hash of + // the public key that group signs under. It carries no request + // identifier, so it is read before the request-topic gate below. + if rawLog.Topics[0] == events["GroupRegistered"].ID.Hex() { + if len(rawLog.Topics) != 3 { + continue + } + groupIDBytes, err := decodeCanonicalEthereumBytes( + rawLog.Topics[1], + 32, + ) + if err != nil { + continue + } + groupIDValue := new(big.Int).SetBytes(groupIDBytes) + if !groupIDValue.IsUint64() { + continue + } + groupID := groupIDValue.String() + + keyHash, err := decodeCanonicalEthereumBytes( + rawLog.Topics[2], + 32, + ) + if err != nil { + continue + } + registeredKey := hex.EncodeToString(keyHash) + + if known, seen := logs.registeredGroupKeys[groupID]; seen && + known != registeredKey { + logs.ambiguousGroupRegistrations[groupID] = struct{}{} + delete(logs.registeredGroupKeys, groupID) + continue + } + if _, ambiguous := + logs.ambiguousGroupRegistrations[groupID]; ambiguous { + continue + } + logs.registeredGroupKeys[groupID] = registeredKey + continue + } + + // The request identifier is the sole indexed input of the other + // three events, so each of their logs always has exactly the + // signature topic and the request topic. + if len(rawLog.Topics) != 2 { + continue + } + requestIDBytes, err := decodeCanonicalEthereumBytes( + rawLog.Topics[1], + 32, + ) + if err != nil { + continue + } + requestID := new(big.Int).SetBytes(requestIDBytes).String() + + switch rawLog.Topics[0] { + case events["RelayEntryRequested"].ID.Hex(): + if known, seen := logs.requestBlocks[requestID]; seen && + known != receipt.BlockNumber { + logs.ambiguousRequests[requestID] = struct{}{} + continue + } + logs.requestBlocks[requestID] = receipt.BlockNumber + + // The request's two non-indexed values are the group the + // beacon selected to answer it and the previous entry it + // signs over. The selected group names a registry index + // rather than a public key, so it is joined to the key a + // permit's record carries through GroupRegistered. + event := events["RelayEntryRequested"] + data, err := decodeCanonicalEthereumDynamicBytes(rawLog.Data) + if err != nil { + continue + } + values, err := event.Inputs.NonIndexed().Unpack(data) + if err != nil || len(values) < 2 { + continue + } + selectedGroupID, ok := values[0].(uint64) + if !ok { + continue + } + previousEntry, ok := values[1].([]byte) + if !ok { + continue + } + canonicalPreviousEntry, ok := canonicalRelayPoint(previousEntry) + if !ok { + continue + } + + selectedGroup := strconv.FormatUint(selectedGroupID, 10) + if known, seen := logs.requestGroups[requestID]; seen && + known != selectedGroup { + logs.ambiguousRequestGroups[requestID] = struct{}{} + delete(logs.requestGroups, requestID) + } else if _, ambiguous := + logs.ambiguousRequestGroups[requestID]; !ambiguous { + logs.requestGroups[requestID] = selectedGroup + } + + identity := relayEntryIdentity( + receipt.BlockNumber, + canonicalPreviousEntry, + ) + if known, seen := logs.requestIdentities[identity]; seen && + known != requestID { + logs.ambiguousIdentities[identity] = struct{}{} + continue + } + logs.requestIdentities[identity] = requestID + + // One request identifier answering two different requests + // would let either one's evidence close the other's permit, + // so the identifier binds neither rather than binding both. + if known, seen := logs.identityRequests[requestID]; seen && + known != identity { + logs.ambiguousRequests[requestID] = struct{}{} + continue + } + logs.identityRequests[requestID] = identity + case events["RelayEntrySubmitted"].ID.Hex(): + logs.submittedRequests[requestID] = struct{}{} + + // The accepted entry is the submission's second non-indexed + // value; the first is the submitter, which says who published + // the entry rather than what the beacon accepted. + event := events["RelayEntrySubmitted"] + data, err := decodeCanonicalEthereumDynamicBytes(rawLog.Data) + if err != nil { + continue + } + values, err := event.Inputs.NonIndexed().Unpack(data) + if err != nil || len(values) < 2 { + continue + } + entry, ok := values[1].([]byte) + if !ok { + continue + } + canonicalEntry, ok := canonicalRelayPoint(entry) + if !ok { + continue + } + + if known, seen := logs.submittedEntries[requestID]; seen && + !bytes.Equal(known, canonicalEntry) { + logs.ambiguousSubmissions[requestID] = struct{}{} + delete(logs.submittedEntries, requestID) + continue + } + if _, ambiguous := + logs.ambiguousSubmissions[requestID]; ambiguous { + continue + } + logs.submittedEntries[requestID] = canonicalEntry + case events["RelayEntryTimedOut"].ID.Hex(): + event := events["RelayEntryTimedOut"] + data, err := decodeCanonicalEthereumDynamicBytes(rawLog.Data) + if err != nil { + continue + } + values, err := event.Inputs.NonIndexed().Unpack(data) + if err != nil || len(values) == 0 { + continue + } + terminatedGroupID, ok := values[0].(uint64) + if !ok { + continue + } + + logs.timeouts[requestID+":"+strconv.FormatUint( + terminatedGroupID, + 10, + )] = receipt.BlockNumber + } + } + } + + return logs, nil +} + +// reconcileRelayTimeoutSettlements joins every relay entry timeout penalty the +// node recorded as its permit's result to the beacon's own authenticated logs. +// +// The offline journal pass has already established what a node can be held to +// on its own word: the settlement it names answers the request start block its +// permit was issued for, and one settlement answers one request across the +// whole journal. Neither says the penalty happened. A node that filed a report +// which reverted, was dropped, or lost the race to another reporter can render +// exactly the same reference as one whose report the beacon accepted. +// +// Three authenticated readings decide it, and the node's word supplies none of +// them. A canonical RelayEntryTimedOut log emitted by the expected RandomBeacon +// must name the exact request identifier and terminated group the reference +// carries — the penalty happened. A canonical RelayEntryRequested log for that +// same request must sit in the block the permit was issued for — the penalty is +// this permit's, not a real penalty from some other request borrowed to close +// it. And no RelayEntrySubmitted log may answer that request — a delivered +// entry and a timeout are mutually exclusive endings, so evidence carrying both +// settles nothing. +// +// The enclosing receipt validation has already bound every receipt to an +// attested canonical block with successful status, so a matched log is chain +// state rather than a report the evidence generator wrote. Evidence that simply +// omits the beacon's logs proves nothing either way and blocks the barrier: an +// unproven penalty is exactly what the barrier exists to hold. +func (r *auditRun) reconcileRelayTimeoutSettlements( + record *chainReconciliationEvidence, +) []string { + if r.manifest.ParticipationTerminalOutcomes == nil { + return nil + } + + var violations []string + var logs *relayEntryLifecycleLogs + for i, outcome := range r.manifest.ParticipationTerminalOutcomes.Outcomes { + if outcome.Permit.Ceremony != participation.BeaconTimeoutReport || + outcome.Outcome != participation.TerminalOutcomeCompleted { + continue + } + + referenceStartBlock, requestID, terminatedGroupID, err := + participation.ParseBeaconRelayTimeoutSettlementReference( + outcome.Evidence.Reference, + ) + if err != nil { + // The journal pass reports the unparseable reference itself; this + // pass has nothing left to join it to. + continue + } + + if logs == nil { + logs, err = authenticatedRelayEntryLogs(record) + if err != nil { + violations = append(violations, fmt.Sprintf( + "cannot decode the authenticated beacon relay entry logs: "+ + "[%v]", + err, + )) + break + } + } + + settlementIdentity := requestID.String() + ":" + + strconv.FormatUint(terminatedGroupID, 10) + if _, corroborated := logs.timeouts[settlementIdentity]; !corroborated { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry timeout "+ + "settlement [%s], but no authenticated RandomBeacon "+ + "RelayEntryTimedOut log names that request and terminated "+ + "group", + i, + outcome.Evidence.Reference, + )) + continue + } + + if _, ambiguous := + logs.ambiguousRequests[requestID.String()]; ambiguous { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry timeout "+ + "settlement [%s], but the authenticated logs place request "+ + "[%s] at more than one block, so the terminated request "+ + "cannot be bound to the permit", + i, + outcome.Evidence.Reference, + requestID, + )) + continue + } + requestBlock, requested := logs.requestBlocks[requestID.String()] + if !requested { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry timeout "+ + "settlement [%s], but no authenticated RandomBeacon "+ + "RelayEntryRequested log names request [%s], so the "+ + "terminated request cannot be bound to the permit", + i, + outcome.Evidence.Reference, + requestID, + )) + continue + } + if requestBlock != referenceStartBlock { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry timeout "+ + "settlement [%s], but the authenticated logs make request "+ + "[%s] at block [%d], not the request start block the "+ + "permit was issued for", + i, + outcome.Evidence.Reference, + requestID, + requestBlock, + )) + continue + } + + if _, delivered := + logs.submittedRequests[requestID.String()]; delivered { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry timeout "+ + "settlement [%s], but an authenticated RandomBeacon "+ + "RelayEntrySubmitted log answers request [%s]; a delivered "+ + "entry and a timeout cannot both settle one request", + i, + outcome.Evidence.Reference, + requestID, + )) + } + } + + return violations +} + +// relayGroupKeyHash renders the beacon's own identifier for a group public +// key: the hash of the uncompressed point. +// +// The registry keys groups by that hash and GroupRegistered indexes the key, +// so the hash — not the key bytes — is what an authenticated log carries and +// what a record's compressed key has to be reduced to before the two can be +// compared at all. +func relayGroupKeyHash(compressedGroupPublicKey []byte) (string, error) { + groupPublicKey, err := altbn128.DecompressToG2(compressedGroupPublicKey) + if err != nil { + return "", err + } + + return hex.EncodeToString( + ethereumCrypto.Keccak256(groupPublicKey.Marshal()), + ), nil +} + +// relayEntryGroupSelectionViolation reports why the group a completed relay +// signing outcome names cannot be the group the beacon selected to answer the +// request, or the empty string when the authenticated logs do not contradict +// it. +// +// The recovered entry on its own proves only that some group this node holds +// key material for signed the previous entry, which is all the pairing check in +// the journal pass can establish. It does not say the group was the one this +// request selected. Relay entries are deterministic, so an entry another +// locally held group produced over the same previous entry is a real signature +// that verifies — and without this join it would close the selected group's +// permit, recording work the selected group never did. +// +// The beacon's own RelayEntryRequested log names the selected group by its +// registry index, and GroupRegistered binds that index to the hash of the key +// the group signs under. Together they turn the node's claim into one the chain +// can refuse. +// +// Both halves of that join are mandatory. The registration that binds a group +// is older than the request — blocks or months older — so an evidence bundle +// gathered around the request has no reason to carry it unless the generator is +// told to go and fetch it, and that is exactly what the generator contract now +// requires. Treating its absence as consent would leave the whole join +// optional: a node closing a selected group's permit with another of its +// groups' entries need only hand in evidence without the registration, which is +// the shape an honest bundle would have had. So a missing selection or a +// missing registration blocks, alongside evidence that contradicts itself or +// the record — a request selecting two groups, a group registered under two +// keys, or a selected group whose registered key is not the one the record +// names. +func relayEntryGroupSelectionViolation( + logs *relayEntryLifecycleLogs, + outcomeIndex int, + reference string, + requestID string, + compressedGroupPublicKey []byte, +) string { + if _, ambiguous := logs.ambiguousRequestGroups[requestID]; ambiguous { + return fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated logs select more than one group for request "+ + "[%s], so the group the entry had to come from is ambiguous", + outcomeIndex, + reference, + requestID, + ) + } + selectedGroup, selected := logs.requestGroups[requestID] + if !selected { + return fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but no "+ + "authenticated RandomBeacon RelayEntryRequested log names the "+ + "group selected to answer request [%s], so nothing says the "+ + "entry came from the group the beacon asked", + outcomeIndex, + reference, + requestID, + ) + } + + if _, ambiguous := + logs.ambiguousGroupRegistrations[selectedGroup]; ambiguous { + return fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated logs register group [%s] — the group request "+ + "[%s] selected — under more than one public key", + outcomeIndex, + reference, + selectedGroup, + requestID, + ) + } + registeredKeyHash, registered := logs.registeredGroupKeys[selectedGroup] + if !registered { + return fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but no "+ + "authenticated RandomBeacon GroupRegistered log registers "+ + "group [%s] — the group request [%s] selected — so the key "+ + "the entry names is bound to no selected group", + outcomeIndex, + reference, + selectedGroup, + requestID, + ) + } + + namedKeyHash, err := relayGroupKeyHash(compressedGroupPublicKey) + if err != nil { + // A group key that is not a point on the curve is reported by the + // journal pass, which verifies the entry under it; there is nothing + // this join can add to that. + return "" + } + + if namedKeyHash != registeredKeyHash { + return fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated logs have request [%s] answered by group [%s], "+ + "registered under public key hash [%s] rather than the key "+ + "the entry names", + outcomeIndex, + reference, + requestID, + selectedGroup, + registeredKeyHash, + ) + } + + return "" +} + +// reconcileRelayEntryResults binds every relay entry a node recorded as its +// signing permit's result to the beacon's own request, and refuses one the +// beacon contradicts. +// +// The offline journal pass verifies the entry itself: a threshold BLS signature +// by a group whose key material the snapshot decoded, over the previous entry +// the record names. That proves authorship, and every entry the beacon ever +// produced keeps it forever. What it cannot prove is that the ceremony this +// permit was issued for is the one that produced it. The record's binding to a +// request is a start block the node wrote next to an entry it chose, so a +// historical entry relabelled with a live permit's block satisfies the pairing +// check and the journal-wide replay guard alike — the entry it replays was +// never in this journal to begin with. +// +// A canonical RelayEntryRequested log closes that. The beacon identifies a +// request by the block it was made in and the previous entry it signs over, +// which is exactly the pair a relay entry record names, so the log the beacon +// emitted for this permit's block has to be the one signing over this record's +// previous entry. An entry recovered for another request signs over that +// request's previous entry and matches no request at this block. +// +// The same log names the group the beacon selected to answer the request, and +// that group has to be the one the record names. A node holding memberships of +// several groups can otherwise close a selected group's permit with an entry a +// different one of its groups produced over the same previous entry, which +// verifies and answers this very request but records work by a group that did +// none. The selection names a registry index rather than a key, so the +// GroupRegistered receipt that binds that index is required as well: without it +// the index binds nothing, and leaving it out would be the cheapest way to skip +// the check entirely. +// +// Where the beacon accepted an entry for that request, it must be the entry the +// node named. A submission is not required, though: a group's threshold +// recovers the entry regardless of which member publishes it, and a recovery +// whose on-chain submission then reverted, was dropped, or lost the race is +// still that ceremony's durable result. Requiring one would refuse a completed +// ceremony for a transaction outcome that says nothing about it. +func (r *auditRun) reconcileRelayEntryResults( + record *chainReconciliationEvidence, +) []string { + if r.manifest.ParticipationTerminalOutcomes == nil { + return nil + } + + var violations []string + var logs *relayEntryLifecycleLogs + for i, outcome := range r.manifest.ParticipationTerminalOutcomes.Outcomes { + if outcome.Permit.Ceremony != participation.BeaconRelaySigning || + outcome.Outcome != participation.TerminalOutcomeCompleted { + continue + } + + referenceStartBlock, groupPublicKey, previousEntry, entry, err := + participation.ParseBeaconRelayEntryReference( + outcome.Evidence.Reference, + ) + if err != nil { + // The journal pass reports the unparseable reference itself; this + // pass has nothing left to join it to. + continue + } + + if logs == nil { + logs, err = authenticatedRelayEntryLogs(record) + if err != nil { + violations = append(violations, fmt.Sprintf( + "cannot decode the authenticated beacon relay entry logs: "+ + "[%v]", + err, + )) + break + } + } + + identity := relayEntryIdentity(referenceStartBlock, previousEntry) + if _, ambiguous := logs.ambiguousIdentities[identity]; ambiguous { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated logs make more than one request over that "+ + "previous entry at block [%d], so the answered request "+ + "cannot be bound to the permit", + i, + outcome.Evidence.Reference, + referenceStartBlock, + )) + continue + } + requestID, requested := logs.requestIdentities[identity] + if !requested { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but no "+ + "authenticated RandomBeacon RelayEntryRequested log makes a "+ + "request over that previous entry at block [%d], so the "+ + "entry answers no request the permit was issued for", + i, + outcome.Evidence.Reference, + referenceStartBlock, + )) + continue + } + if _, ambiguous := logs.ambiguousRequests[requestID]; ambiguous { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated logs answer more than one request under "+ + "identifier [%s], so the request the entry answers cannot "+ + "be bound to the permit", + i, + outcome.Evidence.Reference, + requestID, + )) + continue + } + + if selectionViolation := relayEntryGroupSelectionViolation( + logs, + i, + outcome.Evidence.Reference, + requestID, + groupPublicKey, + ); selectionViolation != "" { + violations = append(violations, selectionViolation) + continue + } + + if _, ambiguous := logs.ambiguousSubmissions[requestID]; ambiguous { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated logs record more than one entry accepted "+ + "for request [%s]", + i, + outcome.Evidence.Reference, + requestID, + )) + continue + } + if accepted, submitted := logs.submittedEntries[requestID]; submitted && + !bytes.Equal(accepted, entry) { + violations = append(violations, fmt.Sprintf( + "node-authored outcome [%d] reports relay entry [%s], but the "+ + "authenticated RandomBeacon RelayEntrySubmitted log for "+ + "request [%s] accepted a different entry", + i, + outcome.Evidence.Reference, + requestID, + )) + } + } + + return violations +} + +func expectedTBTCDKGRawLog( + eventName string, + result *tbtcDKGResultEvidence, +) ([]string, string, error) { + parsed, err := ecdsaabi.WalletRegistryMetaData.GetAbi() + if err != nil { + return nil, "", fmt.Errorf( + "cannot load the generated WalletRegistry ABI: [%v]", + err, + ) + } + event, ok := parsed.Events[eventName] + if !ok { + return nil, "", fmt.Errorf( + "generated WalletRegistry ABI has no [%s] event", + eventName, + ) + } + + topics := []string{event.ID.Hex()} + var values []interface{} + switch eventName { + case "DkgStarted": + topics = append(topics, result.Started.Seed) + case "DkgResultSubmitted": + topics = append( + topics, + result.Submitted.ResultHash, + result.Submitted.Seed, + ) + abiValue, err := tbtcDKGResultABIValue(result.Submitted.Result) + if err != nil { + return nil, "", err + } + values = append(values, abiValue) + case "DkgResultApproved": + // The approver is not part of wallet identity, but its indexed address + // must still be present and canonically encoded in the raw log. An + // empty expected topic below is that one explicit wildcard. + topics = append(topics, result.Approved.ResultHash, "") + case "WalletCreated": + topics = append( + topics, + result.WalletCreated.WalletID, + result.WalletCreated.DKGResultHash, + ) + default: + return nil, "", fmt.Errorf( + "unsupported WalletRegistry event [%s]", + eventName, + ) + } + + data, err := event.Inputs.NonIndexed().Pack(values...) + if err != nil { + return nil, "", fmt.Errorf( + "cannot ABI-encode expected [%s] event data: [%v]", + eventName, + err, + ) + } + return topics, "0x" + hex.EncodeToString(data), nil +} + +func validateTBTCDKGAuthenticatedLineage( + walletStorageKey string, + result *tbtcDKGResultEvidence, + record *chainReconciliationEvidence, +) []string { + var violations []string + + receipts := make(map[string]ethereumReceiptEvidence) + for _, receipt := range record.Receipts { + if _, duplicate := receipts[receipt.TransactionHash]; !duplicate { + receipts[receipt.TransactionHash] = receipt + } + } + + for _, observed := range []struct { + name string + log ethereumLogEvidence + }{ + {"DkgStarted", result.Started.ethereumLogEvidence}, + {"DkgResultSubmitted", result.Submitted.ethereumLogEvidence}, + {"DkgResultApproved", result.Approved.ethereumLogEvidence}, + {"WalletCreated", result.WalletCreated.ethereumLogEvidence}, + } { + receipt, ok := receipts[observed.log.TransactionHash] + if !ok { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event transaction [%s] has no "+ + "authenticated receipt", + walletStorageKey, + observed.name, + observed.log.TransactionHash, + )) + continue + } + if receipt.BlockHash != observed.log.BlockHash || + receipt.BlockNumber != observed.log.BlockNumber { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event block identity does not match "+ + "its authenticated receipt", + walletStorageKey, + observed.name, + )) + } + if receipt.Status != 1 { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event belongs to failed receipt [%s]", + walletStorageKey, + observed.name, + observed.log.TransactionHash, + )) + } + + var rawLog *ethereumRawLogEvidence + for i := range receipt.Logs { + if receipt.Logs[i].LogIndex == observed.log.LogIndex { + rawLog = &receipt.Logs[i] + break + } + } + if rawLog == nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event log index [%d] is absent from "+ + "authenticated receipt [%s]", + walletStorageKey, + observed.name, + observed.log.LogIndex, + observed.log.TransactionHash, + )) + continue + } + if rawLog.Address != record.WalletRegistryAddress { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event was emitted by unrelated "+ + "contract [%s], expected WalletRegistry [%s]", + walletStorageKey, + observed.name, + rawLog.Address, + record.WalletRegistryAddress, + )) + } + + expectedTopics, expectedData, err := expectedTBTCDKGRawLog( + observed.name, + result, + ) + if err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] cannot derive expected %s event bytes: [%v]", + walletStorageKey, + observed.name, + err, + )) + continue + } + if len(rawLog.Topics) != len(expectedTopics) { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s raw log has [%d] topics, expected [%d]", + walletStorageKey, + observed.name, + len(rawLog.Topics), + len(expectedTopics), + )) + } else { + for i, expectedTopic := range expectedTopics { + if expectedTopic == "" { + decoded, err := decodeCanonicalEthereumBytes( + rawLog.Topics[i], + 32, + ) + if err == nil { + for _, prefix := range decoded[:12] { + if prefix != 0 { + err = fmt.Errorf( + "address topic is not left-padded with zeroes", + ) + break + } + } + } + if err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s raw approver topic is "+ + "invalid: [%v]", + walletStorageKey, + observed.name, + err, + )) + } + continue + } + if rawLog.Topics[i] != expectedTopic { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s raw topic [%d] [%s] does not "+ + "match decoded event value [%s]", + walletStorageKey, + observed.name, + i, + rawLog.Topics[i], + expectedTopic, + )) + } + } + } + if rawLog.Data != expectedData { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s raw data does not match the decoded "+ + "event value", + walletStorageKey, + observed.name, + )) + } + } + + return violations +} + +func validateTBTCDKGResultEvidence( + walletStorageKey string, + walletID string, + result *tbtcDKGResultEvidence, + record *chainReconciliationEvidence, +) []string { + violations := make([]string, 0) + resultHash := result.resultHash() + violations = append( + violations, + validateTBTCDKGAuthenticatedLineage( + walletStorageKey, + result, + record, + )..., + ) + + for _, event := range []struct { + name string + log ethereumLogEvidence + }{ + {"DkgStarted", result.Started.ethereumLogEvidence}, + {"DkgResultSubmitted", result.Submitted.ethereumLogEvidence}, + {"DkgResultApproved", result.Approved.ethereumLogEvidence}, + {"WalletCreated", result.WalletCreated.ethereumLogEvidence}, + } { + if _, err := decodeCanonicalEthereumBytes( + event.log.TransactionHash, + 32, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event has invalid transaction hash "+ + "[%s]: [%v]", + walletStorageKey, + event.name, + event.log.TransactionHash, + err, + )) + } + if _, err := decodeCanonicalEthereumBytes( + event.log.BlockHash, + 32, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event has invalid block hash [%s]: [%v]", + walletStorageKey, + event.name, + event.log.BlockHash, + err, + )) + } + if event.log.BlockNumber == 0 { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s event has zero block number", + walletStorageKey, + event.name, + )) + } + } + + if _, err := decodeCanonicalEthereumBytes(result.Started.Seed, 32); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgStarted event has invalid seed [%s]: [%v]", + walletStorageKey, + result.Started.Seed, + err, + )) + } + if _, err := decodeCanonicalEthereumBytes(result.Submitted.Seed, 32); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgResultSubmitted event has invalid seed "+ + "[%s]: [%v]", + walletStorageKey, + result.Submitted.Seed, + err, + )) + } + if result.Started.Seed != result.Submitted.Seed { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgStarted seed [%s] does not match "+ + "DkgResultSubmitted seed [%s]", + walletStorageKey, + result.Started.Seed, + result.Submitted.Seed, + )) + } + + for _, field := range []struct { + name string + value string + }{ + {"DkgResultSubmitted result hash", result.Submitted.ResultHash}, + {"DkgResultApproved result hash", result.Approved.ResultHash}, + {"WalletCreated DKG result hash", result.WalletCreated.DKGResultHash}, + {"WalletCreated wallet ID", result.WalletCreated.WalletID}, + } { + if _, err := decodeCanonicalEthereumBytes(field.value, 32); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] %s [%s] is invalid: [%v]", + walletStorageKey, + field.name, + field.value, + err, + )) + } + } + + if result.Approved.ResultHash != result.Submitted.ResultHash { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgResultApproved hash [%s] does not match "+ + "DkgResultSubmitted hash [%s]", + walletStorageKey, + result.Approved.ResultHash, + result.Submitted.ResultHash, + )) + } + if result.WalletCreated.DKGResultHash != result.Submitted.ResultHash { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] WalletCreated DKG result hash [%s] does not "+ + "match DkgResultSubmitted hash [%s]", + walletStorageKey, + result.WalletCreated.DKGResultHash, + result.Submitted.ResultHash, + )) + } + + if result.Started.BlockNumber > result.Submitted.BlockNumber { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgStarted block [%d] is after "+ + "DkgResultSubmitted block [%d]", + walletStorageKey, + result.Started.BlockNumber, + result.Submitted.BlockNumber, + )) + } + if result.Submitted.BlockNumber > result.Approved.BlockNumber { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgResultSubmitted block [%d] is after "+ + "DkgResultApproved block [%d]", + walletStorageKey, + result.Submitted.BlockNumber, + result.Approved.BlockNumber, + )) + } + if result.Approved.TransactionHash != result.WalletCreated.TransactionHash || + result.Approved.BlockHash != result.WalletCreated.BlockHash || + result.Approved.BlockNumber != result.WalletCreated.BlockNumber { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgResultApproved and WalletCreated events "+ + "do not belong to the same approval receipt", + walletStorageKey, + )) + } else if result.WalletCreated.LogIndex <= result.Approved.LogIndex { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] WalletCreated log index [%d] does not follow "+ + "DkgResultApproved log index [%d] in the approval receipt", + walletStorageKey, + result.WalletCreated.LogIndex, + result.Approved.LogIndex, + )) + } + + chainResult := result.Submitted.Result + originalGroupSize := result.originalGroupSize() + if originalGroupSize == 0 || + originalGroupSize > uint16(group.MaxMemberIndex) { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has invalid original group "+ + "size [%d]", + walletStorageKey, + resultHash, + originalGroupSize, + )) + } + if chainResult.SubmitterMemberIndex == 0 || + chainResult.SubmitterMemberIndex > originalGroupSize { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has submitter member [%d] "+ + "outside original group size [%d]", + walletStorageKey, + resultHash, + chainResult.SubmitterMemberIndex, + originalGroupSize, + )) + } + if len(chainResult.Members) > int(group.MaxMemberIndex) { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] carries [%d] original "+ + "members, exceeding [%d]", + walletStorageKey, + resultHash, + len(chainResult.Members), + group.MaxMemberIndex, + )) + } + for i, member := range chainResult.Members { + if member == 0 { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has zero operator ID at "+ + "original member [%d]", + walletStorageKey, + resultHash, + i+1, + )) + } + } + if chainResult.MisbehavedMemberIndexes == nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] omits the complete "+ + "misbehaved-member set", + walletStorageKey, + resultHash, + )) + } + var previous uint8 + for i, memberIndex := range chainResult.MisbehavedMemberIndexes { + if memberIndex == 0 || + uint16(memberIndex) > originalGroupSize { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has misbehaved member "+ + "[%d] outside original group size [%d]", + walletStorageKey, + resultHash, + memberIndex, + originalGroupSize, + )) + } + if i > 0 && memberIndex <= previous { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] misbehaved members are "+ + "not strictly increasing at [%d]", + walletStorageKey, + resultHash, + memberIndex, + )) + } + previous = memberIndex + } + if len(chainResult.MisbehavedMemberIndexes) >= int(originalGroupSize) { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] leaves no final signing-group "+ + "member", + walletStorageKey, + resultHash, + )) + } + + if chainResult.SigningMemberIndexes == nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] omits signing-member indexes", + walletStorageKey, + resultHash, + )) + } else if len(chainResult.SigningMemberIndexes) == 0 { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has no signing member", + walletStorageKey, + resultHash, + )) + } + misbehaved := make(map[uint8]struct{}, len(chainResult.MisbehavedMemberIndexes)) + for _, memberIndex := range chainResult.MisbehavedMemberIndexes { + misbehaved[memberIndex] = struct{}{} + } + if _, excluded := misbehaved[uint8(chainResult.SubmitterMemberIndex)]; excluded { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] names misbehaved member [%d] "+ + "as its submitter", + walletStorageKey, + resultHash, + chainResult.SubmitterMemberIndex, + )) + } + var previousSigning *big.Int + for i, memberIndex := range chainResult.SigningMemberIndexes { + if memberIndex == nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has null signing member "+ + "at position [%d]", + walletStorageKey, + resultHash, + i, + )) + continue + } + if memberIndex.Sign() <= 0 || + !memberIndex.IsUint64() || + memberIndex.Uint64() > uint64(originalGroupSize) { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has signing member [%s] "+ + "outside original group size [%d]", + walletStorageKey, + resultHash, + memberIndex.String(), + originalGroupSize, + )) + } + if memberIndex.IsUint64() { + if _, excluded := misbehaved[uint8(memberIndex.Uint64())]; excluded { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] names misbehaved "+ + "member [%s] as a signer", + walletStorageKey, + resultHash, + memberIndex.String(), + )) + } + } + if previousSigning != nil && memberIndex.Cmp(previousSigning) <= 0 { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] signing members are not "+ + "strictly increasing at [%s]", + walletStorageKey, + resultHash, + memberIndex.String(), + )) + } + previousSigning = memberIndex + } + if signatures, err := decodeCanonicalEthereumDynamicBytes( + chainResult.Signatures, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] has invalid signatures: [%v]", + walletStorageKey, + resultHash, + err, + )) + } else if len(signatures) != 65*len(chainResult.SigningMemberIndexes) { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] carries [%d] signature bytes "+ + "for [%d] signing members", + walletStorageKey, + resultHash, + len(signatures), + len(chainResult.SigningMemberIndexes), + )) + } + + if calculatedMembersHash, err := computeTBTCDKGMembersHash( + chainResult, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] cannot derive DKG members hash: [%v]", + walletStorageKey, + err, + )) + } else if calculatedMembersHash != chainResult.MembersHash { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DKG result [%s] members hash [%s] does not "+ + "match the derived operating-members hash [%s]", + walletStorageKey, + resultHash, + chainResult.MembersHash, + calculatedMembersHash, + )) + } + + if calculatedResultHash, err := computeTBTCDKGResultHash( + chainResult, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] cannot derive DKG result hash from the "+ + "submitted event: [%v]", + walletStorageKey, + err, + )) + } else if calculatedResultHash != result.Submitted.ResultHash { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] DkgResultSubmitted hash [%s] does not match "+ + "keccak256(abi.encode(result)) [%s]", + walletStorageKey, + result.Submitted.ResultHash, + calculatedResultHash, + )) + } + + if calculatedWalletID, err := computeTBTCWalletID( + chainResult.GroupPublicKey, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] cannot derive wallet ID from the submitted "+ + "group public key: [%v]", + walletStorageKey, + err, + )) + } else { + if calculatedWalletID != result.WalletCreated.WalletID { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] WalletCreated ID [%s] does not match "+ + "keccak256(group public key) [%s]", + walletStorageKey, + result.WalletCreated.WalletID, + calculatedWalletID, + )) + } + if strings.TrimPrefix(calculatedWalletID, "0x") != walletID { + violations = append(violations, fmt.Sprintf( + "tbtc wallet [%s] accepted event lineage derives wallet ID "+ + "[%s], but chain reconciliation names [%s]", + walletStorageKey, + strings.TrimPrefix(calculatedWalletID, "0x"), + walletID, + )) + } + } + + return violations +} + +// validateTBTCDKGTerminalLineage binds a completed node-owned DKG permit to +// the canonical accepted result for the persisted wallet it names. tBTC +// permits use the original selected-group index while persisted memberships +// use the compacted final signing-group index; the result's misbehaved seats +// are the only authoritative mapping between those identities. +func validateTBTCDKGTerminalLineage( + auditManifest *manifest, + record *chainReconciliationEvidence, + wallets map[string]int, +) []string { + journal := auditManifest.ParticipationTerminalOutcomes + if journal == nil { + return nil + } + + violations := make([]string, 0) + for i, outcome := range journal.Outcomes { + if outcome.Outcome != participation.TerminalOutcomeCompleted || + outcome.Permit.Ceremony != participation.TBTCDKG || + outcome.Evidence.Kind != + participation.TerminalEvidencePersistedTBTCSinger { + continue + } + + walletIndex, ok := wallets[outcome.Evidence.Reference] + if !ok { + continue + } + wallet := record.Wallets[walletIndex] + if wallet.DKGSettlement != "approved" || wallet.DKGResult == nil { + continue + } + result := wallet.DKGResult + resultSeedHash, _ := result.seedHash() + resultHash := result.resultHash() + resultStartBlock := result.startBlock() + originalGroupSize := result.originalGroupSize() + misbehavedMemberIndexes := result.misbehavedMemberIndexes() + + if outcome.Permit.WorkID != resultSeedHash { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] belongs to "+ + "seed [%s], but persisted wallet [%s] was created by "+ + "canonical result [%s] for seed [%s]", + i, + outcome.Permit.WorkID, + wallet.WalletStorageKey, + resultHash, + resultSeedHash, + )) + } + if outcome.Permit.CanonicalStartBlock != resultStartBlock { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] has "+ + "canonical start block [%d], but persisted wallet [%s] "+ + "was created by result [%s] anchored at [%d]", + i, + outcome.Permit.CanonicalStartBlock, + wallet.WalletStorageKey, + resultHash, + resultStartBlock, + )) + } + + originalMemberIndex, err := strconv.ParseUint( + outcome.Permit.PermitID, + 10, + 8, + ) + if err != nil { + continue + } + finalMemberIndex, included := finalTBTCDKGMembership( + group.MemberIndex(originalMemberIndex), + originalGroupSize, + misbehavedMemberIndexes, + ) + if !included { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] belongs to "+ + "original member [%s], but canonical result [%s] does "+ + "not include that member in its final signing group", + i, + outcome.Permit.PermitID, + resultHash, + )) + } else if outcome.Evidence.MembershipIndex != finalMemberIndex { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] belongs to "+ + "original member [%s], which canonical result [%s] maps "+ + "to final membership [%d], but names persisted "+ + "membership [%d]", + i, + outcome.Permit.PermitID, + resultHash, + finalMemberIndex, + outcome.Evidence.MembershipIndex, + )) + } + + violations = append(violations, validateTBTCDKGTranscriptLineage( + i, + outcome, + resultHash, + originalGroupSize, + misbehavedMemberIndexes, + )...) + } + + return violations +} + +// validateTBTCDKGTranscriptLineage holds the whole seat map a completed tBTC +// DKG record publishes against the one the canonical accepted result implies. +// +// Checking only this permit's own entry leaves the rest of the map unchecked, +// and the rest of the map is what says who the other seats belonged to. A record +// whose remote entries were rewritten still names the right seed, the right +// anchor, the right length and the right local seat, so every other check here +// passes — while a reader translating the transcript through it attributes final +// seats to original members that never held them. That reader is the fleet-wide +// ownership map, and a seat moved in it is a seat some other release appears to +// have supplied. +// +// The expected map is derived rather than trusted: the final signing group is +// the accepted result's members with its misbehaved seats removed, ascending, so +// final seat i came from the i-th survivor and nothing about that is the +// recording node's to state. +func validateTBTCDKGTranscriptLineage( + index int, + outcome participation.TerminalOutcomeRecord, + resultHash string, + originalGroupSize uint16, + misbehavedMemberIndexes []uint8, +) []string { + contribution := outcome.Evidence.Contribution + if contribution == nil { + return nil + } + + survivors := canonicalTBTCDKGSurvivors( + originalGroupSize, + misbehavedMemberIndexes, + ) + expectedFinal := make(participation.MemberIndexes, 0, len(survivors)) + for seat := 1; seat <= len(survivors); seat++ { + expectedFinal = append(expectedFinal, group.MemberIndex(seat)) + } + + violations := make([]string, 0) + if !slices.Equal(contribution.IncorporatedMembers, expectedFinal) { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] names final "+ + "signing group %v, but canonical result [%s] rebuilds group "+ + "%v from its %d accepted members", + index, + contribution.IncorporatedMembers, + resultHash, + expectedFinal, + len(survivors), + )) + } + if !slices.Equal(contribution.PermitSpaceMembers, survivors) { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] maps its final "+ + "signing group back to original members %v, but canonical "+ + "result [%s] leaves survivors %v after removing its "+ + "misbehaved seats", + index, + contribution.PermitSpaceMembers, + resultHash, + survivors, + )) + } + + return violations +} + +// canonicalTBTCDKGSurvivors returns the original DKG memberships an accepted +// result's final signing group was rebuilt from, ascending: entry i is the +// membership that became final seat i+1. +// +// A misbehaved index outside the original group, and a repeat of one already +// counted, are ignored rather than treated as removing a seat, exactly as the +// final-membership derivation beside this does. Counting either would shorten +// the expected group and make every record of a perfectly ordinary result +// disagree with it. +func canonicalTBTCDKGSurvivors( + originalGroupSize uint16, + misbehavedMemberIndexes []uint8, +) participation.MemberIndexes { + misbehaved := make(map[uint8]struct{}, len(misbehavedMemberIndexes)) + for _, rawMisbehaved := range misbehavedMemberIndexes { + if rawMisbehaved == 0 || uint16(rawMisbehaved) > originalGroupSize { + continue + } + misbehaved[rawMisbehaved] = struct{}{} + } + + survivors := make(participation.MemberIndexes, 0, originalGroupSize) + for index := uint16(1); index <= originalGroupSize; index++ { + if _, removed := misbehaved[uint8(index)]; removed { + continue + } + survivors = append(survivors, group.MemberIndex(index)) + } + + return survivors +} + +func finalTBTCDKGMembership( + originalMemberIndex group.MemberIndex, + originalGroupSize uint16, + misbehavedMemberIndexes []uint8, +) (group.MemberIndex, bool) { + if originalMemberIndex == 0 || + uint16(originalMemberIndex) > originalGroupSize { + return 0, false + } + + misbehavedBefore := uint16(0) + seen := make(map[uint8]struct{}, len(misbehavedMemberIndexes)) + for _, rawMisbehaved := range misbehavedMemberIndexes { + if rawMisbehaved == 0 || + uint16(rawMisbehaved) > originalGroupSize { + continue + } + if _, duplicate := seen[rawMisbehaved]; duplicate { + continue + } + seen[rawMisbehaved] = struct{}{} + + misbehaved := group.MemberIndex(rawMisbehaved) + if misbehaved == originalMemberIndex { + return 0, false + } + if misbehaved < originalMemberIndex { + misbehavedBefore++ + } + } + + return group.MemberIndex( + uint16(originalMemberIndex) - misbehavedBefore, + ), true +} + +// validateBitcoinReconciliationEvidence checks the Bitcoin reconciliation +// record: schema, snapshot binding, network identity, an attested-complete +// pending set, and a valid terminal state for every pending transaction. +func (r *auditRun) validateBitcoinReconciliationEvidence( + content []byte, +) []string { + record := &bitcoinReconciliationEvidence{} + if err := strictUnmarshal(content, record); err != nil { + return []string{fmt.Sprintf( + "cannot be decoded as a bitcoin reconciliation record: [%v]", + err, + )} + } + + violations := r.validateEnvelope( + record.evidenceEnvelope, + "bitcoin_reconciliation", + ) + + if record.BitcoinNetwork == "" { + violations = append(violations, "the Bitcoin network is missing") + } else if r.expected.bitcoinNetwork != "" && + record.BitcoinNetwork != r.expected.bitcoinNetwork { + violations = append(violations, fmt.Sprintf( + "reconciled against Bitcoin network [%s], expected [%s]", + record.BitcoinNetwork, + r.expected.bitcoinNetwork, + )) + } + if !record.Complete { + violations = append( + violations, + "the pending transaction set is not attested complete", + ) + } + reconciled := make(map[string]string, len(record.PendingTransactions)) + for i, transaction := range record.PendingTransactions { + if transaction.TransactionHash == "" { + violations = append(violations, fmt.Sprintf( + "pending transaction entry [%d] is missing its hash", + i, + )) + } else if !isCanonicalBitcoinTransactionHash( + transaction.TransactionHash, + ) { + // A hash in any other rendering cannot be matched against the + // node's own record, so an entry could satisfy the coverage check + // below by naming the same transaction in a shape that never + // compares equal. + violations = append(violations, fmt.Sprintf( + "pending transaction entry [%d] hash [%s] is not a canonical "+ + "lowercase transaction hash", + i, + transaction.TransactionHash, + )) + } else { + reconciled[transaction.TransactionHash] = transaction.State + } + if _, ok := validBitcoinTransactionStates[transaction.State]; !ok { + violations = append(violations, fmt.Sprintf( + "pending transaction entry [%d] has unknown state [%s]", + i, + transaction.State, + )) + } + } + + violations = append( + violations, + r.reconcileSignedTransactionCoverage(reconciled)..., + ) + + return violations +} + +// reconcileSignedTransactionCoverage joins every wallet action the node +// recorded as completed to the reconciled pending-transaction set. The node +// authors that record itself: it pins the transaction hash the moment the +// threshold signature is applied, before any broadcast, precisely because a +// permit canceled mid-broadcast may still have put the transaction on the +// network. So a signed transaction the reconciliation never enumerated is the +// ambiguous case the barrier exists to catch — the set attests it is complete, +// and it is missing a transaction the node knows it signed. Without this join +// the node's own token clears the journal and nothing checks it against +// Bitcoin. +func (r *auditRun) reconcileSignedTransactionCoverage( + reconciled map[string]string, +) []string { + if r.manifest.ParticipationTerminalOutcomes == nil { + return nil + } + + var violations []string + for i, outcome := range r.manifest.ParticipationTerminalOutcomes.Outcomes { + if outcome.Outcome != participation.TerminalOutcomeCompleted || + outcome.Permit.Ceremony != participation.TBTCSigning { + continue + } + if outcome.Evidence.Kind != + participation.TerminalEvidenceBitcoinTransaction { + // The evidence-kind rule already blocks this; skipping here keeps + // one defect from being reported twice. + continue + } + if !isCanonicalBitcoinTransactionHash(outcome.Evidence.Reference) { + violations = append(violations, fmt.Sprintf( + "node-authored completed wallet action [%d] names transaction "+ + "[%s], which is not a canonical transaction hash the "+ + "Bitcoin reconciliation could enumerate", + i, + outcome.Evidence.Reference, + )) + continue + } + if _, covered := reconciled[outcome.Evidence.Reference]; !covered { + violations = append(violations, fmt.Sprintf( + "node-authored completed wallet action [%d] signed transaction "+ + "[%s], which the attested-complete pending transaction set "+ + "does not enumerate", + i, + outcome.Evidence.Reference, + )) + } + } + + return violations +} + +// isCanonicalBitcoinTransactionHash reports whether value is the unprefixed +// lowercase hex rendering a Bitcoin transaction hash serializes to. Any other +// shape — mixed case, a 0x prefix, a truncated digest — is an alias that would +// silently fail every set comparison it takes part in. +func isCanonicalBitcoinTransactionHash(value string) bool { + if len(value) != 2*bitcoin.HashByteLength { + return false + } + for i := 0; i < len(value); i++ { + character := value[i] + if (character < '0' || character > '9') && + (character < 'a' || character > 'f') { + return false + } + } + return true +} + +// quarantineIdentity identifies one quarantined local permit. Several DKG +// events can share an anchor and one node can control several members in one +// event, so ceremony, mode, and block are only classifications. The seed hash +// identifies the chain work; the member index identifies the local permit. +type quarantineIdentity struct { + ceremony string + mode string + canonicalStartBlock uint64 + workID string + permitID string +} + +// persistedSignerIdentity identifies one exact active membership record. A +// wallet or beacon group can contain several locally controlled memberships, +// so the group reference alone cannot corroborate several completed DKG +// permits independently. +type persistedSignerIdentity struct { + reference string + membershipIndex group.MemberIndex +} + +// relayEntryClaim records which relay request a recovered entry has already +// been accepted as the result of, and the outcome that first claimed it. +type relayEntryClaim struct { + outcomeIndex int + requestStartBlock uint64 +} + +// relayTimeoutSettlementClaim records which relay request a node already +// claimed a given beacon timeout settlement for. +type relayTimeoutSettlementClaim struct { + outcomeIndex int + requestStartBlock uint64 +} + +// cutoverModeViolation applies the release gate's one-value schedule to +// persisted evidence. It is shared by quiescence records and both quarantine +// namespaces so completed and interrupted work are judged by the same +// boundary rule. +func cutoverModeViolation( + subject string, + mode string, + canonicalStartBlock uint64, + cutoverBlock uint64, +) string { + legacy := participation.ModeLegacy.String() + securityV2 := participation.ModeSecurityV2.String() + + if cutoverBlock > 0 && canonicalStartBlock == 0 { + return fmt.Sprintf( + "%s has zero canonical anchor under armed cutover block [%d]", + subject, + cutoverBlock, + ) + } + + switch mode { + case legacy: + if cutoverBlock > 0 && canonicalStartBlock >= cutoverBlock { + return fmt.Sprintf( + "%s claims mode [%s] with canonical anchor [%d] at or "+ + "after cutover block [%d]", + subject, + legacy, + canonicalStartBlock, + cutoverBlock, + ) + } + case securityV2: + if cutoverBlock == 0 { + return fmt.Sprintf( + "%s claims mode [%s] under a disabled all-zero schedule", + subject, + securityV2, + ) + } + if canonicalStartBlock < cutoverBlock { + return fmt.Sprintf( + "%s claims mode [%s] with canonical anchor [%d] before "+ + "cutover block [%d]", + subject, + securityV2, + canonicalStartBlock, + cutoverBlock, + ) + } + default: + return fmt.Sprintf( + "%s names unknown protocol mode [%s]", + subject, + mode, + ) + } + + return "" +} + +// validateQuiescencePermitIdentity checks the chain-work and local-permit +// portions of one quiescence entry. DKG work is identified by the SHA-256 hash +// of its seed and each DKG or relay-signing permit belongs to one group member; +// other ceremony classes use the stable chain-native/action identifiers +// accepted by the rehearsal driver. +func validateQuiescencePermitIdentity( + index int, + permit quiescencePermitEvidence, +) []string { + violations := make([]string, 0) + + switch permit.Ceremony { + case string(participation.TBTCDKG), + string(participation.BeaconDKG): + if !isCanonicalSHA256Hex(permit.WorkID) { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] chain work identity [%s] is not a "+ + "canonical SHA-256 seed hash of 64 lowercase "+ + "hexadecimal characters", + index, + permit.WorkID, + )) + } + default: + if !isStableEvidenceID(permit.WorkID) { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] chain work identity [%s] is not a "+ + "stable evidence identifier", + index, + permit.WorkID, + )) + } + } + + switch permit.Ceremony { + case string(participation.TBTCDKG), + string(participation.BeaconDKG), + string(participation.BeaconRelaySigning): + if !isCanonicalMemberIndex(permit.PermitID) { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] local permit identity [%s] is not a "+ + "canonical protocol member index from 1 through %d", + index, + permit.PermitID, + group.MaxMemberIndex, + )) + } + default: + if !isStableEvidenceID(permit.PermitID) { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] local permit identity [%s] is not a "+ + "stable evidence identifier", + index, + permit.PermitID, + )) + } + } + + return violations +} + +func inventoryIdentity(permit participation.PermitSnapshot) quarantineIdentity { + return quarantineIdentity{ + ceremony: string(permit.Ceremony), + mode: permit.Mode, + canonicalStartBlock: permit.CanonicalStartBlock, + workID: permit.WorkID, + permitID: permit.PermitID, + } +} + +func outcomeIdentity(permit quiescencePermitEvidence) quarantineIdentity { + return quarantineIdentity{ + ceremony: permit.Ceremony, + mode: permit.Mode, + canonicalStartBlock: permit.CanonicalStartBlock, + workID: permit.WorkID, + permitID: permit.PermitID, + } +} + +// validateQuiescenceReportEvidence checks the quiescence outcome record: +// schema, snapshot binding, the quiescing node's exact artifact identity and +// cutover schedule, a stated cause, and an independently captured gate +// inventory. The inventory and terminal outcomes must cover exactly the same +// unique permit identities and agree with the gate's total and per-mode +// counts. Every quarantined DKG outcome must also be matched by preserved +// quarantine state carrying the same ceremony, protocol mode, canonical +// anchor, chain-work ID, and local permit ID — one preserved output per +// claiming permit, so one real output cannot vouch for several claims. +func (r *auditRun) validateQuiescenceReportEvidence(content []byte) []string { + record := &quiescenceReportEvidence{} + if err := strictUnmarshal(content, record); err != nil { + return []string{fmt.Sprintf( + "cannot be decoded as a quiescence report: [%v]", + err, + )} + } + + violations := r.validateEnvelope( + record.evidenceEnvelope, + "quiescence_report", + ) + + violations = append(violations, exactIdentityViolations( + "quiescing release version", + record.ReleaseVersion, + r.expected.releaseVersion, + )...) + violations = append(violations, exactIdentityViolations( + "quiescing release revision", + record.ReleaseRevision, + r.expected.releaseRevision, + )...) + violations = append(violations, exactIdentityViolations( + "quiescing release epoch", + record.ReleaseEpoch, + r.expected.releaseEpoch, + )...) + if record.CutoverBlock == 0 { + violations = append( + violations, + "the armed cutover block is missing", + ) + } else if r.expected.cutoverBlock > 0 && + record.CutoverBlock != r.expected.cutoverBlock { + violations = append(violations, fmt.Sprintf( + "quiesced under cutover block [%d], expected [%d]", + record.CutoverBlock, + r.expected.cutoverBlock, + )) + } + + if record.QuiesceCause == "" { + violations = append(violations, "the quiescence cause is missing") + } + + knownCeremonies := make(map[string]struct{}) + for _, ceremony := range participation.AllCeremonies() { + knownCeremonies[string(ceremony)] = struct{}{} + } + + snapshot := r.manifest.QuiescenceSnapshot + if snapshot == nil { + violations = append( + violations, + "the audited storage snapshot contains no node-authored "+ + "quiescence gate snapshot", + ) + snapshot = &participation.QuiescenceSnapshot{} + } + + violations = append(violations, exactIdentityViolations( + "node-authored quiescing release version", + snapshot.ReleaseVersion, + record.ReleaseVersion, + )...) + violations = append(violations, exactIdentityViolations( + "node-authored quiescing release revision", + snapshot.ReleaseRevision, + record.ReleaseRevision, + )...) + violations = append(violations, exactIdentityViolations( + "node-authored quiescing release epoch", + snapshot.ReleaseEpoch, + record.ReleaseEpoch, + )...) + if snapshot.CutoverBlock != record.CutoverBlock { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence used cutover block [%d], but the "+ + "terminal report names [%d]", + snapshot.CutoverBlock, + record.CutoverBlock, + )) + } + if snapshot.QuiesceCause != record.QuiesceCause { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence cause [%s] does not match the "+ + "terminal report cause [%s]", + snapshot.QuiesceCause, + record.QuiesceCause, + )) + } + if !record.GeneratedAt.IsZero() && + snapshot.CapturedAt.After(record.GeneratedAt) { + violations = append( + violations, + "the node-authored quiescence gate snapshot was captured after the "+ + "quiescence report was generated", + ) + } + + inventoryPermits := make(map[quarantineIdentity]int) + for i, permit := range snapshot.ActivePermits { + identity := inventoryIdentity(permit) + if firstIndex, duplicate := inventoryPermits[identity]; duplicate { + violations = append(violations, fmt.Sprintf( + "gate inventory entry [%d] duplicates the full permit "+ + "identity first recorded by entry [%d]", + i, + firstIndex, + )) + } else { + inventoryPermits[identity] = i + } + } + + nodeOutcomes := make( + map[quarantineIdentity]participation.TerminalOutcomeRecord, + ) + if r.manifest.ParticipationTerminalOutcomes != nil { + for _, outcome := range r.manifest.ParticipationTerminalOutcomes.Outcomes { + nodeOutcomes[inventoryIdentity(outcome.Permit)] = outcome + } + } + + beaconQuarantined := make(map[quarantineIdentity]int) + for _, quarantined := range r.manifest.BeaconQuarantinedOutputs { + beaconQuarantined[quarantineIdentity{ + ceremony: quarantined.Ceremony, + mode: quarantined.ProtocolMode, + canonicalStartBlock: quarantined.CanonicalStartBlock, + workID: quarantined.SeedHash, + permitID: fmt.Sprint(quarantined.MemberIndex), + }]++ + } + tbtcQuarantined := make(map[quarantineIdentity]int) + for _, quarantined := range r.manifest.TBTCQuarantinedOutputs { + tbtcQuarantined[quarantineIdentity{ + ceremony: quarantined.Ceremony, + mode: quarantined.ProtocolMode, + canonicalStartBlock: quarantined.CanonicalStartBlock, + workID: quarantined.SeedHash, + permitID: fmt.Sprint(quarantined.MemberIndex), + }]++ + } + + seenPermits := make(map[quarantineIdentity]int) + var outcomeLegacy uint64 + var outcomeSecurityV2 uint64 + for i, permit := range record.ActivePermitsAtQuiescence { + if _, ok := knownCeremonies[permit.Ceremony]; !ok { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] names unknown ceremony [%s]", + i, + permit.Ceremony, + )) + } + if permit.Mode != participation.ModeLegacy.String() && + permit.Mode != participation.ModeSecurityV2.String() { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] names unknown protocol mode [%s]", + i, + permit.Mode, + )) + } else { + if permit.Mode == participation.ModeLegacy.String() { + outcomeLegacy++ + } else { + outcomeSecurityV2++ + } + if violation := cutoverModeViolation( + fmt.Sprintf("permit entry [%d]", i), + permit.Mode, + permit.CanonicalStartBlock, + record.CutoverBlock, + ); violation != "" { + violations = append(violations, violation) + } + } + if _, ok := validQuiescencePermitOutcomes[permit.Outcome]; !ok { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] has unknown terminal outcome [%s]", + i, + permit.Outcome, + )) + continue + } + + violations = append( + violations, + validateQuiescencePermitIdentity(i, permit)..., + ) + + identity := outcomeIdentity(permit) + if firstIndex, duplicate := seenPermits[identity]; duplicate { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] duplicates the full permit identity "+ + "first recorded by entry [%d] [ceremony=%s] [mode=%s] "+ + "[canonicalStartBlock=%d] [workID=%s] [permitID=%s]", + i, + firstIndex, + permit.Ceremony, + permit.Mode, + permit.CanonicalStartBlock, + permit.WorkID, + permit.PermitID, + )) + } else { + seenPermits[identity] = i + } + if _, inventoried := inventoryPermits[identity]; !inventoried { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] has no matching identity in the "+ + "at-quiescence gate inventory", + i, + )) + } + nodeOutcome, nodeAuthored := nodeOutcomes[identity] + if !nodeAuthored { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] has no matching node-authored terminal "+ + "outcome", + i, + )) + } else if string(nodeOutcome.Outcome) != permit.Outcome { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] claims terminal outcome [%s], but the "+ + "node-authored journal records [%s]", + i, + permit.Outcome, + nodeOutcome.Outcome, + )) + } + + if permit.Outcome != "quarantined" || !r.manifest.Interpreted { + continue + } + switch permit.Ceremony { + case string(participation.BeaconDKG): + if beaconQuarantined[identity] == 0 { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] claims a quarantined [%s] output "+ + "[mode=%s] [canonicalStartBlock=%d] [workID=%s] "+ + "[permitID=%s] but the beacon quarantine namespace "+ + "holds none matching that exact local permit", + i, + permit.Ceremony, + permit.Mode, + permit.CanonicalStartBlock, + permit.WorkID, + permit.PermitID, + )) + continue + } + beaconQuarantined[identity]-- + case string(participation.TBTCDKG): + if tbtcQuarantined[identity] == 0 { + violations = append(violations, fmt.Sprintf( + "permit entry [%d] claims a quarantined [%s] output "+ + "[mode=%s] [canonicalStartBlock=%d] [workID=%s] "+ + "[permitID=%s] but the tbtc quarantine namespace "+ + "holds none matching that exact local permit", + i, + permit.Ceremony, + permit.Mode, + permit.CanonicalStartBlock, + permit.WorkID, + permit.PermitID, + )) + continue + } + tbtcQuarantined[identity]-- + } + } + + if uint64(len(record.ActivePermitsAtQuiescence)) != + snapshot.ActiveCeremonies { + violations = append(violations, fmt.Sprintf( + "the terminal outcome list contains [%d] permits, but the "+ + "node-authored gate snapshot declares total [%d]", + len(record.ActivePermitsAtQuiescence), + snapshot.ActiveCeremonies, + )) + } + if outcomeLegacy != snapshot.ActiveLegacyCeremonies { + violations = append(violations, fmt.Sprintf( + "the terminal outcome list contains [%d] legacy permits, but "+ + "the node-authored gate snapshot declares [%d]", + outcomeLegacy, + snapshot.ActiveLegacyCeremonies, + )) + } + if outcomeSecurityV2 != snapshot.ActiveSecurityV2Ceremonies { + violations = append(violations, fmt.Sprintf( + "the terminal outcome list contains [%d] security-v2 permits, "+ + "but the node-authored gate snapshot declares [%d]", + outcomeSecurityV2, + snapshot.ActiveSecurityV2Ceremonies, + )) + } + for identity, inventoryIndex := range inventoryPermits { + if _, reported := seenPermits[identity]; !reported { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] has no terminal outcome "+ + "[ceremony=%s] [mode=%s] "+ + "[canonicalStartBlock=%d] [workID=%s] [permitID=%s]", + inventoryIndex, + identity.ceremony, + identity.mode, + identity.canonicalStartBlock, + identity.workID, + identity.permitID, + )) + } + } + + return violations +} + +// validatePriorReaderCompatibilityEvidence checks the prior-reader record: +// schema, snapshot binding, the exactly pinned prior and current release +// artifacts on both sides of the test, and an explicit compatible result for +// every schema this release writes — each schema at most once, so one result +// cannot be shadowed by a contradicting duplicate. Any missing or +// incompatible schema means the prior-binary rollback is not an accepted +// mechanism. +func (r *auditRun) validatePriorReaderCompatibilityEvidence( + content []byte, +) []string { + record := &priorReaderCompatibilityEvidence{} + if err := strictUnmarshal(content, record); err != nil { + return []string{fmt.Sprintf( + "cannot be decoded as a prior-reader compatibility record: [%v]", + err, + )} + } + + violations := r.validateEnvelope( + record.evidenceEnvelope, + "prior_reader_compatibility", + ) + + violations = append(violations, exactIdentityViolations( + "tested prior version", + record.PriorVersion, + r.expected.priorVersion, + )...) + violations = append(violations, exactIdentityViolations( + "tested prior revision", + record.PriorRevision, + r.expected.priorRevision, + )...) + violations = append(violations, digestViolations( + "tested prior image digest", + record.PriorImageDigest, + r.expected.priorImageDigest, + )...) + violations = append(violations, exactIdentityViolations( + "writing release version", + record.ReleaseVersion, + r.expected.releaseVersion, + )...) + violations = append(violations, exactIdentityViolations( + "writing release revision", + record.ReleaseRevision, + r.expected.releaseRevision, + )...) + violations = append(violations, digestViolations( + "writing release image digest", + record.ReleaseImageDigest, + r.expected.releaseImageDigest, + )...) + + results := make(map[string]bool) + for i, result := range record.SchemaResults { + if result.Schema == "" { + violations = append(violations, fmt.Sprintf( + "schema result entry [%d] is missing its schema name", + i, + )) + continue + } + if _, duplicate := results[result.Schema]; duplicate { + violations = append(violations, fmt.Sprintf( + "schema [%s] is covered more than once; duplicate results "+ + "cannot prove compatibility", + result.Schema, + )) + continue + } + results[result.Schema] = result.Compatible + } + for _, schema := range requiredPriorReaderSchemas { + compatible, covered := results[schema] + if !covered { + violations = append(violations, fmt.Sprintf( + "required schema [%s] is not covered", + schema, + )) + continue + } + if !compatible { + violations = append(violations, fmt.Sprintf( + "the prior release cannot read schema [%s]", + schema, + )) + } + } + + return violations +} + +// strictUnmarshal decodes JSON while rejecting unknown fields and trailing +// content, so a placeholder or mistyped record cannot pass as evidence. +func strictUnmarshal(content []byte, target interface{}) error { + decoder := json.NewDecoder(bytes.NewReader(content)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(target); err != nil { + return err + } + if decoder.More() { + return fmt.Errorf("trailing content after the record") + } + return nil +} + +// interpretKeyStoreNamespaces decodes the beacon active, beacon quarantine, +// and tBTC active namespaces through the standard encrypted persistence +// handles, cross-validates every record against its storage location and its +// paired records, and reports every failure as a finding. +func interpretKeyStoreNamespaces( + storageDir string, + password string, + run *auditRun, +) error { + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + password, + ) + if err != nil { + return fmt.Errorf("cannot open the storage snapshot: [%w]", err) + } + + activeGroups, err := interpretBeaconActiveNamespace(diskStorage, run) + if err != nil { + return err + } + if err := interpretBeaconQuarantineNamespace( + diskStorage, + run, + activeGroups, + ); err != nil { + return err + } + activeWallets, err := interpretTBTCActiveNamespace(diskStorage, run) + if err != nil { + return err + } + if err := interpretTBTCQuarantineNamespace( + diskStorage, + run, + activeWallets, + ); err != nil { + return err + } + if err := interpretParticipationQuiescenceSnapshot( + diskStorage, + run, + ); err != nil { + return err + } + + sortRecords(run.manifest) + + return nil +} + +// interpretParticipationQuiescenceSnapshot reads the node-authored gate +// capture from encrypted work storage. This is the authoritative active-permit +// inventory. Its companion terminal-outcome journal supplies the authoritative +// dispositions; the external quiescence report can only corroborate them and +// cannot replace or shorten either record. +func interpretParticipationQuiescenceSnapshot( + diskStorage storage.Storage, + run *auditRun, +) error { + handle, err := diskStorage.InitializeWorkPersistence("participation") + if err != nil { + return fmt.Errorf( + "cannot open the participation work namespace: [%w]", + err, + ) + } + + descriptors, descriptorErrors := handle.ReadAll() + errorsDone := make(chan struct{}) + go func() { + defer close(errorsDone) + for err := range descriptorErrors { + run.finding( + "participation work namespace read error: [%v]", + err, + ) + } + }() + + snapshotCount := 0 + journalCount := 0 + for descriptor := range descriptors { + if descriptor.Directory() != + participation.QuiescenceSnapshotStorageDirectory { + run.finding( + "participation work record [%s/%s] is not in the recognized "+ + "node-authored quiescence directory", + descriptor.Directory(), + descriptor.Name(), + ) + continue + } + + content, err := descriptor.Content() + if err != nil { + run.finding( + "node-authored quiescence snapshot cannot be decrypted: [%v]", + err, + ) + continue + } + + switch descriptor.Name() { + case participation.QuiescenceSnapshotStorageFile: + snapshotCount++ + snapshot := &participation.QuiescenceSnapshot{} + if err := strictUnmarshal(content, snapshot); err != nil { + run.finding( + "node-authored quiescence snapshot cannot be decoded: [%v]", + err, + ) + continue + } + if run.manifest.QuiescenceSnapshot != nil { + run.finding( + "more than one node-authored quiescence snapshot is present", + ) + continue + } + + run.manifest.QuiescenceSnapshot = snapshot + for _, violation := range validateNodeQuiescenceSnapshot(snapshot) { + run.finding("%s", violation) + } + if snapshot.CapturedAt.After(run.manifest.GeneratedAt) { + run.finding( + "node-authored quiescence snapshot capture time is after " + + "the offline audit time", + ) + } else if run.expected.maxEvidenceAge > 0 && + run.manifest.GeneratedAt.Sub(snapshot.CapturedAt) > + run.expected.maxEvidenceAge { + run.finding( + "node-authored quiescence snapshot is older than the "+ + "maximum evidence age [%s]", + run.expected.maxEvidenceAge, + ) + } + case participation.TerminalOutcomeJournalStorageFile: + journalCount++ + journal := &participation.TerminalOutcomeJournal{} + if err := strictUnmarshal(content, journal); err != nil { + run.finding( + "node-authored terminal-outcome journal cannot be "+ + "decoded: [%v]", + err, + ) + continue + } + if run.manifest.ParticipationTerminalOutcomes != nil { + run.finding( + "more than one node-authored terminal-outcome journal " + + "is present", + ) + continue + } + run.manifest.ParticipationTerminalOutcomes = journal + default: + run.finding( + "participation work record [%s/%s] is not a recognized "+ + "node-authored quiescence artifact", + descriptor.Directory(), + descriptor.Name(), + ) + } + } + <-errorsDone + + if snapshotCount == 0 { + run.finding( + "the node-authored participation quiescence snapshot is missing", + ) + } + if journalCount == 0 { + run.finding( + "the node-authored participation terminal-outcome journal is missing", + ) + } + for _, violation := range validateNodeTerminalOutcomes(run.manifest) { + run.finding("%s", violation) + } + + return nil +} + +func validateNodeQuiescenceSnapshot( + snapshot *participation.QuiescenceSnapshot, +) []string { + violations := make([]string, 0) + + if snapshot.SchemaVersion != participation.QuiescenceSnapshotSchemaVersion { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence snapshot schema [%d] is not [%d]", + snapshot.SchemaVersion, + participation.QuiescenceSnapshotSchemaVersion, + )) + } + if snapshot.CapturedAt.IsZero() { + violations = append( + violations, + "node-authored quiescence snapshot capture time is missing", + ) + } + if snapshot.ReleaseVersion == "" { + violations = append( + violations, + "node-authored quiescence snapshot release version is missing", + ) + } + if snapshot.ReleaseRevision == "" { + violations = append( + violations, + "node-authored quiescence snapshot release revision is missing", + ) + } + if snapshot.ReleaseEpoch != participation.CompiledEpoch.String() { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence snapshot epoch [%s] is not the "+ + "compiled epoch [%s]", + snapshot.ReleaseEpoch, + participation.CompiledEpoch, + )) + } + if snapshot.CutoverBlock == 0 { + violations = append( + violations, + "node-authored quiescence snapshot cutover block is zero", + ) + } + if snapshot.State != participation.StateQuiescing.String() { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence snapshot state [%s] is not [%s]", + snapshot.State, + participation.StateQuiescing, + )) + } + if snapshot.QuiesceCause == "" { + violations = append( + violations, + "node-authored quiescence snapshot cause is missing", + ) + } + + if snapshot.ActiveLegacyCeremonies > + snapshot.ActiveCeremonies || + snapshot.ActiveSecurityV2Ceremonies > + snapshot.ActiveCeremonies- + snapshot.ActiveLegacyCeremonies || + snapshot.ActiveLegacyCeremonies+ + snapshot.ActiveSecurityV2Ceremonies != + snapshot.ActiveCeremonies { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence snapshot mode counts [%d legacy, "+ + "%d security-v2] do not sum to total [%d]", + snapshot.ActiveLegacyCeremonies, + snapshot.ActiveSecurityV2Ceremonies, + snapshot.ActiveCeremonies, + )) + } + if snapshot.ActiveCeremonies != uint64(len(snapshot.ActivePermits)) { + violations = append(violations, fmt.Sprintf( + "node-authored quiescence snapshot inventories [%d] permits, "+ + "but declares total [%d]", + len(snapshot.ActivePermits), + snapshot.ActiveCeremonies, + )) + } + + knownCeremonies := make(map[string]struct{}) + for _, ceremony := range participation.AllCeremonies() { + knownCeremonies[string(ceremony)] = struct{}{} + } + + identities := make(map[quarantineIdentity]int) + var legacy uint64 + var securityV2 uint64 + for i, permit := range snapshot.ActivePermits { + if _, ok := knownCeremonies[string(permit.Ceremony)]; !ok { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] names unknown "+ + "ceremony [%s]", + i, + permit.Ceremony, + )) + } + switch permit.Mode { + case participation.ModeLegacy.String(): + legacy++ + case participation.ModeSecurityV2.String(): + securityV2++ + default: + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] names unknown "+ + "protocol mode [%s]", + i, + permit.Mode, + )) + } + if !permit.IdentityBound { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] was issued "+ + "without a stable work and permit identity", + i, + )) + } + if violation := cutoverModeViolation( + fmt.Sprintf("node-authored gate inventory entry [%d]", i), + permit.Mode, + permit.CanonicalStartBlock, + snapshot.CutoverBlock, + ); violation != "" { + violations = append(violations, violation) + } + violations = append( + violations, + validateQuiescencePermitIdentity( + i, + quiescencePermitEvidence{ + Ceremony: string(permit.Ceremony), + Mode: permit.Mode, + CanonicalStartBlock: permit.CanonicalStartBlock, + WorkID: permit.WorkID, + PermitID: permit.PermitID, + }, + )..., + ) + // The issuance-time side of the seat ownership statement, checked + // against the shape its own ceremony can have. The audit reads it out of + // a stopped node's storage rather than watching it being issued, so + // nothing has held it to that shape yet. + if err := participation.ValidatePermitOperatedShape( + permit.Ceremony, + permit.PermitID, + permit.OperatedMembers, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] claims operated "+ + "memberships its ceremony cannot have: [%v]", + i, + err, + )) + } + + identity := inventoryIdentity(permit) + if firstIndex, duplicate := identities[identity]; duplicate { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] duplicates the "+ + "full permit identity first recorded by entry [%d]", + i, + firstIndex, + )) + } else { + identities[identity] = i + } + + if i > 0 { + previous := snapshot.ActivePermits[i-1] + if permitSnapshotLess(permit, previous) { + violations = append( + violations, + "node-authored gate inventory is not deterministically sorted", + ) + } + } + } + + if legacy != snapshot.ActiveLegacyCeremonies { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory contains [%d] legacy permits, "+ + "but declares [%d]", + legacy, + snapshot.ActiveLegacyCeremonies, + )) + } + if securityV2 != snapshot.ActiveSecurityV2Ceremonies { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory contains [%d] security-v2 "+ + "permits, but declares [%d]", + securityV2, + snapshot.ActiveSecurityV2Ceremonies, + )) + } + + return violations +} + +// validateNodeTerminalOutcomes reconciles the ceremony-owner-authored terminal +// journal with the immutable permit inventory captured by the gate. It also +// corroborates DKG completion against persisted signer state and quarantine +// against the protected signer namespaces. An external quiescence report is +// deliberately not consulted here. +func validateNodeTerminalOutcomes( + auditManifest *manifest, +) []string { + journal := auditManifest.ParticipationTerminalOutcomes + if journal == nil { + return nil + } + + violations := make([]string, 0) + if journal.SchemaVersion != + participation.TerminalOutcomeJournalSchemaVersion { + violations = append(violations, fmt.Sprintf( + "node-authored terminal-outcome journal schema [%d] is not [%d]", + journal.SchemaVersion, + participation.TerminalOutcomeJournalSchemaVersion, + )) + } + + snapshot := auditManifest.QuiescenceSnapshot + if snapshot == nil { + violations = append( + violations, + "node-authored terminal-outcome journal has no gate snapshot to bind", + ) + return violations + } + if !journal.SnapshotCapturedAt.Equal(snapshot.CapturedAt) { + violations = append(violations, fmt.Sprintf( + "node-authored terminal-outcome journal binds snapshot time [%s], "+ + "but the gate snapshot was captured at [%s]", + journal.SnapshotCapturedAt, + snapshot.CapturedAt, + )) + } + + inventory := make(map[quarantineIdentity]int) + for i, permit := range snapshot.ActivePermits { + inventory[inventoryIdentity(permit)] = i + } + + activeTBTCSigners := make(map[persistedSignerIdentity]struct{}) + for _, wallet := range auditManifest.TBTCActiveWallets { + for _, memberIndex := range wallet.MemberIndexes { + activeTBTCSigners[persistedSignerIdentity{ + reference: wallet.WalletStorageKey, + membershipIndex: group.MemberIndex(memberIndex), + }] = struct{}{} + } + } + activeBeaconSigners := make(map[persistedSignerIdentity]struct{}) + beaconGroupKeys := make(map[string]struct{}) + for _, membership := range auditManifest.BeaconActiveMemberships { + activeBeaconSigners[persistedSignerIdentity{ + reference: membership.GroupPublicKey, + membershipIndex: group.MemberIndex(membership.MemberIndex), + }] = struct{}{} + beaconGroupKeys[membership.GroupPublicKey] = struct{}{} + } + // A relay entry can be signed by a group whose signer the rollback later + // quarantined; quarantining the key material does not unmake the entry the + // group already produced. + for _, quarantined := range auditManifest.BeaconQuarantinedOutputs { + if quarantined.GroupPublicKey == "" { + continue + } + beaconGroupKeys[quarantined.GroupPublicKey] = struct{}{} + } + claimedTBTCSigners := make(map[persistedSignerIdentity]int) + claimedBeaconSigners := make(map[persistedSignerIdentity]int) + // One relay entry answers one relay request, so the journal-wide record of + // which request each entry was already used for is what catches an entry + // replayed onto a second request. + claimedRelayEntries := make(map[string]relayEntryClaim) + // A beacon terminates one request once, so the journal-wide record of which + // request each settlement was already used for is what catches a real + // penalty replayed onto a second request. + claimedTimeoutSettlements := make(map[string]relayTimeoutSettlementClaim) + tbtcQuarantined := make(map[quarantineIdentity]struct{}) + for _, quarantined := range auditManifest.TBTCQuarantinedOutputs { + tbtcQuarantined[quarantineIdentity{ + ceremony: quarantined.Ceremony, + mode: quarantined.ProtocolMode, + canonicalStartBlock: quarantined.CanonicalStartBlock, + workID: quarantined.SeedHash, + permitID: fmt.Sprint(quarantined.MemberIndex), + }] = struct{}{} + } + beaconQuarantined := make(map[quarantineIdentity]struct{}) + for _, quarantined := range auditManifest.BeaconQuarantinedOutputs { + beaconQuarantined[quarantineIdentity{ + ceremony: quarantined.Ceremony, + mode: quarantined.ProtocolMode, + canonicalStartBlock: quarantined.CanonicalStartBlock, + workID: quarantined.SeedHash, + permitID: fmt.Sprint(quarantined.MemberIndex), + }] = struct{}{} + } + + seen := make(map[quarantineIdentity]int) + for i, outcome := range journal.Outcomes { + identity := inventoryIdentity(outcome.Permit) + if firstIndex, duplicate := seen[identity]; duplicate { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] duplicates the full "+ + "permit identity first recorded by outcome [%d]", + i, + firstIndex, + )) + } else { + seen[identity] = i + } + if inventoryIndex, inventoried := inventory[identity]; !inventoried { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] has no matching permit "+ + "in the at-quiescence gate inventory", + i, + )) + } else if issued := snapshot.ActivePermits[inventoryIndex]; !slices.Equal( + outcome.Permit.OperatedMembers, + issued.OperatedMembers, + ) { + // The two sides of the seat ownership statement: the seats the + // snapshot recorded this permit holding while it was live, and the + // seats the journal record for the same permit carries. Both are + // copies of one set fixed at issuance, so a disagreement is an edit + // to whichever of them a reader is about to build an ownership map + // from, and there is no rule for choosing between them that is not a + // guess. + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] claims operated "+ + "memberships %v, but the at-quiescence gate inventory "+ + "issued the same permit %v", + i, + outcome.Permit.OperatedMembers, + issued.OperatedMembers, + )) + } + // The shape the ceremony can have, reapplied to the journal's own copy. + // An inventory entry it agrees with is not enough on its own: the same + // edit made to both sides would pass the comparison above. + if err := participation.ValidatePermitOperatedShape( + outcome.Permit.Ceremony, + outcome.Permit.PermitID, + outcome.Permit.OperatedMembers, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] claims operated "+ + "memberships its ceremony cannot have: [%v]", + i, + err, + )) + } + if outcome.RecordedAt.IsZero() { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] has no record time", + i, + )) + } + if !outcome.Permit.IdentityBound { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] belongs to an unbound "+ + "permit identity", + i, + )) + } + violations = append( + violations, + validateQuiescencePermitIdentity( + i, + quiescencePermitEvidence{ + Ceremony: string(outcome.Permit.Ceremony), + Mode: outcome.Permit.Mode, + CanonicalStartBlock: outcome.Permit.CanonicalStartBlock, + WorkID: outcome.Permit.WorkID, + PermitID: outcome.Permit.PermitID, + }, + )..., + ) + if outcome.Outcome != "unresolved" { + if err := participation.ValidateTerminalOutcome( + outcome.Permit.Ceremony, + outcome.Permit.WorkID, + outcome.Outcome, + outcome.Evidence, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] evidence is invalid: [%v]", + i, + err, + )) + } + // The same binding the live gate applies at record time, reapplied + // to the journal it wrote. A record that reached storage with a + // transcript seat outside its permit's operated set was either + // written by a gate this audit cannot vouch for or edited + // afterwards, and both are reasons to refuse it rather than to read + // its seats into an ownership map. + if err := participation.ValidatePermitOperatedOwnership( + outcome.Permit.Ceremony, + outcome.Permit.OperatedMembers, + outcome.Outcome, + outcome.Evidence, + ); err != nil { + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] claims a membership "+ + "outside its permit: [%v]", + i, + err, + )) + } + } + + switch outcome.Outcome { + case participation.TerminalOutcomeCompleted: + switch outcome.Permit.Ceremony { + case participation.TBTCDKG: + signerIdentity := persistedSignerIdentity{ + reference: outcome.Evidence.Reference, + membershipIndex: outcome.Evidence.MembershipIndex, + } + if outcome.Evidence.Kind != + participation.TerminalEvidencePersistedTBTCSinger { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] does "+ + "not name persisted tbtc signer evidence", + i, + )) + } else if _, ok := activeTBTCSigners[signerIdentity]; !ok { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcome [%d] names "+ + "persisted signer [%s] membership [%d], but the "+ + "active tbtc namespace holds no matching signer", + i, + outcome.Evidence.Reference, + outcome.Evidence.MembershipIndex, + )) + } else if firstOutcome, duplicate := + claimedTBTCSigners[signerIdentity]; duplicate { + violations = append(violations, fmt.Sprintf( + "node-authored completed tbtc DKG outcomes [%d] and "+ + "[%d] claim the same persisted signer [%s] "+ + "membership [%d]", + firstOutcome, + i, + outcome.Evidence.Reference, + outcome.Evidence.MembershipIndex, + )) + } else { + claimedTBTCSigners[signerIdentity] = i + } + case participation.BeaconDKG: + signerIdentity := persistedSignerIdentity{ + reference: outcome.Evidence.Reference, + membershipIndex: outcome.Evidence.MembershipIndex, + } + if outcome.Evidence.Kind != + participation.TerminalEvidencePersistedBeaconSigner { + violations = append(violations, fmt.Sprintf( + "node-authored completed beacon DKG outcome [%d] does "+ + "not name persisted beacon signer evidence", + i, + )) + } else if _, ok := activeBeaconSigners[signerIdentity]; !ok { + violations = append(violations, fmt.Sprintf( + "node-authored completed beacon DKG outcome [%d] names "+ + "persisted signer [%s] membership [%d], but the "+ + "active beacon namespace holds no matching signer", + i, + outcome.Evidence.Reference, + outcome.Evidence.MembershipIndex, + )) + } else if firstOutcome, duplicate := + claimedBeaconSigners[signerIdentity]; duplicate { + violations = append(violations, fmt.Sprintf( + "node-authored completed beacon DKG outcomes [%d] and "+ + "[%d] claim the same persisted signer [%s] "+ + "membership [%d]", + firstOutcome, + i, + outcome.Evidence.Reference, + outcome.Evidence.MembershipIndex, + )) + } else { + claimedBeaconSigners[signerIdentity] = i + } + permitMemberIndex, err := strconv.ParseUint( + outcome.Permit.PermitID, + 10, + 8, + ) + if err == nil && + outcome.Evidence.MembershipIndex != + group.MemberIndex(permitMemberIndex) { + violations = append(violations, fmt.Sprintf( + "node-authored completed beacon DKG outcome [%d] "+ + "belongs to permit member [%s], but names "+ + "persisted membership [%d]", + i, + outcome.Permit.PermitID, + outcome.Evidence.MembershipIndex, + )) + } + case participation.BeaconRelaySigning: + violations = append( + violations, + validateRelayEntryTerminalResult( + i, + outcome.Permit.WorkID, + outcome.Evidence.Reference, + beaconGroupKeys, + claimedRelayEntries, + )..., + ) + case participation.BeaconTimeoutReport: + violations = append( + violations, + validateRelayTimeoutSettlement( + i, + outcome.Permit.WorkID, + outcome.Evidence.Reference, + claimedTimeoutSettlements, + )..., + ) + default: + if outcome.Evidence.Kind == + participation.TerminalEvidenceNoThreshold || + outcome.Evidence.Kind == "" { + violations = append(violations, fmt.Sprintf( + "node-authored completed outcome [%d] has no durable "+ + "result evidence", + i, + )) + } + } + if outcome.Evidence.Reference != "" && + !isStableEvidenceID(outcome.Evidence.Reference) { + violations = append(violations, fmt.Sprintf( + "node-authored completed outcome [%d] evidence reference "+ + "[%s] is not a stable evidence identifier", + i, + outcome.Evidence.Reference, + )) + } + case participation.TerminalOutcomeQuarantined: + switch outcome.Permit.Ceremony { + case participation.TBTCDKG: + if outcome.Evidence.Kind != + participation.TerminalEvidenceQuarantinedTBTCSinger { + violations = append(violations, fmt.Sprintf( + "node-authored quarantined tbtc DKG outcome [%d] has "+ + "the wrong evidence kind [%s]", + i, + outcome.Evidence.Kind, + )) + } + if _, ok := tbtcQuarantined[identity]; !ok { + violations = append(violations, fmt.Sprintf( + "node-authored quarantined tbtc DKG outcome [%d] has "+ + "no exact protected signer record", + i, + )) + } + case participation.BeaconDKG: + if outcome.Evidence.Kind != + participation.TerminalEvidenceQuarantinedBeaconSigner { + violations = append(violations, fmt.Sprintf( + "node-authored quarantined beacon DKG outcome [%d] has "+ + "the wrong evidence kind [%s]", + i, + outcome.Evidence.Kind, + )) + } + if _, ok := beaconQuarantined[identity]; !ok { + violations = append(violations, fmt.Sprintf( + "node-authored quarantined beacon DKG outcome [%d] has "+ + "no exact protected signer record", + i, + )) + } + default: + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] claims quarantine "+ + "for ceremony [%s], which has no protected signer "+ + "namespace", + i, + outcome.Permit.Ceremony, + )) + } + case participation.TerminalOutcomeExhausted: + if outcome.Evidence.Kind != + participation.TerminalEvidenceNoThreshold || + outcome.Evidence.Reference != "" { + violations = append(violations, fmt.Sprintf( + "node-authored exhausted outcome [%d] is not an explicit "+ + "no-threshold result", + i, + )) + } + default: + violations = append(violations, fmt.Sprintf( + "node-authored terminal outcome [%d] is unresolved or unknown "+ + "[%s]", + i, + outcome.Outcome, + )) + } + + if i > 0 && + permitSnapshotLess( + outcome.Permit, + journal.Outcomes[i-1].Permit, + ) { + violations = append( + violations, + "node-authored terminal-outcome journal is not "+ + "deterministically sorted", + ) + } + } + + if len(journal.Outcomes) != len(snapshot.ActivePermits) { + violations = append(violations, fmt.Sprintf( + "node-authored terminal-outcome journal contains [%d] outcomes, "+ + "but the gate snapshot inventories [%d] permits", + len(journal.Outcomes), + len(snapshot.ActivePermits), + )) + } + for identity, inventoryIndex := range inventory { + if _, recorded := seen[identity]; !recorded { + violations = append(violations, fmt.Sprintf( + "node-authored gate inventory entry [%d] has no node-authored "+ + "terminal outcome [ceremony=%s] [mode=%s] "+ + "[canonicalStartBlock=%d] [workID=%s] [permitID=%s]", + inventoryIndex, + identity.ceremony, + identity.mode, + identity.canonicalStartBlock, + identity.workID, + identity.permitID, + )) + } + } + + return violations +} + +func permitSnapshotLess( + left participation.PermitSnapshot, + right participation.PermitSnapshot, +) bool { + if left.Ceremony != right.Ceremony { + return left.Ceremony < right.Ceremony + } + if left.CanonicalStartBlock != right.CanonicalStartBlock { + return left.CanonicalStartBlock < right.CanonicalStartBlock + } + if left.WorkID != right.WorkID { + return left.WorkID < right.WorkID + } + return left.PermitID < right.PermitID +} + +// interpretBeaconActiveNamespace decodes every active-namespace record as a +// membership — exactly what the client's own active-group scan assumes on +// start — and cross-checks each record against the directory and file name it +// is stored under. It returns the set of active group public keys for the +// quarantine overlap check. +func interpretBeaconActiveNamespace( + diskStorage storage.Storage, + run *auditRun, +) (map[string]struct{}, error) { + activeHandle, err := diskStorage.InitializeKeyStorePersistence("beacon") + if err != nil { + return nil, fmt.Errorf( + "cannot open the beacon keystore namespace: [%w]", + err, + ) + } + + activeGroups := make(map[string]struct{}) + + activeData, activeErrors := activeHandle.ReadAll() + activeDone := make(chan struct{}) + go func() { + defer close(activeDone) + for err := range activeErrors { + run.finding("beacon active namespace read error: [%v]", err) + } + }() + for descriptor := range activeData { + content, err := descriptor.Content() + if err != nil { + run.finding( + "beacon active record [%s/%s] cannot be decrypted: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + + membership := ®istry.Membership{} + if err := membership.Unmarshal(content); err != nil { + run.finding( + "beacon active record [%s/%s] cannot be decoded as a "+ + "membership: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + + groupPublicKey := hex.EncodeToString( + membership.Signer.GroupPublicKeyBytesCompressed(), + ) + memberIndex := uint8(membership.Signer.MemberID()) + + // The client's active scan trusts the storage location; a record + // whose content disagrees with its directory or member file name + // belongs to a different group or member than the layout claims. + if descriptor.Directory() != groupPublicKey { + run.finding( + "beacon active record [%s/%s] contains group [%s], not the "+ + "group its directory claims", + descriptor.Directory(), + descriptor.Name(), + groupPublicKey, + ) + } + if expected := fmt.Sprintf( + "membership_%d", + memberIndex, + ); descriptor.Name() != expected { + run.finding( + "beacon active record [%s/%s] contains member [%d], not the "+ + "member its file name claims", + descriptor.Directory(), + descriptor.Name(), + memberIndex, + ) + } + + activeGroups[groupPublicKey] = struct{}{} + run.manifest.BeaconActiveMemberships = append( + run.manifest.BeaconActiveMemberships, + beaconMembershipRecord{ + GroupPublicKey: groupPublicKey, + MemberIndex: memberIndex, + ChannelName: membership.ChannelName, + }, + ) + } + <-activeDone + + return activeGroups, nil +} + +// beaconQuarantineEntry pairs the two halves of one quarantined output while +// the namespace is scanned. +type beaconQuarantineEntry struct { + directory string + memberSuffix string + metadata *registry.QuarantinedSignerMetadata + membership *registry.Membership + // handoffMetadata and handoffMembership are the halves carried by the + // combined record preservation writes when the namespace would not take the + // pair. They stand in for whichever half the pair is missing, so an output + // preserved that way is as complete a piece of evidence as a paired one. + handoffMetadata *registry.QuarantinedSignerMetadata + handoffMembership *registry.Membership + // membershipBytes and handoffMembershipBytes are the key material as each + // form stored it. The two forms encode the membership identically, so when + // both name the same seat the stored bytes are what says whether they hold + // the same share — a question the decoded values cannot be asked, since a + // membership carries private scalars this audit must not compare field by + // field or report. + membershipBytes []byte + handoffMembershipBytes []byte +} + +// interpretBeaconQuarantineNamespace decodes the quarantine namespace, pairs +// metadata and membership halves by directory and member suffix, and +// cross-validates the metadata against its schema, this release's identity, +// the cutover arithmetic, the storage location, the decrypted membership, and +// the active namespace. +func interpretBeaconQuarantineNamespace( + diskStorage storage.Storage, + run *auditRun, + activeGroups map[string]struct{}, +) error { + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + return fmt.Errorf( + "cannot open the beacon quarantine namespace: [%w]", + err, + ) + } + + quarantineEntries := make(map[string]*beaconQuarantineEntry) + entryFor := func(directory, name, prefix string) *beaconQuarantineEntry { + suffix := strings.TrimPrefix(name, prefix) + key := directory + "/" + suffix + if _, ok := quarantineEntries[key]; !ok { + quarantineEntries[key] = &beaconQuarantineEntry{ + directory: directory, + memberSuffix: suffix, + } + } + return quarantineEntries[key] + } + + quarantineData, quarantineErrors := quarantineHandle.ReadAll() + quarantineDone := make(chan struct{}) + go func() { + defer close(quarantineDone) + for err := range quarantineErrors { + run.finding("beacon quarantine namespace read error: [%v]", err) + } + }() + for descriptor := range quarantineData { + content, err := descriptor.Content() + if err != nil { + run.finding( + "beacon quarantine record [%s/%s] cannot be decrypted: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + + switch { + case strings.HasPrefix(descriptor.Name(), "metadata_"): + metadata := ®istry.QuarantinedSignerMetadata{} + if err := json.Unmarshal(content, metadata); err != nil { + run.finding( + "beacon quarantine metadata [%s/%s] cannot be decoded: "+ + "[%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + entryFor( + descriptor.Directory(), + descriptor.Name(), + "metadata_", + ).metadata = metadata + case strings.HasPrefix(descriptor.Name(), "membership_"): + membership := ®istry.Membership{} + if err := membership.Unmarshal(content); err != nil { + run.finding( + "beacon quarantine membership [%s/%s] cannot be decoded: "+ + "[%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + beaconEntry := entryFor( + descriptor.Directory(), + descriptor.Name(), + "membership_", + ) + beaconEntry.membership = membership + beaconEntry.membershipBytes = content + case strings.HasPrefix(descriptor.Name(), "handoff_"): + handoff, err := registry.DecodeQuarantinedSignerHandoff(content) + if err != nil { + run.finding( + "beacon quarantine handoff [%s/%s] cannot be decoded: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + membership := ®istry.Membership{} + if err := membership.Unmarshal(handoff.Membership); err != nil { + run.finding( + "beacon quarantine handoff [%s/%s] carries key material "+ + "that cannot be decoded: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + entry := entryFor( + descriptor.Directory(), + descriptor.Name(), + "handoff_", + ) + metadata := handoff.Metadata + entry.handoffMetadata = &metadata + entry.handoffMembership = membership + entry.handoffMembershipBytes = handoff.Membership + default: + run.finding( + "beacon quarantine record [%s/%s] has an unknown name", + descriptor.Directory(), + descriptor.Name(), + ) + } + } + <-quarantineDone + + keys := make([]string, 0, len(quarantineEntries)) + for key := range quarantineEntries { + keys = append(keys, key) + } + sort.Strings(keys) + + for _, key := range keys { + entry := quarantineEntries[key] + + // The combined record stands in for whichever half of the pair the + // namespace would not take. It is resolved before validation so the + // audit reports what the namespace actually holds of an output rather + // than which of the two layouts it was written in. + hasHandoff := entry.handoffMembership != nil + reconcileBeaconQuarantineForms(run, entry) + + validateQuarantineEntry(run, entry, activeGroups) + + if entry.metadata == nil { + continue + } + run.manifest.BeaconQuarantinedOutputs = append( + run.manifest.BeaconQuarantinedOutputs, + beaconQuarantineRecord{ + QuarantinedSignerMetadata: *entry.metadata, + HasMembershipRecord: entry.membership != nil, + HasHandoffRecord: hasHandoff, + }, + ) + } + + return nil +} + +// reconcileBeaconQuarantineForms settles one output the namespace holds in both +// preserved forms and fills whichever half the pair is missing from the handoff. +// +// Preservation writes the pair first and falls back on the combined record for +// what the namespace would not take, so the two forms overlap by design: a run +// that got the membership down, was refused the metadata, and then wrote the +// handoff leaves a standalone half beside a handoff carrying its own copy of +// that same half. Reading past the duplicate is not free. The forms are only +// interchangeable while they agree, and preferring whichever one this scan +// happened to decode first lets a stale or half-overwritten standalone record +// stand in for a complete handoff that contradicts it — the audit answering +// with evidence it never checked. +// +// So a duplicate is compared rather than deduplicated on sight, and a +// disagreement is reported instead of resolved. Which copy is the true one is +// not a question this tool can answer offline: it is what an operator has to +// settle before a rollback trusts either. +func reconcileBeaconQuarantineForms( + run *auditRun, + entry *beaconQuarantineEntry, +) { + if entry.metadata != nil && entry.handoffMetadata != nil { + same, err := sameQuarantineDocument(entry.metadata, entry.handoffMetadata) + switch { + case err != nil: + run.finding( + "beacon quarantined output [%s/%s] carries audit metadata in "+ + "both preserved forms and they cannot be compared: [%v]", + entry.directory, + entry.memberSuffix, + err, + ) + case !same: + run.finding( + "beacon quarantined output [%s/%s] carries audit metadata in "+ + "both preserved forms and they disagree; the standalone "+ + "record and the handoff describe the same seat differently, "+ + "so neither can be used as evidence until an operator "+ + "establishes which one the namespace should keep", + entry.directory, + entry.memberSuffix, + ) + } + } + if entry.metadata == nil { + entry.metadata = entry.handoffMetadata + } + + if entry.membership != nil && entry.handoffMembership != nil && + !bytes.Equal(entry.membershipBytes, entry.handoffMembershipBytes) { + run.finding( + "beacon quarantined output [%s/%s] carries key material in both "+ + "preserved forms and the two copies differ; a rollback cannot "+ + "tell which share this seat holds until an operator establishes "+ + "which record the namespace should keep", + entry.directory, + entry.memberSuffix, + ) + } + if entry.membership == nil { + entry.membership = entry.handoffMembership + } +} + +// sameQuarantineDocument reports whether two decoded quarantine documents carry +// the same values. +// +// They are compared through one encoder rather than field by field or as stored +// bytes: the standalone record and the handoff's copy travel to disk by +// different routes, so equal documents need not be equal bytes, and a +// field-by-field comparison would silently stop covering any field a later +// schema adds. +func sameQuarantineDocument(left, right interface{}) (bool, error) { + leftBytes, err := json.Marshal(left) + if err != nil { + return false, err + } + + rightBytes, err := json.Marshal(right) + if err != nil { + return false, err + } + + return bytes.Equal(leftBytes, rightBytes), nil +} + +// validateQuarantineEntry cross-validates one paired quarantine output. The +// metadata exists for the offline audit alone, so any half or field that +// contradicts the rest of the record makes the output untrustworthy evidence. +func validateQuarantineEntry( + run *auditRun, + entry *beaconQuarantineEntry, + activeGroups map[string]struct{}, +) { + key := entry.directory + "/" + entry.memberSuffix + + // A quarantined group visible in the active namespace is exactly the + // ambiguity the quarantine exists to prevent: the same key material would + // be both activated and marked interrupted. + if _, active := activeGroups[entry.directory]; active { + run.finding( + "beacon quarantine output [%s] belongs to group [%s] that is "+ + "also present in the active namespace", + key, + entry.directory, + ) + } + + if entry.membership != nil { + membershipGroup := hex.EncodeToString( + entry.membership.Signer.GroupPublicKeyBytesCompressed(), + ) + if membershipGroup != entry.directory { + run.finding( + "beacon quarantine membership [%s] contains group [%s], not "+ + "the group its directory claims", + key, + membershipGroup, + ) + } + if suffix := fmt.Sprint( + entry.membership.Signer.MemberID(), + ); suffix != entry.memberSuffix { + run.finding( + "beacon quarantine membership [%s] contains member [%s], "+ + "not the member its file name claims", + key, + suffix, + ) + } + } + + if entry.metadata == nil { + run.finding( + "beacon quarantine output [%s] has a membership record "+ + "without audit metadata", + key, + ) + return + } + + metadata := entry.metadata + if entry.membership == nil { + run.finding( + "beacon quarantine output [%s] has audit metadata without "+ + "a membership record; the key material was not preserved", + key, + ) + } + + if metadata.SchemaVersion != registry.QuarantineSchemaVersion { + run.finding( + "beacon quarantine metadata [%s] has schema version [%d], "+ + "expected [%d]", + key, + metadata.SchemaVersion, + registry.QuarantineSchemaVersion, + ) + } + if metadata.ReleaseEpoch != participation.CompiledEpoch.String() { + run.finding( + "beacon quarantine metadata [%s] was written by release epoch "+ + "[%s], not by this audit's epoch [%s]", + key, + metadata.ReleaseEpoch, + participation.CompiledEpoch, + ) + } + if metadata.Ceremony != string(participation.BeaconDKG) { + run.finding( + "beacon quarantine metadata [%s] names ceremony [%s]; only "+ + "[%s] outputs are quarantined", + key, + metadata.Ceremony, + participation.BeaconDKG, + ) + } + if !isCanonicalSHA256Hex(metadata.SeedHash) { + run.finding( + "beacon quarantine metadata [%s] seed hash [%s] is not a "+ + "canonical SHA-256 digest of 64 lowercase hexadecimal "+ + "characters", + key, + metadata.SeedHash, + ) + } + if metadata.MemberIndex == 0 { + run.finding( + "beacon quarantine metadata [%s] names invalid member index [0]", + key, + ) + } + if metadata.GroupPublicKey != entry.directory { + run.finding( + "beacon quarantine metadata [%s] names group [%s], not the "+ + "group its directory claims", + key, + metadata.GroupPublicKey, + ) + } + if suffix := fmt.Sprint(metadata.MemberIndex); suffix != entry.memberSuffix { + run.finding( + "beacon quarantine metadata [%s] names member [%s], not the "+ + "member its file name claims", + key, + suffix, + ) + } + + validateQuarantineMode(run, key, metadata) + + if entry.membership != nil { + if member := uint8( + entry.membership.Signer.MemberID(), + ); member != metadata.MemberIndex { + run.finding( + "beacon quarantine output [%s] pairs metadata for member "+ + "[%d] with a membership of member [%d]", + key, + metadata.MemberIndex, + member, + ) + } + membershipGroup := hex.EncodeToString( + entry.membership.Signer.GroupPublicKeyBytesCompressed(), + ) + if membershipGroup != metadata.GroupPublicKey { + run.finding( + "beacon quarantine output [%s] pairs metadata for group "+ + "[%s] with a membership of group [%s]", + key, + metadata.GroupPublicKey, + membershipGroup, + ) + } + } +} + +// validateQuarantineMode checks the recorded protocol mode against the +// recorded cutover arithmetic — the mode is pinned from the canonical +// anchor, so a record that contradicts that rule was not produced by the +// release gate — and the recorded cutover block against the expected armed +// schedule: a record preserved under a different cutover block belongs to a +// different deployment than the one being rolled back. +func validateQuarantineMode( + run *auditRun, + key string, + metadata *registry.QuarantinedSignerMetadata, +) { + if run.expected.cutoverBlock > 0 && + metadata.CutoverBlock != run.expected.cutoverBlock { + run.finding( + "beacon quarantine metadata [%s] was preserved under cutover "+ + "block [%d], not the expected cutover block [%d]", + key, + metadata.CutoverBlock, + run.expected.cutoverBlock, + ) + } + + if violation := cutoverModeViolation( + fmt.Sprintf("beacon quarantine metadata [%s]", key), + metadata.ProtocolMode, + metadata.CanonicalStartBlock, + metadata.CutoverBlock, + ); violation != "" { + run.finding("%s", violation) + } +} + +// interpretTBTCActiveNamespace decodes every tBTC keystore record with the +// same decode the wallet registry loader uses and cross-checks each record +// against the wallet directory and member file name it is stored under, the +// signing group bounds, and its sibling records. It returns the set of active +// wallet storage keys for the quarantine overlap check. +func interpretTBTCActiveNamespace( + diskStorage storage.Storage, + run *auditRun, +) (map[string]struct{}, error) { + tbtcHandle, err := diskStorage.InitializeKeyStorePersistence("tbtc") + if err != nil { + return nil, fmt.Errorf( + "cannot open the tbtc keystore namespace: [%w]", + err, + ) + } + + wallets := make(map[string]*tbtcWalletRecord) + seenMembers := make(map[string]map[uint8]struct{}) + + tbtcData, tbtcErrors := tbtcHandle.ReadAll() + tbtcDone := make(chan struct{}) + go func() { + defer close(tbtcDone) + for err := range tbtcErrors { + run.finding("tbtc keystore namespace read error: [%v]", err) + } + }() + for descriptor := range tbtcData { + content, err := descriptor.Content() + if err != nil { + run.finding( + "tbtc active record [%s/%s] cannot be decrypted: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + + record, err := tbtc.DecodeSignerAuditRecord(content) + if err != nil { + run.finding( + "tbtc active record [%s/%s] cannot be decoded the way the "+ + "wallet registry loader decodes it: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + + if record.WalletStorageKey != descriptor.Directory() { + run.finding( + "tbtc active record [%s/%s] contains wallet [%s], not the "+ + "wallet its directory claims", + descriptor.Directory(), + descriptor.Name(), + record.WalletStorageKey, + ) + } + + // The registry saves each signer under "membership_"; a record + // whose content disagrees with its file name belongs to a different + // member than the layout claims. + if expected := fmt.Sprintf( + "membership_%d", + record.MemberIndex, + ); descriptor.Name() != expected { + run.finding( + "tbtc active record [%s/%s] contains member [%d], not the "+ + "member its file name claims", + descriptor.Directory(), + descriptor.Name(), + record.MemberIndex, + ) + } + + if record.SigningGroupSize <= 0 || + int(record.MemberIndex) < 1 || + int(record.MemberIndex) > record.SigningGroupSize { + run.finding( + "tbtc active record [%s/%s] claims member index [%d] outside "+ + "the signing group bounds [1, %d]", + descriptor.Directory(), + descriptor.Name(), + record.MemberIndex, + record.SigningGroupSize, + ) + } + + wallet, ok := wallets[record.WalletStorageKey] + if !ok { + wallet = &tbtcWalletRecord{ + WalletStorageKey: record.WalletStorageKey, + WalletID: record.WalletID, + WalletPublicKeyHash: record.WalletPublicKeyHash, + SigningGroupSize: record.SigningGroupSize, + } + wallets[record.WalletStorageKey] = wallet + seenMembers[record.WalletStorageKey] = make(map[uint8]struct{}) + } + if wallet.SigningGroupSize != record.SigningGroupSize { + run.finding( + "tbtc active record [%s/%s] claims signing group size [%d] "+ + "while another record of the same wallet claims [%d]", + descriptor.Directory(), + descriptor.Name(), + record.SigningGroupSize, + wallet.SigningGroupSize, + ) + } + if _, duplicate := seenMembers[record.WalletStorageKey][uint8( + record.MemberIndex, + )]; duplicate { + run.finding( + "tbtc active record [%s/%s] duplicates member index [%d] of "+ + "the same wallet", + descriptor.Directory(), + descriptor.Name(), + record.MemberIndex, + ) + } + seenMembers[record.WalletStorageKey][uint8( + record.MemberIndex, + )] = struct{}{} + wallet.MemberIndexes = append( + wallet.MemberIndexes, + uint8(record.MemberIndex), + ) + } + <-tbtcDone + + activeWallets := make(map[string]struct{}) + for _, wallet := range wallets { + sort.Slice(wallet.MemberIndexes, func(i, j int) bool { + return wallet.MemberIndexes[i] < wallet.MemberIndexes[j] + }) + run.manifest.TBTCActiveWallets = append( + run.manifest.TBTCActiveWallets, + *wallet, + ) + activeWallets[wallet.WalletStorageKey] = struct{}{} + } + + return activeWallets, nil +} + +// tbtcQuarantineEntry pairs the two halves of one quarantined tBTC signer +// output while the namespace is scanned. +type tbtcQuarantineEntry struct { + directory string + memberSuffix string + metadata *tbtc.QuarantinedSignerMetadata + signer *tbtc.SignerAuditRecord + // handoffMetadata and handoffSigner are the halves carried by the combined + // record preservation writes when the namespace would not take the pair. + // They stand in for whichever half the pair is missing, so an output + // preserved that way is as complete a piece of evidence as a paired one. + handoffMetadata *tbtc.QuarantinedSignerMetadata + handoffSigner *tbtc.SignerAuditRecord + // signerBytes and handoffSignerBytes are the key material as each form + // stored it. The two forms encode the signer identically, so when both name + // the same seat the stored bytes are what says whether they hold the same + // share — a question the decoded records cannot be asked, since the audit + // record deliberately carries only the public identity of a signer whose + // private half it must never compare or report. + signerBytes []byte + handoffSignerBytes []byte +} + +// interpretTBTCQuarantineNamespace decodes the tBTC quarantine namespace, +// pairs metadata and signer halves by wallet directory and member suffix, and +// cross-validates the metadata against its schema, this release's identity, +// the cutover arithmetic, the storage location, the decoded signer, and the +// active namespace. +func interpretTBTCQuarantineNamespace( + diskStorage storage.Storage, + run *auditRun, + activeWallets map[string]struct{}, +) error { + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + return fmt.Errorf( + "cannot open the tbtc quarantine namespace: [%w]", + err, + ) + } + + quarantineEntries := make(map[string]*tbtcQuarantineEntry) + entryFor := func(directory, name, prefix string) *tbtcQuarantineEntry { + suffix := strings.TrimPrefix(name, prefix) + key := directory + "/" + suffix + if _, ok := quarantineEntries[key]; !ok { + quarantineEntries[key] = &tbtcQuarantineEntry{ + directory: directory, + memberSuffix: suffix, + } + } + return quarantineEntries[key] + } + + quarantineData, quarantineErrors := quarantineHandle.ReadAll() + quarantineDone := make(chan struct{}) + go func() { + defer close(quarantineDone) + for err := range quarantineErrors { + run.finding("tbtc quarantine namespace read error: [%v]", err) + } + }() + for descriptor := range quarantineData { + content, err := descriptor.Content() + if err != nil { + run.finding( + "tbtc quarantine record [%s/%s] cannot be decrypted: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + + switch { + case strings.HasPrefix(descriptor.Name(), "metadata_"): + metadata := &tbtc.QuarantinedSignerMetadata{} + if err := json.Unmarshal(content, metadata); err != nil { + run.finding( + "tbtc quarantine metadata [%s/%s] cannot be decoded: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + entryFor( + descriptor.Directory(), + descriptor.Name(), + "metadata_", + ).metadata = metadata + case strings.HasPrefix(descriptor.Name(), "membership_"): + record, err := tbtc.DecodeSignerAuditRecord(content) + if err != nil { + run.finding( + "tbtc quarantine membership [%s/%s] cannot be decoded the "+ + "way the wallet registry loader decodes it: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + tbtcEntry := entryFor( + descriptor.Directory(), + descriptor.Name(), + "membership_", + ) + tbtcEntry.signer = record + tbtcEntry.signerBytes = content + case strings.HasPrefix(descriptor.Name(), "handoff_"): + handoff, err := tbtc.DecodeQuarantinedSignerHandoff(content) + if err != nil { + run.finding( + "tbtc quarantine handoff [%s/%s] cannot be decoded: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + record, err := tbtc.DecodeSignerAuditRecord(handoff.Signer) + if err != nil { + run.finding( + "tbtc quarantine handoff [%s/%s] carries key material that "+ + "cannot be decoded the way the wallet registry loader "+ + "decodes it: [%v]", + descriptor.Directory(), + descriptor.Name(), + err, + ) + continue + } + entry := entryFor( + descriptor.Directory(), + descriptor.Name(), + "handoff_", + ) + metadata := handoff.Metadata + entry.handoffMetadata = &metadata + entry.handoffSigner = record + entry.handoffSignerBytes = handoff.Signer + default: + run.finding( + "tbtc quarantine record [%s/%s] has an unknown name", + descriptor.Directory(), + descriptor.Name(), + ) + } + } + <-quarantineDone + + keys := make([]string, 0, len(quarantineEntries)) + for key := range quarantineEntries { + keys = append(keys, key) + } + sort.Strings(keys) + + for _, key := range keys { + entry := quarantineEntries[key] + + // The combined record stands in for whichever half of the pair the + // namespace would not take. It is resolved before validation so the + // audit reports what the namespace actually holds of an output rather + // than which of the two layouts it was written in. + hasHandoff := entry.handoffSigner != nil + reconcileTBTCQuarantineForms(run, entry) + + validateTBTCQuarantineEntry(run, entry, activeWallets) + + if entry.metadata == nil { + continue + } + signerWalletID := "" + signerWalletPublicKeyHash := "" + if entry.signer != nil { + signerWalletID = entry.signer.WalletID + signerWalletPublicKeyHash = entry.signer.WalletPublicKeyHash + } + run.manifest.TBTCQuarantinedOutputs = append( + run.manifest.TBTCQuarantinedOutputs, + tbtcQuarantineRecord{ + QuarantinedSignerMetadata: *entry.metadata, + WalletStorageKey: entry.directory, + SignerWalletID: signerWalletID, + SignerWalletPublicKeyHash: signerWalletPublicKeyHash, + HasMembershipRecord: entry.signer != nil, + HasHandoffRecord: hasHandoff, + }, + ) + } + + return nil +} + +// reconcileTBTCQuarantineForms settles one tBTC output the namespace holds in +// both preserved forms and fills whichever half the pair is missing from the +// handoff. +// +// Preservation writes the pair first and falls back on the combined record for +// what the namespace would not take, so the two forms overlap by design: a run +// that got the membership down, was refused the metadata, and then wrote the +// handoff leaves a standalone half beside a handoff carrying its own copy of +// that same half. Reading past the duplicate is not free. The forms are only +// interchangeable while they agree, and preferring whichever one this scan +// happened to decode first lets a stale or half-overwritten standalone record +// stand in for a complete handoff that contradicts it — the audit answering +// with evidence it never checked. +// +// So a duplicate is compared rather than deduplicated on sight, and a +// disagreement is reported instead of resolved. Which copy is the true one is +// not a question this tool can answer offline: it is what an operator has to +// settle before a rollback trusts either. +func reconcileTBTCQuarantineForms( + run *auditRun, + entry *tbtcQuarantineEntry, +) { + if entry.metadata != nil && entry.handoffMetadata != nil { + same, err := sameQuarantineDocument(entry.metadata, entry.handoffMetadata) + switch { + case err != nil: + run.finding( + "tbtc quarantined output [%s/%s] carries audit metadata in "+ + "both preserved forms and they cannot be compared: [%v]", + entry.directory, + entry.memberSuffix, + err, + ) + case !same: + run.finding( + "tbtc quarantined output [%s/%s] carries audit metadata in "+ + "both preserved forms and they disagree; the standalone "+ + "record and the handoff describe the same seat differently, "+ + "so neither can be used as evidence until an operator "+ + "establishes which one the namespace should keep", + entry.directory, + entry.memberSuffix, + ) + } + } + if entry.metadata == nil { + entry.metadata = entry.handoffMetadata + } + + if entry.signer != nil && entry.handoffSigner != nil && + !bytes.Equal(entry.signerBytes, entry.handoffSignerBytes) { + run.finding( + "tbtc quarantined output [%s/%s] carries key material in both "+ + "preserved forms and the two copies differ; a rollback cannot "+ + "tell which share this seat holds until an operator establishes "+ + "which record the namespace should keep", + entry.directory, + entry.memberSuffix, + ) + } + if entry.signer == nil { + entry.signer = entry.handoffSigner + } +} + +// validateTBTCQuarantineEntry cross-validates one paired tBTC quarantine +// output. The metadata exists for the offline audit alone, so any half or +// field that contradicts the rest of the record makes the output +// untrustworthy evidence. +func validateTBTCQuarantineEntry( + run *auditRun, + entry *tbtcQuarantineEntry, + activeWallets map[string]struct{}, +) { + key := entry.directory + "/" + entry.memberSuffix + + // A quarantined wallet visible in the active namespace is exactly the + // ambiguity the quarantine exists to prevent: the same wallet would be + // both activated and marked interrupted. + if _, active := activeWallets[entry.directory]; active { + run.finding( + "tbtc quarantine output [%s] belongs to wallet [%s] that is "+ + "also present in the active namespace", + key, + entry.directory, + ) + } + + if entry.signer != nil { + if entry.signer.WalletStorageKey != entry.directory { + run.finding( + "tbtc quarantine membership [%s] contains wallet [%s], not "+ + "the wallet its directory claims", + key, + entry.signer.WalletStorageKey, + ) + } + if suffix := fmt.Sprint( + entry.signer.MemberIndex, + ); suffix != entry.memberSuffix { + run.finding( + "tbtc quarantine membership [%s] contains member [%s], not "+ + "the member its file name claims", + key, + suffix, + ) + } + } + + if entry.metadata == nil { + run.finding( + "tbtc quarantine output [%s] has a membership record without "+ + "audit metadata", + key, + ) + return + } + + metadata := entry.metadata + if entry.signer == nil { + run.finding( + "tbtc quarantine output [%s] has audit metadata without a "+ + "membership record; the key material was not preserved", + key, + ) + } + + if metadata.SchemaVersion != tbtc.QuarantineSchemaVersion { + run.finding( + "tbtc quarantine metadata [%s] has schema version [%d], "+ + "expected [%d]", + key, + metadata.SchemaVersion, + tbtc.QuarantineSchemaVersion, + ) + } + if metadata.ReleaseEpoch != participation.CompiledEpoch.String() { + run.finding( + "tbtc quarantine metadata [%s] was written by release epoch "+ + "[%s], not by this audit's epoch [%s]", + key, + metadata.ReleaseEpoch, + participation.CompiledEpoch, + ) + } + if metadata.Ceremony != string(participation.TBTCDKG) { + run.finding( + "tbtc quarantine metadata [%s] names ceremony [%s]; only [%s] "+ + "outputs are quarantined", + key, + metadata.Ceremony, + participation.TBTCDKG, + ) + } + if suffix := fmt.Sprint(metadata.MemberIndex); suffix != entry.memberSuffix { + run.finding( + "tbtc quarantine metadata [%s] names member [%s], not the "+ + "member its file name claims", + key, + suffix, + ) + } + if !isCanonicalSHA256Hex(metadata.SeedHash) { + run.finding( + "tbtc quarantine metadata [%s] seed hash [%s] is not a "+ + "canonical SHA-256 digest of 64 lowercase hexadecimal "+ + "characters", + key, + metadata.SeedHash, + ) + } + if metadata.MemberIndex == 0 { + run.finding( + "tbtc quarantine metadata [%s] names invalid member index [0]", + key, + ) + } + if metadata.WalletPublicKeyHash == "" { + run.finding( + "tbtc quarantine metadata [%s] is missing the wallet public "+ + "key hash", + key, + ) + } + + validateTBTCQuarantineMode(run, key, metadata) + + if entry.signer == nil { + return + } + + if uint8(entry.signer.MemberIndex) != metadata.MemberIndex { + run.finding( + "tbtc quarantine output [%s] pairs metadata for member [%d] "+ + "with a membership of member [%d]", + key, + metadata.MemberIndex, + entry.signer.MemberIndex, + ) + } + if metadata.WalletID != "" && + metadata.WalletID != entry.signer.WalletID { + run.finding( + "tbtc quarantine metadata [%s] names wallet ID [%s], but its "+ + "membership decodes to wallet ID [%s]", + key, + metadata.WalletID, + entry.signer.WalletID, + ) + } + if metadata.WalletPublicKeyHash != "" && + metadata.WalletPublicKeyHash != entry.signer.WalletPublicKeyHash { + run.finding( + "tbtc quarantine metadata [%s] names wallet public key hash "+ + "[%s], but its membership decodes to [%s]", + key, + metadata.WalletPublicKeyHash, + entry.signer.WalletPublicKeyHash, + ) + } +} + +// validateTBTCQuarantineMode checks the recorded protocol mode against the +// recorded cutover arithmetic — the mode is pinned from the canonical +// anchor, so a record that contradicts that rule was not produced by the +// release gate — and the recorded cutover block against the expected armed +// schedule: a record preserved under a different cutover block belongs to a +// different deployment than the one being rolled back. +func validateTBTCQuarantineMode( + run *auditRun, + key string, + metadata *tbtc.QuarantinedSignerMetadata, +) { + if run.expected.cutoverBlock > 0 && + metadata.CutoverBlock != run.expected.cutoverBlock { + run.finding( + "tbtc quarantine metadata [%s] was preserved under cutover "+ + "block [%d], not the expected cutover block [%d]", + key, + metadata.CutoverBlock, + run.expected.cutoverBlock, + ) + } + + if violation := cutoverModeViolation( + fmt.Sprintf("tbtc quarantine metadata [%s]", key), + metadata.ProtocolMode, + metadata.CanonicalStartBlock, + metadata.CutoverBlock, + ); violation != "" { + run.finding("%s", violation) + } +} + +// sortRecords orders the interpreted records deterministically so two audits +// of the same snapshot produce byte-identical manifests apart from the +// generation time. +func sortRecords(auditManifest *manifest) { + sort.Slice(auditManifest.BeaconActiveMemberships, func(i, j int) bool { + left := auditManifest.BeaconActiveMemberships[i] + right := auditManifest.BeaconActiveMemberships[j] + if left.GroupPublicKey != right.GroupPublicKey { + return left.GroupPublicKey < right.GroupPublicKey + } + return left.MemberIndex < right.MemberIndex + }) + sort.Slice(auditManifest.BeaconQuarantinedOutputs, func(i, j int) bool { + left := auditManifest.BeaconQuarantinedOutputs[i] + right := auditManifest.BeaconQuarantinedOutputs[j] + if left.GroupPublicKey != right.GroupPublicKey { + return left.GroupPublicKey < right.GroupPublicKey + } + return left.MemberIndex < right.MemberIndex + }) + sort.Slice(auditManifest.TBTCActiveWallets, func(i, j int) bool { + return auditManifest.TBTCActiveWallets[i].WalletStorageKey < + auditManifest.TBTCActiveWallets[j].WalletStorageKey + }) + sort.Slice(auditManifest.TBTCQuarantinedOutputs, func(i, j int) bool { + left := auditManifest.TBTCQuarantinedOutputs[i] + right := auditManifest.TBTCQuarantinedOutputs[j] + if left.WalletPublicKeyHash != right.WalletPublicKeyHash { + return left.WalletPublicKeyHash < right.WalletPublicKeyHash + } + return left.MemberIndex < right.MemberIndex + }) +} diff --git a/cmd/participation-state-audit/main_test.go b/cmd/participation-state-audit/main_test.go new file mode 100644 index 0000000000..95127b4486 --- /dev/null +++ b/cmd/participation-state-audit/main_test.go @@ -0,0 +1,7821 @@ +package main + +import ( + "bytes" + "context" + "crypto/ecdsa" + "crypto/ed25519" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "math/big" + "os" + "path/filepath" + "slices" + "sort" + "strconv" + "strings" + "testing" + "time" + + "github.com/ethereum/go-ethereum/accounts/abi" + "github.com/ethereum/go-ethereum/common" + ethereumCrypto "github.com/ethereum/go-ethereum/crypto" + bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + "google.golang.org/protobuf/proto" + + "github.com/keep-network/keep-common/pkg/persistence" + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/altbn128" + "github.com/keep-network/keep-core/pkg/beacon/dkg" + "github.com/keep-network/keep-core/pkg/beacon/registry" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/bls" + "github.com/keep-network/keep-core/pkg/chain" + beaconabi "github.com/keep-network/keep-core/pkg/chain/ethereum/beacon/gen/abi" + ecdsaabi "github.com/keep-network/keep-core/pkg/chain/ethereum/ecdsa/gen/abi" + "github.com/keep-network/keep-core/pkg/crypto/secp256k1" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/storage" + "github.com/keep-network/keep-core/pkg/tbtc" + tbtcpb "github.com/keep-network/keep-core/pkg/tbtc/gen/pb" + "github.com/keep-network/keep-core/pkg/tecdsa" + tecdsapb "github.com/keep-network/keep-core/pkg/tecdsa/gen/pb" +) + +const testPassword = "audit-test-password" + +const ( + testWalletRegistryAddress = "0x1111111111111111111111111111111111111111" + testRandomBeaconAddress = "0x3333333333333333333333333333333333333333" + testFinalizedEthereumBlock = uint64(10_000) + testChainEvidencePrivateKeyByte = byte(0x42) +) + +func testCanonicalEthereumBlockHash(block uint64) string { + return fmt.Sprintf("0x%064x", block+1) +} + +func testChainEvidencePrivateKey() ed25519.PrivateKey { + seed := make([]byte, ed25519.SeedSize) + for i := range seed { + seed[i] = testChainEvidencePrivateKeyByte + } + return ed25519.NewKeyFromSeed(seed) +} + +func testChainEvidencePublicKey() string { + publicKey := testChainEvidencePrivateKey().Public().(ed25519.PublicKey) + return hex.EncodeToString(publicKey) +} + +type auditGateBlockCounter struct { + block uint64 +} + +func (c *auditGateBlockCounter) CurrentBlock() (uint64, error) { + return c.block, nil +} + +func (c *auditGateBlockCounter) WaitForBlockHeight(uint64) error { + return nil +} + +func (c *auditGateBlockCounter) BlockHeightWaiter( + uint64, +) (<-chan uint64, error) { + result := make(chan uint64, 1) + result <- c.block + close(result) + return result, nil +} + +func (c *auditGateBlockCounter) WatchBlocks( + ctx context.Context, +) <-chan uint64 { + result := make(chan uint64) + go func() { + <-ctx.Done() + close(result) + }() + return result +} + +type auditGateMetrics struct{} + +func (auditGateMetrics) IncrementCounter(string, float64) {} +func (auditGateMetrics) SetGauge(string, float64) {} + +func newTestSigner( + t *testing.T, + memberIndex group.MemberIndex, + groupSecret int64, +) *dkg.ThresholdSigner { + t.Helper() + + groupPublicKey := new(bn256.G2).ScalarBaseMult(big.NewInt(groupSecret)) + + return dkg.NewThresholdSigner( + memberIndex, + groupPublicKey, + big.NewInt(7), + map[group.MemberIndex]*bn256.G2{ + memberIndex: new(bn256.G2).ScalarBaseMult(big.NewInt(7)), + }, + []chain.Address{"0x0000000000000000000000000000000000000001"}, + ) +} + +func groupPublicKeyHex(membership *registry.Membership) string { + return hex.EncodeToString( + membership.Signer.GroupPublicKeyBytesCompressed(), + ) +} + +// newTestStorage builds a storage snapshot with one active beacon membership +// and one quarantined output of a different group, written through the +// production persistence paths and layout, and returns its root directory. +func newTestStorage(t *testing.T) string { + return newTestStorageWithQuiescencePermits(t, nil) +} + +func newTestStorageWithQuiescencePermits( + t *testing.T, + permits []quiescencePermitEvidence, +) string { + t.Helper() + + storageDir := t.TempDir() + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + + activeHandle, err := diskStorage.InitializeKeyStorePersistence("beacon") + if err != nil { + t.Fatal(err) + } + activeMembership := ®istry.Membership{ + Signer: newTestSigner(t, group.MemberIndex(1), testActiveGroupSecret), + ChannelName: "test-channel", + } + activeBytes, err := activeMembership.Marshal() + if err != nil { + t.Fatal(err) + } + if err := activeHandle.Save( + activeBytes, + groupPublicKeyHex(activeMembership), + "/membership_1", + ); err != nil { + t.Fatal(err) + } + + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + t.Fatal(err) + } + quarantine := registry.NewQuarantine( + context.Background(), + &testutils.MockLogger{}, + quarantineHandle, + ) + if _, err := quarantine.Preserve( + ®istry.Membership{ + Signer: newTestSigner(t, group.MemberIndex(2), 43), + ChannelName: "test-channel", + }, + registry.QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + ProtocolMode: "legacy", + CutoverBlock: 1_000, + CanonicalStartBlock: 900, + Ceremony: "beacon_dkg", + SeedHash: strings.Repeat("a", 64), + FailedOperation: "beacon_dkg_result_publication", + LastObservedBlock: 950, + }, + nil, + ); err != nil { + t.Fatal(err) + } + + participationHandle, err := + diskStorage.InitializeWorkPersistence("participation") + if err != nil { + t.Fatal(err) + } + recorder, err := + participation.NewPersistenceQuiescenceSnapshotRecorder( + participationHandle, + ) + if err != nil { + t.Fatal(err) + } + + snapshot := participation.QuiescenceSnapshot{ + SchemaVersion: participation.QuiescenceSnapshotSchemaVersion, + CapturedAt: time.Now().UTC().Add(-time.Minute), + ReleaseVersion: "v2.1.0", + ReleaseRevision: strings.Repeat("ef", 20), + ReleaseEpoch: participation.CompiledEpoch.String(), + CutoverBlock: 1_000, + CurrentBlock: 1_100, + ClockAvailable: true, + State: participation.StateQuiescing.String(), + QuiesceCause: "rollback drill", + ActiveCeremonies: uint64(len(permits)), + } + for _, permit := range permits { + snapshot.ActivePermits = append( + snapshot.ActivePermits, + participation.PermitSnapshot{ + Ceremony: participation.Ceremony(permit.Ceremony), + Mode: permit.Mode, + CanonicalStartBlock: permit.CanonicalStartBlock, + WorkID: permit.WorkID, + PermitID: permit.PermitID, + IdentityBound: true, + OperatedMembers: testOperatedMembers( + participation.Ceremony(permit.Ceremony), + permit.PermitID, + ), + }, + ) + switch permit.Mode { + case participation.ModeLegacy.String(): + snapshot.ActiveLegacyCeremonies++ + case participation.ModeSecurityV2.String(): + snapshot.ActiveSecurityV2Ceremonies++ + } + } + sort.Slice(snapshot.ActivePermits, func(i, j int) bool { + return permitSnapshotLess( + snapshot.ActivePermits[i], + snapshot.ActivePermits[j], + ) + }) + if err := recorder.Record(snapshot); err != nil { + t.Fatal(err) + } + for i, permit := range permits { + record := testTerminalOutcomeRecord( + t, + permit, + participation.PermitSnapshot{ + Ceremony: participation.Ceremony(permit.Ceremony), + Mode: permit.Mode, + CanonicalStartBlock: permit.CanonicalStartBlock, + WorkID: permit.WorkID, + PermitID: permit.PermitID, + IdentityBound: true, + OperatedMembers: testOperatedMembers( + participation.Ceremony(permit.Ceremony), + permit.PermitID, + ), + }, + fmt.Sprintf("test-result-%d", i), + groupPublicKeyHex(activeMembership), + ) + if err := recorder.RecordTerminalOutcome(record); err != nil { + t.Fatal(err) + } + } + + return storageDir +} + +// testPermitSeat reports the seat a per-seat ceremony names in its own permit +// identity, and zero for the permits that name none. A DKG member, a beacon +// group member and a relay signing membership each run one seat under one +// permit, so the identity is where their seat lives. +func testPermitSeat( + ceremony participation.Ceremony, + permitID string, +) group.MemberIndex { + switch ceremony { + case participation.TBTCDKG, + participation.BeaconDKG, + participation.BeaconRelaySigning: + default: + return 0 + } + + seat, err := strconv.ParseUint(permitID, 10, 8) + if err != nil || seat == 0 { + return 0 + } + + return group.MemberIndex(seat) +} + +// testOperatedMembers renders the seats a fixture permit's holder operates, as +// the production node names them at issuance. The per-seat ceremonies take +// theirs from the permit identity; a wallet action takes the seat its transcript +// is written for; and the permits that operate none — a forwarder, a timeout +// monitor — name none. +func testOperatedMembers( + ceremony participation.Ceremony, + permitID string, +) participation.MemberIndexes { + switch ceremony { + case participation.BeaconRelayForwarding, + participation.BeaconTimeoutReport: + return nil + } + + if seat := testPermitSeat(ceremony, permitID); seat != 0 { + return participation.MemberIndexes{seat} + } + + return participation.MemberIndexes{group.MemberIndex(1)} +} + +// testTranscriptContribution renders the transcript a completed outcome of the +// given ceremony must carry, and nil for the ceremonies whose owners author +// none. The local membership is the one the record persists, so the two halves +// of the fixture describe one ceremony and a test that means to break the +// binding has to say so. +// testTranscriptContribution renders the transcript a completed outcome must +// carry. permitSeat is the seat in the permits' own index space that produced the +// local membership; it is read only for the ceremonies whose record speaks in a +// different space than their permits, and 0 stands for "the same seat". +func testTranscriptContribution( + ceremony participation.Ceremony, + local group.MemberIndex, + permitSeat group.MemberIndex, +) *participation.TranscriptContribution { + if !participation.AuthorsTranscriptContribution(ceremony) { + return nil + } + + if local == 0 { + local = group.MemberIndex(1) + } + + incorporated := participation.MemberIndexes{local} + for _, peer := range []group.MemberIndex{1, 2, 3} { + if peer != local { + incorporated = append(incorporated, peer) + } + } + slices.Sort(incorporated) + + return &participation.TranscriptContribution{ + IncorporatedMembers: incorporated, + LocalMembers: participation.MemberIndexes{local}, + PermitSpaceMembers: testPermitSpaceMembers( + ceremony, + incorporated, + local, + permitSeat, + ), + } +} + +// testPermitSpaceMembers renders a mapping from a transcript's seats back to the +// index space this work's permits were issued in, placing permitSeat under local +// and running consecutively either side of it, and nil for the ceremonies whose +// result already speaks in the permits' space. +// +// permitSeat must leave room for the seats below local, which every fixture here +// satisfies. A mapping that does not is left malformed rather than quietly +// adjusted: the gate's own set validation then refuses it, which is the loud +// failure a fixture that cannot mean what it says deserves. +func testPermitSpaceMembers( + ceremony participation.Ceremony, + incorporated participation.MemberIndexes, + local group.MemberIndex, + permitSeat group.MemberIndex, +) participation.MemberIndexes { + if ceremony != participation.TBTCDKG { + return nil + } + if permitSeat == 0 { + permitSeat = local + } + + position := slices.Index(incorporated, local) + mapping := make(participation.MemberIndexes, len(incorporated)) + for i := range mapping { + mapping[i] = group.MemberIndex(int(permitSeat) + i - position) + } + + return mapping +} + +func testTerminalOutcomeRecord( + t *testing.T, + permit quiescencePermitEvidence, + snapshot participation.PermitSnapshot, + resultReference string, + beaconSignerReference string, +) participation.TerminalOutcomeRecord { + t.Helper() + + evidence := participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: resultReference, + } + switch participation.TerminalOutcome(permit.Outcome) { + case participation.TerminalOutcomeCompleted: + // Each ceremony settles on the evidence class its result actually + // lives in, mirroring what the node records in production. A fixture + // that settled everything on a node-authored protocol digest would + // exercise a shape the gate's own validator rejects. + switch snapshot.Ceremony { + case participation.TBTCDKG: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: "test-tbtc-signer", + MembershipIndex: group.MemberIndex(1), + } + case participation.BeaconDKG: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedBeaconSigner, + Reference: beaconSignerReference, + MembershipIndex: group.MemberIndex(1), + } + case participation.TBTCSigning: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceBitcoinTransaction, + Reference: testBitcoinTransactionHash(resultReference), + } + case participation.TBTCInactivityClaim: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceEthereumTransaction, + Reference: resultReference, + } + case participation.BeaconRelaySigning: + // A relay entry answers exactly one request, and the audit holds + // the record to the request its permit names, so the fixture + // derives the request from the permit rather than picking one. + requestStartBlock, err := participation.ParseBeaconRelayWorkID( + snapshot.WorkID, + ) + if err != nil { + t.Fatalf( + "relay fixture work identity [%s] names no relay "+ + "request: [%v]", + snapshot.WorkID, + err, + ) + } + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: testRelayEntryReference( + requestStartBlock, + testActiveGroupSecret, + resultReference, + ), + } + case participation.BeaconRelayForwarding: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceForwarderClosed, + } + } + // The transcript's local seat is the seat the permit was issued for + // wherever the two share an index space. tBTC DKG is the exception: + // its permit names a DKG index while its transcript is in the final + // signing group's, so there the persisted membership is the local seat + // and the permit's own seat is what the transcript maps it back to. + local := evidence.MembershipIndex + permitSeat := testPermitSeat(snapshot.Ceremony, snapshot.PermitID) + if snapshot.Ceremony != participation.TBTCDKG && permitSeat != 0 { + local = permitSeat + } + evidence.Contribution = testTranscriptContribution( + snapshot.Ceremony, + local, + permitSeat, + ) + case participation.TerminalOutcomeQuarantined: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceQuarantinedTBTCSinger, + } + if snapshot.Ceremony == participation.BeaconDKG { + evidence.Kind = + participation.TerminalEvidenceQuarantinedBeaconSigner + } + case participation.TerminalOutcomeExhausted: + evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceNoThreshold, + } + } + + return participation.TerminalOutcomeRecord{ + RecordedAt: time.Now().UTC(), + Permit: snapshot, + Outcome: participation.TerminalOutcome(permit.Outcome), + Evidence: evidence, + } +} + +// testActiveGroupSecret is the scalar behind the active beacon membership every +// test storage holds. The fixture group's public key is its base multiple, so a +// test can produce entries that genuinely verify under the group the snapshot +// decoded rather than merely well-formed ones. +const testActiveGroupSecret = int64(42) + +// testFixtureSelectedGroupID is the registry index the fixture beacon selects +// for every relay request it logs. It is the index the group behind +// testActiveGroupSecret is registered under, since the audit refuses a +// completed entry whose key is not the one the selected index is bound to. +const testFixtureSelectedGroupID = uint64(1) + +// testRelayEntryReference derives a relay entry identity from a fixture label, +// signed for real by the given group. Passing the group's own secret is what +// makes the reference survive the audit's signature verification: the check is +// a pairing over actual curve points, so a shaped-but-fabricated identity is +// exactly what it exists to reject. +func testRelayEntryReference( + requestStartBlock uint64, + groupSecret int64, + label string, +) string { + secret := big.NewInt(groupSecret) + previousEntry := altbn128.G1HashToPoint([]byte(label)) + + reference, err := participation.BeaconRelayEntryReference( + requestStartBlock, + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(secret), + }.Compress(), + previousEntry.Marshal(), + bls.SignG1(secret, previousEntry).Marshal(), + ) + if err != nil { + panic(err) + } + + return reference +} + +// testBitcoinTransactionHash derives a canonical, unprefixed lowercase +// transaction hash from a fixture label, so a wallet action's evidence has the +// shape the Bitcoin reconciliation set can actually enumerate. +func testBitcoinTransactionHash(label string) string { + digest := sha256.Sum256([]byte(label)) + return hex.EncodeToString(digest[:]) +} + +func persistRealGateQuiescenceSnapshot( + t *testing.T, + storageDir string, + permits []quiescencePermitEvidence, +) { + t.Helper() + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + handle, err := diskStorage.InitializeWorkPersistence("participation") + if err != nil { + t.Fatal(err) + } + recorder, err := + participation.NewPersistenceQuiescenceSnapshotRecorder(handle) + if err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: 1_000}, + &auditGateBlockCounter{block: 1_100}, + auditGateMetrics{}, + participation.WithArtifactIdentity( + "v2.1.0", + strings.Repeat("ef", 20), + ), + participation.WithQuiescenceSnapshotRecorder(recorder), + ) + if err != nil { + t.Fatal(err) + } + defer gate.Close() + + active := make([]participation.Permit, 0, len(permits)) + for _, expected := range permits { + permit, err := gate.Begin( + participation.Ceremony(expected.Ceremony), + expected.CanonicalStartBlock, + participation.PermitIdentity{ + WorkID: expected.WorkID, + PermitID: expected.PermitID, + OperatedMembers: testOperatedMembers( + participation.Ceremony(expected.Ceremony), + expected.PermitID, + ), + }, + ) + if err != nil { + t.Fatal(err) + } + if permit.Mode().String() != expected.Mode { + t.Fatalf( + "test permit mode [%s] does not match gate-selected mode [%s]", + expected.Mode, + permit.Mode(), + ) + } + active = append(active, permit) + } + + gate.Quiesce(fmt.Errorf("rollback drill")) + for i, permit := range active { + record := testTerminalOutcomeRecord( + t, + permits[i], + participation.PermitSnapshot{ + Ceremony: permit.Ceremony(), + Mode: permit.Mode().String(), + CanonicalStartBlock: permit.CanonicalStartBlock(), + WorkID: permit.WorkID(), + PermitID: permit.PermitID(), + IdentityBound: true, + OperatedMembers: testOperatedMembers( + permit.Ceremony(), + permit.PermitID(), + ), + }, + fmt.Sprintf("real-gate-result-%d", i), + "", + ) + if err := permit.RecordTerminalOutcome( + record.Outcome, + record.Evidence, + ); err != nil { + t.Fatal(err) + } + permit.Close() + } +} + +// newPlaceholderEvidence writes one placeholder text file per external +// rollback input and returns the populated inputs. Placeholder bytes satisfy +// no evidence schema and must stay blocking. +func newPlaceholderEvidence(t *testing.T) evidenceInputs { + t.Helper() + + evidenceDir := t.TempDir() + write := func(name string) string { + path := filepath.Join(evidenceDir, name) + if err := os.WriteFile(path, []byte(name+" evidence"), 0o600); err != nil { + t.Fatal(err) + } + return path + } + + return evidenceInputs{ + chainReconciliation: write("chain-reconciliation"), + bitcoinReconciliation: write("bitcoin-reconciliation"), + quiescenceReport: write("quiescence-report"), + priorReaderCompatibility: write("prior-reader-compatibility"), + } +} + +// newValidTBTCDKGResultEvidence constructs a complete accepted-event lineage +// and returns its derived wallet ID and seed work identity. +func newValidTBTCDKGResultEvidence( + t *testing.T, + seed *big.Int, + startBlock uint64, + originalGroupSize uint16, + misbehavedMemberIndexes []uint8, +) (*tbtcDKGResultEvidence, string, string) { + t.Helper() + + misbehaved := make(map[uint8]struct{}, len(misbehavedMemberIndexes)) + for _, memberIndex := range misbehavedMemberIndexes { + misbehaved[memberIndex] = struct{}{} + } + + members := make([]uint32, originalGroupSize) + signingMemberIndexes := make([]*big.Int, 0, originalGroupSize) + for i := uint16(0); i < originalGroupSize; i++ { + members[i] = uint32(10_000 + i) + if _, excluded := misbehaved[uint8(i+1)]; !excluded { + signingMemberIndexes = append( + signingMemberIndexes, + new(big.Int).SetUint64(uint64(i+1)), + ) + } + } + if len(signingMemberIndexes) == 0 { + t.Fatal("test DKG result needs an operating member") + } + + seedHex := fmt.Sprintf("0x%064x", seed) + seedTag := seed.Uint64() + log := func(blockNumber uint64, logIndex uint64) ethereumLogEvidence { + return ethereumLogEvidence{ + TransactionHash: fmt.Sprintf( + "0x%064x", + (seedTag<<16)+(blockNumber<<2)+logIndex+1, + ), + BlockHash: testCanonicalEthereumBlockHash(blockNumber), + BlockNumber: blockNumber, + LogIndex: logIndex, + } + } + + chainResult := tbtcDKGChainResultEvidence{ + SubmitterMemberIndex: uint16(signingMemberIndexes[0].Uint64()), + GroupPublicKey: "0x" + + "79be667ef9dcbbac55a06295ce870b07029bfcdb2dce28d959f2815b16f81798" + + "483ada7726a3c4655da4fbfc0e1108a8fd17b448a68554199c47d08ffb10d4b8", + MisbehavedMemberIndexes: append( + []uint8(nil), + misbehavedMemberIndexes..., + ), + Signatures: "0x" + strings.Repeat( + "02", + 65*len(signingMemberIndexes), + ), + SigningMemberIndexes: signingMemberIndexes, + Members: members, + } + membersHash, err := computeTBTCDKGMembersHash(chainResult) + if err != nil { + t.Fatal(err) + } + chainResult.MembersHash = membersHash + + resultHash, err := computeTBTCDKGResultHash(chainResult) + if err != nil { + t.Fatal(err) + } + walletID, err := computeTBTCWalletID(chainResult.GroupPublicKey) + if err != nil { + t.Fatal(err) + } + + approvalLog := log(startBlock+2, 0) + walletCreatedLog := approvalLog + walletCreatedLog.LogIndex = 1 + + result := &tbtcDKGResultEvidence{ + Started: tbtcDKGStartedEventEvidence{ + ethereumLogEvidence: log(startBlock, 0), + Seed: seedHex, + }, + Submitted: tbtcDKGResultSubmittedEventEvidence{ + ethereumLogEvidence: log(startBlock+1, 0), + ResultHash: resultHash, + Seed: seedHex, + Result: chainResult, + }, + Approved: tbtcDKGResultApprovedEventEvidence{ + ethereumLogEvidence: approvalLog, + ResultHash: resultHash, + }, + WalletCreated: tbtcWalletCreatedEventEvidence{ + ethereumLogEvidence: walletCreatedLog, + WalletID: walletID, + DKGResultHash: resultHash, + }, + } + seedHash, err := result.seedHash() + if err != nil { + t.Fatal(err) + } + + return result, strings.TrimPrefix(walletID, "0x"), seedHash +} + +func resignTestChainReconciliationEvidence( + t *testing.T, + record *chainReconciliationEvidence, +) { + t.Helper() + + record.CollectorAttestation.Signature = "" + payload, err := chainReconciliationSignaturePayload(record) + if err != nil { + t.Fatal(err) + } + record.CollectorAttestation.Signature = hex.EncodeToString( + ed25519.Sign(testChainEvidencePrivateKey(), payload), + ) +} + +func authenticateTestChainReconciliationEvidence( + t *testing.T, + record *chainReconciliationEvidence, +) { + t.Helper() + + record.WalletRegistryAddress = testWalletRegistryAddress + record.RandomBeaconAddress = testRandomBeaconAddress + record.Receipts = nil + + canonicalBlocks := map[uint64]string{ + testFinalizedEthereumBlock: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + } + receiptIndexes := make(map[string]int) + transactionIndex := uint64(0) + + addEvent := func( + name string, + event ethereumLogEvidence, + result *tbtcDKGResultEvidence, + ) { + t.Helper() + + topics, data, err := expectedTBTCDKGRawLog(name, result) + if err != nil { + t.Fatal(err) + } + for i, topic := range topics { + if topic == "" { + topics[i] = "0x" + strings.Repeat("0", 24) + + strings.TrimPrefix(testWalletRegistryAddress, "0x") + } + } + rawLog := ethereumRawLogEvidence{ + Address: testWalletRegistryAddress, + Topics: topics, + Data: data, + LogIndex: event.LogIndex, + } + + receiptIndex, ok := receiptIndexes[event.TransactionHash] + if !ok { + receiptIndex = len(record.Receipts) + receiptIndexes[event.TransactionHash] = receiptIndex + record.Receipts = append(record.Receipts, ethereumReceiptEvidence{ + TransactionHash: event.TransactionHash, + BlockHash: event.BlockHash, + BlockNumber: event.BlockNumber, + TransactionIndex: transactionIndex, + Status: 1, + }) + transactionIndex++ + } + record.Receipts[receiptIndex].Logs = append( + record.Receipts[receiptIndex].Logs, + rawLog, + ) + canonicalBlocks[event.BlockNumber] = event.BlockHash + } + + for _, wallet := range record.Wallets { + if wallet.DKGResult == nil { + continue + } + result := wallet.DKGResult + addEvent( + "DkgStarted", + result.Started.ethereumLogEvidence, + result, + ) + addEvent( + "DkgResultSubmitted", + result.Submitted.ethereumLogEvidence, + result, + ) + addEvent( + "DkgResultApproved", + result.Approved.ethereumLogEvidence, + result, + ) + addEvent( + "WalletCreated", + result.WalletCreated.ethereumLogEvidence, + result, + ) + } + + blockNumbers := make([]uint64, 0, len(canonicalBlocks)) + for blockNumber := range canonicalBlocks { + blockNumbers = append(blockNumbers, blockNumber) + } + sort.Slice(blockNumbers, func(i, j int) bool { + return blockNumbers[i] < blockNumbers[j] + }) + + record.CollectorAttestation = ethereumCollectorAttestation{ + FinalizedBlockNumber: testFinalizedEthereumBlock, + FinalizedBlockHash: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + } + for _, blockNumber := range blockNumbers { + record.CollectorAttestation.CanonicalBlocks = append( + record.CollectorAttestation.CanonicalBlocks, + ethereumCanonicalBlockEvidence{ + BlockNumber: blockNumber, + BlockHash: canonicalBlocks[blockNumber], + }, + ) + } + resignTestChainReconciliationEvidence(t, record) +} + +func TestComputeTBTCDKGResultHashMatchesGeneratedWalletRegistryABI( + t *testing.T, +) { + evidence, _, _ := newValidTBTCDKGResultEvidence( + t, + big.NewInt(42), + 1_000, + 4, + []uint8{1}, + ) + result := evidence.Submitted.Result + + parsed, err := ecdsaabi.WalletRegistryMetaData.GetAbi() + if err != nil { + t.Fatal(err) + } + event, ok := parsed.Events["DkgResultSubmitted"] + if !ok { + t.Fatal("generated WalletRegistry ABI has no DkgResultSubmitted event") + } + if len(event.Inputs) != 3 { + t.Fatalf( + "expected DkgResultSubmitted to have 3 inputs, got %d", + len(event.Inputs), + ) + } + + groupPublicKey, err := decodeCanonicalEthereumBytes( + result.GroupPublicKey, + 64, + ) + if err != nil { + t.Fatal(err) + } + signatures, err := decodeCanonicalEthereumDynamicBytes(result.Signatures) + if err != nil { + t.Fatal(err) + } + membersHashBytes, err := decodeCanonicalEthereumBytes( + result.MembersHash, + 32, + ) + if err != nil { + t.Fatal(err) + } + var membersHash [32]byte + copy(membersHash[:], membersHashBytes) + signingMemberIndexes := make([]*big.Int, len(result.SigningMemberIndexes)) + for i, memberIndex := range result.SigningMemberIndexes { + signingMemberIndexes[i] = new(big.Int).Set(memberIndex) + } + + encoded, err := (abi.Arguments{{Type: event.Inputs[2].Type}}).Pack( + ecdsaabi.EcdsaDkgResult{ + SubmitterMemberIndex: new(big.Int).SetUint64( + uint64(result.SubmitterMemberIndex), + ), + GroupPubKey: groupPublicKey, + MisbehavedMembersIndices: result.MisbehavedMemberIndexes, + Signatures: signatures, + SigningMembersIndices: signingMemberIndexes, + Members: result.Members, + MembersHash: membersHash, + }, + ) + if err != nil { + t.Fatal(err) + } + expected := "0x" + hex.EncodeToString(ethereumCrypto.Keccak256(encoded)) + + actual, err := computeTBTCDKGResultHash(result) + if err != nil { + t.Fatal(err) + } + if actual != expected { + t.Fatalf( + "result hash disagrees with generated WalletRegistry ABI: "+ + "expected [%s], got [%s]", + expected, + actual, + ) + } +} + +// newValidEvidence builds every mandatory external rollback input, bound to +// the given already-audited manifest: every persisted wallet and group the +// manifest interprets is reconciled as registered and settled, and the prior +// reader covers every required schema. +// testRelayRequestID derives the beacon request identifier a fixture request +// carries from the block it was made in. One request per block is what the +// fixtures model, so the block identifies the request as well as the beacon's +// own counter would. +func testRelayRequestID(requestStartBlock uint64) *big.Int { + return new(big.Int).SetUint64(requestStartBlock*1_000 + 7) +} + +// addTestBeaconRelayLogs appends the RandomBeacon logs the chain reconciliation +// needs to corroborate every beacon relay outcome the manifest's journal +// records as completed: the request each recovered entry answers, and the +// termination each accepted timeout report earned. +// +// The audit reads a relay result as this permit's only when the beacon's own +// request log sits in the permit's block over the entry it signs over, so a +// fixture that omits the log describes a node whose result answers no request +// the beacon ever made. +func addTestBeaconRelayLogs( + t *testing.T, + record *chainReconciliationEvidence, + auditManifest *manifest, +) { + t.Helper() + + if auditManifest.ParticipationTerminalOutcomes == nil { + return + } + + // Memberships of one request legitimately share it, so each request is + // logged once however many outcomes name it. + requested := make(map[string]struct{}) + addRequest := func(startBlock uint64, previousEntry []byte) { + t.Helper() + + identity := relayEntryIdentity(startBlock, previousEntry) + if _, done := requested[identity]; done { + return + } + requested[identity] = struct{}{} + + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntryRequested", + testRelayRequestID(startBlock), + startBlock, + testFixtureSelectedGroupID, + previousEntry, + ) + } + + // Every request above selects the one group these fixtures sign under, and + // the audit holds a completed entry to the key that group is registered + // against, so the registration has to be in the bundle for a corroborated + // entry to settle at all. One registration covers every request: the + // receipt is the group's, not the round's. + registered := false + registerSigningGroup := func() { + t.Helper() + + if registered { + return + } + registered = true + + addTestGroupRegisteredReceipt( + t, + record, + testRandomBeaconAddress, + testFixtureSelectedGroupID, + new(bn256.G2).ScalarBaseMult( + big.NewInt(testActiveGroupSecret), + ).Marshal(), + uint64(1), + ) + } + + for _, outcome := range auditManifest.ParticipationTerminalOutcomes.Outcomes { + if outcome.Outcome != participation.TerminalOutcomeCompleted { + continue + } + switch outcome.Permit.Ceremony { + case participation.BeaconRelaySigning: + startBlock, _, previousEntry, _, err := participation. + ParseBeaconRelayEntryReference(outcome.Evidence.Reference) + if err != nil { + continue + } + addRequest(startBlock, previousEntry) + registerSigningGroup() + case participation.BeaconTimeoutReport: + startBlock, requestID, terminatedGroupID, err := participation. + ParseBeaconRelayTimeoutSettlementReference( + outcome.Evidence.Reference, + ) + if err != nil { + continue + } + // A terminated request signs over a previous entry like any + // other, but no permit record names it, so the fixture supplies a + // point of its own rather than deriving one. + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntryRequested", + requestID, + startBlock, + uint64(1), + new(bn256.G1).ScalarBaseMult( + new(big.Int).SetUint64(startBlock+1), + ).Marshal(), + ) + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntryTimedOut", + requestID, + startBlock+1, + terminatedGroupID, + ) + } + } +} + +func newValidEvidence(t *testing.T, auditManifest *manifest) evidenceInputs { + t.Helper() + + evidenceDir := t.TempDir() + write := func(name string, record interface{}) string { + path := filepath.Join(evidenceDir, name) + content, err := json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, content, 0o600); err != nil { + t.Fatal(err) + } + return path + } + + envelope := func(evidenceType string) evidenceEnvelope { + return evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: evidenceType, + GeneratedAt: time.Now().UTC(), + SnapshotAggregateSHA256: auditManifest.Snapshot.AggregateSHA256, + } + } + + chainRecord := &chainReconciliationEvidence{ + evidenceEnvelope: envelope("chain_reconciliation"), + EthereumChainID: "1", + } + for walletIndex := range auditManifest.TBTCActiveWallets { + wallet := &auditManifest.TBTCActiveWallets[walletIndex] + startBlock := uint64(1) + if auditManifest.ParticipationTerminalOutcomes != nil { + for _, outcome := range auditManifest.ParticipationTerminalOutcomes.Outcomes { + if outcome.Outcome == + participation.TerminalOutcomeCompleted && + outcome.Permit.Ceremony == participation.TBTCDKG && + outcome.Evidence.Reference == wallet.WalletStorageKey { + startBlock = outcome.Permit.CanonicalStartBlock + break + } + } + } + dkgResult, walletID, _ := newValidTBTCDKGResultEvidence( + t, + big.NewInt(int64(walletIndex+1)), + startBlock, + uint16(wallet.SigningGroupSize), + []uint8{}, + ) + if wallet.WalletID != walletID { + t.Fatalf( + "test wallet [%s] has ID [%s], but its synthetic accepted "+ + "event lineage derives [%s]", + wallet.WalletStorageKey, + wallet.WalletID, + walletID, + ) + } + chainRecord.Wallets = append(chainRecord.Wallets, tbtcWalletChainEvidence{ + WalletStorageKey: wallet.WalletStorageKey, + WalletID: walletID, + Registered: true, + DKGSettlement: "approved", + DKGResult: dkgResult, + }) + } + for _, quarantined := range auditManifest.TBTCQuarantinedOutputs { + walletID := quarantined.SignerWalletID + if walletID == "" { + walletID = quarantined.WalletID + } + chainRecord.Wallets = append(chainRecord.Wallets, tbtcWalletChainEvidence{ + WalletStorageKey: quarantined.WalletStorageKey, + WalletID: walletID, + Registered: false, + DKGSettlement: "none", + }) + } + for _, membership := range auditManifest.BeaconActiveMemberships { + chainRecord.BeaconGroups = append(chainRecord.BeaconGroups, struct { + GroupPublicKey string `json:"group_public_key"` + Registered bool `json:"registered"` + }{ + GroupPublicKey: membership.GroupPublicKey, + Registered: true, + }) + } + for _, quarantined := range auditManifest.BeaconQuarantinedOutputs { + chainRecord.BeaconGroups = append(chainRecord.BeaconGroups, struct { + GroupPublicKey string `json:"group_public_key"` + Registered bool `json:"registered"` + }{ + GroupPublicKey: quarantined.GroupPublicKey, + Registered: false, + }) + } + authenticateTestChainReconciliationEvidence(t, chainRecord) + addTestBeaconRelayLogs(t, chainRecord, auditManifest) + + bitcoinRecord := &bitcoinReconciliationEvidence{ + evidenceEnvelope: envelope("bitcoin_reconciliation"), + BitcoinNetwork: "mainnet", + Complete: true, + } + // A complete reconciliation enumerates every transaction the audited + // wallets signed, including the ones the node recorded as its own wallet + // actions' durable results. + if auditManifest.ParticipationTerminalOutcomes != nil { + for _, outcome := range auditManifest.ParticipationTerminalOutcomes.Outcomes { + if outcome.Outcome != participation.TerminalOutcomeCompleted || + outcome.Evidence.Kind != + participation.TerminalEvidenceBitcoinTransaction { + continue + } + bitcoinRecord.PendingTransactions = append( + bitcoinRecord.PendingTransactions, + struct { + TransactionHash string `json:"transaction_hash"` + State string `json:"state"` + }{ + TransactionHash: outcome.Evidence.Reference, + State: "mined", + }, + ) + } + } + + quiescenceEnvelope := envelope("quiescence_report") + quiescenceRecord := &quiescenceReportEvidence{ + evidenceEnvelope: quiescenceEnvelope, + ReleaseVersion: "v2.1.0", + ReleaseRevision: strings.Repeat("ef", 20), + ReleaseEpoch: participation.CompiledEpoch.String(), + CutoverBlock: 1_000, + QuiesceCause: "rollback drill", + } + if auditManifest.QuiescenceSnapshot != nil { + for _, permit := range auditManifest.QuiescenceSnapshot.ActivePermits { + outcome := participation.TerminalOutcomeCompleted + if auditManifest.ParticipationTerminalOutcomes != nil { + for _, terminal := range auditManifest.ParticipationTerminalOutcomes.Outcomes { + if terminal.Permit.Equal(permit) { + outcome = terminal.Outcome + break + } + } + } + quiescenceRecord.ActivePermitsAtQuiescence = append( + quiescenceRecord.ActivePermitsAtQuiescence, + quiescencePermitEvidence{ + Ceremony: string(permit.Ceremony), + Mode: permit.Mode, + CanonicalStartBlock: permit.CanonicalStartBlock, + WorkID: permit.WorkID, + PermitID: permit.PermitID, + Outcome: string(outcome), + }, + ) + } + } + + priorReaderRecord := &priorReaderCompatibilityEvidence{ + evidenceEnvelope: envelope("prior_reader_compatibility"), + PriorVersion: "v2.0.0", + PriorRevision: strings.Repeat("ab", 20), + PriorImageDigest: "sha256:" + strings.Repeat("11", 32), + ReleaseVersion: "v2.1.0", + ReleaseRevision: strings.Repeat("ef", 20), + ReleaseImageDigest: "sha256:" + strings.Repeat("22", 32), + } + for _, schema := range requiredPriorReaderSchemas { + priorReaderRecord.SchemaResults = append( + priorReaderRecord.SchemaResults, + struct { + Schema string `json:"schema"` + Compatible bool `json:"compatible"` + }{Schema: schema, Compatible: true}, + ) + } + + return evidenceInputs{ + chainReconciliation: write("chain-reconciliation", chainRecord), + bitcoinReconciliation: write("bitcoin-reconciliation", bitcoinRecord), + quiescenceReport: write("quiescence-report", quiescenceRecord), + priorReaderCompatibility: write( + "prior-reader-compatibility", + priorReaderRecord, + ), + } +} + +// testExpectedIdentity returns the expected-identity inputs matching the +// values newValidEvidence and newTestStorage write, so identity binding +// passes unless a test deliberately mismatches it. +func testExpectedIdentity() expectedIdentityInputs { + return expectedIdentityInputs{ + ethereumChainID: "1", + walletRegistryAddress: testWalletRegistryAddress, + randomBeaconAddress: testRandomBeaconAddress, + finalizedEthereumBlockNumber: testFinalizedEthereumBlock, + finalizedEthereumBlockHash: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + chainEvidencePublicKey: testChainEvidencePublicKey(), + bitcoinNetwork: "mainnet", + priorVersion: "v2.0.0", + priorRevision: strings.Repeat("ab", 20), + priorImageDigest: "sha256:" + strings.Repeat("11", 32), + releaseVersion: "v2.1.0", + releaseRevision: strings.Repeat("ef", 20), + releaseImageDigest: "sha256:" + strings.Repeat("22", 32), + releaseEpoch: participation.CompiledEpoch.String(), + cutoverBlock: 1_000, + maxEvidenceAge: 24 * time.Hour, + } +} + +func hasBlocker(auditManifest *manifest, fragment string) bool { + for _, blocker := range auditManifest.RollbackBlockers { + if strings.Contains(blocker, fragment) { + return true + } + } + return false +} + +func hasFinding(auditManifest *manifest, fragment string) bool { + for _, finding := range auditManifest.Findings { + if strings.Contains(finding, fragment) { + return true + } + } + return false +} + +func containsSubstring(values []string, fragment string) bool { + for _, value := range values { + if strings.Contains(value, fragment) { + return true + } + } + return false +} + +func updateQuiescenceReport( + t *testing.T, + evidence evidenceInputs, + update func(*quiescenceReportEvidence), +) { + t.Helper() + + content, err := os.ReadFile(evidence.quiescenceReport) + if err != nil { + t.Fatal(err) + } + + record := &quiescenceReportEvidence{} + if err := json.Unmarshal(content, record); err != nil { + t.Fatal(err) + } + update(record) + + content, err = json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(evidence.quiescenceReport, content, 0o600); err != nil { + t.Fatal(err) + } +} + +func setQuiescencePermits( + record *quiescenceReportEvidence, + permits []quiescencePermitEvidence, +) { + record.ActivePermitsAtQuiescence = append( + []quiescencePermitEvidence(nil), + permits..., + ) +} + +func TestRunAudit_ConsistentSnapshot(t *testing.T) { + storageDir := newTestStorage(t) + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if !auditManifest.Interpreted { + t.Error("expected the manifest to be interpreted") + } + if !auditManifest.Consistent { + t.Errorf( + "expected a consistent manifest, findings: %v", + auditManifest.Findings, + ) + } + + if got := len(auditManifest.BeaconActiveMemberships); got != 1 { + t.Fatalf("expected [1] active membership, got [%d]", got) + } + active := auditManifest.BeaconActiveMemberships[0] + if active.MemberIndex != 1 { + t.Errorf("expected active member index [1], got [%d]", active.MemberIndex) + } + if active.ChannelName != "test-channel" { + t.Errorf("unexpected channel name [%s]", active.ChannelName) + } + + if got := len(auditManifest.BeaconQuarantinedOutputs); got != 1 { + t.Fatalf("expected [1] quarantined output, got [%d]", got) + } + quarantined := auditManifest.BeaconQuarantinedOutputs[0] + if !quarantined.HasMembershipRecord { + t.Error("expected the quarantined output to have its membership record") + } + if quarantined.MemberIndex != 2 { + t.Errorf( + "expected quarantined member index [2], got [%d]", + quarantined.MemberIndex, + ) + } + if quarantined.ProtocolMode != "legacy" { + t.Errorf( + "expected the quarantined mode [legacy], got [%s]", + quarantined.ProtocolMode, + ) + } + if quarantined.CanonicalStartBlock != 900 { + t.Errorf( + "expected the canonical start block [900], got [%d]", + quarantined.CanonicalStartBlock, + ) + } + + // The active membership must never surface from the quarantine namespace + // and vice versa: the two interpreted sets are namespace-disjoint. + for _, namespace := range auditManifest.Namespaces { + if namespace.Name == "keystore/beacon-quarantine" && !namespace.Present { + t.Error("expected the quarantine namespace to be present") + } + for _, file := range namespace.Files { + if namespace.Name == "keystore/beacon" && + strings.Contains(file.Path, "beacon-quarantine") { + t.Errorf( + "quarantine file inventoried under the active "+ + "namespace: [%s]", + file.Path, + ) + } + } + } + + if auditManifest.Snapshot.AggregateSHA256 == "" { + t.Error("expected the snapshot aggregate checksum to be recorded") + } + if auditManifest.Snapshot.TotalFiles == 0 { + t.Error("expected the snapshot to count its inventoried files") + } + + // A consistent snapshot alone must never read as rollback-ready: every + // external evidence input is missing and each missing one is a blocker. + if auditManifest.RollbackBarrierReady { + t.Error("a consistent snapshot without evidence must not be barrier-ready") + } + if got := len(auditManifest.RollbackBlockers); got != 4 { + t.Errorf( + "expected [4] rollback blockers without evidence, got [%d]: %v", + got, + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_ValidEvidenceSatisfiesBarrier(t *testing.T) { + storageDir := newTestStorage(t) + + // The two-phase workflow: the first audit produces the snapshot identity + // and interpreted inventory the external evidence must bind to and cover; + // the second audit validates the produced evidence. + firstPass, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, firstPass), + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if !auditManifest.Consistent { + t.Fatalf( + "expected a consistent manifest, findings: %v", + auditManifest.Findings, + ) + } + if !auditManifest.RollbackBarrierReady { + t.Errorf( + "expected the barrier to be ready with valid evidence supplied, "+ + "blockers: %v", + auditManifest.RollbackBlockers, + ) + } + for _, record := range auditManifest.ExternalEvidence { + if !record.Supplied || !record.Valid || record.SHA256 == "" { + t.Errorf( + "expected evidence [%s] to be recorded as supplied and valid "+ + "with its checksum", + record.Name, + ) + } + } +} + +func TestRunAudit_PlaceholderEvidenceIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + auditManifest, err := runAudit( + storageDir, + testPassword, + newPlaceholderEvidence(t), + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("placeholder evidence must never authorize the barrier") + } + for _, record := range auditManifest.ExternalEvidence { + if !record.Supplied { + t.Errorf("expected evidence [%s] to be recorded as supplied", record.Name) + } + if record.Valid { + t.Errorf("expected placeholder evidence [%s] to be invalid", record.Name) + } + } + if !hasBlocker(auditManifest, "cannot be decoded") { + t.Errorf( + "expected undecodable-evidence blockers, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_EvidenceBoundToDifferentSnapshotIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + // Rebind the otherwise valid evidence to a different snapshot identity. + foreign := *firstPass + foreign.Snapshot.AggregateSHA256 = strings.Repeat("00", 32) + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, &foreign), + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("evidence bound to another snapshot must not authorize the barrier") + } + if !hasBlocker(auditManifest, "not to this audited snapshot") { + t.Errorf( + "expected a snapshot-binding blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_UncoveredPersistedGroupIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + // Drop the persisted beacon group from the reconciliation coverage. + uncovered := *firstPass + uncovered.BeaconActiveMemberships = nil + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, &uncovered), + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("an unreconciled persisted group must not authorize the barrier") + } + if !hasBlocker(auditManifest, "is not reconciled") { + t.Errorf( + "expected a coverage blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_IncompatiblePriorReaderIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + + // Rewrite the prior-reader record with one incompatible required schema. + record := &priorReaderCompatibilityEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "prior_reader_compatibility", + GeneratedAt: time.Now().UTC(), + SnapshotAggregateSHA256: firstPass.Snapshot.AggregateSHA256, + }, + PriorVersion: "v2.0.0", + PriorRevision: strings.Repeat("ab", 20), + } + for i, schema := range requiredPriorReaderSchemas { + record.SchemaResults = append(record.SchemaResults, struct { + Schema string `json:"schema"` + Compatible bool `json:"compatible"` + }{Schema: schema, Compatible: i != 0}) + } + content, err := json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.priorReaderCompatibility, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidence, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("an unreadable prior-reader schema must not authorize the barrier") + } + if !hasBlocker(auditManifest, "cannot read schema") { + t.Errorf( + "expected a prior-reader blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_QuarantinedClaimWithoutQuarantineStateIsBlocking( + t *testing.T, +) { + permits := []quiescencePermitEvidence{{ + Ceremony: "tbtc_dkg", + Mode: "security_v2", + CanonicalStartBlock: 1_000, + WorkID: strings.Repeat("d", 64), + PermitID: "1", + Outcome: "quarantined", + }} + storageDir := newTestStorageWithQuiescencePermits(t, permits) + + firstPass, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + + // Claim a quarantined tBTC DKG output; the snapshot's tbtc quarantine + // namespace holds none. + record := &quiescenceReportEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "quiescence_report", + GeneratedAt: time.Now().UTC(), + SnapshotAggregateSHA256: firstPass.Snapshot.AggregateSHA256, + }, + QuiesceCause: "rollback drill", + } + setQuiescencePermits( + record, + permits, + ) + content, err := json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.quiescenceReport, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidence, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error( + "an unevidenced quarantined-output claim must not authorize " + + "the barrier", + ) + } + if !hasBlocker( + auditManifest, + "the tbtc quarantine namespace holds none", + ) { + t.Errorf( + "expected a quarantine cross-check blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// TestRunAudit_BeaconQuarantineMembershipWithoutMetadataIsAFinding proves the +// audit names preserved key material that no audit record explains. +// +// This is the state a metadata write failure leaves behind: preservation +// attempts both records independently, so a namespace that took the share and +// refused its metadata keeps the share. The share is the half worth keeping and +// the node reports it as preserved, but nothing on disk says which ceremony +// generated it or why it was withheld — so the audit has to raise it rather +// than count it as an ordinary quarantined output. +// newBeaconQuarantineNamespace opens the beacon quarantine namespace of a fresh +// storage snapshot, returning the snapshot directory and the handle to write +// preserved records through. +func newBeaconQuarantineNamespace( + t *testing.T, +) (string, persistence.ProtectedHandle) { + t.Helper() + + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + t.Fatal(err) + } + + return storageDir, quarantineHandle +} + +// newBeaconQuarantineMembership builds the membership encoding a preserved +// beacon output carries, together with the group directory it belongs under. +// +// The channel name is what a caller varies to get two records under one group +// and seat whose stored bytes differ: it leaves the group public key alone, and +// two copies of one output's key material only need comparing when they agree on +// where they belong. +func newBeaconQuarantineMembership( + t *testing.T, + memberIndex group.MemberIndex, + groupSecret int64, + channelName string, +) ([]byte, string) { + t.Helper() + + membership := ®istry.Membership{ + Signer: newTestSigner(t, memberIndex, groupSecret), + ChannelName: channelName, + } + + membershipBytes, err := membership.Marshal() + if err != nil { + t.Fatal(err) + } + + return membershipBytes, groupPublicKeyHex(membership) +} + +// newBeaconQuarantineMetadata builds the audit record a preserved beacon output +// travels with, describing the seat of the given group. +func newBeaconQuarantineMetadata( + memberIndex group.MemberIndex, + groupPublicKey string, +) registry.QuarantinedSignerMetadata { + return registry.QuarantinedSignerMetadata{ + SchemaVersion: registry.QuarantineSchemaVersion, + ReleaseEpoch: participation.CompiledEpoch.String(), + ProtocolMode: "legacy", + CutoverBlock: 1_000, + CanonicalStartBlock: 900, + Ceremony: string(participation.BeaconDKG), + SeedHash: strings.Repeat("a", 64), + MemberIndex: uint8(memberIndex), + GroupPublicKey: groupPublicKey, + FailedOperation: "beacon_dkg_group_registration", + LastObservedBlock: 950, + } +} + +// storeBeaconQuarantineHandoffRecord writes one combined beacon handoff carrying +// exactly the metadata and membership it is given, so a test can put a handoff +// beside a standalone record that contradicts it. +func storeBeaconQuarantineHandoffRecord( + t *testing.T, + handle persistence.ProtectedHandle, + groupDirectory string, + memberIndex group.MemberIndex, + metadata registry.QuarantinedSignerMetadata, + membership []byte, +) { + t.Helper() + + handoff, err := json.Marshal(registry.QuarantinedSignerHandoff{ + SchemaVersion: registry.QuarantineHandoffSchemaVersion, + Metadata: metadata, + Membership: membership, + }) + if err != nil { + t.Fatal(err) + } + + if err := handle.Save( + handoff, + groupDirectory, + "/handoff_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } +} + +// TestRunAudit_BeaconQuarantineDisagreeingDuplicateKeyMaterialIsAFinding proves +// the audit reports two copies of one beacon output's key material that do not +// match, rather than picking one of them. +// +// The stake is higher here than for a wallet seat. A beacon group whose result +// was already accepted on chain loses usable threshold for every member that +// cannot produce its share, so which of two disagreeing copies is the real one +// decides whether that group still signs. It is not a question this tool can +// settle offline, and it is not one it may answer by preferring whichever record +// it happened to decode first. +func TestRunAudit_BeaconQuarantineDisagreeingDuplicateKeyMaterialIsAFinding( + t *testing.T, +) { + storageDir, quarantineHandle := newBeaconQuarantineNamespace(t) + + const memberIndex = group.MemberIndex(4) + + handoffMembership, groupDirectory := newBeaconQuarantineMembership( + t, + memberIndex, + 44, + "test-channel", + ) + + storeBeaconQuarantineHandoffRecord( + t, + quarantineHandle, + groupDirectory, + memberIndex, + newBeaconQuarantineMetadata(memberIndex, groupDirectory), + handoffMembership, + ) + + // The same group and the same seat, holding a different share. + standaloneMembership, standaloneDirectory := newBeaconQuarantineMembership( + t, + memberIndex, + 44, + "a-different-channel", + ) + if standaloneDirectory != groupDirectory { + t.Fatal("both copies must be filed under the same group") + } + if bytes.Equal(standaloneMembership, handoffMembership) { + t.Fatal("the two copies of the key material must differ") + } + if err := quarantineHandle.Save( + standaloneMembership, + groupDirectory, + "/membership_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "carries key material in both preserved forms and the two copies differ", + ) { + t.Errorf( + "expected a disagreeing-duplicate-key-material finding, "+ + "findings: %v", + auditManifest.Findings, + ) + } +} + +// TestRunAudit_BeaconQuarantineDisagreeingDuplicateMetadataIsAFinding proves the +// audit reports two accounts of one beacon output that describe it differently. +// +// The metadata is what lets a rollback reconcile a preserved share against the +// chain — the mode it was generated under, the anchor it was measured from, the +// operation that refused it. Two records disagreeing about those describe two +// different histories for one seat, and publishing either without saying so +// would hand the rollback a settled answer it does not have. +func TestRunAudit_BeaconQuarantineDisagreeingDuplicateMetadataIsAFinding( + t *testing.T, +) { + storageDir, quarantineHandle := newBeaconQuarantineNamespace(t) + + const memberIndex = group.MemberIndex(4) + + membership, groupDirectory := newBeaconQuarantineMembership( + t, + memberIndex, + 44, + "test-channel", + ) + + metadata := newBeaconQuarantineMetadata(memberIndex, groupDirectory) + storeBeaconQuarantineHandoffRecord( + t, + quarantineHandle, + groupDirectory, + memberIndex, + metadata, + membership, + ) + + // The same output, measured from a different anchor. + disagreeing := metadata + disagreeing.CanonicalStartBlock = metadata.CanonicalStartBlock + 17 + disagreeingBytes, err := json.Marshal(disagreeing) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + disagreeingBytes, + groupDirectory, + "/metadata_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "carries audit metadata in both preserved forms and they disagree", + ) { + t.Errorf( + "expected a disagreeing-duplicate-metadata finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_BeaconQuarantineMembershipWithoutMetadataIsAFinding( + t *testing.T, +) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + t.Fatal(err) + } + + // A group of its own, so the orphan is the only thing wrong with it: the + // fixture's quarantined output already occupies its own directory, and + // reusing that one would pair this membership with that metadata. + orphan := ®istry.Membership{ + Signer: newTestSigner(t, group.MemberIndex(4), 44), + ChannelName: "test-channel", + } + orphanBytes, err := orphan.Marshal() + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + orphanBytes, + groupPublicKeyHex(orphan), + "/membership_4", + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "has a membership record without audit metadata", + ) { + t.Errorf( + "expected an unexplained-key-material finding, findings: %v", + auditManifest.Findings, + ) + } + + // The orphan is not published as a quarantined output: the manifest's + // records are built from the metadata, and there is none to build from. + // The finding is the whole account of it, which is why it must be raised. + for _, output := range auditManifest.BeaconQuarantinedOutputs { + if output.MemberIndex == 4 { + t.Error( + "an output with no audit metadata was published as a " + + "quarantined output", + ) + } + } +} + +func TestRunAudit_TBTCQuarantineMetadataWithoutMembershipIsAFinding( + t *testing.T, +) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + []byte(`{`+ + `"schema_version":1,`+ + `"release_epoch":"security_v2_cutover",`+ + `"protocol_mode":"security_v2",`+ + `"cutover_block":100,`+ + `"canonical_start_block":900,`+ + `"ceremony":"tbtc_dkg",`+ + `"seed_hash":"aa",`+ + `"member_index":3,`+ + `"wallet_id":"bb",`+ + `"wallet_public_key_hash":"cc",`+ + `"failed_operation":"tbtc_dkg_signer_activation",`+ + `"last_observed_block":950,`+ + `"preserved_at":"2026-01-01T00:00:00Z"}`), + "orphaned-wallet-directory", + "/metadata_3", + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "tbtc quarantine output [orphaned-wallet-directory/3] has audit "+ + "metadata without a membership record", + ) { + t.Errorf( + "expected an orphaned-metadata finding, findings: %v", + auditManifest.Findings, + ) + } + if !hasFinding( + auditManifest, + "seed hash [aa] is not a canonical SHA-256 digest of 64 lowercase "+ + "hexadecimal characters", + ) { + t.Errorf( + "expected a malformed seed-hash finding, findings: %v", + auditManifest.Findings, + ) + } + + if got := len(auditManifest.TBTCQuarantinedOutputs); got != 1 { + t.Fatalf("expected [1] tbtc quarantined output, got [%d]", got) + } + if auditManifest.TBTCQuarantinedOutputs[0].HasMembershipRecord { + t.Error("expected the output to report its missing membership record") + } +} + +// storeTBTCQuarantineMembership writes one tBTC quarantine membership record +// that decodes the way the wallet registry loader decodes it, and returns the +// wallet storage key it was filed under. +// +// The record is assembled from the wire types the node persists rather than from +// a signer built in memory: the tECDSA key-share fixtures live in an internal +// package this command cannot import, and what the audit has to survive is the +// encoding on disk, not the struct behind it. The private key share carries the +// smallest payload that still decodes — the audit decodes it only to prove the +// record parses in full, and never looks inside — while the wallet public key is +// a real curve point, because every identity the audit derives from the record +// comes out of it. +func storeTBTCQuarantineMembership( + t *testing.T, + handle persistence.ProtectedHandle, + memberIndex group.MemberIndex, + walletScalar int64, +) string { + t.Helper() + + membership, walletStorageKey, _ := newTBTCQuarantineMembership( + t, + memberIndex, + walletScalar, + ) + + if err := handle.Save( + membership, + walletStorageKey, + "/membership_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + return walletStorageKey +} + +// newTBTCQuarantineMembership builds the membership encoding a preserved tBTC +// signer output carries, together with the wallet directory it belongs under +// and the wallet public key every identity in it is derived from. +func newTBTCQuarantineMembership( + t *testing.T, + memberIndex group.MemberIndex, + walletScalar int64, +) ([]byte, string, *ecdsa.PublicKey) { + t.Helper() + + return newTBTCQuarantineMembershipForOperators( + t, + memberIndex, + walletScalar, + []string{"0xAA", "0xBB", "0xCC"}, + ) +} + +// newTBTCQuarantineMembershipForOperators builds the same membership encoding +// for a named signing group, so a test can put two records under one wallet and +// seat whose stored bytes differ. +// +// The operators are what varies because they leave the wallet public key alone: +// the directory a record is filed under and the seat it names both come out of +// that key, and two copies of one output's key material only need comparing when +// they agree on where they belong. +func newTBTCQuarantineMembershipForOperators( + t *testing.T, + memberIndex group.MemberIndex, + walletScalar int64, + signingGroupOperators []string, +) ([]byte, string, *ecdsa.PublicKey) { + t.Helper() + + x, y := tecdsa.Curve.ScalarBaseMult(big.NewInt(walletScalar).Bytes()) + walletPublicKey := &ecdsa.PublicKey{Curve: tecdsa.Curve, X: x, Y: y} + + privateKeyShare, err := proto.Marshal(&tecdsapb.PrivateKeyShare{ + Data: &tecdsapb.LocalPartySaveData{ + EcdsaPub: &tecdsapb.LocalPartySaveData_ECPoint{ + X: x.Bytes(), + Y: y.Bytes(), + }, + }, + }) + if err != nil { + t.Fatal(err) + } + + membership, err := proto.Marshal(&tbtcpb.Signer{ + Wallet: &tbtcpb.Wallet{ + PublicKey: secp256k1.Marshal(walletPublicKey), + SigningGroupOperators: signingGroupOperators, + }, + SigningGroupMemberIndex: uint32(memberIndex), + PrivateKeyShare: privateKeyShare, + }) + if err != nil { + t.Fatal(err) + } + + // The registry strips the uncompressed-point prefix to keep the directory + // name usable, and the audit compares the directory against the key it + // derives from the record itself. + walletStorageKey := hex.EncodeToString( + secp256k1.Marshal(walletPublicKey), + )[2:] + + return membership, walletStorageKey, walletPublicKey +} + +// TestRunAudit_TBTCQuarantineHandoffIsAWholeOutput proves an output preserved +// as the single combined record is read as complete evidence, not as an +// incomplete pair. +// +// A namespace that will not take one of the two records preservation prefers +// leaves the node writing the output whole under a name of its own. What is on +// disk afterwards is one record rather than two, and it carries both the key +// material a rollback has to account for and the mode, anchor, ceremony, and +// refused operation that let the audit reconcile it against the chain. An audit +// that only knew the pair would report exactly the opposite of the truth here: +// a missing half, over an output nothing is missing from. +func TestRunAudit_TBTCQuarantineHandoffIsAWholeOutput(t *testing.T) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + t.Fatal(err) + } + + walletStorageKey := storeTBTCQuarantineHandoff( + t, + quarantineHandle, + group.MemberIndex(3), + 7, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + for _, orphan := range []string{ + "has a membership record without audit metadata", + "has audit metadata without a membership record", + } { + if hasFinding(auditManifest, orphan) { + t.Errorf( + "an output preserved whole must not be reported as an "+ + "incomplete pair, findings: %v", + auditManifest.Findings, + ) + } + } + + if got := len(auditManifest.TBTCQuarantinedOutputs); got != 1 { + t.Fatalf("expected [1] tbtc quarantined output, got [%d]", got) + } + quarantined := auditManifest.TBTCQuarantinedOutputs[0] + + if !quarantined.HasMembershipRecord { + t.Error( + "the combined record carries key material, so the output must " + + "report it as preserved", + ) + } + if !quarantined.HasHandoffRecord { + t.Error("expected the output to report how it was preserved") + } + if quarantined.WalletStorageKey != walletStorageKey { + t.Errorf( + "quarantined output filed under wallet [%s], expected [%s]", + quarantined.WalletStorageKey, + walletStorageKey, + ) + } + if quarantined.SignerWalletID == "" { + t.Error( + "the identity chain reconciliation matches must be derived from " + + "the key material the combined record carries", + ) + } + if quarantined.ProtocolMode != "legacy" { + t.Errorf( + "expected the quarantined mode [legacy], got [%s]", + quarantined.ProtocolMode, + ) + } + if quarantined.CanonicalStartBlock != 900 { + t.Errorf( + "expected the canonical start block [900], got [%d]", + quarantined.CanonicalStartBlock, + ) + } + if quarantined.FailedOperation != "tbtc_dkg_signer_activation" { + t.Errorf( + "expected the refused operation to travel with the material, "+ + "got [%s]", + quarantined.FailedOperation, + ) + } +} + +// storeTBTCQuarantineHandoff writes one tBTC quarantine output as the single +// combined record, carrying the same membership encoding the pair would have +// used, and returns the wallet storage key it was filed under. +func storeTBTCQuarantineHandoff( + t *testing.T, + handle persistence.ProtectedHandle, + memberIndex group.MemberIndex, + walletScalar int64, +) string { + t.Helper() + + membership, walletStorageKey, walletPublicKey := newTBTCQuarantineMembership( + t, + memberIndex, + walletScalar, + ) + + storeTBTCQuarantineHandoffRecord( + t, + handle, + walletStorageKey, + memberIndex, + newTBTCQuarantineMetadata(memberIndex, walletPublicKey), + membership, + ) + + return walletStorageKey +} + +// newTBTCQuarantineMetadata builds the audit record a preserved tBTC signer +// output travels with, describing the seat the given wallet key belongs to. +func newTBTCQuarantineMetadata( + memberIndex group.MemberIndex, + walletPublicKey *ecdsa.PublicKey, +) tbtc.QuarantinedSignerMetadata { + walletPublicKeyHash := bitcoin.PublicKeyHash(walletPublicKey) + + return tbtc.QuarantinedSignerMetadata{ + SchemaVersion: tbtc.QuarantineSchemaVersion, + ReleaseEpoch: participation.CompiledEpoch.String(), + ProtocolMode: "legacy", + CutoverBlock: 1_000, + CanonicalStartBlock: 900, + Ceremony: string(participation.TBTCDKG), + SeedHash: strings.Repeat("a", 64), + MemberIndex: uint8(memberIndex), + WalletPublicKeyHash: hex.EncodeToString(walletPublicKeyHash[:]), + FailedOperation: "tbtc_dkg_signer_activation", + LastObservedBlock: 950, + } +} + +// storeTBTCQuarantineHandoffRecord writes one combined tBTC handoff carrying +// exactly the metadata and membership it is given, so a test can put a handoff +// beside a standalone record that contradicts it. +func storeTBTCQuarantineHandoffRecord( + t *testing.T, + handle persistence.ProtectedHandle, + walletStorageKey string, + memberIndex group.MemberIndex, + metadata tbtc.QuarantinedSignerMetadata, + membership []byte, +) { + t.Helper() + + handoff, err := json.Marshal(tbtc.QuarantinedSignerHandoff{ + SchemaVersion: tbtc.QuarantineHandoffSchemaVersion, + Metadata: metadata, + Signer: membership, + }) + if err != nil { + t.Fatal(err) + } + + if err := handle.Save( + handoff, + walletStorageKey, + "/handoff_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } +} + +// newTBTCQuarantineNamespace opens the tBTC quarantine namespace of a fresh +// storage snapshot, returning the snapshot directory and the handle to write +// preserved records through. +func newTBTCQuarantineNamespace(t *testing.T) (string, persistence.ProtectedHandle) { + t.Helper() + + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + t.Fatal(err) + } + + return storageDir, quarantineHandle +} + +// TestRunAudit_TBTCQuarantineDisagreeingDuplicateKeyMaterialIsAFinding proves +// the audit reports two copies of one output's key material that do not match, +// rather than picking one of them. +// +// Preservation writes the record pair first and falls back on the combined +// handoff for whatever the namespace refused, so an output can legitimately +// leave a standalone membership beside a handoff carrying its own copy of that +// same share. While they agree, either answers for the seat. When they do not — +// a record left by an interrupted preservation, a name an operator's repair +// wrote over — the namespace holds two different accounts of what this seat is, +// and only one of them can be the share a rollback would have to settle. An +// audit that preferred whichever record it decoded first would answer that +// question with evidence it never compared, and would answer it the same way +// whichever copy was stale. +func TestRunAudit_TBTCQuarantineDisagreeingDuplicateKeyMaterialIsAFinding( + t *testing.T, +) { + storageDir, quarantineHandle := newTBTCQuarantineNamespace(t) + + const memberIndex = group.MemberIndex(3) + + handoffMembership, walletStorageKey, walletPublicKey := + newTBTCQuarantineMembership(t, memberIndex, 7) + + storeTBTCQuarantineHandoffRecord( + t, + quarantineHandle, + walletStorageKey, + memberIndex, + newTBTCQuarantineMetadata(memberIndex, walletPublicKey), + handoffMembership, + ) + + // The same wallet and the same seat, holding a different share. + standaloneMembership, _, _ := newTBTCQuarantineMembershipForOperators( + t, + memberIndex, + 7, + []string{"0xDD", "0xEE", "0xFF"}, + ) + if bytes.Equal(standaloneMembership, handoffMembership) { + t.Fatal("the two copies of the key material must differ") + } + if err := quarantineHandle.Save( + standaloneMembership, + walletStorageKey, + "/membership_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "carries key material in both preserved forms and the two copies differ", + ) { + t.Errorf( + "expected a disagreeing-duplicate-key-material finding, "+ + "findings: %v", + auditManifest.Findings, + ) + } +} + +// TestRunAudit_TBTCQuarantineDisagreeingDuplicateMetadataIsAFinding proves the +// audit reports two accounts of one output that describe it differently. +// +// The metadata is the whole of what lets a rollback reconcile a preserved share +// against the chain: the mode it was generated under, the anchor it was measured +// from, the operation that refused it. Two records disagreeing about those +// describe two different histories for one seat, and an audit that published +// either of them without saying so would be handing the rollback a settled +// answer it does not have. +func TestRunAudit_TBTCQuarantineDisagreeingDuplicateMetadataIsAFinding( + t *testing.T, +) { + storageDir, quarantineHandle := newTBTCQuarantineNamespace(t) + + const memberIndex = group.MemberIndex(3) + + membership, walletStorageKey, walletPublicKey := + newTBTCQuarantineMembership(t, memberIndex, 7) + + metadata := newTBTCQuarantineMetadata(memberIndex, walletPublicKey) + storeTBTCQuarantineHandoffRecord( + t, + quarantineHandle, + walletStorageKey, + memberIndex, + metadata, + membership, + ) + + // The same output, measured from a different anchor. + disagreeing := metadata + disagreeing.CanonicalStartBlock = metadata.CanonicalStartBlock + 17 + disagreeingBytes, err := json.Marshal(disagreeing) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + disagreeingBytes, + walletStorageKey, + "/metadata_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "carries audit metadata in both preserved forms and they disagree", + ) { + t.Errorf( + "expected a disagreeing-duplicate-metadata finding, findings: %v", + auditManifest.Findings, + ) + } +} + +// TestRunAudit_TBTCQuarantineAgreeingDuplicateFormsAreOneOutput proves the +// duplicate check does not turn the layout preservation actually writes into a +// finding. +// +// A namespace that took the membership, refused the metadata, and then took the +// handoff holds both forms of the same output, and every copy in it agrees. That +// is a preserved output with nothing wrong with it, filed once. Only a +// comparison can tell it apart from the contradictory case, which is the reason +// the comparison is made rather than assumed either way. +func TestRunAudit_TBTCQuarantineAgreeingDuplicateFormsAreOneOutput( + t *testing.T, +) { + storageDir, quarantineHandle := newTBTCQuarantineNamespace(t) + + const memberIndex = group.MemberIndex(3) + + membership, walletStorageKey, walletPublicKey := + newTBTCQuarantineMembership(t, memberIndex, 7) + + metadata := newTBTCQuarantineMetadata(memberIndex, walletPublicKey) + storeTBTCQuarantineHandoffRecord( + t, + quarantineHandle, + walletStorageKey, + memberIndex, + metadata, + membership, + ) + + if err := quarantineHandle.Save( + membership, + walletStorageKey, + "/membership_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + metadataBytes, err := json.Marshal(metadata) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + metadataBytes, + walletStorageKey, + "/metadata_"+fmt.Sprint(memberIndex), + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + for _, disagreement := range []string{ + "carries key material in both preserved forms", + "carries audit metadata in both preserved forms", + } { + if hasFinding(auditManifest, disagreement) { + t.Errorf( + "copies that agree must not be reported as a disagreement, "+ + "findings: %v", + auditManifest.Findings, + ) + } + } + + if got := len(auditManifest.TBTCQuarantinedOutputs); got != 1 { + t.Fatalf( + "one seat preserved in both forms is one output, got [%d]", + got, + ) + } +} + +// TestRunAudit_TBTCQuarantineMembershipWithoutMetadataIsAFinding proves the +// audit raises the other half of the incomplete-pair check: preserved key +// material with no audit record explaining it. +// +// The two orphans are opposite states. Metadata without a membership is a share +// that was lost; a membership without metadata is a share that survived while +// the record naming it did not, which is exactly what a node leaves behind when +// the metadata write of a quarantine is refused. Nothing in the record itself +// says which mode, anchor, or ceremony produced it, so a rollback cannot +// reconcile it against the chain and the audit has to say so. +// +// It runs through the stored namespace rather than against the validator so the +// whole path is covered: enumeration, decoding the membership the way the +// wallet registry loader does, pairing it with the metadata that is not there, +// and the blocking verdict. +func TestRunAudit_TBTCQuarantineMembershipWithoutMetadataIsAFinding( + t *testing.T, +) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + t.Fatal(err) + } + + walletStorageKey := storeTBTCQuarantineMembership( + t, + quarantineHandle, + group.MemberIndex(3), + 7, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + fmt.Sprintf( + "tbtc quarantine output [%s/3] has a membership record without "+ + "audit metadata", + walletStorageKey, + ), + ) { + t.Errorf( + "expected an unexplained-key-material finding, findings: %v", + auditManifest.Findings, + ) + } + + // The finding has to come from a record the audit actually read. A stored + // membership it could not decode, or filed under a wallet it could not + // match, raises its own finding — and would make the orphan finding above + // pass for reasons that have nothing to do with the missing metadata. + for _, finding := range auditManifest.Findings { + if strings.Contains(finding, "cannot be decoded") || + strings.Contains(finding, "not the wallet its directory claims") || + strings.Contains(finding, "not the member its file name claims") || + strings.Contains(finding, "has an unknown name") { + t.Errorf("the stored membership was not read as one: %s", finding) + } + } + + // The orphan is not published as a quarantined output: the manifest's + // records are built from the metadata, and there is none to build from. + // The finding is the whole account of it, which is why it must be raised. + for _, output := range auditManifest.TBTCQuarantinedOutputs { + if output.WalletStorageKey == walletStorageKey { + t.Error( + "an output with no audit metadata was published as a " + + "quarantined output", + ) + } + } +} + +// TestValidateTBTCQuarantineEntry_MembershipWithoutMetadataIsAFinding covers the +// same incomplete pair at the validator, where the record's identity fields can +// be varied without re-deriving a wallet for each one. +func TestValidateTBTCQuarantineEntry_MembershipWithoutMetadataIsAFinding( + t *testing.T, +) { + run := &auditRun{ + manifest: &manifest{}, + expected: testExpectedIdentity(), + } + + validateTBTCQuarantineEntry( + run, + &tbtcQuarantineEntry{ + directory: "orphaned-wallet-directory", + memberSuffix: "3", + signer: &tbtc.SignerAuditRecord{ + WalletStorageKey: "orphaned-wallet-directory", + WalletID: "aa", + WalletPublicKeyHash: "bb", + MemberIndex: group.MemberIndex(3), + SigningGroupSize: 2, + }, + }, + map[string]struct{}{}, + ) + + if !hasFinding( + run.manifest, + "tbtc quarantine output [orphaned-wallet-directory/3] has a "+ + "membership record without audit metadata", + ) { + t.Errorf( + "expected an unexplained-key-material finding, findings: %v", + run.manifest.Findings, + ) + } +} + +// TestValidateTBTCQuarantineEntry_CompletePairIsNotAFinding proves the +// incomplete-pair checks do not fire on a record whose halves both landed, so +// the finding above reports the missing record rather than every quarantine. +func TestValidateTBTCQuarantineEntry_CompletePairIsNotAFinding(t *testing.T) { + run := &auditRun{ + manifest: &manifest{}, + expected: testExpectedIdentity(), + } + + validateTBTCQuarantineEntry( + run, + &tbtcQuarantineEntry{ + directory: "paired-wallet-directory", + memberSuffix: "3", + signer: &tbtc.SignerAuditRecord{ + WalletStorageKey: "paired-wallet-directory", + WalletID: "aa", + WalletPublicKeyHash: "bb", + MemberIndex: group.MemberIndex(3), + SigningGroupSize: 2, + }, + metadata: &tbtc.QuarantinedSignerMetadata{ + SchemaVersion: tbtc.QuarantineSchemaVersion, + MemberIndex: 3, + }, + }, + map[string]struct{}{}, + ) + + for _, finding := range run.manifest.Findings { + if strings.Contains(finding, "membership record without audit metadata") || + strings.Contains(finding, "audit metadata without a membership record") { + t.Errorf("a complete pair raised an incomplete-pair finding: %s", finding) + } + } +} + +func TestRunAudit_UndecodableTBTCQuarantineMembershipIsAFinding( + t *testing.T, +) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + []byte("not a signer record"), + "some-wallet-directory", + "/membership_1", + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding( + auditManifest, + "tbtc quarantine membership [some-wallet-directory/membership_1] "+ + "cannot be decoded", + ) { + t.Errorf( + "expected a quarantine decode finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_UnreadableEvidenceIsAnError(t *testing.T) { + storageDir := newTestStorage(t) + + _, err := runAudit( + storageDir, + testPassword, + evidenceInputs{ + chainReconciliation: filepath.Join(t.TempDir(), "does-not-exist"), + }, + testExpectedIdentity(), + ) + if err == nil { + t.Error("expected an error for an unreadable evidence reference") + } +} + +func TestRunAudit_MetadataWithoutMembershipIsAFinding(t *testing.T) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + []byte(`{"schema_version":1,"member_index":3}`), + "orphaned-group-directory", + "/metadata_3", + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding(auditManifest, "audit metadata without a membership") { + t.Errorf( + "expected an orphaned-metadata finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_QuarantineMetadataCrossChecks(t *testing.T) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + t.Fatal(err) + } + quarantine := registry.NewQuarantine( + context.Background(), + &testutils.MockLogger{}, + quarantineHandle, + ) + + // Written through the production quarantine path, so directory, group, + // and member all pair up — but the metadata's own fields contradict the + // release identity and the cutover arithmetic. + if _, err := quarantine.Preserve( + ®istry.Membership{ + Signer: newTestSigner(t, group.MemberIndex(4), 44), + ChannelName: "test-channel", + }, + registry.QuarantinedSignerMetadata{ + ReleaseEpoch: "some_other_epoch", + ProtocolMode: "security_v2", + CutoverBlock: 1_000, + CanonicalStartBlock: 900, + Ceremony: "not_a_ceremony", + SeedHash: strings.Repeat("b", 64), + FailedOperation: "beacon_dkg_result_publication", + LastObservedBlock: 950, + }, + nil, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + for _, fragment := range []string{ + "was written by release epoch [some_other_epoch]", + "names ceremony [not_a_ceremony]", + "claims mode [security_v2] with canonical anchor [900] before " + + "cutover block [1000]", + } { + if !hasFinding(auditManifest, fragment) { + t.Errorf( + "expected a finding containing [%s], findings: %v", + fragment, + auditManifest.Findings, + ) + } + } +} + +func TestRunAudit_QuarantinedGroupAlsoActiveIsAFinding(t *testing.T) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + t.Fatal(err) + } + quarantine := registry.NewQuarantine( + context.Background(), + &testutils.MockLogger{}, + quarantineHandle, + ) + + // Group secret 42 is the group the fixture also activates. + if _, err := quarantine.Preserve( + ®istry.Membership{ + Signer: newTestSigner(t, group.MemberIndex(5), 42), + ChannelName: "test-channel", + }, + registry.QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + ProtocolMode: "legacy", + CutoverBlock: 1_000, + CanonicalStartBlock: 900, + Ceremony: "beacon_dkg", + SeedHash: strings.Repeat("c", 64), + FailedOperation: "beacon_dkg_result_publication", + LastObservedBlock: 950, + }, + nil, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding(auditManifest, "also present in the active namespace") { + t.Errorf( + "expected an active-overlap finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_MisplacedActiveMembershipIsAFinding(t *testing.T) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + activeHandle, err := diskStorage.InitializeKeyStorePersistence("beacon") + if err != nil { + t.Fatal(err) + } + + misplaced := ®istry.Membership{ + Signer: newTestSigner(t, group.MemberIndex(6), 45), + ChannelName: "test-channel", + } + misplacedBytes, err := misplaced.Marshal() + if err != nil { + t.Fatal(err) + } + if err := activeHandle.Save( + misplacedBytes, + "not-the-group-directory", + "/membership_7", + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding(auditManifest, "not the group its directory claims") { + t.Errorf( + "expected a directory-mismatch finding, findings: %v", + auditManifest.Findings, + ) + } + if !hasFinding(auditManifest, "not the member its file name claims") { + t.Errorf( + "expected a member-name finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_UndecodableTBTCRecordIsAFinding(t *testing.T) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + tbtcHandle, err := diskStorage.InitializeKeyStorePersistence("tbtc") + if err != nil { + t.Fatal(err) + } + if err := tbtcHandle.Save( + []byte("not a signer record"), + "some-wallet-directory", + "/membership_1", + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding(auditManifest, "wallet registry loader") { + t.Errorf( + "expected a tbtc decode finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_UnexpectedNamespaceIsAFinding(t *testing.T) { + storageDir := newTestStorage(t) + + if err := os.MkdirAll( + filepath.Join(storageDir, "keystore", "rogue-namespace"), + 0o700, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit(storageDir, testPassword, evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Consistent { + t.Error("expected an inconsistent manifest") + } + if !hasFinding(auditManifest, "unexpected entry [rogue-namespace]") { + t.Errorf( + "expected an unexpected-entry finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestRunAudit_WithoutPasswordInventoriesOnly(t *testing.T) { + storageDir := newTestStorage(t) + + auditManifest, err := runAudit(storageDir, "", evidenceInputs{}, testExpectedIdentity()) + if err != nil { + t.Fatal(err) + } + + if auditManifest.Interpreted { + t.Error("expected an uninterpreted manifest without the password") + } + if auditManifest.Consistent { + t.Error("an uninterpreted manifest must not classify as consistent") + } + if auditManifest.RollbackBarrierReady { + t.Error("an uninterpreted manifest must not be barrier-ready") + } + if len(auditManifest.BeaconActiveMemberships) != 0 { + t.Error("expected no interpreted memberships without the password") + } + + var beaconFiles, quarantineFiles int + for _, namespace := range auditManifest.Namespaces { + switch namespace.Name { + case "keystore/beacon": + beaconFiles = len(namespace.Files) + case "keystore/beacon-quarantine": + quarantineFiles = len(namespace.Files) + } + } + if beaconFiles == 0 { + t.Error("expected the active namespace inventory to list files") + } + if quarantineFiles == 0 { + t.Error("expected the quarantine namespace inventory to list files") + } +} + +func TestRunAudit_MissingExpectedIdentityIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, firstPass), + expectedIdentityInputs{}, + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("missing expected identities must not authorize the barrier") + } + for _, fragment := range []string{ + "expected Ethereum chain ID is not supplied", + "expected WalletRegistry address is not supplied", + "expected RandomBeacon address is not supplied", + "expected Bitcoin network is not supplied", + "expected prior version is not supplied", + "expected prior revision is not supplied", + "expected prior image digest is not supplied", + "expected release version is not supplied", + "expected release revision is not supplied", + "expected release image digest is not supplied", + "expected release epoch is not supplied", + "expected cutover block is not supplied", + "no evidence freshness bound is supplied", + } { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected the [%s] blocker, blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } +} + +func TestRunAudit_MismatchedExpectedIdentityIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + // The evidence itself is schema-valid and records chain [1], network + // [mainnet], version [v2.0.0]; the audit expects a different operational + // target for each. + mismatched := expectedIdentityInputs{ + ethereumChainID: "11155111", + walletRegistryAddress: "0x0000000000000000000000000000000000000001", + randomBeaconAddress: "0x0000000000000000000000000000000000000002", + bitcoinNetwork: "testnet", + priorVersion: "v1.9.9", + priorRevision: strings.Repeat("cd", 20), + maxEvidenceAge: 24 * time.Hour, + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, firstPass), + mismatched, + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("mismatched identities must not authorize the barrier") + } + for _, fragment := range []string{ + "reconciled against Ethereum chain [1], expected [11155111]", + "the WalletRegistry address [" + testWalletRegistryAddress + "]", + "the RandomBeacon address [" + testRandomBeaconAddress + "]", + "reconciled against Bitcoin network [mainnet], expected [testnet]", + "tested prior version [v2.0.0], expected [v1.9.9]", + "tested prior revision", + } { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected the [%s] blocker, blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } +} + +func TestRunAudit_StaleEvidenceIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + // The evidence was generated a moment ago; a one-nanosecond freshness + // bound makes every record stale. + stale := testExpectedIdentity() + stale.maxEvidenceAge = time.Nanosecond + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, firstPass), + stale, + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("stale evidence must not authorize the barrier") + } + if !hasBlocker(auditManifest, "evidence freshness bound") { + t.Errorf( + "expected a freshness blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_UncoveredQuarantinedOutputIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + // Evidence generated from a manifest stripped of the quarantined output + // reconciles the active state only. + uncovered := *firstPass + uncovered.BeaconQuarantinedOutputs = nil + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, &uncovered), + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("an unreconciled quarantined output must not authorize the barrier") + } + if !hasBlocker(auditManifest, "quarantined beacon group") || + !hasBlocker(auditManifest, "is not reconciled") { + t.Errorf( + "expected a quarantined-coverage blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_EvidenceForStateTheSnapshotDoesNotHoldIsBlocking( + t *testing.T, +) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + // Evidence generated from a manifest holding one extra beacon group + // reconciles state this snapshot does not hold. + padded := *firstPass + padded.BeaconActiveMemberships = append( + append( + []beaconMembershipRecord{}, + firstPass.BeaconActiveMemberships..., + ), + beaconMembershipRecord{ + GroupPublicKey: strings.Repeat("ee", 64), + MemberIndex: 9, + ChannelName: "foreign-channel", + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, &padded), + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("evidence for foreign state must not authorize the barrier") + } + if !hasBlocker(auditManifest, "that the snapshot does not hold") { + t.Errorf( + "expected a foreign-state blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_RegisteredQuarantinedOnlyShareIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + + // Flip the quarantined group's chain state to registered: the share + // exists only in quarantine, so a prior binary would run the group + // without it. + content, err := os.ReadFile(evidence.chainReconciliation) + if err != nil { + t.Fatal(err) + } + record := &chainReconciliationEvidence{} + if err := json.Unmarshal(content, record); err != nil { + t.Fatal(err) + } + quarantinedGroup := firstPass.BeaconQuarantinedOutputs[0].GroupPublicKey + for i := range record.BeaconGroups { + if record.BeaconGroups[i].GroupPublicKey == quarantinedGroup { + record.BeaconGroups[i].Registered = true + } + } + content, err = json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.chainReconciliation, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error( + "a registered group with a quarantine-only share must not " + + "authorize the barrier", + ) + } + if !hasBlocker(auditManifest, "preserved only in quarantine") { + t.Errorf( + "expected a quarantine-only-share blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_QuarantinedClaimWithMismatchedWorkIdentityIsBlocking( + t *testing.T, +) { + permits := []quiescencePermitEvidence{{ + Ceremony: "beacon_dkg", + Mode: "legacy", + CanonicalStartBlock: 900, + WorkID: strings.Repeat("f", 64), + PermitID: "2", + Outcome: "quarantined", + }} + storageDir := newTestStorageWithQuiescencePermits(t, permits) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + + // The snapshot's only quarantined beacon output has this ceremony, mode, + // anchor, and member index. The report substitutes another DKG seed hash; + // those classifications and local member alone must not vouch for work + // from another chain event. + record := &quiescenceReportEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "quiescence_report", + GeneratedAt: time.Now().UTC(), + SnapshotAggregateSHA256: firstPass.Snapshot.AggregateSHA256, + }, + QuiesceCause: "rollback drill", + } + setQuiescencePermits( + record, + permits, + ) + content, err := json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.quiescenceReport, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error( + "a work-mismatched quarantined-output claim must not " + + "authorize the barrier", + ) + } + if !hasBlocker( + auditManifest, + "the beacon quarantine namespace holds none matching", + ) { + t.Errorf( + "expected an exact-permit-matching blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// TestRunAudit_DuplicateCompletedPermitIdentityIsBlocking proves permit +// uniqueness is enforced for completed outcomes as well as quarantined ones. +// Otherwise a report with the expected entry count could repeat one completed +// permit while omitting a different permit that was still active. +func TestRunAudit_DuplicateCompletedPermitIdentityIsBlocking(t *testing.T) { + completed := quiescencePermitEvidence{ + Ceremony: string(participation.TBTCSigning), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 900, + WorkID: "wallet-action-1", + PermitID: "wallet-action-1", + Outcome: "completed", + } + storageDir := newTestStorageWithQuiescencePermits( + t, + []quiescencePermitEvidence{completed}, + ) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + setQuiescencePermits( + record, + []quiescencePermitEvidence{completed}, + ) + record.ActivePermitsAtQuiescence = []quiescencePermitEvidence{ + completed, + completed, + } + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error( + "a duplicated completed permit must not authorize the barrier", + ) + } + if !hasBlocker( + auditManifest, + "duplicates the full permit identity first recorded by entry [0]", + ) { + t.Errorf( + "expected a duplicate-permit blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// TestRunAudit_MalformedQuiescencePermitIdentitiesAreBlocking proves the audit +// does not treat aliases, truncated seed hashes, or separator-bearing labels +// as exact local permit identities. +func TestRunAudit_MalformedQuiescencePermitIdentitiesAreBlocking(t *testing.T) { + permits := []quiescencePermitEvidence{ + { + Ceremony: string(participation.TBTCDKG), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 900, + WorkID: strings.Repeat("A", 64), + PermitID: "01", + Outcome: "completed", + }, + { + Ceremony: string( + participation.BeaconRelaySigning, + ), + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: participation.BeaconRelayWorkID(1), + PermitID: "member-1", + Outcome: "completed", + }, + { + Ceremony: string(participation.TBTCSigning), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 900, + WorkID: "wallet/action", + PermitID: "wallet~1", + Outcome: "completed", + }, + } + storageDir := newTestStorageWithQuiescencePermits(t, permits) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + setQuiescencePermits(record, permits) + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("malformed permit identities must not authorize the barrier") + } + for _, fragment := range []string{ + "is not a canonical SHA-256 seed hash", + "local permit identity [01] is not a canonical protocol member index", + "local permit identity [member-1] is not a canonical protocol member index", + "chain work identity [wallet/action] is not a stable evidence identifier", + "local permit identity [wallet~1] is not a stable evidence identifier", + } { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected identity-format blocker [%s], blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } +} + +// TestRunAudit_CompleteNonemptyQuiescenceInventorySatisfiesBarrier proves a +// one-to-one terminal-outcome list with matching aggregate counts remains an +// authorizing record. +func TestRunAudit_CompleteNonemptyQuiescenceInventorySatisfiesBarrier( + t *testing.T, +) { + permits := []quiescencePermitEvidence{ + { + Ceremony: string( + participation.TBTCSigning, + ), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 999, + WorkID: "wallet-action-legacy", + PermitID: "wallet-action-legacy", + Outcome: "completed", + }, + { + Ceremony: string( + participation.BeaconRelaySigning, + ), + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: participation.BeaconRelayWorkID(1_000), + PermitID: "2", + Outcome: "completed", + }, + } + storageDir := newTestStorageWithQuiescencePermits(t, permits) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + setQuiescencePermits(record, permits) + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if !auditManifest.RollbackBarrierReady { + t.Errorf( + "expected a complete nonempty quiescence inventory to authorize "+ + "the barrier, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_ExternalCompletedCannotReplaceNodeExhaustedOutcome( + t *testing.T, +) { + permit := quiescencePermitEvidence{ + Ceremony: string(participation.TBTCSigning), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 999, + WorkID: "wallet-action-exhausted", + PermitID: "wallet-exhausted", + Outcome: string(participation.TerminalOutcomeExhausted), + } + storageDir := newTestStorageWithQuiescencePermits( + t, + []quiescencePermitEvidence{permit}, + ) + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + evidence := newValidEvidence(t, firstPass) + + // An external generator labels the permit completed and supplies an + // unrelated successful Bitcoin transaction. Neither can replace the + // ceremony owner's durable no-threshold outcome. + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + record.ActivePermitsAtQuiescence[0].Outcome = + string(participation.TerminalOutcomeCompleted) + }, + ) + bitcoinRecord := &bitcoinReconciliationEvidence{} + content, err := os.ReadFile(evidence.bitcoinReconciliation) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(content, bitcoinRecord); err != nil { + t.Fatal(err) + } + bitcoinRecord.PendingTransactions = append( + bitcoinRecord.PendingTransactions, + struct { + TransactionHash string `json:"transaction_hash"` + State string `json:"state"` + }{ + TransactionHash: strings.Repeat("ab", 32), + State: "mined", + }, + ) + content, err = json.MarshalIndent(bitcoinRecord, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.bitcoinReconciliation, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if auditManifest.RollbackBarrierReady { + t.Error( + "an external completed label and unrelated successful transaction " + + "must not replace the node-authored exhausted outcome", + ) + } + if !hasBlocker( + auditManifest, + "claims terminal outcome [completed], but the node-authored journal records [exhausted]", + ) { + t.Errorf( + "expected node-authored outcome mismatch blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestRunAudit_NodeCompletedDKGRequiresPersistedSigner(t *testing.T) { + permit := quiescencePermitEvidence{ + Ceremony: string(participation.TBTCDKG), + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: strings.Repeat("d", 64), + PermitID: "1", + Outcome: string(participation.TerminalOutcomeCompleted), + } + storageDir := newTestStorageWithQuiescencePermits( + t, + []quiescencePermitEvidence{permit}, + ) + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + evidence := newValidEvidence(t, firstPass) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if auditManifest.RollbackBarrierReady { + t.Error( + "a node-authored completed DKG outcome without its persisted " + + "signer must not authorize the barrier", + ) + } + if !hasFinding( + auditManifest, + "active tbtc namespace holds no matching signer", + ) { + t.Errorf( + "expected persisted-signer corroboration finding, findings: %v", + auditManifest.Findings, + ) + } +} + +func TestValidateNodeTerminalOutcomes_DKGCompletionIsMembershipExact( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + workID := strings.Repeat("d", 64) + permits := []participation.PermitSnapshot{ + { + Ceremony: participation.TBTCDKG, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: workID, + PermitID: "1", + IdentityBound: true, + }, + { + Ceremony: participation.TBTCDKG, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: workID, + PermitID: "2", + IdentityBound: true, + }, + } + + newManifest := func( + secondReference string, + secondMembership group.MemberIndex, + ) *manifest { + return &manifest{ + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + CapturedAt: capturedAt, + ActivePermits: permits, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permits[0], + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: group.MemberIndex(1), + }, + }, + { + RecordedAt: time.Now().UTC(), + Permit: permits[1], + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: secondReference, + MembershipIndex: secondMembership, + }, + }, + }, + }, + TBTCActiveWallets: []tbtcWalletRecord{ + { + WalletStorageKey: "wallet-storage-key", + WalletID: "wallet-id", + MemberIndexes: []uint8{1}, + SigningGroupSize: 2, + }, + }, + } + } + + tests := map[string]struct { + manifest *manifest + expectedFinding string + }{ + "missing second persisted membership": { + manifest: newManifest( + "wallet-storage-key", + group.MemberIndex(2), + ), + expectedFinding: "membership [2], but the active tbtc namespace holds no matching signer", + }, + "one persisted membership reused by two permits": { + manifest: newManifest( + "wallet-storage-key", + group.MemberIndex(1), + ), + expectedFinding: "claim the same persisted signer [wallet-storage-key] membership [1]", + }, + "wallet ID alias cannot replace the signer storage record": { + manifest: newManifest( + "wallet-id", + group.MemberIndex(1), + ), + expectedFinding: "persisted signer [wallet-id] membership [1], but the active tbtc namespace holds no matching signer", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + violations := validateNodeTerminalOutcomes(test.manifest) + for _, violation := range violations { + if strings.Contains(violation, test.expectedFinding) { + return + } + } + t.Fatalf( + "expected exact-membership violation containing [%s], got: %v", + test.expectedFinding, + violations, + ) + }) + } +} + +// TestValidateNodeTerminalOutcomes_OperatedMembershipsAreReconciled proves the +// offline audit holds a permit's operated seats to both records that carry them. +// +// The seats are fixed at issuance and copied into two places: the gate snapshot +// captured while the permit was live, and the journal record written when it +// closed. A reader building a fleet seat ownership map picks one of the two, and +// nothing else in either record constrains the field — an entry whose seats were +// widened, narrowed, or reassigned after the fact still names a real ceremony, a +// real permit and a real outcome. So the audit reconciles the two copies against +// each other, holds each to the shape its ceremony can have, and holds the +// transcript to the copy it travelled with. +func TestValidateNodeTerminalOutcomes_OperatedMembershipsAreReconciled( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + workID := strings.Repeat("d", 64) + + // Two DKG seats of one ceremony, both surviving into a two-member final + // group: DKG seat 2 holds final seat 1 and DKG seat 3 holds final seat 2. + newPermit := func( + permitID string, + operated participation.MemberIndexes, + ) participation.PermitSnapshot { + return participation.PermitSnapshot{ + Ceremony: participation.TBTCDKG, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: workID, + PermitID: permitID, + IdentityBound: true, + OperatedMembers: operated, + } + } + newRecord := func( + permit participation.PermitSnapshot, + membership group.MemberIndex, + permitSeat group.MemberIndex, + ) participation.TerminalOutcomeRecord { + return participation.TerminalOutcomeRecord{ + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: membership, + Contribution: testTranscriptContribution( + participation.TBTCDKG, + membership, + permitSeat, + ), + }, + } + } + newManifest := func( + inventory []participation.PermitSnapshot, + outcomes []participation.TerminalOutcomeRecord, + ) *manifest { + return &manifest{ + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + SchemaVersion: participation.QuiescenceSnapshotSchemaVersion, + CapturedAt: capturedAt, + ActivePermits: inventory, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: outcomes, + }, + TBTCActiveWallets: []tbtcWalletRecord{ + { + WalletStorageKey: "wallet-storage-key", + WalletID: "wallet-id", + MemberIndexes: []uint8{1, 2}, + SigningGroupSize: 2, + }, + }, + } + } + + honestFirst := newPermit("2", participation.MemberIndexes{2}) + honestSecond := newPermit("3", participation.MemberIndexes{3}) + honestInventory := []participation.PermitSnapshot{honestFirst, honestSecond} + + // The baseline both sides agree on, so a mutation below is the only reason + // any of these findings can appear. + baseline := validateNodeTerminalOutcomes(newManifest( + honestInventory, + []participation.TerminalOutcomeRecord{ + newRecord(honestFirst, group.MemberIndex(1), group.MemberIndex(2)), + newRecord(honestSecond, group.MemberIndex(2), group.MemberIndex(3)), + }, + )) + for _, violation := range baseline { + if strings.Contains(violation, "operated membership") || + strings.Contains(violation, "outside its permit") { + t.Errorf( + "a journal agreeing with its own gate inventory was refused: [%s]", + violation, + ) + } + } + + tests := map[string]struct { + inventory []participation.PermitSnapshot + outcomes []participation.TerminalOutcomeRecord + expectedFinding string + }{ + // A schema-1 snapshot carries no operated seats at all, so a journal + // that names them is the only account of them and there is nothing left + // to reconcile it against. + "the gate inventory carries no operated seats": { + inventory: []participation.PermitSnapshot{ + newPermit("2", nil), + newPermit("3", nil), + }, + outcomes: []participation.TerminalOutcomeRecord{ + newRecord(honestFirst, group.MemberIndex(1), group.MemberIndex(2)), + newRecord(honestSecond, group.MemberIndex(2), group.MemberIndex(3)), + }, + expectedFinding: "but the at-quiescence gate inventory issued the same permit", + }, + "the journal widened one permit's operated seats": { + inventory: honestInventory, + outcomes: []participation.TerminalOutcomeRecord{ + newRecord( + newPermit("2", participation.MemberIndexes{2, 3}), + group.MemberIndex(1), + group.MemberIndex(2), + ), + newRecord(honestSecond, group.MemberIndex(2), group.MemberIndex(3)), + }, + expectedFinding: "but the at-quiescence gate inventory issued the same permit", + }, + // The same edit applied to both copies passes the reconciliation above, + // so the shape its ceremony can have has to be reapplied to each. + "both copies claim a second seat for a one-seat ceremony": { + inventory: []participation.PermitSnapshot{ + newPermit("2", participation.MemberIndexes{2, 3}), + honestSecond, + }, + outcomes: []participation.TerminalOutcomeRecord{ + newRecord( + newPermit("2", participation.MemberIndexes{2, 3}), + group.MemberIndex(1), + group.MemberIndex(2), + ), + newRecord(honestSecond, group.MemberIndex(2), group.MemberIndex(3)), + }, + expectedFinding: "runs one seat per permit", + }, + "both copies claim a seat that is not the permit's own": { + inventory: []participation.PermitSnapshot{ + newPermit("2", participation.MemberIndexes{4}), + honestSecond, + }, + outcomes: []participation.TerminalOutcomeRecord{ + newRecord( + newPermit("2", participation.MemberIndexes{4}), + group.MemberIndex(1), + group.MemberIndex(4), + ), + newRecord(honestSecond, group.MemberIndex(2), group.MemberIndex(3)), + }, + expectedFinding: "runs one seat per permit", + }, + "both copies carry a malformed operated set": { + inventory: []participation.PermitSnapshot{ + newPermit("2", participation.MemberIndexes{2, 2}), + honestSecond, + }, + outcomes: []participation.TerminalOutcomeRecord{ + newRecord( + newPermit("2", participation.MemberIndexes{2, 2}), + group.MemberIndex(1), + group.MemberIndex(2), + ), + newRecord(honestSecond, group.MemberIndex(2), group.MemberIndex(3)), + }, + expectedFinding: "operated memberships", + }, + // The persisted memberships swapped under one shared, honest mapping. + // This is the swap a reader of the raw seat numbers cannot see and the + // mapping makes local: final seat 2 was produced by DKG seat 3, which is + // not the seat this permit was issued to operate. + "the persisted memberships were swapped under one mapping": { + inventory: honestInventory, + outcomes: []participation.TerminalOutcomeRecord{ + newRecord(honestFirst, group.MemberIndex(2), group.MemberIndex(3)), + newRecord(honestSecond, group.MemberIndex(1), group.MemberIndex(2)), + }, + expectedFinding: "which is not among the memberships", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + violations := validateNodeTerminalOutcomes( + newManifest(test.inventory, test.outcomes), + ) + if !containsSubstring(violations, test.expectedFinding) { + t.Fatalf( + "expected an operated-membership violation containing "+ + "[%s], got: %v", + test.expectedFinding, + violations, + ) + } + }) + } +} + +// TestValidateNodeTerminalOutcomes_CompletedEvidenceKindIsPinnedPerCeremony +// proves the offline audit refuses a settlement recorded in the wrong evidence +// class. A wallet action's durable result is a Bitcoin transaction the audit +// can reconcile against the chain; a node-authored protocol digest in its place +// would clear the rollback journal on the node's own say-so after an ambiguous +// submission, with nothing canonical left to check it against. +func TestValidateNodeTerminalOutcomes_CompletedEvidenceKindIsPinnedPerCeremony( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + permit := participation.PermitSnapshot{ + Ceremony: participation.TBTCSigning, + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 999, + WorkID: "wallet-action-legacy", + PermitID: "wallet-action-legacy", + IdentityBound: true, + } + + newManifest := func(evidence participation.TerminalEvidence) *manifest { + return &manifest{ + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + CapturedAt: capturedAt, + ActivePermits: []participation.PermitSnapshot{permit}, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeCompleted, + Evidence: evidence, + }, + }, + }, + } + } + + forged := validateNodeTerminalOutcomes(newManifest( + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: strings.Repeat("a", 64), + }, + )) + rejected := false + for _, violation := range forged { + if strings.Contains(violation, "requires evidence kind [bitcoin_transaction]") { + rejected = true + break + } + } + if !rejected { + t.Errorf( + "a completed wallet action settled on a node-authored protocol "+ + "digest was not rejected, violations: %v", + forged, + ) + } + + honest := validateNodeTerminalOutcomes(newManifest( + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceBitcoinTransaction, + Reference: strings.Repeat("a", 64), + Contribution: testTranscriptContribution( + participation.TBTCSigning, + group.MemberIndex(1), + group.MemberIndex(1), + ), + }, + )) + for _, violation := range honest { + if strings.Contains(violation, "evidence is invalid") { + t.Errorf( + "the wallet action's own evidence class was rejected: [%s]", + violation, + ) + } + } +} + +// TestRunAudit_SignedTransactionMustBeEnumeratedByBitcoinReconciliation proves +// a wallet action the node recorded as completed cannot clear the barrier +// unless the attested-complete pending set names the exact transaction it +// signed. The node pins that hash before broadcasting, so a transaction missing +// from a complete reconciliation is precisely the ambiguous submission the +// rollback barrier exists to catch. +func TestRunAudit_SignedTransactionMustBeEnumeratedByBitcoinReconciliation( + t *testing.T, +) { + permits := []quiescencePermitEvidence{ + { + Ceremony: string(participation.TBTCSigning), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 999, + WorkID: "wallet-action-legacy", + PermitID: "wallet-action-legacy", + Outcome: "completed", + }, + } + + tests := map[string]struct { + mutate func(*bitcoinReconciliationEvidence) + expectedBlocker string + }{ + "the signed transaction is dropped from a complete set": { + mutate: func(record *bitcoinReconciliationEvidence) { + record.PendingTransactions = nil + }, + expectedBlocker: "does not enumerate", + }, + "an unrelated transaction stands in for the signed one": { + mutate: func(record *bitcoinReconciliationEvidence) { + record.PendingTransactions[0].TransactionHash = + strings.Repeat("ab", 32) + }, + expectedBlocker: "does not enumerate", + }, + "the signed transaction is named in a noncanonical shape": { + mutate: func(record *bitcoinReconciliationEvidence) { + record.PendingTransactions[0].TransactionHash = + strings.ToUpper( + record.PendingTransactions[0].TransactionHash, + ) + }, + expectedBlocker: "is not a canonical lowercase transaction hash", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + storageDir := newTestStorageWithQuiescencePermits(t, permits) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + setQuiescencePermits(record, permits) + }, + ) + + // The unmutated evidence must authorize the barrier, otherwise the + // mutation below proves nothing about this check. + baseline, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if !baseline.RollbackBarrierReady { + t.Fatalf( + "the unmutated evidence does not authorize the barrier, "+ + "blockers: %v", + baseline.RollbackBlockers, + ) + } + + updateBitcoinReconciliation(t, evidence, test.mutate) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if auditManifest.RollbackBarrierReady { + t.Fatalf( + "a signed transaction the complete reconciliation does not " + + "cover authorized the barrier", + ) + } + for _, blocker := range auditManifest.RollbackBlockers { + if strings.Contains(blocker, test.expectedBlocker) { + return + } + } + t.Errorf( + "expected a blocker containing [%s], blockers: %v", + test.expectedBlocker, + auditManifest.RollbackBlockers, + ) + }) + } +} + +// updateBitcoinReconciliation rewrites the supplied Bitcoin reconciliation +// evidence in place. +func updateBitcoinReconciliation( + t *testing.T, + evidence evidenceInputs, + mutate func(*bitcoinReconciliationEvidence), +) { + t.Helper() + + record := &bitcoinReconciliationEvidence{} + content, err := os.ReadFile(evidence.bitcoinReconciliation) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(content, record); err != nil { + t.Fatal(err) + } + + mutate(record) + + content, err = json.MarshalIndent(record, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.bitcoinReconciliation, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } +} + +func TestValidateChainReconciliationEvidence_TBTCDKGPermitLineage( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + canonicalSeed := big.NewInt(42) + canonicalResult, _, seedHash := newValidTBTCDKGResultEvidence( + t, + canonicalSeed, + 1_000, + 4, + []uint8{1}, + ) + resultHash := canonicalResult.resultHash() + permits := []participation.PermitSnapshot{ + { + Ceremony: participation.TBTCDKG, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: seedHash, + PermitID: "2", + IdentityBound: true, + OperatedMembers: participation.MemberIndexes{2}, + }, + { + Ceremony: participation.TBTCDKG, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: seedHash, + PermitID: "3", + IdentityBound: true, + OperatedMembers: participation.MemberIndexes{3}, + }, + } + + newRunAndRecord := func( + firstMembership group.MemberIndex, + secondMembership group.MemberIndex, + resultSeed *big.Int, + ) (*auditRun, *chainReconciliationEvidence) { + dkgResult, resultWalletID, _ := newValidTBTCDKGResultEvidence( + t, + resultSeed, + 1_000, + 4, + []uint8{1}, + ) + auditManifest := &manifest{ + GeneratedAt: time.Now().UTC(), + Snapshot: snapshotIdentity{ + AggregateSHA256: strings.Repeat("c", 64), + }, + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + SchemaVersion: participation.QuiescenceSnapshotSchemaVersion, + CapturedAt: capturedAt, + ActivePermits: permits, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permits[0], + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: firstMembership, + // Each record maps its own final seat back to the + // DKG seat its own permit was issued for, so the + // journal is internally consistent whichever way the + // two persisted memberships are assigned. The two + // mappings then disagree about how one final group + // was rebuilt, and only the accepted result on chain + // says which of them is the real one. + Contribution: testTranscriptContribution( + participation.TBTCDKG, + firstMembership, + group.MemberIndex(2), + ), + }, + }, + { + RecordedAt: time.Now().UTC(), + Permit: permits[1], + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: secondMembership, + Contribution: testTranscriptContribution( + participation.TBTCDKG, + secondMembership, + group.MemberIndex(3), + ), + }, + }, + }, + }, + TBTCActiveWallets: []tbtcWalletRecord{ + { + WalletStorageKey: "wallet-storage-key", + WalletID: resultWalletID, + MemberIndexes: []uint8{ + uint8(firstMembership), + uint8(secondMembership), + }, + SigningGroupSize: 3, + }, + }, + } + run := &auditRun{ + manifest: auditManifest, + expected: expectedIdentityInputs{ + ethereumChainID: "1", + walletRegistryAddress: testWalletRegistryAddress, + randomBeaconAddress: testRandomBeaconAddress, + finalizedEthereumBlockNumber: testFinalizedEthereumBlock, + finalizedEthereumBlockHash: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + chainEvidencePublicKey: testChainEvidencePublicKey(), + maxEvidenceAge: time.Hour, + }, + } + record := &chainReconciliationEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "chain_reconciliation", + GeneratedAt: auditManifest.GeneratedAt, + SnapshotAggregateSHA256: auditManifest.Snapshot.AggregateSHA256, + }, + EthereumChainID: "1", + Wallets: []tbtcWalletChainEvidence{ + { + WalletStorageKey: "wallet-storage-key", + WalletID: resultWalletID, + Registered: true, + DKGSettlement: "approved", + DKGResult: dkgResult, + }, + }, + } + authenticateTestChainReconciliationEvidence(t, record) + return run, record + } + + t.Run("canonical original-to-final mapping", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + if violations := run.validateChainReconciliationEvidence(content); len(violations) != 0 { + t.Fatalf("expected exact DKG lineage to pass, got: %v", violations) + } + }) + + t.Run("self-consistent lineage from untrusted generator", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + + seed := make([]byte, ed25519.SeedSize) + for i := range seed { + seed[i] = 0x24 + } + untrustedKey := ed25519.NewKeyFromSeed(seed) + record.CollectorAttestation.Signature = "" + payload, err := chainReconciliationSignaturePayload(record) + if err != nil { + t.Fatal(err) + } + record.CollectorAttestation.Signature = hex.EncodeToString( + ed25519.Sign(untrustedKey, payload), + ) + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "not signed by the independently trusted finalized-chain collector", + ) { + t.Fatalf( + "expected an unauthenticated-collector violation, got: %v", + violations, + ) + } + }) + + t.Run("correct event bytes from unrelated contract", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + record.Receipts[0].Logs[0].Address = + "0x2222222222222222222222222222222222222222" + resignTestChainReconciliationEvidence(t, record) + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "event was emitted by unrelated contract", + ) { + t.Fatalf( + "expected an unrelated-contract violation, got: %v", + violations, + ) + } + }) + + t.Run("failed receipt", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + record.Receipts[0].Status = 0 + resignTestChainReconciliationEvidence(t, record) + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring(violations, "has failed status [0]") || + !containsSubstring(violations, "belongs to failed receipt") { + t.Fatalf( + "expected failed-receipt violations, got: %v", + violations, + ) + } + }) + + t.Run("non-canonical receipt block", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + record.Receipts[0].BlockHash = "0x" + strings.Repeat("f", 64) + resignTestChainReconciliationEvidence(t, record) + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring(violations, "names non-canonical block hash") { + t.Fatalf( + "expected a non-canonical-block violation, got: %v", + violations, + ) + } + }) + + t.Run("full-width signing member index is rejected by group bounds", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + fullWidthIndex := new(big.Int).Lsh(big.NewInt(1), 128) + record.Wallets[0].DKGResult.Submitted.Result. + SigningMemberIndexes[0] = fullWidthIndex + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "signing member ["+fullWidthIndex.String()+ + "] outside original group size", + ) { + t.Fatalf( + "expected full-width index to decode then fail group bounds, "+ + "got: %v", + violations, + ) + } + }) + + t.Run("caller-supplied result hash is recomputed", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + forgedHash := "0x" + strings.Repeat("f", 64) + record.Wallets[0].DKGResult.Submitted.ResultHash = forgedHash + record.Wallets[0].DKGResult.Approved.ResultHash = forgedHash + record.Wallets[0].DKGResult.WalletCreated.DKGResultHash = forgedHash + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "does not match keccak256(abi.encode(result))", + ) { + t.Fatalf( + "expected a derived-result-hash violation, got: %v", + violations, + ) + } + }) + + t.Run("members hash is derived from the submitted result", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + result := record.Wallets[0].DKGResult + result.Submitted.Result.MembersHash = + "0x" + strings.Repeat("e", 64) + // Keep the event's indexed result hash internally consistent with the + // forged tuple. The independent operating-members derivation must + // still reject it. + forgedHash, err := computeTBTCDKGResultHash(result.Submitted.Result) + if err != nil { + t.Fatal(err) + } + result.Submitted.ResultHash = forgedHash + result.Approved.ResultHash = forgedHash + result.WalletCreated.DKGResultHash = forgedHash + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "does not match the derived operating-members hash", + ) { + t.Fatalf( + "expected a derived-members-hash violation, got: %v", + violations, + ) + } + }) + + t.Run("approval and wallet creation share one receipt", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + record.Wallets[0].DKGResult.WalletCreated.TransactionHash = + "0x" + strings.Repeat("d", 64) + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "do not belong to the same approval receipt", + ) { + t.Fatalf( + "expected an accepted-event-lineage violation, got: %v", + violations, + ) + } + }) + + t.Run("swapped persisted memberships", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(2), + group.MemberIndex(1), + canonicalSeed, + ) + // Storage existence and one-to-one claims alone accept this swap: + // both final memberships really exist and neither is reused. + if violations := validateNodeTerminalOutcomes(run.manifest); len(violations) != 0 { + t.Fatalf( + "expected node-local membership existence to be insufficient, got: %v", + violations, + ) + } + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + for _, expected := range []string{ + "maps to final membership [1], but names persisted membership [2]", + "maps to final membership [2], but names persisted membership [1]", + } { + if !containsSubstring(violations, expected) { + t.Errorf( + "expected swapped-membership violation containing [%s], got: %v", + expected, + violations, + ) + } + } + }) + + // The seat map is one statement about every seat of the final group, not + // only about the one its author sat in. A rewrite that leaves the author's + // own entry alone, and the length alone, satisfies every other check here: + // the seed, the anchor and the persisted membership all still agree, the + // author's operated seat still maps to its own final seat, and the map is + // still an ascending set of the right size. What it changes is who the other + // final seats belonged to — and a fleet-wide ownership map translated through + // it hands those seats to original members that never held them, which reads + // afterwards as seats some other release supplied. + t.Run("rewritten remote seat mapping", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + outcome := &run.manifest.ParticipationTerminalOutcomes.Outcomes[0] + mapping := outcome.Evidence.Contribution.PermitSpaceMembers + if !slices.Equal( + mapping, + participation.MemberIndexes{2, 3, 4}, + ) { + t.Fatalf( + "fixture no longer maps the canonical survivors, got: %v", + mapping, + ) + } + // Only the last entry moves. The author sits in final seat 1, so its own + // mapping is the first entry and is left exactly as it was. + mapping[len(mapping)-1] = group.MemberIndex(5) + + // Everything the audit checked before this still passes, which is what + // makes the rewrite worth refusing rather than a shape error. + if violations := validateNodeTerminalOutcomes(run.manifest); len(violations) != 0 { + t.Fatalf( + "expected a rewritten remote entry to survive node-local "+ + "validation, got: %v", + violations, + ) + } + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "maps its final signing group back to original members [2 3 5], "+ + "but canonical result ["+resultHash+"] leaves survivors [2 3 4]", + ) { + t.Fatalf( + "expected a rewritten-seat-map violation, got: %v", + violations, + ) + } + }) + + // And the other half of the same map. A record naming a final group the + // accepted result did not rebuild describes a different ceremony, however + // well its own seat lines up inside it. + t.Run("final signing group the result did not rebuild", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + canonicalSeed, + ) + contribution := run.manifest. + ParticipationTerminalOutcomes.Outcomes[0].Evidence.Contribution + contribution.IncorporatedMembers = participation.MemberIndexes{1, 2} + contribution.PermitSpaceMembers = participation.MemberIndexes{2, 3} + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + if !containsSubstring( + violations, + "names final signing group [1 2], but canonical result ["+ + resultHash+"] rebuilds group [1 2 3] from its 3 accepted members", + ) { + t.Fatalf( + "expected a rebuilt-group violation, got: %v", + violations, + ) + } + }) + + t.Run("unrelated approved wallet", func(t *testing.T) { + run, record := newRunAndRecord( + group.MemberIndex(1), + group.MemberIndex(2), + big.NewInt(43), + ) + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + violations := run.validateChainReconciliationEvidence(content) + _, _, unrelatedSeedHash := newValidTBTCDKGResultEvidence( + t, + big.NewInt(43), + 1_000, + 4, + []uint8{1}, + ) + if !containsSubstring( + violations, + "persisted wallet [wallet-storage-key] was created by canonical "+ + "result ["+resultHash+"] for seed ["+unrelatedSeedHash+"]", + ) { + t.Fatalf( + "expected unrelated-approved-wallet violation, got: %v", + violations, + ) + } + }) +} + +func TestValidateNodeTerminalOutcomes_DKGExhaustionNeedsChainProof( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + permit := participation.PermitSnapshot{ + Ceremony: participation.TBTCDKG, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: strings.Repeat("d", 64), + PermitID: "1", + IdentityBound: true, + } + auditManifest := &manifest{ + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + CapturedAt: capturedAt, + ActivePermits: []participation.PermitSnapshot{permit}, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeExhausted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceNoThreshold, + }, + }, + }, + }, + } + + violations := validateNodeTerminalOutcomes(auditManifest) + for _, violation := range violations { + if strings.Contains( + violation, + "no chain-derived proof that another member did not publish", + ) { + return + } + } + t.Fatalf( + "expected unauthenticated DKG exhaustion to be rejected, got: %v", + violations, + ) +} + +func TestRunAudit_NodeCompletedBeaconDKGMatchesPersistedSigner(t *testing.T) { + permit := quiescencePermitEvidence{ + Ceremony: string(participation.BeaconDKG), + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: strings.Repeat("d", 64), + PermitID: "1", + Outcome: string(participation.TerminalOutcomeCompleted), + } + storageDir := newTestStorageWithQuiescencePermits( + t, + []quiescencePermitEvidence{permit}, + ) + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + evidence := newValidEvidence(t, firstPass) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if !auditManifest.RollbackBarrierReady { + t.Errorf( + "a node-authored completed beacon DKG outcome matching its active "+ + "signer should authorize the barrier, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +func TestValidateNodeTerminalOutcomes_RejectsUnsupportedEvidence( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + permit := participation.PermitSnapshot{ + Ceremony: participation.TBTCSigning, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: "wallet-action", + PermitID: "wallet", + IdentityBound: true, + } + auditManifest := &manifest{ + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + CapturedAt: capturedAt, + ActivePermits: []participation.PermitSnapshot{permit}, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceKind( + "fabricated_result", + ), + Reference: "fabricated-result", + }, + }, + }, + }, + } + + violations := validateNodeTerminalOutcomes(auditManifest) + for _, violation := range violations { + if strings.Contains(violation, "evidence is invalid") { + return + } + } + t.Fatalf( + "expected unsupported terminal evidence violation, got: %v", + violations, + ) +} + +// TestRunAudit_QuiescenceOutcomesMustCoverGateInventory proves terminal +// outcomes cannot authorize the barrier when they omit permits independently +// captured by the gate at the quiescence transition. +func TestRunAudit_QuiescenceOutcomesMustCoverGateInventory(t *testing.T) { + permits := []quiescencePermitEvidence{ + { + Ceremony: string(participation.TBTCSigning), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 999, + WorkID: "wallet-action-legacy", + PermitID: "wallet-action-legacy", + Outcome: "completed", + }, + { + Ceremony: string(participation.BeaconRelaySigning), + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: participation.BeaconRelayWorkID(1_000), + PermitID: "2", + Outcome: "completed", + }, + } + storageDir := newTestStorageWithQuiescencePermits(t, permits) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + tests := map[string]struct { + outcomes []quiescencePermitEvidence + blockerFragments []string + }{ + "empty outcomes over nonempty inventory": { + outcomes: nil, + blockerFragments: []string{ + "contains [0] permits, but the node-authored gate snapshot declares total [2]", + "contains [0] legacy permits, but the node-authored gate snapshot declares [1]", + "contains [0] security-v2 permits, but the node-authored gate snapshot declares [1]", + }, + }, + "shortened outcomes without duplicates": { + outcomes: permits[:1], + blockerFragments: []string{ + "contains [1] permits, but the node-authored gate snapshot declares total [2]", + "contains [0] security-v2 permits, but the node-authored gate snapshot declares [1]", + "node-authored gate inventory entry [0] has no terminal outcome", + }, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + setQuiescencePermits(record, permits) + record.ActivePermitsAtQuiescence = append( + []quiescencePermitEvidence(nil), + test.outcomes..., + ) + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error( + "an incomplete terminal-outcome list must not " + + "authorize the barrier", + ) + } + for _, fragment := range test.blockerFragments { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected coverage blocker [%s], blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } + }) + } +} + +// TestValidateRelayEntryTerminalResult asserts a beacon relay outcome is +// checked rather than believed. +// +// This is the one node-authored protocol result the offline audit can verify +// outright: a relay entry is a threshold BLS signature by the group over the +// previous entry, so the pairing itself decides. Anything a node could author +// alone — an entry it invented, an entry lifted from another group, an entry +// re-pointed at a different previous entry — fails that check, and a group the +// snapshot holds no membership of says nothing about what this node did even +// when its signature is genuine. +func TestValidateRelayEntryTerminalResult(t *testing.T) { + const ( + heldGroupSecret = int64(42) + foreignGroupSecret = int64(77) + ) + + const requestStartBlock = uint64(1_000) + + heldGroupKey := hex.EncodeToString( + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(big.NewInt(heldGroupSecret)), + }.Compress(), + ) + beaconGroupKeys := map[string]struct{}{heldGroupKey: {}} + + // reference renders an entry the given group really signed over the given + // previous entry, answering the given request, so only the deliberate + // mismatches below are wrong. + reference := func( + t *testing.T, + requestStartBlock uint64, + groupSecret int64, + signingSecret int64, + previousEntryLabel string, + namedPreviousEntryLabel string, + ) string { + t.Helper() + + signed := altbn128.G1HashToPoint([]byte(previousEntryLabel)) + named := altbn128.G1HashToPoint([]byte(namedPreviousEntryLabel)) + + rendered, err := participation.BeaconRelayEntryReference( + requestStartBlock, + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(big.NewInt(groupSecret)), + }.Compress(), + named.Marshal(), + bls.SignG1(big.NewInt(signingSecret), signed).Marshal(), + ) + if err != nil { + t.Fatal(err) + } + + return rendered + } + + tests := map[string]struct { + workID string + reference func(t *testing.T) string + valid bool + }{ + "an entry the named group signed over the named previous entry": { + reference: func(t *testing.T) string { + return reference(t, requestStartBlock, heldGroupSecret, heldGroupSecret, "seed", "seed") + }, + valid: true, + }, + // The forgery the check exists for: a node naming a group it belongs + // to and an entry that group never produced. + "an entry signed by nobody's threshold key": { + reference: func(t *testing.T) string { + return reference(t, requestStartBlock, heldGroupSecret, foreignGroupSecret, "seed", "seed") + }, + }, + // A genuine signature re-pointed at a previous entry it was not made + // over would let one relay round's result settle another's permit. + "a genuine entry over a different previous entry": { + reference: func(t *testing.T) string { + return reference(t, requestStartBlock, heldGroupSecret, heldGroupSecret, "seed", "other-seed") + }, + }, + // Genuine and self-consistent, but produced by a group this snapshot + // has no membership of, so it reports nothing about this node. + "a genuine entry of a group the snapshot does not hold": { + reference: func(t *testing.T) string { + return reference( + t, + requestStartBlock, + foreignGroupSecret, + foreignGroupSecret, + "seed", + "seed", + ) + }, + }, + // The replay the request binding exists for: an entry this node's + // group really produced, for a request this permit was not issued + // for. The signature is genuine and stays genuine forever, so nothing + // but the request it names contradicts it. + "a genuine entry answering an earlier request": { + reference: func(t *testing.T) string { + return reference(t, requestStartBlock-1, heldGroupSecret, heldGroupSecret, "seed", "seed") + }, + }, + "a genuine entry answering a later request": { + reference: func(t *testing.T) string { + return reference(t, requestStartBlock+1, heldGroupSecret, heldGroupSecret, "seed", "seed") + }, + }, + "a permit whose work identity names no relay request": { + workID: "wallet-action", + reference: func(t *testing.T) string { + return reference(t, requestStartBlock, heldGroupSecret, heldGroupSecret, "seed", "seed") + }, + }, + "a permit whose request is not canonically rendered": { + workID: "relay-request-0" + strconv.FormatUint(requestStartBlock, 10), + reference: func(t *testing.T) string { + return reference(t, requestStartBlock, heldGroupSecret, heldGroupSecret, "seed", "seed") + }, + }, + "a reference that is not a canonical entry identity": { + reference: func(*testing.T) string { return "relay-entry-1" }, + }, + "a well-formed reference whose components are not curve points": { + reference: func(t *testing.T) string { + notAPoint := bytes.Repeat([]byte{0xff}, 64) + rendered, err := participation.BeaconRelayEntryReference( + requestStartBlock, + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult( + big.NewInt(heldGroupSecret), + ), + }.Compress(), + notAPoint, + notAPoint, + ) + if err != nil { + t.Fatal(err) + } + return rendered + }, + }, + // The degenerate forgery: the point at infinity pairs trivially, so an + // entry and previous entry of all zeros would verify under any group + // key without knowing anything about it. + "the point at infinity, which verifies under any group": { + reference: func(t *testing.T) string { + infinity := make([]byte, 64) + rendered, err := participation.BeaconRelayEntryReference( + requestStartBlock, + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult( + big.NewInt(heldGroupSecret), + ), + }.Compress(), + infinity, + infinity, + ) + if err != nil { + t.Fatal(err) + } + return rendered + }, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + workID := test.workID + if workID == "" { + workID = participation.BeaconRelayWorkID(requestStartBlock) + } + + violations := validateRelayEntryTerminalResult( + 0, + workID, + test.reference(t), + beaconGroupKeys, + make(map[string]relayEntryClaim), + ) + + if test.valid && len(violations) != 0 { + t.Fatalf( + "expected a verifiable relay entry to pass, got: %v", + violations, + ) + } + if !test.valid && len(violations) == 0 { + t.Fatal("expected the relay entry to be rejected") + } + }) + } +} + +// TestValidateRelayEntryTerminalResult_OneEntryAnswersOneRequest asserts a +// relay result cannot close more than one relay request across the journal. +// +// A relay entry is deterministic for a given previous entry, so it belongs to +// exactly one request; the signature stays valid whatever request it is filed +// under. Without this the same genuine entry could settle every relay permit a +// node ever held, which is what the rollback barrier reads as completed work. +// Memberships of the same request legitimately recover and record the same +// entry, so those must still pass. +func TestValidateRelayEntryTerminalResult_OneEntryAnswersOneRequest( + t *testing.T, +) { + const ( + groupSecret = int64(42) + firstRequestStartBlock = uint64(1_000) + secondRequestStartBlock = uint64(2_000) + ) + + beaconGroupKeys := map[string]struct{}{ + hex.EncodeToString(altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(big.NewInt(groupSecret)), + }.Compress()): {}, + } + claimed := make(map[string]relayEntryClaim) + + record := func(outcomeIndex int, requestStartBlock uint64) []string { + return validateRelayEntryTerminalResult( + outcomeIndex, + participation.BeaconRelayWorkID(requestStartBlock), + testRelayEntryReference(requestStartBlock, groupSecret, "seed"), + beaconGroupKeys, + claimed, + ) + } + + if violations := record(0, firstRequestStartBlock); len(violations) != 0 { + t.Fatalf("the first use of an entry was rejected: %v", violations) + } + + // A second membership of the same request recovers the very same entry. + if violations := record(1, firstRequestStartBlock); len(violations) != 0 { + t.Errorf( + "a second membership of the same request was rejected: %v", + violations, + ) + } + + if violations := record(2, secondRequestStartBlock); len(violations) == 0 { + t.Error( + "an entry already used for one request was accepted as the " + + "result of another", + ) + } +} + +// testTimeoutSettlementReference renders the beacon settlement identity a +// completed timeout report outcome carries. +func testTimeoutSettlementReference( + t *testing.T, + requestStartBlock uint64, + requestID int64, + terminatedGroupID uint64, +) string { + t.Helper() + + reference, err := participation.BeaconRelayTimeoutSettlementReference( + requestStartBlock, + big.NewInt(requestID), + terminatedGroupID, + ) + if err != nil { + t.Fatal(err) + } + return reference +} + +// TestValidateRelayTimeoutSettlement asserts the offline audit holds a +// completed timeout report to a settlement identity it can join to exactly one +// authenticated beacon log, and to the request its own permit was issued for. +func TestValidateRelayTimeoutSettlement(t *testing.T) { + const requestStartBlock = uint64(1_000) + + tests := map[string]struct { + workID string + reference string + expectViolation bool + }{ + "a settlement terminating the permit's own request": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: testTimeoutSettlementReference( + t, + requestStartBlock, + 11, + 4, + ), + }, + // A real penalty, earned by another request. Nothing about the log it + // joins to is wrong; what is wrong is the permit it is settling. + "a settlement terminating another request": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: testTimeoutSettlementReference( + t, + requestStartBlock+1, + 11, + 4, + ), + expectViolation: true, + }, + "a permit that names no relay request": { + workID: "not-a-relay-request", + reference: testTimeoutSettlementReference( + t, + requestStartBlock, + 11, + 4, + ), + expectViolation: true, + }, + // A digest is exactly what the record must not be: it names no log an + // operator could fetch. + "a digest standing in for a settlement": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: participation.TerminalResultReference( + "domain", + []byte("result"), + ), + expectViolation: true, + }, + "a settlement identity missing its terminated group": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: "1000:11", + expectViolation: true, + }, + "a settlement identity with a padded request start block": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: "01000:11:4", + expectViolation: true, + }, + "a settlement identity with a padded request identifier": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: "1000:011:4", + expectViolation: true, + }, + "a settlement identity with a negative request identifier": { + workID: participation.BeaconRelayWorkID(requestStartBlock), + reference: "1000:-11:4", + expectViolation: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + violations := validateRelayTimeoutSettlement( + 0, + test.workID, + test.reference, + make(map[string]relayTimeoutSettlementClaim), + ) + + if hasViolation := len(violations) != 0; hasViolation != + test.expectViolation { + t.Errorf( + "unexpected violations\nexpected any: [%t]\nactual: %v", + test.expectViolation, + violations, + ) + } + }) + } +} + +// TestValidateRelayTimeoutSettlement_OneSettlementAnswersOneRequest asserts a +// beacon settlement cannot settle two permits issued for different requests. +// The beacon terminates a request once, so the same request identifier and +// terminated group standing as a second request's result is a claim on a +// penalty that request never earned. +func TestValidateRelayTimeoutSettlement_OneSettlementAnswersOneRequest( + t *testing.T, +) { + const ( + firstRequestStartBlock = uint64(1_000) + secondRequestStartBlock = uint64(2_000) + ) + + claimed := make(map[string]relayTimeoutSettlementClaim) + + record := func(outcomeIndex int, requestStartBlock uint64) []string { + return validateRelayTimeoutSettlement( + outcomeIndex, + participation.BeaconRelayWorkID(requestStartBlock), + testTimeoutSettlementReference(t, requestStartBlock, 11, 4), + claimed, + ) + } + + if violations := record(0, firstRequestStartBlock); len(violations) != 0 { + t.Fatalf("the first use of a settlement was rejected: %v", violations) + } + + // The same permit's record written twice — a retried journal write — is the + // same claim, not a replay. + if violations := record(1, firstRequestStartBlock); len(violations) != 0 { + t.Errorf( + "a repeated record of the same settlement was rejected: %v", + violations, + ) + } + + if violations := record(2, secondRequestStartBlock); len(violations) == 0 { + t.Error( + "a settlement already used for one request was accepted as the " + + "result of another", + ) + } +} + +// TestRunAudit_SelfAttestedEqualOmissionCannotHideRealGatePermit proves the +// prior report-only attack is closed: even if an external generator reports a +// shortened outcome list and would have shortened its own duplicate counts +// and inventory too, the audit reconciles against the independently persisted +// registry captured by the production gate. +func TestRunAudit_SelfAttestedEqualOmissionCannotHideRealGatePermit( + t *testing.T, +) { + permits := []quiescencePermitEvidence{ + { + Ceremony: string(participation.TBTCSigning), + Mode: participation.ModeLegacy.String(), + CanonicalStartBlock: 999, + WorkID: "wallet-action-real-legacy", + PermitID: "wallet-real-legacy", + Outcome: "completed", + }, + { + Ceremony: string(participation.BeaconRelaySigning), + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: participation.BeaconRelayWorkID(1_000), + PermitID: "2", + Outcome: "completed", + }, + } + + storageDir := newTestStorage(t) + persistRealGateQuiescenceSnapshot(t, storageDir, permits) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + record.ActivePermitsAtQuiescence = + []quiescencePermitEvidence{permits[0]} + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if auditManifest.RollbackBarrierReady { + t.Error( + "an external equal omission must not hide a permit captured by " + + "the production gate", + ) + } + for _, fragment := range []string{ + "contains [1] permits, but the node-authored gate snapshot declares total [2]", + "node-authored gate inventory entry [0] has no terminal outcome", + } { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected node-authored inventory blocker [%s], blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } +} + +func TestRunAudit_QuiescenceReportCannotPredateGateTransition(t *testing.T) { + storageDir := newTestStorage(t) + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if firstPass.QuiescenceSnapshot == nil { + t.Fatal("test storage has no node-authored quiescence snapshot") + } + + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + record.GeneratedAt = + firstPass.QuiescenceSnapshot.CapturedAt.Add(-time.Second) + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + if auditManifest.RollbackBarrierReady { + t.Error( + "a terminal report generated before the node quiesced must not " + + "authorize the barrier", + ) + } + if !hasBlocker( + auditManifest, + "node-authored quiescence gate snapshot was captured after the quiescence report was generated", + ) { + t.Errorf( + "expected transition-time blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// TestRunAudit_QuiescencePermitModeMustMatchCutoverBoundary proves completed +// permits are subject to the same canonical-anchor arithmetic as quarantined +// outputs. +func TestRunAudit_QuiescencePermitModeMustMatchCutoverBoundary(t *testing.T) { + tests := map[string]struct { + mode participation.ProtocolMode + anchor uint64 + blockerFragment string + }{ + "legacy at C": { + mode: participation.ModeLegacy, + anchor: 1_000, + blockerFragment: "permit entry [0] claims mode [legacy] with canonical anchor [1000] at or after cutover block [1000]", + }, + "security-v2 below C": { + mode: participation.ModeSecurityV2, + anchor: 999, + blockerFragment: "permit entry [0] claims mode [security_v2] with canonical anchor [999] before cutover block [1000]", + }, + "zero canonical anchor": { + mode: participation.ModeLegacy, + anchor: 0, + blockerFragment: "permit entry [0] has zero canonical anchor under armed cutover block [1000]", + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + permits := []quiescencePermitEvidence{{ + Ceremony: string( + participation.TBTCSigning, + ), + Mode: test.mode.String(), + CanonicalStartBlock: test.anchor, + WorkID: "wallet-action-boundary", + PermitID: "wallet-action-boundary", + Outcome: "completed", + }} + storageDir := newTestStorageWithQuiescencePermits(t, permits) + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + updateQuiescenceReport( + t, + evidence, + func(record *quiescenceReportEvidence) { + setQuiescencePermits(record, permits) + }, + ) + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error( + "a permit whose mode contradicts its canonical " + + "anchor must not authorize the barrier", + ) + } + if !hasBlocker(auditManifest, test.blockerFragment) { + t.Errorf( + "expected cutover-arithmetic blocker [%s], blockers: %v", + test.blockerFragment, + auditManifest.RollbackBlockers, + ) + } + }) + } +} + +// TestRunAudit_WrongExpectedReleaseEpochIsBlocking proves an expected release +// epoch differing from the audit build's own compiled epoch is a rollback +// blocker: the wrong audit tool cannot judge the audited state. +func TestRunAudit_WrongExpectedReleaseEpochIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + expected := testExpectedIdentity() + expected.releaseEpoch = "some_other_epoch" + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + expected, + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("a wrong expected epoch must not authorize the barrier") + } + if !hasBlocker( + auditManifest, + "does not match this audit build's compiled epoch", + ) { + t.Errorf( + "expected a compiled-epoch blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// TestRunAudit_MutableExpectedImageDigestIsBlocking proves an expected image +// reference that is not an immutable sha256 digest — a tag, a malformed +// digest — is a rollback blocker: a mutable reference cannot pin the +// artifact the rollback restores or leaves. +func TestRunAudit_MutableExpectedImageDigestIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + expected := testExpectedIdentity() + expected.priorImageDigest = "keep-client:latest" + expected.releaseImageDigest = "sha256:not-a-hex-digest" + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + expected, + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("a mutable image reference must not authorize the barrier") + } + if !hasBlocker( + auditManifest, + "the expected prior image digest [keep-client:latest] is not an "+ + "immutable sha256 image digest", + ) { + t.Errorf( + "expected a prior-digest blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } + if !hasBlocker( + auditManifest, + "the expected release image digest [sha256:not-a-hex-digest] is "+ + "not an immutable sha256 image digest", + ) { + t.Errorf( + "expected a release-digest blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// TestRunAudit_MismatchedArtifactIdentityIsBlocking proves schema-valid +// evidence recording a different release artifact, prior image, or cutover +// block than the expected one blocks the barrier, and that quarantine +// metadata preserved under a different cutover block is a finding of its +// own. +func TestRunAudit_MismatchedArtifactIdentityIsBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + // The evidence records the artifact identities newValidEvidence writes; + // the audit expects a different candidate build and cutover schedule. + mismatched := testExpectedIdentity() + mismatched.priorImageDigest = "sha256:" + strings.Repeat("44", 32) + mismatched.releaseVersion = "v9.9.9" + mismatched.releaseRevision = strings.Repeat("00", 20) + mismatched.releaseImageDigest = "sha256:" + strings.Repeat("33", 32) + mismatched.cutoverBlock = 2_000 + + auditManifest, err := runAudit( + storageDir, + testPassword, + newValidEvidence(t, firstPass), + mismatched, + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("mismatched artifact identities must not authorize the barrier") + } + for _, fragment := range []string{ + "quiescing release version [v2.1.0], expected [v9.9.9]", + "quiescing release revision", + "quiesced under cutover block [1000], expected [2000]", + "tested prior image digest", + "writing release version [v2.1.0], expected [v9.9.9]", + "writing release revision", + "writing release image digest", + } { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected the [%s] blocker, blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } + if !hasFinding( + auditManifest, + "was preserved under cutover block [1000], not the expected "+ + "cutover block [2000]", + ) { + t.Errorf( + "expected a quarantine cutover-binding finding, findings: %v", + auditManifest.Findings, + ) + } +} + +// TestRunAudit_DuplicateReconciliationEntriesAreBlocking proves duplicate +// wallet, wallet-ID, beacon-group, and schema-result entries in otherwise +// valid evidence are violations: duplicates cannot prove one-to-one coverage +// and can shadow a contradicting result. +func TestRunAudit_DuplicateReconciliationEntriesAreBlocking(t *testing.T) { + storageDir := newTestStorage(t) + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + + content, err := os.ReadFile(evidence.chainReconciliation) + if err != nil { + t.Fatal(err) + } + chainRecord := &chainReconciliationEvidence{} + if err := json.Unmarshal(content, chainRecord); err != nil { + t.Fatal(err) + } + // Duplicate the persisted beacon group's entry, reconcile one fabricated + // wallet twice, and claim its wallet ID from a second fabricated wallet. + chainRecord.BeaconGroups = append( + chainRecord.BeaconGroups, + chainRecord.BeaconGroups[0], + ) + walletEntry := tbtcWalletChainEvidence{ + WalletStorageKey: "duplicated-wallet", + WalletID: strings.Repeat("aa", 32), + Registered: false, + DKGSettlement: "none", + } + chainRecord.Wallets = append(chainRecord.Wallets, walletEntry, walletEntry) + walletEntry.WalletStorageKey = "identity-thief-wallet" + chainRecord.Wallets = append(chainRecord.Wallets, walletEntry) + content, err = json.MarshalIndent(chainRecord, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.chainReconciliation, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + content, err = os.ReadFile(evidence.priorReaderCompatibility) + if err != nil { + t.Fatal(err) + } + priorReaderRecord := &priorReaderCompatibilityEvidence{} + if err := json.Unmarshal(content, priorReaderRecord); err != nil { + t.Fatal(err) + } + // The duplicate contradicts the authoritative first result; it must be + // rejected, not silently shadow it. + priorReaderRecord.SchemaResults = append( + priorReaderRecord.SchemaResults, + struct { + Schema string `json:"schema"` + Compatible bool `json:"compatible"` + }{Schema: requiredPriorReaderSchemas[0], Compatible: false}, + ) + content, err = json.MarshalIndent(priorReaderRecord, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.priorReaderCompatibility, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("duplicate reconciliation entries must not authorize the barrier") + } + for _, fragment := range []string{ + "beacon group [" + + firstPass.BeaconActiveMemberships[0].GroupPublicKey + + "] is reconciled more than once", + "tbtc wallet [duplicated-wallet] is reconciled more than once", + "is claimed by both tbtc wallet [duplicated-wallet] and tbtc " + + "wallet [identity-thief-wallet]", + "schema [" + requiredPriorReaderSchemas[0] + "] is covered more " + + "than once", + } { + if !hasBlocker(auditManifest, fragment) { + t.Errorf( + "expected the [%s] blocker, blockers: %v", + fragment, + auditManifest.RollbackBlockers, + ) + } + } +} + +// TestRunAudit_QuarantinedUnsettledOrMisidentifiedWalletIsBlocking proves a +// quarantined-only tBTC wallet whose reconciled DKG settlement is anything +// but an explicit no-result state — or whose reconciled wallet ID differs +// from the preserved output's identity — blocks the barrier: an unsettled +// result may still hand the prior binary a wallet whose share exists only in +// quarantine, and evidence for a different wallet proves nothing about this +// one. +func TestRunAudit_QuarantinedUnsettledOrMisidentifiedWalletIsBlocking( + t *testing.T, +) { + storageDir := newTestStorage(t) + + diskStorage, err := storage.Initialize( + storage.Config{Dir: storageDir}, + testPassword, + ) + if err != nil { + t.Fatal(err) + } + quarantineHandle, err := diskStorage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + t.Fatal(err) + } + if err := quarantineHandle.Save( + []byte(`{`+ + `"schema_version":1,`+ + `"release_epoch":"security_v2_cutover",`+ + `"protocol_mode":"legacy",`+ + `"cutover_block":1000,`+ + `"canonical_start_block":900,`+ + `"ceremony":"tbtc_dkg",`+ + `"seed_hash":"aa",`+ + `"member_index":3,`+ + `"wallet_id":"bb",`+ + `"wallet_public_key_hash":"cc",`+ + `"failed_operation":"tbtc_dkg_signer_activation",`+ + `"last_observed_block":950,`+ + `"preserved_at":"2026-01-01T00:00:00Z"}`), + "interrupted-wallet-directory", + "/metadata_3", + ); err != nil { + t.Fatal(err) + } + + firstPass, err := runAudit( + storageDir, + testPassword, + evidenceInputs{}, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + evidence := newValidEvidence(t, firstPass) + + content, err := os.ReadFile(evidence.chainReconciliation) + if err != nil { + t.Fatal(err) + } + chainRecord := &chainReconciliationEvidence{} + if err := json.Unmarshal(content, chainRecord); err != nil { + t.Fatal(err) + } + // The quarantined wallet's result is reported as still pending, under a + // wallet ID that is not the preserved output's identity. + for i := range chainRecord.Wallets { + if chainRecord.Wallets[i].WalletStorageKey == + "interrupted-wallet-directory" { + chainRecord.Wallets[i].DKGSettlement = "pending" + chainRecord.Wallets[i].WalletID = "ff" + } + } + content, err = json.MarshalIndent(chainRecord, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile( + evidence.chainReconciliation, + content, + 0o600, + ); err != nil { + t.Fatal(err) + } + + auditManifest, err := runAudit( + storageDir, + testPassword, + evidence, + testExpectedIdentity(), + ) + if err != nil { + t.Fatal(err) + } + + if auditManifest.RollbackBarrierReady { + t.Error("an unsettled quarantined wallet must not authorize the barrier") + } + if !hasBlocker( + auditManifest, + "quarantined tbtc wallet [interrupted-wallet-directory] has DKG "+ + "settlement [pending], expected [none]", + ) { + t.Errorf( + "expected a settlement blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } + if !hasBlocker( + auditManifest, + "quarantined tbtc wallet [interrupted-wallet-directory] is "+ + "reconciled under wallet ID [ff], but its preserved output "+ + "carries wallet ID [bb]", + ) { + t.Errorf( + "expected a wallet-identity blocker, blockers: %v", + auditManifest.RollbackBlockers, + ) + } +} + +// addTestInactivityClaimedReceipt appends an authenticated receipt carrying a +// real ABI-encoded InactivityClaimed log for the given wallet and nonce, then +// re-attests and re-signs the record. The log is built from the generated +// WalletRegistry ABI so the audit's decode path is exercised against the exact +// bytes the contract emits. +func addTestInactivityClaimedReceipt( + t *testing.T, + record *chainReconciliationEvidence, + address string, + walletID [32]byte, + nonce *big.Int, + blockNumber uint64, +) { + t.Helper() + + parsed, err := ecdsaabi.WalletRegistryMetaData.GetAbi() + if err != nil { + t.Fatal(err) + } + event, ok := parsed.Events["InactivityClaimed"] + if !ok { + t.Fatal("generated WalletRegistry ABI has no InactivityClaimed event") + } + + data, err := event.Inputs.NonIndexed().Pack( + nonce, + common.HexToAddress(testWalletRegistryAddress), + ) + if err != nil { + t.Fatal(err) + } + + transactionHash := fmt.Sprintf("0x%064x", blockNumber*7+3) + record.Receipts = append(record.Receipts, ethereumReceiptEvidence{ + TransactionHash: transactionHash, + BlockHash: testCanonicalEthereumBlockHash(blockNumber), + BlockNumber: blockNumber, + TransactionIndex: uint64(len(record.Receipts)), + Status: 1, + Logs: []ethereumRawLogEvidence{ + { + Address: address, + Topics: []string{ + strings.ToLower(event.ID.Hex()), + "0x" + hex.EncodeToString(walletID[:]), + }, + Data: "0x" + hex.EncodeToString(data), + LogIndex: 0, + }, + }, + }) + + record.CollectorAttestation.CanonicalBlocks = append( + record.CollectorAttestation.CanonicalBlocks, + ethereumCanonicalBlockEvidence{ + BlockNumber: blockNumber, + BlockHash: testCanonicalEthereumBlockHash(blockNumber), + }, + ) + sort.Slice( + record.CollectorAttestation.CanonicalBlocks, + func(i, j int) bool { + return record.CollectorAttestation.CanonicalBlocks[i].BlockNumber < + record.CollectorAttestation.CanonicalBlocks[j].BlockNumber + }, + ) + resignTestChainReconciliationEvidence(t, record) +} + +// TestValidateChainReconciliationEvidence_InactivityClaimSettlement asserts a +// penalty a heartbeat filed is reconciled against the WalletRegistry rather +// than accepted on the node's own record. The claim runs under the heartbeat's +// permit and leaves no separate journal entry, so an unobserved or fabricated +// settlement must block the barrier instead of clearing it. +func TestValidateChainReconciliationEvidence_InactivityClaimSettlement( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + walletID := [32]byte{ + 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, + 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, + 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x20, 0x21, + 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28, 0x29, + } + const claimBlock = uint64(9_000) + nonce := big.NewInt(17) + + settledReference, err := participation.InactivityClaimSettlementReference( + walletID[:], + nonce, + ) + if err != nil { + t.Fatal(err) + } + + // A tBTC permit names itself with the wallet public key hash, which is what + // the snapshot's own signer material resolves to the registry wallet ID. + walletPublicKeyHash := strings.Repeat("ab", 20) + + permit := participation.PermitSnapshot{ + Ceremony: participation.TBTCHeartbeat, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: 8_000, + WorkID: strings.Repeat("d", 64), + PermitID: walletPublicKeyHash, + IdentityBound: true, + } + + newRunAndRecord := func( + settlement *participation.ChainSettlementRecord, + ) (*auditRun, *chainReconciliationEvidence) { + auditManifest := &manifest{ + GeneratedAt: time.Now().UTC(), + Snapshot: snapshotIdentity{ + AggregateSHA256: strings.Repeat("c", 64), + }, + TBTCActiveWallets: []tbtcWalletRecord{ + { + WalletStorageKey: strings.Repeat("f", 40), + WalletID: hex.EncodeToString(walletID[:]), + WalletPublicKeyHash: walletPublicKeyHash, + MemberIndexes: []uint8{1}, + SigningGroupSize: 1, + }, + }, + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + SchemaVersion: participation.QuiescenceSnapshotSchemaVersion, + CapturedAt: capturedAt, + ActivePermits: []participation.PermitSnapshot{permit}, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: strings.Repeat("e", 64), + ChainSettlement: settlement, + }, + }, + }, + }, + } + run := &auditRun{ + manifest: auditManifest, + expected: expectedIdentityInputs{ + ethereumChainID: "1", + walletRegistryAddress: testWalletRegistryAddress, + randomBeaconAddress: testRandomBeaconAddress, + finalizedEthereumBlockNumber: testFinalizedEthereumBlock, + finalizedEthereumBlockHash: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + chainEvidencePublicKey: testChainEvidencePublicKey(), + maxEvidenceAge: time.Hour, + }, + } + record := &chainReconciliationEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "chain_reconciliation", + GeneratedAt: auditManifest.GeneratedAt, + SnapshotAggregateSHA256: auditManifest.Snapshot.AggregateSHA256, + }, + EthereumChainID: "1", + } + authenticateTestChainReconciliationEvidence(t, record) + return run, record + } + + validate := func( + t *testing.T, + run *auditRun, + record *chainReconciliationEvidence, + ) []string { + t.Helper() + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + return run.validateChainReconciliationEvidence(content) + } + + // The negative cases below differ from the passing one only in the + // settlement or the claim log, so requiring the reason pins each failure to + // the reconciliation instead of to some unrelated envelope defect. + assertBlockedBy := func(t *testing.T, violations []string, reason string) { + t.Helper() + + for _, violation := range violations { + if strings.Contains(violation, reason) { + return + } + } + t.Fatalf( + "expected a violation containing [%s], got: %v", + reason, + violations, + ) + } + + // The wallet these outcomes punish is present only to identify itself, not + // to be reconciled as persisted state, so the coverage that persisted + // wallets owe the chain evidence is another test's subject. The passing + // cases assert only that the settlement reconciliation itself is silent. + assertSettles := func(t *testing.T, violations []string) { + t.Helper() + + for _, violation := range violations { + if strings.Contains(violation, "inactivity claim") { + t.Fatalf( + "expected a corroborated settlement to reconcile, got: %s", + violation, + ) + } + } + } + + t.Run("settlement corroborated by a canonical claim log", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + walletID, + nonce, + claimBlock, + ) + assertSettles(t, validate(t, run, record)) + }) + + t.Run("submission whose settlement stayed unresolved", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + }) + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + walletID, + nonce, + claimBlock, + ) + // Even with the claim present on chain, an unresolved submission is + // unreconciled: the node cannot say the log it never resolved is its + // own. + assertBlockedBy( + t, + validate(t, run, record), + "could not resolve", + ) + }) + + // The check that keeps a real penalty from being borrowed: the log is + // authentic, canonical, and emitted by the expected registry — it just + // punishes a different wallet than the permit is about. Corroboration + // alone cannot tell those apart, because every real claim on the chain + // corroborates equally. + t.Run("valid claim log belonging to another wallet", func(t *testing.T) { + otherWalletID := walletID + otherWalletID[0] ^= 0xff + + borrowedReference, err := participation. + InactivityClaimSettlementReference(otherWalletID[:], nonce) + if err != nil { + t.Fatal(err) + } + + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: borrowedReference, + }) + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + otherWalletID, + nonce, + claimBlock, + ) + assertBlockedBy( + t, + validate(t, run, record), + "reports settling inactivity claim", + ) + }) + + // A penalty against a wallet this node holds no key material for is not a + // claim its heartbeat could have filed, so it cannot be bound and must not + // pass on the corroborating log alone. + t.Run("permit naming a wallet the snapshot does not hold", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + run.manifest.TBTCActiveWallets = nil + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + walletID, + nonce, + claimBlock, + ) + assertBlockedBy( + t, + validate(t, run, record), + "holds no signer material identifying that wallet", + ) + }) + + // Quarantined material identifies the punished wallet just as well: a + // rollback that quarantines a signer does not unmake the penalty its + // heartbeat filed. + t.Run("permit bound through quarantined material", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + run.manifest.TBTCActiveWallets = nil + run.manifest.TBTCQuarantinedOutputs = []tbtcQuarantineRecord{ + { + WalletStorageKey: strings.Repeat("f", 40), + SignerWalletID: hex.EncodeToString(walletID[:]), + SignerWalletPublicKeyHash: walletPublicKeyHash, + HasMembershipRecord: true, + }, + } + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + walletID, + nonce, + claimBlock, + ) + assertSettles(t, validate(t, run, record)) + }) + + t.Run("settlement with no matching claim log", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated WalletRegistry InactivityClaimed log", + ) + }) + + t.Run("claim log at a different nonce", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + walletID, + new(big.Int).Add(nonce, big.NewInt(1)), + claimBlock, + ) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated WalletRegistry InactivityClaimed log", + ) + }) + + t.Run("claim log for a different wallet", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + otherWalletID := walletID + otherWalletID[0] ^= 0xff + addTestInactivityClaimedReceipt( + t, + record, + testWalletRegistryAddress, + otherWalletID, + nonce, + claimBlock, + ) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated WalletRegistry InactivityClaimed log", + ) + }) + + t.Run("claim log from an unrelated contract", func(t *testing.T) { + run, record := newRunAndRecord(&participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + Reference: settledReference, + }) + addTestInactivityClaimedReceipt( + t, + record, + "0x2222222222222222222222222222222222222222", + walletID, + nonce, + claimBlock, + ) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated WalletRegistry InactivityClaimed log", + ) + }) +} + +// addTestRelayEntryReceipt appends an authenticated receipt carrying a real +// ABI-encoded RandomBeacon relay lifecycle log, then re-attests and re-signs +// the record. The log is built from the generated RandomBeacon ABI so the +// audit's decode path is exercised against the exact bytes the contract emits. +// +// The non-indexed values are the ones each event actually carries: the +// terminated group for a timeout, the selected group and previous entry for a +// request, and the submitter and entry for a delivery. +func addTestRelayEntryReceipt( + t *testing.T, + record *chainReconciliationEvidence, + address string, + eventName string, + requestID *big.Int, + blockNumber uint64, + values ...interface{}, +) { + t.Helper() + + parsed, err := beaconabi.RandomBeaconMetaData.GetAbi() + if err != nil { + t.Fatal(err) + } + event, ok := parsed.Events[eventName] + if !ok { + t.Fatalf("generated RandomBeacon ABI has no %s event", eventName) + } + + data, err := event.Inputs.NonIndexed().Pack(values...) + if err != nil { + t.Fatal(err) + } + + var requestTopic [32]byte + requestID.FillBytes(requestTopic[:]) + + addTestBeaconLogReceipt( + t, + record, + address, + []string{ + strings.ToLower(event.ID.Hex()), + "0x" + hex.EncodeToString(requestTopic[:]), + }, + data, + blockNumber, + ) +} + +// addTestGroupRegisteredReceipt appends an authenticated receipt carrying the +// RandomBeacon's own registration of a group: the registry index a request +// names the group by, and the hash of the public key it signs under. Both +// inputs are indexed, so the log carries them as topics and no data. +func addTestGroupRegisteredReceipt( + t *testing.T, + record *chainReconciliationEvidence, + address string, + groupID uint64, + groupPublicKey []byte, + blockNumber uint64, +) { + t.Helper() + + parsed, err := beaconabi.RandomBeaconMetaData.GetAbi() + if err != nil { + t.Fatal(err) + } + event, ok := parsed.Events["GroupRegistered"] + if !ok { + t.Fatal("generated RandomBeacon ABI has no GroupRegistered event") + } + + var groupTopic [32]byte + new(big.Int).SetUint64(groupID).FillBytes(groupTopic[:]) + + addTestBeaconLogReceipt( + t, + record, + address, + []string{ + strings.ToLower(event.ID.Hex()), + "0x" + hex.EncodeToString(groupTopic[:]), + "0x" + hex.EncodeToString(ethereumCrypto.Keccak256(groupPublicKey)), + }, + nil, + blockNumber, + ) +} + +// addTestBeaconLogReceipt appends an authenticated receipt carrying one raw +// RandomBeacon log, then re-attests and re-signs the record. +func addTestBeaconLogReceipt( + t *testing.T, + record *chainReconciliationEvidence, + address string, + topics []string, + data []byte, + blockNumber uint64, +) { + t.Helper() + + transactionHash := fmt.Sprintf( + "0x%064x", + blockNumber*31+uint64(len(record.Receipts))+1, + ) + record.Receipts = append(record.Receipts, ethereumReceiptEvidence{ + TransactionHash: transactionHash, + BlockHash: testCanonicalEthereumBlockHash(blockNumber), + BlockNumber: blockNumber, + TransactionIndex: uint64(len(record.Receipts)), + Status: 1, + Logs: []ethereumRawLogEvidence{ + { + Address: address, + Topics: topics, + Data: "0x" + hex.EncodeToString(data), + LogIndex: 0, + }, + }, + }) + + record.CollectorAttestation.CanonicalBlocks = append( + record.CollectorAttestation.CanonicalBlocks, + ethereumCanonicalBlockEvidence{ + BlockNumber: blockNumber, + BlockHash: testCanonicalEthereumBlockHash(blockNumber), + }, + ) + sort.Slice( + record.CollectorAttestation.CanonicalBlocks, + func(i, j int) bool { + return record.CollectorAttestation.CanonicalBlocks[i].BlockNumber < + record.CollectorAttestation.CanonicalBlocks[j].BlockNumber + }, + ) + deduplicated := record.CollectorAttestation.CanonicalBlocks[:0] + for i, block := range record.CollectorAttestation.CanonicalBlocks { + if i > 0 && block.BlockNumber == deduplicated[len(deduplicated)-1]. + BlockNumber { + continue + } + deduplicated = append(deduplicated, block) + } + record.CollectorAttestation.CanonicalBlocks = deduplicated + + resignTestChainReconciliationEvidence(t, record) +} + +// TestValidateChainReconciliationEvidence_RelayTimeoutSettlement asserts a +// relay entry timeout penalty a node recorded as its permit's result is +// reconciled against the RandomBeacon's own logs rather than accepted on the +// node's word. +// +// A node that filed a report which reverted, was dropped, or lost the race to +// another reporter renders exactly the same reference as one whose report the +// beacon accepted, so nothing in the journal tells the two apart. The +// authenticated logs do: the timeout must exist, the request it terminated +// must be the one this permit was issued for, and no delivered entry may +// answer that same request. +func TestValidateChainReconciliationEvidence_RelayTimeoutSettlement( + t *testing.T, +) { + capturedAt := time.Now().UTC().Add(-time.Minute) + + const requestStartBlock = uint64(9_100) + const terminatedGroupID = uint64(4) + const timeoutBlock = uint64(9_200) + requestID := big.NewInt(77) + + // The previous entry the terminated request was signing over. The beacon + // carries it in the request log; the audit only needs the log to exist at + // the permit's block, so any well-formed byte string serves here. + previousEntry := []byte{0x0a, 0x0b, 0x0c} + + permit := participation.PermitSnapshot{ + Ceremony: participation.BeaconTimeoutReport, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: requestStartBlock, + WorkID: participation.BeaconRelayWorkID(requestStartBlock), + PermitID: "monitor", + IdentityBound: true, + } + + newRunAndRecord := func( + reference string, + ) (*auditRun, *chainReconciliationEvidence) { + auditManifest := &manifest{ + GeneratedAt: time.Now().UTC(), + Snapshot: snapshotIdentity{ + AggregateSHA256: strings.Repeat("c", 64), + }, + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + SchemaVersion: participation.QuiescenceSnapshotSchemaVersion, + CapturedAt: capturedAt, + ActivePermits: []participation.PermitSnapshot{permit}, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceEthereumTransaction, + Reference: reference, + }, + }, + }, + }, + } + run := &auditRun{ + manifest: auditManifest, + expected: expectedIdentityInputs{ + ethereumChainID: "1", + walletRegistryAddress: testWalletRegistryAddress, + randomBeaconAddress: testRandomBeaconAddress, + finalizedEthereumBlockNumber: testFinalizedEthereumBlock, + finalizedEthereumBlockHash: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + chainEvidencePublicKey: testChainEvidencePublicKey(), + maxEvidenceAge: time.Hour, + }, + } + record := &chainReconciliationEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "chain_reconciliation", + GeneratedAt: auditManifest.GeneratedAt, + SnapshotAggregateSHA256: auditManifest.Snapshot.AggregateSHA256, + }, + EthereumChainID: "1", + } + authenticateTestChainReconciliationEvidence(t, record) + return run, record + } + + addRequest := func( + t *testing.T, + record *chainReconciliationEvidence, + blockNumber uint64, + ) { + t.Helper() + + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntryRequested", + requestID, + blockNumber, + terminatedGroupID, + previousEntry, + ) + } + + addTimeout := func( + t *testing.T, + record *chainReconciliationEvidence, + address string, + groupID uint64, + ) { + t.Helper() + + addTestRelayEntryReceipt( + t, + record, + address, + "RelayEntryTimedOut", + requestID, + timeoutBlock, + groupID, + ) + } + + validate := func( + t *testing.T, + run *auditRun, + record *chainReconciliationEvidence, + ) []string { + t.Helper() + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + return run.validateChainReconciliationEvidence(content) + } + + // The negative cases below differ from the passing one only in the logs the + // evidence carries, so requiring the reason pins each failure to the + // settlement reconciliation instead of to some unrelated envelope defect. + assertBlockedBy := func(t *testing.T, violations []string, reason string) { + t.Helper() + + for _, violation := range violations { + if strings.Contains(violation, reason) { + return + } + } + t.Fatalf( + "expected a violation containing [%s], got: %v", + reason, + violations, + ) + } + + assertSettles := func(t *testing.T, violations []string) { + t.Helper() + + for _, violation := range violations { + if strings.Contains(violation, "relay entry timeout settlement") { + t.Fatalf( + "expected a corroborated settlement to reconcile, got: %s", + violation, + ) + } + } + } + + settledReference := testTimeoutSettlementReference( + t, + requestStartBlock, + requestID.Int64(), + terminatedGroupID, + ) + + t.Run("settlement corroborated by canonical beacon logs", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock) + addTimeout(t, record, testRandomBeaconAddress, terminatedGroupID) + assertSettles(t, validate(t, run, record)) + }) + + // The report the node filed is all the journal ever holds. Without the + // beacon's own termination log the penalty may never have happened, and an + // unproven penalty is exactly what the barrier exists to hold. + t.Run("settlement with no matching timeout log", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryTimedOut log", + ) + }) + + t.Run("timeout log terminating a different group", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock) + addTimeout(t, record, testRandomBeaconAddress, terminatedGroupID+1) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryTimedOut log", + ) + }) + + // An identically shaped event from an attacker-deployed contract names no + // penalty the beacon ever applied. + t.Run("timeout log from an unrelated contract", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock) + addTimeout(t, record, testWalletRegistryAddress, terminatedGroupID) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryTimedOut log", + ) + }) + + // The termination is real and this node may even have reported it, but + // nothing yet says the request it terminated is the one this permit was + // issued for. Without the request log the settlement cannot be bound to the + // permit at all. + t.Run("settlement with no matching request log", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addTimeout(t, record, testRandomBeaconAddress, terminatedGroupID) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryRequested log", + ) + }) + + // The check that keeps a real penalty from being borrowed: every genuine + // termination on the chain corroborates equally, so the request it belongs + // to has to sit in the block this permit names. + t.Run("request log at another block", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock+1) + addTimeout(t, record, testRandomBeaconAddress, terminatedGroupID) + assertBlockedBy( + t, + validate(t, run, record), + "not the request start block the permit was issued for", + ) + }) + + // Nothing in the logs says which block a caller meant, so a request the + // evidence places twice binds no permit rather than binding it to either. + t.Run("request logged at two blocks", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock) + addRequest(t, record, requestStartBlock+1) + addTimeout(t, record, testRandomBeaconAddress, terminatedGroupID) + assertBlockedBy( + t, + validate(t, run, record), + "at more than one block", + ) + }) + + // A delivered entry and a timeout are mutually exclusive endings. Evidence + // carrying both settles nothing, whichever one the node recorded. + t.Run("delivered entry answering the same request", func(t *testing.T) { + run, record := newRunAndRecord(settledReference) + addRequest(t, record, requestStartBlock) + addTimeout(t, record, testRandomBeaconAddress, terminatedGroupID) + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntrySubmitted", + requestID, + timeoutBlock, + common.HexToAddress(testRandomBeaconAddress), + []byte{0x01, 0x02}, + ) + assertBlockedBy( + t, + validate(t, run, record), + "a delivered entry and a timeout cannot both settle one request", + ) + }) + + // A monitor that ended without a penalty records no settlement at all, so + // there is nothing for this pass to corroborate and the absent beacon logs + // are not held against it. + t.Run("exhausted report naming no settlement", func(t *testing.T) { + run, record := newRunAndRecord("") + outcomes := run.manifest.ParticipationTerminalOutcomes.Outcomes + outcomes[0].Outcome = participation.TerminalOutcomeExhausted + outcomes[0].Evidence = participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceNoThreshold, + } + assertSettles(t, validate(t, run, record)) + }) +} + +// TestValidateChainReconciliationEvidence_RelayEntryResult asserts a recovered +// relay entry is bound to the beacon's own request before it closes a signing +// permit. +// +// The journal pass proves the entry is a threshold signature by a group whose +// key the snapshot holds, and every entry the beacon ever produced keeps that +// property forever. What it cannot prove is that this permit's ceremony +// produced it: the record's only tie to a request is a start block the node +// wrote beside an entry it chose. A historical entry relabelled with a live +// permit's block verifies exactly as well and was never in this journal for the +// replay guard to catch. +func TestValidateChainReconciliationEvidence_RelayEntryResult(t *testing.T) { + capturedAt := time.Now().UTC().Add(-time.Minute) + + const requestStartBlock = uint64(9_400) + const groupSecret = int64(0x5eed) + const otherGroupSecret = int64(0x5eee) + const selectedGroupID = uint64(1) + const otherGroupID = uint64(7) + + // The uncompressed point is the form the registry stores and the beacon + // hashes a group's identity from; the reference carries the compressed one. + onChainGroupPublicKey := func(secret int64) []byte { + return new(bn256.G2).ScalarBaseMult(big.NewInt(secret)).Marshal() + } + + reference := testRelayEntryReference( + requestStartBlock, + groupSecret, + "relay-entry-reconciliation", + ) + _, _, previousEntry, entry, err := participation. + ParseBeaconRelayEntryReference(reference) + if err != nil { + t.Fatal(err) + } + requestID := testRelayRequestID(requestStartBlock) + + permit := participation.PermitSnapshot{ + Ceremony: participation.BeaconRelaySigning, + Mode: participation.ModeSecurityV2.String(), + CanonicalStartBlock: requestStartBlock, + WorkID: participation.BeaconRelayWorkID(requestStartBlock), + PermitID: "1", + IdentityBound: true, + } + + newRunAndRecord := func() (*auditRun, *chainReconciliationEvidence) { + auditManifest := &manifest{ + GeneratedAt: time.Now().UTC(), + Snapshot: snapshotIdentity{ + AggregateSHA256: strings.Repeat("c", 64), + }, + QuiescenceSnapshot: &participation.QuiescenceSnapshot{ + SchemaVersion: participation.QuiescenceSnapshotSchemaVersion, + CapturedAt: capturedAt, + ActivePermits: []participation.PermitSnapshot{permit}, + }, + ParticipationTerminalOutcomes: &participation.TerminalOutcomeJournal{ + SchemaVersion: participation.TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: capturedAt, + Outcomes: []participation.TerminalOutcomeRecord{ + { + RecordedAt: time.Now().UTC(), + Permit: permit, + Outcome: participation.TerminalOutcomeCompleted, + Evidence: participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: reference, + }, + }, + }, + }, + } + run := &auditRun{ + manifest: auditManifest, + expected: expectedIdentityInputs{ + ethereumChainID: "1", + walletRegistryAddress: testWalletRegistryAddress, + randomBeaconAddress: testRandomBeaconAddress, + finalizedEthereumBlockNumber: testFinalizedEthereumBlock, + finalizedEthereumBlockHash: testCanonicalEthereumBlockHash( + testFinalizedEthereumBlock, + ), + chainEvidencePublicKey: testChainEvidencePublicKey(), + maxEvidenceAge: time.Hour, + }, + } + chainRecord := &chainReconciliationEvidence{ + evidenceEnvelope: evidenceEnvelope{ + SchemaVersion: evidenceSchemaVersion, + EvidenceType: "chain_reconciliation", + GeneratedAt: auditManifest.GeneratedAt, + SnapshotAggregateSHA256: auditManifest.Snapshot.AggregateSHA256, + }, + EthereumChainID: "1", + } + authenticateTestChainReconciliationEvidence(t, chainRecord) + return run, chainRecord + } + + addRequestFromGroup := func( + t *testing.T, + record *chainReconciliationEvidence, + id *big.Int, + blockNumber uint64, + selectedGroupID uint64, + requestPreviousEntry []byte, + ) { + t.Helper() + + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntryRequested", + id, + blockNumber, + selectedGroupID, + requestPreviousEntry, + ) + } + + addRequest := func( + t *testing.T, + record *chainReconciliationEvidence, + id *big.Int, + blockNumber uint64, + requestPreviousEntry []byte, + ) { + t.Helper() + + addRequestFromGroup( + t, + record, + id, + blockNumber, + selectedGroupID, + requestPreviousEntry, + ) + } + + // The selected group's registration is what binds the index the request + // names to the key the record signs under, and the audit requires it, so + // every case that is meant to settle has to supply it. + addRegistration := func( + t *testing.T, + record *chainReconciliationEvidence, + groupID uint64, + secret int64, + ) { + t.Helper() + + addTestGroupRegisteredReceipt( + t, + record, + testRandomBeaconAddress, + groupID, + onChainGroupPublicKey(secret), + requestStartBlock-100, + ) + } + + addSelectedRegistration := func( + t *testing.T, + record *chainReconciliationEvidence, + ) { + t.Helper() + + addRegistration(t, record, selectedGroupID, groupSecret) + } + + addSubmission := func( + t *testing.T, + record *chainReconciliationEvidence, + acceptedEntry []byte, + ) { + t.Helper() + + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntrySubmitted", + requestID, + requestStartBlock+2, + common.HexToAddress(testRandomBeaconAddress), + acceptedEntry, + ) + } + + validate := func( + t *testing.T, + run *auditRun, + record *chainReconciliationEvidence, + ) []string { + t.Helper() + + content, err := json.Marshal(record) + if err != nil { + t.Fatal(err) + } + return run.validateChainReconciliationEvidence(content) + } + + assertBlockedBy := func(t *testing.T, violations []string, reason string) { + t.Helper() + + for _, violation := range violations { + if strings.Contains(violation, reason) { + return + } + } + t.Fatalf( + "expected a violation containing [%s], got: %v", + reason, + violations, + ) + } + + assertSettles := func(t *testing.T, violations []string) { + t.Helper() + + for _, violation := range violations { + if strings.Contains(violation, "reports relay entry") { + t.Fatalf( + "expected a corroborated relay entry to reconcile, got: %s", + violation, + ) + } + } + } + + t.Run("entry answering the permit's own request", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addSelectedRegistration(t, record) + assertSettles(t, validate(t, run, record)) + }) + + // The entry verifies under its group and answers the block the permit + // names, and still nothing says the beacon ever made that request. + t.Run("entry answering no logged request", func(t *testing.T) { + run, record := newRunAndRecord() + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryRequested log makes a "+ + "request over that previous entry", + ) + }) + + // The relabelling case: a real entry from an earlier request carries that + // request's previous entry, which the request at this permit's block is not + // signing over. + t.Run("request at the block over another previous entry", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest( + t, + record, + requestID, + requestStartBlock, + new(bn256.G1).ScalarBaseMult(big.NewInt(3)).Marshal(), + ) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryRequested log makes a "+ + "request over that previous entry", + ) + }) + + // The request the entry answers is real, but it was made in a block this + // permit was not issued for. + t.Run("request at another block", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock+1, previousEntry) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon RelayEntryRequested log makes a "+ + "request over that previous entry", + ) + }) + + // The beacon's request names the group it selected by a registry index, and + // the registration binds that index to the key the group signs under. Here + // the two agree with the record. + t.Run("registration binding the selected group's own key", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addTestGroupRegisteredReceipt( + t, + record, + testRandomBeaconAddress, + selectedGroupID, + onChainGroupPublicKey(groupSecret), + requestStartBlock-100, + ) + assertSettles(t, validate(t, run, record)) + }) + + // The entry is a real threshold signature over this very request's previous + // entry, and the pairing check passes, because the node holds a membership + // of the group that produced it. It is simply not the group the beacon + // selected, so the work it records is work the selected group never did. + t.Run("entry signed by a group the request did not select", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addTestGroupRegisteredReceipt( + t, + record, + testRandomBeaconAddress, + selectedGroupID, + onChainGroupPublicKey(otherGroupSecret), + requestStartBlock-100, + ) + assertBlockedBy( + t, + validate(t, run, record), + "registered under public key hash", + ) + }) + + t.Run("selected group registered under two keys", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addTestGroupRegisteredReceipt( + t, + record, + testRandomBeaconAddress, + selectedGroupID, + onChainGroupPublicKey(groupSecret), + requestStartBlock-100, + ) + addTestGroupRegisteredReceipt( + t, + record, + testRandomBeaconAddress, + selectedGroupID, + onChainGroupPublicKey(otherGroupSecret), + requestStartBlock-99, + ) + assertBlockedBy( + t, + validate(t, run, record), + "under more than one public key", + ) + }) + + t.Run("one request selecting two groups", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addRequestFromGroup( + t, + record, + requestID, + requestStartBlock, + otherGroupID, + previousEntry, + ) + assertBlockedBy( + t, + validate(t, run, record), + "select more than one group for request", + ) + }) + + // One identifier answering two requests would let either request's evidence + // close the other's permit. + t.Run("one identifier answering two requests", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addRequest( + t, + record, + requestID, + requestStartBlock, + new(bn256.G1).ScalarBaseMult(big.NewInt(3)).Marshal(), + ) + assertBlockedBy( + t, + validate(t, run, record), + "more than one request under identifier", + ) + }) + + // The registration is older than the request, so nothing about gathering + // evidence around the request produces it — the generator has to be told to + // fetch it, and the contract says so. A registration for some other group + // binds the selected index to nothing, which is the same position as having + // supplied none at all. + t.Run("registration of an unrelated group", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addRegistration(t, record, otherGroupID, otherGroupSecret) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon GroupRegistered log registers "+ + "group [1]", + ) + }) + + // The discriminating case. Everything here is exactly what an honest + // bundle looks like — a real request at the permit's block, a real + // threshold entry over its previous entry that the pairing check accepts — + // except that the group which produced the entry is not the one the beacon + // selected. Nothing in the bundle says so, because the receipt that would + // have said so is the one left out. Were absence read as consent, omitting + // it would be all it takes to close a selected group's permit with another + // group's work. + t.Run("wrong group with the registration omitted", func(t *testing.T) { + run, record := newRunAndRecord() + addRequestFromGroup( + t, + record, + requestID, + requestStartBlock, + otherGroupID, + previousEntry, + ) + assertBlockedBy( + t, + validate(t, run, record), + "no authenticated RandomBeacon GroupRegistered log registers "+ + "group [7]", + ) + }) + + // The same omission on the selection side. Every log that indexes a request + // identity also records the group it selected, so the audit's own decoder + // cannot produce this pair; the join is asked directly rather than through + // evidence that cannot express it. It still has to block, because what + // makes the selection mandatory is that an entry with no selected group to + // be held to is exactly the entry that was never checked. + t.Run("request naming no selected group", func(t *testing.T) { + violation := relayEntryGroupSelectionViolation( + &relayEntryLifecycleLogs{ + requestGroups: make(map[string]string), + ambiguousRequestGroups: make(map[string]struct{}), + registeredGroupKeys: make(map[string]string), + ambiguousGroupRegistrations: make(map[string]struct{}), + }, + 0, + reference, + requestID.String(), + nil, + ) + if !strings.Contains( + violation, + "names the group selected to answer request", + ) { + t.Fatalf( + "expected an unselected group to block, got: [%s]", + violation, + ) + } + }) + + t.Run("two requests sharing one identity", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addRequest( + t, + record, + new(big.Int).Add(requestID, big.NewInt(1)), + requestStartBlock, + previousEntry, + ) + assertBlockedBy( + t, + validate(t, run, record), + "more than one request over that previous entry", + ) + }) + + t.Run("submission accepting the entry the node named", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addSelectedRegistration(t, record) + addSubmission(t, record, entry) + assertSettles(t, validate(t, run, record)) + }) + + t.Run("submission accepting a different entry", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addSelectedRegistration(t, record) + addSubmission( + t, + record, + new(bn256.G1).ScalarBaseMult(big.NewInt(5)).Marshal(), + ) + assertBlockedBy( + t, + validate(t, run, record), + "accepted a different entry", + ) + }) + + t.Run("two submissions accepting different entries", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addSelectedRegistration(t, record) + addSubmission(t, record, entry) + addSubmission( + t, + record, + new(bn256.G1).ScalarBaseMult(big.NewInt(5)).Marshal(), + ) + assertBlockedBy( + t, + validate(t, run, record), + "more than one entry accepted for request", + ) + }) + + // A group's threshold recovers the entry whoever publishes it, so a + // recovery whose submission reverted, was dropped, or lost the race is + // still that ceremony's durable result. Requiring a submission would refuse + // a completed ceremony for a transaction outcome that says nothing about + // it. + t.Run("recovered entry no submission answers", func(t *testing.T) { + run, record := newRunAndRecord() + addRequest(t, record, requestID, requestStartBlock, previousEntry) + addSelectedRegistration(t, record) + addTestRelayEntryReceipt( + t, + record, + testRandomBeaconAddress, + "RelayEntryTimedOut", + requestID, + requestStartBlock+2, + uint64(1), + ) + assertSettles(t, validate(t, run, record)) + }) +} diff --git a/cmd/participation_bounds_test.go b/cmd/participation_bounds_test.go new file mode 100644 index 0000000000..ffe1cd417c --- /dev/null +++ b/cmd/participation_bounds_test.go @@ -0,0 +1,41 @@ +package cmd + +import ( + "testing" + + "github.com/keep-network/keep-core/pkg/beacon" + "github.com/keep-network/keep-core/pkg/chain/ethereum" + "github.com/keep-network/keep-core/pkg/tbtc" +) + +// TestMaximumLegacyCompletionBlocksAcrossProtocols is the cross-package drift +// assertion for the combined in-flight completion bound: the starting input +// for quiesce deadlines, roster retention, and the release-manifest grace +// derivation. The beacon side is derived from the configuration the +// production Ethereum adapter actually supplies, so adapter drift fails the +// assertion, not only a changed literal. It fails when either protocol's +// bound moves without those derived values being deliberately re-reviewed. +func TestMaximumLegacyCompletionBlocksAcrossProtocols(t *testing.T) { + tbtcBound := tbtc.MaximumLegacyCompletionBlocks() + beaconBound, err := beacon.MaximumLegacyCompletionBlocks( + (ðereum.BeaconChain{}).GetConfig(), + ) + if err != nil { + t.Fatalf("unexpected beacon completion bound error: [%v]", err) + } + + combined := tbtcBound + if beaconBound > combined { + combined = beaconBound + } + + if tbtcBound != 1200 { + t.Errorf("tBTC completion bound changed: expected [1200], got [%d]", tbtcBound) + } + if beaconBound != 136 { + t.Errorf("beacon completion bound changed: expected [136], got [%d]", beaconBound) + } + if combined != 1200 { + t.Errorf("combined completion bound changed: expected [1200], got [%d]", combined) + } +} diff --git a/cmd/quiesce_lifecycle_test.go b/cmd/quiesce_lifecycle_test.go new file mode 100644 index 0000000000..6f34801d73 --- /dev/null +++ b/cmd/quiesce_lifecycle_test.go @@ -0,0 +1,570 @@ +package cmd + +import ( + "context" + "errors" + "math" + "os" + "strings" + "sync/atomic" + "syscall" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +func TestAwaitQuiesce_NaturalCompletion(t *testing.T) { + quiesceDone := make(chan struct{}) + close(quiesceDone) + + reason := awaitQuiesce(quiesceDone, make(chan os.Signal), time.Hour) + if reason != "completed" { + t.Errorf("expected reason [completed], got [%s]", reason) + } +} + +func TestAwaitQuiesce_SecondSignalForces(t *testing.T) { + signals := make(chan os.Signal, 1) + signals <- syscall.SIGTERM + + reason := awaitQuiesce(make(chan struct{}), signals, time.Hour) + if reason != "forced_by_signal" { + t.Errorf("expected reason [forced_by_signal], got [%s]", reason) + } +} + +func TestAwaitQuiesce_BackstopDeadline(t *testing.T) { + reason := awaitQuiesce( + make(chan struct{}), + make(chan os.Signal), + time.Millisecond, + ) + if reason != "backstop_deadline" { + t.Errorf("expected reason [backstop_deadline], got [%s]", reason) + } +} + +// TestSignalLifecycleController_FirstSignalPreventsNewPermits proves the +// controller acts on the first termination signal the moment it arrives — +// the model of a signal received while startup is still initializing +// components: the gate refuses every subsequent Begin, the in-flight permit +// keeps draining, and only after it completes does the controller report +// shutdown and cancel the run context. +func TestSignalLifecycleController_FirstSignalPreventsNewPermits(t *testing.T) { + localChain := local_v1.Connect(10, 5) + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: 1_000_000}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + defer gate.Close() + + // The in-flight ceremony that must survive the first signal and keep the + // drain open until it completes. + permit, err := gate.Begin(participation.BeaconDKG, 1) + if err != nil { + t.Fatalf("cannot begin the in-flight ceremony: [%v]", err) + } + + runCtx, cancelRunCtx := context.WithCancel(context.Background()) + defer cancelRunCtx() + + signals := make(chan os.Signal, 2) + evidenceWindow := participation.NewCutoverEvidenceWindowSignal() + shutdownChan := startSignalLifecycleController( + runCtx, + cancelRunCtx, + gate, + evidenceWindow, + signals, + time.Hour, + time.Hour, + ) + + signals <- syscall.SIGTERM + + // The controller quiesces asynchronously; a Begin that still succeeds + // lost the race with the quiesce transition and its permit is returned + // before retrying. Once the refusal appears it must be the quiesce + // sentinel. + deadline := time.Now().Add(10 * time.Second) + for { + extraPermit, err := gate.Begin(participation.BeaconDKG, 1) + if err != nil { + if !errors.Is(err, participation.ErrQuiescing) { + t.Fatalf("expected the quiesce refusal, got [%v]", err) + } + break + } + extraPermit.Close() + + if time.Now().After(deadline) { + t.Fatal("the gate never started refusing new permits") + } + time.Sleep(10 * time.Millisecond) + } + + if !evidenceWindow.Active() { + t.Fatal("the termination signal did not open the rollback evidence window") + } + if evidenceWindow.SetActive(false) { + t.Fatal("the rollback evidence window was not held active during quiescence") + } + + // The drain must wait for the in-flight permit: no shutdown report and no + // run-context cancellation may occur while it is open. + select { + case err := <-shutdownChan: + t.Fatalf("shutdown reported while a permit was active: [%v]", err) + case <-runCtx.Done(): + t.Fatal("run context canceled while a permit was active") + default: + } + + permit.Close() + + select { + case err := <-shutdownChan: + if err == nil || !strings.Contains(err.Error(), "signal") { + t.Errorf("expected a signal shutdown report, got [%v]", err) + } + case <-time.After(10 * time.Second): + t.Fatal("no shutdown report after the drain completed") + } + + select { + case <-runCtx.Done(): + case <-time.After(10 * time.Second): + t.Fatal("the run context was not canceled after the shutdown report") + } +} + +func TestAwaitForcedCancellationCleanup_Drained(t *testing.T) { + drained := make(chan struct{}) + close(drained) + + reason := awaitForcedCancellationCleanup(drained, time.Hour) + if reason != cleanupReasonDrained { + t.Errorf( + "expected reason [%s], got [%s]", + cleanupReasonDrained, + reason, + ) + } +} + +func TestAwaitForcedCancellationCleanup_AllowanceExceeded(t *testing.T) { + reason := awaitForcedCancellationCleanup( + make(chan struct{}), + time.Millisecond, + ) + if reason != cleanupReasonAllowanceExceeded { + t.Errorf( + "expected reason [%s], got [%s]", + cleanupReasonAllowanceExceeded, + reason, + ) + } +} + +// TestForcedCancellationAllowance_BoundToManifestAllowance pins the runtime +// phase-two wait to the compiled allowance the release manifest adds on top +// of the in-process backstop, and pins the manifest's grace to strictly +// outlast that wait: the room the termination grace reserves after the +// backstop must be the full cleanup allowance the controller actually waits +// plus the positive exit headroom for the teardown running outside both +// timers — never exactly the allowance, or SIGKILL can land inside teardown. +// The same identity is then checked against the checked-in repository +// manifest, so a reviewed document drifting away from the runtime wait fails +// here even though its own numbers are internally coherent. +func TestForcedCancellationAllowance_BoundToManifestAllowance(t *testing.T) { + grace, err := deriveTerminationGrace( + compiledForcedCancellationAllowanceSeconds, + ) + if err != nil { + t.Fatalf("unexpected derivation error: [%v]", err) + } + + expected := time.Duration(grace.ForcedCancellationAllowanceSeconds) * + time.Second + if got := forcedCancellationAllowance(); got != expected { + t.Errorf( + "expected the runtime allowance [%s] to match the manifest "+ + "allowance, got [%s]", + expected, + got, + ) + } + + graceBeyondBackstop := grace.TerminationGracePeriodSeconds - + grace.InProcessBackstopSeconds + runtimeWaitSeconds := uint64(forcedCancellationAllowance() / time.Second) + if graceBeyondBackstop != runtimeWaitSeconds+grace.ProcessExitHeadroomSeconds { + t.Errorf( + "the termination grace reserves [%d]s after the backstop, but "+ + "the runtime waits [%s] and the exit headroom is [%d]s", + graceBeyondBackstop, + forcedCancellationAllowance(), + grace.ProcessExitHeadroomSeconds, + ) + } + if graceBeyondBackstop <= runtimeWaitSeconds { + t.Errorf( + "the termination grace must reserve strictly more than the "+ + "[%d]s runtime cleanup wait after the backstop, got [%d]s", + runtimeWaitSeconds, + graceBeyondBackstop, + ) + } + + // The derivation above proves the arithmetic; the repository manifest + // must also record exactly the allowance the runtime consumes, otherwise + // the reviewed scaffolds derived from it budget a cleanup window the + // process does not observe. Loading the checked-in file makes this a + // drift test against the repository document, not only against the + // in-memory derivation. + repositoryManifest, err := loadReleaseManifest(releaseManifestRepositoryPath) + if err != nil { + t.Fatalf("cannot load the repository manifest: [%v]", err) + } + recordedAllowanceSeconds := + repositoryManifest.TerminationGrace.ForcedCancellationAllowanceSeconds + if recordedAllowanceSeconds != runtimeWaitSeconds { + t.Errorf( + "the repository manifest records a cleanup allowance of [%d]s, "+ + "but the runtime cleanup wait consumes [%d]s", + recordedAllowanceSeconds, + runtimeWaitSeconds, + ) + } +} + +// TestSignalLifecycleController_TeardownFitsExitHeadroom walks the forced +// shutdown from the original termination instant to exit readiness — the +// shutdown report delivered and the run context canceled — with both timed +// waits fully consumed by a wedged permit owner, and requires everything +// outside those two waits (controller scheduling, the quiesce and close +// calls, logging, report delivery, context cancellation) to fit within the +// exit headroom the release manifest reserves for it. The service manager +// counts its grace from the same instant this test starts counting, so this +// is the in-process proof that the manifest's grace bounds the complete +// sequence, not just the sum of the two timers. +func TestSignalLifecycleController_TeardownFitsExitHeadroom(t *testing.T) { + localChain := local_v1.Connect(10, 5) + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: 1_000_000}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + defer gate.Close() + + // Never released: the drain can only end through the backstop and the + // cleanup wait can only end through the allowance, so the elapsed time + // beyond those two budgets is exactly the controller's own overhead. + permit, err := gate.Begin(participation.BeaconDKG, 1) + if err != nil { + t.Fatalf("cannot begin the in-flight ceremony: [%v]", err) + } + defer permit.Close() + + runCtx, cancelRunCtx := context.WithCancel(context.Background()) + defer cancelRunCtx() + + backstop := 50 * time.Millisecond + allowance := 50 * time.Millisecond + signals := make(chan os.Signal, 1) + shutdownChan := startSignalLifecycleController( + runCtx, + cancelRunCtx, + gate, + participation.NewCutoverEvidenceWindowSignal(), + signals, + backstop, + allowance, + ) + + grace, err := deriveTerminationGrace( + compiledForcedCancellationAllowanceSeconds, + ) + if err != nil { + t.Fatalf("unexpected derivation error: [%v]", err) + } + exitHeadroom := time.Duration(grace.ProcessExitHeadroomSeconds) * + time.Second + + terminationInstant := time.Now() + signals <- syscall.SIGTERM + + select { + case err := <-shutdownChan: + if err == nil || !strings.Contains(err.Error(), "signal") { + t.Errorf("expected a signal shutdown report, got [%v]", err) + } + case <-time.After(backstop + allowance + exitHeadroom): + t.Fatal( + "no shutdown report within the backstop, the allowance, and " + + "the exit headroom", + ) + } + + select { + case <-runCtx.Done(): + case <-time.After(exitHeadroom): + t.Fatal("the run context was not canceled after the shutdown report") + } + + overhead := time.Since(terminationInstant) - backstop - allowance + if overhead >= exitHeadroom { + t.Errorf( + "the controller consumed [%s] beyond the two timed waits, more "+ + "than the [%s] exit headroom the termination grace reserves", + overhead, + exitHeadroom, + ) + } +} + +// TestSignalLifecycleController_JoinsForcedCancellationCleanup proves the +// forced path is two-phase: after the second signal ends the drain, the +// controller must not report shutdown or cancel the run context until the +// owner of the force-canceled permit has finished its delayed cleanup — the +// model of a quarantine persistence write racing process exit — and released +// the permit. +func TestSignalLifecycleController_JoinsForcedCancellationCleanup(t *testing.T) { + localChain := local_v1.Connect(10, 5) + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: 1_000_000}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + defer gate.Close() + + permit, err := gate.Begin(participation.BeaconDKG, 1) + if err != nil { + t.Fatalf("cannot begin the in-flight ceremony: [%v]", err) + } + + runCtx, cancelRunCtx := context.WithCancel(context.Background()) + defer cancelRunCtx() + + signals := make(chan os.Signal, 2) + shutdownChan := startSignalLifecycleController( + runCtx, + cancelRunCtx, + gate, + participation.NewCutoverEvidenceWindowSignal(), + signals, + time.Hour, + time.Hour, + ) + + // The permit owner: it starts its cleanup only when the gate cancels the + // permit, holds the permit across a deliberately slow persistence write, + // and releases it only afterwards — exactly the shape of the quarantine + // paths in the beacon and tBTC nodes. + quarantineFinished := make(chan struct{}) + var runCtxEndedDuringCleanup atomic.Bool + go func() { + <-permit.Context().Done() + time.Sleep(200 * time.Millisecond) + if runCtx.Err() != nil { + runCtxEndedDuringCleanup.Store(true) + } + close(quarantineFinished) + permit.Close() + }() + + // The first signal quiesces; the second forces the drain to end while the + // permit is still held. + signals <- syscall.SIGTERM + signals <- syscall.SIGTERM + + select { + case err := <-shutdownChan: + select { + case <-quarantineFinished: + default: + t.Fatal( + "shutdown reported before the delayed cleanup finished", + ) + } + if err == nil || !strings.Contains(err.Error(), "signal") { + t.Errorf("expected a signal shutdown report, got [%v]", err) + } + case <-time.After(10 * time.Second): + t.Fatal("no shutdown report after the cleanup finished") + } + + if runCtxEndedDuringCleanup.Load() { + t.Fatal("run context canceled while the cleanup was still running") + } + + select { + case <-runCtx.Done(): + case <-time.After(10 * time.Second): + t.Fatal("the run context was not canceled after the shutdown report") + } +} + +// TestSignalLifecycleController_CancellationAllowanceBoundsTheWait proves the +// cleanup join cannot wedge the shutdown: a permit owner that never releases +// its canceled permit delays the shutdown report by exactly the reviewed +// allowance, not forever. +func TestSignalLifecycleController_CancellationAllowanceBoundsTheWait( + t *testing.T, +) { + localChain := local_v1.Connect(10, 5) + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: 1_000_000}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + defer gate.Close() + + // Deliberately never released before the shutdown report: the owner is + // modeled as wedged. + permit, err := gate.Begin(participation.BeaconDKG, 1) + if err != nil { + t.Fatalf("cannot begin the in-flight ceremony: [%v]", err) + } + defer permit.Close() + + runCtx, cancelRunCtx := context.WithCancel(context.Background()) + defer cancelRunCtx() + + allowance := 150 * time.Millisecond + signals := make(chan os.Signal, 2) + shutdownChan := startSignalLifecycleController( + runCtx, + cancelRunCtx, + gate, + participation.NewCutoverEvidenceWindowSignal(), + signals, + time.Hour, + allowance, + ) + + waitStart := time.Now() + signals <- syscall.SIGTERM + signals <- syscall.SIGTERM + + select { + case err := <-shutdownChan: + if elapsed := time.Since(waitStart); elapsed < allowance { + t.Errorf( + "shutdown reported after [%s], before the [%s] allowance "+ + "was consumed", + elapsed, + allowance, + ) + } + if err == nil || !strings.Contains(err.Error(), "signal") { + t.Errorf("expected a signal shutdown report, got [%v]", err) + } + case <-time.After(10 * time.Second): + t.Fatal("the allowance did not bound the cleanup wait") + } +} + +// TestQuiesceBackstopDeadline_DominatesCompletionBound pins the wall-clock +// backstop to the block-derived completion bound plus the reviewed block +// margin: the drain must always be given at least the conservative wall-clock +// equivalent of the longest legitimately in-flight work, plus the margins. +func TestQuiesceBackstopDeadline_DominatesCompletionBound(t *testing.T) { + bound := uint64(1200) + expected := time.Duration(bound+quiesceReviewedMarginBlocks)* + quiesceUpperBlockIntervalSeconds*time.Second + + quiesceBackstopMargin + + got, err := quiesceBackstopDeadline(bound) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if got != expected { + t.Errorf( + "expected backstop [%s] for bound [%d], got [%s]", + expected, + bound, + got, + ) + } + if got <= quiesceBackstopMargin { + t.Error("the backstop must exceed the margin for a nonzero bound") + } +} + +// TestQuiesceBackstopDeadline_RejectsOverflow proves every step of the +// deadline calculation is overflow-checked: a bound that cannot be converted +// to a wall-clock deadline is a startup error, never a silently truncated +// grace period. +func TestQuiesceBackstopDeadline_RejectsOverflow(t *testing.T) { + overflowingBounds := map[string]uint64{ + "block margin addition overflows": math.MaxUint64 - 1, + "seconds multiplication overflows": math.MaxUint64/ + quiesceUpperBlockIntervalSeconds - 1, + "duration conversion overflows": math.MaxInt64/ + uint64(time.Second) + 1, + } + + for name, bound := range overflowingBounds { + t.Run(name, func(t *testing.T) { + if _, err := quiesceBackstopDeadline(bound); err == nil { + t.Errorf( + "expected an overflow error for bound [%d]", + bound, + ) + } + }) + } +} diff --git a/cmd/releasemanifest.go b/cmd/releasemanifest.go new file mode 100644 index 0000000000..3f40f3d0a2 --- /dev/null +++ b/cmd/releasemanifest.go @@ -0,0 +1,966 @@ +package cmd + +import ( + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + "io" + "math" + "os" + "regexp" + "strings" + "time" + + "github.com/spf13/cobra" + + commonEthereum "github.com/keep-network/keep-common/pkg/chain/ethereum" + "github.com/keep-network/keep-core/pkg/beacon" + "github.com/keep-network/keep-core/pkg/chain/ethereum" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/tbtc" +) + +// releaseManifestSchemaVersion is the only release manifest schema this +// binary understands. A manifest carrying any other version is rejected +// instead of being partially interpreted. Version 2 added the release identity +// section: a manifest that only binds the termination grace can no longer be +// read as a complete reviewed record. +const releaseManifestSchemaVersion = uint64(2) + +// compiledForcedCancellationAllowanceSeconds is the reviewed wall-clock +// allowance between the in-process quiesce backstop firing and the service +// manager escalating to SIGKILL. It covers the audited forced-cancellation +// path that runs after the backstop: canceling the permits that outlived the +// drain, persisting their audit records, closing the gate, and letting the +// process exit. It deliberately mirrors the RPC/processing margin used inside +// the backstop itself; both absorb the same order of local skew. The runtime +// cleanup wait consumes exactly this constant (forcedCancellationAllowance in +// start.go), and manifest validation rejects a manifest recording any other +// allowance, so no reviewed document can promise the service manager a +// cleanup window the running process does not observe. +const compiledForcedCancellationAllowanceSeconds = uint64(300) + +// processExitHeadroomSeconds is the reviewed headroom the external +// termination grace adds on top of the two in-process waits. The service +// manager counts its grace from signal delivery, but the backstop timer arms +// only after the lifecycle controller has been scheduled and has quiesced +// the gate, the cancellation-allowance timer arms only after the gate has +// closed, and the shutdown logging, run-context teardown, and process exit +// run after both. This headroom budgets that overhead — purely local work +// with no chain or network waits, sized far above what such work needs even +// under heavy load — so the external SIGKILL deadline ends strictly after +// the complete internal shutdown sequence instead of exactly at the sum of +// the two timed waits. +const processExitHeadroomSeconds = uint64(60) + +// sourceCommitPattern is the shape of a full git commit hash. An abbreviated +// hash is rejected because it names a prefix rather than a commit: prefixes +// collide, and the whole point of recording the source here is that a reviewer +// can fetch exactly what was built. +var sourceCommitPattern = regexp.MustCompile(`^[0-9a-f]{40}$`) + +// imageDigestPattern is the shape of an OCI content digest. The algorithm is +// pinned to sha256 rather than accepted from the document: the digest is what +// makes the reference immutable, so the manifest does not get to name the +// function it is immutable under. +var imageDigestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +// releaseImage names one runtime image the release publishes. +// +// The reference is what a deployment scaffold actually pulls, and it has to +// carry the digest. A tag is a name a registry can repoint at any time, so a +// manifest recording a digest beside a tagged reference documents one artifact +// while deployment runs whatever the tag resolves to at pull time — and the +// acceptance evidence, which is stated against the digest, would describe an +// image no node is necessarily running. +type releaseImage struct { + Platform string `json:"platform"` + Reference string `json:"reference"` + Digest string `json:"digest"` +} + +// releaseIdentity names the chain and block the reviewed cutover happens at. +// +// The termination grace binds this document to the compiled bounds of the +// binary validating it. That establishes the numbers are right for some build +// of this source; it does not establish which chain and block the cutover it +// describes is for. Every statement made about this release afterwards — +// smoke-gate evidence, the fleet inventory's per-instance digest attestation, a +// rollback decision taken against a block height — is stated against those +// identities, so the reviewed record carries them and validation holds them +// against what the binary compiles in. +// +// What the release was built into is deliberately absent. The commit finally +// built and the immutable image digests are outputs of a build over this +// document's own bytes, so a manifest recording them would have to contain a +// hash of the tree containing it: writing the value changes the commit the +// value names. Those two live in the detached release provenance instead — a +// document produced after the source commit and the images exist, bound back to +// this one by its hash. See releaseProvenance. +type releaseIdentity struct { + ChainID uint64 `json:"chain_id"` + CutoverBlock uint64 `json:"cutover_block"` + Notes string `json:"notes,omitempty"` +} + +// relocatedIdentityFields are the release-identity keys that used to live in +// the manifest and now live in the detached provenance. Strict decoding already +// rejects a manifest carrying them, but it rejects them as misspellings; naming +// them here is what turns that into the one instruction a reviewer holding a +// pre-relocation manifest needs. +var relocatedIdentityFields = []string{"source_commit", "images"} + +// releaseProvenance is the detached record of what the reviewed release was +// actually built into: the commit it was built from, and the immutable image +// digests the fleet runs. +// +// It exists separately from the manifest because those values cannot be inside +// the tree they describe. A reviewed manifest is part of the commit it would +// have to name, so filling the field changes the answer — which is why the +// manifest carries only what is reviewable ahead of the build, and this +// document, generated afterwards and never committed to the tree it describes, +// carries the rest. +// +// ManifestSHA256 is what keeps the two one record. Provenance naming a source +// commit and a set of images says nothing on its own about which reviewed +// bounds those artifacts were built under; hashing the manifest bytes into it +// means acceptance can refuse provenance produced against some other reviewed +// document, and refuse a manifest edited after provenance was taken over it. +type releaseProvenance struct { + SchemaVersion uint64 `json:"schema_version"` + GeneratedAt string `json:"generated_at"` + ManifestSHA256 string `json:"manifest_sha256"` + SourceCommit string `json:"source_commit"` + Images []releaseImage `json:"images"` + Notes string `json:"notes,omitempty"` +} + +// releaseProvenanceSchemaVersion is the only provenance schema this binary +// understands. +const releaseProvenanceSchemaVersion = uint64(1) + +// manifestDigestPattern is the shape of the manifest hash provenance binds +// itself to: the lowercase hexadecimal sha256 the release scripts compute over +// the manifest bytes, bare rather than algorithm-prefixed because it names a +// file rather than a registry object. +var manifestDigestPattern = regexp.MustCompile(`^[0-9a-f]{64}$`) + +// beaconCompletionInputs records the beacon chain configuration from which +// the beacon completion bound was derived, so a manifest reviewer can retrace +// the arithmetic without reading the adapter source. +type beaconCompletionInputs struct { + GroupSize uint64 `json:"group_size"` + ResultPublicationBlockStep uint64 `json:"result_publication_block_step"` + RelayEntryTimeoutBlocks uint64 `json:"relay_entry_timeout_blocks"` +} + +// terminationGrace is the manifest section binding the service manager's +// external termination grace to the in-process quiesce deadline. Every field +// is derived from this binary's compiled protocol bounds — the allowance is +// the compiled constant the runtime cleanup wait consumes, recorded here so +// the reviewed document names it — and the grace period is the checked sum +// of the backstop, the allowance, and the compiled process-exit headroom. +type terminationGrace struct { + TBTCCompletionBlocks uint64 `json:"tbtc_completion_blocks"` + BeaconCompletionBlocks uint64 `json:"beacon_completion_blocks"` + BeaconInputs beaconCompletionInputs `json:"beacon_inputs"` + MaximumLegacyCompletionBlocks uint64 `json:"maximum_legacy_completion_blocks"` + ReviewedMarginBlocks uint64 `json:"reviewed_margin_blocks"` + UpperBlockIntervalSeconds uint64 `json:"upper_block_interval_seconds"` + RPCProcessingAllowanceSeconds uint64 `json:"rpc_processing_allowance_seconds"` + InProcessBackstopSeconds uint64 `json:"in_process_backstop_seconds"` + ForcedCancellationAllowanceSeconds uint64 `json:"forced_cancellation_allowance_seconds"` + ProcessExitHeadroomSeconds uint64 `json:"process_exit_headroom_seconds"` + TerminationGracePeriodSeconds uint64 `json:"termination_grace_period_seconds"` + Notes string `json:"notes,omitempty"` +} + +// releaseManifest is the reviewed record from which deployment scaffolds take +// the authoritative service-manager termination grace. The client never reads +// it at runtime: its protocol bounds are compiled in, and this manifest exists +// so the external SIGKILL deadline is derived from those same bounds instead +// of being configured by hand. +type releaseManifest struct { + SchemaVersion uint64 `json:"schema_version"` + GeneratedAt string `json:"generated_at"` + ProtocolEpoch string `json:"protocol_epoch"` + ReleaseIdentity releaseIdentity `json:"release_identity"` + TerminationGrace terminationGrace `json:"termination_grace"` +} + +// deriveReleaseIdentity returns the release identity this binary states on its +// own: the mainnet chain the compiled cutover constant is for, and the cutover +// block compiled into it. +// +// Both are compiled values, which is the whole of what the manifest records. +// The source commit and the published image digests are outputs of the build +// rather than values inside it — a binary cannot honestly name the tree it was +// produced from or the registry content addresses it was packaged into — so +// they are recorded by the detached provenance and checked there. +func deriveReleaseIdentity() releaseIdentity { + return releaseIdentity{ + ChainID: uint64(commonEthereum.Mainnet.ChainID()), + CutoverBlock: participation.MainnetCutoverBlock, + } +} + +// deriveTerminationGrace computes the termination grace section from this +// binary's compiled protocol bounds: the tBTC and beacon completion bounds, +// the reviewed quiesce margin, the upper block interval, the RPC/processing +// allowance, and the in-process backstop produced by the same checked +// arithmetic the node uses at startup. Every production caller passes the +// compiled forced-cancellation allowance — the very value the runtime +// cleanup wait consumes — and validation separately requires a manifest to +// record exactly that value. A zero allowance is rejected because the +// external grace must end strictly after the in-process backstop for the +// audited forced-cancellation path to run before SIGKILL. +func deriveTerminationGrace( + forcedCancellationAllowanceSeconds uint64, +) (terminationGrace, error) { + if forcedCancellationAllowanceSeconds == 0 { + return terminationGrace{}, fmt.Errorf( + "forced-cancellation allowance must be positive: the external " + + "termination grace must end strictly after the in-process " + + "backstop", + ) + } + + beaconConfig := (ðereum.BeaconChain{}).GetConfig() + beaconBound, err := beacon.MaximumLegacyCompletionBlocks(beaconConfig) + if err != nil { + return terminationGrace{}, fmt.Errorf( + "cannot derive the beacon completion bound: [%v]", + err, + ) + } + + tbtcBound := tbtc.MaximumLegacyCompletionBlocks() + maximumBound := tbtcBound + if beaconBound > maximumBound { + maximumBound = beaconBound + } + + // The backstop is produced by the exact function the node runs at + // startup, so the manifest can never encode a deadline the client would + // not actually arm. + backstop, err := quiesceBackstopDeadline(maximumBound) + if err != nil { + return terminationGrace{}, fmt.Errorf( + "cannot derive the in-process backstop: [%v]", + err, + ) + } + backstopSeconds := uint64(backstop / time.Second) + + if forcedCancellationAllowanceSeconds > + math.MaxUint64-processExitHeadroomSeconds { + return terminationGrace{}, fmt.Errorf( + "termination grace overflows: allowance [%d]s plus exit "+ + "headroom [%d]s", + forcedCancellationAllowanceSeconds, + processExitHeadroomSeconds, + ) + } + graceBeyondBackstop := forcedCancellationAllowanceSeconds + + processExitHeadroomSeconds + if backstopSeconds > math.MaxUint64-graceBeyondBackstop { + return terminationGrace{}, fmt.Errorf( + "termination grace overflows: backstop [%d]s plus allowance "+ + "[%d]s plus exit headroom [%d]s", + backstopSeconds, + forcedCancellationAllowanceSeconds, + processExitHeadroomSeconds, + ) + } + + return terminationGrace{ + TBTCCompletionBlocks: tbtcBound, + BeaconCompletionBlocks: beaconBound, + BeaconInputs: beaconCompletionInputs{ + GroupSize: uint64(beaconConfig.GroupSize), + ResultPublicationBlockStep: beaconConfig.ResultPublicationBlockStep, + RelayEntryTimeoutBlocks: beaconConfig.RelayEntryTimeout, + }, + MaximumLegacyCompletionBlocks: maximumBound, + ReviewedMarginBlocks: quiesceReviewedMarginBlocks, + UpperBlockIntervalSeconds: uint64(quiesceUpperBlockIntervalSeconds), + RPCProcessingAllowanceSeconds: uint64(quiesceBackstopMargin / time.Second), + InProcessBackstopSeconds: backstopSeconds, + ForcedCancellationAllowanceSeconds: forcedCancellationAllowanceSeconds, + ProcessExitHeadroomSeconds: processExitHeadroomSeconds, + TerminationGracePeriodSeconds: backstopSeconds + graceBeyondBackstop, + }, nil +} + +// loadReleaseManifest reads and strictly decodes a release manifest: unknown +// fields, trailing content, and non-integer numbers are all rejected so a +// misspelled or hand-mangled input cannot pass as a reviewed manifest. +func loadReleaseManifest(path string) (releaseManifest, error) { + file, err := os.Open(path) // #nosec G304 -- operator-supplied manifest path + if err != nil { + return releaseManifest{}, fmt.Errorf( + "cannot open the release manifest: [%v]", + err, + ) + } + defer func() { + _ = file.Close() + }() + + decoder := json.NewDecoder(file) + decoder.DisallowUnknownFields() + + var manifest releaseManifest + if err := decoder.Decode(&manifest); err != nil { + // A manifest predating the provenance split fails here as a set of + // unknown fields, which reads as a typo rather than as the one thing it + // is. Say what actually happened, and where those values went. + if relocated := relocatedIdentityKeys(path); len(relocated) > 0 { + return releaseManifest{}, fmt.Errorf( + "release manifest [%s] records release_identity.%s, which the "+ + "manifest no longer carries: the commit built and the "+ + "image digests are outputs of the build over this "+ + "document's own bytes, so they moved to the detached "+ + "release provenance generated after the build. Remove "+ + "them here and record them there; "+ + "`release-manifest verify-provenance` checks that "+ + "document against this one. Decoding reported: [%v]", + path, + strings.Join(relocated, ", release_identity."), + err, + ) + } + return releaseManifest{}, fmt.Errorf( + "cannot decode the release manifest [%s]: [%v]", + path, + err, + ) + } + // The end-of-document check must ask for the next token, not use More: + // More only reports whether another value begins next, so a stray closing + // delimiter after the manifest object would pass it. Token consumes + // whatever actually follows — a value, a delimiter, or malformed bytes — + // and only clean EOF is an intact single-document manifest. + if _, err := decoder.Token(); !errors.Is(err, io.EOF) { + return releaseManifest{}, fmt.Errorf( + "release manifest [%s] carries trailing content after the "+ + "manifest object", + path, + ) + } + + return manifest, nil +} + +// relocatedIdentityKeys reports which of the keys that moved to the detached +// provenance a document still carries under release_identity. It is diagnostic +// only — the caller has already refused the document — so an unreadable or +// unparseable file simply reports nothing rather than masking the real error +// with one about reading it a second time. +func relocatedIdentityKeys(path string) []string { + raw, err := os.ReadFile(path) // #nosec G304 -- operator-supplied path + if err != nil { + return nil + } + + var document struct { + ReleaseIdentity map[string]json.RawMessage `json:"release_identity"` + } + if err := json.Unmarshal(raw, &document); err != nil { + return nil + } + + var found []string + for _, field := range relocatedIdentityFields { + if _, present := document.ReleaseIdentity[field]; present { + found = append(found, field) + } + } + return found +} + +// hashReleaseManifest returns the sha256 the release scripts bind records to: +// the digest of the exact manifest bytes, lowercase hexadecimal. Hashing the +// bytes rather than a re-encoding of the decoded document is deliberate — the +// binding has to be to the file a reviewer read and a record names, not to this +// binary's idea of how that file should be formatted. +func hashReleaseManifest(path string) (string, error) { + raw, err := os.ReadFile(path) // #nosec G304 -- operator-supplied path + if err != nil { + return "", fmt.Errorf( + "cannot read the release manifest [%s] to hash it: [%v]", + path, + err, + ) + } + return fmt.Sprintf("%x", sha256.Sum256(raw)), nil +} + +// loadReleaseProvenance reads and strictly decodes a detached provenance +// document, on the same terms as the manifest: unknown fields, trailing +// content, and non-integer numbers are all rejected. +func loadReleaseProvenance(path string) (releaseProvenance, error) { + file, err := os.Open(path) // #nosec G304 -- operator-supplied path + if err != nil { + return releaseProvenance{}, fmt.Errorf( + "cannot open the release provenance: [%v]", + err, + ) + } + defer func() { + _ = file.Close() + }() + + decoder := json.NewDecoder(file) + decoder.DisallowUnknownFields() + + var provenance releaseProvenance + if err := decoder.Decode(&provenance); err != nil { + return releaseProvenance{}, fmt.Errorf( + "cannot decode the release provenance [%s]: [%v]", + path, + err, + ) + } + if _, err := decoder.Token(); !errors.Is(err, io.EOF) { + return releaseProvenance{}, fmt.Errorf( + "release provenance [%s] carries trailing content after the "+ + "provenance object", + path, + ) + } + + return provenance, nil +} + +// validateReleaseManifest checks a manifest against this binary's compiled +// bounds and reports every violation, not only the first: a reviewer fixing a +// stale manifest sees the complete distance to the current code in one run. +// The manifest is authoritative only when this returns nil. +func validateReleaseManifest(manifest releaseManifest) error { + var violations []error + + if manifest.SchemaVersion != releaseManifestSchemaVersion { + violations = append(violations, fmt.Errorf( + "schema_version must be [%d], got [%d]", + releaseManifestSchemaVersion, + manifest.SchemaVersion, + )) + } + + if _, err := time.Parse(time.RFC3339, manifest.GeneratedAt); err != nil { + violations = append(violations, fmt.Errorf( + "generated_at must be an RFC 3339 timestamp: [%v]", + err, + )) + } + + expectedEpoch := participation.CompiledEpoch.String() + if manifest.ProtocolEpoch != expectedEpoch { + violations = append(violations, fmt.Errorf( + "protocol_epoch must be [%s], got [%s]", + expectedEpoch, + manifest.ProtocolEpoch, + )) + } + + violations = append( + violations, + releaseIdentityViolations(manifest.ReleaseIdentity)..., + ) + + // Derivation must consume the compiled allowance, never the manifest's + // own recorded value: the runtime cleanup wait is bound to the compiled + // constant, so a manifest carrying any other allowance — even one whose + // grace and scaffold values were recomputed coherently around it — would + // grant the service manager a SIGKILL deadline the running process does + // not honor. The recorded allowance is checked like every other number, + // against the compiled value. + derived, err := deriveTerminationGrace( + compiledForcedCancellationAllowanceSeconds, + ) + if err != nil { + violations = append(violations, err) + return errors.Join(violations...) + } + + recorded := manifest.TerminationGrace + // Notes are the only free-form field; every number must equal the value + // derived from the compiled bounds so neither a stale manifest nor a + // changed constant can pass unnoticed. + recordedNumbers, derivedNumbers := recorded, derived + recordedNumbers.Notes, derivedNumbers.Notes = "", "" + if recordedNumbers != derivedNumbers { + for _, mismatch := range []struct { + field string + recorded uint64 + derived uint64 + }{ + {"tbtc_completion_blocks", recorded.TBTCCompletionBlocks, derived.TBTCCompletionBlocks}, + {"beacon_completion_blocks", recorded.BeaconCompletionBlocks, derived.BeaconCompletionBlocks}, + {"beacon_inputs.group_size", recorded.BeaconInputs.GroupSize, derived.BeaconInputs.GroupSize}, + {"beacon_inputs.result_publication_block_step", recorded.BeaconInputs.ResultPublicationBlockStep, derived.BeaconInputs.ResultPublicationBlockStep}, + {"beacon_inputs.relay_entry_timeout_blocks", recorded.BeaconInputs.RelayEntryTimeoutBlocks, derived.BeaconInputs.RelayEntryTimeoutBlocks}, + {"maximum_legacy_completion_blocks", recorded.MaximumLegacyCompletionBlocks, derived.MaximumLegacyCompletionBlocks}, + {"reviewed_margin_blocks", recorded.ReviewedMarginBlocks, derived.ReviewedMarginBlocks}, + {"upper_block_interval_seconds", recorded.UpperBlockIntervalSeconds, derived.UpperBlockIntervalSeconds}, + {"rpc_processing_allowance_seconds", recorded.RPCProcessingAllowanceSeconds, derived.RPCProcessingAllowanceSeconds}, + {"in_process_backstop_seconds", recorded.InProcessBackstopSeconds, derived.InProcessBackstopSeconds}, + {"forced_cancellation_allowance_seconds", recorded.ForcedCancellationAllowanceSeconds, derived.ForcedCancellationAllowanceSeconds}, + {"process_exit_headroom_seconds", recorded.ProcessExitHeadroomSeconds, derived.ProcessExitHeadroomSeconds}, + {"termination_grace_period_seconds", recorded.TerminationGracePeriodSeconds, derived.TerminationGracePeriodSeconds}, + } { + if mismatch.recorded != mismatch.derived { + violations = append(violations, fmt.Errorf( + "%s must be [%d] as derived from the compiled bounds, "+ + "got [%d]", + mismatch.field, + mismatch.derived, + mismatch.recorded, + )) + } + } + } + + return errors.Join(violations...) +} + +// releaseIdentityViolations holds the recorded identity against what this +// binary compiles in. +// +// Both fields are compiled values, so both are checked outright rather than +// merely when present: unlike the build outputs that moved to the detached +// provenance, neither has a legitimate unrecorded state. +func releaseIdentityViolations(identity releaseIdentity) []error { + var violations []error + + derived := deriveReleaseIdentity() + + if identity.ChainID != derived.ChainID { + violations = append(violations, fmt.Errorf( + "release_identity.chain_id must be [%d], the mainnet chain the "+ + "compiled cutover block is for, got [%d]", + derived.ChainID, + identity.ChainID, + )) + } + + // Held against the compiled constant rather than merely required to be + // present: the client resolves the mainnet schedule from what it compiles + // in and reads no manifest at runtime, so a document naming a different + // block would send operators to a cutover height no node observes. + if identity.CutoverBlock != derived.CutoverBlock { + violations = append(violations, fmt.Errorf( + "release_identity.cutover_block must be [%d] as compiled into "+ + "this binary, got [%d]", + derived.CutoverBlock, + identity.CutoverBlock, + )) + } + + return violations +} + +// releaseImageViolations checks a recorded image list against the shape a +// reference has to have to name one immutable artifact. The field name is +// passed in because the same list is checked in the detached provenance, and a +// violation has to tell the reviewer which document to go edit. +func releaseImageViolations(field string, images []releaseImage) []error { + var violations []error + + platforms := make(map[string]struct{}, len(images)) + for index, image := range images { + if image.Platform == "" { + violations = append(violations, fmt.Errorf( + "%s[%d].platform must name the platform the image was built "+ + "for, as the architecture[/variant] the registry manifest "+ + "lists it under", + field, + index, + )) + } else if _, duplicate := platforms[image.Platform]; duplicate { + // One platform cannot have two images in one release: a scaffold + // choosing between them would be choosing between artifacts, which + // is the decision this document exists to have already made. + violations = append(violations, fmt.Errorf( + "%s[%d] repeats platform [%s]", + field, + index, + image.Platform, + )) + } else { + platforms[image.Platform] = struct{}{} + } + + if !imageDigestPattern.MatchString(image.Digest) { + violations = append(violations, fmt.Errorf( + "%s[%d].digest must be a sha256 content digest, got [%s]", + field, + index, + image.Digest, + )) + // The reference check below is a comparison against this digest, + // so a malformed digest has nothing to check the reference against. + continue + } + + // A tag is a name the registry may repoint at any time. A reference + // carrying one pulls whatever it resolves to at deployment time, which + // is not necessarily the artifact this document — or the acceptance + // evidence stated against its digest — describes. + if !strings.HasSuffix(image.Reference, "@"+image.Digest) { + violations = append(violations, fmt.Errorf( + "%s[%d].reference must be pinned to its digest as "+ + "[repository]@%s, got [%s]", + field, + index, + image.Digest, + image.Reference, + )) + } + } + + return violations +} + +// releaseReadyViolations reports what still stands between a manifest that is +// internally valid and one a release-acceptance decision may be taken against. +// +// The two are deliberately separate checks. Validity is a property of the +// document and the binary reading it, and it holds throughout development; +// readiness is a property of the release, and it cannot hold until the block +// the cutover happens at — a value that does not exist during development — has +// been reviewed and compiled in. Answering them with one verdict would either +// fail every development run or let a placeholder manifest pass as accepted. +// +// Readiness is the whole of what the manifest can answer. It says the reviewed +// document names a real cutover; it cannot say which artifact runs it, because +// the artifact does not exist when this document is reviewed. That half is the +// detached provenance's, and validateReleaseProvenance is where it is asked. +func releaseReadyViolations(manifest releaseManifest) []error { + var violations []error + + if manifest.ReleaseIdentity.CutoverBlock == 0 { + violations = append(violations, fmt.Errorf( + "release_identity.cutover_block is the zero placeholder: a "+ + "reviewed release commit must set the mainnet cutover block "+ + "in the client before a manifest can be release-ready", + )) + } + + return violations +} + +// validateReleaseProvenance checks a detached provenance document against the +// reviewed manifest whose bytes hash to manifestSHA256, reporting every +// violation rather than only the first. +// +// Unlike the manifest, provenance has no valid half-recorded state. It is +// generated after the build it describes, so a document missing the commit or +// the images is not an early draft — it is a claim about a release whose +// artifacts its author did not have. +func validateReleaseProvenance( + provenance releaseProvenance, + manifestSHA256 string, +) error { + var violations []error + + if provenance.SchemaVersion != releaseProvenanceSchemaVersion { + violations = append(violations, fmt.Errorf( + "schema_version must be [%d], got [%d]", + releaseProvenanceSchemaVersion, + provenance.SchemaVersion, + )) + } + + if _, err := time.Parse(time.RFC3339, provenance.GeneratedAt); err != nil { + violations = append(violations, fmt.Errorf( + "generated_at must be an RFC 3339 timestamp: [%v]", + err, + )) + } + + // Both halves of the binding, separately reported. A malformed hash is a + // document that never named a manifest; a well-formed one naming another + // manifest is provenance for a release reviewed under different bounds, and + // telling those two apart is what tells the operator which file to fix. + switch { + case !manifestDigestPattern.MatchString(provenance.ManifestSHA256): + violations = append(violations, fmt.Errorf( + "manifest_sha256 must be a 64-character lowercase hexadecimal "+ + "sha256 over the reviewed manifest bytes, got [%s]", + provenance.ManifestSHA256, + )) + case provenance.ManifestSHA256 != manifestSHA256: + violations = append(violations, fmt.Errorf( + "manifest_sha256 is [%s], but the reviewed manifest hashes to "+ + "[%s]; this provenance was taken over a different reviewed "+ + "document, or the manifest was edited after it was taken", + provenance.ManifestSHA256, + manifestSHA256, + )) + } + + if !sourceCommitPattern.MatchString(provenance.SourceCommit) { + violations = append(violations, fmt.Errorf( + "source_commit must be a full 40-character lowercase hexadecimal "+ + "commit naming the tree the release was built from, got [%s]", + provenance.SourceCommit, + )) + } + + if len(provenance.Images) == 0 { + violations = append(violations, fmt.Errorf( + "images is empty: provenance must record the immutable image "+ + "digests the fleet runs and the acceptance evidence is "+ + "collected against", + )) + } + violations = append( + violations, + releaseImageViolations("images", provenance.Images)..., + ) + + return errors.Join(violations...) +} + +// ReleaseManifestCommand contains the definition of the release-manifest +// command-line subcommand and its own subcommands. +var ReleaseManifestCommand = &cobra.Command{ + Use: "release-manifest", + Short: "Derive and validate the release manifest termination grace", + Long: "The release-manifest command derives the service-manager " + + "termination grace from this binary's compiled protocol bounds and " + + "validates a reviewed release manifest against them. The external " + + "grace must end strictly after the complete in-process shutdown " + + "sequence — the quiesce backstop, the forced-cancellation cleanup " + + "allowance, and the process teardown budgeted by the compiled exit " + + "headroom — so the audited forced-cancellation path and its writes " + + "always finish before the service manager escalates to SIGKILL.", +} + +var releaseManifestPath string + +// releaseManifestRequireReleaseReady turns validate into the release-acceptance +// check. It is off by default so the same command stays usable throughout +// development, where the identity a release records cannot yet exist. +var releaseManifestRequireReleaseReady bool + +// The derive subcommand deliberately takes no allowance flag: the runtime +// cleanup wait consumes the compiled allowance, so the only manifest worth +// deriving — and the only one validation accepts — is the one recording +// exactly that constant. +var releaseManifestDeriveCommand = &cobra.Command{ + Use: "derive", + Short: "Print the release manifest derived from the compiled bounds", + RunE: func(cmd *cobra.Command, args []string) error { + grace, err := deriveTerminationGrace( + compiledForcedCancellationAllowanceSeconds, + ) + if err != nil { + return err + } + + // The identity is entirely compiled values. What the build produced is + // not derived here and is not recorded here at all: it belongs to the + // detached provenance, generated once the build this document is + // reviewed ahead of has actually happened. + manifest := releaseManifest{ + SchemaVersion: releaseManifestSchemaVersion, + GeneratedAt: time.Now().UTC().Format(time.RFC3339), + ProtocolEpoch: participation.CompiledEpoch.String(), + ReleaseIdentity: deriveReleaseIdentity(), + TerminationGrace: grace, + } + + encoded, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + return fmt.Errorf("cannot encode the derived manifest: [%v]", err) + } + fmt.Fprintln(cmd.OutOrStdout(), string(encoded)) + return nil + }, +} + +var releaseManifestValidateCommand = &cobra.Command{ + Use: "validate", + Short: "Validate a release manifest against the compiled bounds", + RunE: func(cmd *cobra.Command, args []string) error { + manifest, err := loadReleaseManifest(releaseManifestPath) + if err != nil { + return err + } + + if err := validateReleaseManifest(manifest); err != nil { + return fmt.Errorf( + "release manifest [%s] rejected:\n%v", + releaseManifestPath, + err, + ) + } + + if releaseManifestRequireReleaseReady { + if violations := releaseReadyViolations( + manifest, + ); len(violations) > 0 { + return fmt.Errorf( + "release manifest [%s] is valid but not release-ready:\n%v", + releaseManifestPath, + errors.Join(violations...), + ) + } + } + + fmt.Fprintf( + cmd.OutOrStdout(), + "release manifest [%s] validated against the compiled bounds\n"+ + "protocol epoch: %s\n"+ + "chain id: %d\n"+ + "cutover block: %d\n"+ + "in-process backstop: %ds\n"+ + "service-manager termination grace: %ds\n", + releaseManifestPath, + manifest.ProtocolEpoch, + manifest.ReleaseIdentity.ChainID, + manifest.ReleaseIdentity.CutoverBlock, + manifest.TerminationGrace.InProcessBackstopSeconds, + manifest.TerminationGrace.TerminationGracePeriodSeconds, + ) + return nil + }, +} + +var releaseProvenancePath string + +// verify-provenance is the release-acceptance half the manifest cannot answer. +// validate --release-ready says the reviewed document names a real cutover; +// this says which artifact runs it, and that the artifact was built under those +// same reviewed bounds. +var releaseManifestVerifyProvenanceCommand = &cobra.Command{ + Use: "verify-provenance", + Short: "Verify detached release provenance against a reviewed manifest", + Long: "The verify-provenance command checks the detached provenance " + + "document — the commit the release was built from and the immutable " + + "image digests it publishes — against the reviewed release manifest " + + "it was taken over. Those values are outputs of a build over the " + + "manifest's own bytes, so they cannot live inside it: writing the " + + "commit into the tree changes the commit. Provenance is therefore " + + "generated after the build, never committed to the tree it describes, " + + "and bound back to the reviewed document by its hash.", + RunE: func(cmd *cobra.Command, args []string) error { + manifest, err := loadReleaseManifest(releaseManifestPath) + if err != nil { + return err + } + + // Provenance for a manifest this binary rejects is provenance for a + // release that would not pass its own validation, so the reviewed + // document is held to the compiled bounds first — and to readiness, + // because provenance exists only for a release being accepted. + if err := validateReleaseManifest(manifest); err != nil { + return fmt.Errorf( + "release manifest [%s] rejected:\n%v", + releaseManifestPath, + err, + ) + } + if violations := releaseReadyViolations(manifest); len(violations) > 0 { + return fmt.Errorf( + "release manifest [%s] is valid but not release-ready, so no "+ + "provenance can be verified against it:\n%v", + releaseManifestPath, + errors.Join(violations...), + ) + } + + manifestSHA256, err := hashReleaseManifest(releaseManifestPath) + if err != nil { + return err + } + + provenance, err := loadReleaseProvenance(releaseProvenancePath) + if err != nil { + return err + } + if err := validateReleaseProvenance( + provenance, + manifestSHA256, + ); err != nil { + return fmt.Errorf( + "release provenance [%s] rejected:\n%v", + releaseProvenancePath, + err, + ) + } + + fmt.Fprintf( + cmd.OutOrStdout(), + "release provenance [%s] verified against [%s]\n"+ + "reviewed manifest sha256: %s\n"+ + "source commit: %s\n"+ + "cutover block: %d\n", + releaseProvenancePath, + releaseManifestPath, + manifestSHA256, + provenance.SourceCommit, + manifest.ReleaseIdentity.CutoverBlock, + ) + for _, image := range provenance.Images { + fmt.Fprintf( + cmd.OutOrStdout(), + "image %-16s %s\n", + image.Platform, + image.Reference, + ) + } + return nil + }, +} + +func init() { + releaseManifestValidateCommand.Flags().StringVar( + &releaseManifestPath, + "manifest", + "", + "Path to the release manifest JSON document.", + ) + releaseManifestValidateCommand.Flags().BoolVar( + &releaseManifestRequireReleaseReady, + "release-ready", + false, + "Additionally require the reviewed cutover block to be the nonzero "+ + "block compiled into this binary. What the release was built "+ + "into is checked separately, by verify-provenance.", + ) + if err := releaseManifestValidateCommand.MarkFlagRequired("manifest"); err != nil { + logger.Fatalf("cannot mark the manifest flag required: [%v]", err) + } + + releaseManifestVerifyProvenanceCommand.Flags().StringVar( + &releaseManifestPath, + "manifest", + "", + "Path to the reviewed release manifest JSON document.", + ) + releaseManifestVerifyProvenanceCommand.Flags().StringVar( + &releaseProvenancePath, + "provenance", + "", + "Path to the detached release provenance JSON document, generated "+ + "after the build and never committed to the tree it describes.", + ) + for _, flag := range []string{"manifest", "provenance"} { + if err := releaseManifestVerifyProvenanceCommand.MarkFlagRequired( + flag, + ); err != nil { + logger.Fatalf("cannot mark the %s flag required: [%v]", flag, err) + } + } + + ReleaseManifestCommand.AddCommand( + releaseManifestDeriveCommand, + releaseManifestValidateCommand, + releaseManifestVerifyProvenanceCommand, + ) +} diff --git a/cmd/releasemanifest_test.go b/cmd/releasemanifest_test.go new file mode 100644 index 0000000000..3e5c7b0000 --- /dev/null +++ b/cmd/releasemanifest_test.go @@ -0,0 +1,1326 @@ +package cmd + +import ( + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + "math" + "os" + "path/filepath" + "reflect" + "regexp" + "strconv" + "strings" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +const releaseManifestRepositoryPath = "../scripts/release/pr4109/release-manifest.json" + +const releaseManifestDeployDirectory = "../scripts/release/pr4109/deploy" + +const releaseManifestSchemaPath = "../scripts/release/pr4109/release-manifest.schema.json" + +// releaseProvenanceSchemaPath is the published schema of the detached +// provenance document: what the release was built into, recorded outside the +// tree it describes. +const releaseProvenanceSchemaPath = "../scripts/release/pr4109/release-provenance.schema.json" + +const rehearsalEvidenceSchemaPath = "../scripts/release/pr4109/rehearsal-evidence.schema.json" + +// validReleaseManifestForTests builds a manifest that must pass validation: +// the derived termination grace under the reviewed default allowance, +// wrapped in the identity fields of the current artifact. +func validReleaseManifestForTests(t *testing.T) releaseManifest { + t.Helper() + + grace, err := deriveTerminationGrace(compiledForcedCancellationAllowanceSeconds) + if err != nil { + t.Fatalf("unexpected derivation error: [%v]", err) + } + + return releaseManifest{ + SchemaVersion: releaseManifestSchemaVersion, + GeneratedAt: "2026-07-27T23:11:28Z", + ProtocolEpoch: participation.CompiledEpoch.String(), + ReleaseIdentity: deriveReleaseIdentity(), + TerminationGrace: grace, + } +} + +// validReleaseProvenanceForTests is a complete detached provenance document +// over a manifest hashing to manifestSHA256: the commit built and one image per +// platform, pinned by digest. +func validReleaseProvenanceForTests(manifestSHA256 string) releaseProvenance { + return releaseProvenance{ + SchemaVersion: releaseProvenanceSchemaVersion, + GeneratedAt: "2026-07-27T23:11:28Z", + ManifestSHA256: manifestSHA256, + SourceCommit: "0123456789abcdef0123456789abcdef01234567", + Images: []releaseImage{ + { + Platform: "amd64", + Reference: "gcr.io/keep-test/keep-client@sha256:" + + strings.Repeat("a", 64), + Digest: "sha256:" + strings.Repeat("a", 64), + }, + { + Platform: "arm64", + Reference: "gcr.io/keep-test/keep-client@sha256:" + + strings.Repeat("b", 64), + Digest: "sha256:" + strings.Repeat("b", 64), + }, + }, + } +} + +// manifestSHAForTests is a well-formed manifest hash. Provenance is checked +// against the hash of real manifest bytes wherever that is what the case is +// about; this stands in where the case is about something else. +const manifestSHAForTests = "1f" + + "00000000000000000000000000000000000000000000000000000000000000" + +// TestReleaseManifestDeriveMatchesCompiledBounds is the drift assertion for +// the manifest derivation: every number the manifest records is pinned to a +// reviewed literal, and the backstop and grace are re-checked against the +// documented arithmetic identity. It fails whenever a compiled bound moves +// without the manifest chain being deliberately re-reviewed. +func TestReleaseManifestDeriveMatchesCompiledBounds(t *testing.T) { + grace, err := deriveTerminationGrace(compiledForcedCancellationAllowanceSeconds) + if err != nil { + t.Fatalf("unexpected derivation error: [%v]", err) + } + + for _, assertion := range []struct { + field string + got uint64 + expected uint64 + }{ + {"tbtc_completion_blocks", grace.TBTCCompletionBlocks, 1200}, + {"beacon_completion_blocks", grace.BeaconCompletionBlocks, 136}, + {"beacon_inputs.group_size", grace.BeaconInputs.GroupSize, 64}, + {"beacon_inputs.result_publication_block_step", grace.BeaconInputs.ResultPublicationBlockStep, 1}, + {"beacon_inputs.relay_entry_timeout_blocks", grace.BeaconInputs.RelayEntryTimeoutBlocks, 64}, + {"maximum_legacy_completion_blocks", grace.MaximumLegacyCompletionBlocks, 1200}, + {"reviewed_margin_blocks", grace.ReviewedMarginBlocks, 100}, + {"upper_block_interval_seconds", grace.UpperBlockIntervalSeconds, 15}, + {"rpc_processing_allowance_seconds", grace.RPCProcessingAllowanceSeconds, 300}, + {"in_process_backstop_seconds", grace.InProcessBackstopSeconds, 19800}, + {"forced_cancellation_allowance_seconds", grace.ForcedCancellationAllowanceSeconds, 300}, + {"process_exit_headroom_seconds", grace.ProcessExitHeadroomSeconds, 60}, + {"termination_grace_period_seconds", grace.TerminationGracePeriodSeconds, 20160}, + } { + if assertion.got != assertion.expected { + t.Errorf( + "%s changed: expected [%d], got [%d]", + assertion.field, + assertion.expected, + assertion.got, + ) + } + } + + backstopIdentity := (grace.MaximumLegacyCompletionBlocks+ + grace.ReviewedMarginBlocks)*grace.UpperBlockIntervalSeconds + + grace.RPCProcessingAllowanceSeconds + if grace.InProcessBackstopSeconds != backstopIdentity { + t.Errorf( + "backstop identity broken: (bound+margin)*interval+allowance is "+ + "[%d], recorded backstop is [%d]", + backstopIdentity, + grace.InProcessBackstopSeconds, + ) + } + + backstopDuration, err := quiesceBackstopDeadline( + grace.MaximumLegacyCompletionBlocks, + ) + if err != nil { + t.Fatalf("unexpected backstop error: [%v]", err) + } + if grace.InProcessBackstopSeconds != uint64(backstopDuration/time.Second) { + t.Errorf( + "manifest backstop [%d]s diverged from the runtime deadline [%s]", + grace.InProcessBackstopSeconds, + backstopDuration, + ) + } + + graceIdentity := grace.InProcessBackstopSeconds + + grace.ForcedCancellationAllowanceSeconds + + grace.ProcessExitHeadroomSeconds + if grace.TerminationGracePeriodSeconds != graceIdentity { + t.Errorf( + "grace identity broken: backstop+allowance+headroom is [%d], "+ + "recorded grace is [%d]", + graceIdentity, + grace.TerminationGracePeriodSeconds, + ) + } + // The strict inequality is the entire point of the exit headroom: the + // service manager counts its grace from signal delivery, so a grace that + // merely equals the sum of the two in-process waits leaves controller + // scheduling, the quiesce and close calls, logging, and teardown + // unbudgeted and lets SIGKILL land inside them. + internalWaits := grace.InProcessBackstopSeconds + + grace.ForcedCancellationAllowanceSeconds + if grace.TerminationGracePeriodSeconds <= internalWaits { + t.Errorf( + "grace [%d]s must end strictly after the complete internal wait "+ + "[%d]s (backstop plus cancellation allowance)", + grace.TerminationGracePeriodSeconds, + internalWaits, + ) + } + if grace.ProcessExitHeadroomSeconds == 0 { + t.Error("the process-exit headroom must be positive") + } +} + +func TestReleaseManifestDeriveRejectsZeroAllowance(t *testing.T) { + _, err := deriveTerminationGrace(0) + if err == nil { + t.Fatal("expected a zero allowance to be rejected") + } + if !strings.Contains(err.Error(), "must be positive") { + t.Errorf("unexpected rejection message: [%v]", err) + } +} + +func TestReleaseManifestDeriveRejectsOverflowingAllowance(t *testing.T) { + _, err := deriveTerminationGrace(math.MaxUint64) + if err == nil { + t.Fatal("expected an overflowing allowance to be rejected") + } + if !strings.Contains(err.Error(), "termination grace overflows") { + t.Errorf("unexpected rejection message: [%v]", err) + } +} + +// TestReleaseManifestFileMatchesCompiledDerivation pins the checked-in +// manifest to the compiled bounds: a stale manifest and a changed constant +// both fail here, forcing the regenerate-and-re-review step described in the +// manifest's own notes. +func TestReleaseManifestFileMatchesCompiledDerivation(t *testing.T) { + manifest, err := loadReleaseManifest(releaseManifestRepositoryPath) + if err != nil { + t.Fatalf("cannot load the repository manifest: [%v]", err) + } + + if err := validateReleaseManifest(manifest); err != nil { + t.Errorf( + "repository manifest rejected against the compiled bounds:\n%v", + err, + ) + } +} + +// TestReleaseManifestDeploymentScaffoldMatchesManifest closes the chain from +// the compiled bounds through the manifest into the deployment scaffold: +// every scaffold file must carry exactly the manifest's termination grace at +// its expected number of sites — once per service-manager fragment, once per +// R1 rehearsal node — and the systemd drop-in must keep SIGTERM as the stop +// signal because that is the signal the lifecycle controller quiesces on. +func TestReleaseManifestDeploymentScaffoldMatchesManifest(t *testing.T) { + manifest, err := loadReleaseManifest(releaseManifestRepositoryPath) + if err != nil { + t.Fatalf("cannot load the repository manifest: [%v]", err) + } + expected := manifest.TerminationGrace.TerminationGracePeriodSeconds + + scaffolds := []struct { + path string + pattern *regexp.Regexp + occurrences int + }{ + { + filepath.Join( + releaseManifestDeployDirectory, + "keep-client-termination-grace.k8s-patch.yaml", + ), + regexp.MustCompile(`(?m)^\s*terminationGracePeriodSeconds:\s*(\d+)\s*$`), + 1, + }, + { + filepath.Join( + releaseManifestDeployDirectory, + "keep-client-termination-grace.systemd-dropin.conf", + ), + regexp.MustCompile(`(?m)^TimeoutStopSec=(\d+)$`), + 1, + }, + { + "../scripts/release/pr4109/compose.rehearsal.yaml", + regexp.MustCompile(`(?m)^\s*stop_grace_period:\s*(\d+)s\s*$`), + 2, + }, + } + + for _, scaffold := range scaffolds { + content, err := os.ReadFile(scaffold.path) + if err != nil { + t.Fatalf("cannot read the deployment scaffold: [%v]", err) + } + + matches := scaffold.pattern.FindAllStringSubmatch(string(content), -1) + if len(matches) != scaffold.occurrences { + t.Errorf( + "[%s] must configure the grace at exactly [%d] site(s), "+ + "found [%d]", + scaffold.path, + scaffold.occurrences, + len(matches), + ) + continue + } + + for _, match := range matches { + configured, err := strconv.ParseUint(match[1], 10, 64) + if err != nil { + t.Errorf( + "[%s] carries a non-integer grace [%s]", + scaffold.path, + match[1], + ) + continue + } + if configured != expected { + t.Errorf( + "[%s] configures a grace of [%d]s, the validated "+ + "manifest requires [%d]s", + scaffold.path, + configured, + expected, + ) + } + } + } + + systemdPath := filepath.Join( + releaseManifestDeployDirectory, + "keep-client-termination-grace.systemd-dropin.conf", + ) + systemdContent, err := os.ReadFile(systemdPath) + if err != nil { + t.Fatalf("cannot read the systemd drop-in: [%v]", err) + } + killSignal := regexp.MustCompile(`(?m)^KillSignal=(\S+)$`). + FindAllStringSubmatch(string(systemdContent), -1) + if len(killSignal) != 1 || killSignal[0][1] != "SIGTERM" { + t.Errorf( + "the systemd drop-in must keep KillSignal=SIGTERM exactly once, "+ + "got [%v]", + killSignal, + ) + } +} + +// TestReleaseManifestSchemaMatchesTheDecodedDocument is the drift check for +// the published schema. The schema is a hand-written mirror of the types the +// client decodes, and nothing but this test holds the two together. +// +// Drift here is quiet and one-directional. The strict Go loader rejects a field +// the schema forgot, so a manifest is never accepted on a mismatch; what +// happens instead is that external tooling validating against the schema passes +// a document the release binary refuses, or — worse — reports a document +// complete while a required section it never learned about is missing from it. +func TestReleaseManifestSchemaMatchesTheDecodedDocument(t *testing.T) { + assertSchemaMatchesTypes( + t, + releaseManifestSchemaPath, + releaseManifestSchemaVersion, + []schemaSection{ + {"the manifest", nil, releaseManifest{}}, + { + "release_identity", + []string{"properties", "release_identity"}, + releaseIdentity{}, + }, + { + "termination_grace", + []string{"properties", "termination_grace"}, + terminationGrace{}, + }, + { + "termination_grace.beacon_inputs", + []string{ + "properties", "termination_grace", "properties", + "beacon_inputs", + }, + beaconCompletionInputs{}, + }, + }, + ) +} + +// TestReleaseProvenanceSchemaMatchesTheDecodedDocument holds the detached +// provenance schema to the same standard as the manifest's. The provenance +// document is where the build outputs went when they could no longer live in +// the tree they describe, so a schema drifting from the loader here is the same +// failure as before, only relocated: a document every tool but the binary +// accepts. +func TestReleaseProvenanceSchemaMatchesTheDecodedDocument(t *testing.T) { + assertSchemaMatchesTypes( + t, + releaseProvenanceSchemaPath, + releaseProvenanceSchemaVersion, + []schemaSection{ + {"the provenance", nil, releaseProvenance{}}, + { + "images entries", + []string{"properties", "images", "items"}, + releaseImage{}, + }, + }, + ) +} + +// schemaSection names one object in a schema and the Go type that must decode +// it, so one walk can check any number of documents. +type schemaSection struct { + name string + path []string + recorded any +} + +// assertSchemaMatchesTypes checks that a published JSON Schema describes +// exactly what the loader decodes: the same members, required exactly where the +// encoder cannot omit them, closed to anything else, and pinned to the schema +// version the binary accepts. +func assertSchemaMatchesTypes( + t *testing.T, + schemaPath string, + schemaVersion uint64, + sections []schemaSection, +) { + t.Helper() + + content, err := os.ReadFile(schemaPath) + if err != nil { + t.Fatalf("cannot read the schema [%s]: [%v]", schemaPath, err) + } + + // Decoded as an object graph rather than into a shape of its own: the check + // walks the same nesting the document types have, and a schema that renamed + // or moved a section must fail rather than decode into zero values. + var schema map[string]any + if err := json.Unmarshal(content, &schema); err != nil { + t.Fatalf("cannot decode the schema [%s]: [%v]", schemaPath, err) + } + + if version := schemaObject(t, schema, "properties", "schema_version"); version != nil { + if got, want := version["const"], float64(schemaVersion); got != want { + t.Errorf( + "the schema pins schema_version to [%v], the binary accepts "+ + "only [%v]", + got, + want, + ) + } + } + + for _, section := range sections { + object := schemaObject(t, schema, section.path...) + if object == nil { + t.Errorf("the schema defines no [%s] section", section.name) + continue + } + + // A schema that accepts unknown properties describes a laxer document + // than the loader does, which is the direction that lets a hand-edited + // manifest read as valid to everything except the binary. + if closed, _ := object["additionalProperties"].(bool); closed { + t.Errorf("[%s] must not accept additional properties", section.name) + } + + declared := schemaObject(t, object, "properties") + requiredList, _ := object["required"].([]any) + required := make(map[string]struct{}, len(requiredList)) + for _, entry := range requiredList { + name, _ := entry.(string) + required[name] = struct{}{} + } + + for field, optional := range jsonFields(section.recorded) { + if _, present := declared[field]; !present { + t.Errorf("[%s] must define [%s]", section.name, field) + } + // Optional exactly where the encoder may omit it. A required field + // the schema left optional is a section a document may drop; an + // optional one the schema required is a document the binary + // produces and the schema rejects. + if _, present := required[field]; present == optional { + if optional { + t.Errorf( + "[%s] must not require [%s]: the client omits it when "+ + "it is empty", + section.name, + field, + ) + } else { + t.Errorf("[%s] must require [%s]", section.name, field) + } + } + } + + for field := range declared { + if _, recorded := jsonFields(section.recorded)[field]; !recorded { + t.Errorf( + "[%s] defines [%s], which the client does not decode", + section.name, + field, + ) + } + } + } +} + +// schemaObject walks a decoded schema to the object at the given path, +// reporting nil when the path does not lead to one. +func schemaObject(t *testing.T, schema map[string]any, path ...string) map[string]any { + t.Helper() + + object := schema + for _, step := range path { + next, ok := object[step].(map[string]any) + if !ok { + return nil + } + object = next + } + + return object +} + +// jsonFields lists the JSON member names a manifest type encodes, saying of +// each whether the encoder may leave it out. +func jsonFields(recorded any) map[string]bool { + fields := make(map[string]bool) + + structType := reflect.TypeOf(recorded) + for index := range structType.NumField() { + tag := structType.Field(index).Tag.Get("json") + name, options, _ := strings.Cut(tag, ",") + if name == "" || name == "-" { + continue + } + fields[name] = strings.Contains(options, "omitempty") + } + + return fields +} + +// TestRehearsalEvidenceSchemaRequiresManifestBinding pins the evidence +// schema's release-manifest binding: every accepted rehearsal record must +// name the hash of the reviewed manifest and the grace it ran under. +// Dropping the requirement from the schema would silently sever the link +// between the termination-grace record and the source SHA, image digests, +// and chain identity the record carries. +func TestRehearsalEvidenceSchemaRequiresManifestBinding(t *testing.T) { + content, err := os.ReadFile(rehearsalEvidenceSchemaPath) + if err != nil { + t.Fatalf("cannot read the evidence schema: [%v]", err) + } + + var schema struct { + Required []string `json:"required"` + Properties struct { + ReleaseManifest struct { + Required []string `json:"required"` + Properties map[string]json.RawMessage `json:"properties"` + } `json:"release_manifest"` + } `json:"properties"` + } + if err := json.Unmarshal(content, &schema); err != nil { + t.Fatalf("cannot decode the evidence schema: [%v]", err) + } + + contains := func(list []string, want string) bool { + for _, entry := range list { + if entry == want { + return true + } + } + return false + } + + if !contains(schema.Required, "release_manifest") { + t.Error("the evidence schema must require the release_manifest binding") + } + for _, field := range []string{ + "sha256", + "termination_grace_period_seconds", + } { + if !contains(schema.Properties.ReleaseManifest.Required, field) { + t.Errorf("the release_manifest binding must require [%s]", field) + } + if _, present := schema.Properties.ReleaseManifest.Properties[field]; !present { + t.Errorf("the release_manifest binding must define [%s]", field) + } + } +} + +func TestReleaseManifestValidateAcceptsDerived(t *testing.T) { + if err := validateReleaseManifest(validReleaseManifestForTests(t)); err != nil { + t.Errorf("derived manifest rejected: [%v]", err) + } +} + +// TestReleaseManifestValidateFailsClosed mutates every field of a valid +// manifest in turn and requires validation to name the exact violation, so +// no single stale number can survive a validate run. +func TestReleaseManifestValidateFailsClosed(t *testing.T) { + tests := map[string]struct { + mutate func(*releaseManifest) + expectedMessage string + }{ + "wrong schema version": { + func(m *releaseManifest) { m.SchemaVersion = 1 }, + "schema_version must be [2]", + }, + "unparseable generation timestamp": { + func(m *releaseManifest) { m.GeneratedAt = "yesterday" }, + "generated_at must be an RFC 3339 timestamp", + }, + "wrong protocol epoch": { + func(m *releaseManifest) { m.ProtocolEpoch = "legacy" }, + "protocol_epoch must be [security_v2_cutover]", + }, + "stale tbtc bound": { + func(m *releaseManifest) { m.TerminationGrace.TBTCCompletionBlocks++ }, + "tbtc_completion_blocks must be [1200]", + }, + "stale beacon bound": { + func(m *releaseManifest) { m.TerminationGrace.BeaconCompletionBlocks-- }, + "beacon_completion_blocks must be [136]", + }, + "stale beacon group size": { + func(m *releaseManifest) { m.TerminationGrace.BeaconInputs.GroupSize++ }, + "beacon_inputs.group_size must be [64]", + }, + "stale beacon publication step": { + func(m *releaseManifest) { + m.TerminationGrace.BeaconInputs.ResultPublicationBlockStep++ + }, + "beacon_inputs.result_publication_block_step must be [1]", + }, + "stale beacon relay entry timeout": { + func(m *releaseManifest) { + m.TerminationGrace.BeaconInputs.RelayEntryTimeoutBlocks++ + }, + "beacon_inputs.relay_entry_timeout_blocks must be [64]", + }, + "stale combined bound": { + func(m *releaseManifest) { + m.TerminationGrace.MaximumLegacyCompletionBlocks++ + }, + "maximum_legacy_completion_blocks must be [1200]", + }, + "stale reviewed margin": { + func(m *releaseManifest) { m.TerminationGrace.ReviewedMarginBlocks++ }, + "reviewed_margin_blocks must be [100]", + }, + "stale block interval": { + func(m *releaseManifest) { + m.TerminationGrace.UpperBlockIntervalSeconds++ + }, + "upper_block_interval_seconds must be [15]", + }, + "stale rpc allowance": { + func(m *releaseManifest) { + m.TerminationGrace.RPCProcessingAllowanceSeconds++ + }, + "rpc_processing_allowance_seconds must be [300]", + }, + "stale backstop": { + func(m *releaseManifest) { m.TerminationGrace.InProcessBackstopSeconds++ }, + "in_process_backstop_seconds must be [19800]", + }, + "zero forced-cancellation allowance": { + func(m *releaseManifest) { + m.TerminationGrace.ForcedCancellationAllowanceSeconds = 0 + }, + "forced_cancellation_allowance_seconds must be [300]", + }, + "stale forced-cancellation allowance": { + func(m *releaseManifest) { + m.TerminationGrace.ForcedCancellationAllowanceSeconds++ + }, + "forced_cancellation_allowance_seconds must be [300]", + }, + "stale exit headroom": { + func(m *releaseManifest) { + m.TerminationGrace.ProcessExitHeadroomSeconds++ + }, + "process_exit_headroom_seconds must be [60]", + }, + "exit headroom dropped": { + func(m *releaseManifest) { + m.TerminationGrace.ProcessExitHeadroomSeconds = 0 + }, + "process_exit_headroom_seconds must be [60]", + }, + "grace not equal to the full internal sequence": { + func(m *releaseManifest) { + m.TerminationGrace.TerminationGracePeriodSeconds++ + }, + "termination_grace_period_seconds must be [20160]", + }, + "grace truncated to the backstop": { + func(m *releaseManifest) { + m.TerminationGrace.TerminationGracePeriodSeconds = + m.TerminationGrace.InProcessBackstopSeconds + }, + "termination_grace_period_seconds must be [20160]", + }, + "grace truncated to the two timed waits": { + func(m *releaseManifest) { + m.TerminationGrace.TerminationGracePeriodSeconds = + m.TerminationGrace.InProcessBackstopSeconds + + m.TerminationGrace.ForcedCancellationAllowanceSeconds + }, + "termination_grace_period_seconds must be [20160]", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + manifest := validReleaseManifestForTests(t) + test.mutate(&manifest) + + err := validateReleaseManifest(manifest) + if err == nil { + t.Fatal("expected the mutated manifest to be rejected") + } + if !strings.Contains(err.Error(), test.expectedMessage) { + t.Errorf( + "rejection must name the violation [%s], got:\n%v", + test.expectedMessage, + err, + ) + } + }) + } +} + +// TestReleaseManifestValidateRejectsCoherentlyChangedAllowance proves the +// allowance identity cannot be bypassed by a self-consistent edit: a manifest +// whose allowance, grace, and therefore scaffold-facing sum were all +// recomputed coherently around a different allowance still names a cleanup +// window the runtime does not wait, so validation must reject it against the +// compiled constant — never re-derive around the manifest's own value. +func TestReleaseManifestValidateRejectsCoherentlyChangedAllowance(t *testing.T) { + tests := map[string]struct { + allowanceSeconds uint64 + expectedRejection []string + }{ + "allowance lowered below the runtime cleanup wait": { + 120, + []string{ + "forced_cancellation_allowance_seconds must be [300] as " + + "derived from the compiled bounds, got [120]", + "termination_grace_period_seconds must be [20160] as " + + "derived from the compiled bounds, got [19980]", + }, + }, + "allowance raised above the runtime cleanup wait": { + 600, + []string{ + "forced_cancellation_allowance_seconds must be [300] as " + + "derived from the compiled bounds, got [600]", + "termination_grace_period_seconds must be [20160] as " + + "derived from the compiled bounds, got [20460]", + }, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + manifest := validReleaseManifestForTests(t) + grace := &manifest.TerminationGrace + grace.ForcedCancellationAllowanceSeconds = test.allowanceSeconds + // Recompute the grace exactly as a coherent hand-edit would, so + // the only remaining inconsistency is with the compiled constant + // the runtime waits. + grace.TerminationGracePeriodSeconds = grace.InProcessBackstopSeconds + + test.allowanceSeconds + grace.ProcessExitHeadroomSeconds + + err := validateReleaseManifest(manifest) + if err == nil { + t.Fatal("expected the coherently changed allowance to be rejected") + } + for _, expectedMessage := range test.expectedRejection { + if !strings.Contains(err.Error(), expectedMessage) { + t.Errorf( + "rejection must name the violation [%s], got:\n%v", + expectedMessage, + err, + ) + } + } + }) + } +} + +func TestReleaseManifestValidateReportsEveryViolation(t *testing.T) { + manifest := validReleaseManifestForTests(t) + manifest.SchemaVersion = 7 + manifest.ReleaseIdentity.ChainID++ + manifest.TerminationGrace.ReviewedMarginBlocks++ + manifest.TerminationGrace.TerminationGracePeriodSeconds++ + + err := validateReleaseManifest(manifest) + if err == nil { + t.Fatal("expected the mutated manifest to be rejected") + } + for _, expectedMessage := range []string{ + "schema_version must be [2]", + "release_identity.chain_id must be [1]", + "reviewed_margin_blocks must be [100]", + "termination_grace_period_seconds must be [20160]", + } { + if !strings.Contains(err.Error(), expectedMessage) { + t.Errorf( + "rejection must accumulate the violation [%s], got:\n%v", + expectedMessage, + err, + ) + } + } +} + +// TestReleaseManifestValidateBindsIdentityToTheCompiledClient proves the +// recorded chain and cutover block cannot drift from what the binary compiles +// in, and that a recorded source commit or image reference has to name one +// immutable thing. +// +// The client reads no manifest at runtime: it resolves the mainnet schedule +// from its compiled constant. A document naming a different block would send +// operators to a cutover height no node observes, and a reference carrying a +// tag would let deployment pull whatever the registry resolves it to rather +// than the artifact the acceptance evidence describes. +func TestReleaseManifestValidateBindsIdentityToTheCompiledClient(t *testing.T) { + tests := map[string]struct { + mutate func(*releaseIdentity) + expectedMessage string + }{ + "chain other than the one the compiled cutover block is for": { + func(i *releaseIdentity) { i.ChainID = 11155111 }, + "release_identity.chain_id must be [1]", + }, + "cutover block the client does not compile in": { + func(i *releaseIdentity) { + i.CutoverBlock = participation.MainnetCutoverBlock + 1 + }, + "release_identity.cutover_block must be [" + strconv.FormatUint( + participation.MainnetCutoverBlock, + 10, + ) + "]", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + manifest := validReleaseManifestForTests(t) + test.mutate(&manifest.ReleaseIdentity) + + err := validateReleaseManifest(manifest) + if err == nil { + t.Fatal("expected the mutated identity to be rejected") + } + if !strings.Contains(err.Error(), test.expectedMessage) { + t.Errorf( + "rejection must name the violation [%s], got:\n%v", + test.expectedMessage, + err, + ) + } + }) + } +} + +// TestReleaseManifestRejectsRelocatedBuildOutputs is the standing refusal of +// the shape this document used to have. +// +// A manifest naming the commit it is part of cannot be produced: writing the +// hash changes the tree, so the value is stale the moment it is written. That +// is why the build outputs moved to the detached provenance, and a manifest +// still carrying them has to be refused with the one instruction that resolves +// it rather than with an unknown-field complaint that reads like a typo. +func TestReleaseManifestRejectsRelocatedBuildOutputs(t *testing.T) { + valid, err := json.Marshal(validReleaseManifestForTests(t)) + if err != nil { + t.Fatalf("cannot encode the valid manifest: [%v]", err) + } + + var document map[string]any + if err := json.Unmarshal(valid, &document); err != nil { + t.Fatalf("cannot decode the valid manifest: [%v]", err) + } + + for _, field := range relocatedIdentityFields { + t.Run(field, func(t *testing.T) { + identity, ok := document["release_identity"].(map[string]any) + if !ok { + t.Fatal("the encoded manifest carries no release_identity") + } + // A fresh copy per case: the loop must not leave the previous + // field behind and test two relocations as one. + mutated := make(map[string]any, len(identity)+1) + for key, value := range identity { + mutated[key] = value + } + mutated[field] = "0123456789abcdef0123456789abcdef01234567" + + withField := make(map[string]any, len(document)) + for key, value := range document { + withField[key] = value + } + withField["release_identity"] = mutated + + encoded, err := json.Marshal(withField) + if err != nil { + t.Fatalf("cannot encode the mutated manifest: [%v]", err) + } + path := filepath.Join(t.TempDir(), "release-manifest.json") + if err := os.WriteFile(path, encoded, 0o600); err != nil { + t.Fatalf("cannot write the mutated manifest: [%v]", err) + } + + _, err = loadReleaseManifest(path) + if err == nil { + t.Fatalf( + "a manifest recording release_identity.%s was accepted; "+ + "that value cannot be inside the tree it names", + field, + ) + } + for _, expected := range []string{ + "release_identity." + field, + "detached release provenance", + } { + if !strings.Contains(err.Error(), expected) { + t.Errorf( + "the refusal must say [%s] so the reviewer knows "+ + "where the value went, got:\n%v", + expected, + err, + ) + } + } + }) + } +} + +// TestReleaseProvenanceValidateAcceptsACompleteDocument proves a complete +// provenance passes, so the refusals below reject what is wrong with a document +// rather than the act of recording one. +func TestReleaseProvenanceValidateAcceptsACompleteDocument(t *testing.T) { + provenance := validReleaseProvenanceForTests(manifestSHAForTests) + + if err := validateReleaseProvenance( + provenance, + manifestSHAForTests, + ); err != nil { + t.Errorf("complete provenance rejected: [%v]", err) + } +} + +// TestReleaseProvenanceValidateFailsClosed drives every way a provenance +// document can fail to name one artifact built under one set of reviewed +// bounds. These are the checks that used to sit on the manifest identity, now +// asked where the values actually live — plus the binding that only exists +// because they live apart: the hash tying this document to the manifest it was +// taken over. +func TestReleaseProvenanceValidateFailsClosed(t *testing.T) { + tests := map[string]struct { + mutate func(*releaseProvenance) + expectedMessage string + }{ + "schema version the binary does not understand": { + func(p *releaseProvenance) { + p.SchemaVersion = releaseProvenanceSchemaVersion + 1 + }, + "schema_version must be", + }, + "generation timestamp that is not RFC 3339": { + func(p *releaseProvenance) { p.GeneratedAt = "yesterday" }, + "generated_at must be an RFC 3339 timestamp", + }, + "unrecorded manifest hash": { + func(p *releaseProvenance) { p.ManifestSHA256 = "" }, + "manifest_sha256 must be a 64-character lowercase hexadecimal", + }, + "manifest hash that is not hexadecimal": { + func(p *releaseProvenance) { + p.ManifestSHA256 = strings.Repeat("z", 64) + }, + "manifest_sha256 must be a 64-character lowercase hexadecimal", + }, + "well-formed hash of some other reviewed manifest": { + func(p *releaseProvenance) { + p.ManifestSHA256 = strings.Repeat("c", 64) + }, + "this provenance was taken over a different reviewed document", + }, + "unrecorded source commit": { + func(p *releaseProvenance) { p.SourceCommit = "" }, + "source_commit must be a full 40-character", + }, + "abbreviated source commit": { + func(p *releaseProvenance) { p.SourceCommit = "0123456" }, + "source_commit must be a full 40-character", + }, + "source commit that is not hexadecimal": { + func(p *releaseProvenance) { + p.SourceCommit = strings.Repeat("z", 40) + }, + "source_commit must be a full 40-character", + }, + "no images at all": { + func(p *releaseProvenance) { p.Images = nil }, + "images is empty", + }, + "image digest under another algorithm": { + func(p *releaseProvenance) { + p.Images[0].Digest = "sha512:" + strings.Repeat("a", 64) + p.Images[0].Reference = "gcr.io/keep-test/keep-client@" + + p.Images[0].Digest + }, + "images[0].digest must be a sha256 content digest", + }, + "image reference carrying a tag instead of its digest": { + func(p *releaseProvenance) { + p.Images[0].Reference = "gcr.io/keep-test/keep-client:v2.1.0" + }, + "images[0].reference must be pinned to its digest", + }, + "image reference pinned to a different digest": { + func(p *releaseProvenance) { + p.Images[0].Reference = "gcr.io/keep-test/keep-client@sha256:" + + strings.Repeat("c", 64) + }, + "images[0].reference must be pinned to its digest", + }, + "image without a platform": { + func(p *releaseProvenance) { p.Images[0].Platform = "" }, + "images[0].platform must name the platform", + }, + "two images for one platform": { + func(p *releaseProvenance) { + p.Images[1].Platform = p.Images[0].Platform + }, + "images[1] repeats platform [amd64]", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + provenance := validReleaseProvenanceForTests(manifestSHAForTests) + test.mutate(&provenance) + + err := validateReleaseProvenance(provenance, manifestSHAForTests) + if err == nil { + t.Fatal("expected the mutated provenance to be rejected") + } + if !strings.Contains(err.Error(), test.expectedMessage) { + t.Errorf( + "rejection must name the violation [%s], got:\n%v", + test.expectedMessage, + err, + ) + } + }) + } +} + +// TestReleaseProvenanceBindsToTheReviewedManifestBytes closes the loop the two +// documents exist in: the hash provenance records has to be the hash of the +// exact reviewed bytes, so editing the manifest after provenance was taken over +// it invalidates the pair rather than silently re-pointing it. +func TestReleaseProvenanceBindsToTheReviewedManifestBytes(t *testing.T) { + manifestSHA, err := hashReleaseManifest(releaseManifestRepositoryPath) + if err != nil { + t.Fatalf("cannot hash the repository manifest: [%v]", err) + } + + // The same digest the release scripts bind evidence records to, computed + // the same way: over the bytes on disk, not over a re-encoding of them. + raw, err := os.ReadFile(releaseManifestRepositoryPath) + if err != nil { + t.Fatalf("cannot read the repository manifest: [%v]", err) + } + if expected := fmt.Sprintf("%x", sha256.Sum256(raw)); manifestSHA != expected { + t.Fatalf( + "the manifest hash must be taken over the file bytes: got [%s], "+ + "the bytes hash to [%s]", + manifestSHA, + expected, + ) + } + + provenance := validReleaseProvenanceForTests(manifestSHA) + if err := validateReleaseProvenance(provenance, manifestSHA); err != nil { + t.Errorf("provenance over the reviewed bytes rejected: [%v]", err) + } + + // One byte of the reviewed document changing is enough: the pair is no + // longer one record, and nothing downstream may treat it as one. + edited := filepath.Join(t.TempDir(), "release-manifest.json") + if err := os.WriteFile(edited, append(raw, '\n'), 0o600); err != nil { + t.Fatalf("cannot write the edited manifest: [%v]", err) + } + editedSHA, err := hashReleaseManifest(edited) + if err != nil { + t.Fatalf("cannot hash the edited manifest: [%v]", err) + } + if editedSHA == manifestSHA { + t.Fatal("editing the manifest must change its hash") + } + if err := validateReleaseProvenance(provenance, editedSHA); err == nil { + t.Error( + "provenance taken over the reviewed manifest was accepted " + + "against an edited one", + ) + } +} + +// TestReleaseManifestRepositoryFileIsNotReleaseReady is the standing statement +// of what release acceptance is still waiting for, checked rather than +// asserted in prose. +// +// The checked-in manifest is a valid document — every number in it matches the +// compiled bounds — and it is deliberately not a release-ready one. It is +// reviewed before the build it will name exists, and the cutover block it +// records is the compiled zero placeholder. This test fails the moment that +// stops being true in either direction: a manifest that stops validating, or +// one that starts passing release-readiness while its identity is still empty. +func TestReleaseManifestRepositoryFileIsNotReleaseReady(t *testing.T) { + manifest, err := loadReleaseManifest(releaseManifestRepositoryPath) + if err != nil { + t.Fatalf("cannot load the repository manifest: [%v]", err) + } + + if err := validateReleaseManifest(manifest); err != nil { + t.Fatalf("the repository manifest must be a valid document: [%v]", err) + } + + violations := releaseReadyViolations(manifest) + + // The one blocker readiness can still report lives in the client rather + // than in the document: no edit to the manifest can clear it, only a + // reviewed release commit setting C. Once that commit lands the manifest is + // ready, and this test says so rather than failing — what it must never + // permit is readiness while the placeholder stands. + if participation.MainnetCutoverBlock == 0 { + if len(violations) == 0 { + t.Fatal( + "the repository manifest passed release-readiness over the " + + "compiled zero cutover block: the readiness check stopped " + + "checking the one value it owns", + ) + } + joined := errors.Join(violations...).Error() + if !strings.Contains( + joined, + "release_identity.cutover_block is the zero placeholder", + ) { + t.Errorf( + "release-readiness must still report the zero placeholder, "+ + "got:\n%s", + joined, + ) + } + return + } + + if len(violations) != 0 { + t.Errorf( + "the reviewed cutover block is set, so the repository manifest "+ + "must be release-ready, got: %v", + violations, + ) + } +} + +// TestReleaseManifestReleaseReadyIsSeparateFromValidity proves the two verdicts +// stay apart: a document can be entirely valid and still not be something an +// acceptance decision may be taken against. +// +// It also pins where the boundary now runs. Readiness answers only what the +// manifest owns — that the reviewed cutover is a real one — and cannot be +// reached by editing the file, because the value it turns on is compiled in. +// Which artifact runs that cutover is not a question this verdict answers at +// all; that is the detached provenance's, and a readiness check that started +// answering it would be back to demanding the tree name its own commit. +func TestReleaseManifestReleaseReadyIsSeparateFromValidity(t *testing.T) { + manifest := validReleaseManifestForTests(t) + + if err := validateReleaseManifest(manifest); err != nil { + t.Fatalf("the derived manifest must be valid: [%v]", err) + } + + violations := releaseReadyViolations(manifest) + + if participation.MainnetCutoverBlock == 0 { + if len(violations) != 1 { + t.Fatalf( + "the compiled zero cutover block must be the only blocker, "+ + "got: %v", + violations, + ) + } + if !strings.Contains( + violations[0].Error(), + "release_identity.cutover_block is the zero placeholder", + ) { + t.Errorf("unexpected blocker: [%v]", violations[0]) + } + return + } + + if len(violations) != 0 { + t.Errorf( + "a manifest over a set cutover block must be release-ready, "+ + "got: %v", + violations, + ) + } +} + +// TestReleaseManifestLoadFailsClosed drives the strict decoder: unknown +// fields, trailing content, and non-integer numbers are exactly the shapes a +// hand-edited manifest degrades into, and each must be rejected outright. +func TestReleaseManifestLoadFailsClosed(t *testing.T) { + valid, err := json.Marshal(validReleaseManifestForTests(t)) + if err != nil { + t.Fatalf("cannot encode the valid manifest: [%v]", err) + } + + tests := map[string]struct { + content string + expectedMessage string + }{ + "unknown field": { + strings.Replace( + string(valid), + `"schema_version"`, + `"surprise_field":true,"schema_version"`, + 1, + ), + "unknown field", + }, + "trailing content": { + string(valid) + "{}", + "trailing content", + }, + // A stray closing delimiter is the shape a hand-edit most easily + // leaves behind, and the one a More-style check waves through: More + // only asks whether another value begins next, and a bare delimiter + // does not. + "trailing closing brace": { + string(valid) + "}", + "trailing content", + }, + "trailing closing bracket": { + string(valid) + "]", + "trailing content", + }, + "trailing number": { + string(valid) + "\n7", + "trailing content", + }, + "trailing string": { + string(valid) + ` "note"`, + "trailing content", + }, + "trailing boolean": { + string(valid) + " true", + "trailing content", + }, + "fractional grace": { + strings.Replace( + string(valid), + `"termination_grace_period_seconds":20160`, + `"termination_grace_period_seconds":20160.5`, + 1, + ), + "cannot decode", + }, + "negative margin": { + strings.Replace( + string(valid), + `"reviewed_margin_blocks":100`, + `"reviewed_margin_blocks":-100`, + 1, + ), + "cannot decode", + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + // The replacement patterns above must actually hit, otherwise the + // case silently degrades into loading a valid manifest. + if test.content == string(valid) { + t.Fatal("mutation did not change the manifest encoding") + } + + path := filepath.Join(t.TempDir(), "release-manifest.json") + if err := os.WriteFile(path, []byte(test.content), 0o600); err != nil { + t.Fatalf("cannot write the manifest fixture: [%v]", err) + } + + _, err := loadReleaseManifest(path) + if err == nil { + t.Fatal("expected the malformed manifest to be rejected") + } + if !strings.Contains(err.Error(), test.expectedMessage) { + t.Errorf( + "rejection must name the defect [%s], got: [%v]", + test.expectedMessage, + err, + ) + } + }) + } +} + +// TestReleaseManifestLoadAcceptsTrailingWhitespace pins the boundary of the +// end-of-document check: insignificant whitespace after the manifest object — +// the newline every editor and generator appends — is not trailing content. +func TestReleaseManifestLoadAcceptsTrailingWhitespace(t *testing.T) { + valid, err := json.Marshal(validReleaseManifestForTests(t)) + if err != nil { + t.Fatalf("cannot encode the valid manifest: [%v]", err) + } + + path := filepath.Join(t.TempDir(), "release-manifest.json") + content := append(valid, " \t\r\n\n"...) + if err := os.WriteFile(path, content, 0o600); err != nil { + t.Fatalf("cannot write the manifest fixture: [%v]", err) + } + + manifest, err := loadReleaseManifest(path) + if err != nil { + t.Fatalf("expected the whitespace-terminated manifest to load: [%v]", err) + } + if err := validateReleaseManifest(manifest); err != nil { + t.Errorf("expected the loaded manifest to validate: [%v]", err) + } +} + +func TestReleaseManifestLoadRejectsMissingFile(t *testing.T) { + _, err := loadReleaseManifest( + filepath.Join(t.TempDir(), "absent-manifest.json"), + ) + if err == nil { + t.Fatal("expected a missing manifest to be rejected") + } + if !strings.Contains(err.Error(), "cannot open") { + t.Errorf("unexpected rejection message: [%v]", err) + } +} diff --git a/cmd/start.go b/cmd/start.go index c5bc8902f2..61ec50f5d7 100644 --- a/cmd/start.go +++ b/cmd/start.go @@ -3,6 +3,10 @@ package cmd import ( "context" "fmt" + "math" + "os" + "os/signal" + "syscall" "time" "github.com/keep-network/keep-core/pkg/tbtcpg" @@ -25,6 +29,7 @@ import ( "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/net/libp2p" "github.com/keep-network/keep-core/pkg/net/retransmission" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tbtc" ) @@ -63,7 +68,66 @@ Environment variables: // start starts a node func start(cmd *cobra.Command) error { + // Two-context lifecycle: the gate and the cutover roster live on the root + // context and are closed explicitly, while every protocol component + // receives runCtx, which the signal controller cancels only after + // quiescence has run its course. Handing protocol code an + // already-canceled signal context would defeat graceful completion. ctx := context.Background() + runCtx, cancelRunCtx := context.WithCancel(ctx) + defer cancelRunCtx() + + // The roster evidence window is operational observability only. SIGUSR1 + // opens it for go/no-go capture and SIGUSR2 closes it; the shutdown + // controller holds it open before rollback quiescence. Neither signal is + // visible to the participation gate or any protocol-mode decision. + cutoverEvidenceWindow := participation.NewCutoverEvidenceWindowSignal() + evidenceWindowSignalChan := make(chan os.Signal, 2) + signal.Notify( + evidenceWindowSignalChan, + syscall.SIGUSR1, + syscall.SIGUSR2, + ) + defer signal.Stop(evidenceWindowSignalChan) + + // Signal capture is installed before anything else so that no window of + // the startup sequence is left to the default signal action: a signal + // arriving before the participation gate exists is held in the buffered + // channel and acted on the moment the lifecycle controller starts, + // immediately after the gate is constructed. + signalChan := make(chan os.Signal, 2) + signal.Notify(signalChan, syscall.SIGTERM, syscall.SIGINT) + defer signal.Stop(signalChan) + + // Resolve the protocol participation schedule before connecting anywhere: + // these are configuration-only checks, and a misconfigured cutover block + // must terminate startup before any component can send protocol traffic. + participationSchedule, err := participation.ResolveAndValidate( + clientConfig.Ethereum.Network, + clientConfig.ProtocolParticipation, + ) + if err != nil { + return fmt.Errorf( + "protocol participation schedule rejected: [%v]", + err, + ) + } + + cutoverBlockSource := "release_baked" + if clientConfig.ProtocolParticipation.CutoverBlockSet { + cutoverBlockSource = "non_mainnet_override" + } + logger.Infof( + "protocol participation schedule resolved [version=%s] "+ + "[revision=%s] [epoch=%s] [cutoverBlock=%d] [source=%s] "+ + "[disabled=%t]", + build.Version, + build.Revision, + participation.CompiledEpoch, + participationSchedule.CutoverBlock, + cutoverBlockSource, + participationSchedule.Disabled(), + ) beaconChain, tbtcChain, blockCounter, signing, operatorPrivateKey, err := ethereum.Connect(ctx, clientConfig.Ethereum) @@ -71,28 +135,182 @@ func start(cmd *cobra.Command) error { return fmt.Errorf("error connecting to Ethereum node: [%v]", err) } - netProvider, err := initializeNetwork( + // The client-info registry and its chain-bound observers start first: the + // participation gate and cutover roster constructed below need a real + // metrics sink before the network provider exists. The network-bound + // observers attach right after the network initializes. + clientInfoRegistry := initializeClientInfo(runCtx, clientConfig, blockCounter) + + var perfMetrics *clientinfo.PerformanceMetrics + if clientInfoRegistry != nil { + perfMetrics = clientinfo.NewPerformanceMetrics(runCtx, clientInfoRegistry) + + // Wire performance metrics into firewall validation so live on-chain + // IsRecognized calls are counted. The recorder is a package-level sink + // read at validation time, so setting it before the network provider + // is constructed loses no events. + firewall.SetMetricsRecorder(perfMetrics) + } + + // Construct the production participation gate and the cutover peer roster + // from the shared block counter immediately after the Ethereum connection + // and before the network provider, beacon, or tBTC can send protocol + // traffic. The gate performs its first synchronous chain-clock read here + // and a clock error refuses startup. With client-info disabled both record + // to a no-op sink so their logs and state machines still function. + var gateMetrics participation.GateMetricsRecorder + if perfMetrics != nil { + gateMetrics = perfMetrics + } else { + gateMetrics = &clientinfo.NoOpPerformanceMetrics{} + } + + // The gate's quiescence capture is persisted in its own encrypted work + // namespace. Initialize it before constructing the gate so even a signal + // received during the rest of startup produces a node-authored inventory + // in the storage snapshot later handed to the rollback audit. + participationPersistence, err := initializeParticipationPersistence() + if err != nil { + return fmt.Errorf( + "cannot initialize participation persistence: [%w]", + err, + ) + } + quiescenceRecorder, err := + participation.NewPersistenceQuiescenceSnapshotRecorder( + participationPersistence, + ) + if err != nil { + return fmt.Errorf( + "cannot construct the quiescence snapshot recorder: [%w]", + err, + ) + } + + participationGate, err := participation.NewGate( ctx, - []firewall.Application{beaconChain, tbtcChain}, - operatorPrivateKey, + participationSchedule, blockCounter, + gateMetrics, + participation.WithArtifactIdentity(build.Version, build.Revision), + participation.WithQuiescenceSnapshotRecorder(quiescenceRecorder), ) if err != nil { - return fmt.Errorf("cannot initialize network: [%v]", err) + return fmt.Errorf("cannot construct the participation gate: [%v]", err) } + defer participationGate.Close() - clientInfoRegistry := initializeClientInfo( + rosterRetentionBlocks, err := tbtc.CutoverPeerRosterRetentionBlocks() + if err != nil { + return fmt.Errorf( + "cannot derive cutover peer roster retention: [%v]", + err, + ) + } + var rosterMetrics participation.CutoverRosterMetricsRecorder + if perfMetrics != nil { + rosterMetrics = perfMetrics + } else { + rosterMetrics = &clientinfo.NoOpPerformanceMetrics{} + } + cutoverRoster, err := participation.NewCutoverPeerRoster( ctx, - clientConfig, - netProvider, - signing, blockCounter, + rosterRetentionBlocks, + rosterMetrics, + participation.WithCutoverSchedule(participationSchedule), + participation.WithCutoverEvidenceWindowSignal(cutoverEvidenceWindow), + ) + if err != nil { + return fmt.Errorf("cannot create cutover peer roster: [%v]", err) + } + defer cutoverRoster.Close() + + startEvidenceWindowSignalController( + runCtx, + cutoverEvidenceWindow, + evidenceWindowSignalChan, ) - // Wire performance metrics into network provider if available - var perfMetrics *clientinfo.PerformanceMetrics if clientInfoRegistry != nil { - perfMetrics = clientinfo.NewPerformanceMetrics(ctx, clientInfoRegistry) + // The chain identity handed to the diagnostics contract is the + // connected beacon chain itself, so every scrape reports the chain + // this node actually reached rather than the one it was configured to + // reach. + participation.RegisterDiagnosticSources( + clientInfoRegistry, + participationGate, + cutoverRoster, + beaconChain, + cutoverBlockSource, + ) + } + + beaconCompletionBound, err := beacon.MaximumLegacyCompletionBlocks( + beaconChain.GetConfig(), + ) + if err != nil { + return fmt.Errorf("cannot derive the beacon completion bound: [%v]", err) + } + maximumCompletionBound := tbtc.MaximumLegacyCompletionBlocks() + if beaconCompletionBound > maximumCompletionBound { + maximumCompletionBound = beaconCompletionBound + } + + // The quiesce backstop is derived at startup with checked arithmetic: an + // overflowing deadline calculation refuses startup instead of producing a + // silently truncated grace period at shutdown time. + quiesceBackstop, err := quiesceBackstopDeadline(maximumCompletionBound) + if err != nil { + return fmt.Errorf("cannot derive the quiesce backstop: [%v]", err) + } + + // The lifecycle controller arms while only the gate and roster exist — + // before the network provider, beacon, or tBTC can begin protocol work — + // so a termination signal received at any later point of startup quiesces + // the gate immediately instead of waiting for initialization to finish. + shutdownChan := startSignalLifecycleController( + runCtx, + cancelRunCtx, + participationGate, + cutoverEvidenceWindow, + signalChan, + quiesceBackstop, + forcedCancellationAllowance(), + ) + + gateSnapshot := participationGate.State() + logger.Infof( + "protocol participation gate started [state=%s] [currentBlock=%d] "+ + "[cutoverBlock=%d] [revision=%s] [epoch=%s] "+ + "[maximumLegacyCompletionBlocks=%d] [quiesceMarginBlocks=%d] "+ + "[quiesceUpperBlockIntervalSeconds=%d] [quiesceBackstop=%s] "+ + "[source=%s]", + gateSnapshot.State, + gateSnapshot.CurrentBlock, + gateSnapshot.CutoverBlock, + build.Revision, + participation.CompiledEpoch, + maximumCompletionBound, + quiesceReviewedMarginBlocks, + quiesceUpperBlockIntervalSeconds, + quiesceBackstop, + cutoverBlockSource, + ) + + netProvider, err := initializeNetwork( + runCtx, + []firewall.Application{beaconChain, tbtcChain}, + operatorPrivateKey, + blockCounter, + ) + if err != nil { + return fmt.Errorf("cannot initialize network: [%v]", err) + } + + registerNetworkClientInfo(clientConfig, clientInfoRegistry, netProvider, signing) + + if perfMetrics != nil { // Type assert to libp2p provider to set metrics recorder // The provider struct is not exported, so we use interface assertion if setter, ok := netProvider.(interface { @@ -104,23 +322,21 @@ func start(cmd *cobra.Command) error { }); ok { setter.SetMetricsRecorder(perfMetrics) } - - // Wire performance metrics into firewall validation so live - // on-chain IsRecognized calls are counted. - firewall.SetMetricsRecorder(perfMetrics) } // Initialize beacon and tbtc only for non-bootstrap nodes. // Skip initialization for bootstrap nodes as they are only used for network // discovery. if !isBootstrap() { - btcChain, err := electrum.Connect(ctx, clientConfig.Bitcoin.Electrum) + btcChain, err := electrum.Connect(runCtx, clientConfig.Bitcoin.Electrum) if err != nil { return fmt.Errorf("could not connect to Electrum chain: [%v]", err) } beaconKeyStorePersistence, + beaconQuarantinePersistence, tbtcKeyStorePersistence, + tbtcQuarantinePersistence, tbtcDataPersistence, err := initializePersistence() if err != nil { @@ -143,15 +359,18 @@ func start(cmd *cobra.Command) error { btcChain, clientConfig.ClientInfo.RPCHealthCheckInterval, ) - rpcHealthChecker.Start(ctx) + rpcHealthChecker.Start(runCtx) } err = beacon.Initialize( - ctx, + runCtx, beaconChain, netProvider, beaconKeyStorePersistence, + beaconQuarantinePersistence, scheduler, + participationGate, + gateMetrics, ) if err != nil { return fmt.Errorf("error initializing beacon: [%v]", err) @@ -163,11 +382,12 @@ func start(cmd *cobra.Command) error { ) err = tbtc.Initialize( - ctx, + runCtx, tbtcChain, btcChain, netProvider, tbtcKeyStorePersistence, + tbtcQuarantinePersistence, tbtcDataPersistence, scheduler, proposalGenerator, @@ -175,6 +395,8 @@ func start(cmd *cobra.Command) error { clientInfoRegistry, perfMetrics, // Pass the existing performance metrics instance to avoid duplicate registrations clientConfig.Ethereum.Network, + participationGate, + cutoverRoster, ) if err != nil { return fmt.Errorf("error initializing TBTC: [%v]", err) @@ -188,8 +410,311 @@ func start(cmd *cobra.Command) error { clientConfig.Ethereum, ) - <-ctx.Done() - return fmt.Errorf("shutting down the node because its context has ended") + // The lifecycle controller has been driving signals since right after the + // gate was constructed; from here the main goroutine only waits for its + // shutdown report or for the run context to end for another reason. + select { + case err := <-shutdownChan: + return err + case <-runCtx.Done(): + // The controller sends its shutdown report before it cancels the run + // context, so a pending report is always preferred over the bare + // context end. + select { + case err := <-shutdownChan: + return err + default: + } + return fmt.Errorf("shutting down the node because its context has ended") + } +} + +// startSignalLifecycleController launches the signal-driven shutdown +// controller. It must be started as soon as the participation gate exists, +// before the network provider or either application can begin protocol work: +// the first SIGTERM/SIGINT — including one that arrives while startup is +// still initializing components — immediately quiesces the gate, so no new +// permit is issued from that moment on, while existing ceremonies run to +// natural completion. A second signal or the in-process backstop deadline +// forces the remainder through the gate's audited forced-cancellation path. +// That path is two-phase: Close cancels the permits that outlived the drain, +// and the controller then keeps the run context alive until their owners +// finish the cancellation cleanup — the quarantine and audit writes included +// — and release them, bounded by the reviewed forced-cancellation allowance. +// Only then is the shutdown cause reported and the run context canceled, so +// in-flight protocol work and its cleanup keep network, chain, and +// persistence access for the whole shutdown. The returned channel reports +// the shutdown cause once the drive completes. +func startSignalLifecycleController( + runCtx context.Context, + cancelRunCtx context.CancelFunc, + gate participation.Gate, + evidenceWindow *participation.CutoverEvidenceWindowSignal, + signals <-chan os.Signal, + backstop time.Duration, + cancellationAllowance time.Duration, +) <-chan error { + shutdown := make(chan error, 1) + + go func() { + select { + case receivedSignal := <-signals: + // Rollback evidence must include empty roster snapshots throughout + // the drain. Hold the logging-only signal open before the gate + // enters quiescence so a later SIGUSR2 cannot suppress that evidence. + evidenceWindow.HoldActive() + logger.Infof( + "protocol cutover evidence window held active "+ + "[source=quiescence] [signal=%v]", + receivedSignal, + ) + + quiesceCause := fmt.Errorf("received signal [%v]", receivedSignal) + quiesceDone := gate.Quiesce(quiesceCause) + + reason := awaitQuiesce(quiesceDone, signals, backstop) + logger.Infof( + "protocol participation quiescence ended [reason=%s] "+ + "[signal=%v]", + reason, + receivedSignal, + ) + + // Close force-cancels any permit that outlived the drain and + // stops the clock supervisor. The canceled permits stay counted + // until their owners finish the cancellation cleanup and release + // them, and the gate reports that through its drained channel. + gate.Close() + + // Phase two of the forced cancellation: join the owners of the + // canceled permits before letting the process exit, so quarantine + // and audit writes are never cut off mid-flight. The wait is + // bounded by the reviewed forced-cancellation allowance — the + // same wall-clock room the release manifest grants the service + // manager between the in-process deadline and SIGKILL. After + // natural completion the drained channel is already closed and + // the wait returns immediately. + cleanupReason := awaitForcedCancellationCleanup( + gate.Drained(), + cancellationAllowance, + ) + if cleanupReason == cleanupReasonAllowanceExceeded { + logger.Warnf( + "protocol participation forced-cancellation cleanup did "+ + "not finish within the [%s] allowance; shutting down "+ + "with permits still held [signal=%v]", + cancellationAllowance, + receivedSignal, + ) + } else { + logger.Infof( + "protocol participation forced-cancellation cleanup "+ + "ended [reason=%s] [signal=%v]", + cleanupReason, + receivedSignal, + ) + } + + // The shutdown report is sent before the run context is canceled + // so the report is already pending whenever the main goroutine + // observes the context end. + shutdown <- fmt.Errorf( + "shutting down the node after signal [%v]", + receivedSignal, + ) + cancelRunCtx() + case <-runCtx.Done(): + // The process is ending for another reason; there is no drain to + // drive. + } + }() + + return shutdown +} + +// startEvidenceWindowSignalController applies the operator's logging-only +// evidence-window controls. SIGUSR1 opens the window and SIGUSR2 closes it. +// The signal changes only the roster's decision to log an empty periodic +// snapshot; protocol authorization and mode selection remain exclusively +// owned by the participation gate. +func startEvidenceWindowSignalController( + ctx context.Context, + evidenceWindow *participation.CutoverEvidenceWindowSignal, + signals <-chan os.Signal, +) <-chan struct{} { + done := make(chan struct{}) + + go func() { + defer close(done) + + for { + select { + case <-ctx.Done(): + return + case receivedSignal, ok := <-signals: + if !ok { + return + } + + var ( + active bool + changed bool + ) + switch receivedSignal { + case syscall.SIGUSR1: + active = true + changed = evidenceWindow.SetActive(true) + case syscall.SIGUSR2: + active = false + changed = evidenceWindow.SetActive(false) + default: + continue + } + + if changed { + logger.Infof( + "protocol cutover evidence window changed "+ + "[active=%t] [signal=%v]", + active, + receivedSignal, + ) + } + } + } + }() + + return done +} + +// quiesceUpperBlockIntervalSeconds is the conservative upper bound on the +// Ethereum block interval used to convert the block-clock completion bound +// into the in-process wall-clock backstop. The release manifest derives the +// authoritative external termination grace from reviewed production evidence; +// this value only sizes the last-resort in-process deadline. +const quiesceUpperBlockIntervalSeconds = 15 + +// quiesceReviewedMarginBlocks is the reviewed block margin added on top of the +// maximum legacy completion bound before the conversion to wall time: it +// absorbs chain-clock jitter and late block delivery around the completion +// bound so a ceremony finishing exactly at its protocol deadline is not +// force-canceled by the backstop. The release manifest records this margin +// beside the completion bound and the block-interval bound as the inputs of +// the external termination grace. +const quiesceReviewedMarginBlocks = uint64(100) + +// quiesceBackstopMargin absorbs RPC and processing skew on top of the +// block-derived backstop. +const quiesceBackstopMargin = 5 * time.Minute + +// quiesceBackstopDeadline converts the maximum legacy completion bound plus +// the reviewed block margin into the in-process wall-clock backstop for the +// quiesce drain, with every step overflow-checked. The service manager's +// configured termination grace, derived in the release manifest from the same +// inputs, remains the authoritative external deadline; this backstop only +// guarantees the audited forced-cancellation path runs even if no second +// signal ever arrives. +func quiesceBackstopDeadline(completionBoundBlocks uint64) (time.Duration, error) { + totalBlocks := completionBoundBlocks + quiesceReviewedMarginBlocks + if totalBlocks < completionBoundBlocks { + return 0, fmt.Errorf( + "quiesce backstop block bound overflows: completion bound [%d] "+ + "plus margin [%d]", + completionBoundBlocks, + quiesceReviewedMarginBlocks, + ) + } + + totalSeconds := totalBlocks * quiesceUpperBlockIntervalSeconds + if totalSeconds/quiesceUpperBlockIntervalSeconds != totalBlocks { + return 0, fmt.Errorf( + "quiesce backstop seconds overflow: [%d] blocks at [%d] "+ + "seconds per block", + totalBlocks, + quiesceUpperBlockIntervalSeconds, + ) + } + + if totalSeconds > uint64(math.MaxInt64/time.Second) { + return 0, fmt.Errorf( + "quiesce backstop duration overflows: [%d] seconds", + totalSeconds, + ) + } + backstop := time.Duration(totalSeconds) * time.Second + + if backstop > math.MaxInt64-quiesceBackstopMargin { + return 0, fmt.Errorf( + "quiesce backstop duration overflows with the [%s] margin", + quiesceBackstopMargin, + ) + } + + return backstop + quiesceBackstopMargin, nil +} + +// awaitQuiesce waits for the quiesce drain to end and reports why: natural +// completion of every active permit, a second operator signal forcing +// shutdown, or the in-process backstop deadline. +func awaitQuiesce( + quiesceDone <-chan struct{}, + signals <-chan os.Signal, + backstop time.Duration, +) string { + backstopTimer := time.NewTimer(backstop) + defer backstopTimer.Stop() + + select { + case <-quiesceDone: + return "completed" + case <-signals: + return "forced_by_signal" + case <-backstopTimer.C: + return "backstop_deadline" + } +} + +// The two ways the forced-cancellation cleanup wait can end: every canceled +// permit was released by its owner, or the reviewed allowance elapsed first. +const ( + cleanupReasonDrained = "drained" + cleanupReasonAllowanceExceeded = "allowance_exceeded" +) + +// forcedCancellationAllowance is the wall-clock bound on the second phase of +// a forced shutdown: the time the controller keeps the process alive after +// canceling the remaining permits so their owners can finish quarantine and +// audit writes. It is the same compiled allowance the release manifest adds +// on top of the in-process backstop when deriving the service manager's +// termination grace — together with the compiled process-exit headroom that +// budgets the controller scheduling, quiesce and close calls, shutdown +// logging, and teardown running outside both in-process timers — so the +// external SIGKILL deadline, counted from signal delivery, always ends +// strictly after this wait does. Manifest validation rejects a manifest +// recording any other allowance, so a reviewed grace always budgets exactly +// this wait. Deliberately not a signal-escapable wait: an operator hammering +// the terminal must not be able to cut off key-material persistence. +func forcedCancellationAllowance() time.Duration { + return time.Duration(compiledForcedCancellationAllowanceSeconds) * + time.Second +} + +// awaitForcedCancellationCleanup waits for the owners of force-canceled +// permits to finish their cancellation cleanup and release them, bounded by +// the reviewed forced-cancellation allowance, and reports which of the two +// ended the wait. +func awaitForcedCancellationCleanup( + drained <-chan struct{}, + allowance time.Duration, +) string { + allowanceTimer := time.NewTimer(allowance) + defer allowanceTimer.Stop() + + select { + case <-drained: + return cleanupReasonDrained + case <-allowanceTimer.C: + return cleanupReasonAllowanceExceeded + } } func isBootstrap() bool { @@ -224,11 +749,14 @@ func initializeNetwork( return netProvider, nil } +// initializeClientInfo starts the client-info registry and attaches the +// chain-bound observers. It runs before the network provider exists because +// the participation gate and cutover roster need its metrics sink from the +// first chain-clock read; the network-bound observers attach later through +// registerNetworkClientInfo. func initializeClientInfo( ctx context.Context, config *config.Config, - netProvider net.Provider, - signing chain.Signing, blockCounter chain.BlockCounter, ) *clientinfo.Registry { registry, isConfigured := clientinfo.Initialize(ctx, config.ClientInfo.Port) @@ -237,6 +765,40 @@ func initializeClientInfo( return nil } + registry.ObserveEthConnectivity( + blockCounter, + config.ClientInfo.EthereumMetricsTick, + ) + + registry.RegisterMetricClientInfo( + build.Version, + build.Revision, + participation.CompiledEpoch.String(), + ) + + registry.RegisterEthChainInfoSource(blockCounter) + + logger.Infof( + "enabled client info endpoint on port [%v]", + config.ClientInfo.Port, + ) + + return registry +} + +// registerNetworkClientInfo attaches the network-bound client-info observers +// once the network provider exists. It is a no-op when the client-info +// endpoint is not configured. +func registerNetworkClientInfo( + config *config.Config, + registry *clientinfo.Registry, + netProvider net.Provider, + signing chain.Signing, +) { + if registry == nil { + return + } + registry.ObserveConnectedPeersCount( netProvider, config.ClientInfo.NetworkMetricsTick, @@ -248,13 +810,6 @@ func initializeClientInfo( config.ClientInfo.NetworkMetricsTick, ) - registry.ObserveEthConnectivity( - blockCounter, - config.ClientInfo.EthereumMetricsTick, - ) - - registry.RegisterMetricClientInfo(build.Version) - registry.RegisterConnectedPeersSource(netProvider, signing) registry.RegisterClientInfoSource( @@ -263,20 +818,13 @@ func initializeClientInfo( build.Version, build.Revision, ) - - registry.RegisterEthChainInfoSource(blockCounter) - - logger.Infof( - "enabled client info endpoint on port [%v]", - config.ClientInfo.Port, - ) - - return registry } func initializePersistence() ( beaconKeyStorePersistence persistence.ProtectedHandle, + beaconQuarantinePersistence persistence.ProtectedHandle, tbtcKeyStorePersistence persistence.ProtectedHandle, + tbtcQuarantinePersistence persistence.ProtectedHandle, tbtcDataPersistence persistence.BasicHandle, err error, ) { @@ -285,32 +833,61 @@ func initializePersistence() ( clientConfig.Ethereum.KeyFilePassword, ) if err != nil { - return nil, nil, nil, fmt.Errorf("cannot initialize storage: [%w]", err) + return nil, nil, nil, nil, nil, fmt.Errorf( + "cannot initialize storage: [%w]", + err, + ) } beaconKeyStorePersistence, err = storage.InitializeKeyStorePersistence( "beacon", ) if err != nil { - return nil, nil, nil, fmt.Errorf( + return nil, nil, nil, nil, nil, fmt.Errorf( "cannot initialize beacon keystore persistence: [%w]", err, ) } + // The quarantine namespace is a sibling of the active beacon keystore, so + // no release's active-group scan — which reads only the "beacon" directory + // — can load a quarantined signer output as an active signer. + beaconQuarantinePersistence, err = storage.InitializeKeyStorePersistence( + "beacon-quarantine", + ) + if err != nil { + return nil, nil, nil, nil, nil, fmt.Errorf( + "cannot initialize beacon quarantine persistence: [%w]", + err, + ) + } + tbtcKeyStorePersistence, err = storage.InitializeKeyStorePersistence( "tbtc", ) if err != nil { - return nil, nil, nil, fmt.Errorf( + return nil, nil, nil, nil, nil, fmt.Errorf( "cannot initialize tbtc keystore persistence: [%w]", err, ) } + // The quarantine namespace is a sibling of the active tbtc keystore, so + // no release's active-wallet scan — which reads only the "tbtc" directory + // — can load a quarantined signer output as an active signer. + tbtcQuarantinePersistence, err = storage.InitializeKeyStorePersistence( + "tbtc-quarantine", + ) + if err != nil { + return nil, nil, nil, nil, nil, fmt.Errorf( + "cannot initialize tbtc quarantine persistence: [%w]", + err, + ) + } + tbtcDataPersistence, err = storage.InitializeWorkPersistence("tbtc") if err != nil { - return nil, nil, nil, fmt.Errorf( + return nil, nil, nil, nil, nil, fmt.Errorf( "cannot initialize tbtc data persistence: [%w]", err, ) @@ -318,3 +895,30 @@ func initializePersistence() ( return } + +func initializeParticipationPersistence() ( + persistence.BasicHandle, + error, +) { + diskStorage, err := storage.Initialize( + clientConfig.Storage, + clientConfig.Ethereum.KeyFilePassword, + ) + if err != nil { + return nil, fmt.Errorf( + "cannot initialize storage: [%w]", + err, + ) + } + + participationPersistence, err := + diskStorage.InitializeWorkPersistence("participation") + if err != nil { + return nil, fmt.Errorf( + "cannot initialize participation work persistence: [%w]", + err, + ) + } + + return participationPersistence, nil +} diff --git a/config/category.go b/config/category.go index f6b3f2ab0c..edcc2160b6 100644 --- a/config/category.go +++ b/config/category.go @@ -12,6 +12,7 @@ const ( Tbtc Maintainer Developer + ProtocolParticipation ) // StartCmdCategories are categories needed for the start command. @@ -23,6 +24,7 @@ var StartCmdCategories = []Category{ Storage, ClientInfo, Tbtc, + ProtocolParticipation, Developer, } @@ -30,6 +32,7 @@ var StartCmdCategories = []Category{ var MaintainerCategories = []Category{ Ethereum, BitcoinElectrum, + ClientInfo, Maintainer, } @@ -43,5 +46,6 @@ var AllCategories = []Category{ ClientInfo, Tbtc, Maintainer, + ProtocolParticipation, Developer, } diff --git a/config/config.go b/config/config.go index 92081b2f10..b7c451c41a 100644 --- a/config/config.go +++ b/config/config.go @@ -23,6 +23,7 @@ import ( "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/maintainer" "github.com/keep-network/keep-core/pkg/net/libp2p" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/storage" "github.com/keep-network/keep-core/pkg/tbtc" ) @@ -45,13 +46,14 @@ const ( // Config is the top level config structure. type Config struct { - Ethereum commonEthereum.Config - Bitcoin BitcoinConfig - LibP2P libp2p.Config `mapstructure:"network"` - Storage storage.Config - ClientInfo clientinfo.Config - Maintainer maintainer.Config - Tbtc tbtc.Config + Ethereum commonEthereum.Config + Bitcoin BitcoinConfig + LibP2P libp2p.Config `mapstructure:"network"` + Storage storage.Config + ClientInfo clientinfo.Config + Maintainer maintainer.Config + Tbtc tbtc.Config + ProtocolParticipation participation.Config `mapstructure:"protocolParticipation"` } // BitcoinConfig defines the configuration for Bitcoin. @@ -141,6 +143,17 @@ func (c *Config) ReadConfig(configFilePath string, flagSet *pflag.FlagSet, categ return fmt.Errorf("unable to unmarshal config: %w", err) } + // Record whether the protocol participation cutover block was explicitly + // supplied at all: mainnet rejection is keyed on this presence — an + // explicit zero must be rejected too — so the decoded numeric value alone + // is not enough. Viper's IsSet deliberately ignores unchanged flag + // defaults, so this is true only for a config-file key or an explicitly + // changed flag. + c.ProtocolParticipation.CutoverBlockSet = + viper.IsSet("protocolParticipation.cutoverBlock") || + (flagSet != nil && + flagSet.Changed("protocolParticipation.cutoverBlock")) + // Resolve contracts addresses. c.resolveContractsAddresses() diff --git a/config/config_test.go b/config/config_test.go index f8de558c4e..3d7aacbfe4 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -255,6 +255,31 @@ func TestReadConfigFromFile(t *testing.T) { } } +// TestReadConfig_ClientInfoPortZero pins the lower-level configuration path: an +// explicit `[clientInfo] Port = 0` in a TOML file must unmarshal to zero. This +// supplements, and does not replace, the two command-path explicit-zero tests in +// cmd/flags_test.go, because Viper unmarshalling and flag binding follow different +// precedence rules. +func TestReadConfig_ClientInfoPortZero(t *testing.T) { + t.Setenv(EthereumPasswordEnvVariable, "test-password") + + cfg := &Config{} + if err := cfg.ReadConfig( + "../test/config_clientinfo_zero.toml", + nil, + AllCategories..., + ); err != nil { + t.Fatalf("failed to read test config: [%v]", err) + } + + if cfg.ClientInfo.Port != 0 { + t.Errorf( + "expected clientInfo.port to be 0, got [%d]", + cfg.ClientInfo.Port, + ) + } +} + func TestReadConfig_ReadPassword(t *testing.T) { expectToPrompt := "expect-to-prompt" diff --git a/configs/config.toml.SAMPLE b/configs/config.toml.SAMPLE index 9ef7e220e9..00fe4ad77f 100644 --- a/configs/config.toml.SAMPLE +++ b/configs/config.toml.SAMPLE @@ -104,6 +104,13 @@ Dir = "/my/secure/location" # Diagnostics module exposes the following information: # - list of connected peers along with their network id and ethereum operator address # - information about the client's network id and ethereum operator address +# +# The metrics/diagnostics HTTP server listens on the compatibility default port +# 9601. This is a temporary compatibility default for the coordinated security +# release; set Port = 0 to explicitly disable the server. The endpoint is +# unauthenticated, so expose it only over a trusted network path (firewall/VPN +# or an authenticated proxy). Operators MUST commit an explicit Port value before +# the follow-up R2 release flips the default back to 0 (disabled). [clientInfo] Port = 9601 # NetworkMetricsTick = 60 @@ -118,6 +125,14 @@ Port = 9601 # PreParamsGenerationConcurrency = 1 # KeyGenerationConcurrency = 1 +# Protocol cutover block override for NON-MAINNET networks only. Mainnet always +# uses the compiled release constant and refuses to start when this setting is +# present, including with an explicit zero. Testnet release rehearsals must set +# a nonzero value; developer mode may use 0 to disable the cutover schedule. +# +# [protocolParticipation] +# CutoverBlock = 124000 + # Developer options to work with locally deployed contracts # # [developer] diff --git a/docs/performance-metrics.adoc b/docs/performance-metrics.adoc index af2a7132bc..0276cadf65 100644 --- a/docs/performance-metrics.adoc +++ b/docs/performance-metrics.adoc @@ -7,11 +7,19 @@ through the `/metrics` endpoint when the client info endpoint is configured. == Metrics Endpoint Metrics are exposed via HTTP at the `/metrics` endpoint on the port configured -in the `ClientInfo` section of the configuration file (default: `9601`). +in the `ClientInfo` section of the configuration file. For the coordinated +security release the client-info server listens on the compatibility default +port `9601`; this is a temporary compatibility default. Set an explicit +`[clientInfo] Port = 9601` (or another port) to keep it, or `[clientInfo] +Port = 0` to explicitly disable it. The endpoint is unauthenticated and MUST be +exposed only on a trusted/private network path (firewall/VPN or an +authenticated proxy). The follow-up R2 release will flip the default back to +`0` (disabled) once the monitoring migration is complete; commit an explicit +`clientInfo.port` value before then. Example: ---- -curl http://localhost:9601/metrics +curl http://localhost:/metrics ---- == Metric Types @@ -276,4 +284,187 @@ For each action type, the following metrics are available: ==== `performance_relay_entry_timeout_reported_total` *Type*: Counter *Description*: Total number of relay entry timeouts reported on-chain -*Labels*: None \ No newline at end of file +*Labels*: None + +=== Cutover-Readiness Observability Metrics + +The coordinated security release adds stranded/legacy-peer observability so that +cutover readiness can identify which operators remain nonconverged. The +following node-local metrics are recorded by the tBTC DKG and signing announcer +mismatch observer and the node-local cutover peer roster +(`pkg/protocol/participation`). They deduplicate every post-cutover legacy-wire +sighting down to the normalized operator address; they never carry operator, +session, or peer labels. + +NOTE: These observability metrics do not require the block-height cutover gate. +The gate ships in this build as well; its `performance_participation_*` metrics +are documented in the Protocol Participation Gate Metrics section below. + +==== `performance_announcer_session_id_mismatch_total` +*Type*: Counter +*Description*: Unique membership-valid senders per announce call whose announced session ID differs from the local one +*Labels*: None + +==== `performance_announcer_cross_format_peer_total` +*Type*: Counter +*Description*: Subset of session-ID mismatches classified as a legacy-versus-hardened cross-format difference +*Labels*: None + +==== `performance_announcer_legacy_peers_current` +*Type*: Gauge +*Description*: Deduplicated operator addresses currently retained in the local cutover roster +*Labels*: None + +==== `performance_announcer_legacy_peer_oldest_age_blocks` +*Type*: Gauge +*Description*: Current block minus the oldest retained first-seen block +*Labels*: None + +==== `performance_announcer_legacy_peer_roster_revision` +*Type*: Gauge +*Description*: Monotonic process-local roster revision +*Labels*: None + +==== `performance_announcer_legacy_peer_additions_total` +*Type*: Counter +*Description*: Absent-to-present operator transitions in the local cutover roster +*Labels*: None + +==== `performance_announcer_legacy_peer_evictions_total` +*Type*: Counter +*Description*: Final-sighting retention evictions from the local cutover roster +*Labels*: None + +The authoritative fleet view is produced by the separate `cutover-roster` +aggregator (`cmd/cutover-roster`), which exposes its own +`performance_cutover_fleet_*` and `performance_cutover_operator_*` metrics on a +monitoring-only endpoint. Those metrics are documented with that tool and are +not served from a node's `/metrics` endpoint. + +=== Protocol Participation Gate Metrics + +The chain-clocked participation gate (`pkg/protocol/participation`) decides which +cryptographic format each ceremony runs, from the ceremony's own canonical chain +anchor. These metrics are the evidence the cutover go/no-go and rollback +decisions are made on: what format the node is issuing, what it is still running, +and what it refused. + +Every metric below is registered at its zero value with the fixed performance +family, so a scraper sees the complete set from process start and an absent +metric means the instrument is absent — not that nothing has happened yet. None +of them carries an operator, session, wallet, or peer label. + +==== `performance_participation_gate_state` +*Type*: Gauge +*Description*: Current participation state: `0` disabled, `1` open_legacy, `2` open_security_v2, `3` quiescing, `4` clock_unavailable +*Labels*: None + +==== `performance_participation_current_block` +*Type*: Gauge +*Description*: Latest successfully read Ethereum block height +*Labels*: None + +==== `performance_participation_cutover_block` +*Type*: Gauge +*Description*: Resolved cutover block at and above which ceremonies run the hardened format +*Labels*: None + +==== `performance_participation_allowed` +*Type*: Gauge +*Description*: `1` when a new permit can be issued, `0` while quiescing or without a usable chain clock +*Labels*: None + +==== `performance_participation_active_ceremonies` +*Type*: Gauge +*Description*: Permits currently held, in either format +*Labels*: None + +==== `performance_participation_active_legacy_ceremonies` +*Type*: Gauge +*Description*: Permits currently held that were issued in the legacy format +*Labels*: None + +==== `performance_participation_active_security_v2_ceremonies` +*Type*: Gauge +*Description*: Permits currently held that were issued in the hardened format +*Labels*: None + +==== `performance_participation_mode_legacy_total` +*Type*: Counter +*Description*: Permits issued in the legacy format +*Labels*: None + +==== `performance_participation_mode_security_v2_total` +*Type*: Counter +*Description*: Permits issued in the hardened format +*Labels*: None + +==== `performance_participation_legacy_completions_after_cutover_total` +*Type*: Counter +*Description*: Legacy-format completions committed at or after the cutover block — a ceremony that began before it and was allowed to finish +*Labels*: None + +==== `performance_participation_refusals_total` +*Type*: Counter +*Description*: Permits refused for an invalid anchor, an unusable chain clock, or quiescence +*Labels*: None + +==== `performance_participation_refusals__total` +*Type*: Counter +*Description*: The aggregate refusal counter broken out per gated ceremony, as one fixed counter for each: `tbtc_dkg`, `tbtc_wallet_coordination`, `tbtc_signing`, `tbtc_heartbeat`, `tbtc_inactivity_claim`, `beacon_dkg`, `beacon_relay_signing`, `beacon_relay_forwarding`, `beacon_timeout_report`. The ceremony is part of the metric name rather than a label, so the set is closed and a refusal is always attributable +*Labels*: None + +==== `performance_participation_commit_refusals_total` +*Type*: Counter +*Description*: Completion commit fences that failed, refusing to record a result under a format the chain anchor does not permit +*Labels*: None + +==== `performance_participation_clock_errors_total` +*Type*: Counter +*Description*: Failed chain-clock reads, from the block waiter, the supervisor poll, or a synchronous per-operation read +*Labels*: None + +==== `performance_participation_clock_aborts_total` +*Type*: Counter +*Description*: Held permits canceled because the chain clock became unusable +*Labels*: None + +==== `performance_participation_quiesce_total` +*Type*: Counter +*Description*: Transitions into process quiescence +*Labels*: None + +==== `performance_participation_quiesce_forced_aborts_total` +*Type*: Counter +*Description*: Permits still held when the shutdown deadline expired and were canceled by it +*Labels*: None + +==== `performance_participation_tbtc_quarantine_preservation_failures_total` +*Type*: Counter +*Description*: tBTC signer-output preservation episodes whose protected quarantine was still incomplete after the write-grace rounds. The counter increments while the process is alive and retrying (or on return when no live observer ran) and remains cumulative if a later retry makes the full output durable. Read it with `performance_participation_tbtc_quarantine_incomplete_outputs`: a nonzero counter with a zero live gauge is recovered history, not an output known to remain incomplete +*Labels*: None + +==== `performance_participation_beacon_quarantine_preservation_failures_total` +*Type*: Counter +*Description*: Beacon signer-output preservation episodes whose protected quarantine was still incomplete after the write-grace rounds. The counter increments while the process is alive and retrying (or on return when no live observer ran) and remains cumulative if a later retry makes the full output durable. Read it with `performance_participation_beacon_quarantine_incomplete_outputs`: a nonzero counter with a zero live gauge is recovered history, not an output known to remain incomplete +*Labels*: None + +==== `performance_participation_tbtc_quarantine_incomplete_outputs` +*Type*: Gauge +*Description*: tBTC signer outputs the running process is still holding after write-grace exhaustion while the protected namespace lacks either key material or the audit record explaining it. The gauge clears only when the complete output becomes durable. A nonzero or unreadable value refuses both single-release cutover and rollback acceptance +*Labels*: None + +==== `performance_participation_beacon_quarantine_incomplete_outputs` +*Type*: Gauge +*Description*: Beacon signer outputs the running process is still holding after write-grace exhaustion while the protected namespace lacks either key material or the audit record explaining it. The gauge clears only when the complete output becomes durable. A nonzero or unreadable value refuses both single-release cutover and rollback acceptance +*Labels*: None + +==== `performance_participation_quarantined_tbtc_signers` +*Type*: Gauge +*Description*: tBTC signer outputs held in the protected quarantine namespace that this process has not activated in its wallet cache — generated key material a gate refusal preserved, which a rollback still has to account for. Published by tBTC rather than by the gate, and recounted from the namespace, so it covers what earlier processes on the host preserved and not only this one's. It remains at its pre-registered zero on a process running no tBTC +*Labels*: None + +==== `performance_heartbeat_penalty_suppressed_total` +*Type*: Counter +*Description*: Inactivity penalties for a low-activity heartbeat that were suppressed because the ceremony ran the legacy format after the cutover block, or because the process was quiescing +*Labels*: None diff --git a/docs/resources/client-start-help b/docs/resources/client-start-help index 76fcba6f86..3a5220748a 100644 --- a/docs/resources/client-start-help +++ b/docs/resources/client-start-help @@ -18,13 +18,13 @@ Flags: --bitcoin.electrum.requestTimeout duration Timeout for a single attempt of Electrum protocol request. (default 30s) --bitcoin.electrum.requestRetryTimeout duration Timeout for Electrum protocol request retries. (default 2m0s) --bitcoin.electrum.keepAliveInterval duration Interval for connection keep alive requests. (default 5m0s) - --network.bootstrap Run the client in bootstrap mode. + --network.bootstrap [DEPRECATED: remove in v3.0] Run the client in bootstrap mode. This flag is deprecated and will be removed in v3.0. --network.peers strings Addresses of the network bootstrap nodes. -p, --network.port int Keep client listening port. (default 3919) --network.announcedAddresses strings Overwrites the default Keep client address announced in the network. Should be used for NAT or when more advanced firewall rules are applied. --network.disseminationTime int Specifies courtesy message dissemination time in seconds for topics the node is not subscribed to. Should be used only on selected bootstrap nodes. (0 = none) --storage.dir string Location to store the Keep client key shares and other sensitive data. - --clientInfo.port int Client Info HTTP server listening port. (default 9601) + --clientInfo.port int Client Info HTTP server listening port. Set to 0 to disable; expose only on a trusted network. (default 9601) --clientInfo.networkMetricsTick duration Client Info network metrics check tick in seconds. (default 1m0s) --clientInfo.ethereumMetricsTick duration Client info Ethereum metrics check tick in seconds. (default 10m0s) --tbtc.preParamsPoolSize int tECDSA pre-parameters pool size. (default 1000) @@ -32,6 +32,7 @@ Flags: --tbtc.preParamsGenerationDelay duration tECDSA pre-parameters generation delay. (default 10s) --tbtc.preParamsGenerationConcurrency int tECDSA pre-parameters generation concurrency. (default 1) --tbtc.keyGenerationConcurrency int tECDSA key generation concurrency. (default number of cores) + --protocolParticipation.cutoverBlock uint Protocol cutover block override for non-mainnet networks. Mainnet always uses the compiled release constant and rejects this setting; testnet requires a nonzero value; developer mode may use 0 to disable the cutover schedule. --developer.bridgeAddress string Address of the Bridge smart contract --developer.maintainerProxyAddress string Address of the MaintainerProxy smart contract --developer.lightRelayAddress string Address of the LightRelay smart contract diff --git a/docs/resources/docker-start-mainnet-sample b/docs/resources/docker-start-mainnet-sample index 3a428281eb..923b76a9c7 100644 --- a/docs/resources/docker-start-mainnet-sample +++ b/docs/resources/docker-start-mainnet-sample @@ -6,6 +6,12 @@ OPERATOR_KEY_FILE_PASSWORD="" CONFIG_DIR=$(pwd)/config STORAGE_DIR=$(pwd)/storage +# Only the public P2P port (3919) is published to the host below. The +# unauthenticated client-info server still listens inside the container on the +# compatibility default port 9601 (metrics/diagnostics) unless you set +# `--clientInfo.port 0`. Do NOT publish 9601 to 0.0.0.0; reach it only over a +# trusted path (a private Docker network, a firewall/VPN, or an authenticated +# mTLS reverse proxy). docker run --detach \ --volume $CONFIG_DIR:/mnt/keep/config \ --volume $STORAGE_DIR:/mnt/keep/storage \ @@ -14,7 +20,6 @@ docker run --detach \ --log-opt max-size=100m \ --log-opt max-file=3 \ -p 3919:3919 \ - -p 9601:9601 \ thresholdnetwork/keep-client:latest \ start \ --ethereum.url $ETHEREUM_WS_URL \ diff --git a/docs/resources/docker-start-testnet-sample b/docs/resources/docker-start-testnet-sample index f09029e473..e40cc0be12 100644 --- a/docs/resources/docker-start-testnet-sample +++ b/docs/resources/docker-start-testnet-sample @@ -6,6 +6,12 @@ OPERATOR_KEY_FILE_PASSWORD="" CONFIG_DIR=$(pwd)/config STORAGE_DIR=$(pwd)/storage +# Only the public P2P port (3919) is published to the host below. The +# unauthenticated client-info server still listens inside the container on the +# compatibility default port 9601 (metrics/diagnostics) unless you set +# `--clientInfo.port 0`. Do NOT publish 9601 to 0.0.0.0; reach it only over a +# trusted path (a private Docker network, a firewall/VPN, or an authenticated +# mTLS reverse proxy). docker run --detach \ --volume $CONFIG_DIR:/mnt/keep/config \ --volume $STORAGE_DIR:/mnt/keep/storage \ @@ -14,7 +20,6 @@ docker run --detach \ --log-opt max-size=100m \ --log-opt max-file=3 \ -p 3919:3919 \ - -p 9601:9601 \ us-docker.pkg.dev/keep-test-f3e0/public/keep-client:latest \ start \ --testnet \ diff --git a/docs/run-keep-node.adoc b/docs/run-keep-node.adoc index 85c5d26319..338532a82a 100644 --- a/docs/run-keep-node.adoc +++ b/docs/run-keep-node.adoc @@ -165,7 +165,9 @@ monitoring. A *Network* Port has to be exposed publicly, so the peers can connect to your node. // TODO: Add link to the Rewards Allocation documentation. -A *Diagnostics* Port has to be exposed publicly, for the Rewards Allocation. +A *Diagnostics* Port must be reachable from the Rewards Allocation prober via a +trusted network path; do not expose it publicly. See <> for the +trusted-network requirement and the temporary 9601 compatibility default. IMPORTANT: Please update your firewall rules if necessary. @@ -309,8 +311,20 @@ startup log. When sharing remember to substitute the `/ipv4/` address with the [#clientInfo] == Client Info -The client exposes metrics and diagnostics on a configurable port (default: `9601`) -under `/metrics` and `/diagnostics` resources. +The client exposes metrics and diagnostics on a configurable port under the +`/metrics` and `/diagnostics` resources. Note that `clientInfo.port` (the +non-public client-info port) is distinct from the public P2P `network.port` +(default `3919`): the client-info endpoint is unauthenticated and MUST be +reachable only over a trusted network path — a firewall/VPN or an authenticated +proxy in front of it. + +IMPORTANT: For the coordinated security release the client-info server listens +on the compatibility default port `9601`. This is a temporary compatibility +window: set `clientInfo.port = 0` to explicitly disable the server, or set an +explicit port to keep it. Commit an explicit `clientInfo.port` value now and +migrate every scrape target onto its trusted path; the follow-up R2 release +will sunset this window and flip the default back to `0` (disabled) once the +monitoring migration is complete. The data can be consumed by Prometheus to monitor the state of a node. @@ -323,15 +337,15 @@ The client exposes the following metrics: - connected bootstraps count, - Ethereum client connectivity status (if a simple read-only CALL can be executed). -Metrics are enabled once the client starts. It is possible to customize the port -at which metrics endpoint is exposed as well as the frequency with which -the metrics are collected. +Metrics are enabled once the client info endpoint is configured. It is possible +to customize the port at which metrics endpoint is exposed as well as the +frequency with which the metrics are collected. Exposed metrics contain the value and timestamp at which they were collected. Example metrics endpoint call result: ``` -$ curl localhost:9601/metrics +$ curl localhost:/metrics # TYPE connected_peers_count gauge connected_peers_count 108 1623235129569 @@ -350,12 +364,12 @@ The client exposes the following diagnostics: - list of connected peers along with their network id and Ethereum operator address, - information about the client's network id and Ethereum operator address. -Diagnostics are enabled once the client starts. It is possible to customize -the port at which diagnostics endpoint is exposed. +Diagnostics are enabled once the client info endpoint is configured. It is +possible to customize the port at which diagnostics endpoint is exposed. Example diagnostics endpoint call result: ``` -$ curl localhost:9601/diagnostics +$ curl localhost:/diagnostics { "client_info" { "ethereum_address":"0xDcd4199e22d09248cA2583cBDD2759b2acD22381", diff --git a/go.mod b/go.mod index 4ab16da8bb..2045975a46 100644 --- a/go.mod +++ b/go.mod @@ -1,21 +1,20 @@ module github.com/keep-network/keep-core -go 1.24.0 +go 1.25.7 -toolchain go1.24.1 +toolchain go1.25.10 replace ( - github.com/bnb-chain/tss-lib => github.com/threshold-network/tss-lib v0.0.0-20230901144531-2e712689cfbe + github.com/bnb-chain/tss-lib => github.com/threshold-network/tss-lib v0.0.0-20260729021955-d847ce003019 // btcd in version v.0.23 extracted `btcd/btcec` to a separate package `btcd/btcec/v2`. // Some of the dependencies still require the old version, which we workaround // here: github.com/btcsuite/btcd => github.com/btcsuite/btcd v0.22.3 github.com/btcsuite/btcd/v2 => github.com/btcsuite/btcd v0.23.4 github.com/checksum0/go-electrum => github.com/keep-network/go-electrum v0.0.0-20240206170935-6038cb594daa - github.com/keep-network/keep-common => github.com/threshold-network/keep-common v1.7.1-tlabs.0 - // Temporary replacement until v1.28.2 is released containing `protodelim` package. - // See https://github.com/protocolbuffers/protobuf-go/commit/fb0abd915897428ccfdd6b03b48ad8219751ee54 - google.golang.org/protobuf/dev => google.golang.org/protobuf v1.28.2-0.20220831092852-f930b1dc76e8 + // v1.7.1-tlabs.1 fixes the //go:linkname targets in the Ethereum codegen + // (bind -> abigen) so it links against go-ethereum v1.16+. + github.com/keep-network/keep-common => github.com/threshold-network/keep-common v1.7.1-tlabs.1 ) require ( @@ -26,8 +25,8 @@ require ( github.com/btcsuite/btcd/v2 v2.0.0-00010101000000-000000000000 github.com/btcsuite/btcutil v1.0.3-0.20201208143702-a53e38424cce github.com/checksum0/go-electrum v0.0.0-20220912200153-b862ac442cf9 - github.com/ethereum/go-ethereum v1.13.15 - github.com/ferranbt/fastssz v0.1.2 + github.com/ethereum/go-ethereum v1.17.3 + github.com/ferranbt/fastssz v0.1.4 github.com/go-test/deep v1.0.8 github.com/google/gofuzz v1.2.0 github.com/graph-gophers/graphql-go v1.3.0 @@ -35,32 +34,36 @@ require ( github.com/influxdata/influxdb-client-go/v2 v2.4.0 github.com/influxdata/influxdb1-client v0.0.0-20220302092344-a9ab5670611c github.com/ipfs/go-datastore v0.6.0 - github.com/ipfs/go-ipfs-config v0.0.4 github.com/ipfs/go-log v1.0.5 github.com/ipfs/go-log/v2 v2.5.1 github.com/jbenet/goprocess v0.1.4 github.com/keep-network/keep-common v1.7.1-0.20240424094333-bd36cd25bb74 - github.com/libp2p/go-addr-util v0.2.0 github.com/libp2p/go-libp2p v0.38.2 github.com/libp2p/go-libp2p-kad-dht v0.29.0 github.com/libp2p/go-libp2p-pubsub v0.13.0 github.com/mitchellh/mapstructure v1.5.0 github.com/multiformats/go-multiaddr v0.14.0 github.com/peterh/liner v1.1.1-0.20190123174540-a2c9a5303de7 - github.com/spf13/cobra v1.5.0 - github.com/spf13/pflag v1.0.5 + github.com/quasilyte/go-ruleguard/dsl v0.3.23 + github.com/spf13/cobra v1.8.1 + github.com/spf13/pflag v1.0.6 github.com/spf13/viper v1.12.0 + go.etcd.io/bbolt v1.3.11 go.uber.org/zap v1.27.0 - golang.org/x/crypto v0.32.0 + golang.org/x/crypto v0.47.0 golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 - golang.org/x/sync v0.10.0 - golang.org/x/term v0.28.0 - google.golang.org/protobuf v1.36.3 - google.golang.org/protobuf/dev v0.0.0-00010101000000-000000000000 + golang.org/x/sync v0.19.0 + golang.org/x/term v0.39.0 + google.golang.org/protobuf v1.36.11 + pgregory.net/rapid v1.3.0 ) require ( + github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6 // indirect + github.com/crate-crypto/go-eth-kzg v1.5.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect + github.com/emicklei/dot v1.6.2 // indirect + github.com/ethereum/c-kzg-4844/v2 v2.1.6 // indirect github.com/go-task/slim-sprig/v3 v3.0.0 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect github.com/pion/datachannel v1.5.10 // indirect @@ -83,32 +86,28 @@ require ( github.com/pmezard/go-difflib v1.0.0 // indirect github.com/stretchr/testify v1.11.1 // indirect github.com/wlynxg/anet v0.0.5 // indirect + golang.org/x/telemetry v0.0.0-20251203150158-8fff8a5912fc // indirect ) require ( - github.com/Microsoft/go-winio v0.6.1 // indirect + github.com/Microsoft/go-winio v0.6.2 // indirect github.com/StackExchange/wmi v1.2.1 // indirect github.com/aead/siphash v1.0.1 // indirect - github.com/agl/ed25519 v0.0.0-20170116200512-5312a6153412 // indirect github.com/benbjohnson/clock v1.3.5 // indirect github.com/beorn7/perks v1.0.1 // indirect - github.com/bits-and-blooms/bitset v1.10.0 // indirect + github.com/bits-and-blooms/bitset v1.20.0 // indirect github.com/btcsuite/btcd/btcutil v1.1.1 // indirect github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect - github.com/consensys/bavard v0.1.13 // indirect - github.com/consensys/gnark-crypto v0.12.1 // indirect + github.com/consensys/gnark-crypto v0.18.1 // indirect github.com/containerd/cgroups v1.1.0 // indirect github.com/coreos/go-systemd/v22 v22.5.0 // indirect - github.com/crate-crypto/go-kzg-4844 v0.7.0 // indirect github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c // indirect - github.com/deckarep/golang-set/v2 v2.1.0 // indirect - github.com/decred/dcrd/dcrec/edwards/v2 v2.0.0 // indirect + github.com/deckarep/golang-set/v2 v2.6.0 // indirect github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 // indirect github.com/deepmap/oapi-codegen v1.6.0 // indirect github.com/docker/go-units v0.5.0 // indirect github.com/elastic/gosigar v0.14.3 // indirect - github.com/ethereum/c-kzg-4844 v0.4.0 // indirect github.com/flynn/noise v1.1.0 // indirect github.com/francoispqt/gojay v1.2.13 // indirect github.com/fsnotify/fsnotify v1.6.0 // indirect @@ -125,13 +124,12 @@ require ( github.com/hashicorp/golang-lru v1.0.2 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect github.com/hashicorp/hcl v1.0.0 // indirect - github.com/holiman/uint256 v1.2.4 // indirect + github.com/holiman/uint256 v1.3.2 // indirect github.com/huin/goupnp v1.3.0 // indirect - github.com/inconshreveable/mousetrap v1.0.0 // indirect + github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/influxdata/line-protocol v0.0.0-20200327222509-2487e7298839 // indirect github.com/ipfs/boxo v0.27.2 // indirect github.com/ipfs/go-cid v0.5.0 // indirect - github.com/ipfs/go-ipfs-addr v0.0.1 // indirect github.com/ipld/go-ipld-prime v0.21.0 // indirect github.com/jackpal/go-nat-pmp v1.0.2 // indirect github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect @@ -143,9 +141,7 @@ require ( github.com/libp2p/go-cidranger v1.1.0 // indirect github.com/libp2p/go-flow-metrics v0.2.0 // indirect github.com/libp2p/go-libp2p-asn-util v0.4.1 // indirect - github.com/libp2p/go-libp2p-crypto v0.0.2 // indirect github.com/libp2p/go-libp2p-kbucket v0.6.4 // indirect - github.com/libp2p/go-libp2p-peer v0.1.1 // indirect github.com/libp2p/go-libp2p-record v0.3.1 // indirect github.com/libp2p/go-libp2p-routing-helpers v0.7.4 // indirect github.com/libp2p/go-msgio v0.3.0 // indirect @@ -161,8 +157,6 @@ require ( github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b // indirect github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc // indirect github.com/minio/sha256-simd v1.0.1 // indirect - github.com/mitchellh/go-homedir v1.1.0 // indirect - github.com/mmcloughlin/addchain v0.4.0 // indirect github.com/mr-tron/base58 v1.2.0 // indirect github.com/multiformats/go-base32 v0.1.0 // indirect github.com/multiformats/go-base36 v0.2.0 // indirect @@ -182,7 +176,7 @@ require ( github.com/pelletier/go-toml/v2 v2.0.9 // indirect github.com/pkg/errors v0.9.1 // indirect github.com/polydawn/refmt v0.89.0 // indirect - github.com/prometheus/client_golang v1.20.5 // indirect + github.com/prometheus/client_golang v1.20.5 github.com/prometheus/client_model v0.6.1 // indirect github.com/prometheus/common v0.62.0 // indirect github.com/prometheus/procfs v0.15.1 // indirect @@ -192,14 +186,12 @@ require ( github.com/raulk/go-watchdog v1.3.0 // indirect github.com/rivo/uniseg v0.2.0 // indirect github.com/shirou/gopsutil v3.21.4-0.20210419000835-c7a38de76ee5+incompatible - github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a // indirect - github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 // indirect github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/spf13/afero v1.8.2 // indirect github.com/spf13/cast v1.5.0 // indirect github.com/spf13/jwalterweatherman v1.1.0 // indirect github.com/subosito/gotenv v1.3.0 // indirect - github.com/supranational/blst v0.3.11 // indirect + github.com/supranational/blst v0.3.16 // indirect github.com/tklauser/go-sysconf v0.3.12 // indirect github.com/tklauser/numcpus v0.6.1 // indirect github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect @@ -212,16 +204,15 @@ require ( go.uber.org/fx v1.23.0 // indirect go.uber.org/mock v0.5.0 // indirect go.uber.org/multierr v1.11.0 // indirect - golang.org/x/mod v0.22.0 // indirect - golang.org/x/net v0.34.0 // indirect - golang.org/x/sys v0.29.0 // indirect - golang.org/x/text v0.21.0 // indirect - golang.org/x/time v0.5.0 - golang.org/x/tools v0.29.0 // indirect + golang.org/x/mod v0.31.0 // indirect + golang.org/x/net v0.49.0 // indirect + golang.org/x/sys v0.40.0 // indirect + golang.org/x/text v0.33.0 // indirect + golang.org/x/time v0.9.0 + golang.org/x/tools v0.40.0 // indirect gonum.org/v1/gonum v0.15.1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/yaml.v2 v2.4.0 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect lukechampine.com/blake3 v1.3.0 // indirect - rsc.io/tmplfunc v0.0.3 // indirect ) diff --git a/go.sum b/go.sum index 03b0d5a436..cff4d96e06 100644 --- a/go.sum +++ b/go.sum @@ -49,16 +49,16 @@ github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo= github.com/DataDog/zstd v1.4.5 h1:EndNeuB0l9syBZhut0wns3gV1hL8zX8LIu6ZiVHWLIQ= github.com/DataDog/zstd v1.4.5/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo= -github.com/Microsoft/go-winio v0.6.1 h1:9/kr64B9VUZrLm5YYwbGtUJnMgqWVOdUAXu6Migciow= -github.com/Microsoft/go-winio v0.6.1/go.mod h1:LRdKpFKfdobln8UmuiYcKPot9D2v6svN5+sAH+4kjUM= +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6 h1:1zYrtlhrZ6/b6SAjLSfKzWtdgqK0U+HtH/VcBWh1BaU= +github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime v0.0.0-20251001021608-1fe7b43fc4d6/go.mod h1:ioLG6R+5bUSO1oeGSDxOV3FADARuMoytZCSX6MEMQkI= github.com/StackExchange/wmi v1.2.1 h1:VIkavFPXSjcnS+O8yTq7NI32k0R5Aj+v39y29VYDOSA= github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8= -github.com/VictoriaMetrics/fastcache v1.12.1 h1:i0mICQuojGDL3KblA7wUNlY5lOK6a4bwt3uRKnkZU40= -github.com/VictoriaMetrics/fastcache v1.12.1/go.mod h1:tX04vaqcNoQeGLD+ra5pU5sWkuxnzWhEzLwhP9w653o= +github.com/VictoriaMetrics/fastcache v1.13.0 h1:AW4mheMR5Vd9FkAPUv+NH6Nhw+fmbTMGMsNAoA/+4G0= +github.com/VictoriaMetrics/fastcache v1.13.0/go.mod h1:hHXhl4DA2fTL2HTZDJFXWgW0LNjo6B+4aj2Wmng3TjU= github.com/aead/siphash v1.0.1 h1:FwHfE/T45KPKYuuSAKyyvE+oPWcaQ+CUmFW0bPlM+kg= github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII= -github.com/agl/ed25519 v0.0.0-20170116200512-5312a6153412 h1:w1UutsfOrms1J05zt7ISrnJIXKzwaspym5BTKGx93EI= -github.com/agl/ed25519 v0.0.0-20170116200512-5312a6153412/go.mod h1:WPjqKcmVOxf0XSf3YxCJs6N6AOSrOx3obionmG7T0y0= github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c= github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= github.com/benbjohnson/clock v1.3.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= @@ -67,8 +67,8 @@ github.com/benbjohnson/clock v1.3.5/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZx github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= -github.com/bits-and-blooms/bitset v1.10.0 h1:ePXTeiPEazB5+opbv5fr8umg2R/1NlzgDsyepwsSr88= -github.com/bits-and-blooms/bitset v1.10.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8= +github.com/bits-and-blooms/bitset v1.20.0 h1:2F+rfL86jE2d/bmw7OhqUg2Sj/1rURkBn3MdfoPyRVU= +github.com/bits-and-blooms/bitset v1.20.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8= github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g= github.com/btcsuite/btcd v0.22.3 h1:kYNaWFvOw6xvqP0vR20RP1Zq1DVMBxEO8QN5d1/EfNg= github.com/btcsuite/btcd v0.22.3/go.mod h1:wqgTSL29+50LRkmOVknEdmt8ZojIzhuWvgu/iptuN7Y= @@ -108,22 +108,20 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= -github.com/cockroachdb/errors v1.8.1 h1:A5+txlVZfOqFBDa4mGz2bUWSp0aHElvHX2bKkdbQu+Y= -github.com/cockroachdb/errors v1.8.1/go.mod h1:qGwQn6JmZ+oMjuLwjWzUNqblqk0xl4CVV3SQbGwK7Ac= -github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f h1:o/kfcElHqOiXqcou5a3rIlMc7oJbMQkeLk0VQJ7zgqY= -github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f/go.mod h1:i/u985jwjWRlyHXQbwatDASoW0RMlZ/3i9yJHE2xLkI= -github.com/cockroachdb/pebble v0.0.0-20230928194634-aa077af62593 h1:aPEJyR4rPBvDmeyi+l/FS/VtA00IWvjeFvjen1m1l1A= -github.com/cockroachdb/pebble v0.0.0-20230928194634-aa077af62593/go.mod h1:6hk1eMY/u5t+Cf18q5lFMUA1Rc+Sm5I6Ra1QuPyxXCo= -github.com/cockroachdb/redact v1.0.8 h1:8QG/764wK+vmEYoOlfobpe12EQcS81ukx/a4hdVMxNw= -github.com/cockroachdb/redact v1.0.8/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg= -github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 h1:IKgmqgMQlVJIZj19CdocBeSfSaiCbEBZGKODaixqtHM= -github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2/go.mod h1:8BT+cPK6xvFOcRlk0R8eg+OTkcqI6baNH4xAkpiYVvQ= +github.com/cockroachdb/errors v1.11.3 h1:5bA+k2Y6r+oz/6Z/RFlNeVCesGARKuC6YymtcDrbC/I= +github.com/cockroachdb/errors v1.11.3/go.mod h1:m4UIW4CDjx+R5cybPsNrRbreomiFqt8o1h1wUVazSd8= +github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce h1:giXvy4KSc/6g/esnpM7Geqxka4WSqI1SZc7sMJFd3y4= +github.com/cockroachdb/fifo v0.0.0-20240606204812-0bbfbd93a7ce/go.mod h1:9/y3cnZ5GKakj/H4y9r9GTjCvAFta7KLgSHPJJYc52M= +github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b h1:r6VH0faHjZeQy818SGhaone5OnYfxFR/+AzdY3sf5aE= +github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b/go.mod h1:Vz9DsVWQQhf3vs21MhPMZpMGSht7O/2vFW2xusFUVOs= +github.com/cockroachdb/pebble v1.1.5 h1:5AAWCBWbat0uE0blr8qzufZP5tBjkRyy/jWe1QWLnvw= +github.com/cockroachdb/pebble v1.1.5/go.mod h1:17wO9el1YEigxkP/YtV8NtCivQDgoCyBg5c4VR/eOWo= +github.com/cockroachdb/redact v1.1.5 h1:u1PMllDkdFfPWaNGMyLD1+so+aq3uUItthCFqzwPJ30= +github.com/cockroachdb/redact v1.1.5/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg= github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 h1:zuQyyAKVxetITBuuhv3BI9cMrmStnpT18zmgmTxunpo= github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06/go.mod h1:7nc4anLGjupUW/PeY5qiNYsdNXj7zopG+eqsS7To5IQ= -github.com/consensys/bavard v0.1.13 h1:oLhMLOFGTLdlda/kma4VOJazblc7IM5y5QPd2A/YjhQ= -github.com/consensys/bavard v0.1.13/go.mod h1:9ItSMtA/dXMAiL7BG6bqW2m3NdSEObYWoH223nGHukI= -github.com/consensys/gnark-crypto v0.12.1 h1:lHH39WuuFgVHONRl3J0LRBtuYdQTumFSDtJF7HpyG8M= -github.com/consensys/gnark-crypto v0.12.1/go.mod h1:v2Gy7L/4ZRosZ7Ivs+9SfUDr0f5UlG+EM5t7MPHiLuY= +github.com/consensys/gnark-crypto v0.18.1 h1:RyLV6UhPRoYYzaFnPQA4qK3DyuDgkTgskDdoGqFt3fI= +github.com/consensys/gnark-crypto v0.18.1/go.mod h1:L3mXGFTe1ZN+RSJ+CLjUt9x7PNdx8ubaYfDROyp2Z8c= github.com/containerd/cgroups v0.0.0-20201119153540-4cbc285b3327/go.mod h1:ZJeTFisyysqgcCdecO57Dj79RfL0LNeGiFUqLYQRYLE= github.com/containerd/cgroups v1.1.0 h1:v8rEWFl6EoqHB+swVNjVoCJE8o3jX7e8nqBGPLaDFBM= github.com/containerd/cgroups v1.1.0/go.mod h1:6ppBcbh/NOOUU+dMKrykgaBnK9lCIBxHqJDGwsa1mIw= @@ -133,25 +131,24 @@ github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8 github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= -github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w= -github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= -github.com/crate-crypto/go-ipa v0.0.0-20231025140028-3c0104f4b233 h1:d28BXYi+wUpz1KBmiF9bWrjEMacUEREV6MBi2ODnrfQ= -github.com/crate-crypto/go-ipa v0.0.0-20231025140028-3c0104f4b233/go.mod h1:geZJZH3SzKCqnz5VT0q/DyIG/tvu/dZk+VIfXicupJs= -github.com/crate-crypto/go-kzg-4844 v0.7.0 h1:C0vgZRk4q4EZ/JgPfzuSoxdCq3C3mOZMBShovmncxvA= -github.com/crate-crypto/go-kzg-4844 v0.7.0/go.mod h1:1kMhvPgI0Ky3yIa+9lFySEBUBXkYxeOi8ZF1sYioxhc= +github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/cpuguy83/go-md2man/v2 v2.0.5 h1:ZtcqGrnekaHpVLArFSe4HK5DoKx1T0rq2DwVB0alcyc= +github.com/cpuguy83/go-md2man/v2 v2.0.5/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= +github.com/crate-crypto/go-eth-kzg v1.5.0 h1:FYRiJMJG2iv+2Dy3fi14SVGjcPteZ5HAAUe4YWlJygc= +github.com/crate-crypto/go-eth-kzg v1.5.0/go.mod h1:J9/u5sWfznSObptgfa92Jq8rTswn6ahQWEuiLHOjCUI= github.com/cyberdelia/templates v0.0.0-20141128023046-ca7fffd4298c/go.mod h1:GyV+0YP4qX0UQ7r2MoYZ+AvYDp12OF5yg4q8rGnyNh4= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c h1:pFUpOrbxDR6AkioZ1ySsx5yxlDQZ8stG2b88gTPxgJU= github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c/go.mod h1:6UhI8N9EjYm1c2odKpFpAYeR8dsBeM7PtzQhRgxRr9U= -github.com/deckarep/golang-set/v2 v2.1.0 h1:g47V4Or+DUdzbs8FxCCmgb6VYd+ptPAngjM6dtGktsI= -github.com/deckarep/golang-set/v2 v2.1.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4= +github.com/dchest/siphash v1.2.3 h1:QXwFc8cFOR2dSa/gE6o/HokBMWtLUaNDVd+22aKHeEA= +github.com/dchest/siphash v1.2.3/go.mod h1:0NvQU092bT0ipiFN++/rXm69QG9tVxLAlQHIXMPAkHc= +github.com/deckarep/golang-set/v2 v2.6.0 h1:XfcQbWM1LlMB8BsJ8N9vW5ehnnPVIw0je80NsVHagjM= +github.com/deckarep/golang-set/v2 v2.6.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4= github.com/decred/dcrd/crypto/blake256 v1.0.0/go.mod h1:sQl2p6Y26YV+ZOcSTP6thNdn47hh8kt6rqSlvmrXFAc= github.com/decred/dcrd/crypto/blake256 v1.0.1 h1:7PltbUIQB7u/FfZ39+DGa/ShuMyJ5ilcvdfma9wOH6Y= github.com/decred/dcrd/crypto/blake256 v1.0.1/go.mod h1:2OfgNZ5wDpcsFmHmCK5gZTPcCXqlm2ArzUIkw9czNJo= -github.com/decred/dcrd/dcrec/edwards/v2 v2.0.0 h1:E5KszxGgpjpmW8vN811G6rBAZg0/S/DftdGqN4FW5x4= -github.com/decred/dcrd/dcrec/edwards/v2 v2.0.0/go.mod h1:d0H8xGMWbiIQP7gN3v2rByWUcuZPm9YsgmnfoxgbINc= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1/go.mod h1:hyedUtir6IdtD/7lIxGeCxkaw7y45JueMRL4DIyJDKs= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 h1:rpfIENRNNilwHwZeG5+P150SMrnNEcHYvcCuK6dPZSg= github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0/go.mod h1:v57UDF4pDQJcEfFUCRop3lJL149eHGSe9Jvczhzjo/0= @@ -166,21 +163,22 @@ github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25Kn github.com/elastic/gosigar v0.12.0/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs= github.com/elastic/gosigar v0.14.3 h1:xwkKwPia+hSfg9GqrCUKYdId102m9qTJIIr7egmK/uo= github.com/elastic/gosigar v0.14.3/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs= +github.com/emicklei/dot v1.6.2 h1:08GN+DD79cy/tzN6uLCT84+2Wk9u+wvqP+Hkx/dIR8A= +github.com/emicklei/dot v1.6.2/go.mod h1:DeV7GvQtIw4h2u73RKBkkFdvVAz0D9fzeJrgPW6gy/s= github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po= github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= -github.com/ethereum/c-kzg-4844 v0.4.0 h1:3MS1s4JtA868KpJxroZoepdV0ZKBp3u/O5HcZ7R3nlY= -github.com/ethereum/c-kzg-4844 v0.4.0/go.mod h1:VewdlzQmpT5QSrVhbBuGoCdFJkpaJlO1aQputP83wc0= -github.com/ethereum/go-ethereum v1.13.15 h1:U7sSGYGo4SPjP6iNIifNoyIAiNjrmQkz6EwQG+/EZWo= -github.com/ethereum/go-ethereum v1.13.15/go.mod h1:TN8ZiHrdJwSe8Cb6x+p0hs5CxhJZPbqB7hHkaUXcmIU= -github.com/facebookgo/atomicfile v0.0.0-20151019160806-2de1f203e7d5/go.mod h1:JpoxHjuQauoxiFMl1ie8Xc/7TfLuMZ5eOCONd1sUBHg= -github.com/ferranbt/fastssz v0.1.2 h1:Dky6dXlngF6Qjc+EfDipAkE83N5I5DE68bY6O0VLNPk= -github.com/ferranbt/fastssz v0.1.2/go.mod h1:X5UPrE2u1UJjxHA8X54u04SBwdAQjG2sFtWs39YxyWs= -github.com/fjl/memsize v0.0.2 h1:27txuSD9or+NZlnOWdKUxeBzTAUkWCVh+4Gf2dWFOzA= -github.com/fjl/memsize v0.0.2/go.mod h1:VvhXpOYNQvB+uIk2RvXzuaQtkQJzzIx6lSBe1xv7hi0= +github.com/ethereum/c-kzg-4844/v2 v2.1.6 h1:xQymkKCT5E2Jiaoqf3v4wsNgjZLY0lRSkZn27fRjSls= +github.com/ethereum/c-kzg-4844/v2 v2.1.6/go.mod h1:8HMkUZ5JRv4hpw/XUrYWSQNAUzhHMg2UDb/U+5m+XNw= +github.com/ethereum/go-bigmodexpfix v0.0.0-20250911101455-f9e208c548ab h1:rvv6MJhy07IMfEKuARQ9TKojGqLVNxQajaXEp/BoqSk= +github.com/ethereum/go-bigmodexpfix v0.0.0-20250911101455-f9e208c548ab/go.mod h1:IuLm4IsPipXKF7CW5Lzf68PIbZ5yl7FFd74l/E0o9A8= +github.com/ethereum/go-ethereum v1.17.3 h1:Ev/sQHH+UdKZHWjuVzhu2pxhi/sXaPZl23Q+Q5LDd4Q= +github.com/ethereum/go-ethereum v1.17.3/go.mod h1:f2EhRwqewIZkGoQekywI2Y2RZAMTSavLNkD9qItFy1A= +github.com/ferranbt/fastssz v0.1.4 h1:OCDB+dYDEQDvAgtAGnTSidK1Pe2tW3nFV40XyMkTeDY= +github.com/ferranbt/fastssz v0.1.4/go.mod h1:Ea3+oeoRGGLGm5shYAeDgu6PGUlcvQhE2fILyD9+tGg= github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc= github.com/flynn/noise v1.1.0 h1:KjPQoQCEFdZDiP03phOvGi11+SVVhBG2wOWAorLsstg= github.com/flynn/noise v1.1.0/go.mod h1:xbMo+0i6+IGbYdJhF31t2eR1BIU0CYc12+BNAKwUTag= @@ -193,9 +191,9 @@ github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4 github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw= github.com/gballet/go-libpcsclite v0.0.0-20191108122812-4678299bea08 h1:f6D9Hr8xV8uYKlyuj8XIruxlh9WjVjdh1gIicAS7ays= github.com/gballet/go-libpcsclite v0.0.0-20191108122812-4678299bea08/go.mod h1:x7DCsMOv1taUwEWCzT4cmDeAkigA5/QCwUodaVOe8Ww= -github.com/gballet/go-verkle v0.1.1-0.20231031103413-a67434b50f46 h1:BAIP2GihuqhwdILrV+7GJel5lyPV3u1+PgzrWLc0TkE= -github.com/gballet/go-verkle v0.1.1-0.20231031103413-a67434b50f46/go.mod h1:QNpY22eby74jVhqH4WhDLDwxc/vqsern6pW+u2kbkpc= github.com/getkin/kin-openapi v0.53.0/go.mod h1:7Yn5whZr5kJi6t+kShccXS8ae1APpYTW6yheSwk8Yi4= +github.com/getsentry/sentry-go v0.27.0 h1:Pv98CIbtB3LkMWmXi4Joa5OOcwbmnX88sF5qbK3r3Ps= +github.com/getsentry/sentry-go v0.27.0/go.mod h1:lc76E2QywIyW8WuBnwl8Lc4bkmQH4+w1gwTf25trprY= github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0= github.com/go-chi/chi/v5 v5.0.0/go.mod h1:BBug9lr0cqtdAhsu6R4AAdvufI0/XBzAQSsUqJpoZOs= @@ -222,15 +220,15 @@ github.com/godbus/dbus/v5 v5.0.3/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5x github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= github.com/godbus/dbus/v5 v5.1.0 h1:4KLkAxT3aOY8Li4FRJe/KvhoNFFxo0m6fNuFUO8QJUk= github.com/godbus/dbus/v5 v5.1.0/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= -github.com/gofrs/flock v0.8.1 h1:+gYjHKf32LDeiEEFhQaotPbLuUXjY5ZqxKgXy7n59aw= -github.com/gofrs/flock v0.8.1/go.mod h1:F1TvTiK9OcQqauNUHlbJvyl9Qa1QvF/gOUDKA14jxHU= +github.com/gofrs/flock v0.12.1 h1:MTLVXXHf8ekldpJk3AKicLij9MdwOWkZ+a/jHHZby9E= +github.com/gofrs/flock v0.12.1/go.mod h1:9zxTsyu5xtJ9DK+1tFZyibEV7y3uwDxPPfbxeeHCoD0= github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4= github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg= -github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= +github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI= +github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= @@ -260,8 +258,8 @@ github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QD github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb h1:PBC98N2aIaM3XXiurYmW7fx4GZkL8feAMVq7nEjURHk= -github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs= +github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golangci/lint-1 v0.0.0-20181222135242-d2cdd8c08219/go.mod h1:/X8TswGSh1pIozq4ZwCfxS0WA5JGXguxk94ar/4c87Y= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= @@ -300,7 +298,6 @@ github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad h1:a6HEuzUHeKH6hwfN/ZoQgRgVIWFJljSWa/zetS2WTvg= github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= -github.com/google/subcommands v1.2.0/go.mod h1:ZjhPrFU+Olkh9WazFPsl27BQ4UPiG37m3yTrtFlrHVk= github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= @@ -316,12 +313,14 @@ github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c/go.mod h1:wJfORR github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So= github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/grafana/pyroscope-go v1.2.7 h1:VWBBlqxjyR0Cwk2W6UrE8CdcdD80GOFNutj0Kb1T8ac= +github.com/grafana/pyroscope-go v1.2.7/go.mod h1:o/bpSLiJYYP6HQtvcoVKiE9s5RiNgjYTj1DhiddP2Pc= +github.com/grafana/pyroscope-go/godeltaprof v0.1.9 h1:c1Us8i6eSmkW+Ez05d3co8kasnuOY813tbMN8i/a3Og= +github.com/grafana/pyroscope-go/godeltaprof v0.1.9/go.mod h1:2+l7K7twW49Ct4wFluZD3tZ6e0SjanjcUUBPVD/UuGU= github.com/graph-gophers/graphql-go v1.3.0 h1:Eb9x/q6MFpCLz7jBCiP/WTxjSDrYLR1QY41SORZyNJ0= github.com/graph-gophers/graphql-go v1.3.0/go.mod h1:9CQHMSxwO4MprSdzoIEobiHpoLtHm77vfxsvsIN5Vuc= github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= github.com/grpc-ecosystem/grpc-gateway v1.5.0/go.mod h1:RSKVYQBd5MCa4OVpNdGskqpgL2+G+NZTnrVHpWWfpdw= -github.com/gxed/hashland/keccakpg v0.0.1/go.mod h1:kRzw3HkwxFU1mpmPP8v1WyQzwdGfmKFJ6tItnhQ67kU= -github.com/gxed/hashland/murmur3 v0.0.1/go.mod h1:KjXop02n4/ckmZSnY2+HKcLud/tcmvhST0bie/0lS48= github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= @@ -338,18 +337,18 @@ github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= -github.com/holiman/billy v0.0.0-20240216141850-2abb0c79d3c4 h1:X4egAf/gcS1zATw6wn4Ej8vjuVGxeHdan+bRb2ebyv4= -github.com/holiman/billy v0.0.0-20240216141850-2abb0c79d3c4/go.mod h1:5GuXa7vkL8u9FkFuWdVvfR5ix8hRB7DbOAaYULamFpc= +github.com/holiman/billy v0.0.0-20250707135307-f2f9b9aae7db h1:IZUYC/xb3giYwBLMnr8d0TGTzPKFGNTCGgGLoyeX330= +github.com/holiman/billy v0.0.0-20250707135307-f2f9b9aae7db/go.mod h1:xTEYN9KCHxuYHs+NmrmzFcnvHMzLLNiGFafCb1n3Mfg= github.com/holiman/bloomfilter/v2 v2.0.3 h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao= github.com/holiman/bloomfilter/v2 v2.0.3/go.mod h1:zpoh+gs7qcpqrHr3dB55AMiJwo0iURXE7ZOP9L9hSkA= -github.com/holiman/uint256 v1.2.4 h1:jUc4Nk8fm9jZabQuqr2JzednajVmBpC+oiTiXZJEApU= -github.com/holiman/uint256 v1.2.4/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E= +github.com/holiman/uint256 v1.3.2 h1:a9EgMPSC1AAaj1SZL5zIQD3WbwTuHrMGOerLjGmM/TA= +github.com/holiman/uint256 v1.3.2/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E= github.com/huin/goupnp v1.3.0 h1:UvLUlWDNpoUdYzb2TCn+MuTWtcjXKSza2n6CBdQ0xXc= github.com/huin/goupnp v1.3.0/go.mod h1:gnGPsThkYa7bFi/KWmEysQRf48l2dvR5bxr2OFckNX8= github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc= -github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= -github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= +github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/influxdata/influxdb-client-go/v2 v2.4.0 h1:HGBfZYStlx3Kqvsv1h2pJixbCl/jhnFtxpKFAv9Tu5k= github.com/influxdata/influxdb-client-go/v2 v2.4.0/go.mod h1:vLNHdxTJkIf2mSLvGrpj8TCcISApPoXkaxP8g9uRlW8= github.com/influxdata/influxdb1-client v0.0.0-20220302092344-a9ab5670611c h1:qSHzRbhzK8RdXOsAdfDgO49TtqC1oZ+acxPrkfTxcCs= @@ -360,25 +359,18 @@ github.com/ipfs/boxo v0.27.2 h1:sGo4KdwBaMjdBjH08lqPJyt27Z4CO6sugne3ryX513s= github.com/ipfs/boxo v0.27.2/go.mod h1:qEIRrGNr0bitDedTCzyzBHxzNWqYmyuHgK8LG9Q83EM= github.com/ipfs/go-block-format v0.2.0 h1:ZqrkxBA2ICbDRbK8KJs/u0O3dlp6gmAuuXUJNiW1Ycs= github.com/ipfs/go-block-format v0.2.0/go.mod h1:+jpL11nFx5A/SPpsoBn6Bzkra/zaArfSmsknbPMYgzM= -github.com/ipfs/go-cid v0.0.7/go.mod h1:6Ux9z5e+HpkQdckYoX1PG/6xqKspzlEIR5SDmgqgC/I= github.com/ipfs/go-cid v0.5.0 h1:goEKKhaGm0ul11IHA7I6p1GmKz8kEYniqFopaB5Otwg= github.com/ipfs/go-cid v0.5.0/go.mod h1:0L7vmeNXpQpUS9vt+yEARkJ8rOg43DF3iPgn4GIN0mk= github.com/ipfs/go-datastore v0.6.0 h1:JKyz+Gvz1QEZw0LsX1IBn+JFCJQH4SJVFtM4uWU0Myk= github.com/ipfs/go-datastore v0.6.0/go.mod h1:rt5M3nNbSO/8q1t4LNkLyUwRs8HupMeN/8O4Vn9YAT8= github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= -github.com/ipfs/go-ipfs-addr v0.0.1 h1:DpDFybnho9v3/a1dzJ5KnWdThWD1HrFLpQ+tWIyBaFI= -github.com/ipfs/go-ipfs-addr v0.0.1/go.mod h1:uKTDljHT3Q3SUWzDLp3aYUi8MrY32fgNgogsIa0npjg= -github.com/ipfs/go-ipfs-config v0.0.4 h1:zOWk1gGvIOptjHvvu0qSC8psB2IBKO/FbQArFnmm0LM= -github.com/ipfs/go-ipfs-config v0.0.4/go.mod h1:KDbHjNyg4e6LLQSQpkgQMBz6Jf4LXiWAcmnkcwmH0DU= -github.com/ipfs/go-ipfs-util v0.0.1/go.mod h1:spsl5z8KUnrve+73pOhSVZND1SIxPW5RyBCNzQxlJBc= github.com/ipfs/go-ipfs-util v0.0.3 h1:2RFdGez6bu2ZlZdI+rWfIdbQb1KudQp3VGwPtdNCmE0= github.com/ipfs/go-ipfs-util v0.0.3/go.mod h1:LHzG1a0Ig4G+iZ26UUOMjHd+lfM84LZCrn17xAKWBvs= github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM= github.com/ipfs/go-log v1.0.5 h1:2dOuUCB1Z7uoczMWgAyDck5JLb72zHzrMnGnCNNbvY8= github.com/ipfs/go-log v1.0.5/go.mod h1:j0b8ZoR+7+R99LD9jZ6+AJsrzkPbSXbZfGakb5JPtIo= github.com/ipfs/go-log/v2 v2.1.3/go.mod h1:/8d0SH3Su5Ooc31QlL1WysJhvyOTDCjcCZ9Axpmri6g= -github.com/ipfs/go-log/v2 v2.4.0/go.mod h1:nPZnh7Cj7lwS3LpRU5Mwr2ol1c2gXIEXuF6aywqrtmo= github.com/ipfs/go-log/v2 v2.5.1 h1:1XdUzF7048prq4aBjDQQ4SL5RxftpRGdXhNRwKSAlcY= github.com/ipfs/go-log/v2 v2.5.1/go.mod h1:prSpmC1Gpllc9UYWxDiZDreBYw7zp4Iqp1kOLU9U5UI= github.com/ipfs/go-test v0.0.4 h1:DKT66T6GBB6PsDFLoO56QZPrOmzJkqU1FZH5C9ySkew= @@ -429,10 +421,8 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/labstack/echo/v4 v4.2.1/go.mod h1:AA49e0DZ8kk5jTOOCKNuPR6oTnBS0dYiM4FW1e6jwpg= github.com/labstack/gommon v0.3.0/go.mod h1:MULnywXg0yavhxWKc+lOruYdAhDwPK9wf0OL7NoOu+k= -github.com/leanovate/gopter v0.2.9 h1:fQjYxZaynp97ozCzfOyOuAGOU4aU/z37zf/tOujFk7c= -github.com/leanovate/gopter v0.2.9/go.mod h1:U2L/78B+KVFIx2VmW6onHJQzXtFb+p5y3y2Sh+Jxxv8= -github.com/libp2p/go-addr-util v0.2.0 h1:nwPtbrJEujbrmQm7tMxjsFY+PjZ0YWFeb9jVdpjjiuc= -github.com/libp2p/go-addr-util v0.2.0/go.mod h1:lsJiu306BQNNAUWgzNiwNGFunP1/swKhRvTLzpPveD0= +github.com/leanovate/gopter v0.2.11 h1:vRjThO1EKPb/1NsDXuDrzldR28RLkBflWYcU9CvzWu4= +github.com/leanovate/gopter v0.2.11/go.mod h1:aK3tzZP/C+p1m3SPRE4SYZFGP7jjkuSI4f7Xvpt0S9c= github.com/libp2p/go-buffer-pool v0.1.0 h1:oK4mSFcQz7cTQIfqbe4MIj9gLW+mnanjyFtc6cdF0Y8= github.com/libp2p/go-buffer-pool v0.1.0/go.mod h1:N+vh8gMqimBzdKkSMVuydVDq+UV5QTWy5HSiZacSbPg= github.com/libp2p/go-cidranger v1.1.0 h1:ewPN8EZ0dd1LSnrtuwd4709PXVcITVeuwbag38yPW7c= @@ -443,16 +433,10 @@ github.com/libp2p/go-libp2p v0.38.2 h1:9SZQDOCi82A25An4kx30lEtr6kGTxrtoaDkbs5xrK github.com/libp2p/go-libp2p v0.38.2/go.mod h1:QWV4zGL3O9nXKdHirIC59DoRcZ446dfkjbOJ55NEWFo= github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl950SO9L6n94= github.com/libp2p/go-libp2p-asn-util v0.4.1/go.mod h1:d/NI6XZ9qxw67b4e+NgpQexCIiFYJjErASrYW4PFDN8= -github.com/libp2p/go-libp2p-crypto v0.0.1/go.mod h1:yJkNyDmO341d5wwXxDUGO0LykUVT72ImHNUqh5D/dBE= -github.com/libp2p/go-libp2p-crypto v0.0.2 h1:TTdJ4y6Uoa6NxQcuEaVkQfFRcQeCE2ReDk8Ok4I0Fyw= -github.com/libp2p/go-libp2p-crypto v0.0.2/go.mod h1:eETI5OUfBnvARGOHrJz2eWNyTUxEGZnBxMcbUjfIj4I= github.com/libp2p/go-libp2p-kad-dht v0.29.0 h1:045eW21lGlMSD9aKSZZGH4fnBMIInPwQLxIQ35P962I= github.com/libp2p/go-libp2p-kad-dht v0.29.0/go.mod h1:mIci3rHSwDsxQWcCjfmxD8vMTgh5xLuvwb1D5WP8ZNk= github.com/libp2p/go-libp2p-kbucket v0.6.4 h1:OjfiYxU42TKQSB8t8WYd8MKhYhMJeO2If+NiuKfb6iQ= github.com/libp2p/go-libp2p-kbucket v0.6.4/go.mod h1:jp6w82sczYaBsAypt5ayACcRJi0lgsba7o4TzJKEfWA= -github.com/libp2p/go-libp2p-peer v0.0.1/go.mod h1:nXQvOBbwVqoP+T5Y5nCjeH4sP9IX/J0AMzcDUVruVoo= -github.com/libp2p/go-libp2p-peer v0.1.1 h1:qGCWD1a+PyZcna6htMPo26jAtqirVnJ5NvBQIKV7rRY= -github.com/libp2p/go-libp2p-peer v0.1.1/go.mod h1:jkF12jGB4Gk/IOo+yomm+7oLWxF278F7UnrYUQ1Q8es= github.com/libp2p/go-libp2p-pubsub v0.13.0 h1:RmFQ2XAy3zQtbt2iNPy7Tt0/3fwTnHpCQSSnmGnt1Ps= github.com/libp2p/go-libp2p-pubsub v0.13.0/go.mod h1:m0gpUOyrXKXdE7c8FNQ9/HLfWbxaEw7xku45w+PaqZo= github.com/libp2p/go-libp2p-record v0.3.1 h1:cly48Xi5GjNw5Wq+7gmjfBiG9HCzQVkiZOUZ8kUl+Fg= @@ -461,7 +445,6 @@ github.com/libp2p/go-libp2p-routing-helpers v0.7.4 h1:6LqS1Bzn5CfDJ4tzvP9uwh42IB github.com/libp2p/go-libp2p-routing-helpers v0.7.4/go.mod h1:we5WDj9tbolBXOuF1hGOkR+r7Uh1408tQbAKaT5n1LE= github.com/libp2p/go-libp2p-testing v0.12.0 h1:EPvBb4kKMWO29qP4mZGyhVzUyR25dvfUIK5WDu6iPUA= github.com/libp2p/go-libp2p-testing v0.12.0/go.mod h1:KcGDRXyN7sQCllucn1cOOS+Dmm7ujhfEyXQL5lvkcPg= -github.com/libp2p/go-maddr-filter v0.1.0/go.mod h1:VzZhTXkMucEGGEOSKddrwGiOv0tUhgnKqNEmIAz/bPU= github.com/libp2p/go-msgio v0.3.0 h1:mf3Z8B1xcFN314sWX+2vOTShIE0Mmn2TXn3YCUQGNj0= github.com/libp2p/go-msgio v0.3.0/go.mod h1:nyRM819GmVaF9LX3l03RMh10QdOroF++NBbxAb0mmDM= github.com/libp2p/go-nat v0.2.0 h1:Tyz+bUFAYqGyJ/ppPPymMGbIgNRH+WqC5QrT5fKrrGk= @@ -508,68 +491,44 @@ github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b/go.mod h1:lxPUiZwKo github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc h1:PTfri+PuQmWDqERdnNMiD9ZejrlswWrCpBEZgWOiTrc= github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc/go.mod h1:cGKTAVKx4SxOuR/czcZ/E2RSJ3sfHs8FpHhQ5CWMf9s= github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ= -github.com/minio/sha256-simd v0.0.0-20190131020904-2d45a736cd16/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= -github.com/minio/sha256-simd v0.0.0-20190328051042-05b4dd3047e5/go.mod h1:2FMWW+8GMoPweT6+pI63m9YE3Lmw4J71hV56Chs1E/U= github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM= github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8= -github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y= -github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY= github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= github.com/mitchellh/pointerstructure v1.2.0 h1:O+i9nHnXS3l/9Wu7r4NrEdwA2VFTicjUEN1uBnDo34A= github.com/mitchellh/pointerstructure v1.2.0/go.mod h1:BRAsLI5zgXmw97Lf6s25bs8ohIXc3tViBH44KcwB2g4= -github.com/mmcloughlin/addchain v0.4.0 h1:SobOdjm2xLj1KkXN5/n0xTIWyZA2+s99UCY1iPfkHRY= -github.com/mmcloughlin/addchain v0.4.0/go.mod h1:A86O+tHqZLMNO4w6ZZ4FlVQEadcoqkyU72HC5wJ4RlU= -github.com/mmcloughlin/profile v0.1.1/go.mod h1:IhHD7q1ooxgwTgjxQYkACGA77oFTDdFVejUS1/tS/qU= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= -github.com/mr-tron/base58 v1.1.0/go.mod h1:xcD2VGqlgYjBdcBLw+TuYLr8afG+Hj8g2eTVqeSzSU8= github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= -github.com/mr-tron/base58 v1.1.3/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o= github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= -github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA= github.com/multiformats/go-base32 v0.1.0 h1:pVx9xoSPqEIQG8o+UbAe7DNi51oej1NtK+aGkbLYxPE= github.com/multiformats/go-base32 v0.1.0/go.mod h1:Kj3tFY6zNr+ABYMqeUNeGvkIC/UYgtWibDcT0rExnbI= -github.com/multiformats/go-base36 v0.1.0/go.mod h1:kFGE83c6s80PklsHO9sRn2NCoffoRdUUOENyW/Vv6sM= github.com/multiformats/go-base36 v0.2.0 h1:lFsAbNOGeKtuKozrtBsAkSVhv1p9D0/qedU9rQyccr0= github.com/multiformats/go-base36 v0.2.0/go.mod h1:qvnKE++v+2MWCfePClUEjE78Z7P2a1UV0xHgWc0hkp4= -github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44= github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo= -github.com/multiformats/go-multiaddr v0.2.2/go.mod h1:NtfXiOtHvghW9KojvtySjH5y0u0xW5UouOmQQrn6a3Y= -github.com/multiformats/go-multiaddr v0.3.3/go.mod h1:lCKNGP1EQ1eZ35Za2wlqnabm9xQkib3fyB+nZXHLag0= github.com/multiformats/go-multiaddr v0.14.0 h1:bfrHrJhrRuh/NXH5mCnemjpbGjzRw/b+tJFOD41g2tU= github.com/multiformats/go-multiaddr v0.14.0/go.mod h1:6EkVAxtznq2yC3QT5CM1UTAwG0GTP3EWAIcjHuzQ+r4= -github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/9403szc57zJPs45fmnznu3Q= github.com/multiformats/go-multiaddr-dns v0.4.1 h1:whi/uCLbDS3mSEUMb1MsoT4uzUeZB0N32yzufqS0i5M= github.com/multiformats/go-multiaddr-dns v0.4.1/go.mod h1:7hfthtB4E4pQwirrz+J0CcDUfbWzTqEzVyYKKIKpgkc= github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E= github.com/multiformats/go-multiaddr-fmt v0.1.0/go.mod h1:hGtDIW4PU4BqJ50gW2quDuPVjyWNZxToGUh/HwTZYJo= -github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPwIdYQD509ZjSb5y9Oc= github.com/multiformats/go-multibase v0.2.0 h1:isdYCVLvksgWlMW9OZRYJEa9pZETFivncJHmHnnd87g= github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6oPyjtnzK/XTFDPkNuk= github.com/multiformats/go-multicodec v0.9.0 h1:pb/dlPnzee/Sxv/j4PmkDRxCOi3hXTz3IbPKOXWJkmg= github.com/multiformats/go-multicodec v0.9.0/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k= -github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= -github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= -github.com/multiformats/go-multihash v0.0.13/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc= -github.com/multiformats/go-multihash v0.0.14/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc= github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7BFvVU9RSh+U= github.com/multiformats/go-multihash v0.2.3/go.mod h1:dXgKXCXjBzdscBLk9JkjINiEsCKRVch90MdaGiKsvSM= github.com/multiformats/go-multistream v0.6.0 h1:ZaHKbsL404720283o4c/IHQXiS6gb8qAN5EIJ4PN5EA= github.com/multiformats/go-multistream v0.6.0/go.mod h1:MOyoG5otO24cHIg8kf9QW2/NozURlkP/rvi2FQJyCPg= -github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= -github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/multiformats/go-varint v0.0.7 h1:sWSGR+f/eu5ABZA2ZpYKBILXTTs9JWpdEM/nEGOHFS8= github.com/multiformats/go-varint v0.0.7/go.mod h1:r8PUYw/fD/SjBCiKOoDlGF6QawOELpZAu9eioSos/OU= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo= github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM= -github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec= -github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY= github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= github.com/onsi/ginkgo/v2 v2.22.0 h1:Yed107/8DjTr0lKCNt7Dn8yQ6ybuDRQoMGrNFKzMfHg= github.com/onsi/ginkgo/v2 v2.22.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo= @@ -626,6 +585,8 @@ github.com/pion/srtp/v2 v2.0.20 h1:HNNny4s+OUmG280ETrCdgFndp4ufx3/uy85EawYEhTk= github.com/pion/srtp/v2 v2.0.20/go.mod h1:0KJQjA99A6/a0DOVTu1PhDSw0CXF2jTkqOoMg3ODqdA= github.com/pion/stun v0.6.1 h1:8lp6YejULeHBF8NmV8e2787BogQhduZugh5PdhDyyN4= github.com/pion/stun v0.6.1/go.mod h1:/hO7APkX4hZKu/D0f2lHzNyvdkTGtIy3NDmLR7kSz/8= +github.com/pion/stun/v2 v2.0.0 h1:A5+wXKLAypxQri59+tmQKVs7+l6mMM+3d+eER9ifRU0= +github.com/pion/stun/v2 v2.0.0/go.mod h1:22qRSh08fSEttYUmJZGlriq9+03jtVmXNODgLccj8GQ= github.com/pion/transport/v2 v2.2.1/go.mod h1:cXXWavvCnFF6McHTft3DWS9iic2Mftcz1Aq29pGcU5g= github.com/pion/transport/v2 v2.2.3/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= github.com/pion/transport/v2 v2.2.4/go.mod h1:q2U/tf9FEfnSBGSW6w5Qp5PFWRLRj3NjLhCCgpRK4p0= @@ -660,8 +621,10 @@ github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkq github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk= github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc= github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= -github.com/prysmaticlabs/gohashtree v0.0.1-alpha.0.20220714111606-acbb2962fb48 h1:cSo6/vk8YpvkLbk9v3FO97cakNmUoxwi2KMP8hd5WIw= -github.com/prysmaticlabs/gohashtree v0.0.1-alpha.0.20220714111606-acbb2962fb48/go.mod h1:4pWaT30XoEx1j8KNJf3TV+E3mQkaufn7mf+jRNb/Fuk= +github.com/prysmaticlabs/gohashtree v0.0.4-beta h1:H/EbCuXPeTV3lpKeXGPpEV9gsUpkqOOVnWapUyeWro4= +github.com/prysmaticlabs/gohashtree v0.0.4-beta/go.mod h1:BFdtALS+Ffhg3lGQIHv9HDWuHS8cTvHZzrHWxwOtGOs= +github.com/quasilyte/go-ruleguard/dsl v0.3.23 h1:lxjt5B6ZCiBeeNO8/oQsegE6fLeCzuMRoVWSkXC4uvY= +github.com/quasilyte/go-ruleguard/dsl v0.3.23/go.mod h1:KeCP03KrjuSO0H1kTuZQCWlQPulDV6YMIXmpQss17rU= github.com/quic-go/qpack v0.5.1 h1:giqksBPnT/HDtZ6VhtFKgoLOWmlyo9Ei6u9PqzIMbhI= github.com/quic-go/qpack v0.5.1/go.mod h1:+PC4XFrEskIVkcLzpEkbLqq1uCoxPhQuvK5rH1ZgaEg= github.com/quic-go/quic-go v0.48.2 h1:wsKXZPeGWpMpCGSWqOcqpW2wZYic/8T3aqiOID0/KWE= @@ -715,26 +678,21 @@ github.com/smartystreets/goconvey v1.7.2 h1:9RBaZCeXEQ3UselpuwUQHltGVXvdwm6cv1hg github.com/smartystreets/goconvey v1.7.2/go.mod h1:Vw0tHAZW6lzCRk3xgdin6fKYcG+G3Pg9vgXWeJpQFMM= github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d/go.mod h1:UdhH50NIW0fCiwBSr0co2m7BnFLdv4fQTgdqdJTHFeE= github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e/go.mod h1:HuIsMU8RRBOtsCgI77wP899iHVBQpCmg4ErYMZB+2IA= -github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a h1:/eS3yfGjQKG+9kayBkj0ip1BGhq6zJ3eaVksphxAaek= -github.com/spacemonkeygo/openssl v0.0.0-20181017203307-c2dcc5cca94a/go.mod h1:7AyxJNCJ7SBZ1MfVQCWD6Uqo2oubI2Eq2y2eqf+A5r0= -github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 h1:RC6RW7j+1+HkWaX/Yh71Ee5ZHaHYt7ZP4sQgUrm6cDU= -github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572/go.mod h1:w0SWMsp6j9O/dk4/ZpIhL+3CkG8ofA2vuv7k+ltqUMc= github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spf13/afero v1.8.2 h1:xehSyVa0YnHWsJ49JFljMpg1HX19V6NDZ1fkm1Xznbo= github.com/spf13/afero v1.8.2/go.mod h1:CtAatgMJh6bJEIs48Ay/FOnkljP3WeGUG0MC1RfAqwo= github.com/spf13/cast v1.5.0 h1:rj3WzYc11XZaIZMPKmwP96zkFEnnAmV8s6XbB2aY32w= github.com/spf13/cast v1.5.0/go.mod h1:SpXXQ5YoyJw6s3/6cMTQuxvgRl3PCJiyaX9p6b155UU= -github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU= -github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM= +github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM= +github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y= github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk= github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo= -github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o= +github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/viper v1.12.0 h1:CZ7eSOd3kZoaYDLbXnmzgQI5RlciuXBMA+18HwHRfZQ= github.com/spf13/viper v1.12.0/go.mod h1:b6COn30jlNxbm/V2IqWiNWkJ+vZNiMNksliPCiuKtSI= -github.com/status-im/keycard-go v0.2.0 h1:QDLFswOQu1r5jsycloeQh3bVU8n/NatHHaZobtDnDzA= -github.com/status-im/keycard-go v0.2.0/go.mod h1:wlp8ZLbsmrF6g6WjugPAx+IzoLrkdf9+mHxBEeo3Hbg= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= @@ -754,26 +712,24 @@ github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= github.com/subosito/gotenv v1.3.0 h1:mjC+YW8QpAdXibNi+vNWgzmgBH4+5l5dCXv8cNysBLI= github.com/subosito/gotenv v1.3.0/go.mod h1:YzJjq/33h7nrwdY+iHMhEOEEbW0ovIz0tB6t6PwAXzs= -github.com/supranational/blst v0.3.11 h1:LyU6FolezeWAhvQk0k6O/d49jqgO52MSDDfYgbeoEm4= -github.com/supranational/blst v0.3.11/go.mod h1:jZJtfjgudtNl4en1tzwPIV3KjUnQUvG3/j+w+fVonLw= +github.com/supranational/blst v0.3.16 h1:bTDadT+3fK497EvLdWRQEjiGnUtzJ7jjIUMF0jqwYhE= +github.com/supranational/blst v0.3.16/go.mod h1:jZJtfjgudtNl4en1tzwPIV3KjUnQUvG3/j+w+fVonLw= github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 h1:epCh84lMvA70Z7CTTCmYQn2CKbY8j86K7/FAIr141uY= github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7/go.mod h1:q4W45IWZaF22tdD+VEXcAWRA037jwmWEB5VWYORlTpc= github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cbyO7IOYJZWg1U88JhDg3PB6klq9Hg2pA= -github.com/threshold-network/keep-common v1.7.1-tlabs.0 h1:E3Qy3yoeA3+9Ybi08Bb1Xm1D2fFxoberQwUjw+UEK8k= -github.com/threshold-network/keep-common v1.7.1-tlabs.0/go.mod h1:OmaZrnZODf6RJ95yUn2kBjy8Z4u2npPJQkSiyimluto= -github.com/threshold-network/tss-lib v0.0.0-20230901144531-2e712689cfbe h1:dOKhoYxZjXwFIyGnxgU+Sa1obZPMHRhu6e44oOLkzU4= -github.com/threshold-network/tss-lib v0.0.0-20230901144531-2e712689cfbe/go.mod h1:o3zAAo7A88ZJnCE1qpjy1hTqPn+GPQlxRsj8soz14UU= +github.com/threshold-network/keep-common v1.7.1-tlabs.1 h1:GcaQUb/5TOdc1Vhs4ZsbLM5a1C0CXx7Nmqv4npNKTag= +github.com/threshold-network/keep-common v1.7.1-tlabs.1/go.mod h1:BufGmgx5NVFeOjsb6aKI0MUv8vTzuNRbMluWtwPb9E8= +github.com/threshold-network/tss-lib v0.0.0-20260729021955-d847ce003019 h1:EmD85fdfi20RKON39+Hho5zmB57gHj7EWd7pTYwsqRY= +github.com/threshold-network/tss-lib v0.0.0-20260729021955-d847ce003019/go.mod h1:V6jseKmLMG1hHD9Qws8WEPaJ+ui1tWISyDGDe0eMkQk= github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU= github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI= github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk= github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY= -github.com/tyler-smith/go-bip39 v1.1.0 h1:5eUemwrMargf3BSLRRCalXT93Ns6pQJIjYQN2nyfOP8= -github.com/tyler-smith/go-bip39 v1.1.0/go.mod h1:gUYDtqQw1JS3ZJ8UWVcGTGqqr6YIN3CWg+kkNaLt55U= github.com/urfave/cli v1.22.2/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0= github.com/urfave/cli v1.22.10 h1:p8Fspmz3iTctJstry1PYS3HVdllxnEzTEsgIgtxTrCk= github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0= -github.com/urfave/cli/v2 v2.25.7 h1:VAzn5oq403l5pHjc4OhD54+XGO9cdKVL/7lDjF+iKUs= -github.com/urfave/cli/v2 v2.25.7/go.mod h1:8qnjx1vcq5s2/wpsqoZFndg2CE5tNFyrTvS6SinrnYQ= +github.com/urfave/cli/v2 v2.27.5 h1:WoHEJLdsXr6dDWoJgMq/CboDmyY/8HMMH1fTECbih+w= +github.com/urfave/cli/v2 v2.27.5/go.mod h1:3Sevf16NykTbInEnD0yKkjDAeZDS0A6bzhBH5hrMvTQ= github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= github.com/valyala/fasttemplate v1.0.1/go.mod h1:UQGH1tvbgY+Nz5t2n7tXsz52dQxojPUpymEIMZ47gx8= github.com/valyala/fasttemplate v1.2.1/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ= @@ -787,14 +743,16 @@ github.com/whyrusleeping/go-logging v0.0.0-20170515211332-0457bb6b88fc/go.mod h1 github.com/wlynxg/anet v0.0.3/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA= github.com/wlynxg/anet v0.0.5 h1:J3VJGi1gvo0JwZ/P1/Yc/8p63SoW98B5dHkYDmpgvvU= github.com/wlynxg/anet v0.0.5/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA= -github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU= -github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673/go.mod h1:N3UwUGtsrSj3ccvlPHLoLsHnpR27oXr4ZE984MbSER8= +github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4= +github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= +go.etcd.io/bbolt v1.3.11 h1:yGEzV1wPz2yVCLsD8ZAiGHhHVlczyC9d1rP43/VCRJ0= +go.etcd.io/bbolt v1.3.11/go.mod h1:dksAq7YMXoljX0xu6VF5DMZGbhYYoLUalEiSySYAS4I= go.opencensus.io v0.18.0/go.mod h1:vKdFvxhtzZ9onBp9VKHK8z/sRpBMnKAsufL7wlDrCOA= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8= @@ -810,6 +768,8 @@ go.opentelemetry.io/otel v1.41.0 h1:YlEwVsGAlCvczDILpUXpIpPSL/VPugt7zHThEMLce1c= go.opentelemetry.io/otel v1.41.0/go.mod h1:Yt4UwgEKeT05QbLwbyHXEwhnjxNO6D8L5PQP51/46dE= go.opentelemetry.io/otel/metric v1.41.0 h1:rFnDcs4gRzBcsO9tS8LCpgR0dxg4aaxWlJxCno7JlTQ= go.opentelemetry.io/otel/metric v1.41.0/go.mod h1:xPvCwd9pU0VN8tPZYzDZV/BMj9CM9vs00GuBjeKhJps= +go.opentelemetry.io/otel/sdk v1.40.0 h1:KHW/jUzgo6wsPh9At46+h4upjtccTmuZCFAc9OJ71f8= +go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE= go.opentelemetry.io/otel/trace v1.41.0 h1:Vbk2co6bhj8L59ZJ6/xFTskY+tGAbOnCtQGVVa9TIN0= go.opentelemetry.io/otel/trace v1.41.0/go.mod h1:U1NU4ULCoxeDKc09yCWdWe+3QoyweJcISEVa1RBzOis= go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= @@ -836,11 +796,8 @@ go4.org v0.0.0-20180809161055-417644f6feb5/go.mod h1:MkTOUMDaeVYJUOUsaDXIhWPZYa1 golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d/go.mod h1:OWs+y06UdEOHN4y+MfF/py+xQ/tYqIWW03b70/CG9Rw= golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20181030102418-4d3f4d9ffa16/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= -golang.org/x/crypto v0.0.0-20190225124518-7f87c0fbb88b/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190313024323-a1f597ede03a/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= @@ -858,8 +815,8 @@ golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5y golang.org/x/crypto v0.8.0/go.mod h1:mRqEX+O9/h5TFCrQhkgjo2yKi0yYA+9ecGkdQoHrywE= golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg= -golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc= -golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc= +golang.org/x/crypto v0.47.0 h1:V6e3FRj+n4dbpw86FJ8Fv7XVOql7TEwpHapKoMJ/GO8= +golang.org/x/crypto v0.47.0/go.mod h1:ff3Y9VzzKbwSSEzWqJsJVBnWmRwRSHt/6Op5n9bQc4A= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -899,8 +856,8 @@ golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.22.0 h1:D4nJWe9zXqHOmWqj4VMOJhvzj7bEZg4wEYa759z1pH4= -golang.org/x/mod v0.22.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY= +golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI= +golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg= golang.org/x/net v0.0.0-20180719180050-a680a1efc54d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= @@ -947,8 +904,8 @@ golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= golang.org/x/net v0.14.0/go.mod h1:PpSgVXXLK0OxS0F31C1/tv6XNguvCrnXIDrFMspZIUI= golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY= -golang.org/x/net v0.34.0 h1:Mb7Mrk043xzHgnRM88suvJFwzVrRfHEHJEl5/71CKw0= -golang.org/x/net v0.34.0/go.mod h1:di0qlW3YNM5oh6GqDGQr92MyTozJPmybPK4Ev/Gm31k= +golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o= +golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181017192945-9dcd33a902f4/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= @@ -974,14 +931,13 @@ golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ= -golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4= +golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= golang.org/x/sys v0.0.0-20180810173357-98c5dad5d1a0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181029174526-d69651ed3497/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190219092855-153ac476189d/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190316082340-a2f829d7f35f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1039,8 +995,10 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU= -golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ= +golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= +golang.org/x/telemetry v0.0.0-20251203150158-8fff8a5912fc h1:bH6xUXay0AIFMElXG2rQ4uiE+7ncwtiOdPfYK1NK2XA= +golang.org/x/telemetry v0.0.0-20251203150158-8fff8a5912fc/go.mod h1:hKdjCMrbv9skySur+Nek8Hd0uJ0GuxJIoIX2payrIdQ= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= @@ -1049,8 +1007,8 @@ golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.11.0/go.mod h1:zC9APTIj3jG3FdV/Ons+XE1riIZXG4aZ4GTHiPZJPIU= golang.org/x/term v0.16.0/go.mod h1:yn7UURbUtPyrVJPGPq404EukNFxcm/foM+bV/bfcDsY= -golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg= -golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek= +golang.org/x/term v0.39.0 h1:RclSuaJf32jOqZz74CkPA9qFuVTX7vhLlpfj/IGWlqY= +golang.org/x/term v0.39.0/go.mod h1:yxzUCTP/U+FzoxfdKmLaA0RV1WgE0VY7hXBwKtY/4ww= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -1063,16 +1021,16 @@ golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo= -golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ= +golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE= +golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20201208040808-7e3f01d25324/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= -golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= -golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= +golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY= +golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -1132,8 +1090,8 @@ golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/tools v0.29.0 h1:Xx0h3TtM9rzQpQuR4dKLrdglAmCEN5Oi+P74JdhdzXE= -golang.org/x/tools v0.29.0/go.mod h1:KMQVMRsVxU6nHCFXrBPhDB8XncLNLM0lIy/F14RP588= +golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA= +golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -1242,10 +1200,8 @@ google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGj google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.28.2-0.20220831092852-f930b1dc76e8 h1:KR8+MyP7/qOlV+8Af01LtjL04bu7on42eVsxT4EyBQk= -google.golang.org/protobuf v1.28.2-0.20220831092852-f930b1dc76e8/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= -google.golang.org/protobuf v1.36.3 h1:82DV7MYdb8anAVi3qge1wSnMDrnKK7ebr+I0hHRN1BU= -google.golang.org/protobuf v1.36.3/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -1255,8 +1211,8 @@ gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw= gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA= gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k= -gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8= -gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k= +gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= +gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= @@ -1278,10 +1234,10 @@ honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9 honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k= lukechampine.com/blake3 v1.3.0 h1:sJ3XhFINmHSrYCgl958hscfIa3bw8x4DqMP3u1YvoYE= lukechampine.com/blake3 v1.3.0/go.mod h1:0OFRp7fBtAylGVCO40o87sbupkyIGgbpv1+M1k1LM6k= +pgregory.net/rapid v1.3.0 h1:vBvO0VSqti75J1jjYqpgPNBLKMd1+gxa9fYo7vk/Exc= +pgregory.net/rapid v1.3.0/go.mod h1:dPlE4OBBxgXPqkP79flB6sJL1dx5azpI7HQ9MY9Z7uk= rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= -rsc.io/tmplfunc v0.0.3 h1:53XFQh69AfOa8Tw0Jm7t+GV7KZhOi6jzsCzTtKbMvzU= -rsc.io/tmplfunc v0.0.3/go.mod h1:AG3sTPzElb1Io3Yg4voV9AGZJuleGAwaVRxL9M49PhA= sourcegraph.com/sourcegraph/go-diff v0.5.0/go.mod h1:kuch7UrkMzY0X+p9CRK03kfuPQ2zzQcaEFbx8wA8rck= sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4/go.mod h1:ketZ/q3QxT9HOBeFhu6RdvsftgpsbFHBF5Cas6cDKZ0= diff --git a/infrastructure/eth-networks/private-testnet/bundles/bundle-guide.adoc b/infrastructure/eth-networks/private-testnet/bundles/bundle-guide.adoc index 0ef60af252..47858c0c76 100644 --- a/infrastructure/eth-networks/private-testnet/bundles/bundle-guide.adoc +++ b/infrastructure/eth-networks/private-testnet/bundles/bundle-guide.adoc @@ -93,6 +93,16 @@ To validate the running client check the metrics for the number of connected pee The client should connect to the bootstrap nodes (at least 2) and other nodes that are working in the network. There should be at least 10 connections. +For the coordinated security release the metrics endpoint listens on the +compatibility default port `9601`. Set an explicit port in the client +configuration to pin it (recommended), or set `clientInfo.port = 0` to disable +it; then probe the configured port over a trusted network path only: + +``` +[clientInfo] +Port = 9601 +``` + ``` -curl localhost:9601/metrics +curl localhost:/metrics ``` diff --git a/infrastructure/kube/keep-dev/eth-tx-rpc-ws-networkpolicy.yaml b/infrastructure/kube/keep-dev/eth-tx-rpc-ws-networkpolicy.yaml new file mode 100644 index 0000000000..49237f7cc7 --- /dev/null +++ b/infrastructure/kube/keep-dev/eth-tx-rpc-ws-networkpolicy.yaml @@ -0,0 +1,23 @@ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: eth-tx-rpc-ws-ingress + namespace: ropsten + labels: + app: geth + type: tx +spec: + podSelector: + matchLabels: + app: geth + type: tx + policyTypes: + - Ingress + ingress: + - from: + - podSelector: {} + ports: + - protocol: TCP + port: 8545 + - protocol: TCP + port: 8546 diff --git a/infrastructure/kube/keep-prd/monitoring/README.adoc b/infrastructure/kube/keep-prd/monitoring/README.adoc index bc9f79b764..0af33a9230 100644 --- a/infrastructure/kube/keep-prd/monitoring/README.adoc +++ b/infrastructure/kube/keep-prd/monitoring/README.adoc @@ -15,9 +15,48 @@ The monitoring stack has the following components: 1. Prometheus 2. Trickster 3. Grafana +4. Alertmanager (cutover-roster alert routing) +5. cutover-roster (coordinated-cutover fleet readiness collector) The production monitoring is based on the configuration described in the link:../../keep-test/monitoring/README.adoc[keep-test monitoring documentation]. +## Cutover-roster fleet readiness + +The `cutover-roster` collector answers the coordinated-cutover go/no-go question +("which ceremony-eligible instance has not reported the exact cutover release?"). +It exposes the `performance_cutover_*` metrics (Prometheus job `cutover-roster`) +and a readiness API (`GET /api/v1/cutover-readiness`). The **Cutover Readiness** +Grafana dashboard reads the fleet gauges from Prometheus and the per-instance +reconciliation reasons from the readiness API via the Infinity datasource. The +`cutover-roster` alerts (`prometheus/config/rules.yaml`, group `cutover-roster`) +route to Alertmanager, whose tree matches their `route_to` label and fans them to +the Release and Operator Coordination receivers. + +Three operational details make the stack internally deployable: + +* The Infinity datasource plugin is pinned to `2.3.1` in +`grafana/grafana-deployment.yaml` — the newest Infinity release compatible with +the pinned Grafana `9.2.5` (its `grafanaDependency` is `>=8.4.7`). An unversioned +install resolves to the latest Infinity, which requires Grafana `>= 11.6`. Drop +the pin if Grafana is upgraded. +* The collector serves `/healthz` OUTSIDE the `--allowedCIDRs` boundary, and the +`cutover-roster` Deployment's liveness/readiness probes target it. Kubelet probes +originate from the node IP (not the monitoring pod CIDR, not loopback), so probing +an allowlisted data endpoint would return `403` and keep the pod permanently +unready. The data endpoints (readiness API + `/metrics`) stay behind the allowlist. +* A third alert, `CutoverRosterCollectorDown` +(`up{job="cutover-roster"} == 0 or absent(...)`), fires when the collector target +is down or absent. Without it a dead collector makes every `performance_cutover_*` +series vanish, so the other two roster alerts would evaluate absent and never fire. + +NOTE: The `cutover-roster/deployment.yaml` and `alertmanager/` manifests are +reviewable skeletons. Before apply, fill the `REPLACE_` placeholders: the +collector image `@sha256:` digest, the monitoring pod CIDR (`--allowedCIDRs`), +the authoritative inventory `ConfigMap` / secrets (Ethereum RPC URL, +WalletRegistry address, expected revision/digest, cutover block), and the +Alertmanager receiver integrations. These are operator-supplied and are +intentionally not committed. + Resources are exposed publicly under the following URLs: [cols="^1s,2m"] diff --git a/infrastructure/kube/keep-prd/monitoring/alertmanager/alertmanager-deployment.yaml b/infrastructure/kube/keep-prd/monitoring/alertmanager/alertmanager-deployment.yaml new file mode 100644 index 0000000000..22d3ed23da --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/alertmanager/alertmanager-deployment.yaml @@ -0,0 +1,65 @@ +--- +# Alertmanager for the cutover-roster fleet alerts. Prometheus (config.yaml +# alerting block) forwards alerts here; the routing tree in +# config/alertmanager.yaml matches their route_to label and fans them out to the +# Release and Operator Coordination receivers. +# +# FOLLOW-UP BEFORE APPLY: fill the receiver integrations in +# config/alertmanager.yaml from a Secret (Slack/PagerDuty/email); this skeleton +# routes but does not deliver until a receiver is configured. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: alertmanager +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: alertmanager + type: monitoring + template: + spec: + securityContext: + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + runAsNonRoot: true + containers: + - name: alertmanager + image: prom/alertmanager:v0.26.0 + args: + - --config.file=/etc/alertmanager/alertmanager.yaml + - --storage.path=/alertmanager + - --web.external-url=/alertmanager/ + ports: + - name: alertmanager + containerPort: 9093 + readinessProbe: + httpGet: + path: /alertmanager/-/ready + port: alertmanager + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 2 + resources: + limits: + cpu: 200m + memory: 256Mi + requests: + cpu: 50m + memory: 64Mi + volumeMounts: + - name: alertmanager-config-volume + mountPath: /etc/alertmanager/ + - name: alertmanager-storage-volume + mountPath: /alertmanager + securityContext: + readOnlyRootFilesystem: true + volumes: + - name: alertmanager-config-volume + configMap: + name: alertmanager-config + - name: alertmanager-storage-volume + emptyDir: {} diff --git a/infrastructure/kube/keep-prd/monitoring/alertmanager/alertmanager-service.yaml b/infrastructure/kube/keep-prd/monitoring/alertmanager/alertmanager-service.yaml new file mode 100644 index 0000000000..ab581cfa59 --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/alertmanager/alertmanager-service.yaml @@ -0,0 +1,15 @@ +--- +apiVersion: v1 +kind: Service +metadata: + name: alertmanager +spec: + type: ClusterIP + selector: + app: alertmanager + type: monitoring + ports: + - name: alertmanager + port: 9093 + targetPort: alertmanager + protocol: TCP diff --git a/infrastructure/kube/keep-prd/monitoring/alertmanager/config/alertmanager.yaml b/infrastructure/kube/keep-prd/monitoring/alertmanager/config/alertmanager.yaml new file mode 100644 index 0000000000..84b36dfee5 --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/alertmanager/config/alertmanager.yaml @@ -0,0 +1,38 @@ +# Alertmanager routing for the cutover-roster fleet alerts. +# +# The cutover-roster rules (prometheus/config/rules.yaml, group cutover-roster; +# mirrored by cutoverroster.AlertRules()) set a route_to label of +# "release,operator-coordination". The routing tree below matches that label and +# fans the alert out to both teams. The receiver integrations themselves +# (Slack/PagerDuty/email endpoints) are REPLACE_ placeholders — fill them in from +# a Secret at deploy time; do not commit real webhook URLs. +global: + resolve_timeout: 5m + +route: + receiver: default + group_by: ["alertname", "team"] + group_wait: 30s + group_interval: 5m + repeat_interval: 4h + routes: + # Cutover alerts carry route_to=release,operator-coordination. Match the + # "release" audience and continue so the operator-coordination route below + # also fires for the same alert. + - matchers: + - route_to =~ ".*release.*" + receiver: release + continue: true + - matchers: + - route_to =~ ".*operator-coordination.*" + receiver: operator-coordination + continue: true + +receivers: + - name: default + - name: release + # REPLACE_WITH_RELEASE_RECEIVER: e.g. a slack_configs / pagerduty_configs / + # email_configs block for the Release team, sourced from a Secret. + - name: operator-coordination + # REPLACE_WITH_OPERATOR_COORDINATION_RECEIVER: the Operator Coordination + # team's integration, sourced from a Secret. diff --git a/infrastructure/kube/keep-prd/monitoring/alertmanager/kustomization.yaml b/infrastructure/kube/keep-prd/monitoring/alertmanager/kustomization.yaml new file mode 100644 index 0000000000..09a8fe260c --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/alertmanager/kustomization.yaml @@ -0,0 +1,19 @@ +resources: + - alertmanager-deployment.yaml + - alertmanager-service.yaml + +namespace: monitoring + +commonLabels: + app: alertmanager + type: monitoring + +configMapGenerator: + - name: alertmanager-config + files: + - config/alertmanager.yaml + +generatorOptions: + disableNameSuffixHash: true + annotations: + note: generated diff --git a/infrastructure/kube/keep-prd/monitoring/cutover-roster/deployment.yaml b/infrastructure/kube/keep-prd/monitoring/cutover-roster/deployment.yaml new file mode 100644 index 0000000000..859d6387c1 --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/cutover-roster/deployment.yaml @@ -0,0 +1,138 @@ +--- +# cutover-roster fleet collector Deployment. +# +# FOLLOW-UP BEFORE APPLY (see monitoring/README.adoc): this manifest is a +# reviewable skeleton. The values marked REPLACE_ must be filled at release time: +# * spec.template.spec.containers[cutover-roster].image — pin the collector +# image by an immutable @sha256: digest (never a mutable tag), matching the +# single-release immutable-digest requirement. +# * --allowedCIDRs — the monitoring/Prometheus pod CIDR that is allowed to +# scrape the API (the collector refuses a non-loopback bind without it). +# * the cutover-roster-inventory ConfigMap and cutover-roster-secrets Secret — +# the authoritative inventory JSON, attested digests, quarantine evidence, +# Ethereum RPC URL, and WalletRegistry address. These are operator-supplied +# and are intentionally not committed here. +# * --expectedRevision / --expectedImageDigest / --cutoverBlock / --chainID — +# become meaningful once the real cutover release ships. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cutover-roster +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: cutover-roster + type: monitoring + template: + spec: + securityContext: + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + runAsNonRoot: true + containers: + - name: cutover-roster + # REPLACE_WITH_IMMUTABLE_DIGEST: pin by @sha256: digest before apply. + image: keepnetwork/cutover-roster@sha256:REPLACE_WITH_IMMUTABLE_DIGEST + args: + - --apiAddr=0.0.0.0:9701 + # REPLACE_WITH_MONITORING_POD_CIDR: the Prometheus/monitoring pod + # network allowed to reach the DATA endpoints (readiness API + /metrics). + # Required for a non-loopback bind. It need NOT include the node/kubelet + # IPs: the liveness/readiness probes below target /healthz, which is + # served outside this allowlist, so probes from the node IP still succeed. + - --allowedCIDRs=REPLACE_WITH_MONITORING_POD_CIDR + - --dbPath=/var/lib/cutover-roster/roster.db + - --inventoryFile=/etc/cutover-roster/inventory.json + - --serviceDiscoveryFile=/etc/prometheus/sd/keep-sd.json + - --attestedDigestsFile=/etc/cutover-roster/attested-digests.json + - --quarantineEvidenceFile=/etc/cutover-roster/quarantine-evidence.json + - --expectedRevision=REPLACE_WITH_RELEASE_REVISION + - --expectedImageDigest=REPLACE_WITH_RELEASE_IMAGE_DIGEST + - --expectedEpoch=security_v2_cutover + - --cutoverBlock=REPLACE_WITH_CUTOVER_BLOCK + - --chainID=1 + # REPLACE_ these from the cutover-roster-secrets Secret at deploy time + # (the collector reads them as flags; template them in rather than + # committing the RPC URL / registry address). Both are REQUIRED for a + # complete readiness determination. + - --ethereumRPC=REPLACE_WITH_ETHEREUM_RPC_URL + - --walletRegistryAddress=REPLACE_WITH_WALLET_REGISTRY_ADDRESS + ports: + - name: api + containerPort: 9701 + # Probes target /healthz, which the collector serves OUTSIDE the + # --allowedCIDRs boundary (the data endpoints stay behind it). The kubelet + # sends probes from the node IP — not on the monitoring pod CIDR and not + # loopback — so probing an allowlisted data endpoint would return 403 and + # keep the pod permanently unready. + readinessProbe: + httpGet: + path: /healthz + port: api + initialDelaySeconds: 10 + periodSeconds: 30 + timeoutSeconds: 2 + livenessProbe: + httpGet: + path: /healthz + port: api + initialDelaySeconds: 15 + periodSeconds: 30 + timeoutSeconds: 2 + resources: + limits: + cpu: 500m + memory: 512Mi + requests: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cutover-roster-db + mountPath: /var/lib/cutover-roster/ + - name: cutover-roster-inventory + mountPath: /etc/cutover-roster/ + readOnly: true + - name: cutover-roster-sd + mountPath: /etc/prometheus/sd/ + readOnly: true + securityContext: + readOnlyRootFilesystem: true + # keep-sd sidecar produces the same keep-sd.json service-discovery target + # file the collector reconciles against, so an eligible operator absent + # from discovery is offline_unknown and discovered per-instance targets + # (keyed by network id) supply the report scrape URLs. + - name: keep-sd + image: keepnetwork/keep-prometheus-sd + args: + - --output.file=/etc/prometheus/sd/keep-sd.json + - --source.address=bst-a01.tbtc.boar.network:9601 + - --source.address=bst-b01.tbtc.boar.network:9601 + - --refresh.interval=5m + - --scan.timeout=3s + - --log.json + resources: + limits: + cpu: 500m + memory: 256Mi + requests: + cpu: 100m + memory: 128Mi + volumeMounts: + - name: cutover-roster-sd + mountPath: /etc/prometheus/sd/ + securityContext: + readOnlyRootFilesystem: true + volumes: + - name: cutover-roster-db + persistentVolumeClaim: + claimName: cutover-roster-pvc + - name: cutover-roster-inventory + configMap: + name: cutover-roster-inventory + optional: true + - name: cutover-roster-sd + emptyDir: {} diff --git a/infrastructure/kube/keep-prd/monitoring/cutover-roster/kustomization.yaml b/infrastructure/kube/keep-prd/monitoring/cutover-roster/kustomization.yaml new file mode 100644 index 0000000000..c3de20d0ea --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/cutover-roster/kustomization.yaml @@ -0,0 +1,10 @@ +resources: + - deployment.yaml + - service.yaml + - pvc.yaml + +namespace: monitoring + +commonLabels: + app: cutover-roster + type: monitoring diff --git a/infrastructure/kube/keep-prd/monitoring/cutover-roster/pvc.yaml b/infrastructure/kube/keep-prd/monitoring/cutover-roster/pvc.yaml new file mode 100644 index 0000000000..e6c8473d94 --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/cutover-roster/pvc.yaml @@ -0,0 +1,15 @@ +--- +# Durable storage for the collector's bbolt central state (roster.db). Central +# state (resolved/blocking/quarantined operator history) must survive collector +# restarts, so it is persisted rather than kept in an emptyDir. +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cutover-roster-pvc +spec: + storageClassName: monitoring-storage + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi diff --git a/infrastructure/kube/keep-prd/monitoring/cutover-roster/service.yaml b/infrastructure/kube/keep-prd/monitoring/cutover-roster/service.yaml new file mode 100644 index 0000000000..2c3e3db7d2 --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/cutover-roster/service.yaml @@ -0,0 +1,20 @@ +--- +# Monitoring-only Service for the cutover-roster fleet collector. It fronts the +# readiness API and /metrics endpoint (port 9701) so Prometheus (job +# cutover-roster) and the Grafana Infinity datasource (Cutover Roster API) can +# reach it by DNS name on the monitoring network. It is a ClusterIP service and +# is never exposed publicly; the readiness data is authoritative but not public. +apiVersion: v1 +kind: Service +metadata: + name: cutover-roster +spec: + type: ClusterIP + selector: + app: cutover-roster + type: monitoring + ports: + - name: api + port: 9701 + targetPort: api + protocol: TCP diff --git a/infrastructure/kube/keep-prd/monitoring/grafana/config/datasources.yaml b/infrastructure/kube/keep-prd/monitoring/grafana/config/datasources.yaml index ef00731e62..714d3ece23 100644 --- a/infrastructure/kube/keep-prd/monitoring/grafana/config/datasources.yaml +++ b/infrastructure/kube/keep-prd/monitoring/grafana/config/datasources.yaml @@ -8,3 +8,20 @@ datasources: url: http://trickster:8480/prometheus version: 1 isDefault: true + # JSON datasource for the cutover-roster readiness API. The per-instance + # reconciliation detail (instance_statuses) is exposed only by the API — node + # and collector metrics deliberately omit per-instance/session labels — so the + # instance-reason table on the Cutover Readiness dashboard reads it here rather + # than from Prometheus. Reaches the collector Service on the monitoring network. + - name: Cutover Roster API + uid: cutover-roster-api + type: yesoreyeram-infinity-datasource + access: proxy + editable: true + orgId: 1 + url: http://cutover-roster:9701 + version: 1 + jsonData: + auth_method: none + allowedHosts: + - http://cutover-roster:9701 diff --git a/infrastructure/kube/keep-prd/monitoring/grafana/dashboards/keep/cutover-readiness.json b/infrastructure/kube/keep-prd/monitoring/grafana/dashboards/keep/cutover-readiness.json new file mode 100644 index 0000000000..dbd58474a8 --- /dev/null +++ b/infrastructure/kube/keep-prd/monitoring/grafana/dashboards/keep/cutover-readiness.json @@ -0,0 +1,153 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { "type": "grafana", "uid": "-- Grafana --" }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "description": "Coordinated protocol cutover fleet readiness: blocking, quarantined, and recently-resolved operators from the cutover-roster collector. Per-instance reasons are in the readiness API (GET /api/v1/cutover-readiness); node metrics deliberately omit instance/session labels.", + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "links": [], + "liveNow": false, + "panels": [ + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } }, "overrides": [] }, + "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 }, + "id": 1, + "options": { "colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_fleet_blocking_operators", "instant": true, "refId": "A" } ], + "title": "Blocking operators", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 1 } ] } }, "overrides": [] }, + "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 }, + "id": 2, + "options": { "colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_fleet_observed_legacy", "instant": true, "refId": "A" } ], + "title": "Observed legacy (post-cutover)", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 1 } ] } }, "overrides": [] }, + "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 }, + "id": 3, + "options": { "colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_reporters_stale", "instant": true, "refId": "A" } ], + "title": "Reporters stale", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } }, "overrides": [] }, + "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 }, + "id": 4, + "options": { "colorMode": "value", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_inventory_unreconciled", "instant": true, "refId": "A" } ], + "title": "Inventory unreconciled", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "description": "Operators in any blocking status (offline_unknown, noncutover_revision, observed_legacy) with their staking provider and last-seen block.", + "fieldConfig": { "defaults": { "custom": { "align": "auto", "displayMode": "auto" } }, "overrides": [] }, + "gridPos": { "h": 9, "w": 24, "x": 0, "y": 4 }, + "id": 5, + "options": { "showHeader": true }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_operator_last_seen_block{status=~\"offline_unknown|noncutover_revision|observed_legacy\"}", "format": "table", "instant": true, "refId": "A" } ], + "title": "Blocking operators", + "transformations": [ { "id": "labelsToFields", "options": {} }, { "id": "organize", "options": { "excludeByName": { "Time": true, "__name__": true, "job": true, "instance": true }, "renameByName": { "Value": "last_seen_block" } } } ], + "type": "table" + }, + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "description": "Operators whose otherwise-blocking instances have independently-verified network/eligibility quarantine or removal evidence.", + "fieldConfig": { "defaults": { "custom": { "align": "auto", "displayMode": "auto" } }, "overrides": [] }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 }, + "id": 6, + "options": { "showHeader": true }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_operator_last_seen_block{status=\"quarantined\"}", "format": "table", "instant": true, "refId": "A" } ], + "title": "Quarantined operators", + "transformations": [ { "id": "labelsToFields", "options": {} }, { "id": "organize", "options": { "excludeByName": { "Time": true, "__name__": true, "job": true, "instance": true }, "renameByName": { "Value": "last_seen_block" } } } ], + "type": "table" + }, + { + "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, + "description": "Operators confirmed resolved_current (retained for 30 days as go/no-go evidence).", + "fieldConfig": { "defaults": { "custom": { "align": "auto", "displayMode": "auto" } }, "overrides": [] }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 }, + "id": 7, + "options": { "showHeader": true }, + "pluginVersion": "9.3.0", + "targets": [ { "datasource": { "type": "prometheus", "uid": "P09205B1DD12FB1C6" }, "expr": "performance_cutover_operator_last_seen_block{status=\"resolved_current\"}", "format": "table", "instant": true, "refId": "A" } ], + "title": "Recently resolved operators", + "transformations": [ { "id": "labelsToFields", "options": {} }, { "id": "organize", "options": { "excludeByName": { "Time": true, "__name__": true, "job": true, "instance": true }, "renameByName": { "Value": "last_seen_block" } } } ], + "type": "table" + }, + { + "datasource": { "type": "yesoreyeram-infinity-datasource", "uid": "cutover-roster-api" }, + "description": "Per-instance reconciliation detail for every blocking operator, read from the collector readiness API (GET /api/v1/cutover-readiness). Node/collector metrics deliberately omit per-instance labels, so this table sources the API directly rather than Prometheus.", + "fieldConfig": { "defaults": { "custom": { "align": "auto", "displayMode": "auto" } }, "overrides": [] }, + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 }, + "id": 8, + "options": { "showHeader": true }, + "pluginVersion": "9.3.0", + "targets": [ + { + "datasource": { "type": "yesoreyeram-infinity-datasource", "uid": "cutover-roster-api" }, + "refId": "A", + "type": "json", + "source": "url", + "format": "table", + "parser": "backend", + "url": "http://cutover-roster:9701/api/v1/cutover-readiness", + "url_options": { "method": "GET" }, + "root_selector": "blocking.instance_statuses.{\"operator_address\": %.operator_address, \"staking_provider\": %.staking_provider, \"instance_id\": instance_id, \"class\": class, \"reason\": reason, \"reported_this_cycle\": reported_this_cycle, \"expected_revision\": expected_revision, \"observed_revision\": observed_revision, \"expected_image_digest\": expected_image_digest, \"observed_image_digest\": observed_image_digest, \"quarantined\": quarantined}", + "columns": [ + { "selector": "operator_address", "text": "Operator", "type": "string" }, + { "selector": "staking_provider", "text": "Staking provider", "type": "string" }, + { "selector": "instance_id", "text": "Instance", "type": "string" }, + { "selector": "class", "text": "Class", "type": "string" }, + { "selector": "reason", "text": "Reason", "type": "string" }, + { "selector": "reported_this_cycle", "text": "Reported this cycle", "type": "string" }, + { "selector": "expected_revision", "text": "Expected rev", "type": "string" }, + { "selector": "observed_revision", "text": "Observed rev", "type": "string" }, + { "selector": "quarantined", "text": "Quarantined", "type": "string" } + ] + } + ], + "title": "Instance-level reasons (blocking operators)", + "type": "table" + } + ], + "refresh": "1m", + "schemaVersion": 37, + "style": "dark", + "tags": ["keep", "cutover", "release"], + "templating": { "list": [] }, + "time": { "from": "now-6h", "to": "now" }, + "timepicker": {}, + "timezone": "", + "title": "Cutover Readiness", + "uid": "cutover-readiness", + "version": 1, + "weekStart": "" +} diff --git a/infrastructure/kube/keep-prd/monitoring/grafana/grafana-deployment.yaml b/infrastructure/kube/keep-prd/monitoring/grafana/grafana-deployment.yaml index d9d39b4acd..5c64d1812c 100644 --- a/infrastructure/kube/keep-prd/monitoring/grafana/grafana-deployment.yaml +++ b/infrastructure/kube/keep-prd/monitoring/grafana/grafana-deployment.yaml @@ -19,6 +19,20 @@ spec: - name: grafana image: grafana/grafana:9.2.5 env: + # The Cutover Readiness dashboard's instance-reason table reads the + # cutover-roster readiness API (JSON), which needs the Infinity + # datasource plugin. It installs into the writable grafana PVC at + # /var/lib/grafana/plugins, so readOnlyRootFilesystem is preserved. + # + # The version is PINNED to 2.3.1 (the newest Infinity release whose + # plugin.json grafanaDependency is ">=8.4.7", so it loads on the pinned + # Grafana 9.2.5 below). An unversioned install resolves to the latest + # Infinity, which now requires Grafana >= 11.6 and would silently fail + # to load here, leaving the instance-reason table broken. The plugin id + # and version are space-separated per Grafana's docker install format. + # If Grafana is upgraded to >= 11.6, this pin can be dropped. + - name: GF_INSTALL_PLUGINS + value: yesoreyeram-infinity-datasource 2.3.1 - name: GF_SERVER_DOMAIN value: monitoring.threshold.network - name: GF_SERVER_ROOT_URL diff --git a/infrastructure/kube/keep-prd/monitoring/grafana/kustomization.yaml b/infrastructure/kube/keep-prd/monitoring/grafana/kustomization.yaml index e1ca15444f..12164a00a6 100644 --- a/infrastructure/kube/keep-prd/monitoring/grafana/kustomization.yaml +++ b/infrastructure/kube/keep-prd/monitoring/grafana/kustomization.yaml @@ -19,6 +19,7 @@ configMapGenerator: files: - dashboards/keep/keep-nodes-public.json - dashboards/keep/keep-nodes.json + - dashboards/keep/cutover-readiness.json generatorOptions: disableNameSuffixHash: true diff --git a/infrastructure/kube/keep-prd/monitoring/monitoring-ingress.yaml b/infrastructure/kube/keep-prd/monitoring/monitoring-ingress.yaml index bfa25808cb..71daef72fa 100644 --- a/infrastructure/kube/keep-prd/monitoring/monitoring-ingress.yaml +++ b/infrastructure/kube/keep-prd/monitoring/monitoring-ingress.yaml @@ -25,20 +25,6 @@ spec: name: grafana port: number: 3000 - - path: "/prometheus" - pathType: Prefix - backend: - service: - name: trickster - port: - number: 8480 - - path: "/trickster" - pathType: Prefix - backend: - service: - name: trickster - port: - number: 8480 --- apiVersion: networking.gke.io/v1 kind: ManagedCertificate diff --git a/infrastructure/kube/keep-prd/monitoring/prometheus/config/config.yaml b/infrastructure/kube/keep-prd/monitoring/prometheus/config/config.yaml index caafb7470f..ee9bfe18a5 100644 --- a/infrastructure/kube/keep-prd/monitoring/prometheus/config/config.yaml +++ b/infrastructure/kube/keep-prd/monitoring/prometheus/config/config.yaml @@ -4,7 +4,27 @@ global: evaluation_interval: 1m rule_files: - /etc/prometheus/rules.yaml +# Route the cutover-roster alerts (rules.yaml, group cutover-roster) to +# Alertmanager, whose routing tree matches their route_to label +# (monitoring/alertmanager/config/alertmanager.yaml). The Alertmanager workload +# itself is provisioned under monitoring/alertmanager. +alerting: + alertmanagers: + - static_configs: + - targets: + - alertmanager:9093 scrape_configs: + # The authoritative cutover-roster fleet collector. It exposes the + # performance_cutover_* metrics the cutover-roster rules alert on. The + # keep-discovered-nodes job below scrapes individual nodes and does NOT cover + # these fleet metrics, so this dedicated job is their committed source. + - job_name: cutover-roster + honor_timestamps: true + metrics_path: /metrics + scheme: http + static_configs: + - targets: + - cutover-roster:9701 - job_name: keep-discovered-nodes honor_timestamps: true metrics_path: /metrics diff --git a/infrastructure/kube/keep-prd/monitoring/prometheus/config/rules.yaml b/infrastructure/kube/keep-prd/monitoring/prometheus/config/rules.yaml index 668044bd92..a008d5bee9 100644 --- a/infrastructure/kube/keep-prd/monitoring/prometheus/config/rules.yaml +++ b/infrastructure/kube/keep-prd/monitoring/prometheus/config/rules.yaml @@ -50,3 +50,59 @@ groups: breakdown (performance_network_join_requests_failed_*_total) to tell genuine non-recognition (firewall_unrecognized) apart from firewall RPC errors, timeouts, and connection resets. + + # Cutover-readiness fleet alerts, emitted by the cutover-roster collector + # (pkg/monitoring/cutoverroster). These mirror cutoverroster.AlertRules(), + # which is the programmatic source of truth; keep the two in sync. Both fire + # only after two consecutive one-minute evaluations and are routed to the + # Release and Operator Coordination teams. + - name: cutover-roster + rules: + - alert: CutoverBlockingOperatorsPresent + expr: performance_cutover_fleet_blocking_operators > 0 + for: 2m + labels: + severity: critical + team: release + route_to: release,operator-coordination + annotations: + summary: Cutover-eligible operators remain in a blocking status. + description: >- + One or more authoritative operators are not exact-R1 or + independently quarantined. Cutover readiness is not met. + - alert: CutoverRosterIncomplete + expr: >- + performance_cutover_fleet_blocking_operators > 0 + or performance_cutover_reporters_stale > 0 + or performance_cutover_inventory_unreconciled > 0 + for: 2m + labels: + severity: warning + team: release + route_to: release,operator-coordination + annotations: + summary: Cutover fleet roster is incomplete. + description: >- + Blocking operators, stale reporters, or unreconciled inventory are + present. The go/no-go completeness criteria are not met. + # A dead or unscraped collector makes every performance_cutover_* series + # vanish, so the two alerts above would evaluate absent and never fire. This + # up/absent() alert catches that hole: it fires when the collector target is + # down (up == 0) or has disappeared from the scrape config (absent). + - alert: CutoverRosterCollectorDown + expr: >- + up{job="cutover-roster"} == 0 + or absent(up{job="cutover-roster"}) + for: 2m + labels: + severity: critical + team: release + route_to: release,operator-coordination + annotations: + summary: Cutover-roster collector scrape target is down or absent. + description: >- + Prometheus cannot scrape the cutover-roster collector (job + cutover-roster): the target is down or has disappeared from the scrape + config. Every performance_cutover_* series is therefore stale or + absent, so the other roster alerts can be silently absent. Cutover + readiness cannot be evaluated until the collector is restored. diff --git a/keep-core-release/threshold-network/keep-common/16.md b/keep-core-release/threshold-network/keep-common/16.md new file mode 100644 index 0000000000..e0ae07e154 --- /dev/null +++ b/keep-core-release/threshold-network/keep-common/16.md @@ -0,0 +1,64 @@ +# PR #16 — `fix(codegen): point //go:linkname at abigen for go-ethereum v1.16+` + +- **Repo:** threshold-network/keep-common +- **Branch:** `fix/go-ethereum-1.17-linkname` → `main` +- **URL:** https://github.com/threshold-network/keep-common/pull/16 +- **Status:** open, no reviews, no comments +- **Diff size:** 6 files, +295 / -151 (mostly `go.sum`) +- **Tag target:** `v1.7.1-tlabs.1` + +## What this PR is + +`keep-common` is a **Go library** consumed by `keep-core`. It contains no smart contracts, no long-running services, and no operator-facing binaries beyond the build-time `tools/generators/ethereum` codegen tool. Its release artifact is a Git tag that downstream modules vendor. + +Functional payload: + +1. `tools/generators/ethereum/contract_parsing.go` — re-point three `//go:linkname` directives from the deprecated `accounts/abi/bind` package (v1.15 and earlier) to the renamed/moved `accounts/abi/abigen` package (v1.16+). +2. `go.mod` / `go.sum` — bump `github.com/ethereum/go-ethereum` to **v1.17.3** (the version downstream needs for CVE remediation in tlabs-xyz/keep-core-security#13). +3. `.github/workflows/{client,release}.yml` — read Go version from `go.mod` instead of pinning `1.22`. +4. `CHANGELOG.md` — Unreleased entry noting the Go 1.24 toolchain bump. + +## Breaking changes + +| Surface | Breaking? | Detail | +|---|---|---| +| Public Go API of `keep-common` | No | No exported types, functions, methods, or signatures change. The linkname helpers (`bindStructTypeGo`, `bindTopicTypeGo`, `structured`) are package-private to `tools/generators/ethereum` and not importable. | +| Smart contracts / ABIs / events | N/A | This repo ships no contracts. | +| Generated contract bindings | No | Bindings are regenerated by downstream consumers from their own ABIs; codegen tool still emits the same shape. | +| Runtime behavior of `pkg/chain/ethereum/ethutil/*` | No direct change in this PR | No source edits in `pkg/`. Behavior *can* differ because the underlying `go-ethereum` dep moves from 1.13.x → 1.17.3 (gas estimation, RPC client, `bind.TransactOpts` defaults). That delta is owned by the downstream consumer PR (keep-core-security#13), not this one. | +| Minimum Go toolchain | **Yes** | `go.mod` declares `go 1.24.0` (was `1.18`). Forced by go-ethereum v1.17.3, which itself declares `go 1.24.0`. Consumers building from source need Go 1.24+ available. | +| Transitive `go-ethereum` version | **Yes (transitively)** | Consumers pinning go-ethereum at 1.13.x in their own `go.mod` will be forced to bump to ≥ 1.17.3 once they pull this tag. New transitive deps include `github.com/holiman/uint256`, `github.com/supranational/blst`, and `github.com/ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime`. | + +## Deployment / redeployment + +| Component | Action required | +|---|---| +| Smart contracts | **None.** No contracts in this repo. | +| `keep-core` nodes (client / beacon) | **No redeploy required for this PR alone.** This PR is just the tag. Redeploy happens when the downstream PR (keep-core-security#13) bumps its `keep-common` dep to `v1.7.1-tlabs.1`, regenerates bindings, rebuilds, and that node release is cut. | +| Off-chain services / monitoring | **None for this PR.** Same reasoning. | +| CI / build infrastructure | Build hosts and developer machines must have **Go 1.24+** installed. GitHub Actions runners get this automatically via `actions/setup-go` + `go-version-file: go.mod`; self-hosted runners and local devs may need to update. | + +## Release safety + +Safe to tag `v1.7.1-tlabs.1` from this branch once merged. Caveats: + +1. **This PR is a library tag, not a deployment.** Operational risk lives in the downstream consumer (keep-core-security#13), which is the PR that actually swaps `go-ethereum` 1.13 → 1.17 in the running node binary. Reviewing that PR's runtime delta (gas estimation, RPC client behavior, block-header API which already landed in keep-common commit fd88e7b) is where the real risk-assessment effort should go. +2. **Codegen tool's linkname approach remains fragile.** Acknowledged in the PR body. The local declarations use `map[string]struct{}` while upstream uses `map[string]*tmplStruct`. This works because both map values are pointer-sized and the linker doesn't type-check linkname targets — but any future upstream signature change (added param, value-type swap, return-tuple change) will produce silent memory corruption at codegen time, not a build error. Worth filing a follow-up to either inline the helpers (~30 LoC each) or switch to the public `abigen.Bind` entrypoint. +3. **No verification beyond compile/link.** This PR was validated by `go build ./tools/generators/ethereum/...` (clean), `go vet` (clean), full link of the codegen binary against v1.17.3 (succeeds), and the full `go build ./...` of the keep-common module (clean). No generated-bindings diff was inspected — the assumption is that downstream's `make generate` step in keep-core-security#13 will produce equivalent output and any divergence will surface there. + +## Review scrutiny — findings worth acting on + +Reviewed against a multi-agent review pass; PR has zero external review comments. Findings filtered to valid items only: + +- **Linkname signature drift hazard** — real and acknowledged. Recommended follow-up issue, not a blocker. +- **Go 1.24 toolchain floor not surfaced** — addressed by the CHANGELOG entry added in this PR. + +Dismissed: "lower `go.mod` to 1.23" (impossible — upstream forces 1.24), doc-comment wording nits, `ProjectZKM/Ziren` transitive dep concern (out of scope; belongs to downstream security PR's review surface). + +## Bottom line + +- **No contract redeploy.** No contracts in this repo. +- **No node redeploy from this PR.** Node redeploy is triggered by the downstream consumer PR (keep-core-security#13), not by tagging keep-common alone. +- **No service redeploy from this PR.** +- **Breaking changes are confined to build-time:** Go 1.24 floor, and transitive go-ethereum version bump for any consumer that re-vendors this tag. +- **Safe to tag and release `v1.7.1-tlabs.1`.** All real deployment risk transfers to the downstream PR consuming this tag. diff --git a/keep-core-release/threshold-network/keep-common/17.md b/keep-core-release/threshold-network/keep-common/17.md new file mode 100644 index 0000000000..2f481f03e0 --- /dev/null +++ b/keep-core-release/threshold-network/keep-common/17.md @@ -0,0 +1,72 @@ +# PR #17 — `fix: clear gosec 2.24.7 findings and stabilize Go 1.24 / go-ethereum 1.17 CI` + +- **Repo:** threshold-network/keep-common +- **Branch:** `fix/gosec-findings` → `main` +- **URL:** https://github.com/threshold-network/keep-common/pull/17 +- **Status:** open, CI green (`client-build-and-test`, `client-scan`, `client-lint` all SUCCESS), `MERGEABLE / CLEAN` +- **Diff size:** 7 files, +18 / -9 +- **Tag candidate:** `v1.7.1-tlabs.2` (next patch after the upcoming `v1.7.1-tlabs.1`) + +## What this PR is + +A bundled cleanup PR. Has three logical parts: + +1. **gosec 2.24.7 findings (3 fixes).** Surfaced by running the bumped scanner against current main. +2. **CI stabilization after PR #16 (4 fixes).** Pre-existing Go 1.24 `go vet` failures and a mock-interface drift from the go-ethereum v1.17.3 bump. +3. **gosec action bump (1 fix).** Supersedes PR #15. Necessary because gosec v2.19.0's Docker image ships Go 1.21.3, which can't load this module's `go 1.24.0` go.mod. + +`keep-common` is a **Go library** consumed by `keep-core`. No smart contracts, no long-running services. Release artifact is a Git tag. + +## Breaking changes + +| Surface | Breaking? | Detail | +|---|---|---| +| Public Go API of `keep-common` | No | No exported types, functions, methods, or signatures change. | +| Smart contracts / ABIs / events | N/A | No contracts in this repo. | +| Generated contract bindings | No | Codegen tool emits the same shape (only internal vet/format-string fixes). | +| Runtime behavior — `BlockCounter` | **Yes, narrow** | `block_counter.go` switches `strconv.ParseInt(_, 0, 32) + uint64()` → `strconv.ParseUint(_, 0, 64)`. Two behavior deltas: (a) negative-string inputs now produce a parse error and are skipped instead of being silently converted to a huge `uint64`; (b) block numbers above `MaxInt32` (~2.15B) are now accepted instead of failing the parse. Neither delta is reachable today: `block.Number` is always populated from `(*big.Int).String()` of an Ethereum block header, which is non-negative and well below `MaxInt32` for the next ~800 years. **No observable behavior change in practice.** | +| Runtime behavior — disk persistence | **Yes, forward-only** | `disk_persistence.go` tightens `EnsureDirectoryExists` from `os.ModePerm` (0o777) → `0o750`. Affects newly-created node data and keystore directories. **Existing deployed nodes keep their current 0o777 permissions on already-created directories.** Same-group access still permitted (backup, sidecar). "Other" excluded. | +| Test code | No | Mock interface satisfaction; format-string vet fixes. Test-only. | +| CI workflow (`securego/gosec@v2.24.7`) | No | CI-internal. Will surface more analyzer rules on future PRs. | +| Minimum Go toolchain | No change from #16 | Already at `go 1.24.0` from PR #16. | +| go-ethereum version | No change | Still v1.17.3. | + +## Deployment / redeployment + +| Component | Action required | +|---|---| +| Smart contracts | **None.** No contracts in this repo. | +| `keep-core` nodes (client / beacon) | **No redeploy required for this PR alone.** Library tag only. Redeploy happens when downstream PR (tlabs-xyz/keep-core-security#13 or its successor) bumps its `keep-common` dep to the new tag and that node release is cut. | +| Off-chain services / monitoring | **None.** | +| CI infrastructure | None. The gosec action bump is self-contained (action pulls its own Docker image with Go toolchain). | +| Existing node persistence directories | **No action required.** New permissions (0o750) apply only to directories created after the bump lands in a deployed node. Operators who want consistency on existing dirs can `chmod 0750 ` manually, but it's not required for correctness. | + +## Release safety + +Safe to tag `v1.7.1-tlabs.2` from this branch (after the planned `v1.7.1-tlabs.1` from #16 is cut). Caveats: + +1. **Library tag, not a deployment.** Runtime risk lives in the downstream consumer's adoption PR, not here. The two behavior deltas (block counter parser, dir perms) only take effect once a `keep-core` release consumes this version. +2. **Block counter behavior delta is theoretically observable but practically inert.** The "rejects negative inputs" and "accepts > MaxInt32" branches require malformed RPC responses or block numbers ~800 years in the future, neither of which is reachable. Worth one sentence in the changelog so operators know the parser changed, but no operator-facing action needed. +3. **Directory permission tightening is forward-only.** New keystore/work dirs created on fresh node deployments will be 0o750. Existing operators with custom UID/group setups for backup or sidecar access should verify their access pattern uses the same group as the node process. Worth a CHANGELOG note for operators. +4. **gosec action bump may surface new findings on future PRs.** v2.24.7 added ~10 new analyzers (G117 expanded, G118-G123, G408, G705, G707). This PR cleared the three findings present on current main, but future PRs touching code those analyzers cover may produce new flags. Operationally manageable — triage as they appear. + +## Review scrutiny — findings worth acting on + +CI green, all checks pass. No external reviews on the PR yet. Self-scrutinized via: + +- Multi-agent review pattern applied to each gosec finding individually before fixing. +- Advisor scrutiny called before each fix. +- Local gosec v2.24.7 against the full repo: `Issues: 0`. +- `go vet ./...` clean on Go 1.24. +- `go test ./...` — 342 pass, 1 pre-existing flake in `pkg/clientinfo` (passes on retry; not introduced here). + +**Nothing further to act on before merge.** + +## Bottom line + +- **No contract redeploy.** No contracts in this repo. +- **No node redeploy from this PR alone.** Node redeploy is triggered by the downstream consumer adopting the resulting tag. +- **No service redeploy from this PR.** +- **Two narrow runtime deltas (block counter parser, dir perms 0o750)** — both forward-only and operationally inert for current deployments. +- **Supersedes PR #15** (gosec action bump bundled here); close #15 as obsolete after this lands. +- **Safe to tag and release `v1.7.1-tlabs.2`** after #16's `v1.7.1-tlabs.1` is cut. diff --git a/keep-core-release/threshold-network/keep-core/3945.md b/keep-core-release/threshold-network/keep-core/3945.md new file mode 100644 index 0000000000..61aec591e3 --- /dev/null +++ b/keep-core-release/threshold-network/keep-core/3945.md @@ -0,0 +1,62 @@ +# PR #3945 — `refactor(wallet-registry): update withdrawRewards` + +- **Repo:** threshold-network/keep-core +- **Branch:** `feat/walletRegistry-withdraw` → `main` +- **URL:** https://github.com/threshold-network/keep-core/pull/3945 +- **Status:** approved (lrsaturnino), no review comments +- **Diff size:** 3 files, +81 / -90 + +## What changes on-chain + +Exactly one runtime line in `solidity/ecdsa/contracts/WalletRegistry.sol:474`: + +```solidity +- (, address beneficiary, ) = staking.rolesOf(stakingProvider); ++ (, address beneficiary, ) = _currentAuthorizationSource().rolesOf(stakingProvider); +``` + +All other changes are NatSpec rewrites and Hardhat tests. No state variables, no function signatures, no events, no errors changed. + +## Breaking changes + +| Surface | Breaking? | Detail | +|---|---|---| +| ABI / function selectors | No | `withdrawRewards(address)` signature unchanged | +| Events | No | `RewardsWithdrawn(stakingProvider, amount)` unchanged | +| Storage layout | No | `allowlist` was added by the prior V2 upgrade; this PR adds nothing | +| Go client bindings | No | Generated bindings in `pkg/chain/ethereum/.../WalletRegistry.go` consume the same ABI | +| User-visible behavior | Yes, bounded | After `initializeV2`, beneficiary resolves via `Allowlist.rolesOf` (returns `stakingProvider`) instead of `TokenStaking.rolesOf` (could return a delegated beneficiary). On chains where `allowlist == 0`, behavior is identical to before. ECDSA application rewards are HALTED per TIP-092, so realized blast radius is near zero unless rewards are reactivated. | + +## Deployment / redeployment + +| Component | Action required | +|---|---| +| **WalletRegistry contract** | **YES — requires a proxy upgrade.** Deploy a new implementation and execute `ProxyAdmin.upgrade(proxy, newImpl)`. Do **not** use `upgradeAndCall` with `initializeV2` again — `initializeV2` is `reinitializer(2)` and is already consumed on chains where V2 shipped. The existing `solidity/ecdsa/deploy/17_upgrade_wallet_registry_v2.ts` script is for the *initial* V2 upgrade and short-circuits when `allowlist` is already set; a new script (or one-shot `cast send`) is needed for this implementation swap. On mainnet, the upgrade still goes through the 24h Timelock. | +| Allowlist contract | No change | +| RandomBeacon / sortition pool | No change. RandomBeacon was never migrated to TIP-092 dual-mode authorization (no `_currentAuthorizationSource()`, no `allowlist`); this PR intentionally only touches WalletRegistry. | +| Keep-core nodes (Go client) | **No redeploy.** ABI identical; the node never calls `withdrawRewards` itself and doesn't subscribe to a new event. | +| Off-chain services / monitoring | **No redeploy.** No event/selector changes. | + +## Release safety + +Safe to release on its own. Caveats: + +1. **Operator comms:** any staker who had `beneficiary != stakingProvider` configured in TokenStaking should be told that after this upgrade lands their `withdrawRewards` payouts (if any) go to `stakingProvider` going forward. Per the PR description, ECDSA rewards are halted, so this is forward-looking documentation, not a refund issue. +2. **Upgrade transaction is not in this PR.** The PR contains only code/tests; the operational steps to push the new implementation onto Sepolia/mainnet are not included. +3. **One-line scope** means the audit/canary risk is tiny — same authorization source pattern used by `joinSortitionPool`, `updateOperatorStatus`, etc. throughout the contract. + +## Review scrutiny — findings worth acting on + +Reviewed against a multi-agent review pass and PR comments. **No findings warrant action:** + +- "RandomBeacon parity" claim — **invalid.** RandomBeacon has no `_currentAuthorizationSource()` and no `allowlist` (grep returns 0 matches). It was never migrated to TIP-092 dual-mode authorization, so there's no sister-contract divergence. +- "Silent beneficiary redirection" — **already documented** in the PR body's auditor/reviewer notes. Adding `beneficiary` to the `RewardsWithdrawn` event would break event ABI for a refactor-only PR. +- "Missing TokenStaking-branch test" — **invalid.** Pre-existing tests at `solidity/ecdsa/test/WalletRegistry.Rewards.test.ts:107–122` already cover the default-fixture (TokenStaking) branch. +- "`.to.be.gt(0)` is weak" — the existing TokenStaking test at line 110 uses the same pattern; tightening only the new test creates inconsistency. +- NatSpec wording nits — cosmetic; current wording already conveys exclusivity. + +## Bottom line + +- **Contract upgrade required:** new `WalletRegistry` implementation + `ProxyAdmin.upgrade` (no re-init). +- **No node, service, or other contract redeploy required.** +- **No client-side breaking change.** Behavioral change exists but is forward-looking under current halted-rewards state. diff --git a/keep-core-release/threshold-network/keep-core/3948.md b/keep-core-release/threshold-network/keep-core/3948.md new file mode 100644 index 0000000000..aa7431b1aa --- /dev/null +++ b/keep-core-release/threshold-network/keep-core/3948.md @@ -0,0 +1,59 @@ +# PR #3948 — Breaking change & redeploy safety analysis + +PR: https://github.com/threshold-network/keep-core/pull/3948 +Title: Harden non-sensitive validation paths +Branch: `codex/non-sensitive-hardening-fixes` → `main` + +## Verdict + +**No source-level breaking changes**, but two areas need coordinated rollout, and the Solidity fix has non-trivial deployment implications and an unaddressed sibling-contract bug. + +## Go API surface + +- All `pkg/*` changes either tighten error returns on functions that already return `error`, or shrink/bound caches. **No exported signature changes.** +- Removed constant `PositiveIsRecognizedCachePeriod` is not imported anywhere outside `pkg/firewall` (verified) — safe. + +## Coordination-sensitive change — `pkg/tecdsa/retry/retry.go` + +This is the highest-risk runtime change. `excludeOperatorTriplets` is consumed by `EvaluateRetryParticipantsForSigning` in `pkg/tbtc/signing_loop.go:491`. All operators rerun the same selection over a seeded RNG and must agree on the eligible triplet set to converge on the same retry participants. + +- Old code counted the middle operator's seats twice and ignored the right operator's seats. The set of "eligible" triplets therefore differs between old and new versions whenever the middle and right operators have different seat counts. +- **Mixed-version operator fleets can disagree on which triplet to exclude**, leading to split retry attempts that fail to reach the honest threshold. +- Action: roll out across the operator set in a coordinated window, not gradually. Treat like a consensus-affecting upgrade. + +## Retransmission cache bounding — `pkg/net/retransmission/retransmission.go` + +- Cache is per `Recv()` handler (verified in `pkg/net/libp2p/channel.go:160`), not global. 10k IDs per subscription is generous relative to per-phase message volumes. +- Theoretical regression: a stale retransmission whose ID was evicted from the cache would be re-handled. For idempotent protocol handlers this is harmless; if any downstream handler is not idempotent, it could double-process. Worth a quick scan if you want, but the protocols are designed around retransmission tolerance. + +## Firewall — `pkg/firewall/firewall.go` + +Not breaking, but operationally meaningful: + +- Recognized peers now hit `IsRecognized` (chain RPC) on every reconnect/validation instead of once per 12h. **Watch ETH RPC quota/latency after rollout.** +- The negative cache only rate-limits repeated misses *for the same key*; many distinct unknown peer keys can still cause sustained RPC load. PR description calls this out explicitly. + +## Contract — `solidity/ecdsa/contracts/WalletRegistryGovernance.sol` + +`WalletRegistryGovernance` is `Ownable`, **not upgradeable** (no proxy, no UUPS/initializer). Redeploy implications: + +1. Deploy a new `WalletRegistryGovernance` instance. +2. The deployed `WalletRegistry` (mainnet) currently has the old governance as owner — confirmed by `solidity/ecdsa/deployments/mainnet/WalletRegistryGovernance.json`. Ownership must be transferred from old governance to new governance, which itself requires calling the old governance's transfer flow (subject to its governance delay). +3. **Any in-flight governance proposals (pending parameter changes) in the old contract are lost** — their state is in the old instance's storage and does not migrate. +4. If `finalizeAuthorizationDecreaseDelayUpdate` was ever called against the live `WalletRegistry`, the on-chain `authorizationDecreaseChangePeriod` was overwritten with the previous delay value and remains corrupt until corrected via a fresh `beginAuthorizationDecreaseChangePeriodUpdate` → finalize cycle. **Check on-chain history before assuming nothing needs remediation.** + +## Sibling-contract bug not fixed + +`solidity/random-beacon/contracts/RandomBeaconGovernance.sol` has the **identical destructuring bug** in its `finalizeAuthorizationDecreaseDelayUpdate`. This PR does not touch it. `RandomBeaconGovernance.json` exists in mainnet deployments. Recommend a follow-up fix + redeploy plan for the random-beacon side as well, or the same data-corruption hazard remains live there. + +## Deploy script behavior change + +`solidity/ecdsa/deploy/16_initialize_allowlist_weights.ts` now `throw`s on ownership-transfer failure. Environments that previously appeared to deploy "successfully" while logging a warning will now fail loudly. Correct behavior, but expect any latent permission misconfig to surface on the next deploy. + +## Action checklist before redeploying + +- [ ] Coordinate the Go binary rollout across the operator fleet in a single window (driven by the retry-eligibility fix). +- [ ] Plan the `WalletRegistryGovernance` redeploy: new instance + old-governance ownership transfer to it + replay any in-flight pending changes. +- [ ] Check whether `finalizeAuthorizationDecreaseDelayUpdate` was ever invoked on the live `WalletRegistry`; if so, schedule a corrective change-period update. +- [ ] File a follow-up to patch the identical bug in `RandomBeaconGovernance.sol` (mainnet) and plan its redeploy. +- [ ] Capacity-plan ETH RPC headroom for the firewall change before flipping the new binary on validators. diff --git a/keep-core-release/threshold-network/keep-core/3952.md b/keep-core-release/threshold-network/keep-core/3952.md new file mode 100644 index 0000000000..493cfe499d --- /dev/null +++ b/keep-core-release/threshold-network/keep-core/3952.md @@ -0,0 +1,90 @@ +# PR #3952 — Breaking change & redeploy safety analysis + +PR: https://github.com/threshold-network/keep-core/pull/3952 +Title: test: comprehensive test coverage audit remediation +Branch: `test/audit-coverage` → `main` +Head SHA at analysis time: `aaf2b5baf197a99419768a98a131c9bb078403e1` + +## Verdict + +**No breaking changes. No coordinated rollout required.** Operator nodes can be upgraded on an individual cadence — old and new versions interoperate without protocol divergence. No contract or service redeploys. + +## Surface area + +- **Production Go code:** 8 files, ~91 lines changed (3 bug fixes, 1 test-path defensive fix, 4 refactors) +- **Tests:** ~3.7k lines, all additive +- **Solidity:** zero `.sol` source changes; one Hardhat gas-estimate constant in `WalletRegistry.Inactivity.test.ts` (`1_240_000 → 1_175_000`) +- **Workflows:** `client.yml` only — adds Go coverage profile + artifact upload; broadens integration-test job's `needs:` +- **Build/deps:** no `go.mod`, `go.sum`, `Dockerfile`, `Makefile`, or env-var changes + +## Go API surface + +- No exported function signature changes in any modified file +- `BackoffStrategy` struct gains a private `sync.Mutex` field — additive, zero-value safe; positional struct literals are disallowed for structs with unexported fields outside their own package, so no caller breaks +- New types `pubsubSubscription` (interface) and `snapshotQueueSizes` (method) in `pkg/net/libp2p/channel.go` are private (lowercase) — no external impact + +## Production bug fixes (operator-visible behavior changes) + +### 1. `pkg/tbtc/signing_done.go` — map data race in `signingDoneCheck` +- Pre-fix: `waitUntilAllDone` read `len(sdc.doneSigners)` and iterated the map without holding `doneSignersMutex`, while `onDoneMessage` (writer) held the lock. Real Go map race. +- Worst-case symptom in production: `fatal error: concurrent map iteration and write` crash, or silent corruption of the signature-aggregation loop. +- Fix: both read and write paths now serialize through `doneSignersMutex`. + +### 2. `pkg/tbtc/inactivity.go` — TOCTOU in `SubmitClaim` +- Pre-fix: nonce read once before the per-member delay wait; if a peer submitted during the wait, the loser broadcast a doomed tx that the chain rejected with `wrong inactivity claim nonce`. +- Fix: re-read `GetInactivityClaimNonce` after the wait; abort with a log line if it advanced. +- Net effect: one extra read-only `eth_call` per submission attempt; doomed txs no longer broadcast; alerting noise reduced. + +### 3. `pkg/tbtc/dkg_submit.go` — TOCTOU in `SubmitResult` (added in this branch's final commit) +- Same class of bug as #2, in DKG result submission. Surfaced during multi-agent review of the PR. +- Pre-fix: DKG state read once before the per-member delay wait; loser broadcast a tx that the chain rejected with `not awaiting DKG result`. +- Fix: re-read `GetDKGState` after the wait; abort if state moved away from `AwaitingResult`. +- Regression tests cover both fixes via a hooked `waitForBlockFn` that simulates a competing submission landing during the wait window. + +### 4. `pkg/net/retransmission/strategy.go` — `BackoffStrategy` data race +- Pre-fix: concurrent `Tick` callbacks (overlapping when `retransmitFn` was slow) mutated `tickCounter`, `delay`, `retransmitTick` with no synchronization. +- Fix: `sync.Mutex` guards counter mutation; released before `retransmitFn()` so the I/O call doesn't block other ticks. + +### 5. `pkg/chain/local_v1/blockcounter.go` — `closeOnce` defensive guard +- Test-only code path (`local_v1`). Prevents `close of closed channel` panic when context cancellation races with watcher iteration. No production impact. + +## Refactors (no behavior change) + +- `pkg/net/libp2p/channel.go`: extracts `pubsubSubscription` interface and `snapshotQueueSizes` method for test injection. Metric output is byte-identical. +- `pkg/net/retransmission/retransmission.go`: removes a redundant outer goroutine around `ticker.onTick(...)`; comment now documents the synchronous-registration invariant. +- `pkg/tbtcpg/internal/test/marshaling.go`: `fmt.Errorf(s)` → `errors.New(s)` lint fix in test fixture. + +## On-chain / protocol impact + +- Zero contract source changes → **no contract redeploy**. +- Zero gossipsub / wire-format / RPC-interface changes → **mixed-version operator fleet works without divergence**. This is *not* a consensus-affecting upgrade (contrast with PR #3948's `pkg/tecdsa/retry` change). +- Two new chain reads added (one `GetDKGState`, one `GetInactivityClaimNonce`) per losing submission attempt — read-only `eth_call`s, negligible RPC load. + +## Deployment recommendations + +- **Contracts: no action.** +- **Services / coordinators: no action.** +- **Operator nodes: upgrade recommended, not required.** + - Quality-of-life and correctness win, especially the `signing_done` map race (only finding with crash potential). + - No forced upgrade window. Operators can roll on their own cadence; heterogenous network is safe. + +## Risk + +Low. All production behavior changes fail closed: a member that detects a lost race returns `nil` and logs an info line rather than producing a noisy chain rejection. The mutex additions serialize previously-unsynchronized state; no new lock-ordering concerns (no nested locks introduced; `BackoffStrategy.mu` is released before the user-supplied `retransmitFn` runs). + +## CI + +All required checks green on the merge commit `aaf2b5baf`: + +- Client ✓ +- Solidity ECDSA ✓ +- Solidity ECDSA docs ✓ +- Solidity Random Beacon ✓ +- Solidity Random Beacon docs ✓ + +PR mergeable; `mergeStateStatus: BLOCKED` reflects required review approval (the prior `LGTM` was dismissed after later commits landed). + +## Caveats + +- The PR description claims removal of whole-project Go coverage gates and `continue-on-error` from contracts coverage jobs. Neither is in the diff vs `main` — those CI-discipline changes were apparently descoped or landed elsewhere. The actual workflow change is limited to adding Go coverage artifact upload. +- Race-detector failures in `pkg/net/retransmission/ticker_test.go` and `pkg/chain/local_v1/blockcounter_test.go` exist on this branch tip but are pre-existing on `main` and unrelated to this PR; CI does not run with `-race`, so they don't block the workflow. diff --git a/keep-core-release/threshold-network/tss-lib/4.md b/keep-core-release/threshold-network/tss-lib/4.md new file mode 100644 index 0000000000..ef4fde84e5 --- /dev/null +++ b/keep-core-release/threshold-network/tss-lib/4.md @@ -0,0 +1,107 @@ +# PR #4 — Breaking change & redeploy safety analysis + +PR: https://github.com/threshold-network/tss-lib/pull/4 +Title: Backport tBTC-relevant BNB v4 hardening +Branch: `codex/bnb-332-tbtc-fixes` → `integrate-bnb-hardening` (stacked on PR #2) +Head SHA at analysis time: `5e9e99e9dcd71fa6ec92bec5913546663c343e04` +Status: Draft + +## Verdict + +**This PR by itself: no operator-visible breaking changes; it only tightens validation against malformed peers.** Honest peers running the base branch (PR #2) and honest peers running PR #4 interoperate cleanly. + +**The combined PR #4 + base PR #2 release: yes, wire-format breaking, coordinated rollout required.** This is dictated entirely by base PR #2 (tagged challenges, session context, `SessionNonce` requirement), not by PR #4. PR #4 cannot ship without PR #2 because it is stacked on it. + +No contract redeploy. Library-only change. + +## Surface area + +- **Production Go code:** 12 files, +186/-43 lines +- **Tests:** 6 files, +286 lines (all additive: factor proof, schnorr, vss, dlnproof, ecpoint, keygen messages, resharing messages, round_9 helper) +- **Solidity / contracts:** none — `tss-lib` is a pure Go cryptography library; no on-chain artifacts +- **Workflows:** none modified +- **Build / deps:** no `go.mod`, `go.sum`, `Makefile` changes; no new dependencies +- **Commit count:** 5 (3 authored by `maclane@nucypher.com`, 2 added during this review by `piotr@tnetworklabs.com`) + +## Diff by area + +| File | Change | Operator impact | +|---|---|---| +| `crypto/dlnproof/proof.go` | Verifier: nil h1/h2/N guards; collapsed `Alpha ∈ (1, N)` check (drops prior `Mod(Alpha, N)` normalization) | Rejects malformed peers sending un-reduced Alpha. Honest peers sample Alpha = `h1^a mod N` ∈ (0, N) so unaffected. | +| `crypto/ecpoint.go` | `ScalarMult`/`ScalarBaseMult` return `nil` on invalid input instead of panicking. Removed unused `ScalarMultErr`/`ScalarBaseMultErr` exports. | Prover paths use `GetRandomPositiveInt(N)` (non-zero) — `nil` never returned in honest flows. Hardens panic-DoS surface. | +| `crypto/mta/proofs.go`, `crypto/mta/range_proof.go` | `S2 > maxS2` → `S2 >= maxS2`; ec/pk/NTilde nil guards; nil result check after `X.ScalarMult(e).Add(pf.U)` made explicit. | `S2 == maxS2` is unreachable in honest provers (`S2 = e·ρ + ρ' < 2·q³·NTilde` strictly). No honest false-reject. | +| `crypto/mta/share_protocol.go` | Adds `mta.ErrRangeProofVerify` sentinel | Pure error attribution. No wire impact. | +| `crypto/paillier/factor_proof.go` | Adds DoS bounds on W1, W2, Sigma, V *before* modular exponentiation. Z1/Z2 bound checks moved earlier. | Honest provers produce values within bounds (verified against CGGMP'21 §28 honest-sampling math). Bounds are looser than tight (4× margin on V), so no honest false-reject. | +| `crypto/schnorr/schnorr_proof.go` | Verifier rejects nil/off-curve points and zero/out-of-range scalars before `ScalarMult` | Honest provers always produce `T ∈ (0, q)` via `RejectionSample(q)`. No honest false-reject. | +| `crypto/vss/feldman_vss.go` | `Share.Verify`: rejects `share == nil`, `share == 0`, `share >= q`, nil vs[j], `ScalarMult` nil result | Honest dealer produces `share = f(id) mod q` ∈ (0, q) with overwhelming probability (`share == 0` only with prob ~2^-256). No practical false-reject. | +| `ecdsa/keygen/messages.go` | `KGRound1Message.ValidateBasic` adds `hasBitLen(PaillierN, 2048) && hasBitLen(NTilde, 2048)` (exact-equality) | Matches the long-standing `BitLen() != 2048` check in `ecdsa/keygen/round_2.go:53,59` and `ecdsa/resharing/messages.go:148`. Local `safe_prime.go` generator guarantees `BitLen == 2048` (top 2 bits of 1023-bit Sophie Germain prime forced), so honest keygens are unaffected. | +| `ecdsa/signing/round_2.go` | Wraps `BobMid` / `BobMidWC` errors via `attributeBobMidErr` closure so peer-attributable range-proof rejections are tagged with the correct culprit. | Error-message improvement; no protocol change. | +| `ecdsa/signing/round_4.go` | Rejects nil `thetaInverse` | Defensive guard; `thetaInverse` is internally derived. No honest false-reject. | +| `ecdsa/signing/round_9.go` | **Logic fix:** `if !ok && len(values) != 4` → `if !ok || len(values) != 4`. Helper `decommitFour` extracted to make the guard testable. | Closes a latent bug where a malicious peer could commit to any number ≠ 4 of secrets, send them as the decommitment, and have round 9 silently take `values[0..3]` as attacker-chosen Uj/Tj coordinates (bypassing the `U==T` integrity check). **No hash collision required.** This bug is not present in BNB upstream either — Threshold caught it independently. | + +## Go API surface + +- **No exported function signature changes** in any production file. +- **Removed exports:** `crypto.ScalarMultErr`, `crypto.ScalarBaseMultErr` (unused public API surface; verified no in-tree callers). +- **New sentinel:** `mta.ErrRangeProofVerify` (exported error variable, additive, returned via `errors.Is`). +- **Behavior change on returned nil:** `ScalarMult` / `ScalarBaseMult` now return `nil` instead of panicking. All in-tree callers either nil-check the result or pass it to a receiver-nil-safe method (`Equals`, `Add` with nil guard, `SetCurve`). External consumers must update if they relied on panic semantics — but the explicit Threshold callers (keep-core, etc.) consume only the high-level keygen/signing APIs, not these primitives directly. + +## On-chain / protocol impact + +- **Contracts: zero.** No `.sol` files in this repo. Nothing to deploy. +- **Wire format:** PR #4 introduces no new field on the wire. It rejects values that were already provably malformed under the protocol spec. Honest peers running PR #2 (without PR #4) produce messages that pass PR #4's stricter `ValidateBasic`. +- **Wire compatibility regression to consider — base PR #2's notice, inherited:** + > "This is a protocol/wire compatibility break for proof transcripts. Proofs whose Fiat-Shamir challenges now use tagged hashing or session context will not verify across mixed old/new versions … Operators should roll this out as a coordinated protocol upgrade." + This statement applies to the combined release. PR #4 does **not** add to the break. +- **Operator-controlled requirement (inherited from PR #2):** All callers must invoke `Parameters.SetSessionNonce()` / `SetSessionNonceBytes()` before starting keygen, signing, or ECDSA resharing. The protocol now fails closed without it. PR #4 does not change this requirement. +- **CGGMP'21 paper vs. implementation:** PR #4's FactorProof W/V/Sigma DoS bounds are stricter than the CGGMP'21 paper specification and the BNB / LFDT-Lockness reference implementations (both bound only Z1, Z2). The added bounds reject pathologically oversized response scalars before modular exponentiation. Strictly more defensive than the reference protocol; cannot reduce interoperability with conformant implementations. + +## Consumer impact (keep-core / tBTC nodes) + +- **keep-core** vendors `tss-lib` via `go.mod`. Bumping the dependency past the PR #2 + PR #4 cut is a coordinated protocol upgrade across the operator set. This is governed by base PR #2's break, not by PR #4. +- **Caller-side breaking source change required by base PR #2:** any keep-core code that constructs `tss.Parameters` and then runs keygen/signing must call `SetSessionNonceBytes(...)` before round 1 starts. If the keep-core integration of PR #2 is already in flight (or merged), PR #4 piggybacks on it with zero additional caller changes. +- **No `mta.ErrRangeProofVerify` adoption required** in keep-core: the existing error-handling path (`tss.Error.Culprits()`) still surfaces the offending peer ID; the new sentinel just improves the wrapped message text. + +## Deployment recommendations + +- **Contracts: no action.** +- **Coordinators / services: no action specific to this PR.** Inherits from PR #2 the requirement to provide a unique `SessionNonce` per ceremony. +- **Operator nodes:** + - PR #4 cannot be released alone; it ships with PR #2. + - When PR #2 lands and the keep-core operator fleet upgrades to a version that vendors it, PR #4 is a free hardening that ships in the same protocol-cut release. No second coordinated rollout. + - Old nodes that do not upgrade past PR #2 will already be incompatible with new ones (per PR #2's wire-format break notice). PR #4 does not widen this gap. +- **Rollback:** PR #4 alone is cleanly revertible (additive defensive checks + one logic fix in a stacked branch). Reverting `round_9.go`'s `||` → `&&` re-opens the latent bug — do not do so without replacing it with an equivalent guard. + +## Risk + +**Low for PR #4 in isolation.** All operator-facing behavior changes either (a) fail closed against malformed peers without affecting honest ones (verified by enumerating the honest sampling ranges and the local safe-prime generator's invariants), or (b) replace a panic with a `nil` return (`ScalarMult`/`ScalarBaseMult`) for inputs that honest paths never produce. + +**Inherited from PR #2: medium.** Wire-format break, coordinated upgrade, new mandatory `SessionNonce` caller contract. Assess separately when PR #2 is gated for release. + +The one real bug-fix in PR #4 (`round_9.go`) closes an exploit that requires: +- A malicious party with a valid keygen share (insider). +- Their commitment-only message in round 7 binding to ≠ 4 secrets. +- A choice of 4 attacker-controlled values that satisfy the `U == T` integrity check. + +Cost to the attacker is free (no hash collision). Outcome is bypassing the round-9 cross-check that ties their per-party `bigVi`/`bigAi` contributions to honest behavior, which the protocol uses to detect provably-deviating signers. Severity: medium for honest-majority assumption; low if the deployment also relies on independent on-chain detection of misbehavior. + +## Tests added in this review pass + +Two commits added in the review session (`9e272cc`, `5e9e99e`): + +- `ecdsa/signing/round_9_test.go` — new file. `TestDecommitFour` with 4 subtests (4 secrets accepted, 3/6 rejected, mismatched commit rejected). Mutation-detectable: reverting `||` → `&&` fails `rejects_three_secrets` and `rejects_six_secrets`. +- `ecdsa/keygen/messages_test.go` & `ecdsa/resharing/messages_test.go` — added `BitLen == 2047` boundary assertions for both `PaillierN` and `NTilde`, pinning the just-below-2048 case alongside the existing `BitLen=1` and `BitLen=2049` cases. + +## CI + +- Workflows `Go-fmt` and `Go Test` run via `workflow_dispatch` (they only auto-trigger for PRs targeting `master`; this PR targets `integrate-bnb-hardening`). +- Manually triggered on the new HEAD `5e9e99e`: + - Go-fmt: ✅ success (run 26333181166) + - Go Test: in progress at time of writing (run 26333180587) +- Local validation passed: `go test ./ecdsa/signing` (13 tests, 10.7s), `go test ./ecdsa/keygen ./ecdsa/resharing` (36 tests, ~9 min). + +## Caveats + +- **PR title understates the scope.** Three of the most impactful changes (`round_9.go` `&&` → `||`, `ECPoint` nil-return contract, FactorProof W/V/Sigma DoS bounds) are Threshold-originated, not present in BNB upstream. The PR body's "Backport BNB v4 hardening" framing is technically a backport-plus, but reviewers should not assume each diff has a BNB precedent. +- **EdDSA paths are touched but not tested adversarially.** PR #4's shared-crypto changes (VSS, Schnorr, ECPoint) flow into the EdDSA keygen/signing/resharing rounds. Honest EdDSA flows are unaffected (curve-correct `ec.Params().N` usage throughout), but no negative-path tests pin the new guards in the EdDSA path. Out of stated PR scope ("tBTC-relevant" = ECDSA), so this is documentation, not a defect. +- **Constant-time Paillier (BNB v4's `EnableConstantTimeOps`) is deliberately not ported.** Treated as a separate side-channel hardening project per the PR body. diff --git a/keep-core-release/threshold-network/tss-lib/5.md b/keep-core-release/threshold-network/tss-lib/5.md new file mode 100644 index 0000000000..a51c1fb266 --- /dev/null +++ b/keep-core-release/threshold-network/tss-lib/5.md @@ -0,0 +1,85 @@ +# PR #5 — Breaking change & redeploy safety analysis + +PR: https://github.com/threshold-network/tss-lib/pull/5 +Title: Remove unused EdDSA and resharing protocols +Branch: `codex/remove-unused-protocols` → `codex/bnb-332-tbtc-fixes` (stacked on PR #4, which is stacked on PR #2) +Head SHA at analysis time: `1b42437e49a5216a95b51c13df22cc23b7e78604` +Status: Approved (1 review) + +## Verdict + +**This PR by itself: no wire-format breaking changes, no caller-visible runtime breaking changes for keep-core-security.** It removes unused Go packages, exported symbols, proto definitions, and two Go module dependencies. Per the downstream audit recorded in `BNB_HARDENING_INTEGRATION.md`, `keep-core-security` imports only `ecdsa/keygen`, `ecdsa/signing`, and shared `common`/`crypto`/`tss` packages — none of the removed surface — so no source changes are required in `keep-core-security`. The Go binary still must be rebuilt against the new `tss-lib` version. + +**The combined PR #5 + base PR #4 + base PR #2 release: yes, wire-format breaking, coordinated rollout required.** The break is dictated entirely by base PR #2 (tagged challenges, session context, `SessionNonce` requirement). PR #5 does not widen this gap. + +No contract redeploy. Library-only change. + +## Surface area (PR #5 commits only, not the cumulative stack) + +- **Production Go code:** 35 files deleted (entire `eddsa/{keygen,signing,resharing}` and `ecdsa/resharing` package trees); 4 surviving production files touched (`tss/curve.go`, `tss/params.go`, `tss/message.go`, `tss/message.pb.go`, `crypto/ecpoint.go`) +- **Tests:** `crypto/ecpoint_test.go` — `TestEdwardsEcpointJsonSerialization` replaced with `TestP256EcpointJsonSerialization`. `eddsa/*` test files deleted. `ecdsa/resharing/local_party_test.go` deleted. +- **Test fixtures:** 22 EdDSA keygen fixture JSON files deleted. +- **Proto definitions deleted:** `protob/eddsa-keygen.proto`, `protob/eddsa-signing.proto`, `protob/eddsa-resharing.proto`, `protob/ecdsa-resharing.proto`. `protob/message.proto` retained with fields `is_to_old_committee` (=2) and `is_to_old_and_new_committees` (=5) preserved for wire-layout stability. +- **Solidity / contracts:** none — `tss-lib` is a pure Go cryptography library. +- **Workflows:** none modified. +- **Build / deps:** `go.mod` drops `github.com/agl/ed25519` and `github.com/decred/dcrd/dcrec/edwards/v2`; corresponding `go.sum` entries removed. `Makefile` proto-generation loop shortened to `message signature ecdsa-keygen ecdsa-signing`. +- **Docs:** `README.md` rewritten to scope the fork to ECDSA only; `BNB_HARDENING_INTEGRATION.md` updated with a new "Removed Public Surface" section enumerating deletions for downstream upgraders. +- **Commit count:** 2 (`3284c6b` authored by `maclane@nucypher.com`; `1b42437` doc-cleanup follow-up by `piotr@tnetworklabs.com`). + +## Removed public Go surface + +Compile-time breaking for any consumer that imported these. Per the keep-core-security import audit, none of the below are reachable from keep-core-security. + +| Symbol | Kind | Location | +|---|---|---| +| `github.com/bnb-chain/tss-lib/eddsa/keygen` | package | entire tree deleted | +| `github.com/bnb-chain/tss-lib/eddsa/signing` | package | entire tree deleted | +| `github.com/bnb-chain/tss-lib/eddsa/resharing` | package | entire tree deleted | +| `github.com/bnb-chain/tss-lib/ecdsa/resharing` | package | entire tree deleted | +| `tss.Ed25519` | const `CurveName` | `tss/curve.go` | +| `tss.Edwards()` curve registration | runtime registry entry | `tss/curve.go` `init()` | +| `tss.ReSharingParameters` | struct | `tss/params.go` | +| `tss.NewReSharingParameters` | constructor | `tss/params.go` | +| `crypto.ECPoint.EightInvEight` | method on `*ECPoint` | `crypto/ecpoint.go` | +| `crypto.eight`, `crypto.eightInv` | unexported package-level | `crypto/ecpoint.go` | + +## Retained public surface (deliberate) + +- `tss.Message.IsToOldCommittee()` / `tss.Message.IsToOldAndNewCommittees()` interface methods and the matching `MessageImpl` / wire fields: kept so the `MessageWrapper` proto retains field numbers 2 and 5, preserving wire layout for the generic transport message. This fork never sets either to `true`. `MessageImpl.String()` references `IsToOldCommittee()` in diagnostic formatting and would always render the "(To Old Committee)" branch as absent post-PR-#5. The previously-documented rationale now lives in `BNB_HARDENING_INTEGRATION.md`'s "Removed Public Surface" section. + +## Diff to surviving ECDSA keygen/signing code + +- **None of consequence.** PR #5's only edits to `ecdsa/keygen`, `ecdsa/signing`, `crypto/{dlnproof,mta,paillier,schnorr,vss}`, and `common/` are two test-file lines: `ecdsa/keygen/test_utils.go:1` and `crypto/mta/share_protocol_test.go` (cosmetic). +- **Wire format: unchanged.** All ECDSA keygen and signing message types and round logic are byte-for-byte identical to the PR-#4 base. The wire incompatibility highlighted in `BNB_HARDENING_INTEGRATION.md` is inherited from PR #2 and not extended here. +- **`SessionNonce` contract: unchanged.** The fail-closed-without-`SetSessionNonce` requirement is inherited from PR #2. + +## On-chain / protocol impact + +- **Contracts: zero.** No `.sol` files in this repo. Nothing to deploy. +- **Wire format change in this PR: none.** Removing `eddsa-*.proto` and `ecdsa-resharing.proto` deletes definitions for messages that no longer have a Go producer or consumer in this fork. The four removed `.proto` files are not part of any ECDSA keygen/signing flow. +- **Wire compatibility regression inherited from PR #2:** the protocol/transcript break notice from `BNB_HARDENING_INTEGRATION.md` ("This is a protocol/wire compatibility break for proof transcripts…") still applies to the combined release. PR #5 does **not** add to the break. +- **Operator-controlled requirement (inherited from PR #2):** all callers must invoke `Parameters.SetSessionNonce()` / `SetSessionNonceBytes()` before starting keygen and signing. PR #5 does not change this requirement; the only change is that the previously-applicable note about ECDSA *resharing* fail-closed behavior was removed (correctly — that package no longer exists in this fork). +- **Curve registry runtime behavior:** `tss.GetCurveByName("ed25519")` (if any external caller invokes it) now returns `(nil, false)` after PR #5. Honest tBTC code paths use `tss.S256()` exclusively. + +## Consumer impact (keep-core-security / tBTC nodes) + +- **Source-level breakage in keep-core-security: none, per audit.** PR description states the downstream audit confirmed `keep-core-security` imports only ECDSA keygen/signing plus shared `common`/`crypto`/`tss`. The "Removed Public Surface" section added to `BNB_HARDENING_INTEGRATION.md` gives the precise grep targets to re-confirm before cutting a release. +- **Action required if `keep-core-security` ever did import any of the removed paths:** delete that code path entirely (recommended — it was the unused surface the upstream audit identified), or pin to the pre-PR-#5 tss-lib SHA. The PR is incompatible with retaining EdDSA or ECDSA-resharing call sites downstream. +- **Module-graph cleanup:** `keep-core-security` re-vendoring will drop transitive dependencies on `github.com/agl/ed25519` and `github.com/decred/dcrd/dcrec/edwards/v2`. Verify any `go.sum` reduction matches expectation; no functional impact. +- **`SessionNonce` adoption requirement (inherited from PR #2):** unchanged by PR #5. + +## Deployment recommendations + +- **Contracts: no action.** +- **Coordinators / services: no action specific to this PR.** Inherits from PR #2 the requirement to provide a unique `SessionNonce` per ceremony. +- **Operator nodes:** + - PR #5 cannot be released alone; it ships stacked on PR #4 and PR #2. + - When the keep-core-security operator fleet upgrades to a version that vendors the combined stack, PR #5 contributes attack-surface reduction (smaller binary, two fewer transitive dependencies, no dormant EdDSA or ECDSA-resharing code paths) at zero additional caller cost. + - No second coordinated rollout is needed for PR #5; it piggybacks on the PR #2 protocol-cut release. +- **Rollback:** PR #5 alone is cleanly revertible by re-vendoring the pre-PR-#5 SHA. Operationally, if PR #2/#4 are already deployed, reverting PR #5 only restores dead code and does not change wire behavior. There is no scenario in which reverting PR #5 alone is required for safety. + +## Risk + +**Very low for PR #5 in isolation.** The PR is a pure scope-narrowing deletion. No remaining-protocol logic, wire format, exported call signatures on `ecdsa/keygen` / `ecdsa/signing`, or `SessionNonce` contract is changed. Build/vet/tests pass on `1b42437`. + +**Inherited from PR #2: medium.** Wire-format break, coordinated upgrade, new mandatory `SessionNonce` caller contract. Assess separately when PR #2 is gated for release. PR #5 does not amplify this risk. diff --git a/keep-core-release/threshold-network/tss-lib/6.md b/keep-core-release/threshold-network/tss-lib/6.md new file mode 100644 index 0000000000..dd6ef23892 --- /dev/null +++ b/keep-core-release/threshold-network/tss-lib/6.md @@ -0,0 +1,115 @@ +# PR #6 — Breaking change & redeploy safety analysis + +PR: https://github.com/threshold-network/tss-lib/pull/6 +Title: Address residual review items from BNB hardening stack +Branch: `codex/review-residual-cleanup` → `codex/remove-unused-protocols` (stacked on PR #5, which is stacked on PR #4 / PR #2) +Head SHA at analysis time: `f973d1f` +Status: Open, APPROVED by `piotr-roslaniec` (MEMBER) on `f973d1f`; CI green (Go Test, Go-fmt) + +## Verdict + +**This PR by itself: no wire-format breaking changes, no API breaking changes, no behavior breaking for honest callers.** Six of the eight changed files are pure docstring expansions. The only behavioral change is in `crypto/vss/feldman_vss.go`: `Shares.ReConstruct` now returns explicit errors for malformed input (nil shares, nil/zero share IDs, duplicate IDs) instead of panicking through a downstream `ModInverse(0)` nil dereference. Honest callers passing well-formed shares — the only realistic path — observe identical behavior. + +**No nodes, services, or contracts need to be redeployed for PR #6 in isolation.** + +**The combined stack (PR #6 + PR #5 + PR #4 + PR #2) still requires the coordinated rollout described in `5.md` and `4.md`.** That break is dictated entirely by PR #2 (tagged Fiat-Shamir challenges, session-context binding, fail-closed `SessionNonce`). PR #6 does not widen the gap. + +No contract redeploy. Library-only change. + +## Surface area (PR #6 commits only) + +- **Production Go code:** 1 file with behavior change (`crypto/vss/feldman_vss.go`); 6 files docstring-only (`common/hash_utils.go`, `crypto/ecpoint.go`, `crypto/paillier/factor_proof.go`, `ecdsa/keygen/rounds.go`, `ecdsa/signing/rounds.go`, `tss/params.go`). +- **Tests:** `crypto/vss/feldman_vss_test.go` — new table-driven test `TestReconstructRejectsMalformedShares` pinning each rejection path (nil share, nil ID, nil Share, zero ID mod q, duplicate ID) with `NotPanics` + error assertion. +- **Test fixtures:** none. +- **Proto definitions:** none touched. +- **Solidity / contracts:** none — `tss-lib` is a pure Go cryptography library. +- **Workflows / CI:** none modified. +- **Build / deps:** no `go.mod` or `go.sum` changes. +- **Docs:** no top-level doc files touched; all documentation changes are inline package-level Go comments. +- **Commit count:** 2: + - `c9e9c09` (`maclane@nucypher.com`) — original residual-review-items commit (VSS validation + initial docstring batch). + - `f973d1f` (`piotr@tnetworklabs.com`) — follow-up docstring-only correction to `common/hash_utils.go` `RejectionSample` bias paragraph (replaces the loose `q / 2^eHash.BitLen()` bound and inconsistent example with property-based wording and a cross-reference to `HashToN` / `HashToNTagged`). + +## Public Go surface impact + +**Zero removals. Zero signature changes. Zero new exported symbols.** + +| Symbol | Change | +|---|---| +| `vss.Shares.ReConstruct(ec elliptic.Curve)` | Signature unchanged. New error returns for malformed input that previously panicked. | +| `common.RejectionSample`, `common.LiterallyJustMod` | Behavior unchanged. Docstring expanded. | +| `paillier.FactorChallenge` | Behavior unchanged. Docstring added describing the two challenge-distribution branches. | +| `crypto.ECPoint.SetCurve` | Behavior unchanged. Docstring flags the in-place-mutation footgun. | +| `tss.Parameters.SetSessionNonceBytes` | Behavior unchanged (still panics on `<16` bytes). Docstring expanded with per-ceremony uniqueness + entropy guidance. | +| `ecdsa/{keygen,signing}.(*base).getSSID` (unexported) | Behavior unchanged. Docstring pins the round-1-capture invariant. | + +The only API-observable change is that `vss.Shares.ReConstruct` now returns one of four new error strings on malformed input: +- `"vss reconstruct: nil share"` +- `"vss reconstruct: nil share or share field"` +- `"vss reconstruct: share ID is zero mod q"` +- `"vss reconstruct: duplicate share ID %s"` + +All four cases previously produced a nil-pointer-dereference panic in the Lagrange interpolation loop. Replacing panics with errors is strictly a robustness improvement for any caller that already handled errors from this function. + +## Behavior changes (full enumeration) + +1. **`vss.Shares.ReConstruct` defensive validation (`crypto/vss/feldman_vss.go:133-186`).** Pre-existing latent bug: the prior `if shares != nil && shares[0].Threshold+1 > len(shares)` guard checked the slice header but not its first element, so a `Shares{nil}` or any caller passing a slice with `shares[0] == nil` would nil-deref at `shares[0].Threshold`. Additionally, two shares with identical IDs (or IDs equal mod q) produced a zero Lagrange denominator → `ModInverse(0) == nil` → nil-deref in interpolation. The PR adds: + - Explicit `shares[0] == nil` check before threshold validation. + - Per-share nil-field validation in the dedup loop. + - Zero-ID-mod-q rejection (a zero share ID would encode the secret directly). + - Duplicate-ID-mod-q dedup using `map[string]struct{}`. + Honest callers (well-formed shares generated by `vss.Create`) observe no behavior difference; the new error paths are unreachable for any input that satisfies VSS's own invariants. + +2. **Nothing else.** All other touched files are docstring-only edits. No hash inputs, no domain separators, no message encodings, no field layouts changed. + +## Wire format / protocol impact + +- **Wire format: unchanged.** No proto edits, no struct field edits, no serialization edits. Every byte produced on the wire by ECDSA keygen and signing is identical pre- and post-PR. +- **Fiat-Shamir challenge derivation: unchanged.** `RejectionSample` was only annotated with documentation. The underlying call (`LiterallyJustMod`) is byte-identical. The PR's correction of the `RejectionSample` bias docstring (recommended fix to a math error in the new docstring's bound formula — see the multi-agent review) does not affect any computed challenge value. +- **SSID derivation: unchanged.** `getSSID` in both keygen and signing was only annotated; the hash input list (curve params, party IDs, `round.number`, `ssidNonce`) is byte-identical. +- **Paillier `FactorChallenge`: unchanged.** Both the tagged path (`e ∈ [0, 2^256)` via `SHA512_256i_TAGGED` + modular reduction) and legacy path (`e ∈ [-(2^256-1), 2^256)` via `HashToN(2q-1, …) - (q-1)`) are byte-identical; the PR only documents which absolute-value bounds in `FactorVerify` are present to accommodate the legacy signed encoding. +- **`SessionNonce` contract: unchanged.** Still fail-closed when not set, still requires ≥16 bytes via `SetSessionNonceBytes`. PR #6 only expands the docstring with the per-ceremony-uniqueness and high-entropy guidance that reviewers asked for; the runtime contract is identical. + +Two parties — one running pre-PR #6 (i.e. PR #5 HEAD) and one running post-PR #6 — will compute byte-identical SSIDs, byte-identical Fiat-Shamir challenges, byte-identical VSS commitments, and byte-identical wire messages, on every honest input. + +## On-chain / protocol impact + +- **Contracts: zero.** No `.sol` files in this repo. Nothing to deploy. +- **Wire format change in this PR: none.** PR #6 contributes nothing to the wire/transcript break described in PR #2's release notes. +- **Operator-controlled requirements (all inherited from PR #2):** `SetSessionNonce` / `SetSessionNonceBytes` must be called before keygen and signing. Unchanged by PR #6 — the only PR #6 contribution here is clarifying the docstring with explicit "unique per ceremony" and "high-entropy source" guidance. +- **Curve registry: untouched** by PR #6 (PR #5 already removed ed25519). + +## Consumer impact (keep-core-security / tBTC nodes) + +- **Source-level breakage in keep-core-security: none.** Every PR #6 change is either documentation or strictly additive defensive validation on `vss.Shares.ReConstruct`. No removals, no signature changes, no behavior change for honest callers. +- **`ReConstruct` callers downstream:** `ReConstruct` is invoked only inside `tss-lib`'s own tests (VSS tests + `ecdsa/keygen/local_party_test.go`). It is not on any keygen or signing wire path. If `keep-core-security` calls `ReConstruct` for share-backup or recovery flows, those callers will continue to receive valid secrets for honest inputs and will now receive a typed error (instead of a panic) for malformed inputs — a strict improvement. +- **Module-graph: unchanged.** No `go.mod` / `go.sum` deltas. `keep-core-security` re-vendoring this commit will see only the documentation and the VSS validation diff. +- **`SessionNonce` adoption requirement (inherited from PR #2):** unchanged by PR #6. The docstring expansion is informational — the same runtime guard at the same call sites with the same panic conditions. + +## Residual review-item resolution (documentation only) + +The multi-agent review of `c9e9c09` flagged a factual issue in the newly-added `RejectionSample` bias docstring: the stated bound `q / 2^eHash.BitLen()` was loose to the point that it could not support the docstring's own ~2^-128 conclusion for secp256k1, and the "q significantly smaller than 2^256 (e.g., q = 2^256)" example was internally inconsistent. + +**Resolved in `f973d1f`** (this PR's second commit). The paragraph now states the safe regime as a property of q ("close to 2^k from below") rather than via a loose formula or call-site enumeration, drops the unused curve25519 reference whose conclusion is correct but not derivable from the simple bound, and cross-references `HashToN` / `HashToNTagged` for the large-modulus regime that they were introduced to address. + +`f973d1f` is documentation-only — `RejectionSample`'s runtime behavior (`LiterallyJustMod` under the hood) is byte-for-byte unchanged. No computed Fiat-Shamir challenge moves. + +## Deployment recommendations + +- **Contracts: no action.** +- **Coordinators / services: no action specific to this PR.** Inherits from PR #2 the requirement to provide a unique `SessionNonce` per ceremony — unchanged. +- **Operator nodes:** + - PR #6 cannot be released alone; it ships stacked on PR #5 / PR #4 / PR #2. + - When the keep-core-security operator fleet upgrades to a version that vendors the combined stack, PR #6 contributes: (a) a defensive nil/dedup guard on `vss.Shares.ReConstruct` that converts a latent nil-deref into a typed error, and (b) clarifying documentation on `SessionNonce` usage, the BNB-RejectionSample modular-reduction choice, ECPoint mutation semantics, getSSID round-1 invariant, and `FactorChallenge` two-path encoding. + - **No second coordinated rollout is needed for PR #6**; it piggybacks on the PR #2 protocol-cut release. +- **Rollback:** PR #6 alone is cleanly revertible by re-vendoring the PR #5 HEAD SHA. The revert restores the pre-existing latent panic on malformed VSS shares but does not change wire behavior. There is no scenario in which reverting PR #6 alone is required for safety. + +## Validation performed + +- `go build ./...` clean. +- `go vet ./common ./crypto/vss` clean. +- `go test -count=1 ./common ./crypto/vss` passed (including the new `TestReconstructRejectsMalformedShares` table cases). +- GitHub Actions (workflow_dispatch on `f973d1f`): **Go Test** ✓ success, **Go-fmt** ✓ success. +- Wire-format audit: zero changes to `.proto` files, struct field layouts, hash input vectors, or message serialization paths. +- API audit: zero exported symbols removed; zero exported signatures changed; zero new exported symbols. +- Caller audit: `vss.Shares.ReConstruct` is reachable only from tests in this repo; downstream `keep-core-security` consumers are unaffected on the honest path and strictly improved on the malformed-input path. diff --git a/keep-core-release/tlabs-xyz/keep-core-security/10.md b/keep-core-release/tlabs-xyz/keep-core-security/10.md new file mode 100644 index 0000000000..ea99a3b3f1 --- /dev/null +++ b/keep-core-release/tlabs-xyz/keep-core-security/10.md @@ -0,0 +1,140 @@ +# PR #10 — fix(deps): remediate Sysdig keep-client:v2.5.2 image vulnerabilities + +- Repo: tlabs-xyz/keep-core-security +- Branch: `fix/sysdig-tbtc-2.5.2-cves` +- URL: https://github.com/tlabs-xyz/keep-core-security/pull/10 + +## Summary + +CVE-remediation PR. Upgrades Go runtime 1.24.1 → 1.25.10 and refreshes the +dependency tree (libp2p, quic-go, golang.org/x/*, protobuf, multiaddr, pion/*) +to clear vulnerabilities flagged against the `keep-client:v2.5.2` image by +Sysdig. Drops the archived `go-addr-util` package in favor of +`go-multiaddr/net`. Moves `protodelim` from the temporary `dev/` import path +to the stable one (which also picks up CVE-2024-24786 in protobuf). Adds +top-level `permissions:` blocks to the three workflows that use +`dorny/paths-filter`. Pins Docker base images to specific Go patch versions +for build reproducibility. + +Source change is tightly scoped: 3 Go files in `pkg/net/libp2p/`, ~24 lines +diff, semantically equivalent (`addrutil.InterfaceAddresses` and +`manet.InterfaceMultiaddrs` both filter only on `IsIP6LinkLocal`). + +## Breaking changes + +None at the operator / wire / consensus surface. + +| Surface | Changed? | Notes | +|---|---|---| +| Wire protocol (`/keep/handshake/1.0.0`, `authProtocolID="keep"`) | No | `pkg/net/libp2p/transport.go` IDs unchanged | +| Public Go API of `pkg/net/libp2p` | No | Only internal `getListenAddrs` body changed | +| Configuration / flags / env | No | No CLI, config file, or env-var changes | +| Smart contracts (Solidity) | No | Zero contract files touched | +| Persistent state / DB | No | No schema, no on-disk format changes | +| Listen transports | No | Still TCP-only (`/tcp/%d`); QUIC/WebTransport not used by keep-client | +| `go.mod` direct deps | Yes | libp2p 0.38.2→0.48.0, multiaddr 0.14→0.16, crypto 0.32→0.50, protobuf stable path | +| Docker base image | Yes | Now pinned to `golang:1.25.5-alpine3.21` and `golang:1.25.10-bookworm` | +| CI workflow permissions | Yes | Added `contents: read` + `pull-requests: read` to 3 workflows | + +## Behavioral change (intended) + +- IPv6 link-local interface addresses continue to be filtered from the + advertised listen set; behavior matches the deprecated `go-addr-util` + implementation (both filter only on `IsIP6LinkLocal`). +- Test transport in `bootstrap_test.go` changed from the obsolete + `/utp/` to `/quic-v1/`; test intent (one peer reachable via two distinct + transports) preserved. + +## Network compatibility (mixed-version peer network) + +The keep-network is a long-lived P2P validator network. New keep-client +binaries built from this PR will run alongside existing operator nodes still +on v2.5.2 or earlier. Wire-level compatibility was analyzed against the +changelogs: + +- **libp2p 0.38 → 0.48**: Breaking changes in this range are Go-API only + (e.g., `errors.Is(err, network.ErrReset)` in v0.40, identify rate-limiting + in v0.42, WebTransport handshake change in v0.47). The custom Keep + security protocol ID (`/keep/handshake/1.0.0`) is unchanged. WebTransport + is not used by keep-client (TCP-only listen). No wire-level break for + TCP+Keep-security. +- **quic-go 0.48 → 0.59**: All API breaking changes are Go-API only (e.g., + `Connection`→`Conn` struct, `ConnectionTracer` removal). Wire-level + additions (ACK_FREQUENCY frame, IMMEDIATE_ACK frame, `min_ack_delay` + transport parameter) are optional QUIC extensions negotiated via + transport parameters; RFC 9000 mandates that peers ignore unknown + parameters. Mixed-version interop is safe by design. +- **protobuf 1.36.3 → 1.36.6**: Patch-level; wire format unchanged. +- **multiaddr 0.14 → 0.16**: Library-level changes; multiaddr string format + is unchanged. + +## Vulnerabilities addressed (in scope) + +Per the PR description and Socket Security scan (all alerts resolved): + +- Go runtime 1.24.1 → 1.25.10 (multiple Go stdlib CVEs) +- golang.org/x/crypto 0.32 → 0.50 (CVE chain) +- libp2p 0.38.2 → 0.48.0 +- quic-go (indirect) refresh +- google.golang.org/protobuf 1.36.3 → 1.36.6 (picks up CVE-2024-24786 by + dropping the `dev/` replace directive) + +## Deferred / out of scope + +Explicitly called out in the PR description: + +- `go-ethereum` upgrade (1.10.x branch retention required by upstream + abigen tooling; tracked separately) +- `btcd` major version bump +- Alpine 3.21 → 3.22 base image (would unblock `golang:1.25.10-alpine`; + not done here to keep the PR focused) + +## Is it safe to release / redeploy? + +**Yes, with the standard rollout flow.** No contract changes, no state +migration, no operator-visible config changes, no wire-protocol break. + +**Required redeploys:** + +- **keep-client nodes (operators)**: yes — rebuild and roll out the new + image. This is a binary-level dependency refresh; no operator action + beyond pulling the new image and restarting. +- **Smart contracts**: no — zero contract code touched. +- **Off-chain services (relays, observers)**: no — unless they share the + `keep-client` image; in that case, same as operator redeploy. + +**Suggested rollout:** + +1. Build the image; verify Sysdig CVE scan is clean. +2. Deploy to staging; confirm the new client connects to and exchanges + traffic with at least one v2.5.2 peer (TCP transport + Keep auth + handshake). The custom security protocol ID is unchanged, so this + should be a no-op verification. +3. Roll to mainnet operators one node at a time; watch peer-count and + handshake metrics. +4. Hold one v2.5.2 node in the network for several days post-rollout to + confirm sustained interop in mixed-version conditions. + +## Open items (P2, non-blocking) + +- `Dockerfile` `build-sources` stage still relies on `GOTOOLCHAIN=auto` + to fetch Go 1.25.10 at build time because `golang:1.25.10-alpine3.21` + does not exist (Alpine 3.21 + Go 1.25 tops out at 1.25.5; 1.25.6+ + requires alpine3.22). Closing this gap requires bumping Alpine to 3.22, + which was deferred from this PR. Base layer is now pinned to + `golang:1.25.5-alpine3.21` so reproducibility against floating-tag + drift is locked; the toolchain fetch from `dl.google.com` during build + remains. `build-bins` stage is fully pinned (`golang:1.25.10-bookworm`) + and does not fetch. + +## Reviewer notes / risk classification + +- **Code risk**: low — 3-file libp2p refactor, semantically equivalent + to the pre-existing implementation; verified by reading the upstream + filter logic of both `addrutil.InterfaceAddresses` and + `manet.InterfaceMultiaddrs` (both filter only `IsIP6LinkLocal`). +- **Build risk**: low — image pins now in place; base layer + reproducible. +- **Network risk**: low — TCP-only listen, custom Keep security + protocol ID unchanged, QUIC extensions opt-in. +- **Consensus / contract risk**: none — no contract code in this PR. diff --git a/keep-core-release/tlabs-xyz/keep-core-security/11.md b/keep-core-release/tlabs-xyz/keep-core-security/11.md new file mode 100644 index 0000000000..56d61b9b9d --- /dev/null +++ b/keep-core-release/tlabs-xyz/keep-core-security/11.md @@ -0,0 +1,150 @@ +# PR #11 — fix(docker): bump Alpine base 3.21 -> 3.23 for OS-package CVEs + +- Repo: tlabs-xyz/keep-core-security +- Branch: `fix/sysdig-alpine-base-bump` (base: `fix/sysdig-tbtc-2.5.2-cves`, i.e. PR #10) +- URL: https://github.com/tlabs-xyz/keep-core-security/pull/11 + +## Summary + +OS-package CVE-remediation PR. Bumps the Alpine base image from 3.21 to 3.23 +in the two Alpine `FROM` lines of `Dockerfile` (the `build-sources` stage and +the `runtime-docker` stage). Stays inside the Alpine 3.x family — musl 1.2.5 +series, OpenSSL 3.3.x — so it does not introduce a libc or TLS ABI break for +the CGO surface used by keep-client. + +PR-only diff is **2 lines** in `Dockerfile`. Zero Go code, zero Solidity, zero +config, zero CI changes. The local `git diff main...HEAD` looks much larger +only because it transitively rolls up PR #10 (the dependency / Go-runtime +refresh); that work is not this PR's responsibility. + +## Breaking changes + +None at the operator / wire / consensus / API surface. + +| Surface | Changed? | Notes | +|---|---|---| +| Wire protocol (`/keep/handshake/1.0.0`, libp2p, TCP) | No | No Go code touched | +| Public Go API of `keep-client` | No | No Go code touched | +| Configuration / CLI flags / env | No | No config or flag changes | +| Smart contracts (Solidity) | No | Zero contract files touched | +| Persistent state / DB / on-disk format | No | No schema or storage changes | +| `go.mod` / Go dependency graph | No | Unchanged from PR #10 | +| Docker base image (Alpine stages) | Yes | `alpine:3.21` -> `alpine:3.23` in `build-sources` and `runtime-docker` | +| Docker base image (`build-bins` stage) | No | Still `golang:1.25-bookworm` (Debian, untouched by this PR) | +| CI workflow permissions | No | Inherited from PR #10 | + +## Behavioral change (intended) + +- Runtime image ships with newer OS packages by default: OpenSSL 3.3.7-r0+, + musl 1.2.5-r11+, zlib 1.3.2-r0+, plus toolchain (binutils, gcc-runtime) + refresh in the build stage. Application behavior is unchanged — the + keep-client binary itself is byte-identical to PR #10's output, modulo + whatever it links against at runtime via musl + OpenSSL ABI. + +## ABI / runtime compatibility + +The keep-client container is musl-linked (Alpine-built Go binary with CGO +into local libs at runtime: secp256k1, c-kzg-4844-adjacent code, etc.). The +risk surface for a base-image bump is whether musl / OpenSSL ABI drift breaks +runtime symbol resolution. + +- **musl**: Alpine 3.21 ships musl 1.2.5; Alpine 3.22 and 3.23 also stay on + the musl 1.2.5 series. Same major.minor — ABI stable. +- **OpenSSL**: Alpine 3.21 ships OpenSSL 3.3.x; Alpine 3.23 ships OpenSSL + 3.3.x with security patches (3.3.7-r0). Same SONAME family — ABI stable. +- **Build + runtime are aligned**: both stages now use Alpine 3.23, so the + CGO link target at build time matches what the runtime image provides. + There is no cross-Alpine-version musl/OpenSSL surface introduced by this + PR (and the previous state of 3.21 + 3.21 was likewise aligned). + +## Network compatibility (mixed-version peer network) + +Not a wire-level change. Nothing in this PR alters libp2p, the Keep auth +handshake, or any protocol. A keep-client built from this PR is wire-compatible +with v2.5.2 and earlier peers to the same degree PR #10 is — i.e. fully +compatible per the analysis in #10.md. + +## Vulnerabilities addressed (in scope) + +Per the PR description (Sysdig scan of `thresholdnetwork/keep-client:v2.5.2`): + +| CVE | Severity | Package | Fix | +|---|---|---|---| +| CVE-2026-31789 | Critical (9.8) | libcrypto3 / libssl3 | OpenSSL 3.3.7-r0 | +| CVE-2026-28387 (x2) | High (8.1) | libcrypto3 / libssl3 | OpenSSL 3.3.7-r0 | +| CVE-2026-28388/28389/28390 (x2 each) | High (7.5) | libcrypto3 / libssl3 | OpenSSL 3.3.7-r0 | +| CVE-2026-31790 (x2) | High (7.5) | libcrypto3 / libssl3 | OpenSSL 3.3.7-r0 | +| CVE-2026-40200 (x2) | High (8.1) | musl / musl-utils | musl 1.2.5-r11 | +| CVE-2026-22184 | High (7.8) | zlib | zlib 1.3.2-r0 | +| CVE-2026-6042 (x2) | Medium (4.0) | musl / musl-utils | musl 1.2.5-r10 | +| CVE-2026-27171 | Medium (5.5) | zlib | zlib 1.3.2-r0 | + +All findings are in OS packages bundled by the Alpine base layer; bumping the +base layer is the correct fix vector (no application code change required). + +## Is it safe to release / redeploy? + +**Yes, with the standard rollout flow.** No contract changes, no state +migration, no operator-visible config changes, no wire-protocol break, no Go +API change. The change is a Docker base-image refresh that affects only OS +packages inside the runtime container. + +**Required redeploys:** + +- **keep-client nodes (operators)**: yes — to actually consume the CVE + fixes, operators must pull the new image and restart their node container. + Drop-in replacement; no migration, no flag change, no peer churn beyond a + normal node restart. +- **Smart contracts**: no — zero contract code touched. +- **Off-chain services (relays, observers)**: no — unless they share the + `keep-client` image; in that case, same as operator redeploy. +- **Release artifacts (tarballs from `output-bins`)**: not affected. The + `build-bins` stage at `Dockerfile:112` is `golang:1.25-bookworm` (Debian, + glibc) and is untouched by this PR. Consumers of those binaries see no + change. + +**Suggested rollout:** + +1. Build the image; verify Sysdig CVE scan is clean (target: zero OS-package + Critical / High findings). +2. Smoke test in staging — confirm the new client starts, connects to a + v2.5.2 peer, and exchanges traffic (Keep auth handshake + TCP libp2p). + Watch for runtime linker errors on startup (this is the failure mode if + musl / OpenSSL ABI were to drift; none expected within Alpine 3.x). +3. Roll to mainnet operators one node at a time; watch peer-count and + handshake metrics. No coordination window required — rolling restart. +4. After full rollout, no v2.5.2 holdout is needed for this PR specifically + (since wire behavior is unchanged); follow whatever holdout plan #10 + used, since #10 is the change that materially altered the Go binary. + +## Sequencing relative to PR #10 + +This PR is stacked on PR #10. The intended sequence is: + +1. Land PR #10 first (Go-runtime + dependency refresh). +2. Retarget PR #11 to `main`, then land. After retargeting, PR #11's diff + against `main` will collapse back to the same 2 Dockerfile lines. + +If both PRs land before any new image is cut, operators only redeploy once +(combined image). If #10 ships first as its own image, operators redeploy +twice — both are safe rolling restarts. + +## Open items / follow-ups (non-blocking) + +- Retarget to `main` after #10 merges (procedural; called out in PR body). +- Re-run Sysdig scan against the freshly built image post-merge to confirm + zero OS-package Critical / High findings remain. (Listed in PR test plan.) +- The `build-bins` Debian stage and the `runtime-docker` Alpine stage remain + on different libc families. This is pre-existing structure (the deployed + artifact is the Alpine image; `output-bins` produces release tarballs + consumed elsewhere). Out of scope for this PR. + +## Reviewer notes / risk classification + +- **Code risk**: none — zero source code in this PR. +- **Build risk**: low — within-major Alpine bump (3.21 -> 3.23), same musl + 1.2.5 series, same OpenSSL 3.3.x family. CI's `client-build-test-publish` + exercises the full Docker build + Go test suite. +- **Network risk**: none — no wire-protocol or libp2p changes. +- **Consensus / contract risk**: none — no contract code in this PR. +- **Operator risk**: low — drop-in image refresh; standard rolling restart. diff --git a/keep-core-release/tlabs-xyz/keep-core-security/13.md b/keep-core-release/tlabs-xyz/keep-core-security/13.md new file mode 100644 index 0000000000..32b01888b0 --- /dev/null +++ b/keep-core-release/tlabs-xyz/keep-core-security/13.md @@ -0,0 +1,103 @@ +# PR #13 — fix(deps): bump go-ethereum v1.13.15 -> v1.17.3 + +- Repo: tlabs-xyz/keep-core-security +- Branch: `fix/sysdig-go-ethereum-bump` (now based on `main`) +- URL: https://github.com/tlabs-xyz/keep-core-security/pull/13 +- Status at time of writing: rebased onto main after PR #10 merged; CI re-running; `mergeable: MERGEABLE`, `mergeStateStatus: UNSTABLE` (checks pending). + +## Summary + +Follow-up to PR #10, covering the **go-ethereum** bucket that PR #10 deferred. Bumps `github.com/ethereum/go-ethereum` from `v1.13.15` to `v1.17.3` (current latest, published 2026-05-11) to clear 5 High-severity CVEs (CVE-2026-22862, -22868, -26313, -26314, -26315) flagged by the Sysdig scan of `thresholdnetwork/keep-client:v2.5.2`. Also pulls forward the `keep-common` fork (`v1.7.1-tlabs.0` → `v1.7.1-tlabs.1`) so abigen-generated `//go:linkname` targets resolve against go-ethereum v1.16+. CI workflow gains a `free-disk-space` step (SHA-pinned to `jlumbroso/free-disk-space@54081f1`, v1.3.1) because the multi-arch image build exhausts the default ~14 GB on `ubuntu-latest`. + +Scope is tight: 3 files (`go.mod`, `go.sum`, `.github/workflows/client.yml`). **Zero application source changes.** + +## Breaking changes + +None at the operator / wire / consensus / contract surface. + +| Surface | Changed? | Notes | +|---|---|---| +| Wire protocol (libp2p `/keep/handshake/1.0.0`) | No | This PR doesn't touch libp2p, `pkg/net/*`, or the security protocol ID | +| Smart contracts (Solidity) | No | Zero contract files touched; deployed contracts unaffected | +| ABI bindings (`pkg/chain/ethereum/.../gen/contract/*.go`) | No | Generated code unchanged; existing bindings still compile against go-ethereum v1.17 | +| `ethclient` JSON-RPC traffic | No | RPC method set used (`HeaderByNumber`, `TransactionReceipt`, `SuggestGasPrice`, `SubscribeNewHead`, etc.) is API-stable across v1.13–v1.17 | +| Transaction encoding (legacy / dynamic-fee / access-list) | No | go-ethereum v1.14+ added blob-tx support (EIP-4844); legacy encodings are unchanged; keep-client does not emit blob txs | +| Signing (`crypto.Sign` / `crypto.Ecrecover`) | No | secp256k1; signature bytes deterministic across the range | +| Keystore format (V3 JSON) | No | Stable; existing operator keystores load unchanged | +| Configuration / flags / env vars | No | No CLI, config-file, or env-var changes | +| Persistent state / on-disk format | No | keep-core does **not** import `core/rawdb`, `ethdb`, `trie`, `node`, `rpc`, `eth/protocols`, or any go-ethereum storage package (verified by grep) | +| `go.mod` direct deps | Yes | `go-ethereum` 1.13.15 → 1.17.3; `keep-common` replace 1.7.1-tlabs.0 → 1.7.1-tlabs.1 | +| `go.mod` indirect deps (new) | Yes | `crate-crypto/go-eth-kzg`, `ethereum/c-kzg-4844/v2`, `emicklei/dot`, `ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime` (forced by go-ethereum/crypto; not invoked from keep-core's call graph) | +| `go.mod` indirect deps (bumped) | Yes | `gnark-crypto` 0.12 → 0.18, `holiman/uint256` 1.2 → 1.3, `blst` 0.3.11 → 0.3.16, `fastssz` 0.1.2 → 0.1.4, `cobra` 1.5 → 1.8, etc. | +| CI workflow | Yes | New `free-disk-space` step at the start of `client-build-test-publish` (SHA-pinned) | + +## Behavioral changes (intended) + +- **CI runners** now have ~30 GB extra free space before the multi-arch build starts (removes Android / .NET / Haskell toolchains, all unused by keep-client). Without this, the v1.17 build OOM'd on disk. Pinned to commit SHA, not `@main`, since the step runs in a job that later authenticates to Docker Hub, AWS, and GHCR. +- **go-ethereum runtime behavior**: no semantic change to the methods/types keep-core consumes. The v1.13 → v1.17 range introduced blob transactions (EIP-4844), Verkle trie scaffolding, and Amsterdam-fork preparation in upstream — all are server-side concerns; the **client-side** API surface keep-core uses (`types.Transaction`, `bind.BoundContract`, `ethclient.Client`, `crypto.Sign`/`Ecrecover`, ABI binding helpers) is stable. + +## Network compatibility (mixed-version peer network) + +The keep-network is a long-lived P2P validator network. New keep-client binaries built from this PR will run alongside operator nodes still on `v2.5.2` (post-PR-#10) and earlier. + +- **libp2p / handshake**: untouched by this PR (PR #10 already handled the libp2p refresh, which was verified mixed-version safe). Custom security protocol ID `/keep/handshake/1.0.0` unchanged. +- **L1 RPC traffic**: keep-client talks JSON-RPC over HTTP/WebSocket to the operator's chosen Ethereum node (geth / erigon / nethermind). The RPC protocol is independent of the in-process `ethclient` Go API version. Compat with the L1 RPC endpoint is unchanged. +- **On-chain calls**: contract addresses, ABIs, function selectors all unchanged. Existing T / tBTC contracts continue to be called the same way. + +## Supply-chain notes + +- **`ProjectZKM/Ziren/crates/go-runtime/zkvm_runtime`** is a new transitive indirect dep (pseudo-versioned, Apache-2.0, MIPS zkVM runtime). `go mod why` confirms it's reachable via `go-ethereum/crypto` — i.e., it's a forced transitive from upstream. Not invoked directly from any keep-core source. Documented in the PR body for reviewer awareness. Org is legitimate (117 stars, active, Apache-2.0); not a typo-squat. Cannot be severed without forking go-ethereum. +- **`ethereum/c-kzg-4844 v0.4.0`** dropped; replaced by **`ethereum/c-kzg-4844/v2 v2.1.6`** + **`crate-crypto/go-eth-kzg v1.5.0`** (KZG commitments for blob tx support — unused at runtime by keep-client but present in the binary). +- **Socket Security** scan on the PR: go-ethereum vulnerability score +31, supply-chain score +1; no new alerts. + +## Vulnerabilities addressed + +| CVE | Severity | CVSS | Fixed in go-ethereum | +|---|---|---|---| +| CVE-2026-22862 | High | 7.5 | v1.16.8 | +| CVE-2026-22868 | High | 7.5 | v1.16.8 | +| CVE-2026-26313 | High | 7.5 | v1.17.0 | +| CVE-2026-26314 | High | 7.5 | v1.16.9 | +| CVE-2026-26315 | High | 7.5 | v1.16.9 | + +All 5 cleared by pinning to v1.17.3. + +## Is it safe to release / redeploy? + +**Yes, with the standard rollout flow.** No contract changes, no state migration, no operator-visible config changes, no wire-protocol break, no source-level behavior change. + +**Required redeploys:** + +- **keep-client nodes (operators)**: yes — rebuild and roll out the new image to pick up the patched go-ethereum dependency. This is a binary-level refresh; no operator action beyond pulling the new image and restarting. +- **Smart contracts**: no — zero contract code touched. +- **Off-chain services (relays, observers, signers)**: no — unless they share the `keep-client` image; in that case, same as operator redeploy. +- **Operator keystores / wallet files**: no — V3 keystore format is stable across the go-ethereum range. +- **L1 Ethereum node operators run alongside (geth / erigon / nethermind)**: no — independent. + +**Suggested rollout:** + +1. Wait for PR #13 CI (`client-build-test-publish`, `client-scan`, `client-vet`, `client-lint`) to go green on the rebased tip. +2. Run Sysdig rescan against the new image; verify all 5 go-ethereum CVEs from this PR plus the libp2p / Go-runtime CVEs from PR #10 are clear. +3. Deploy to staging; confirm the new client connects to mainnet RPC and exchanges traffic with at least one pre-bump peer. Wire protocol is unchanged so this should be a no-op verification. +4. Smoke-test a `tbtc` end-to-end interaction in staging (deposit / redemption flow); this covers the `ethclient`, `bind`, and `crypto.Sign` paths. +5. Roll to mainnet operators one node at a time; watch peer-count, RPC-error rate, and on-chain tx success metrics. +6. Hold one pre-bump node in the network post-rollout for several days to confirm sustained interop. + +## Open / deferred items + +- **PR-test-plan items 2 and 3** (staging spot-check + Sysdig rescan) remain open at PR-body level. CI item 1 is currently UNSTABLE (re-running post-rebase) — confirm green before merge. +- **Forked `keep-common`** (`threshold-network/keep-common v1.7.1-tlabs.1`) is now a hard prerequisite for go-ethereum 1.16+ in this codebase. Each future go-ethereum bump that crosses an abigen/`go:linkname` change will require a corresponding `tlabs.N` tag. Not a release blocker, but worth tracking long-term (either upstream or accept as a permanent fork). +- **Ziren transitive** stays in the dependency closure as long as go-ethereum keeps it in `crypto`. Re-evaluate at the next go-ethereum bump. + +## Reviewer notes / risk classification + +- **Code risk**: very low — zero application source changes; only `go.mod` / `go.sum` movement and one CI step. +- **Build risk**: low — CI re-running on the rebased tip; main's existing Docker pins (`golang:1.25.5-alpine3.21`, `golang:1.25.10-bookworm`) inherited from PR #10 carry over unchanged. +- **Network risk**: very low — no libp2p, no protocol ID, no RPC contract changed. +- **Consensus / contract risk**: none — no Solidity, no on-chain interaction surface changed. +- **Supply-chain risk**: low-medium — one new untagged transitive (Ziren) forced by upstream; documented; not in keep-core's call graph. Socket scan net-positive. +- **Operational risk for redeploy**: low — drop-in replacement; no state migration; safe rolling deploy. + +## Rebase note (2026-05-23) + +PR #10 squash-merged into main at 2026-05-23T10:59:41Z, which auto-retargeted PR #13 from `fix/sysdig-tbtc-2.5.2-cves` to `main` and deleted the old base branch. The branch was then rebased onto `origin/main` via `git rebase --onto origin/main c6df34aae HEAD`, dropping the 4 pre-squash commits that overlapped with PR #10's content (`bf5307ff4`, `81b42c3d8`, `e2a8c74cb`, `c6df34aae`) and preserving only the 4 PR-13-specific commits. No manual conflict resolution was required. Verified clean afterwards: `go build ./...` clean, `go vet ./...` shows only the two pre-existing warnings in `pkg/tbtcpg/internal/test/marshaling.go` and `pkg/tecdsa/signing/protocol.go` (unchanged by this PR). Force-pushed with `--force-with-lease`. diff --git a/keep-core-release/tlabs-xyz/keep-core-security/2.md b/keep-core-release/tlabs-xyz/keep-core-security/2.md new file mode 100644 index 0000000000..16f5ce2eab --- /dev/null +++ b/keep-core-release/tlabs-xyz/keep-core-security/2.md @@ -0,0 +1,190 @@ +# PR #2 -- Release Risk Assessment + +**Repo:** `tlabs-xyz/keep-core-security` +**PR:** [#2 -- security: whitebox pentesting materials and findings](https://github.com/tlabs-xyz/keep-core-security/pull/2) +**Branch:** `security/whitebox-pentesting-materials` → `main` +**Assessed at:** 2026-05-23 against HEAD commit `6a696002d` (CI green: 21 success, 10 skipped, 0 failed). Originally assessed at `bf1fe2ae0`; the deployment story is unchanged by the test additions since then -- see §11 below. + +## TL;DR + +**This is NOT a safe drop-in release. Two hard-fork-class wire-format changes ship in the Go client (F-02, F-03), and one non-upgradeable on-chain contract (`RandomBeacon`) acquires a new storage slot and new modifier behaviour (F-09).** + +| Surface | Breaking? | Coordination required | +|---------|-----------|------------------------| +| Go client wire protocol (relay-entry BLS signing, GJKR DKG Pedersen H, peer-to-peer share encryption) | **Yes** -- F-02 + F-03 | Coordinated cutover; all operators must upgrade in the same block window | +| RandomBeacon Solidity contract | **Yes** -- new storage slot + new modifier + gas offset bump (F-09) | Fresh deployment at new address; non-proxy; group registry and ownership migration required | +| Persistence on-disk format | No | Existing keystore/work-dir files remain readable | +| Ephemeral session keys (HKDF-derived) | No persistence; regenerated per session | No data migration needed | +| Operator config defaults (`clientInfo.port`) | Soft -- default **retained at `9601`** for this coordinated release (temporary compatibility); explicit `0` disables it | Operators must commit an explicit `clientInfo.port` value, keep the endpoint on a trusted path, and migrate scrape targets before the R2 default-off follow-up | +| libp2p Keep handshake | Local timeout only (15s); no protocol change | None | +| solidity-v1 contracts | Source-only changes; immutable on-chain code untouched | None | + +## 1. Wire-breaking changes in the Go client + +Both changes are deterministic and identical-across-nodes; running a heterogeneous fleet -- some old, some new -- will produce inter-node protocol failures, **not** local errors. A staged rollout is unsafe. + +### F-02 -- `G1HashToPoint` output changes for the same input + +* **File:** `pkg/altbn128/altbn128.go:120-162` +* **Before:** try-and-increment (`x = sha256(m)`; while not on curve, `x += 1`). +* **After:** counter-based hash-and-try (`sha256(m || ctr)`; `ctr` in `[0, 63]`, return first valid point). +* **Wire impact:** + * BLS `Sign()` / `Verify()` -- `pkg/bls/bls.go:51,63`. Old and new nodes will not agree on `H(message)`, so the recovered relay entry signature will fail on-chain BLS verification at `Relay.sol:150-157`. + * GJKR DKG Pedersen generator H -- `pkg/beacon/gjkr/protocol_parameters.go:24`. Old and new nodes will not agree on the commitment generator; commitments will not verify and the DKG will abort. +* **Note from the source:** the in-file comment at `altbn128.go:140-142` explicitly states "Deployment requires a coordinated network upgrade." +* **Residual concern (not new in this PR):** the counter-based loop panics if all 64 attempts fail. Probability per input is `(1/2)^64 ≈ 5e-20`. The panic is identical across nodes by construction (deterministic), so any reachable trigger is a chain-halt class event. All known production callers feed public inputs into this primitive (see F-02.md call-site table), so the panic cannot be deliberately triggered by an attacker. Future RFC 9380 SWU migration (tracked in [issue #4](https://github.com/tlabs-xyz/keep-core-security/issues/4)) is single-pass and eliminates this class. + +### F-03 -- ECDH session-key derivation switches from `sha256` to HKDF-SHA256 with a domain-separation `info` label + +* **File:** `pkg/crypto/ephemeral/symmetric_key.go:24-40` +* **Before:** `key = sha256(btcec.GenerateSharedSecret(priv, pub))` -- no salt, no info, no domain separation across protocols or peer pairs. +* **After:** `key = HKDF-SHA256(ikm = sharedSecret, salt = nil, info = || min(idA,idB) || max(idA,idB))`. Where `` is one of: + * `"gjkr"` (4 callers in `pkg/beacon/gjkr/protocol.go`) + * `"tecdsa-dkg"` (1 caller in `pkg/tecdsa/dkg/protocol.go`) + * `"tecdsa-signing"` (1 caller in `pkg/tecdsa/signing/protocol.go`, also includes `sessionID`) +* **Wire impact:** For the same ECDH shared secret, the old construction and the new construction produce different 32-byte symmetric keys. Old and new nodes will fail to decrypt each other's GJKR and tECDSA peer-to-peer share messages. Both the beacon DKG and the tECDSA DKG/signing protocols will abort on the first encrypted-share exchange. +* **Invariant:** the `info` encoders serialize each `MemberIndex` as a single byte (`byte(id)`). This relies on `group.MemberIndex` being a `uint8`. The dependency is now pinned by a compile-time assertion in `pkg/protocol/group/group.go` and a runtime check in `pkg/protocol/group/member_index_test.go` -- any future widening of `MemberIndex` would be caught at build time. If the type is ever widened, the `*EcdhInfo` encoders must move to a width-independent encoding (e.g. `binary.BigEndian.PutUint16`) in the same coordinated upgrade. +* **No persistence:** ECDH keys are ephemeral, regenerated per session; no migration of stored data. + +### Non-breaking fixes that ship in the same Go-binary cutover + +These ride along with the F-02/F-03 binary upgrade. They're not wire-breaking, but they activate at the same moment, so include them in the cutover release notes: + +* **F-13** -- tBTC event deduplicator TOCTOU fix (`pkg/tbtc/deduplicator.go`). Removes a race window where the same Ethereum event could be processed twice. Behaviour change: no observable difference under normal load; under high-concurrency event delivery, duplicate notifications now collapse to one. +* **F-15** -- `sqrtGfP2` exponent cross-check (`pkg/altbn128/altbn128_test.go`). Adds a test asserting the hardcoded exponent equals `(p^2 + 15) / 32`. Source code unchanged; this is a regression guard only. No runtime impact. + +### Combined coordination requirement + +`SECURITY-BREAKING-CHANGES.md` already documents the cutover checklist. **In this +build both changes activate at the binary level: the binary contains no +block-height cutover gate and reads no chain flag, so the operative cutover is +the operator software upgrade itself — an atomic flag-day, not a code-read +height.** The release-baked cutover block `C` described in §7 (below `C` legacy, +post-`C` security-v2) is the _planned_ end-state that a later change adds; until +that gate lands, the "cutover block" is only an operator-coordinated target for +the simultaneous swap, not a value the binary interprets. + +Minimum operational steps (flag-day model, this build): + +1. Agree a coordinated cutover time/height with all operators (a manual + scheduling target, since the binary does not read it). +2. Stage and dry-run on a testnet with the full fleet. +3. Coordinate a simultaneous binary swap. Rolling, node-by-node upgrades will + cause BLS submissions to revert and DKGs to fail; the swap must be atomic + across the ceremony fleet. +4. Post-cutover monitoring: alert on BLS-verification reverts (`Relay.sol`), on + DKG failure rates, on peer-to-peer share decryption errors, and on the new + `performance_announcer_session_id_mismatch_total` / cutover-roster + stranded-peer signals. + +## 2. On-chain contract changes + +### `RandomBeacon.sol` (F-09 fix) + +* **Storage layout change:** adds `uint256 private _reentrancyStatus`. Initialised to `1` in the constructor. +* **Logic change:** `submitRelayEntry(bytes)` (line 1054) and `submitRelayEntry(bytes, uint32[])` (line 1083) now carry an inline `nonReentrant` modifier; OZ `ReentrancyGuard` is **not** inherited (EIP-170 bytecode budget pressure). +* **Constant change:** `_relayEntrySubmissionGasOffset` constructor default bumped from `11_250` to `13_450` (+2,200 gas) so the relay-entry submitter is fully reimbursed for the additional SSTORE on the modifier's exit path. Tests at `solidity/random-beacon/test/fixtures/index.ts` updated to match. +* **Upgradability:** `RandomBeacon` is deployed via `hardhat-deploy`'s plain `deployments.deploy(...)` (`solidity/random-beacon/deploy/04_deploy_random_beacon.ts:34-53`). No proxy. **The contract is not upgradeable.** Existing mainnet deployment (`0x5499f54b4A1CB4816eefCf78962040461be3D80b`) cannot receive the F-09 fix in-place. + + Deploying the fix on mainnet therefore requires: + 1. Fresh `RandomBeacon` deployment at a new address. + 2. `transferOwnership` of `BeaconSortitionPool` to the new address. + 3. Re-authorisation of the new `RandomBeacon` in `TokenStaking`. + 4. Re-authorisation of the new `RandomBeacon` in `ReimbursementPool`. + 5. Deploy a new `RandomBeaconGovernance` pointing at the new address (`07_deploy_random_beacon_governance.ts`). + 6. Migrate active groups, in-flight relay entries, and authorisations -- or accept that running groups must expire / be rebuilt on the new instance. + 7. Update every on-chain `IRandomBeaconConsumer` to point at the new address (notably `WalletRegistry` for tBTC). + + **This is a major redeployment event.** Treat it the same way the original RandomBeacon launch was treated, including a multi-week operator coordination window. + + **Operational consequence -- the F-09 security benefit lags the Go-binary release.** Between the Go-binary cutover (§1) and the RandomBeacon redeployment, F-09 remains live on mainnet. The Go binary cannot install the on-chain `nonReentrant` modifier; only a new contract deployment can. If the redeployment is deferred indefinitely, the pentest finding is "remediated" in the source tree while the on-chain attack surface is unchanged. Track the redeploy as a deliverable, not a follow-up. + +* **Gas-offset interaction with existing deployments:** the storage slot `_relayEntrySubmissionGasOffset` is governable (`RandomBeacon.sol:666`). An existing deployment could in principle have its gas offset bumped via governance, but without the modifier the bump over-reimburses callers. Don't apply the gas-offset bump independently of the modifier. + +### `solidity-v1` (legacy contracts) + +The PR carries source updates to `KeepRandomBeaconOperator.sol`, `KeepRandomBeaconServiceImplV1.sol`, plus a new `RelayEntryServiceStub.sol` test stub. Per F-14, the v1 contracts are deprecated, **not upgradeable**, and the deployed mainnet code is immutable. Source updates here are historical / advisory only and have no on-chain effect. No deployment action needed. + +### `solidity-v1/yarn.lock` and `solidity/random-beacon/yarn.lock` + +`scryptsy@^2.1.0` removed from the random-beacon yarn lock -- transitive dep cleanup, no on-chain effect. + +## 3. Operator-facing config defaults + +* **`cmd/flags.go`, `cmd/flags_test.go`, `configs/config.toml.SAMPLE`:** `clientInfo.port` default is **retained at `9601`** for this coordinated security release (temporary compatibility). `0` still disables the metrics/diagnostics HTTP server entirely. Keeping the default on preserves the primary evidence channel — a node's exact revision and stranded-peer state — throughout the cutover. (The compiled epoch and active-mode signals are part of the not-yet-landed cutover gate and are not exposed by this build.) A `main` merge had briefly flipped this to `0`; that flip is reverted here and deferred to the follow-up R2 release. +* **Operator-facing impact:** operators keep their metrics endpoint on upgrade. Because the endpoint is unauthenticated, it must be reachable only over a trusted network path; it must never be published on a public interface. +* **Operator runbook update required:** + * Audit operator configs for an explicit `[clientInfo] / Port = ...` entry and commit one now (even if it equals `9601`), so the R2 default-off flip is a no-op for your deployment. + * Where monitoring is intentionally retired, set `Port = 0` explicitly. + * Confirm the endpoint is behind a firewall/VPN or authenticated proxy. +* **R2 follow-up:** a later wire-compatible release flips the default back to `0` (disabled) once the monitoring migration exit criteria are signed off. It must not be bundled into emergency rollback handling. +* Per F-12 guidance, operators should also firewall this port to their scraper's IP range -- it exposes peer topology and operator chain address. + +## 4. Library / dependency-level changes + +### libp2p Keep authentication handshake -- 15s deadline (`pkg/net/libp2p/transport.go`) + +* Adds a 15s absolute deadline to the Keep authentication handshake that runs **after** the TLS upgrade. +* **Why:** without it, a peer that completes TLS and then stalls parks the connection inside a blocking `proto-delim` read, occupying a libp2p resource-manager transient inbound slot until the daemon restarts. This is a DoS pressure-relief, not a wire-protocol change. +* **Protocol compatibility:** old and new clients still complete the same handshake; only the local timeout differs. Slow-but-honest peers within 15s are unaffected. **Not a wire-breaking change.** + +## 5. Persistence and on-disk formats + +No protobuf, serialization-format, or key-storage layout changes. Operators upgrading the binary keep using their existing work directory, keystore, and pre-parameter cache. + +## 6. CI workflow changes (non-shipping) + +`.github/workflows/contracts-{ecdsa,random-beacon}.yml`: the transient `security/whitebox-pentesting-materials` branch entry was removed from `pull_request.branches` before merge -- a CI nudge for the predecessor PR that is no longer needed on `main`. + +`.github/workflows/client.yml`, `contracts-ecdsa-docs.yml`, `contracts-random-beacon-docs.yml`: `permissions:` key order changed during the merge to align with `main` (`contents: read` before `pull-requests: read`). Pure cosmetic. + +## 7. Release / redeploy decision matrix + +| Component | Action | +|-----------|--------| +| `keep-core` Go binary on all operator nodes | **Coordinated upgrade required.** Cutover height agreed; rolling upgrade is unsafe (F-02 + F-03). | +| `RandomBeacon` mainnet contract | **Redeploy at new address.** Non-proxy. Multi-week migration window. Update every `IRandomBeaconConsumer` (notably `WalletRegistry`/tBTC). Or defer the F-09 redeployment to a later batch if the practical exploitability of the unguarded callback is below the redeployment risk. | +| `WalletRegistry`/tBTC ECDSA contracts | No code change in this PR. F-07 explicitly mitigated by design; F-08 accepted post-TIP-092. No on-chain change required. | +| `solidity-v1` contracts | Immutable; no action. | +| Operator config | Audit `clientInfo.port`; commit an explicit value (default retained at `9601` for the release window; `0` to disable) and keep it on a trusted path. | +| Prometheus / monitoring | Scrape targets remain reachable post-upgrade (compatibility `9601` retained). Migrate every target onto its trusted path before the R2 default-off follow-up. | + +## 8. Rollback considerations + +* **Go binary rollback:** rollback is **homogeneous and all-or-nothing**. Before the coordinated swap it is trivial (no upgraded peer exists yet). After the swap, because the prior binary has no cutover gate and resumes legacy participation the moment it starts, **every** upgraded process must be stopped or independently network-quarantined before **any** prior binary becomes ceremony-reachable. A partial, node-by-node rollback recreates the mixed-version (session-ID / HKDF / hash-to-point) hazard in reverse and is prohibited. Have a tested rollback binary path staged before the swap. +* **`RandomBeacon` redeploy rollback:** the new deployment is at a new address. Rolling back means re-pointing consumers at the old address. Practical only if no production traffic has hit the new instance. +* **Operator config rollback:** trivial -- revert config and restart. + +## 9. Open follow-ups (not blocking this release) + +* RFC 9380 SWU hash-to-curve migration -- [issue #4](https://github.com/tlabs-xyz/keep-core-security/issues/4). Eliminates the residual F-02 panic class. Optional; no security impact. +* `WalletRegistry` non-atomic upgrade discipline -- [issue #6](https://github.com/tlabs-xyz/keep-core-security/issues/6). Operational runbook only. +* `keep-common` password-to-key KDF (Argon2id / scrypt / PBKDF2 instead of bare `sha256`). External library, separate release. + +## 11. Changes since the original assessment (non-shipping) + +Between the original assessment SHA `bf1fe2ae0` and current head `6a696002d`, the only additions are tests + tooling, none of which touch production behaviour: + +| File | Kind | Production impact | +|------|------|-------------------| +| `pkg/altbn128/altbn128_test.go` | Go test (`TestG1HashToPointWireFormat`) | None -- pins F-02 output | +| `pkg/tbtc/deduplicator_test.go` | Go tests (3 concurrent regressions) | None -- exercises F-13 race | +| `solidity/random-beacon/contracts/test/ReentrantBeaconConsumer.sol` | Test-only contract under `contracts/test/` | None -- not deployed in production deploy scripts | +| `solidity/random-beacon/test/RandomBeacon.Reentrancy.test.ts` | Hardhat F-09 regression test | None | +| `solidity/random-beacon/test/RandomBeacon.StorageLayout.test.ts` | Hardhat storage-layout pins | None | +| `solidity/random-beacon/hardhat.config.ts` | Adds `storageLayout` to solc `outputSelection`; preserves existing ABI/bytecode/metadata defaults | None -- compiler metadata only; deployed bytecode unchanged | +| `.gitignore` | Adds `.claude/` | None | + +Confirmed: deployed `RandomBeacon` bytecode would be byte-identical between `bf1fe2ae0` and `6a696002d` for the same `solc` version (storage layout output does not alter codegen). + +## 10. Bottom line + +* **Safe to merge to `main` in this repository?** Yes -- the PR's content is correct, tested, and reviewed. +* **Safe to release the binary to mainnet operators without coordination?** **No.** F-02 + F-03 require a synchronized network-wide cutover. +* **Safe to redeploy `RandomBeacon` without migrating consumers?** **No.** Plan the redeploy as a multi-step on-chain event with consumer updates. +* **Recommended sequencing if both deployments proceed:** + 1. Testnet rehearsal with the full fleet to validate F-02/F-03 wire compatibility below the cutover block, per-anchor overlap at the block, and homogeneous security-v2 above it. + 2. Single coordinated mainnet cutover release (Go client): one required operator update to the reviewed R1 digest before one release-baked cutover block `C`. Below `C` participants speak legacy wire formats; canonically post-`C` work speaks security-v2. Treat the operator config audit (`clientInfo.port`, retained at `9601` for observability) and an exact revision/epoch/digest fleet inventory as prerequisites; an un-upgraded binary keeps speaking legacy after `C` and must be externally quarantined. + 3. Rollback, if required, is **homogeneous**: every R1 process must be stopped or network-quarantined before any prior binary becomes ceremony-reachable (there is no predecessor gate). Partial, node-by-node rollback recreates the mixed-version hazard and is prohibited. + 4. `RandomBeacon` redeployment as a separate, later operation -- treated as a fresh contract launch. This step can be deferred without blocking the Go-side cutover, but the F-09 fix only takes effect once the redeploy lands. + 5. R2 client-info default-off is a later wire-compatible follow-up, gated on the monitoring migration; it is not part of the cutover or of rollback handling. diff --git a/keep-core-release/tlabs-xyz/keep-core-security/8.md b/keep-core-release/tlabs-xyz/keep-core-security/8.md new file mode 100644 index 0000000000..a75f686746 --- /dev/null +++ b/keep-core-release/tlabs-xyz/keep-core-security/8.md @@ -0,0 +1,115 @@ +# PR #8 — Release Risk Assessment + +**Repo:** `tlabs-xyz/keep-core-security` +**PR:** [#8 — Integrate tss-lib hardening and bind TECDSA session IDs](https://github.com/tlabs-xyz/keep-core-security/pull/8) +**Branch:** `codex/session-nonce-binding` → `main` +**Assessed at:** 2026-05-23 against HEAD commit `2855ad39b` + +## TL;DR + +**This PR is wire-breaking for the tBTC TECDSA DKG and signing protocols. It is NOT a safe drop-in release. All operators in a wallet's signing group must upgrade together; any mixed pre/post-hardening party in the same ceremony will fail closed.** No contracts, no persisted-state migration, no operator config or CLI surface change. + +| Surface | Breaking? | Coordination required | +|---|---|---| +| tBTC TECDSA DKG wire protocol (proof transcripts, SSID derivation, MtA / range / Paillier / DLN / Schnorr proofs, session-context plumbing) | **Yes** | Coordinated upgrade; all signing-group members must run post-hardening before the next DKG attempt | +| tBTC TECDSA signing wire protocol (same proof surface + required `fullBytesLen` + positive `SetSessionNonce`) | **Yes** | Coordinated upgrade; all signing-group members must run post-hardening before the next signing attempt | +| tBTC DKG/signing application-level session ID format (`signing.go`, `dkg.go`, `signing_loop.go`, `dkg_loop.go`) — protobuf `SessionID` field, used as ceremony match key in `pkg/tecdsa/{dkg,signing}/states.go` | **Yes** | Implicit in the tss-lib upgrade — same operator set must run the new client | +| Random Beacon (relay-entry BLS, GJKR DKG) | **No** | Uses `pkg/beacon/gjkr/`; does **not** depend on `tss-lib`. Untouched by this PR. | +| Smart contracts (Solidity) | **No** | Zero contract files touched. | +| Persistent state / on-disk key shares | **No** | Existing post-DKG key share files remain readable; only the in-protocol message format changes. | +| Operator config / CLI flags / env | **No** | No flag or env additions; behavior is automatic once the binary is upgraded. | +| Public Go API of `keep-client` | **No** (source-compatible) | tss-lib added required-but-variadic params; keep-core call sites are updated in this PR. External Go consumers of `pkg/tecdsa/{dkg,signing}` (none known in this repo) would need to supply the new session-ID semantics. | +| CI workflow permissions (`client.yml`, `contracts-ecdsa-docs.yml`, `contracts-random-beacon-docs.yml`) | **No** | Adds `pull-requests: read` / `contents: read` to detect-changes jobs so `dorny/paths-filter` can run under `GITHUB_TOKEN`. Operator-invisible. | + +## 1. Wire-breaking changes in the tBTC TECDSA stack + +The PR pins `github.com/threshold-network/tss-lib` from `2e712689cfbe` to `ae7075f3409e`, which is the threshold-network fork's hardening branch. Per the upstream `BNB_HARDENING_INTEGRATION.md`: + +> This is a protocol/wire compatibility break for proof transcripts. Proofs whose Fiat-Shamir challenges now use tagged hashing or session context will not verify across mixed old/new versions, even where the Go API remains source-compatible through variadic arguments. Operators should roll this out as a coordinated protocol upgrade rather than mixing parties from before and after this PR in the same keygen, signing, or resharing ceremony. + +Concretely, mixed pre/post-hardening peers will fail in at least these ways: + +| Source | Pre-hardening behavior | Post-hardening behavior | Failure mode in a mixed ceremony | +|---|---|---|---| +| `tss.Parameters.SessionNonce` | Zero fallback (keygen/resharing) or `SHA512_256(messageBytes)` (signing) | **Required positive nonce; fail closed if missing** | Either side computes a different SSID → all subsequent proofs verify against the wrong context → fail closed | +| `SetSessionNonceBytes` | Did not exist | **Required; panics on `<16` bytes** | Post-hardening peer cannot derive SSID without this call (keep-core now calls it in this PR) | +| `fullBytesLen` for ECDSA signing | Not required; library used internal default | **Required at runtime, bounded to curve order byte length** | Post-hardening peer needs an agreed-upon byte width up-front; pre-hardening peer never sent one → message-width mismatch on round 1 | +| DLN / Schnorr / MtA / range / Paillier-mod / Paillier-factor proofs | Untagged Fiat-Shamir hashes; no session context | **Tagged hashing (`common.SHA512_256i_TAGGED`) plus session-context bytes including party index** | Same input produces different challenge bytes pre vs post → proof verification fails on both sides | +| ECDSA resharing | Did not broadcast SSID in `DGRound1Message` | **New committee broadcasts and rejects mismatched SSIDs** | Old committee never broadcasts → new committee aborts | +| Canonical EC coordinates | Accepted any | **Rejects coordinates outside `[0, P)`** | Old peer that ever sent a non-canonical point is rejected by the new peer (one-way strictness) | +| MtA range-proof checks | BNB-upstream level | **GCD, interval, lower-bound, non-one, tagged-challenge checks** | Old proofs lacking the extra invariants fail verification | +| VSS reconstruction | Old constant-time/length contracts | **`threshold+1` reconstruction requirement plus fixture updates** | Old shares constructed without the new contract may fail post-hardening reconstruction in edge cases (Threshold-network keep-core has historically been at-or-above threshold + 1, so this is a hardening, not a regression — but it is enforced strictly now) | + +The PR body explicitly acknowledges this: *"Parties running pre-hardening and post-hardening code in the same ceremony are expected to fail proof verification; rollout must be coordinated."* + +## 2. Application-level session ID format change + +Independent of the tss-lib wire change, keep-core's own session ID **string format** changed in this PR. Session IDs are serialized as the `SessionID` field of every protobuf message in `pkg/tecdsa/dkg/marshaling.go` and `pkg/tecdsa/signing/marshaling.go`, and they are used as a ceremony match key in `pkg/tecdsa/{dkg,signing}/states.go` (e.g. `member.sessionID == protocolMessage.SessionID()`). A peer whose session ID string does not match the message's session ID drops the message. + +| Helper | Pre (`main`) | Post (this PR) | +|---|---|---| +| `dkgAttemptSessionID(seed, n)` | `"-"` | `"dkg--<016x n>"` | +| `signingAttemptSessionID(message, startBlock, n)` | `"-"` | `"signing--<016x startBlock>-<016x n>"` | + +Why this changed: + +- Adds a typed prefix and fixes width so the value clears tss-lib's new 16-byte minimum for `SetSessionNonceBytes`. +- Adds `attemptStartBlock` for signing so repeated same-digest ceremonies do not reuse the GG20 SSID across retries (forensic and security follow-up to the BNB GG20 SSID-uniqueness hardening). +- Computed once per attempt in the retry loop (`signing_loop.go`, `dkg_loop.go`) and threaded through `signingAttemptParams.sessionID` / `dkgAttemptParams.sessionID` so the announcer and the protocol cannot drift. + +**Operator-visibility:** session IDs appear in logs (`zap.String("signedMessage", ...)` etc.). Any external dashboard or alerting rule that pins the old `-` shape will see a string-prefix change. Grep against this repo found no operator-facing consumers; verify against any internal observability tooling before rollout. + +## 3. Required `fullBytesLen` argument + +`pkg/tecdsa/signing/member.go:147-156` now passes `fullBytesLen := (tecdsa.Curve.Params().N.BitLen() + 7) / 8` (32 bytes for secp256k1) into `signing.NewLocalParty`. Per the upstream report: + +> ECDSA/EdDSA signing constructors still accept `fullBytesLen` as a variadic argument for source compatibility, but exactly one positive value is required at runtime so all signers agree on message byte width before the protocol starts. + +If a pre-hardening peer ever signed with messages that started with leading zero bytes, its signatures could be off-by-one bytes shorter than what the chain expects; the fix is to require agreement on `fullBytesLen` up-front. Post-hardening signers all agree on 32 bytes for secp256k1. Pre-hardening signers do not send this and the post-hardening peer will refuse to proceed. + +## 4. Surfaces explicitly NOT changed + +- **Random Beacon stack** (`pkg/beacon/`): does not depend on `tss-lib`; uses keep-core's own GJKR DKG and BLS. Untouched. +- **Smart contracts**: zero `*.sol` files touched (verified by `git diff main...HEAD -- 'solidity/**' 'contracts/**'` returning empty). No proxy upgrade, no new deployment, no storage-slot or constant change. +- **On-disk persistence**: post-DKG key share files remain readable. There is no new field, no marshaling format change to the *stored* key share. Only **in-flight protocol message bytes** change. +- **CLI / config / env**: no flag, env, or config-file change. Operators do not need to edit `keep-client.toml` or similar. +- **libp2p / Keep auth handshake**: no change. Wire compatibility breaks at the *protocol-content* layer (tss-lib message bytes) but not at the transport / handshake layer. + +## 5. Required redeploys + +- **tBTC operator nodes (`keep-client`)**: **yes — mandatory, coordinated**. Every operator that is a member of a tBTC wallet's signing group must run a post-hardening binary before the next DKG or signing ceremony for that wallet, or the ceremony fails closed (timeouts and proof-verification rejections; see §1 table). No contract redeployment; no migration; just the binary replacement and a normal node restart per node. +- **Random Beacon operator nodes**: redeploy is **safe** (no change to beacon code) but **not required for correctness** — the Random Beacon path is untouched by this PR. In practice keep-client is one binary serving both, so the bundled redeploy ships both at once. +- **Smart contracts (tBTC bridge, RandomBeacon, WalletRegistry, etc.)**: **no** — zero Solidity touched. +- **Off-chain services (relays, observers, monitoring)**: **no**, unless they share the keep-client binary; observability services that only watch the chain are unaffected. +- **Release artifacts (Docker images, `output-bins` tarballs)**: standard rebuild and publish. + +## 6. Recommended rollout + +Because the wire break is **proof-content**, not handshake-content, a mixed-version network will not see drop-on-connect failures; it will see DKG attempts that announce successfully but then fail proof verification mid-protocol, retry, and eventually time out. That is harder to debug than an immediate disconnect, so the rollout should be aggressive about ensuring no holdouts. + +1. **Build and tag a new client release** (`vX.Y.Z+1`) that pins `tss-lib` at `ae7075f3409e` (this PR). Publish operator-facing release notes that name the wire break and require a coordinated upgrade. +2. **Staging / devnet dry-run**: stand up a full signing group on the new binary; perform end-to-end DKG and signing ceremonies; confirm the GG20 SSID-uniqueness assertion (retries of the same digest no longer share an SSID); confirm `SetSessionNonceBytes` is called on every code path that constructs a `tss.LocalParty`. Confirm timing behavior: the PR body notes one combined pre-final local run hit DKG outgoing-message timeouts under load — verify staging signing-group block budgets are still comfortable on the new code. +3. **Communicate a hard cutover window** to all tBTC operators. Unlike a wire-compatible rolling restart, this requires every operator who participates in a wallet's signing group to be on the new client **before** the next DKG attempt for that wallet. There is no graceful degradation. Coordinate via the existing operator channel and have on-call ready for the cutover window. +4. **Roll mainnet** at the cutover window. The recommended order: + - First, operators that are NOT in any current wallet's signing group (low blast radius). + - Then operators currently in signing groups, coordinated so that each wallet's group either has 100% old or 100% new for any in-flight signing — never mixed. +5. **Post-rollout monitoring**: watch DKG and signing attempt counters per wallet; any wallet that retries DKG > 1 attempt or signing > 2 attempts in the first hour post-rollout should be inspected. Persistent retry storms indicate at least one holdout operator on the old binary. +6. **Hold the old binary off mainnet** — do not allow re-introduction of pre-hardening clients into the signing groups, because they will deterministically break new ceremonies. + +## 7. Reviewer / risk classification + +- **Code risk**: medium — the keep-core delta is small (15 files, +239/-31, dominated by tests). The dominant risk is in the upstream tss-lib delta, which is separately reviewed per the PR body links (`threshold-network/tss-lib#2`, integration notes in `BNB_HARDENING_INTEGRATION.md`). +- **Wire risk**: **high (coordinated cutover required)** — see §1 and §6. +- **Consensus / contract risk**: none — zero contract changes; the post-DKG public keys produced by previously-completed ceremonies remain valid and signable (the wallets themselves are not invalidated; only future DKG / signing protocol *runs* are wire-incompatible with old peers). +- **State migration risk**: none — no on-disk format change. +- **Operability risk**: medium — wire break is proof-content rather than handshake, so a partial rollout fails opaquely (timeouts + proof rejections), not loudly (disconnect). Mitigation: §6 step 3 (hard cutover window) and step 5 (active monitoring). +- **Security posture**: this PR is itself a security hardening. Pre-hardening, tss-lib had two ceremonies with otherwise-identical inputs deriving the same SSID (zero fallback for keygen / resharing, `SHA512_256(messageBytes)` fallback for signing), breaking the session-binding the proofs rely on. Post-hardening, that fallback is removed and SSID derivation is forced from a per-ceremony nonce. The rollout cost is finite; the risk of *not* rolling is an ongoing transcript-splicing exposure surface. + +## 8. Open items / follow-ups + +- **External observability rule audit**: confirm no dashboard, alerting rule, or log-scraper pins the old session-ID string format (`-`). Grep within this repo found no consumers; out-of-tree observability tooling should be checked separately. (Severity: cosmetic / operability.) +- **Socket security alerts on the PR (`babel-traverse`, `cipher-base`, `elliptic`, `es5-ext`)**: all are npm dev-tooling vulnerabilities not introduced by this PR (it touches no `package.json` or lockfile). Out of scope; tracked at the JS-tooling layer. +- **CodeRabbit auto-review**: paused itself on this branch ("under active development"). Re-trigger before merge if a fresh AI pass is desired. +- **Constant-time follow-up**: upstream PR `#328` (broad constant-time framework) was intentionally **skipped** by the hardening pin per the upstream report; it adds dependency and is default-disabled upstream. Tracked as a separate future security project; not blocking this release. +- **Test load sensitivity**: PR body notes one combined pre-final local run hit DKG outgoing-message timeouts under load. Suggest re-running `go test -count=3 ./pkg/tecdsa/dkg ./pkg/tecdsa/signing ./pkg/tbtc` on a CI agent with realistic CPU contention to confirm no flaky-timeout regression. +- **Resharing**: tss-lib resharing is hardened (SSID broadcast, new committee rejects old-committee broadcasts), but keep-core's tBTC stack does not currently invoke `Resharing`. If a future PR adds it, the integration must call `SetSessionNonceBytes` before `NewLocalParty` for the resharing parameters — same pattern as DKG and signing in this PR. diff --git a/keep-core-release/tlabs-xyz/keep-core-security/9.md b/keep-core-release/tlabs-xyz/keep-core-security/9.md new file mode 100644 index 0000000000..56c2cc71b7 --- /dev/null +++ b/keep-core-release/tlabs-xyz/keep-core-security/9.md @@ -0,0 +1,117 @@ +# PR #9 — fix(libp2p): bound Keep handshake with a connection deadline + +- Repo: tlabs-xyz/keep-core-security +- Branch: `fix/libp2p-handshake-timeout` +- URL: https://github.com/tlabs-xyz/keep-core-security/pull/9 + +## Summary + +Arms an absolute `SetDeadline` on the encrypted connection for the duration of +the Keep authentication handshake (both inbound and outbound), then clears it +once the handshake completes. Closes a DoS surface where a peer that finished +TLS but never sent the first Keep handshake frame parked the responder +goroutine on a blocking `protodelim.UnmarshalFrom`, indefinitely holding the +libp2p resource-manager transient inbound slot. + +The 15s constant matches the libp2p upgrader's existing `defaultAcceptTimeout`. +`crypto/tls.HandshakeContext`'s deadline does not propagate to post-TLS reads, +so the upstream ctx bound alone did not cover the Keep handshake reads — this +fix closes that gap. + +## Breaking changes + +None. + +| Surface | Changed? | Notes | +|---|---|---| +| Wire protocol (`/keep/handshake/1.0.0`, `authProtocolID="keep"`) | No | Same IDs, same frame layout | +| Public Go API (`SecureInbound` / `SecureOutbound` signatures) | No | Helpers `setHandshakeDeadline` / `clearHandshakeDeadline` are package-private | +| Configuration / flags / env | No | 15s hardcoded; matches upstream `defaultAcceptTimeout` | +| `go.mod` / dependencies | No | Zero dep changes | +| Smart contracts (Solidity) | No | No `solidity/`, `*.sol`, or contract dir touched | +| Persistent state / DB | No | No state changes | + +## Behavioral change (intended) + +A peer that completes TLS but stalls during the Keep auth handshake is now +disconnected within `min(15s, ctx.Deadline())` instead of parking the responder +forever. Legitimate handshakes finish in milliseconds — unaffected. + +## Network compatibility + +Fully backward compatible. Pre-fix and post-fix nodes interoperate. No +coordinated cutover required — safe to roll out node-by-node. + +## Redeployment scope + +- **Nodes (`keep-core` client binary):** **yes** — operators must redeploy to + receive the fix. DoS-resilience fix; worth pushing. +- **Off-chain services / infra:** no — no DB migrations, no API contract + changes, no config schema bumps. +- **Smart contracts:** no — Go-only PR. + +## Risk + +Low. + +- Deadline is armed and cleared narrowly around the Keep handshake. Verified + by `TestClearHandshakeDeadlineRemovesArmedDeadline` that post-handshake + stream I/O is not subject to the handshake bound. +- The only construable regression — a peer whose Keep handshake genuinely + takes >15s — was already failing under the upgrader's own 15s ctx; this + just surfaces the failure earlier and frees the transient inbound slot. + +## Tests + +All in `pkg/net/libp2p/transport_test.go`: + +- `TestResponderHandshakeRespectsConnectionDeadline` — regression test for the + pre-fix DoS. Asserts the inbound handshake returns within the armed deadline + window (timing-based, since the underlying error is wrapped with `%v`). +- `TestSetHandshakeDeadlinePicksEarlierOfContextOrDefault` — verifies a tighter + ctx deadline wins over the 15s default. +- `TestSetHandshakeDeadlineUsesDefaultWhenNoContextDeadline` — verifies the + 15s fallback is applied when ctx has no deadline. +- `TestClearHandshakeDeadlineRemovesArmedDeadline` — verifies post-handshake + I/O is not subject to the handshake bound. + +Local: `go test ./pkg/net/libp2p/...` → 37 passed. + +## CI + +Green on `c5cd5659d`: + +- 18 SUCCESS (client-vet, client-lint, client-format, client-scan, + client-build-test-publish, contracts-build-and-test ×2, contracts-slither, + contracts-lint, contracts-deployment-dry-run, docs-publish, etc.) +- 4 intentional SKIPPED (testnet deploy, docs publish gates, client + integration test) +- 0 failures + +## Recommendation + +Safe to release. Standard rolling redeploy of `keep-core` nodes; no service +or contract redeployment needed. + +## Files changed + +``` +.github/workflows/client.yml | 6 + +.github/workflows/contracts-ecdsa-docs.yml | 3 + +.github/workflows/contracts-random-beacon-docs.yml | 3 + +pkg/net/libp2p/transport.go | 70 +++++++- +pkg/net/libp2p/transport_test.go | 182 +++++++++++++++++++++ +5 files changed, 262 insertions(+), 2 deletions(-) +``` + +The CI workflow changes grant `pull-requests: read` to the `dorny/paths-filter` +detect-changes jobs (required for that action on `pull_request` events). +Orthogonal to the libp2p fix; no runtime impact. + +## Commits on branch + +``` +c5cd5659d test(libp2p): tighten handshake-deadline regression test and clarify doc +f664d21ce ci: grant pull-requests: read to path-filter detect-changes jobs +d53a4af26 fix(libp2p): bound Keep handshake with a connection deadline +``` diff --git a/pkg/altbn128/altbn128.go b/pkg/altbn128/altbn128.go index 0455c23b4c..c3d222ec7d 100644 --- a/pkg/altbn128/altbn128.go +++ b/pkg/altbn128/altbn128.go @@ -117,9 +117,64 @@ func G2FromInts(x *gfP2, y *gfP2) (*bn256.G2, error) { return g2, err } -// G1HashToPoint hashes the provided byte slice, maps it into a G1 -// and returns it as a G1 point. +// g1HashToPointMaxAttempts is the maximum number of counter values tried by +// G1HashToPoint. Each attempt has a ~1/2 probability of yielding a valid +// point, so the probability of exhausting all attempts is (1/2)^64 ≈ 5e-20. +const g1HashToPointMaxAttempts = 64 + +// G1HashToPoint hashes the provided byte slice and maps it deterministically +// into a G1 point using a counter-based hash-and-try approach. +// +// For each counter value 0..63 the function computes SHA-256(m || counter), +// treats the digest as a candidate x-coordinate, and checks whether a +// corresponding y exists on the curve. It returns the first valid point found. +// +// This replaces the previous try-and-increment design (increment x until a +// quadratic residue is found) which had variable iteration count proportional +// to the hash output, creating a timing side channel. The counter-based +// approach makes each attempt perform identical work (one SHA-256 and one +// modular square root), bounding (but not normalising) timing across inputs: +// the loop exits on the first valid point, so execution time still varies with +// how many counters are tried. +// +// NOTE: this function produces different output than the previous +// try-and-increment implementation for the same input. Deployment requires a +// coordinated network upgrade. +// +// TODO: replace with a constant-time RFC 9380 SWU implementation. +// See: https://github.com/tlabs-xyz/keep-core-security/issues/4 func G1HashToPoint(m []byte) *bn256.G1 { + buf := make([]byte, len(m)+1) + copy(buf, m) + + for ctr := 0; ctr < g1HashToPointMaxAttempts; ctr++ { + buf[len(m)] = byte(ctr) + h := sha256.Sum256(buf) + x := mod(new(big.Int).SetBytes(h[:]), bn256.P) + if y := yFromX(x); y != nil { + g1, _ := G1FromInts(x, y) + return g1 + } + } + + // Unreachable in practice: probability (1/2)^64. + panic("G1HashToPoint: no valid curve point found for input") +} + +// G1HashToPointLegacy hashes the provided byte slice and maps it into a G1 +// point using the pre-hardening try-and-increment approach: the digest is the +// first candidate x-coordinate and is incremented until a quadratic residue +// is found. +// +// It reproduces, byte for byte, the mapping of the production releases that +// precede the counter-based G1HashToPoint, and exists solely so a ceremony +// pinned to the legacy protocol mode by the participation gate remains +// wire-compatible with peers running such a release. A ceremony uses exactly +// one of the two mappings for its entire lifetime, selected from its permit +// mode. The legacy variant's data-dependent iteration count — the timing side +// channel the counter-based design bounds — is the price of that +// compatibility; new code must use G1HashToPoint. +func G1HashToPointLegacy(m []byte) *bn256.G1 { one := big.NewInt(1) diff --git a/pkg/altbn128/altbn128_test.go b/pkg/altbn128/altbn128_test.go index 304eff948e..5702a5c052 100644 --- a/pkg/altbn128/altbn128_test.go +++ b/pkg/altbn128/altbn128_test.go @@ -2,6 +2,8 @@ package altbn128 import ( "crypto/rand" + "encoding/hex" + "math/big" "testing" bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" @@ -76,6 +78,184 @@ func TestCompressDecompressGivesSameG2Point(t *testing.T) { } } +func TestG1HashToPointDeterministic(t *testing.T) { + msg := []byte("test message for hash-to-point") + p1 := G1HashToPoint(msg) + p2 := G1HashToPoint(msg) + testutils.AssertBytesEqual(t, p1.Marshal(), p2.Marshal()) +} + +func TestG1HashToPointDistinct(t *testing.T) { + p1 := G1HashToPoint([]byte("message one")) + p2 := G1HashToPoint([]byte("message two")) + if string(p1.Marshal()) == string(p2.Marshal()) { + t.Error("distinct inputs produced the same G1 point") + } +} + +func TestG1HashToPointValidPoint(t *testing.T) { + // A valid G1 point can be marshalled and unmarshalled without error. + for _, msg := range [][]byte{ + []byte(""), + []byte("a"), + []byte("hello world"), + make([]byte, 32), + } { + p := G1HashToPoint(msg) + if p == nil { + t.Fatalf("G1HashToPoint returned nil for input %q", msg) + } + // Round-trip through Marshal/Unmarshal to confirm the point is on-curve. + recovered := new(bn256.G1) + if _, err := recovered.Unmarshal(p.Marshal()); err != nil { + t.Errorf("G1HashToPoint produced an invalid G1 point for input %q: %v", msg, err) + } + } +} + +// TestG1HashToPointWireFormat pins the marshalled G1 output for a small set of +// known inputs. G1HashToPoint's only operational consumer is the GJKR DKG +// Pedersen commitment generator H, which every group member derives from the +// shared beacon seed (pkg/beacon/gjkr/protocol_parameters.go); all members must +// derive an identical H, so its output must agree node-to-node. (The relay-entry +// path signs and verifies raw G1 points via bls.SignG1/VerifyG1 and never routes +// through this function.) Any change in its output for the same input is a +// wire-breaking change requiring a coordinated network upgrade (see +// SECURITY-BREAKING-CHANGES.md and F-02.md). If this test fails, do NOT update +// the expected values without scheduling a network cutover. +func TestG1HashToPointWireFormat(t *testing.T) { + vectors := []struct { + input []byte + expectedHex string + }{ + { + input: []byte(""), + expectedHex: "0d6b6eb73d503a452c04b979b8755971498d481ce253a35c0cd08ad866b5a58f25bba4e5ae5ce667d11a0abbe09bd0d8a5dd3cb96d9b1aa6a712522a3864aeb1", + }, + { + input: []byte("keep-core G1 pin"), + expectedHex: "0a20e79a20646662a57a1eada632447c6c966842b7ae285eaaf5ab9d3e51536512d4cb82462b309207a8aa92b8e5aa0e3eb64c1ee1bbafa84f2c1069e4358be7", + }, + { + input: []byte("relay entry v2"), + expectedHex: "0d362375b0d764011cc14db6819cb6ac72dff0c49a9ff88236c4d8ede0120817284575f93cd1444d37faa967de5eeea0fdd696321dabc9e292eb6b075a25196e", + }, + } + + for _, v := range vectors { + got := hex.EncodeToString(G1HashToPoint(v.input).Marshal()) + if got != v.expectedHex { + t.Errorf( + "G1HashToPoint(%q) output drifted -- this is a wire-breaking "+ + "change requiring a coordinated network upgrade.\n"+ + " expected: %s\n"+ + " got: %s\n"+ + "See SECURITY-BREAKING-CHANGES.md and security/findings/F-02.md.", + v.input, v.expectedHex, got, + ) + } + } +} + +// TestG1HashToPointLegacyWireFormat pins the marshalled output of the legacy +// try-and-increment mapping for a small set of known inputs. The legacy +// mapping exists solely so a ceremony pinned to the legacy protocol mode +// remains wire-compatible with peers running a pre-hardening production +// release: its output must stay byte-for-byte what those releases derive. If +// this test fails, the legacy compatibility path is broken — do NOT update +// the expected values; restore the mapping. +func TestG1HashToPointLegacyWireFormat(t *testing.T) { + vectors := []struct { + input []byte + expectedHex string + }{ + { + input: []byte(""), + expectedHex: "221f8a7714359b6db9baddee936a57adc9a8979ec2d46917b41368c0165ec33a2a05536f2b20da52c6ae18e4a02e2aec0a7f35497cfd27b9084ef5c0147b1442", + }, + { + input: []byte("keep-core G1 pin"), + expectedHex: "089fea656fe4bbf194be17dadca92032084f10647fd6f2233028e80d18f025832143081e571616093f5a1f6e352902a438f60cc7f8a75a3fe5bc8783ef2ecd28", + }, + { + input: []byte("relay entry v2"), + expectedHex: "1401d7e9e769a82e1f824e2402f66b7ac1621ede4f02160df4d96ec8000de7b713c2979faf9a76ee254e4c6a0c1c9f5fdd35fdc0533efbe85580074ebd65cf05", + }, + { + input: []byte("beacon group seed"), + expectedHex: "081822c14fff3b1aa5a665ffd7cb7a62a440c7985c931a180fe8bfcc436d7aa416614a46f32a09e169df3f78b678ba5be3ccd3bebce3423bbe1c43b01b06913d", + }, + } + + for _, v := range vectors { + got := hex.EncodeToString(G1HashToPointLegacy(v.input).Marshal()) + if got != v.expectedHex { + t.Errorf( + "G1HashToPointLegacy(%q) output drifted -- the legacy "+ + "compatibility path no longer matches pre-hardening "+ + "releases; restore the mapping instead of updating the "+ + "expected value.\n"+ + " expected: %s\n"+ + " got: %s", + v.input, v.expectedHex, got, + ) + } + } +} + +// TestG1HashToPointLegacyProperties proves the legacy mapping is +// deterministic, produces valid on-curve points, and diverges from the +// hardened counter-based mapping: the two mappings must never be conflated +// for the same ceremony. +func TestG1HashToPointLegacyProperties(t *testing.T) { + for _, msg := range [][]byte{ + []byte(""), + []byte("a"), + []byte("hello world"), + make([]byte, 32), + } { + p1 := G1HashToPointLegacy(msg) + p2 := G1HashToPointLegacy(msg) + testutils.AssertBytesEqual(t, p1.Marshal(), p2.Marshal()) + + recovered := new(bn256.G1) + if _, err := recovered.Unmarshal(p1.Marshal()); err != nil { + t.Errorf( + "G1HashToPointLegacy produced an invalid G1 point for "+ + "input %q: %v", + msg, + err, + ) + } + + hardened := G1HashToPoint(msg) + if string(p1.Marshal()) == string(hardened.Marshal()) { + t.Errorf( + "legacy and hardened mappings coincided for input %q; the "+ + "modes would be indistinguishable on the wire", + msg, + ) + } + } +} + +// TestSqrtGfP2Exponent asserts the hardcoded exponent in sqrtGfP2 equals (p^2+15)/32. +func TestSqrtGfP2Exponent(t *testing.T) { + p2 := new(big.Int).Mul(bn256.P, bn256.P) + expected := new(big.Int).Div(new(big.Int).Add(p2, big.NewInt(15)), big.NewInt(32)) + + hardcoded, ok := new(big.Int).SetString( + "14971724250519463826312126413021210649976634891596900701138993820439690427699319920245032869357433499099632259837909383182382988566862092145199781964622", + 10, + ) + if !ok { + t.Fatal("failed to parse hardcoded exponent") + } + if expected.Cmp(hardcoded) != 0 { + t.Errorf("sqrtGfP2 exponent mismatch:\n expected (p^2+15)/32 = %v\n hardcoded = %v", expected, hardcoded) + } +} + func assertEqual(t *testing.T, n int, n2 int, msg string) { if n != n2 { t.Errorf("%v: [%v] != [%v]", msg, n, n2) diff --git a/pkg/beacon/beacon.go b/pkg/beacon/beacon.go index 753380d697..5e480c0aa8 100644 --- a/pkg/beacon/beacon.go +++ b/pkg/beacon/beacon.go @@ -16,6 +16,7 @@ import ( "github.com/keep-network/keep-core/pkg/beacon/event" "github.com/keep-network/keep-core/pkg/beacon/registry" "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) var logger = log.Logger("keep-beacon") @@ -27,21 +28,59 @@ const ProtocolName = "beacon" // ensuring preconditions like staking are met, and then kicking off the // internal random beacon implementation. Returns an error if this failed, // otherwise enters a blocked loop. +// +// The participation gate is constructed once at process startup, immediately +// after the Ethereum connection, and shared with the tBTC application; this +// function receives that exact instance. A nil gate is forbidden: every +// beacon ceremony's protocol mode must derive from a permit issued by the +// shared gate. +// +// The metrics recorder is the same fixed participation recorder supplied to +// the gate. It remains non-nil when client-info is disabled (as a no-op sink), +// so the beacon quarantine path never needs an optional instrumentation +// branch in production. +// +// The quarantine persistence must be a dedicated protected namespace that no +// release's active-group scan reads: it preserves signer outputs whose +// completion the gate interrupted before an accepted on-chain publication was +// observed, and those records must never load as active signers. func Initialize( ctx context.Context, beaconChain beaconchain.Interface, netProvider net.Provider, persistence persistence.ProtectedHandle, + quarantinePersistence persistence.ProtectedHandle, scheduler *generator.Scheduler, + participationGate participation.Gate, + metricsRecorder participation.GateMetricsRecorder, ) error { + if participationGate == nil { + return fmt.Errorf("the participation gate is required") + } + if metricsRecorder == nil { + return fmt.Errorf("the participation metrics recorder is required") + } + if quarantinePersistence == nil { + return fmt.Errorf("the signer quarantine persistence is required") + } + groupRegistry := registry.NewGroupRegistry(logger, beaconChain, persistence) groupRegistry.LoadExistingGroups() + signerQuarantine := registry.NewQuarantine( + ctx, + logger, + quarantinePersistence, + ) + node := newNode( beaconChain, netProvider, groupRegistry, scheduler, + participationGate, + signerQuarantine, + metricsRecorder, ) err := sortition.MonitorPool( @@ -108,10 +147,14 @@ func Initialize( ) }() } else { - go node.ForwardSignatureShares(request.GroupPublicKey) + go node.ForwardSignatureShares( + request.GroupPublicKey, + request.BlockNumber, + ) } go node.MonitorRelayEntry( + request.PreviousEntry, request.BlockNumber, ) } diff --git a/pkg/beacon/beacon_test.go b/pkg/beacon/beacon_test.go index 9edafa0b84..b250cd5eb0 100644 --- a/pkg/beacon/beacon_test.go +++ b/pkg/beacon/beacon_test.go @@ -247,6 +247,13 @@ func (mbc *mockBeaconChain) IsEntryInProgress() (bool, error) { panic("not implemented") } +func (mbc *mockBeaconChain) RelayEntryTimeoutSettlement( + uint64, + []byte, +) (*event.RelayEntryTimeoutSettlement, error) { + panic("not implemented") +} + func (mbc *mockBeaconChain) CurrentRequestStartBlock() (*big.Int, error) { mbc.currentRequestStartBlockExecutionCount++ startBlock, err := mbc.currentRequestStartBlockFn(mbc.currentRequestStartBlockExecutionCount) diff --git a/pkg/beacon/chain/chain.go b/pkg/beacon/chain/chain.go index 9b08926256..de5ee6508b 100644 --- a/pkg/beacon/chain/chain.go +++ b/pkg/beacon/chain/chain.go @@ -45,6 +45,24 @@ type RelayEntryInterface interface { CurrentRequestPreviousEntry() ([]byte, error) // CurrentRequestGroupPublicKey returns group public key for the current request. CurrentRequestGroupPublicKey() ([]byte, error) + // RelayEntryTimeoutSettlement returns the chain's own record that the relay + // request made at the given block, over the given previous entry, was + // terminated by an accepted timeout report. + // + // It returns a nil settlement and no error whenever the chain holds no such + // record. That covers a request answered by a delivered entry, a report + // that has not been mined or accepted yet, and a canonical chain that no + // longer holds the request at all — the three cases a caller must not read + // as a penalty. An error means the chain could not be asked, which is also + // not a penalty but is worth reporting separately. + // + // Implementations must resolve the record from canonical chain state on + // every call rather than from anything the node observed earlier, so that a + // reorg which removes the request or the timeout takes the record with it. + RelayEntryTimeoutSettlement( + requestBlockNumber uint64, + requestPreviousEntry []byte, + ) (*event.RelayEntryTimeoutSettlement, error) } // GroupSelectionInterface defines the subset of the beacon chain interface that diff --git a/pkg/beacon/dkg/dkg.go b/pkg/beacon/dkg/dkg.go index 5d032a5463..f1fe041811 100644 --- a/pkg/beacon/dkg/dkg.go +++ b/pkg/beacon/dkg/dkg.go @@ -2,8 +2,10 @@ package dkg import ( "bytes" + "context" "fmt" "math/big" + "slices" "sort" "github.com/ipfs/go-log/v2" @@ -14,11 +16,60 @@ import ( "github.com/keep-network/keep-core/pkg/beacon/gjkr" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) -// ExecuteDKG runs the full distributed key generation lifecycle. +// PublicationInterruptedError reports that the release gate stopped the +// ceremony after the group key material was generated but before an accepted +// on-chain publication of the result was observed. The orphaned signer must be +// preserved through the quarantine path: the result may still have been +// accepted on chain by other members, so the share cannot be dropped, and no +// acceptance was observed locally, so the share must not be activated. +type PublicationInterruptedError struct { + // Cause is the gate error that interrupted the publication. + Cause error + // Signer carries the generated key material. Its group operators are the + // full pre-acceptance selection: the accepted result, if any, may exclude + // members, and only the offline state audit may resolve the final roster. + Signer *ThresholdSigner +} + +func (e *PublicationInterruptedError) Error() string { + return fmt.Sprintf( + "DKG result publication interrupted by the release gate "+ + "after key generation: [%v]", + e.Cause, + ) +} + +func (e *PublicationInterruptedError) Unwrap() error { + return e.Cause +} + +// ExecuteDKG runs the full distributed key generation lifecycle. The +// compatibility strategy bundle selects the ceremony's wire-sensitive +// cryptographic behavior and must be supplied explicitly. +// +// The context bounds the execution and must be the ceremony permit's context: +// canceling it aborts the protocol between block waits. The commit guard is +// consulted immediately before the terminal on-chain result submission. A +// successful return means an on-chain publication of the result was observed; +// a *PublicationInterruptedError return carries generated key material whose +// publication the gate interrupted. +// +// Beside the signer it returns the members the key material was generated with: +// the operating members of the accepted result, which are exactly the members +// whose round messages this node authenticated against the group's on-chain +// membership and accepted through every round — a member whose messages did not +// arrive, or arrived without a valid membership behind them, was marked inactive +// and is absent. That is the local view of the transcript, and the only fact +// distinguishing a key generated together with other parties from a share whose +// provenance is one party's word: every member of a finished DKG records the +// same completion and names the same group key whatever population produced it. func ExecuteDKG( + ctx context.Context, logger log.StandardLogger, seed *big.Int, memberIndex group.MemberIndex, @@ -27,12 +78,14 @@ func ExecuteDKG( channel net.BroadcastChannel, membershipValidator *group.MembershipValidator, selectedOperators []chain.Address, -) (*ThresholdSigner, error) { + strategies compatibility.Strategies, + commitGuard participation.CommitGuard, +) (*ThresholdSigner, participation.MemberIndexes, error) { beaconConfig := beaconChain.GetConfig() blockCounter, err := beaconChain.BlockCounter() if err != nil { - return nil, fmt.Errorf("failed to get block counter: [%v]", err) + return nil, nil, fmt.Errorf("failed to get block counter: [%v]", err) } gjkr.RegisterUnmarshallers(channel) @@ -41,6 +94,7 @@ func ExecuteDKG( sessionID := seed.Text(16) gjkrResult, gjkrEndBlockHeight, err := gjkr.Execute( + ctx, logger, seed, sessionID, @@ -50,21 +104,40 @@ func ExecuteDKG( channel, beaconConfig.DishonestThreshold(), membershipValidator, + strategies, startBlockHeight, ) if err != nil { - return nil, fmt.Errorf( - "[member:%v] GJKR execution failed [%v]", + return nil, nil, fmt.Errorf( + "[member:%v] GJKR execution failed [%w]", memberIndex, err, ) } + // From this point on the group key material exists. A gate interruption — + // the permit canceled or a commit fence refused — must surface the signer + // for quarantine instead of dropping it. + interruptedSigner := func(cause error) error { + return &PublicationInterruptedError{ + Cause: cause, + Signer: &ThresholdSigner{ + memberIndex: memberIndex, + groupPublicKey: gjkrResult.GroupPublicKey, + groupPrivateKeyShare: gjkrResult.GroupPrivateKeyShare, + groupPublicKeyShares: gjkrResult.GroupPublicKeyShares(), + groupOperators: selectedOperators, + }, + } + } + startPublicationBlockHeight := gjkrEndBlockHeight operatingMemberIndexes := gjkrResult.Group.OperatingMemberIndexes() - dkgResultChannel := make(chan *event.DKGResultSubmission) + // The buffer lets an in-flight event callback complete after the consumer + // returned on cancellation or timeout, instead of blocking forever. + dkgResultChannel := make(chan *event.DKGResultSubmission, 1) dkgResultSubscription := beaconChain.OnDKGResultSubmitted( func(event *event.DKGResultSubmission) { dkgResultChannel <- event @@ -73,6 +146,7 @@ func ExecuteDKG( defer dkgResultSubscription.Unsubscribe() err = dkgResult.Publish( + ctx, logger, sessionID, memberIndex, @@ -83,8 +157,13 @@ func ExecuteDKG( beaconChain, blockCounter, startPublicationBlockHeight, + commitGuard, ) if err != nil { + if isGateInterruption(ctx, err) { + return nil, nil, interruptedSigner(err) + } + // Result publication failed. It means that either the result this // member proposed is not supported by the majority of group members or // that the chain interaction failed. In either case, we observe the @@ -98,6 +177,7 @@ func ExecuteDKG( ) if operatingMemberIndexes, err = decideMemberFate( + ctx, memberIndex, gjkrResult, dkgResultChannel, @@ -105,7 +185,10 @@ func ExecuteDKG( beaconChain, blockCounter, ); err != nil { - return nil, err + if isGateInterruption(ctx, err) { + return nil, nil, interruptedSigner(err) + } + return nil, nil, err } } @@ -115,16 +198,42 @@ func ExecuteDKG( beaconConfig, ) if err != nil { - return nil, fmt.Errorf("failed to resolve group operators: [%v]", err) + return nil, nil, fmt.Errorf( + "failed to resolve group operators: [%v]", + err, + ) } return &ThresholdSigner{ - memberIndex: memberIndex, - groupPublicKey: gjkrResult.GroupPublicKey, - groupPrivateKeyShare: gjkrResult.GroupPrivateKeyShare, - groupPublicKeyShares: gjkrResult.GroupPublicKeyShares(), - groupOperators: groupOperators, - }, nil + memberIndex: memberIndex, + groupPublicKey: gjkrResult.GroupPublicKey, + groupPrivateKeyShare: gjkrResult.GroupPrivateKeyShare, + groupPublicKeyShares: gjkrResult.GroupPublicKeyShares(), + groupOperators: groupOperators, + }, + operatingMemberships(operatingMemberIndexes), + nil +} + +// operatingMemberships renders the members the key material was generated with, +// ascending. The ordering gives one population exactly one rendering, so two +// members' records of the same DKG compare equal; the copy keeps the rendering +// independent of a caller that reorders the group's own view afterwards. +func operatingMemberships( + operating []group.MemberIndex, +) participation.MemberIndexes { + memberships := make(participation.MemberIndexes, len(operating)) + copy(memberships, operating) + slices.Sort(memberships) + + return memberships +} + +// isGateInterruption distinguishes a release-gate decision from an ordinary +// protocol failure: the ceremony context was canceled by the gate, or the +// error chain carries a gate sentinel from a refused commit fence. +func isGateInterruption(ctx context.Context, err error) bool { + return ctx.Err() != nil || participation.IsGateRefusal(err) } // decideMemberFate decides what the member will do in case it failed @@ -132,6 +241,7 @@ func ExecuteDKG( // supports the same group public key as the one registered on-chain and // the member is not considered as misbehaving by the group. func decideMemberFate( + ctx context.Context, playerIndex group.MemberIndex, gjkrResult *gjkr.Result, dkgResultChannel chan *event.DKGResultSubmission, @@ -140,6 +250,7 @@ func decideMemberFate( blockCounter chain.BlockCounter, ) ([]group.MemberIndex, error) { dkgResultEvent, err := waitForDkgResultEvent( + ctx, dkgResultChannel, startPublicationBlockHeight, beaconChain, @@ -191,6 +302,7 @@ func decideMemberFate( } func waitForDkgResultEvent( + ctx context.Context, dkgResultChannel chan *event.DKGResultSubmission, startPublicationBlockHeight uint64, beaconChain beaconchain.Interface, @@ -212,6 +324,11 @@ func waitForDkgResultEvent( return dkgResultEvent, nil case <-timeoutBlockChannel: return nil, fmt.Errorf("DKG result publication timed out") + case <-ctx.Done(): + return nil, fmt.Errorf( + "waiting for the DKG result event canceled: [%w]", + context.Cause(ctx), + ) } } diff --git a/pkg/beacon/dkg/dkg_test.go b/pkg/beacon/dkg/dkg_test.go index af2037dac0..f31b5750c8 100644 --- a/pkg/beacon/dkg/dkg_test.go +++ b/pkg/beacon/dkg/dkg_test.go @@ -1,6 +1,7 @@ package dkg import ( + "context" "fmt" "math/big" "reflect" @@ -50,6 +51,7 @@ func TestDecideMemberFate_HappyPath(t *testing.T) { } operatingMemberIndexes, err := decideMemberFate( + context.Background(), playerIndex, gjkrResult, dkgResultChannel, @@ -85,6 +87,7 @@ func TestDecideMemberFate_NotSameGroupPublicKey(t *testing.T) { } _, err := decideMemberFate( + context.Background(), playerIndex, gjkrResult, dkgResultChannel, @@ -116,6 +119,7 @@ func TestDecideMemberFate_MemberIsMisbehaved(t *testing.T) { } _, err := decideMemberFate( + context.Background(), playerIndex, gjkrResult, dkgResultChannel, @@ -142,6 +146,7 @@ func TestDecideMemberFate_Timeout(t *testing.T) { setup() _, err := decideMemberFate( + context.Background(), playerIndex, gjkrResult, dkgResultChannel, diff --git a/pkg/beacon/dkg/result/fuzz_test.go b/pkg/beacon/dkg/result/fuzz_test.go new file mode 100644 index 0000000000..8cf372f3c3 --- /dev/null +++ b/pkg/beacon/dkg/result/fuzz_test.go @@ -0,0 +1,15 @@ +package result + +// Fuzz target for the network-message protobuf unmarshaler in this package. +// Asserts that Unmarshal never panics on arbitrary bytes: malformed input must +// return an error, not crash. + +import "testing" + +func FuzzDKGResultHashSignatureMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&DKGResultHashSignatureMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/beacon/dkg/result/publish.go b/pkg/beacon/dkg/result/publish.go index a18f3162a7..23782df27f 100644 --- a/pkg/beacon/dkg/result/publish.go +++ b/pkg/beacon/dkg/result/publish.go @@ -1,6 +1,7 @@ package result import ( + "context" "fmt" "github.com/ipfs/go-log/v2" @@ -10,6 +11,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/protocol/state" ) @@ -28,7 +30,13 @@ func RegisterUnmarshallers(channel net.BroadcastChannel) { // other signatures and results are received and accounted for. Those that match // our own result and added to the list of votes. Finally, we submit the result // along with everyone's votes. +// +// The context bounds the execution: canceling it aborts the publication +// between block waits. The commit guard is consulted immediately before the +// terminal on-chain submission; it must be the permit of the ceremony this +// publication concludes. func Publish( + ctx context.Context, logger log.StandardLogger, sessionID string, memberIndex group.MemberIndex, @@ -39,7 +47,14 @@ func Publish( beaconChain beaconchain.Interface, blockCounter chain.BlockCounter, startBlockHeight uint64, + commitGuard participation.CommitGuard, ) error { + if commitGuard == nil { + // Publishing without a fence would submit a result the release gate + // never authorized; there is no implicit default. + return fmt.Errorf("a commit guard is required to publish a DKG result") + } + initialState := &resultSigningState{ channel: channel, beaconChain: beaconChain, @@ -48,9 +63,10 @@ func Publish( result: convertGjkrResult(result), signatureMessages: make([]*DKGResultHashSignatureMessage, 0), signingStartBlockHeight: startBlockHeight, + commitGuard: commitGuard, } - stateMachine := state.NewSyncMachine(logger, channel, blockCounter, initialState) + stateMachine := state.NewSyncMachine(logger, ctx, channel, blockCounter, initialState) lastState, _, err := stateMachine.Execute(startBlockHeight) if err != nil { diff --git a/pkg/beacon/dkg/result/states.go b/pkg/beacon/dkg/result/states.go index 483605090e..8243ac869d 100644 --- a/pkg/beacon/dkg/result/states.go +++ b/pkg/beacon/dkg/result/states.go @@ -8,6 +8,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/protocol/state" ) @@ -39,6 +40,8 @@ type resultSigningState struct { signatureMessages []*DKGResultHashSignatureMessage signingStartBlockHeight uint64 + + commitGuard participation.CommitGuard } func (rss *resultSigningState) DelayBlocks() uint64 { @@ -107,6 +110,7 @@ func (rss *resultSigningState) Next() (state.SyncState, error) { verificationStartBlockHeight: rss.signingStartBlockHeight + rss.DelayBlocks() + rss.ActiveBlocks(), + commitGuard: rss.commitGuard, }, nil } @@ -133,6 +137,8 @@ type signaturesVerificationState struct { validSignatures map[group.MemberIndex][]byte verificationStartBlockHeight uint64 + + commitGuard participation.CommitGuard } func (svs *signaturesVerificationState) DelayBlocks() uint64 { @@ -171,6 +177,7 @@ func (svs *signaturesVerificationState) Next() (state.SyncState, error) { submissionStartBlockHeight: svs.verificationStartBlockHeight + svs.DelayBlocks() + svs.ActiveBlocks(), + commitGuard: svs.commitGuard, }, nil } @@ -194,6 +201,8 @@ type resultSubmissionState struct { signatures map[group.MemberIndex][]byte submissionStartBlockHeight uint64 + + commitGuard participation.CommitGuard } func (rss *resultSubmissionState) DelayBlocks() uint64 { @@ -210,11 +219,13 @@ func (rss *resultSubmissionState) ActiveBlocks() uint64 { func (rss *resultSubmissionState) Initiate(ctx context.Context) error { return rss.member.SubmitDKGResult( + ctx, rss.result, rss.signatures, rss.beaconChain, rss.blockCounter, rss.submissionStartBlockHeight, + rss.commitGuard, ) } diff --git a/pkg/beacon/dkg/result/submission.go b/pkg/beacon/dkg/result/submission.go index a94d88ead1..6747fe2ba2 100644 --- a/pkg/beacon/dkg/result/submission.go +++ b/pkg/beacon/dkg/result/submission.go @@ -1,13 +1,16 @@ package result import ( + "context" "fmt" + "github.com/ipfs/go-log/v2" beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" "github.com/keep-network/keep-core/pkg/beacon/event" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) // SubmittingMember represents a member submitting a DKG result to the @@ -50,14 +53,26 @@ func NewSubmittingMember( // successfully submitted on chain by the member. In case of failure or result // already submitted by another member it returns `0`. // +// The context bounds the eligibility wait: canceling it aborts the submission +// before the chain call. The commit guard is consulted immediately before the +// terminal on-chain submission; a guard refusal is a release-gate decision, +// not an ordinary submission failure. +// // See Phase 14 of the protocol specification. func (sm *SubmittingMember) SubmitDKGResult( + ctx context.Context, result *beaconchain.DKGResult, signatures map[group.MemberIndex][]byte, chainRelay beaconchain.Interface, blockCounter chain.BlockCounter, startBlockHeight uint64, + commitGuard participation.CommitGuard, ) error { + if commitGuard == nil { + // Submitting without a fence would publish a result the release gate + // never authorized; there is no implicit default. + return fmt.Errorf("a commit guard is required to submit a DKG result") + } config := chainRelay.GetConfig() // Chain rejects the result if it has less than 25% safety margin. @@ -72,7 +87,9 @@ func (sm *SubmittingMember) SubmitDKGResult( ) } - onSubmittedResultChan := make(chan uint64) + // The buffer lets an in-flight event callback complete after the consumer + // returned on cancellation, instead of blocking forever. + onSubmittedResultChan := make(chan uint64, 1) subscription := chainRelay.OnDKGResultSubmitted( func(event *event.DKGResultSubmission) { @@ -117,6 +134,19 @@ func (sm *SubmittingMember) SubmitDKGResult( // submitting the result. subscription.Unsubscribe() + // The last-moment completion fence, immediately before the + // terminal chain call: a ceremony that lost its permit to clock + // failure, quiescence, or the shutdown deadline must not submit. + if err := commitGuard.CheckCommit( + "beacon_dkg_result_submission", + participation.CompletionCommit, + ); err != nil { + return fmt.Errorf( + "DKG result submission refused by the release gate: [%w]", + err, + ) + } + sm.logger.Infof( "[member:%v] submitting DKG result with public key [0x%x] and "+ "[%v] supporting member signatures at block [%v]", @@ -140,6 +170,11 @@ func (sm *SubmittingMember) SubmitDKGResult( // A result has been submitted by other member. Leave without // publishing the result. return nil + case <-ctx.Done(): + return fmt.Errorf( + "DKG result submission canceled: [%w]", + context.Cause(ctx), + ) } } } diff --git a/pkg/beacon/dkg/result/submission_test.go b/pkg/beacon/dkg/result/submission_test.go index 7d2237fa01..73d75c39cd 100644 --- a/pkg/beacon/dkg/result/submission_test.go +++ b/pkg/beacon/dkg/result/submission_test.go @@ -1,6 +1,8 @@ package result import ( + "context" + "errors" "testing" "github.com/keep-network/keep-core/internal/testutils" @@ -8,9 +10,40 @@ import ( beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) +// testCommitPermit issues a real gate permit over the given block counter with +// the developer-only disabled schedule, so submissions exercise the production +// commit fence. The returned permit doubles as the commit guard. +func testCommitPermit( + t *testing.T, + blockCounter chain.BlockCounter, +) participation.Permit { + t.Helper() + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin(participation.BeaconDKG, 0) + if err != nil { + t.Fatal(err) + } + t.Cleanup(permit.Close) + + return permit +} + func TestSubmitDKGResult(t *testing.T) { honestThreshold := 3 groupSize := 5 @@ -77,11 +110,13 @@ func TestSubmitDKGResult(t *testing.T) { } err = member.SubmitDKGResult( + context.Background(), result, signatures, beaconChain, blockCounter, initialBlockHeight, + testCommitPermit(t, blockCounter), ) if err != nil { t.Fatalf("\nexpected: %s\nactual: %s\n", "", err) @@ -159,54 +194,85 @@ func TestConcurrentPublishResult(t *testing.T) { } for testName, test := range tests { t.Run(testName, func(t *testing.T) { - beaconChain, blockCounter, initialBlock, err := - initChainHandle(honestThreshold, groupSize) + // Use the concrete local chain so the test can install a + // subscription-registration signal and remove the race between + // member1's result submission and member2's subscription setup. + chainHandle := local_v1.Connect(groupSize, honestThreshold) + + blockCounter, err := chainHandle.BlockCounter() + if err != nil { + t.Fatal(err) + } + + initialBlockChan, err := blockCounter.BlockHeightWaiter(1) if err != nil { t.Fatal(err) } + initialBlock := <-initialBlockChan - config := beaconChain.GetConfig() + config := chainHandle.GetConfig() tStep := config.ResultPublicationBlockStep expectedBlockEnd1 := initialBlock + test.expectedDuration1(tStep) expectedBlockEnd2 := initialBlock + test.expectedDuration2(tStep) + // member2 (P4) only leaves early by observing member1's (P1) + // submission event. If member1 submits before member2 installs its + // subscription, member2 misses the event and waits until its own + // much later P4 eligibility, making the test flaky under scheduler + // contention. Gate member1 behind a signal proving member2's + // subscription is installed first. The buffer absorbs member1's own + // later registration signal without blocking the chain. + subscriptionRegistered := make(chan struct{}, groupSize) + chainHandle.SetResultSubmissionRegisteredSignal(subscriptionRegistered) + result1Chan := make(chan uint64) defer close(result1Chan) result2Chan := make(chan uint64) defer close(result2Chan) + member2Permit := testCommitPermit(t, blockCounter) go func() { - err := member1.SubmitDKGResult( - test.resultToPublish1, + err := member2.SubmitDKGResult( + context.Background(), + test.resultToPublish2, signatures, - beaconChain, + chainHandle, blockCounter, initialBlock, + member2Permit, ) if err != nil { t.Error(err) } currentBlock, _ := blockCounter.CurrentBlock() - result1Chan <- currentBlock + result2Chan <- currentBlock }() + // Barrier: wait until member2 has installed its subscription before + // releasing member1. This proves both subscriptions are installed + // before member1 can submit. + <-subscriptionRegistered + + member1Permit := testCommitPermit(t, blockCounter) go func() { - err := member2.SubmitDKGResult( - test.resultToPublish2, + err := member1.SubmitDKGResult( + context.Background(), + test.resultToPublish1, signatures, - beaconChain, + chainHandle, blockCounter, initialBlock, + member1Permit, ) if err != nil { t.Error(err) } currentBlock, _ := blockCounter.CurrentBlock() - result2Chan <- currentBlock + result1Chan <- currentBlock }() if result1 := <-result1Chan; result1 != expectedBlockEnd1 { @@ -219,6 +285,193 @@ func TestConcurrentPublishResult(t *testing.T) { } } +// TestSubmitDKGResult_RefusedByGateFence proves the commit fence guards the +// terminal chain call: a permit force-canceled at the gate's shutdown deadline +// refuses the submission with the gate sentinel and nothing reaches the chain. +func TestSubmitDKGResult_RefusedByGateFence(t *testing.T) { + honestThreshold := 3 + groupSize := 5 + + beaconChain, blockCounter, initialBlockHeight, err := initChainHandle( + honestThreshold, + groupSize, + ) + if err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin(participation.BeaconDKG, 0) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + // The terminal shutdown force-cancels the permit: the fence must refuse + // from here on. + gate.Quiesce(errors.New("test shutdown")) + gate.Close() + + result := &beaconchain.DKGResult{ + GroupPublicKey: []byte{124, 46}, + } + signatures := map[group.MemberIndex][]byte{ + 1: {101}, + 2: {102}, + 3: {103}, + 4: {104}, + } + + member := &SubmittingMember{ + logger: &testutils.MockLogger{}, + index: group.MemberIndex(1), + } + + err = member.SubmitDKGResult( + context.Background(), + result, + signatures, + beaconChain, + blockCounter, + initialBlockHeight, + permit, + ) + if !participation.IsGateRefusal(err) { + t.Fatalf("expected a gate refusal, got [%v]", err) + } + + isSubmitted, err := beaconChain.IsGroupRegistered(result.GroupPublicKey) + if err != nil { + t.Fatal(err) + } + if isSubmitted { + t.Error("expected no result submission after a fence refusal") + } +} + +// TestSubmitDKGResult_NilGuardFailsClosed proves a submission without a commit +// guard is refused before any chain interaction: there is no implicit default +// fence. +func TestSubmitDKGResult_NilGuardFailsClosed(t *testing.T) { + honestThreshold := 3 + groupSize := 5 + + beaconChain, blockCounter, initialBlockHeight, err := initChainHandle( + honestThreshold, + groupSize, + ) + if err != nil { + t.Fatal(err) + } + + result := &beaconchain.DKGResult{ + GroupPublicKey: []byte{125, 47}, + } + signatures := map[group.MemberIndex][]byte{ + 1: {101}, + 2: {102}, + 3: {103}, + 4: {104}, + } + + member := &SubmittingMember{ + logger: &testutils.MockLogger{}, + index: group.MemberIndex(1), + } + + err = member.SubmitDKGResult( + context.Background(), + result, + signatures, + beaconChain, + blockCounter, + initialBlockHeight, + nil, + ) + if err == nil { + t.Fatal("expected an error for a nil commit guard") + } + + isSubmitted, err := beaconChain.IsGroupRegistered(result.GroupPublicKey) + if err != nil { + t.Fatal(err) + } + if isSubmitted { + t.Error("expected no result submission without a commit guard") + } +} + +// TestSubmitDKGResult_CanceledContext proves a canceled execution context +// aborts the eligibility wait with the cancellation cause before the chain +// call. +func TestSubmitDKGResult_CanceledContext(t *testing.T) { + honestThreshold := 3 + groupSize := 5 + + beaconChain, blockCounter, initialBlockHeight, err := initChainHandle( + honestThreshold, + groupSize, + ) + if err != nil { + t.Fatal(err) + } + + result := &beaconchain.DKGResult{ + GroupPublicKey: []byte{126, 48}, + } + signatures := map[group.MemberIndex][]byte{ + 1: {101}, + 2: {102}, + 3: {103}, + 4: {104}, + } + + // A later member index keeps the eligibility waiter pending long enough + // for the canceled context to win the select deterministically. + member := &SubmittingMember{ + logger: &testutils.MockLogger{}, + index: group.MemberIndex(5), + } + + cause := errors.New("cancellation cause") + ctx, cancel := context.WithCancelCause(context.Background()) + cancel(cause) + + err = member.SubmitDKGResult( + ctx, + result, + signatures, + beaconChain, + blockCounter, + initialBlockHeight, + testCommitPermit(t, blockCounter), + ) + if !errors.Is(err, cause) { + t.Fatalf( + "expected the cancellation cause in the error chain, got [%v]", + err, + ) + } + + isSubmitted, err := beaconChain.IsGroupRegistered(result.GroupPublicKey) + if err != nil { + t.Fatal(err) + } + if isSubmitted { + t.Error("expected no result submission after cancellation") + } +} + func initChainHandle(honestThreshold int, groupSize int) ( beaconchain.Interface, chain.BlockCounter, diff --git a/pkg/beacon/entry/entry.go b/pkg/beacon/entry/entry.go index d2ddd1ddda..3ab09943df 100644 --- a/pkg/beacon/entry/entry.go +++ b/pkg/beacon/entry/entry.go @@ -4,6 +4,8 @@ import ( "context" "encoding/hex" "fmt" + "slices" + "github.com/keep-network/keep-core/pkg/beacon/event" bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" @@ -14,6 +16,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) // RegisterUnmarshallers initializes the given broadcast channel to be able to @@ -29,7 +32,30 @@ func RegisterUnmarshallers(channel net.BroadcastChannel) { // SignAndSubmit triggers the threshold signature process for the // previous relay entry and publishes the signature to the chain as // a new relay entry. +// +// It returns the marshaled relay entry this member recovered, or nil when the +// local ceremony ended before reaching the honest threshold — because another +// member submitted the entry first, the request timed out, or the permit was +// canceled. A relay entry is deterministic for a given previous entry, so the +// returned value is the ceremony's durable result and stays reconcilable +// against the chain even if the submission below did not go through. +// +// Beside it, the memberships whose signature shares were combined into that +// entry, and nil wherever no entry was recovered. Every one of them was +// authenticated against the group public key share the DKG published for that +// membership and against this exact previous entry, so a membership named there +// held the private share behind its seat and put it into this result. That is +// the local view of the transcript, and the only fact separating a threshold +// entry several parties produced from one this node recovered among its own +// kind: a completed relay ceremony reads identically either way, so a reader +// holding only the completion has to take the population from whichever party +// reported it. +// +// The context bounds the execution and must be the ceremony permit's context: +// canceling it aborts the share exchange and the submission. The commit guard +// is consulted immediately before the terminal on-chain entry submission. func SignAndSubmit( + ctx context.Context, logger log.StandardLogger, blockCounter chain.BlockCounter, channel net.BroadcastChannel, @@ -38,11 +64,22 @@ func SignAndSubmit( honestThreshold int, signer *dkg.ThresholdSigner, startBlockHeight uint64, -) error { - ctx, cancelCtx := context.WithCancel(context.Background()) + commitGuard participation.CommitGuard, +) ([]byte, participation.MemberIndexes, error) { + if commitGuard == nil { + // Submitting without a fence would publish an entry the release gate + // never authorized; there is no implicit default. + return nil, nil, fmt.Errorf( + "a commit guard is required to sign a relay entry", + ) + } + + ctx, cancelCtx := context.WithCancel(ctx) defer cancelCtx() - relayEntrySubmittedChannel := make(chan uint64) + // The buffer lets an in-flight event callback complete after the consumer + // returned on cancellation or timeout, instead of blocking forever. + relayEntrySubmittedChannel := make(chan uint64, 1) subscription := beaconChain.OnRelayEntrySubmitted( func(event *event.RelayEntrySubmitted) { relayEntrySubmittedChannel <- event.BlockNumber @@ -56,20 +93,30 @@ func SignAndSubmit( startBlockHeight + chainConfig.RelayEntryTimeout, ) if err != nil { - return err + return nil, nil, err } previousEntry := new(bn256.G1) _, err = previousEntry.Unmarshal(previousEntryBytes) if err != nil { - return err + return nil, nil, err } selfShare := signer.CalculateSignatureShare(previousEntry) - sessionID := hex.EncodeToString(previousEntryBytes) + // Marshal the local signature share once, on this goroutine, before the + // share is used by both the broadcast goroutine and the signature-recovery + // path. bn256.G1.Marshal normalizes the point in place (MakeAffine), so + // letting broadcastShare marshal the same *bn256.G1 concurrently with + // completeSignature reading it via ScalarMult is a data race. Marshaling + // here and handing broadcastShare only the resulting bytes confines all + // access to the point to this goroutine. Normalizing to affine does not + // change the point's value, so signature recovery is unaffected. + selfShareBytes := selfShare.Marshal() + + sessionID := shareSessionID(previousEntryBytes) - go broadcastShare(ctx, logger, signer.MemberID(), selfShare, channel, sessionID) + go broadcastShare(ctx, logger, signer.MemberID(), selfShareBytes, channel, sessionID) receiveChannel := make(chan net.Message, 64) channel.Recv(ctx, func(netMessage net.Message) { @@ -124,21 +171,34 @@ func SignAndSubmit( signer.MemberID(), blockNumber, ) - return nil + return nil, nil, nil case blockNumber := <-relayEntryTimeoutChannel: - return fmt.Errorf( + return nil, nil, fmt.Errorf( "relay entry timed out at block [%v]; received [%v] valid signature shares", blockNumber, len(receivedValidShares), ) + case <-ctx.Done(): + return nil, nil, fmt.Errorf( + "relay entry signing canceled: [%w]", + context.Cause(ctx), + ) } } signature, err := completeSignature(logger, signer, receivedValidShares, honestThreshold) if err != nil { - return err + return nil, nil, err } + // Read before anything else can add to the map. The loop above exits at + // exactly the honest threshold and every share it holds went into the + // recovery, so this is the population behind the entry rather than a + // population that merely spoke on the channel. + incorporated := incorporatedMemberships(receivedValidShares) + + entryBytes := signature.Marshal() + submitter := &relayEntrySubmitter{ logger: logger, chain: beaconChain, @@ -151,26 +211,74 @@ func SignAndSubmit( // timeout signal appeared while executing the message loop. There is // still a possibility those signals appear in the future so the submitter // must be aware of them and break the execution if they occur. - return submitter.submitRelayEntry( - signature.Marshal(), + // The recovered entry is returned whatever the submission does: the local + // ceremony already reached its threshold result, and a submission refused + // by the release gate or won by another member does not undo it. + return entryBytes, incorporated, submitter.submitRelayEntry( + ctx, + entryBytes, signer.GroupPublicKeyBytes(), startBlockHeight, relayEntrySubmittedChannel, relayEntryTimeoutChannel, + commitGuard, ) } +// incorporatedMemberships renders the memberships whose shares were combined +// into a recovered entry, ascending. The ordering is what gives one population +// exactly one rendering, so two members' records of the same relay compare +// equal and a reader cannot be shown a seat twice. +func incorporatedMemberships( + shares map[group.MemberIndex]*bn256.G1, +) participation.MemberIndexes { + memberships := make(participation.MemberIndexes, 0, len(shares)) + for memberID := range shares { + memberships = append(memberships, memberID) + } + slices.Sort(memberships) + + return memberships +} + +// shareSessionID renders the broadcast session a signature share belongs to. +// +// It is the previous entry and nothing else, and it cannot become anything else +// while a release deriving it this way is on the network: the session ID travels +// on the wire, and both sides of a mixed-release group have to arrive at the same +// string for the same request or each silently filters the other's shares out as +// belonging to some other session. +// +// So a session identifies the value being signed rather than the request that +// asked for it, and the difference is visible in one place. A relay entry is +// deterministic in the previous entry, so a request following one that timed out +// without a submission carries the same previous entry as the request before it — +// and a share produced for the earlier request is then indistinguishable from one +// produced for the later one: same signing key share, same point, same session. +// +// Accepting it is sound, because the entry recovered from it is the same value +// either way. What it means is that the population behind a recovered entry +// attributes seats to that entry and not to the request an audit joined it to: a +// seat in it held the private share behind its membership and put it into this +// result, and a reader taking it for an account of which parties were live at a +// given relay request start block is reading more than the wire carries. Binding +// a share to its request would mean putting the request anchor in the message, +// which is exactly the wire change a compatibility release cannot make. +func shareSessionID(previousEntryBytes []byte) string { + return hex.EncodeToString(previousEntryBytes) +} + func broadcastShare( ctx context.Context, logger log.StandardLogger, memberID group.MemberIndex, - share *bn256.G1, + shareBytes []byte, channel net.BroadcastChannel, sessionID string, ) { message := &SignatureShareMessage{ memberID, - share.Marshal(), + shareBytes, sessionID, } diff --git a/pkg/beacon/entry/fuzz_test.go b/pkg/beacon/entry/fuzz_test.go new file mode 100644 index 0000000000..78b2345af4 --- /dev/null +++ b/pkg/beacon/entry/fuzz_test.go @@ -0,0 +1,15 @@ +package entry + +// Fuzz target for the network-message protobuf unmarshaler in this package. +// Asserts that Unmarshal never panics on arbitrary bytes: malformed input must +// return an error, not crash. + +import "testing" + +func FuzzSignatureShareMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&SignatureShareMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/beacon/entry/session_test.go b/pkg/beacon/entry/session_test.go new file mode 100644 index 0000000000..1ce02fe31f --- /dev/null +++ b/pkg/beacon/entry/session_test.go @@ -0,0 +1,41 @@ +package entry + +import "testing" + +// TestShareSessionID holds the session a signature share is filtered by to the +// encoding a mixed-release group agrees on. +// +// The session ID leaves the node. A peer running the prior release derives it +// from the same previous entry with the same encoding and drops a share carrying +// anything else, so this is a wire contract rather than an internal naming +// choice: changing what goes into it — salting it with the relay request anchor +// to bind a share to one request, or rendering the bytes another way — +// partitions a mixed group into two sessions whose shares never combine. The +// expected value here is written out rather than derived, so that such a change +// fails this test instead of following it. +func TestShareSessionID(t *testing.T) { + previousEntry := []byte{0x0a, 0xff, 0x10, 0x00} + expected := "0aff1000" + + if sessionID := shareSessionID(previousEntry); sessionID != expected { + t.Errorf( + "unexpected session ID\nexpected: [%v]\nactual: [%v]", + expected, + sessionID, + ) + } + + // Two requests are one session exactly when the value being signed is the + // same. That is what lets a share of one request combine into another over + // the same previous entry — the reason a recovered entry's population + // attributes seats to the entry rather than to the request — and what keeps + // a share from crossing between requests over different previous entries. + otherEntry := []byte{0x0a, 0xff, 0x10, 0x01} + + if shareSessionID(previousEntry) == shareSessionID(otherEntry) { + t.Errorf( + "distinct previous entries share the session [%v]", + shareSessionID(previousEntry), + ) + } +} diff --git a/pkg/beacon/entry/submission.go b/pkg/beacon/entry/submission.go index ea7b4bf3b2..19cc30b9c6 100644 --- a/pkg/beacon/entry/submission.go +++ b/pkg/beacon/entry/submission.go @@ -1,13 +1,16 @@ package entry import ( + "context" "fmt" - "github.com/ipfs/go-log/v2" "math/big" + "github.com/ipfs/go-log/v2" + beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) type relayEntrySubmitter struct { @@ -24,12 +27,18 @@ type relayEntrySubmitter struct { // tries to submit after a few blocks if member 1 did not submit and so on. // Relay entry submit process starts at block height defined by startBlockheight // parameter. +// +// The context bounds the submission loop and the commit guard is consulted +// immediately before every terminal chain submission attempt; a guard refusal +// is a release-gate decision, not an ordinary submission failure. func (res *relayEntrySubmitter) submitRelayEntry( + ctx context.Context, newEntry []byte, groupPublicKey []byte, startBlockHeight uint64, relayEntrySubmittedChannel <-chan uint64, relayEntryTimeoutChannel <-chan uint64, + commitGuard participation.CommitGuard, ) error { config := res.chain.GetConfig() @@ -50,6 +59,19 @@ func (res *relayEntrySubmitter) submitRelayEntry( for { select { case blockNumber := <-eligibleToSubmitWaiter: + // The last-moment completion fence, immediately before the + // terminal chain call: a ceremony that lost its permit to clock + // failure, quiescence, or the shutdown deadline must not submit. + if err := commitGuard.CheckCommit( + "beacon_relay_entry_submission", + participation.CompletionCommit, + ); err != nil { + return fmt.Errorf( + "relay entry submission refused by the release gate: [%w]", + err, + ) + } + res.logger.Infof( "[member:%v] submitting relay entry [0x%x] on "+ "behalf of group [0x%x] at block [%v]", @@ -116,6 +138,11 @@ func (res *relayEntrySubmitter) submitRelayEntry( "relay entry timed out at block [%v]", blockNumber, ) + case <-ctx.Done(): + return fmt.Errorf( + "relay entry submission canceled: [%w]", + context.Cause(ctx), + ) } } } diff --git a/pkg/beacon/event/deduplicator.go b/pkg/beacon/event/deduplicator.go index f22fbcdcbf..c3b8d6afed 100644 --- a/pkg/beacon/event/deduplicator.go +++ b/pkg/beacon/event/deduplicator.go @@ -62,16 +62,13 @@ func (d *Deduplicator) NotifyDKGStarted( // The cache key is the hexadecimal representation of the seed. cacheKey := newDKGSeed.Text(16) - // If the key is not in the cache, that means the seed was not handled - // yet and the client should proceed with the execution. - if !d.dkgSeedCache.Has(cacheKey) { - d.dkgSeedCache.Add(cacheKey) - return true - } - - // Otherwise, the DKG seed is a duplicate and the client should not proceed - // with the execution. - return false + // Add is mutex-serialized and atomically checks and inserts the key. It + // returns true only if the seed was not already present, meaning it was + // not handled yet and the client should proceed with the execution. + // Otherwise it returns false and the event is ignored as a duplicate. + // Performing the check and the insertion as a single atomic operation + // avoids a time-of-check to time-of-use race between concurrent callers. + return d.dkgSeedCache.Add(cacheKey) } // NotifyRelayEntryStarted notifies the client wants to start relay entry diff --git a/pkg/beacon/event/deduplicator_test.go b/pkg/beacon/event/deduplicator_test.go index ad36ce9e5c..c2ca539a4b 100644 --- a/pkg/beacon/event/deduplicator_test.go +++ b/pkg/beacon/event/deduplicator_test.go @@ -4,6 +4,8 @@ import ( "encoding/hex" "github.com/keep-network/keep-common/pkg/cache" "math/big" + "sync" + "sync/atomic" "testing" "time" ) @@ -52,6 +54,54 @@ func TestNotifyDKGStarted(t *testing.T) { } } +// TestNotifyDKGStartedConcurrent guards against a time-of-check to time-of-use +// race in NotifyDKGStarted. An earlier implementation performed a separate +// Has() check followed by Add(), so two goroutines racing on the same seed +// could both observe the key as absent and both return true, admitting the +// same DKG instance more than once. The current implementation relies on +// cache.TimeCache.Add() being mutex-serialized and returning true only for the +// first inserter. This test releases many goroutines on the same seed behind a +// barrier and asserts that exactly one caller is allowed to proceed. +func TestNotifyDKGStartedConcurrent(t *testing.T) { + const callers = 100 + + deduplicator := &Deduplicator{ + chain: &testChain{}, + dkgSeedCache: cache.NewTimeCache(testDKGSeedCachePeriod), + } + seed := big.NewInt(42) + + var wins int32 + var ready, start sync.WaitGroup + ready.Add(callers) + start.Add(1) + + results := make(chan bool, callers) + for i := 0; i < callers; i++ { + go func() { + ready.Done() + start.Wait() + results <- deduplicator.NotifyDKGStarted(seed) + }() + } + ready.Wait() + start.Done() + + for i := 0; i < callers; i++ { + if <-results { + atomic.AddInt32(&wins, 1) + } + } + if got := atomic.LoadInt32(&wins); got != 1 { + t.Fatalf( + "%d/%d concurrent NotifyDKGStarted calls returned true; "+ + "want exactly 1", + got, + callers, + ) + } +} + func TestStartRelayEntry_NoPriorRelayEntries(t *testing.T) { chain := &testChain{ currentRequestStartBlockValue: nil, diff --git a/pkg/beacon/event/event.go b/pkg/beacon/event/event.go index 72463d7d4f..8919fe0a78 100644 --- a/pkg/beacon/event/event.go +++ b/pkg/beacon/event/event.go @@ -23,6 +23,37 @@ type RelayEntryRequested struct { BlockNumber uint64 } +// RelayEntryTimeoutSettlement is the beacon's own record that a relay request +// was terminated by an accepted timeout report rather than answered by a +// delivered entry. +// +// Every field is read back off the canonical chain, and that is the point. +// A node knows only that it handed a report transaction to a provider; the +// beacon decides whether a penalty exists. Reading the record back binds the +// claim to chain state that outlives the node's view of it: a chain that no +// longer holds these logs cannot produce the record again, so a settlement +// removed by a reorg stops being claimable instead of surviving in a node's +// process memory. +type RelayEntryTimeoutSettlement struct { + // RequestID is the beacon's own identifier for the terminated request. + // Together with TerminatedGroupID it names exactly one timeout log. + RequestID *big.Int + // TerminatedGroupID is the group the beacon terminated for the timeout. + TerminatedGroupID uint64 + // RequestBlockNumber is the canonical block the terminated request was + // made at. It is what binds the settlement to the permit that reported it. + RequestBlockNumber uint64 + // RequestPreviousEntry is the entry the terminated request was signing + // over, as the beacon recorded it when the request was made. + RequestPreviousEntry []byte + // BlockNumber is the canonical block the beacon recorded the timeout at. + BlockNumber uint64 + // TransactionHash is the transaction the beacon recorded the timeout in. + TransactionHash [32]byte + // ContractAddress is the beacon contract the timeout was read from. + ContractAddress string +} + // DKGStarted represents a DKG start event. type DKGStarted struct { Seed *big.Int diff --git a/pkg/beacon/gjkr/byzantine_strategy_integration_test.go b/pkg/beacon/gjkr/byzantine_strategy_integration_test.go new file mode 100644 index 0000000000..a11313f36b --- /dev/null +++ b/pkg/beacon/gjkr/byzantine_strategy_integration_test.go @@ -0,0 +1,215 @@ +package gjkr_test + +import ( + "testing" + + "github.com/keep-network/keep-core/pkg/beacon/gjkr" + "github.com/keep-network/keep-core/pkg/internal/byzantine" + "github.com/keep-network/keep-core/pkg/internal/dkgtest" + "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// These tests exercise the Tier-2 Byzantine strategy library +// (pkg/internal/byzantine) end to end through a full DKG roundtrip, via +// dkgtest.RunTestWithStrategy. They are the first scenarios built on the +// upgraded interceptor action API. + +// isEphemeralPublicKey matches the GJKR phase-1 message. Withholding it from a +// member models that member being inactive from phase 1 onward. +func isEphemeralPublicKey(m net.TaggedMarshaler) bool { + _, ok := m.(*gjkr.EphemeralPublicKeyMessage) + return ok +} + +// isPeerShares matches the GJKR phase-3 peer-shares message. +func isPeerShares(m net.TaggedMarshaler) bool { + _, ok := m.(*gjkr.PeerSharesMessage) + return ok +} + +// isPublicKeySharePoints matches the GJKR phase-7 public-key-share-points +// message. Withholding it from a member that already provided valid phase-3 +// shares makes that member inactive AFTER it qualified into the QUAL set, which +// is what forces the share-reconstruction path (phases 11-12) to run for it. +func isPublicKeySharePoints(m net.TaggedMarshaler) bool { + _, ok := m.(*gjkr.MemberPublicKeySharePointsMessage) + return ok +} + +// TestByzantine_Withhold_member1_phase1 reproduces the hand-written +// TestExecute_IA_member1_phase1 scenario using byzantine.Withhold, proving the +// typed strategy library yields the same protocol outcome as the bespoke +// interceptor it replaces: member 1 is marked inactive, the remaining four +// members complete DKG and agree on the group key. +func TestByzantine_Withhold_member1_phase1(t *testing.T) { + t.Parallel() + + groupSize := 5 + honestThreshold := 3 + seed := dkgtest.RandomSeed(t) + + strategy := byzantine.Withhold(group.MemberIndex(1), isEphemeralPublicKey) + + result, err := dkgtest.RunTestWithStrategy(groupSize, honestThreshold, seed, strategy) + if err != nil { + t.Fatal(err) + } + + dkgtest.AssertDkgResultPublished(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, groupSize-1) + dkgtest.AssertSuccessfulSigners(t, result, []group.MemberIndex{2, 3, 4, 5}...) + dkgtest.AssertMemberFailuresCount(t, result, 1) + dkgtest.AssertSamePublicKey(t, result) + dkgtest.AssertMisbehavingMembers(t, result, group.MemberIndex(1)) + dkgtest.AssertValidGroupPublicKey(t, result) + dkgtest.AssertResultSupportingMembers(t, result, []group.MemberIndex{2, 3, 4, 5}...) +} + +// TestByzantine_Flood_member1 exercises a capability the legacy modify-or-drop +// interceptor could not express: a member duplicating every message it sends. +// The safety/liveness invariant under test is that the protocol's own +// per-sender deduplication absorbs the flood - with a single over-active member +// and groupSize-1 >= honestThreshold, DKG must still publish a valid, +// agreed-upon group key. The resulting misbehavior classification is observed +// (logged), not asserted, since it is the behavior this scenario is here to +// characterize. +// +// This is a CHARACTERIZATION test, not a Flood regression guard: every +// assertion below also holds for a fully honest run, so a Flood that silently +// regressed to pass-through would still pass here. The guard that Flood +// actually duplicates is the unit test TestFloodDuplicatesTargetMember in +// pkg/internal/byzantine. +// NOTE: deliberately NOT t.Parallel(). This scenario multiplies one member's +// message volume 5x; running it concurrently with the parallel DKG suite raises +// contention and risks the async result handler missing its 5s window - a +// timeout-miss that would surface as a spurious AssertDkgResultPublished +// failure. Per the Tier-2 work-package-0 determinism finding, high-volume +// scenarios run serially. +func TestByzantine_Flood_member1(t *testing.T) { + groupSize := 5 + honestThreshold := 3 + seed := dkgtest.RandomSeed(t) + + strategy := byzantine.Flood(group.MemberIndex(1), 5, byzantine.MatchAll) + + result, err := dkgtest.RunTestWithStrategy(groupSize, honestThreshold, seed, strategy) + if err != nil { + t.Fatal(err) + } + + // What this pins: a member duplicating all of its traffic neither breaks + // the protocol nor gets itself disqualified - every member (the flooder + // included) completes and agrees on the group key. It does not, by itself, + // isolate the absorbing mechanism (per-sender dedup); it shows the protocol + // shrugs the flood off. + dkgtest.AssertDkgResultPublished(t, result) + dkgtest.AssertValidGroupPublicKey(t, result) + dkgtest.AssertSamePublicKey(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, groupSize) + dkgtest.AssertNoMisbehavingMembers(t, result) +} + +// TestByzantine_Corrupt_member4_invalidShares reproduces the hand-written +// TestExecute_DQ_member4_invalidSharesMessage_phase4 scenario using +// byzantine.Corrupt: member 4 broadcasts a peer-shares message missing the +// share for member 1. Receivers detect the malformed message and disqualify +// the sender. The assertions below pin only the observable outcome (member 4 +// disqualified, the other four complete and agree). Reaching the +// accusation/disqualification machinery is the motivation - it is the +// stateful-protocol logic Tier 2 targets, near the contested F-008 +// reconstructed-share path - but this black-box test does not assert that the +// reconstruction path itself executes. +func TestByzantine_Corrupt_member4_invalidShares(t *testing.T) { + t.Parallel() + + groupSize := 5 + honestThreshold := 3 + seed := dkgtest.RandomSeed(t) + + strategy := byzantine.Corrupt( + group.MemberIndex(4), + isPeerShares, + func(m net.TaggedMarshaler) net.TaggedMarshaler { + m.(*gjkr.PeerSharesMessage).RemoveShares(group.MemberIndex(1)) + return m + }, + ) + + result, err := dkgtest.RunTestWithStrategy(groupSize, honestThreshold, seed, strategy) + if err != nil { + t.Fatal(err) + } + + dkgtest.AssertDkgResultPublished(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, groupSize-1) + dkgtest.AssertSuccessfulSigners(t, result, []group.MemberIndex{1, 2, 3, 5}...) + dkgtest.AssertMemberFailuresCount(t, result, 1) + dkgtest.AssertSamePublicKey(t, result) + dkgtest.AssertMisbehavingMembers(t, result, group.MemberIndex(4)) + dkgtest.AssertValidGroupPublicKey(t, result) + dkgtest.AssertResultSupportingMembers(t, result, []group.MemberIndex{1, 2, 3, 5}...) +} + +// TestByzantine_F008_ReconstructionPathExecutes is the execution-verified +// corroboration for the F-008 reachability analysis +// (docs/audits/keep-core/f008-reachability-analysis.md, verdict: false +// positive). It drives a QUAL member into the share-reconstruction path and +// confirms phase 12 completes without the contested nil-deref. +// +// F-008 claims an unguarded ScalarBaseMult(nil) in +// CombiningMember.ComputeGroupPublicKeyShares (gjkr/protocol.go phase 12), +// reachable only via its reconstruction ELSE-branch - which iterates a +// reconstructed member's peerSharesS for every operating member. No existing +// test reaches that branch: the other Byzantine demos disqualify a member in +// phase 4/5 (BEFORE the QUAL set is fixed), so the member is never +// reconstructed and the else-branch never runs. +// +// This scenario withholds member 3's PHASE-7 public-key-share-points message +// AFTER member 3 has already broadcast valid phase-3 shares. Member 3 therefore +// qualifies into QUAL, is then marked inactive in phase 8 for the missing +// points, and so satisfies needsReconstruction (in QUAL, no valid points). The +// honest members reveal their ephemeral keys for it (phase 10-11), reconstruct +// its individual key (phase 11), and at phase 12 take the else-branch for it, +// reading peerSharesS for every operating member - the exact F-008 crash site. +// +// A passing run is the evidence: ComputeGroupPublicKeyShares runs in an +// unrecovered goroutine, so a ScalarBaseMult(nil) panic would crash the test +// binary. Completion with a valid, agreed group key demonstrates that +// peerSharesS was fully populated (no gap) when the else-branch executed - +// exactly what the invariant chain (L1 inactivity gate + L2 completeness check +// + L3 recovery-failure disqualification) guarantees. +func TestByzantine_F008_ReconstructionPathExecutes(t *testing.T) { + t.Parallel() + + groupSize := 5 + honestThreshold := 3 + seed := dkgtest.RandomSeed(t) + + // Member 3 stays silent in phase 7 only; its phase-3 shares pass, so it + // enters QUAL and is reconstructed rather than excluded early. + strategy := byzantine.Withhold(group.MemberIndex(3), isPublicKeySharePoints) + + result, err := dkgtest.RunTestWithStrategy(groupSize, honestThreshold, seed, strategy) + if err != nil { + t.Fatal(err) + } + + // The four honest members complete and agree; member 3 is reconstructed + // (its key recovered from peers) but does not itself complete. + dkgtest.AssertDkgResultPublished(t, result) + dkgtest.AssertValidGroupPublicKey(t, result) + dkgtest.AssertSamePublicKey(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, groupSize-1) + dkgtest.AssertSuccessfulSigners(t, result, []group.MemberIndex{1, 2, 4, 5}...) + dkgtest.AssertMemberFailuresCount(t, result, 1) + dkgtest.AssertMisbehavingMembers(t, result, group.MemberIndex(3)) + dkgtest.AssertResultSupportingMembers(t, result, []group.MemberIndex{1, 2, 4, 5}...) + + // The teeth of the corroboration: the reconstructed-share branch executed + // for member 3 (it is in QUAL, lacks valid phase-7 points), and found + // peerSharesS fully populated - the F-008 guard never fired. Without this + // the PASS would be ambiguous, since the PR #27 guard prevents a crash even + // if the gap occurred. + dkgtest.AssertNoReconstructionGap(t, result) +} diff --git a/pkg/beacon/gjkr/cutover_integration_test.go b/pkg/beacon/gjkr/cutover_integration_test.go new file mode 100644 index 0000000000..62407a8f63 --- /dev/null +++ b/pkg/beacon/gjkr/cutover_integration_test.go @@ -0,0 +1,122 @@ +package gjkr_test + +import ( + "testing" + + "github.com/keep-network/keep-core/pkg/internal/dkgtest" + "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// TestExecute_HomogeneousLegacy proves the full DKG roundtrip succeeds when +// every member runs the legacy compatibility bundle: the pre-cutover wire +// behavior (legacy ECDH derivation and legacy hash-to-point) is a complete, +// working protocol on the production execution path, not only a set of +// primitive fixtures. +func TestExecute_HomogeneousLegacy(t *testing.T) { + t.Parallel() + + groupSize := 5 + honestThreshold := 3 + seed := dkgtest.RandomSeed(t) + + interceptor := func(msg net.TaggedMarshaler) net.TaggedMarshaler { + return msg + } + + result, err := dkgtest.RunTestWithModes( + groupSize, + honestThreshold, + seed, + interceptor, + func(group.MemberIndex) compatibility.Strategies { + return compatibility.Legacy() + }, + ) + if err != nil { + t.Fatal(err) + } + + dkgtest.AssertDkgResultPublished(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, groupSize) + dkgtest.AssertMemberFailuresCount(t, result, 0) + dkgtest.AssertSamePublicKey(t, result) + dkgtest.AssertNoMisbehavingMembers(t, result) + dkgtest.AssertValidGroupPublicKey(t, result) +} + +// TestExecute_HomogeneousSecurityV2Explicit proves the same roundtrip with an +// explicitly selected security-v2 bundle for every member. The default +// harness already pins security-v2, so this pins that the explicit selector +// path is equivalent to it. +func TestExecute_HomogeneousSecurityV2Explicit(t *testing.T) { + t.Parallel() + + groupSize := 5 + honestThreshold := 3 + seed := dkgtest.RandomSeed(t) + + interceptor := func(msg net.TaggedMarshaler) net.TaggedMarshaler { + return msg + } + + result, err := dkgtest.RunTestWithModes( + groupSize, + honestThreshold, + seed, + interceptor, + func(group.MemberIndex) compatibility.Strategies { + return compatibility.SecurityV2() + }, + ) + if err != nil { + t.Fatal(err) + } + + dkgtest.AssertDkgResultPublished(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, groupSize) + dkgtest.AssertMemberFailuresCount(t, result, 0) + dkgtest.AssertSamePublicKey(t, result) + dkgtest.AssertNoMisbehavingMembers(t, result) + dkgtest.AssertValidGroupPublicKey(t, result) +} + +// TestExecute_MixedModeFailsClosed proves a partially incompatible ceremony +// fails closed: with three legacy members and two security-v2 members under a +// four-member honest threshold, neither same-mode cohort can reach the +// threshold, so no member may produce a threshold signer and no result may +// reach the chain. Cross-mode members cannot decrypt each other's shares and +// derive different commitment generators, so both cohorts see the other as +// misbehaving. +func TestExecute_MixedModeFailsClosed(t *testing.T) { + t.Parallel() + + groupSize := 5 + honestThreshold := 4 + seed := dkgtest.RandomSeed(t) + + interceptor := func(msg net.TaggedMarshaler) net.TaggedMarshaler { + return msg + } + + result, err := dkgtest.RunTestWithModes( + groupSize, + honestThreshold, + seed, + interceptor, + func(memberIndex group.MemberIndex) compatibility.Strategies { + if memberIndex <= 3 { + return compatibility.Legacy() + } + return compatibility.SecurityV2() + }, + ) + if err != nil { + t.Fatal(err) + } + + dkgtest.AssertNoDkgResultPublished(t, result) + dkgtest.AssertSuccessfulSignersCount(t, result, 0) + dkgtest.AssertMemberFailuresCount(t, result, groupSize) +} diff --git a/pkg/beacon/gjkr/export_test.go b/pkg/beacon/gjkr/export_test.go index 1a9200e4a0..5c94796146 100644 --- a/pkg/beacon/gjkr/export_test.go +++ b/pkg/beacon/gjkr/export_test.go @@ -133,6 +133,10 @@ func (mekm *MisbehavedEphemeralKeysMessage) RemovePrivateKey( delete(mekm.privateKeys, memberIndex) } +func GjkrEcdhInfo(id1, id2 group.MemberIndex) []byte { + return gjkrEcdhInfo(id1, id2) +} + func GeneratePolynomial(degree int) ([]*big.Int, error) { return generatePolynomial(degree) } diff --git a/pkg/beacon/gjkr/fuzz_test.go b/pkg/beacon/gjkr/fuzz_test.go new file mode 100644 index 0000000000..dedc41415a --- /dev/null +++ b/pkg/beacon/gjkr/fuzz_test.go @@ -0,0 +1,63 @@ +package gjkr + +// Fuzz targets for the network-message protobuf unmarshalers in this package. +// Each asserts that Unmarshal never panics on arbitrary bytes: malformed input +// must return an error, not crash. + +import "testing" + +func FuzzEphemeralPublicKeyMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&EphemeralPublicKeyMessage{}).Unmarshal(data) + }) +} + +func FuzzMemberCommitmentsMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&MemberCommitmentsMessage{}).Unmarshal(data) + }) +} + +func FuzzPeerSharesMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&PeerSharesMessage{}).Unmarshal(data) + }) +} + +func FuzzSecretSharesAccusationsMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&SecretSharesAccusationsMessage{}).Unmarshal(data) + }) +} + +func FuzzMemberPublicKeySharePointsMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&MemberPublicKeySharePointsMessage{}).Unmarshal(data) + }) +} + +func FuzzPointsAccusationsMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&PointsAccusationsMessage{}).Unmarshal(data) + }) +} + +func FuzzMisbehavedEphemeralKeysMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&MisbehavedEphemeralKeysMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/beacon/gjkr/gjkr.go b/pkg/beacon/gjkr/gjkr.go index 99a7518e6b..6376c77b12 100644 --- a/pkg/beacon/gjkr/gjkr.go +++ b/pkg/beacon/gjkr/gjkr.go @@ -1,6 +1,7 @@ package gjkr import ( + "context" "fmt" "math/big" @@ -8,6 +9,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/protocol/state" ) @@ -53,7 +55,16 @@ func RegisterUnmarshallers(channel net.BroadcastChannel) { // If the generation is successful, it returns a threshold group member which // can participate in the signing group; if the generation fails, it returns an // error. +// +// The compatibility strategy bundle carries every wire- and +// transcript-sensitive cryptographic decision of the ceremony — the ECDH +// derivation and the hash-to-point mapping behind the Pedersen generator H — +// and must be supplied explicitly; there is no implicit default mode. +// +// The context bounds the execution: canceling it aborts the protocol between +// block waits and the error carries the cancellation cause. func Execute( + ctx context.Context, logger log.StandardLogger, seed *big.Int, sessionID string, @@ -63,6 +74,7 @@ func Execute( channel net.BroadcastChannel, dishonestThreshold int, membershipValidator *group.MembershipValidator, + strategies compatibility.Strategies, startBlockHeight uint64, ) (*Result, uint64, error) { logger.Debugf("[member:%v] initializing member", memberIndex) @@ -75,6 +87,7 @@ func Execute( membershipValidator, seed, sessionID, + strategies, ) if err != nil { return nil, 0, fmt.Errorf("cannot create a new member: [%v]", err) @@ -85,7 +98,7 @@ func Execute( member: member.InitializeEphemeralKeysGeneration(), } - stateMachine := state.NewSyncMachine(logger, channel, blockCounter, initialState) + stateMachine := state.NewSyncMachine(logger, ctx, channel, blockCounter, initialState) lastState, endBlockHeight, err := stateMachine.Execute(startBlockHeight) if err != nil { diff --git a/pkg/beacon/gjkr/integration_test.go b/pkg/beacon/gjkr/integration_test.go index 12dbccba66..d1812e6759 100644 --- a/pkg/beacon/gjkr/integration_test.go +++ b/pkg/beacon/gjkr/integration_test.go @@ -1351,6 +1351,7 @@ func (mitm *manInTheMiddle) interceptCommunication( keyPair := mitm.ephemeralKeyPairs[publicKeyMessage.SenderID()] symmetricKey := keyPair.PrivateKey.Ecdh( publicKeyMessage.GetPublicKey(mitm.senderIndex), + gjkr.GjkrEcdhInfo(mitm.senderIndex, publicKeyMessage.SenderID()), ) mitm.symmetricKeysMutex.Lock() diff --git a/pkg/beacon/gjkr/member.go b/pkg/beacon/gjkr/member.go index 55cf54c1d9..8c74fb5131 100644 --- a/pkg/beacon/gjkr/member.go +++ b/pkg/beacon/gjkr/member.go @@ -1,12 +1,14 @@ package gjkr import ( + "fmt" "math/big" "github.com/ipfs/go-log/v2" bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" "github.com/keep-network/keep-core/pkg/crypto/ephemeral" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" ) @@ -31,6 +33,12 @@ type memberCore struct { // Cryptographic protocol parameters, the same for all members in the group. protocolParameters *protocolParameters + // Compatibility strategy bundle of the ceremony this member participates + // in. Every wire- and transcript-sensitive decision — the ECDH symmetric + // key derivation and the hash-to-point mapping behind protocolParameters — + // comes from this bundle and is immutable for the member lifetime. + strategies compatibility.Strategies + // Identifier of the particular DKG session this member is part of. sessionID string } @@ -202,6 +210,13 @@ type ReconstructingMember struct { reconstructedIndividualPublicKeys map[group.MemberIndex]*bn256.G2 } +// groupPublicKeySharesResult is the outcome of phase-12 group public key share +// computation. Errors indicate the member cannot produce a valid DKG result. +type groupPublicKeySharesResult struct { + shares map[group.MemberIndex]*bn256.G2 + err error +} + // CombiningMember represents one member in a threshold sharing group who is // combining individual public keys of group members to receive group public key. // @@ -215,7 +230,9 @@ type CombiningMember struct { // Group public key shares calculated for each QUAL group member. // Public key shares calculation is time-expensive so we do it in an async // manner and publish the result to this channel, once ready. - groupPublicKeySharesChannel chan map[group.MemberIndex]*bn256.G2 + groupPublicKeySharesChannel chan groupPublicKeySharesResult + // Populated by combinationState.Initiate on the successful execution path. + computedGroupPublicKeyShares map[group.MemberIndex]*bn256.G2 } // InitializeFinalization returns a member to perform next protocol operations. @@ -231,7 +248,9 @@ type FinalizingMember struct { *CombiningMember } -// NewMember creates a new member in an initial state +// NewMember creates a new member in an initial state. The compatibility +// strategy bundle is required: an implicit cryptographic mode is forbidden, +// so a nil bundle is a construction error rather than a silent default. func NewMember( logger log.StandardLogger, memberID group.MemberIndex, @@ -240,7 +259,14 @@ func NewMember( membershipValidator *group.MembershipValidator, seed *big.Int, sessionID string, + strategies compatibility.Strategies, ) (*LocalMember, error) { + if strategies == nil { + return nil, fmt.Errorf( + "a compatibility strategy bundle is required: the cryptographic " + + "mode must be selected explicitly for the ceremony", + ) + } return &LocalMember{ memberCore: &memberCore{ logger, @@ -248,7 +274,8 @@ func NewMember( group.NewGroup(dishonestThreshold, groupSize), membershipValidator, newDkgEvidenceLog(), - newProtocolParameters(seed), + newProtocolParameters(seed, strategies), + strategies, sessionID, }, }, nil @@ -334,7 +361,7 @@ func (rm *RevealingMember) InitializeReconstruction() *ReconstructingMember { func (rm *ReconstructingMember) InitializeCombining() *CombiningMember { return &CombiningMember{ ReconstructingMember: rm, - groupPublicKeySharesChannel: make(chan map[group.MemberIndex]*bn256.G2), + groupPublicKeySharesChannel: make(chan groupPublicKeySharesResult), } } @@ -376,6 +403,7 @@ func (fm *FinalizingMember) Result() *Result { Group: fm.group, GroupPublicKey: fm.groupPublicKey, // nil if threshold not satisfied GroupPrivateKeyShare: fm.groupPrivateKeyShare, + groupPublicKeyShares: fm.computedGroupPublicKeyShares, groupPublicKeySharesChannel: fm.groupPublicKeySharesChannel, } } diff --git a/pkg/beacon/gjkr/message_filter_test.go b/pkg/beacon/gjkr/message_filter_test.go index 05beb5ba36..d7f4f185e3 100644 --- a/pkg/beacon/gjkr/message_filter_test.go +++ b/pkg/beacon/gjkr/message_filter_test.go @@ -9,6 +9,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain/local_v1" "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" ) @@ -87,6 +88,7 @@ func TestShouldAcceptMessage(t *testing.T) { membershipValdator, big.NewInt(100), "session-1", + compatibility.SecurityV2(), ) if err != nil { t.Fatal(err) diff --git a/pkg/beacon/gjkr/message_test.go b/pkg/beacon/gjkr/message_test.go index a9ae7bce53..d1a1c0c4f9 100644 --- a/pkg/beacon/gjkr/message_test.go +++ b/pkg/beacon/gjkr/message_test.go @@ -105,7 +105,7 @@ func newTestPeerSharesMessage(senderID, receiverID group.MemberIndex, shareS, sh return nil, nil, err } - key := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey) + key := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, gjkrEcdhInfo(senderID, receiverID)) msg := newPeerSharesMessage(senderID, "session-1") if err := msg.addShares(receiverID, shareS, shareT, key); err != nil { diff --git a/pkg/beacon/gjkr/protocol.go b/pkg/beacon/gjkr/protocol.go index 8cedfbcd0b..fa801d5cf8 100644 --- a/pkg/beacon/gjkr/protocol.go +++ b/pkg/beacon/gjkr/protocol.go @@ -115,9 +115,13 @@ func (sm *SymmetricKeyGeneratingMember) GenerateSymmetricKeys( otherMemberEphemeralPublicKey := ephemeralPubKeyMessage.ephemeralPublicKeys[sm.ID] // Create symmetric key for the current group member and the other - // group member by ECDH'ing the public and private key. - symmetricKey := thisMemberEphemeralPrivateKey.Ecdh( + // group member by ECDH'ing the public and private key. The derivation + // comes from the ceremony's compatibility strategy bundle so both + // sides of the exchange agree on it. + symmetricKey := sm.strategies.ECDH( + thisMemberEphemeralPrivateKey, otherMemberEphemeralPublicKey, + gjkrEcdhInfo(sm.ID, otherMember), ) sm.symmetricKeys[otherMember] = symmetricKey } @@ -667,7 +671,7 @@ func (sjm *SharesJustifyingMember) ResolveSecretSharesAccusationsMessages( sjm.discardReceivedShares(accuserID) continue } - symmetricKey := revealedAccuserPrivateKey.Ecdh(accusedPublicKey) + symmetricKey := sjm.strategies.ECDH(revealedAccuserPrivateKey, accusedPublicKey, gjkrEcdhInfo(accuserID, accusedID)) // Get from evidence log peer shares message sent by the accused // member. If the message is not present, this means the accused @@ -1108,7 +1112,7 @@ func (pjm *PointsJustifyingMember) ResolvePublicKeySharePointsAccusationsMessage pjm.group.MarkMemberAsDisqualified(accuserID) continue } - recoveredSymmetricKey := revealedAccuserPrivateKey.Ecdh(accusedPublicKey) + recoveredSymmetricKey := pjm.strategies.ECDH(revealedAccuserPrivateKey, accusedPublicKey, gjkrEcdhInfo(accuserID, accusedID)) // Get from evidence log peer shares message sent by the accused // member. If the message is not present, this means the accused @@ -1464,7 +1468,7 @@ func (rm *ReconstructingMember) recoverMisbehavedShares( rm.group.MarkMemberAsDisqualified(revealingMemberID) continue } - recoveredSymmetricKey := revealedPrivateKey.Ecdh(misbehavedMemberPublicKey) + recoveredSymmetricKey := rm.strategies.ECDH(revealedPrivateKey, misbehavedMemberPublicKey, gjkrEcdhInfo(revealingMemberID, misbehavedMemberID)) // Get from the evidence log peer shares message sent by the member // for which the private key has been revealed. @@ -1739,58 +1743,101 @@ func (cm *CombiningMember) CombineGroupPublicKey() { // from given group member. func (cm *CombiningMember) ComputeGroupPublicKeyShares() { go func() { - cm.logger.Infof( - "[member:%v] starting computation of group public key shares", - cm.ID, - ) + shares, err := cm.computeGroupPublicKeyShares() + cm.groupPublicKeySharesChannel <- groupPublicKeySharesResult{ + shares: shares, + err: err, + } + }() +} - groupPublicKeyShares := make(map[group.MemberIndex]*bn256.G2) +func (cm *CombiningMember) computeGroupPublicKeyShares() ( + map[group.MemberIndex]*bn256.G2, + error, +) { + cm.logger.Infof( + "[member:%v] starting computation of group public key shares", + cm.ID, + ) - // Calculate group public key shares for all other operating members. - for _, operatingMemberID := range cm.group.OperatingMemberIndexes() { - if operatingMemberID == cm.ID { - continue - } + groupPublicKeyShares := make(map[group.MemberIndex]*bn256.G2) - // Calculate the first public key share for the given operating - // member based on the current member public key share points. - sum := cm.publicKeyShare(operatingMemberID, cm.publicKeySharePoints) - - // Iterate through the `QUAL` set and calculate subsequent - // public key share for the given operating member based on... - for qualifiedMemberID := range cm.receivedQualifiedSharesS { - // ...received and valid member's public key share points... - if publicKeySharePoints, ok := cm.receivedValidPeerPublicKeySharePoints[qualifiedMemberID]; ok { - publicKeyShare := cm.publicKeyShare( - operatingMemberID, - publicKeySharePoints, - ) - sum = new(bn256.G2).Add(sum, publicKeyShare) - // ...OR in case given sender didn't send their public key - // share points, take their reconstructed share and recover - // the public key share. - } else { - for _, shares := range cm.revealedMisbehavedMembersShares { - if shares.misbehavedMemberID == qualifiedMemberID { - publicKeyShare := new(bn256.G2).ScalarBaseMult( - shares.peerSharesS[operatingMemberID], + // Calculate group public key shares for all other operating members. + for _, operatingMemberID := range cm.group.OperatingMemberIndexes() { + if operatingMemberID == cm.ID { + continue + } + + // Calculate the first public key share for the given operating + // member based on the current member public key share points. + sum := cm.publicKeyShare(operatingMemberID, cm.publicKeySharePoints) + + // Iterate through the `QUAL` set and calculate subsequent + // public key share for the given operating member based on... + for qualifiedMemberID := range cm.receivedQualifiedSharesS { + // ...received and valid member's public key share points... + if publicKeySharePoints, ok := cm.receivedValidPeerPublicKeySharePoints[qualifiedMemberID]; ok { + publicKeyShare := cm.publicKeyShare( + operatingMemberID, + publicKeySharePoints, + ) + sum = new(bn256.G2).Add(sum, publicKeyShare) + // ...OR in case given sender didn't send their public key + // share points, take their reconstructed share and recover + // the public key share. + } else { + for _, shares := range cm.revealedMisbehavedMembersShares { + if shares.misbehavedMemberID == qualifiedMemberID { + // Defensive guard. The DKG disqualification + // invariants should guarantee a revealed share + // exists here for every operating member. If one is + // missing we must not call ScalarBaseMult on a nil + // *big.Int, which panics and crashes this + // unrecovered goroutine (and so the whole beacon + // node). Fail closed instead of producing a wrong + // share. + peerShareS, ok := shares.peerSharesS[operatingMemberID] + if !ok || peerShareS == nil { + return nil, fmt.Errorf( + "[member:%v] missing revealed share for "+ + "operating member [%v] from misbehaved "+ + "member [%v] (unexpected per DKG invariants)", + cm.ID, + operatingMemberID, + shares.misbehavedMemberID, ) - sum = new(bn256.G2).Add(sum, publicKeyShare) } + + publicKeyShare := new(bn256.G2).ScalarBaseMult( + peerShareS, + ) + sum = new(bn256.G2).Add(sum, publicKeyShare) } } } - - groupPublicKeyShares[operatingMemberID] = sum } - cm.logger.Infof( - "[member:%v] completed computation of group public key shares", - cm.ID, - ) + groupPublicKeyShares[operatingMemberID] = sum + } - cm.groupPublicKeySharesChannel <- groupPublicKeyShares - }() + cm.logger.Infof( + "[member:%v] completed computation of group public key shares", + cm.ID, + ) + + return groupPublicKeyShares, nil +} + +// gjkrEcdhInfo returns the HKDF info label for ECDH-derived keys in the GJKR +// protocol. The pair is sorted so both peers compute the same info regardless +// of which side initiates. Each MemberIndex is encoded as a single byte; the +// compile-time assertion in pkg/protocol/group/group.go enforces the uint8 +// invariant this relies on. +func gjkrEcdhInfo(id1, id2 group.MemberIndex) []byte { + if id1 > id2 { + id1, id2 = id2, id1 + } + return []byte{'g', 'j', 'k', 'r', byte(id1), byte(id2)} } // deduplicateBySender removes duplicated items for the given sender. diff --git a/pkg/beacon/gjkr/protocol_combinations_test.go b/pkg/beacon/gjkr/protocol_combinations_test.go index 19ecf262b4..6f32460e67 100644 --- a/pkg/beacon/gjkr/protocol_combinations_test.go +++ b/pkg/beacon/gjkr/protocol_combinations_test.go @@ -95,7 +95,11 @@ func TestCombineGroupPublicKeyShares(t *testing.T) { } member.ComputeGroupPublicKeyShares() - groupPublicKeyShares := <-member.groupPublicKeySharesChannel + result := <-member.groupPublicKeySharesChannel + if result.err != nil { + t.Fatalf("unexpected error: %v", result.err) + } + groupPublicKeyShares := result.shares expectedGroupPublicKeySharesLength := 2 // groupSize - 1 (combining member) if len(groupPublicKeyShares) != expectedGroupPublicKeySharesLength { @@ -176,7 +180,11 @@ func TestCombineGroupPublicKeyShares_WithReconstruction(t *testing.T) { }} member.ComputeGroupPublicKeyShares() - groupPublicKeyShares := <-member.groupPublicKeySharesChannel + result := <-member.groupPublicKeySharesChannel + if result.err != nil { + t.Fatalf("unexpected error: %v", result.err) + } + groupPublicKeyShares := result.shares expectedGroupPublicKeySharesLength := 1 // groupSize - 1 (combining member) - 1 (inactive member) if len(groupPublicKeyShares) != expectedGroupPublicKeySharesLength { diff --git a/pkg/beacon/gjkr/protocol_ecdh_info_test.go b/pkg/beacon/gjkr/protocol_ecdh_info_test.go new file mode 100644 index 0000000000..31071529df --- /dev/null +++ b/pkg/beacon/gjkr/protocol_ecdh_info_test.go @@ -0,0 +1,51 @@ +package gjkr + +import ( + "bytes" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// TestGjkrEcdhInfoSortSymmetry verifies that the info label is independent of +// argument order: gjkrEcdhInfo(a, b) == gjkrEcdhInfo(b, a). Both peers must +// derive the same session key regardless of who initiates. +func TestGjkrEcdhInfoSortSymmetry(t *testing.T) { + for a := group.MemberIndex(1); a < group.MaxMemberIndex; a++ { + for b := group.MemberIndex(1); b < group.MaxMemberIndex; b++ { + if !bytes.Equal(gjkrEcdhInfo(a, b), gjkrEcdhInfo(b, a)) { + t.Fatalf("info label not symmetric for (%d, %d)", a, b) + } + } + } +} + +// TestGjkrEcdhInfoDistinctPerPair verifies that every distinct sorted member +// pair produces a distinct info label. This is the F-03 invariant that would +// silently break if MemberIndex is ever widened past uint8 without updating +// the encoder: peers whose IDs collide modulo 256 would share a session key. +func TestGjkrEcdhInfoDistinctPerPair(t *testing.T) { + seen := make(map[string][2]group.MemberIndex) + for a := group.MemberIndex(1); a < group.MaxMemberIndex; a++ { + for b := a; b < group.MaxMemberIndex; b++ { + label := string(gjkrEcdhInfo(a, b)) + if prev, ok := seen[label]; ok { + t.Fatalf( + "info label collision: (%d, %d) and (%d, %d) both produce %x", + prev[0], prev[1], a, b, label, + ) + } + seen[label] = [2]group.MemberIndex{a, b} + } + } +} + +// TestGjkrEcdhInfoEncoding pins the wire format. Any change here is a +// protocol-breaking change and requires a coordinated network upgrade. +func TestGjkrEcdhInfoEncoding(t *testing.T) { + got := gjkrEcdhInfo(7, 3) + want := []byte{'g', 'j', 'k', 'r', 3, 7} + if !bytes.Equal(got, want) { + t.Fatalf("encoding drift: got %v, want %v", got, want) + } +} diff --git a/pkg/beacon/gjkr/protocol_ecdh_test.go b/pkg/beacon/gjkr/protocol_ecdh_test.go index 96f6654fc8..260927e46e 100644 --- a/pkg/beacon/gjkr/protocol_ecdh_test.go +++ b/pkg/beacon/gjkr/protocol_ecdh_test.go @@ -9,6 +9,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/crypto/ephemeral" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" ) @@ -146,7 +147,11 @@ func initializeEphemeralKeyPairMembersGroup( ) []*EphemeralKeyPairGeneratingMember { dkgGroup := group.NewGroup(dishonestThreshold, groupSize) - protocolParameters := newProtocolParameters(big.NewInt(18313131145)) + strategies := compatibility.SecurityV2() + protocolParameters := newProtocolParameters( + big.NewInt(18313131145), + strategies, + ) var members []*EphemeralKeyPairGeneratingMember for i := 1; i <= groupSize; i++ { @@ -159,6 +164,7 @@ func initializeEphemeralKeyPairMembersGroup( group: dkgGroup, evidenceLog: newDkgEvidenceLog(), protocolParameters: protocolParameters, + strategies: strategies, sessionID: "session-1", }, }, @@ -223,7 +229,7 @@ func generateGroupWithEphemeralKeys( if member1.ID != member2.ID { privKey := member1.ephemeralKeyPairs[member2.ID].PrivateKey pubKey := member2.ephemeralKeyPairs[member1.ID].PublicKey - member1.symmetricKeys[member2.ID] = privKey.Ecdh(pubKey) + member1.symmetricKeys[member2.ID] = privKey.Ecdh(pubKey, gjkrEcdhInfo(member1.ID, member2.ID)) ephemeralKeys[member2.ID] = member1.ephemeralKeyPairs[member2.ID].PublicKey } diff --git a/pkg/beacon/gjkr/protocol_nilguard_test.go b/pkg/beacon/gjkr/protocol_nilguard_test.go new file mode 100644 index 0000000000..1cb5f775eb --- /dev/null +++ b/pkg/beacon/gjkr/protocol_nilguard_test.go @@ -0,0 +1,67 @@ +package gjkr + +import ( + "math/big" + "testing" + + bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// TestComputeGroupPublicKeyShares_MissingRevealedShare is regression coverage +// for the security-audit finding F-008. ComputeGroupPublicKeyShares runs in an +// unrecovered goroutine; when it falls into the reconstructed-share branch it +// computed ScalarBaseMult(shares.peerSharesS[operatingMemberID]) directly. If +// that map entry were missing, the *big.Int would be nil and ScalarBaseMult +// would panic, taking the whole beacon node down. +// +// The DKG disqualification invariants are expected to make this branch +// missing, the computation fails closed instead of panicking or producing a +// wrong share. Against the unpatched code the goroutine panics and crashes +// the test binary. +func TestComputeGroupPublicKeyShares_MissingRevealedShare(t *testing.T) { + dishonestThreshold := 1 + groupSize := 3 + + members, err := initializeCombiningMembersGroup(dishonestThreshold, groupSize) + if err != nil { + t.Fatal(err) + } + + member := members[0] + + member.publicKeySharePoints = []*bn256.G2{ + new(bn256.G2).ScalarBaseMult(big.NewInt(10)), + new(bn256.G2).ScalarBaseMult(big.NewInt(11)), + new(bn256.G2).ScalarBaseMult(big.NewInt(12)), + } + + member.receivedValidPeerPublicKeySharePoints[2] = []*bn256.G2{ + new(bn256.G2).ScalarBaseMult(big.NewInt(20)), + new(bn256.G2).ScalarBaseMult(big.NewInt(21)), + new(bn256.G2).ScalarBaseMult(big.NewInt(22)), + } + + // Member 3 became inactive and its shares were revealed in phase 11, but + // the revealed shares are MISSING the entry for operating member 2. This + // drives ComputeGroupPublicKeyShares into the reconstructed-share branch + // with shares.peerSharesS[2] == nil. + member.group.MarkMemberAsInactive(3) + delete(member.receivedValidPeerPublicKeySharePoints, 3) + member.revealedMisbehavedMembersShares = []*misbehavedShares{{ + misbehavedMemberID: 3, + peerSharesS: map[group.MemberIndex]*big.Int{ + // intentionally empty: no entry for operating member 2 + }, + }} + + member.ComputeGroupPublicKeyShares() + + result := <-member.groupPublicKeySharesChannel + if result.err == nil { + t.Fatal("expected error for missing revealed share, got nil") + } + if result.shares != nil { + t.Fatalf("expected nil shares on error, got %#v", result.shares) + } +} diff --git a/pkg/beacon/gjkr/protocol_parameters.go b/pkg/beacon/gjkr/protocol_parameters.go index 2393539074..c97674aa62 100644 --- a/pkg/beacon/gjkr/protocol_parameters.go +++ b/pkg/beacon/gjkr/protocol_parameters.go @@ -4,7 +4,8 @@ import ( "math/big" "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" - "github.com/keep-network/keep-core/pkg/altbn128" + + "github.com/keep-network/keep-core/pkg/protocol/compatibility" ) // protocolParameters holds all cryptographic parameters that must be the same @@ -19,8 +20,15 @@ type protocolParameters struct { // provided seed value which can be the previous random beacon's result. // The seed is used to evaluate `H` parameter so that the discrete logarithm of // `H` is unknown. -func newProtocolParameters(seed *big.Int) *protocolParameters { +// +// The hash-to-point mapping deriving `H` is wire-sensitive: every member of a +// group must derive an identical `H`, so the mapping comes from the ceremony's +// compatibility strategy bundle and is fixed for the ceremony lifetime. +func newProtocolParameters( + seed *big.Int, + strategies compatibility.Strategies, +) *protocolParameters { return &protocolParameters{ - H: altbn128.G1HashToPoint(seed.Bytes()), + H: strategies.G1HashToPoint(seed.Bytes()), } } diff --git a/pkg/beacon/gjkr/protocol_sharing_test.go b/pkg/beacon/gjkr/protocol_sharing_test.go index 51f38096c3..75ada89c08 100644 --- a/pkg/beacon/gjkr/protocol_sharing_test.go +++ b/pkg/beacon/gjkr/protocol_sharing_test.go @@ -9,6 +9,7 @@ import ( bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" "github.com/keep-network/keep-core/pkg/crypto/ephemeral" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" ) @@ -65,7 +66,11 @@ func TestCalculatePublicKeySharePoints(t *testing.T) { member := (&LocalMember{ memberCore: &memberCore{ - protocolParameters: newProtocolParameters(big.NewInt(8328121)), + protocolParameters: newProtocolParameters( + big.NewInt(8328121), + compatibility.SecurityV2(), + ), + strategies: compatibility.SecurityV2(), }, }).InitializeEphemeralKeysGeneration(). InitializeSymmetricKeyGeneration(). diff --git a/pkg/beacon/gjkr/result.go b/pkg/beacon/gjkr/result.go index 9896e7ba7b..f38d864bc1 100644 --- a/pkg/beacon/gjkr/result.go +++ b/pkg/beacon/gjkr/result.go @@ -21,7 +21,7 @@ type Result struct { GroupPrivateKeyShare *big.Int groupPublicKeySharesMutex sync.Mutex - groupPublicKeySharesChannel <-chan map[group.MemberIndex]*bn256.G2 + groupPublicKeySharesChannel <-chan groupPublicKeySharesResult groupPublicKeyShares map[group.MemberIndex]*bn256.G2 } @@ -43,7 +43,11 @@ func (r *Result) GroupPublicKeyShares() map[group.MemberIndex]*bn256.G2 { defer r.groupPublicKeySharesMutex.Unlock() if r.groupPublicKeyShares == nil { - r.groupPublicKeyShares = <-r.groupPublicKeySharesChannel + result := <-r.groupPublicKeySharesChannel + if result.err != nil { + return nil + } + r.groupPublicKeyShares = result.shares } return r.groupPublicKeyShares diff --git a/pkg/beacon/gjkr/states.go b/pkg/beacon/gjkr/states.go index 6d3f42919a..4a82911730 100644 --- a/pkg/beacon/gjkr/states.go +++ b/pkg/beacon/gjkr/states.go @@ -2,6 +2,7 @@ package gjkr import ( "context" + "fmt" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" @@ -681,8 +682,23 @@ func (cs *combinationState) ActiveBlocks() uint64 { } func (cs *combinationState) Initiate(ctx context.Context) error { - cs.member.ComputeGroupPublicKeyShares() + resultCh := make(chan groupPublicKeySharesResult, 1) + go func() { + shares, err := cs.member.computeGroupPublicKeyShares() + resultCh <- groupPublicKeySharesResult{shares: shares, err: err} + }() + cs.member.CombineGroupPublicKey() + + result := <-resultCh + if result.err != nil { + return fmt.Errorf( + "failed to compute group public key shares: [%w]", + result.err, + ) + } + + cs.member.computedGroupPublicKeyShares = result.shares return nil } diff --git a/pkg/beacon/integration_test.go b/pkg/beacon/integration_test.go index 760505cc85..b6cd783b1f 100644 --- a/pkg/beacon/integration_test.go +++ b/pkg/beacon/integration_test.go @@ -48,6 +48,14 @@ func TestAllMembersSigning(t *testing.T) { dkgtest.AssertSamePublicKey(t, dkgResult) entrytest.AssertEntryPublished(t, signingResult) entrytest.AssertNoSignerFailures(t, signingResult) + // Who produced the entry, which is what the release evidence records and + // what the deterministic entry value itself cannot say. + entrytest.AssertIncorporatedPopulations( + t, + signingResult, + honestThreshold, + groupSize, + ) groupPublicKey, err := getFirstGroupPublicKey(dkgResult) if err != nil { @@ -81,6 +89,14 @@ func TestHonestThresholdMembersSigning(t *testing.T) { dkgtest.AssertSamePublicKey(t, dkgResult) entrytest.AssertEntryPublished(t, signingResult) entrytest.AssertNoSignerFailures(t, signingResult) + // Who produced the entry, which is what the release evidence records and + // what the deterministic entry value itself cannot say. + entrytest.AssertIncorporatedPopulations( + t, + signingResult, + honestThreshold, + groupSize, + ) groupPublicKey, err := getFirstGroupPublicKey(dkgResult) if err != nil { @@ -153,6 +169,14 @@ func TestInactiveMemberPublicKeySharesReconstructionAndSigning(t *testing.T) { dkgtest.AssertSamePublicKey(t, dkgResult) entrytest.AssertEntryPublished(t, signingResult) entrytest.AssertNoSignerFailures(t, signingResult) + // Who produced the entry, which is what the release evidence records and + // what the deterministic entry value itself cannot say. + entrytest.AssertIncorporatedPopulations( + t, + signingResult, + honestThreshold, + groupSize, + ) groupPublicKey, err := getFirstGroupPublicKey(dkgResult) if err != nil { @@ -207,6 +231,14 @@ func TestInactivePointsAccusationsReconstructionAndSigning(t *testing.T) { dkgtest.AssertSamePublicKey(t, dkgResult) entrytest.AssertEntryPublished(t, signingResult) entrytest.AssertNoSignerFailures(t, signingResult) + // Who produced the entry, which is what the release evidence records and + // what the deterministic entry value itself cannot say. + entrytest.AssertIncorporatedPopulations( + t, + signingResult, + honestThreshold, + groupSize, + ) groupPublicKey, err := getFirstGroupPublicKey(dkgResult) if err != nil { @@ -271,6 +303,14 @@ func TestSigningWithInvalidSignatureShare(t *testing.T) { dkgtest.AssertSamePublicKey(t, dkgResult) entrytest.AssertEntryPublished(t, signingResult) entrytest.AssertNoSignerFailures(t, signingResult) + // Who produced the entry, which is what the release evidence records and + // what the deterministic entry value itself cannot say. + entrytest.AssertIncorporatedPopulations( + t, + signingResult, + honestThreshold, + groupSize, + ) groupPublicKey, err := getFirstGroupPublicKey(dkgResult) if err != nil { diff --git a/pkg/beacon/node.go b/pkg/beacon/node.go index 6bd4054d81..868b295072 100644 --- a/pkg/beacon/node.go +++ b/pkg/beacon/node.go @@ -1,11 +1,19 @@ package beacon import ( + "bytes" + "context" + "crypto/sha256" "encoding/hex" + "errors" "fmt" + "math" "math/big" + "slices" + "sync" bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + "github.com/ipfs/go-log/v2" "go.uber.org/zap" "github.com/keep-network/keep-core/pkg/altbn128" @@ -14,9 +22,13 @@ import ( "github.com/keep-network/keep-core/pkg/beacon/entry" "github.com/keep-network/keep-core/pkg/beacon/event" "github.com/keep-network/keep-core/pkg/beacon/registry" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/generator" "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) // node represents the current state of a beacon node. @@ -25,6 +37,71 @@ type node struct { netProvider net.Provider groupRegistry *registry.Groups protocolLatch *generator.ProtocolLatch + + // participationGate issues the per-ceremony participation permits that pin + // each ceremony's protocol mode from its canonical chain anchor. It is + // constructed once at process startup and shared with the tBTC + // application. + participationGate participation.Gate + + // signerQuarantine preserves signer outputs whose completion the gate + // interrupted before an accepted on-chain publication was observed. It + // writes to a dedicated protected namespace outside the active-group scan. + signerQuarantine *registry.Quarantine + + // metricsRecorder publishes the fixed participation family shared with the + // cutover gate. In particular, it records a live incomplete quarantine + // preservation even when the namespace retained no share for the offline + // audit to enumerate. + metricsRecorder participation.GateMetricsRecorder + + // quarantineMetricsMutex guards incompleteQuarantineOutputs and serializes + // its gauge publication across concurrently preserving DKG members. + quarantineMetricsMutex sync.Mutex + + // incompleteQuarantineOutputs names preservation attempts that exhausted + // their write-grace rounds and are still holding an output whose key + // material and audit record are not both durable. + incompleteQuarantineOutputs map[beaconQuarantinedSigner]struct{} +} + +// beaconQuarantinedSigner is the public, nonsecret identity of one beacon +// output in the quarantine namespace. It deduplicates repeated incomplete +// notifications for the same group seat. +type beaconQuarantinedSigner struct { + groupPublicKey string + memberIndex group.MemberIndex +} + +func beaconDKGPermitIdentity( + seed *big.Int, + memberIndex group.MemberIndex, +) participation.PermitIdentity { + seedHash := sha256.Sum256(seed.Bytes()) + return participation.PermitIdentity{ + WorkID: hex.EncodeToString(seedHash[:]), + PermitID: fmt.Sprint(memberIndex), + // The one seat this permit runs. The beacon group index is the DKG + // index, so this is also the seat the persisted signer will carry. + OperatedMembers: participation.MemberIndexes{memberIndex}, + } +} + +// beaconRelayPermitIdentity binds a relay permit to the request it answers. +// operatedMembers is the seat the permit runs for a signing membership, and +// empty for the relay permits that operate no seat — a forwarder relays other +// members' shares and computes nothing, and a timeout monitor files a penalty +// rather than a contribution. +func beaconRelayPermitIdentity( + requestStartBlock uint64, + localPermitID string, + operatedMembers ...group.MemberIndex, +) participation.PermitIdentity { + return participation.PermitIdentity{ + WorkID: participation.BeaconRelayWorkID(requestStartBlock), + PermitID: localPermitID, + OperatedMembers: participation.MemberIndexes(operatedMembers), + } } // newNode returns an empty node with no group, zero group count, and a nil last @@ -34,15 +111,21 @@ func newNode( netProvider net.Provider, groupRegistry *registry.Groups, scheduler *generator.Scheduler, + participationGate participation.Gate, + signerQuarantine *registry.Quarantine, + metricsRecorder participation.GateMetricsRecorder, ) *node { latch := generator.NewProtocolLatch() scheduler.RegisterProtocol(latch) return &node{ - beaconChain: beaconChain, - netProvider: netProvider, - groupRegistry: groupRegistry, - protocolLatch: latch, + beaconChain: beaconChain, + netProvider: netProvider, + groupRegistry: groupRegistry, + protocolLatch: latch, + participationGate: participationGate, + signerQuarantine: signerQuarantine, + metricsRecorder: metricsRecorder, } } @@ -121,6 +204,24 @@ func (n *node) JoinDKGIfEligible( len(indexes), ) + if n.participationGate == nil { + // The gate is mandatory in production; participating without it + // would select a protocol mode implicitly. Fail closed. + dkgLogger.Errorf( + "no participation gate; refusing to join DKG", + ) + return + } + + if n.signerQuarantine == nil { + // Without a quarantine store a gate interruption after key + // generation would have to drop the generated share. Fail closed. + dkgLogger.Errorf( + "no signer quarantine store; refusing to join DKG", + ) + return + } + broadcastChannel, err := n.netProvider.BroadcastChannelFor(channelName) if err != nil { dkgLogger.Errorf("failed to get broadcast channel: [%v]", err) @@ -147,11 +248,58 @@ func (n *node) JoinDKGIfEligible( // index should be in range [1, groupSize] so we need to add 1. memberIndex := index + 1 + // One participation permit per locally controlled member, issued + // immediately before the member goroutine. The permit pins the + // protocol mode from the ceremony's canonical chain anchor — the + // DKG started event block — for the ceremony's entire lifetime, + // and every wire-sensitive choice derives from the bundle it + // selects. A refusal is a gate decision, not an ordinary DKG + // failure. + permit, err := n.participationGate.Begin( + participation.BeaconDKG, + dkgStartBlockNumber, + beaconDKGPermitIdentity(dkgSeed, memberIndex), + ) + if err != nil { + dkgLogger.Warnf( + "[member:%v] refused by the participation gate: [%v]", + memberIndex, + err, + ) + continue + } + + strategies, err := compatibility.StrategiesFor(permit.Mode()) + if err != nil { + // Unreachable with a well-formed permit; refusing to + // participate is the only safe response to a mode without an + // explicit bundle. + permit.Close() + dkgLogger.Errorf( + "[member:%v] no compatibility strategies for the "+ + "permitted mode: [%v]", + memberIndex, + err, + ) + continue + } + go func() { + defer permit.Close() + n.protocolLatch.Lock() defer n.protocolLatch.Unlock() - signer, err := dkg.ExecuteDKG( + dkgLogger.Infof( + "[member:%v] joining DKG with protocol mode [%s] "+ + "[canonicalStartBlock=%v]", + memberIndex, + permit.Mode(), + permit.CanonicalStartBlock(), + ) + + signer, operatingMembers, err := dkg.ExecuteDKG( + permit.Context(), dkgLogger, dkgSeed, memberIndex, @@ -160,9 +308,35 @@ func (n *node) JoinDKGIfEligible( broadcastChannel, membershipValidator, selectedOperators, + strategies, + permit, ) if err != nil { - dkgLogger.Errorf("failed to execute dkg: [%v]", err) + var interrupted *dkg.PublicationInterruptedError + switch { + case errors.As(err, &interrupted): + // The gate stopped the ceremony after key generation + // but before an accepted publication was observed: + // preserve the orphaned share for the offline audit. + n.quarantineSigner( + dkgLogger, + dkgSeed, + memberIndex, + interrupted, + permit, + ) + case participation.IsGateRefusal(err): + // A gate decision before key generation is not an + // ordinary DKG failure. + dkgLogger.Warnf( + "[member:%v] DKG canceled by the participation "+ + "gate: [%v]", + memberIndex, + err, + ) + default: + dkgLogger.Errorf("failed to execute dkg: [%v]", err) + } return } @@ -170,7 +344,55 @@ func (n *node) JoinDKGIfEligible( signer.GroupPublicKeyBytesCompressed(), ) - // TODO: Consider snapshotting the key material just in case. + // The result reached the chain, so the share must be preserved + // durably in every outcome. The fence decides only whether this + // process may also activate it now: during quiescence or after + // a clock failure the accepted share is saved without + // activation and loads as active on the next start. + err = permit.CheckCommit( + "beacon_dkg_signer_activation", + participation.CompletionCommit, + ) + if err != nil { + dkgLogger.Warnf( + "[member:%v] activation of group [0x%v] refused by "+ + "the release gate; preserving the accepted signer "+ + "without activation: [%v]", + signer.MemberID(), + groupPublicKey, + err, + ) + if saveErr := n.groupRegistry.SaveAcceptedGroup( + signer, + groupPublicKey, + ); saveErr != nil { + dkgLogger.Errorf( + "[member:%v] failed to preserve the accepted "+ + "signer of group [0x%v]; the share is only "+ + "in memory: [%v]", + signer.MemberID(), + groupPublicKey, + saveErr, + ) + } else { + recordBeaconPermitTerminalOutcome( + dkgLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedBeaconSigner, + Reference: groupPublicKey, + MembershipIndex: signer.MemberID(), + Contribution: beaconDKGTranscriptContribution( + signer, + operatingMembers, + ), + }, + ) + } + return + } + err = n.groupRegistry.RegisterGroup(signer, groupPublicKey) if err != nil { dkgLogger.Errorf( @@ -187,6 +409,20 @@ func (n *node) JoinDKGIfEligible( signer.MemberID(), groupPublicKey, ) + recordBeaconPermitTerminalOutcome( + dkgLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedBeaconSigner, + Reference: groupPublicKey, + MembershipIndex: signer.MemberID(), + Contribution: beaconDKGTranscriptContribution( + signer, + operatingMembers, + ), + }, + ) }() } } else { @@ -194,17 +430,811 @@ func (n *node) JoinDKGIfEligible( } } +// quarantineSigner preserves a signer output whose publication the release +// gate interrupted. The share may still be part of a result other members +// published, so it cannot be dropped, and no acceptance was observed locally, +// so it must not be activated; the offline state audit reconciles it against +// the chain. A preservation failure is a WARN-level protocol violation and is +// never suppressed. +func (n *node) quarantineSigner( + dkgLogger log.StandardLogger, + dkgSeed *big.Int, + memberIndex group.MemberIndex, + interrupted *dkg.PublicationInterruptedError, + permit participation.Permit, +) { + dkgLogger.Warnf( + "[member:%v] DKG interrupted by the participation gate after key "+ + "generation; quarantining the signer output: [%v]", + memberIndex, + interrupted.Cause, + ) + + gateSnapshot := n.participationGate.State() + + channelName := hex.EncodeToString( + interrupted.Signer.GroupPublicKeyBytesCompressed(), + ) + quarantineOutput := beaconQuarantinedSigner{ + groupPublicKey: channelName, + memberIndex: memberIndex, + } + seedHash := sha256.Sum256(dkgSeed.Bytes()) + + state, err := n.signerQuarantine.Preserve( + ®istry.Membership{ + Signer: interrupted.Signer, + ChannelName: channelName, + }, + registry.QuarantinedSignerMetadata{ + ReleaseEpoch: participation.CompiledEpoch.String(), + ProtocolMode: permit.Mode().String(), + CutoverBlock: gateSnapshot.CutoverBlock, + CanonicalStartBlock: permit.CanonicalStartBlock(), + Ceremony: string(permit.Ceremony()), + SeedHash: hex.EncodeToString(seedHash[:]), + FailedOperation: "beacon_dkg_result_publication", + LastObservedBlock: gateSnapshot.CurrentBlock, + }, + // Preservation keeps running behind this. It fires once the namespace + // has refused a half for longer than a passing fault would last, so the + // node stops taking new work while it is still holding an output the + // namespace does not fully have. + func(state registry.QuarantineState, cause error) { + n.markIncompleteQuarantine(quarantineOutput) + n.blockOnIncompleteQuarantine( + dkgLogger, + memberIndex, + state, + cause, + ) + }, + ) + + // The callback above normally reports an incomplete output while Preserve + // is still retrying. A failure before the retry loop begins has no grace + // callback, and a process lifetime that ends before grace can return without + // one too, so account for both here. A completed retry removes the output + // from the live gauge; the cumulative failure counter remains as history. + if state.Complete() { + n.resolveIncompleteQuarantine(quarantineOutput) + } else { + n.markIncompleteQuarantine(quarantineOutput) + } + + // The terminal outcome needs the whole output. The audit record is what + // names the mode, canonical anchor, ceremony, seat, and refused operation of + // the preserved share; without it the offline audit cannot reconcile the + // material against the chain, so calling the permit resolved would hand the + // rollback decision a quarantine nothing explains. Either form the namespace + // took it whole in settles this — the record pair or the single handoff + // carrying both. Anything less leaves the permit unresolved, and the offline + // barrier keeps blocking on it until an operator repairs the namespace. + if !state.Complete() { + n.blockOnIncompleteQuarantine(dkgLogger, memberIndex, state, err) + return + } + + recordBeaconPermitTerminalOutcome( + dkgLogger, + permit, + participation.TerminalOutcomeQuarantined, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceQuarantinedBeaconSigner, + }, + ) +} + +// markIncompleteQuarantine publishes a newly observed incomplete preservation. +// The normal call is the live grace-exhaustion callback; the return-time call +// covers failures that reached no callback. The counter records the first +// grace-exhaustion notification for an output while that output remains +// unresolved; repeated notifications for the same group seat coalesce in the +// live gauge. Resolution removes the seat, so a later incomplete episode for +// it is counted again. The gauge remains nonzero for as long as the output lacks +// either key material or its audit record. +func (n *node) markIncompleteQuarantine( + output beaconQuarantinedSigner, +) { + n.quarantineMetricsMutex.Lock() + defer n.quarantineMetricsMutex.Unlock() + + if n.incompleteQuarantineOutputs == nil { + n.incompleteQuarantineOutputs = + make(map[beaconQuarantinedSigner]struct{}) + } + if _, exists := n.incompleteQuarantineOutputs[output]; exists { + return + } + + n.incompleteQuarantineOutputs[output] = struct{}{} + if n.metricsRecorder == nil { + return + } + + n.metricsRecorder.IncrementCounter( + clientinfo.MetricParticipationBeaconQuarantinePreservationFailuresTotal, + 1, + ) + n.metricsRecorder.SetGauge( + clientinfo.MetricParticipationBeaconQuarantineIncompleteOutputs, + float64(len(n.incompleteQuarantineOutputs)), + ) +} + +// resolveIncompleteQuarantine clears the live incomplete-output signal only +// after preservation has made the whole output durable. An output still +// incomplete when the process lifetime ends deliberately remains nonzero in +// the last readable sample; the cumulative counter is never decremented. +func (n *node) resolveIncompleteQuarantine( + output beaconQuarantinedSigner, +) { + n.quarantineMetricsMutex.Lock() + defer n.quarantineMetricsMutex.Unlock() + + if _, exists := n.incompleteQuarantineOutputs[output]; !exists { + return + } + + delete(n.incompleteQuarantineOutputs, output) + if n.metricsRecorder != nil { + n.metricsRecorder.SetGauge( + clientinfo.MetricParticipationBeaconQuarantineIncompleteOutputs, + float64(len(n.incompleteQuarantineOutputs)), + ) + } +} + +// blockOnIncompleteQuarantine stops this node from beginning new ceremonies +// while a preserved output is missing a half the namespace was supposed to hold. +// +// Either half missing leaves an inventory a rollback cannot reconcile, and a +// beacon share is worse to be unsure about than a tBTC one: the group it belongs +// to may already have an accepted result, and a member that cannot produce its +// share permanently reduces that group's usable threshold. A share that reached +// no namespace exists only in the goroutine that generated it. A share preserved +// without its audit metadata is on disk but unexplained: the mode, canonical +// anchor, ceremony, seat, and refused operation that would let the audit match +// it against the chain are exactly what did not land. +// +// Quiescence is the blocking state rather than a new one of its own: it refuses +// every new permit, it is already what the gate-state gauge and the quiesce +// counter report, and it lets the permits still running finish normally. It is +// one-way by design — an operator restarts the node once the namespace is +// repaired — which is also why the preservation behind it is given a grace +// budget first, so a namespace that clears on its own does not cost the fleet a +// node. No terminal outcome is recorded, so this permit closes unresolved and +// blocks the offline barrier on its own. +// +// The returned channel is deliberately ignored. This caller holds a permit of +// its own, so waiting for the active permit count to reach zero here would be +// waiting for itself. +func (n *node) blockOnIncompleteQuarantine( + dkgLogger log.StandardLogger, + memberIndex group.MemberIndex, + state registry.QuarantineState, + cause error, +) { + if state.KeyMaterialPersisted() { + dkgLogger.Errorf( + "[member:%v] the quarantined signer output has no audit record "+ + "explaining it; the share is preserved but a rollback cannot "+ + "reconcile it without the record; refusing new ceremonies on "+ + "this node until an operator repairs the quarantine "+ + "namespace: [%v]", + memberIndex, + cause, + ) + } else { + dkgLogger.Errorf( + "[member:%v] generated key material reached no namespace; the "+ + "share is only in memory [auditMetadataPreserved=%v]; "+ + "refusing new ceremonies on this node until an operator "+ + "resolves the quarantine namespace: [%v]", + memberIndex, + state.MetadataPersisted, + cause, + ) + } + + if n.participationGate == nil { + return + } + + n.participationGate.Quiesce(fmt.Errorf( + "beacon key material could not be preserved with its audit record: [%w]", + cause, + )) +} + +// beaconDKGTranscriptContribution renders the memberships that produced a DKG +// result: the members the key material was generated with, and this node's own +// seat among them. +// +// The local half is the seat this permit was issued for rather than every seat +// this node holds in the group. Each locally controlled member runs its own DKG +// under its own permit and publishes its own record, so the fleet's own +// memberships are covered by the union of those records; a permit may only +// speak for the ceremony it ran. +// +// A seat missing from the operating members is not written down as local. The +// gate refuses a transcript whose local memberships are not among the ones that +// produced the result, and a signer persisted from a ceremony that did not +// include it is incoherent rather than something to record — so the record fails +// closed and the offline barrier blocks on the permit instead. +func beaconDKGTranscriptContribution( + signer *dkg.ThresholdSigner, + operatingMembers participation.MemberIndexes, +) *participation.TranscriptContribution { + local := make(participation.MemberIndexes, 0, 1) + if slices.Contains(operatingMembers, signer.MemberID()) { + local = append(local, signer.MemberID()) + } + + return &participation.TranscriptContribution{ + IncorporatedMembers: operatingMembers, + LocalMembers: local, + } +} + +func recordBeaconPermitTerminalOutcome( + dkgLogger log.StandardLogger, + permit participation.Permit, + outcome participation.TerminalOutcome, + evidence participation.TerminalEvidence, +) { + if err := permit.RecordTerminalOutcome(outcome, evidence); err != nil { + dkgLogger.Warnf( + "could not persist the node-authored beacon terminal outcome "+ + "[member=%s] [outcome=%s]: [%v]", + permit.PermitID(), + outcome, + err, + ) + } +} + +// recordBeaconPermitNoThreshold records that a beacon ceremony ended without +// producing a threshold result or any durable state transition this node owns. +// It is the honest disposition for a ceremony another member finished first, +// one that timed out, and one the release gate canceled. +func recordBeaconPermitNoThreshold( + beaconLogger log.StandardLogger, + permit participation.Permit, +) { + recordBeaconPermitTerminalOutcome( + beaconLogger, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceNoThreshold, + }, + ) +} + +// recordRelayTimeoutTerminalOutcome reports the relay entry monitor's +// node-owned final disposition. The monitor exists only to file the penalty +// report, so the beacon's own record of the terminated request is its durable +// result; a monitor that ended because the group delivered on time, because the +// report failed, or because the release gate canceled it created no penalty +// state and is recorded as exhausted. +// +// A report the node merely handed to a provider is not a result. The submitting +// call returns before the transaction is mined and a transaction that reverts, +// is dropped, or never lands leaves the beacon exactly as it was, so only the +// settlement the beacon itself recorded settles the permit as completed. +// Reading the node's own submission as the penalty would let a failed report +// clear the rollback barrier that exists to hold it. +// +// The evidence carries the beacon's request identifier and terminated group +// rather than a digest of the request this node was watching. Those two fields +// are what a RelayEntryTimedOut log is made of, so the offline audit joins the +// record to an authenticated log instead of checking that a node stayed +// consistent with itself. A settlement whose reference cannot be rendered is +// recorded as exhausted: an unrenderable identity is one the audit could not +// reconcile, and a penalty the audit cannot reconcile has to keep holding the +// barrier. +func recordRelayTimeoutTerminalOutcome( + monitorLogger log.StandardLogger, + permit participation.Permit, + relayRequestBlockNumber uint64, + settlement *event.RelayEntryTimeoutSettlement, +) { + if settlement == nil { + recordBeaconPermitNoThreshold(monitorLogger, permit) + return + } + + reference, err := participation.BeaconRelayTimeoutSettlementReference( + relayRequestBlockNumber, + settlement.RequestID, + settlement.TerminatedGroupID, + ) + if err != nil { + monitorLogger.Errorf( + "the beacon's timeout settlement for the relay request of block "+ + "[%v] has no canonical identity; the report is recorded as "+ + "unsettled: [%v]", + relayRequestBlockNumber, + err, + ) + recordBeaconPermitNoThreshold(monitorLogger, permit) + return + } + + recordBeaconPermitTerminalOutcome( + monitorLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceEthereumTransaction, + Reference: reference, + }, + ) +} + +// relayEntryTimeoutReportResolutionBlocks bounds how long the monitor asks the +// beacon whether the timeout report it filed was accepted. The submitting call +// returns once the transaction reaches the provider and the transaction mines +// afterwards, so the report's effect is not visible on the first read. The +// bound is generous enough for an ordinary inclusion; the permit's context cuts +// the wait short in any case. +const relayEntryTimeoutReportResolutionBlocks = 12 + +// errRelayEntryDeadlineOverflow reports a block deadline the block range cannot +// represent. Both deadlines the relay entry monitor derives are rejected on +// overflow rather than clamped: a wrapped deadline names a block already in the +// past, and every use of one here is a fence that would silently open. +var errRelayEntryDeadlineOverflow = errors.New( + "the block deadline is not representable", +) + +// relayEntryTimeoutBlock returns the block at which the selected group has run +// out of time to deliver, or rejects a sum the block range cannot represent. +// +// The rejection is what keeps the monitor from filing a penalty nobody earned. +// A wrapped timeout block names a block the chain already passed, so the waiter +// on it fires at once and the monitor reports a request that has had no chance +// to be answered. The report is the irreversible step, so the arithmetic that +// authorizes it is checked before the monitor commits to running at all. +func relayEntryTimeoutBlock( + relayRequestBlockNumber uint64, + relayEntryTimeout uint64, +) (uint64, error) { + if relayRequestBlockNumber > math.MaxUint64-relayEntryTimeout { + return 0, fmt.Errorf( + "%w: relay request block [%v] plus the relay entry timeout of "+ + "[%v] blocks", + errRelayEntryDeadlineOverflow, + relayRequestBlockNumber, + relayEntryTimeout, + ) + } + return relayRequestBlockNumber + relayEntryTimeout, nil +} + +// relayEntryTimeoutReportResolutionDeadline returns the block the monitor stops +// asking the beacon about a filed report at, or rejects a sum the block range +// cannot represent. +// +// A wrapped bound reads as already reached, which would end the reconciliation +// before the beacon could answer and leave the permit exhausted. Rejecting it +// reaches the same disposition without pretending a bounded observation +// happened, and clamping it to the top of the range instead would widen the +// window silently. +func relayEntryTimeoutReportResolutionDeadline( + currentBlock uint64, +) (uint64, error) { + if currentBlock > math.MaxUint64-relayEntryTimeoutReportResolutionBlocks { + return 0, fmt.Errorf( + "%w: current block [%v] plus the [%v] block report resolution "+ + "bound", + errRelayEntryDeadlineOverflow, + currentBlock, + relayEntryTimeoutReportResolutionBlocks, + ) + } + return currentBlock + relayEntryTimeoutReportResolutionBlocks, nil +} + +// relayTimeoutReportSettled asks the beacon whether the relay request this +// monitor reported a timeout for was terminated because of the report, and +// returns the beacon's own record of that termination. +// +// A filed report is not a penalty. The submitting call returns once the +// transaction reaches a provider; a transaction that reverts, is dropped, or +// loses the race to another reporter leaves the beacon exactly as it was. Only +// the beacon says whether a penalty exists, so the monitor asks it and claims +// nothing it cannot read back. +// +// What it asks for is the beacon's own settlement record, not an inference from +// the in-flight request slot. The slot cannot carry the answer: a relay entry +// delivered late empties the very same slot an accepted report empties, and a +// chain view taken before the request was made is an empty slot naming the very +// previous entry the request went on to sign over. Telling those apart needs +// history the node accumulated across reads, and process-local history is not a +// canonical binding — a reorg that removes the request leaves the node's memory +// of having seen it intact, and the reading it authorizes then manufactures a +// penalty out of a chain state that no longer exists. +// +// A settlement record has no such gap. It is resolved from canonical logs on +// every call, and every component — the terminated request, the terminating +// log — has to be present in the same view for the record to exist at all. A +// reorg that removes either takes the record with it, so a claim can never +// outlive the chain state it rests on. It is also what the offline audit needs: +// the record names an authenticated log the audit can join to, instead of a +// digest this node derived from its own request. +// +// The subscription is still read, but only to refuse faster: an entry seen at +// any point ends the reconciliation against the report. It can never be what +// establishes one. +// +// Every other reading is a refusal to claim the penalty: an unreadable beacon, +// a chain that exposes no settlement records at all, a record that does not +// answer the request the monitor asked about, a beacon that recorded nothing by +// the time the wait runs out, and a resolution bound that cannot be represented +// all leave the permit exhausted. That direction is the safe one — the rollback +// barrier exists to hold a penalty nobody can account for, so an unproven report +// has to keep holding it. +func (n *node) relayTimeoutReportSettled( + ctx context.Context, + monitorLogger log.StandardLogger, + blockCounter chain.BlockCounter, + relayRequestBlockNumber uint64, + relayRequestPreviousEntry []byte, + entries <-chan *event.RelayEntrySubmitted, +) *event.RelayEntryTimeoutSettlement { + // The beacon identifies the terminated request by the previous entry it was + // signing over, so without that entry there is no request to ask about. + // Refusing here says so plainly instead of polling to a deadline. + if len(relayRequestPreviousEntry) == 0 { + monitorLogger.Warnf( + "the relay request of block [%v] names no previous entry; the "+ + "filed timeout report cannot be reconciled against the beacon "+ + "and is not claimed as settled", + relayRequestBlockNumber, + ) + return nil + } + + // entryDelivered reports whether the group submitted a relay entry. The + // read is non-blocking: the subscription's buffered channel holds a + // delivery that arrived while this loop was elsewhere, and an empty channel + // means nothing has been delivered up to this point in the window. + entryDelivered := func() bool { + select { + case entry := <-entries: + monitorLogger.Warnf( + "a relay entry was submitted at block [%v] while the filed "+ + "timeout report was being resolved; the group delivered, "+ + "so no timeout penalty is claimed", + entry.BlockNumber, + ) + return true + default: + return false + } + } + + // answersReportedRequest holds the beacon's record to the request this + // monitor was issued for. The chain handle is asked about one request, but + // a record that answers another one is a real penalty this permit did not + // earn, so the binding is checked here rather than assumed. + answersReportedRequest := func( + settlement *event.RelayEntryTimeoutSettlement, + ) error { + if settlement.RequestID == nil { + return fmt.Errorf("the settlement names no request identifier") + } + if settlement.RequestBlockNumber != relayRequestBlockNumber { + return fmt.Errorf( + "the settlement terminates the request of block [%v]", + settlement.RequestBlockNumber, + ) + } + if !bytes.Equal( + settlement.RequestPreviousEntry, + relayRequestPreviousEntry, + ) { + return fmt.Errorf( + "the settlement terminates a request signing over a different " + + "previous entry", + ) + } + return nil + } + + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + monitorLogger.Warnf( + "cannot get the current block to resolve the relay entry "+ + "timeout report; the report is not claimed as settled: [%v]", + err, + ) + return nil + } + + deadline, err := relayEntryTimeoutReportResolutionDeadline(currentBlock) + if err != nil { + monitorLogger.Warnf( + "cannot bound the relay entry timeout report resolution; the "+ + "report is not claimed as settled: [%v]", + err, + ) + return nil + } + + for { + if entryDelivered() { + return nil + } + + settlement, err := n.beaconChain.RelayEntryTimeoutSettlement( + relayRequestBlockNumber, + relayRequestPreviousEntry, + ) + if err != nil { + monitorLogger.Warnf( + "cannot read the beacon's record of the relay entry timeout "+ + "report; the report is not claimed as settled: [%v]", + err, + ) + return nil + } + if settlement != nil { + if err := answersReportedRequest(settlement); err != nil { + monitorLogger.Warnf( + "the beacon returned a timeout settlement that does not "+ + "answer the relay request of block [%v]; the filed "+ + "report is not claimed as settled: [%v]", + relayRequestBlockNumber, + err, + ) + return nil + } + + monitorLogger.Infof( + "the beacon terminated group [%v] for the relay request of "+ + "block [%v] at block [%v]; the filed timeout report was "+ + "accepted", + settlement.TerminatedGroupID, + relayRequestBlockNumber, + settlement.BlockNumber, + ) + return settlement + } + + // The beacon holds no record of the request being terminated. That is + // every view the report has not been mined into yet, so the monitor + // keeps asking to the bound. + + height, err := blockCounter.CurrentBlock() + if err != nil { + monitorLogger.Warnf( + "cannot follow the chain while resolving the relay entry "+ + "timeout report; the report is not claimed as "+ + "settled: [%v]", + err, + ) + return nil + } + if height >= deadline { + monitorLogger.Warnf( + "the beacon did not record a penalty for the relay request "+ + "of block [%v] by block [%v]; the filed timeout report is "+ + "not claimed as settled", + relayRequestBlockNumber, + height, + ) + return nil + } + + if err := blockCounter.WaitForBlockHeight(height + 1); err != nil { + monitorLogger.Warnf( + "cannot wait for the next block while resolving the relay "+ + "entry timeout report; the report is not claimed as "+ + "settled: [%v]", + err, + ) + return nil + } + if ctx.Err() != nil { + monitorLogger.Warnf( + "relay entry timeout report resolution ended before the "+ + "beacon confirmed it: [%v]", + context.Cause(ctx), + ) + return nil + } + } +} + +// recordRelayEntryTerminalOutcome reports one relay signing membership's +// node-owned final disposition. A relay entry is deterministic for a given +// previous entry, so the recovered entry itself is the ceremony's durable +// result regardless of which member published it. +// +// The record names the group and the previous entry alongside it, because the +// entry alone is a bare byte string the node asserts. Named together they are a +// threshold BLS signature the offline audit can verify against a group public +// key it decoded from the snapshot's own key material, which no node can +// produce without that group's threshold key. +// +// It names the relay request start block too, because a valid signature says +// which group signed but not which request the result answers. The gate refuses +// a record whose request does not match the permit's own, so an entry recovered +// for one request cannot become another request's result. +func recordRelayEntryTerminalOutcome( + relayLogger log.StandardLogger, + permit participation.Permit, + relayRequestStartBlock uint64, + groupPublicKey []byte, + previousEntry []byte, + relayEntry []byte, + incorporated participation.MemberIndexes, + memberships []*registry.Membership, +) { + if len(relayEntry) == 0 { + recordBeaconPermitNoThreshold(relayLogger, permit) + return + } + + // The previous entry arrives from the chain, so it is normalized to the + // canonical point encoding the audit compares against rather than trusted + // to already be one. Signing has already parsed it as a point, so this + // cannot fail for an entry that was actually signed. + canonicalPreviousEntry := new(bn256.G1) + if _, err := canonicalPreviousEntry.Unmarshal(previousEntry); err != nil { + relayLogger.Errorf( + "the previous entry this relay signed is not a curve point; the "+ + "membership's terminal outcome is left unresolved: [%v]", + err, + ) + return + } + + reference, err := participation.BeaconRelayEntryReference( + relayRequestStartBlock, + groupPublicKey, + canonicalPreviousEntry.Marshal(), + relayEntry, + ) + if err != nil { + // An entry that cannot be named verifiably is worth less than no + // claim of a result at all: recording it would put an unverifiable + // digest where the audit expects a checkable one. The permit closes + // unresolved and the offline barrier blocks on it. + relayLogger.Errorf( + "cannot name the recovered relay entry canonically; the "+ + "membership's terminal outcome is left unresolved: [%v]", + err, + ) + return + } + + recordBeaconPermitTerminalOutcome( + relayLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: reference, + Contribution: relayTranscriptContribution(incorporated, memberships), + }, + ) +} + +// relayTranscriptContribution renders the local view of who produced a relay +// entry: the memberships whose authenticated signature shares were combined +// into it, and the ones among them this node operates in that group. +// +// The local half is read from every membership this node holds in the group +// rather than from the one this permit was issued for. A node running several +// seats supplies several shares, and a record naming only its own seat would +// leave the rest looking like shares some other party had to supply — which is +// exactly the reading a fleet subtracting its own memberships from the +// incorporated population draws to find the parties outside it. +// +// What the incorporated half attests is bounded by the wire, and the bound is +// worth stating where the record is written. A share is authenticated against the +// group public key share published for the membership that sent it and against +// the previous entry being signed, and nothing in it names the request. A relay +// entry is deterministic in its previous entry, so a request that follows one +// which timed out without a submission signs the same previous entry, and a share +// belonging to either is valid for both. A seat named here therefore held the +// private share behind its membership and put it into this recovered entry; that +// it was live at this request's start block is more than the share can say. +// Binding the two would mean carrying the request anchor in the message, which is +// the wire change this release cannot make. +func relayTranscriptContribution( + incorporated participation.MemberIndexes, + memberships []*registry.Membership, +) *participation.TranscriptContribution { + local := make(participation.MemberIndexes, 0, len(memberships)) + for _, membership := range memberships { + memberID := membership.Signer.MemberID() + if slices.Contains(incorporated, memberID) && + !slices.Contains(local, memberID) { + local = append(local, memberID) + } + } + slices.Sort(local) + + return &participation.TranscriptContribution{ + IncorporatedMembers: incorporated, + LocalMembers: local, + } +} + // ForwardSignatureShares enables the ability to forward signature shares // messages to other nodes even if this node is not a part of the group which -// signs the relay entry. -func (n *node) ForwardSignatureShares(groupPublicKeyBytes []byte) { +// signs the relay entry. The forwarding runs under a participation permit +// anchored at the relay request block, so quiescence and clock failure close +// the relay; the permit's mode is telemetry only because forwarding does not +// reinterpret payloads. +func (n *node) ForwardSignatureShares( + groupPublicKeyBytes []byte, + relayRequestBlockNumber uint64, +) { name, err := channelNameForPublicKeyBytes(groupPublicKeyBytes) if err != nil { logger.Warnf("could not forward signature shares: [%v]", err) return } - n.netProvider.BroadcastChannelForwarderFor(name) + if n.participationGate == nil { + logger.Warnf( + "no participation gate; not forwarding signature shares", + ) + return + } + + permit, err := n.participationGate.Begin( + participation.BeaconRelayForwarding, + relayRequestBlockNumber, + beaconRelayPermitIdentity(relayRequestBlockNumber, "forwarder"), + ) + if err != nil { + logger.Warnf( + "signature share forwarding refused by the participation "+ + "gate: [%v]", + err, + ) + return + } + + forwarder, err := n.netProvider.BroadcastChannelForwarderFor(name) + if err != nil { + permit.Close() + logger.Warnf( + "could not start the message forwarder for channel [%v]: [%v]", + name, + err, + ) + return + } + + go func() { + defer permit.Close() + + select { + case <-forwarder.Done(): + // TTL expiry, provider shutdown, or an explicit close ended the + // relay naturally. + case <-permit.Context().Done(): + // Clock failure or forced quiescence closes the relay. + forwarder.Close() + } + + recordBeaconPermitTerminalOutcome( + logger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceForwarderClosed, + }, + ) + }() } // ResumeSigningIfEligible enables a client to rejoin the ongoing signing process @@ -249,10 +1279,15 @@ func (n *node) ResumeSigningIfEligible() { "attempting to rejoin the current signing process [0x%x]", groupPublicKey, ) - n.GenerateRelayEntry( + // The on-chain liveness of the request was verified just above: + // IsEntryInProgress reported an entry in progress and the canonical + // anchor is the on-chain current request start block. That is the + // verification the gate's Resume path requires from its caller. + n.generateRelayEntry( previousEntry, groupPublicKey, entryStartBlock.Uint64(), + true, ) } } @@ -261,11 +1296,60 @@ func (n *node) ResumeSigningIfEligible() { // When a processing group which is supposed to deliver a relay entry does not // fulfill its work, then this node notifies the chain about it. In the case of // delivering a relay entry by a processing group, this node does nothing. +// +// The monitoring runs under a participation permit anchored at the relay +// request block: a timeout report is a penalty commit, so a legacy permit +// cannot report at or after the cutover block and no permit can report once +// quiescence begins. +// +// The previous entry the monitored request is signing over is carried along +// because it is how the beacon identifies that request among the ones made in +// the same block, and the monitor has to name the request exactly to read back +// whether its own report terminated it. func (n *node) MonitorRelayEntry( + relayRequestPreviousEntry []byte, relayRequestBlockNumber uint64, ) { logger.Infof("monitoring chain for a new relay entry") + if n.participationGate == nil { + // The monitor exists only to file the penalty report; without a gate + // there is no fenced way to do that. Fail closed. + logger.Errorf( + "no participation gate; refusing to monitor the relay entry", + ) + return + } + + permit, err := n.participationGate.Begin( + participation.BeaconTimeoutReport, + relayRequestBlockNumber, + beaconRelayPermitIdentity(relayRequestBlockNumber, "timeout-monitor"), + ) + if err != nil { + logger.Warnf( + "relay entry monitoring refused by the participation gate: [%v]", + err, + ) + return + } + // timeoutSettlement holds the beacon's own record of the request this + // monitor's report terminated, which is the only durable state this + // ceremony can create. The deferred recorder below reads its final value. + var timeoutSettlement *event.RelayEntryTimeoutSettlement + + // The terminal outcome is registered after the release so it runs first and + // reaches the permit while it is still open. + defer permit.Close() + defer func() { + recordRelayTimeoutTerminalOutcome( + logger, + permit, + relayRequestBlockNumber, + timeoutSettlement, + ) + }() + blockCounter, err := n.beaconChain.BlockCounter() if err != nil { logger.Errorf("failed to get block counter: [%v]", err) @@ -274,27 +1358,63 @@ func (n *node) MonitorRelayEntry( chainConfig := n.beaconChain.GetConfig() - timeoutWaiterChannel, err := blockCounter.BlockHeightWaiter( - relayRequestBlockNumber + chainConfig.RelayEntryTimeout, + // The block the group runs out of time at is derived before anything is + // watched, because the derivation is what authorizes the report: a timeout + // block that overflowed would name a block already passed and turn the + // waiter below into an immediate false penalty. + timeoutBlock, err := relayEntryTimeoutBlock( + relayRequestBlockNumber, + chainConfig.RelayEntryTimeout, ) + if err != nil { + logger.Errorf( + "refusing to monitor the relay entry of block [%v]; its timeout "+ + "block cannot be derived: [%v]", + relayRequestBlockNumber, + err, + ) + return + } + + timeoutWaiterChannel, err := blockCounter.BlockHeightWaiter(timeoutBlock) if err != nil { logger.Errorf("waiter for a relay entry timeout block failed: [%v]", err) return } - onEntrySubmittedChannel := make(chan *event.RelayEntrySubmitted) + // The buffer lets an in-flight event callback complete after this + // function returned on cancellation, instead of blocking forever. + onEntrySubmittedChannel := make(chan *event.RelayEntrySubmitted, 1) subscription := n.beaconChain.OnRelayEntrySubmitted( func(event *event.RelayEntrySubmitted) { onEntrySubmittedChannel <- event }, ) + // The subscription outlives the timeout branch. A report is only a penalty + // if the request left the beacon because of it, and a late entry delivered + // while the report is being resolved is what distinguishes the two, so the + // monitor keeps watching deliveries until it has stopped asking. + defer subscription.Unsubscribe() for { select { case blockNumber := <-timeoutWaiterChannel: - subscription.Unsubscribe() - close(onEntrySubmittedChannel) + // The last-moment penalty fence: a late legacy timeout at or + // after the cutover block, or any timeout during quiescence, + // must not create new penalty state. + if err := permit.CheckCommit( + "beacon_relay_timeout_report", + participation.PenaltyCommit, + ); err != nil { + logger.Warnf( + "relay entry timeout report refused by the release "+ + "gate: [%v]", + err, + ) + return + } + logger.Warnf( "relay entry was not submitted on time, reporting timeout at block [%v]", blockNumber, @@ -302,7 +1422,19 @@ func (n *node) MonitorRelayEntry( err = n.beaconChain.ReportRelayEntryTimeout() if err != nil { logger.Errorf("could not report a relay entry timeout: [%v]", err) + return } + // The call returning only means a provider took the transaction. + // The beacon decides whether a penalty exists, so the monitor + // asks it before claiming one. + timeoutSettlement = n.relayTimeoutReportSettled( + permit.Context(), + logger, + blockCounter, + relayRequestBlockNumber, + relayRequestPreviousEntry, + onEntrySubmittedChannel, + ) return case entry := <-onEntrySubmittedChannel: logger.Infof( @@ -310,6 +1442,13 @@ func (n *node) MonitorRelayEntry( entry.BlockNumber, ) return + case <-permit.Context().Done(): + logger.Warnf( + "relay entry monitoring canceled by the participation "+ + "gate: [%v]", + context.Cause(permit.Context()), + ) + return } } } @@ -325,6 +1464,20 @@ func (n *node) GenerateRelayEntry( previousEntry []byte, groupPublicKey []byte, startBlockHeight uint64, +) { + n.generateRelayEntry(previousEntry, groupPublicKey, startBlockHeight, false) +} + +// generateRelayEntry runs the relay entry signing for every local membership, +// each under its own participation permit anchored at the on-chain relay +// request start block. The resume flag selects the gate's restart path, which +// requires the caller to have verified on chain that the request is still +// live. +func (n *node) generateRelayEntry( + previousEntry []byte, + groupPublicKey []byte, + startBlockHeight uint64, + resume bool, ) { relayLogger := logger.With( zap.String("groupPublicKey", fmt.Sprintf("0x%x", groupPublicKey)), @@ -337,6 +1490,15 @@ func (n *node) GenerateRelayEntry( return } + if n.participationGate == nil { + // The gate is mandatory in production; signing without it would select + // a protocol mode implicitly. Fail closed. + relayLogger.Errorf( + "no participation gate; refusing to sign the relay entry", + ) + return + } + channel, err := n.netProvider.BroadcastChannelFor(memberships[0].ChannelName) if err != nil { relayLogger.Errorf("could not create broadcast channel: [%v]", err) @@ -375,12 +1537,43 @@ func (n *node) GenerateRelayEntry( chainConfig := n.beaconChain.GetConfig() + issuePermit := n.participationGate.Begin + if resume { + issuePermit = n.participationGate.Resume + } + for _, member := range memberships { - go func(member *registry.Membership) { + // One participation permit per local membership, anchored at the + // on-chain relay request start block: all share exchange and + // submission run under it and a refusal is a gate decision, not an + // ordinary signing failure. + permit, err := issuePermit( + participation.BeaconRelaySigning, + startBlockHeight, + beaconRelayPermitIdentity( + startBlockHeight, + fmt.Sprint(member.Signer.MemberID()), + member.Signer.MemberID(), + ), + ) + if err != nil { + relayLogger.Warnf( + "[member:%v] relay entry signing refused by the "+ + "participation gate: [%v]", + member.Signer.MemberID(), + err, + ) + continue + } + + go func(member *registry.Membership, permit participation.Permit) { + defer permit.Close() + n.protocolLatch.Lock() defer n.protocolLatch.Unlock() - err = entry.SignAndSubmit( + relayEntry, incorporated, err := entry.SignAndSubmit( + permit.Context(), relayLogger, blockCounter, channel, @@ -389,15 +1582,40 @@ func (n *node) GenerateRelayEntry( chainConfig.HonestThreshold, member.Signer, startBlockHeight, + permit, + ) + // The recovered entry, not the submission's fate, is this + // ceremony's node-owned terminal disposition: a member that never + // reached the honest threshold left no result behind. The + // memberships whose shares it was recovered from travel with it, so + // the record says who produced the entry rather than only that one + // exists. + recordRelayEntryTerminalOutcome( + relayLogger, + permit, + startBlockHeight, + member.Signer.GroupPublicKeyBytesCompressed(), + previousEntry, + relayEntry, + incorporated, + memberships, ) if err != nil { + if participation.IsGateRefusal(err) { + relayLogger.Warnf( + "relay entry signing canceled by the participation "+ + "gate: [%v]", + err, + ) + return + } relayLogger.Errorf( "error creating threshold signature: [%v]", err, ) return } - }(member) + }(member, permit) } } diff --git a/pkg/beacon/node_cutover_test.go b/pkg/beacon/node_cutover_test.go new file mode 100644 index 0000000000..b706d04ea1 --- /dev/null +++ b/pkg/beacon/node_cutover_test.go @@ -0,0 +1,1868 @@ +package beacon + +import ( + "context" + "crypto/rand" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "math" + "math/big" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + "github.com/keep-network/keep-common/pkg/persistence" + + "github.com/keep-network/keep-core/internal/testutils" + + beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" + "github.com/keep-network/keep-core/pkg/beacon/dkg" + "github.com/keep-network/keep-core/pkg/beacon/event" + "github.com/keep-network/keep-core/pkg/beacon/gjkr" + "github.com/keep-network/keep-core/pkg/beacon/registry" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/generator" + "github.com/keep-network/keep-core/pkg/net" + netLocal "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// cutoverFakePersistence is an accept-everything persistence handle: signer +// registration succeeds without touching disk. +type cutoverFakePersistence struct{} + +func (cutoverFakePersistence) Save([]byte, string, string) error { return nil } +func (cutoverFakePersistence) Snapshot([]byte, string, string) error { return nil } +func (cutoverFakePersistence) Archive(string) error { return nil } +func (cutoverFakePersistence) ReadAll() ( + <-chan persistence.DataDescriptor, + <-chan error, +) { + data := make(chan persistence.DataDescriptor) + errs := make(chan error) + close(data) + close(errs) + return data, errs +} + +func TestBeaconPermitIdentitiesAreCanonical(t *testing.T) { + seed := big.NewInt(123456789) + seedHash := sha256.Sum256(seed.Bytes()) + + dkgIdentity := beaconDKGPermitIdentity(seed, group.MemberIndex(17)) + if dkgIdentity.WorkID != hex.EncodeToString(seedHash[:]) { + t.Errorf("unexpected DKG work ID [%s]", dkgIdentity.WorkID) + } + if dkgIdentity.PermitID != "17" { + t.Errorf("unexpected DKG permit ID [%s]", dkgIdentity.PermitID) + } + + relayIdentity := beaconRelayPermitIdentity(1234, "17") + if relayIdentity.WorkID != "relay-request-1234" { + t.Errorf("unexpected relay work ID [%s]", relayIdentity.WorkID) + } + if relayIdentity.PermitID != "17" { + t.Errorf("unexpected relay permit ID [%s]", relayIdentity.PermitID) + } +} + +// cutoverRecordingPersistence records every saved file name so tests can +// assert exactly which namespace received signer material. +type cutoverRecordingPersistence struct { + cutoverFakePersistence + + mu sync.Mutex + saves []string + data map[string][]byte +} + +func (p *cutoverRecordingPersistence) Save( + data []byte, + directory string, + name string, +) error { + p.mu.Lock() + defer p.mu.Unlock() + path := directory + name + p.saves = append(p.saves, path) + if p.data == nil { + p.data = make(map[string][]byte) + } + p.data[path] = append([]byte(nil), data...) + return nil +} + +// savesContaining counts recorded saves whose path contains the given marker. +func (p *cutoverRecordingPersistence) savesContaining(marker string) int { + p.mu.Lock() + defer p.mu.Unlock() + count := 0 + for _, save := range p.saves { + if strings.Contains(save, marker) { + count++ + } + } + return count +} + +// savedDataContaining returns copies of the bytes saved to matching paths. +func (p *cutoverRecordingPersistence) savedDataContaining( + marker string, +) [][]byte { + p.mu.Lock() + defer p.mu.Unlock() + result := make([][]byte, 0) + for path, data := range p.data { + if strings.Contains(path, marker) { + result = append(result, append([]byte(nil), data...)) + } + } + return result +} + +// cutoverFailableBlockCounter delegates to the real local chain clock until a +// test induces a synchronous read failure; waiters keep working, matching a +// failing RPC current-height call. +type cutoverFailableBlockCounter struct { + chain.BlockCounter + + failing atomic.Bool +} + +func (c *cutoverFailableBlockCounter) CurrentBlock() (uint64, error) { + if c.failing.Load() { + return 0, fmt.Errorf("induced clock failure") + } + return c.BlockCounter.CurrentBlock() +} + +// cutoverTestChain delegates to the local chain but returns a fixed group +// selection, since the local chain does not implement SelectGroup. +type cutoverTestChain struct { + beaconchain.Interface + selectedOperators chain.Addresses +} + +func (c *cutoverTestChain) SelectGroup(*big.Int) (chain.Addresses, error) { + return c.selectedOperators, nil +} + +// cutoverGateMetrics is a race-safe recording sink for the participation gate. +type cutoverGateMetrics struct { + mu sync.Mutex + counters map[string]float64 + gauges map[string]float64 + updates chan struct{} +} + +func newCutoverGateMetrics() *cutoverGateMetrics { + return &cutoverGateMetrics{ + counters: make(map[string]float64), + gauges: make(map[string]float64), + updates: make(chan struct{}, 1), + } +} + +func (m *cutoverGateMetrics) IncrementCounter(name string, value float64) { + m.mu.Lock() + defer m.mu.Unlock() + m.counters[name] += value +} + +func (m *cutoverGateMetrics) SetGauge(name string, value float64) { + m.mu.Lock() + m.gauges[name] = value + m.mu.Unlock() + + select { + case m.updates <- struct{}{}: + default: + } +} + +func (m *cutoverGateMetrics) counter(name string) float64 { + m.mu.Lock() + defer m.mu.Unlock() + return m.counters[name] +} + +func (m *cutoverGateMetrics) gauge(name string) float64 { + m.mu.Lock() + defer m.mu.Unlock() + return m.gauges[name] +} + +// cutoverLocalChain is the local chain surface the harness needs: the full +// beacon chain interface plus the local result getter. +type cutoverLocalChain interface { + beaconchain.Interface + GetLastDKGResult() ( + *beaconchain.DKGResult, + map[beaconchain.GroupMemberIndex][]byte, + ) +} + +// cutoverNodeHarness bundles everything a node-level cutover test drives. +type cutoverNodeHarness struct { + node *node + localChain cutoverLocalChain + gate participation.Gate + gateMetrics *cutoverGateMetrics + gateClock *cutoverFailableBlockCounter + registryPersistence *cutoverRecordingPersistence + quarantinePersistence *cutoverRecordingPersistence + anchorBlock uint64 + groupSize int +} + +// newCutoverNodeHarness builds a beacon node over the local chain and network +// with a real participation gate. The cutover block is derived from the +// current chain height through cutoverBlockFor, after the chain reached at +// least block one so the anchor is never zero. A nil selection puts the +// node's operator in every seat; a custom selection lets externally driven +// members hold the remaining seats. +func newCutoverNodeHarness( + t *testing.T, + groupSize int, + honestThreshold int, + cutoverBlockFor func(currentBlock uint64) uint64, + selectionFor func(nodeAddress chain.Address) chain.Addresses, +) *cutoverNodeHarness { + t.Helper() + + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + + localChain := local_v1.ConnectWithKey( + groupSize, + honestThreshold, + operatorPrivateKey, + ) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + gateMetrics := newCutoverGateMetrics() + gateClock := &cutoverFailableBlockCounter{BlockCounter: blockCounter} + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: cutoverBlockFor(currentBlock)}, + gateClock, + gateMetrics, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + address, err := localChain.Signing().PublicKeyToAddress(operatorPublicKey) + if err != nil { + t.Fatal(err) + } + var selectedOperators chain.Addresses + if selectionFor != nil { + selectedOperators = selectionFor(address) + } else { + selectedOperators = make(chain.Addresses, groupSize) + for i := range selectedOperators { + selectedOperators[i] = address + } + } + if len(selectedOperators) != groupSize { + t.Fatalf( + "selection has [%d] seats for group size [%d]", + len(selectedOperators), + groupSize, + ) + } + + testChain := &cutoverTestChain{ + Interface: localChain, + selectedOperators: selectedOperators, + } + + registryPersistence := &cutoverRecordingPersistence{} + groupRegistry := registry.NewGroupRegistry( + logger, + testChain, + registryPersistence, + ) + + quarantinePersistence := &cutoverRecordingPersistence{} + signerQuarantine := registry.NewQuarantine( + context.Background(), + logger, + quarantinePersistence, + ) + + node := newNode( + testChain, + netLocal.ConnectWithKey(operatorPublicKey), + groupRegistry, + generator.StartScheduler(), + gate, + signerQuarantine, + gateMetrics, + ) + + return &cutoverNodeHarness{ + node: node, + localChain: localChain, + gate: gate, + gateMetrics: gateMetrics, + gateClock: gateClock, + registryPersistence: registryPersistence, + quarantinePersistence: quarantinePersistence, + anchorBlock: currentBlock, + groupSize: groupSize, + } +} + +func cutoverRandomSeed(t *testing.T) *big.Int { + t.Helper() + seed, err := rand.Int(rand.Reader, big.NewInt(math.MaxInt64)) + if err != nil { + t.Fatal(err) + } + return seed +} + +// runCeremonyToCompletion joins the DKG at the harness anchor and waits for +// the result publication and for every permit to be released. +func (h *cutoverNodeHarness) runCeremonyToCompletion( + t *testing.T, + seed *big.Int, +) { + t.Helper() + + resultChan := make(chan uint64, h.groupSize) + _ = h.localChain.OnDKGResultSubmitted( + func(submission *event.DKGResultSubmission) { + resultChan <- submission.BlockNumber + }, + ) + + h.node.JoinDKGIfEligible(seed, h.anchorBlock) + + select { + case <-resultChan: + case <-time.After(120 * time.Second): + t.Fatal("no DKG result published before the timeout") + } + + // Members close their permits after signer registration; wait for the + // gate to drain so the assertion sees final accounting. + h.waitForPermitRelease(t) +} + +// waitForPermitRelease waits until every member goroutine released its permit, +// so assertions see the final gate accounting and every quarantine or +// registration write has happened. +func (h *cutoverNodeHarness) waitForPermitRelease(t *testing.T) { + t.Helper() + + deadline := time.Now().Add(60 * time.Second) + for time.Now().Before(deadline) { + if h.gate.State().ActiveCeremonies == 0 { + return + } + time.Sleep(50 * time.Millisecond) + } + t.Fatal("permits were not released") +} + +// TestJoinDKGIfEligible_AnchorBelowCutoverRunsLegacyCeremony proves the node +// path end to end for a pre-cutover anchor: every locally controlled member +// receives a legacy permit from the shared gate and the homogeneous legacy +// ceremony completes, publishing a result. +func TestJoinDKGIfEligible_AnchorBelowCutoverRunsLegacyCeremony(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 5, + 3, + // The anchor stays far below the cutover block for the whole run. + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + harness.runCeremonyToCompletion(t, cutoverRandomSeed(t)) + + legacy := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeLegacyTotal, + ) + securityV2 := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeSecurityV2Total, + ) + if legacy != float64(harness.groupSize) { + t.Errorf( + "expected [%d] legacy permits, got [%f]", + harness.groupSize, + legacy, + ) + } + if securityV2 != 0 { + t.Errorf("expected no security-v2 permits, got [%f]", securityV2) + } +} + +// TestJoinDKGIfEligible_AnchorAtCutoverRunsSecurityV2Ceremony proves the exact +// boundary through the node path: an anchor equal to the cutover block pins +// security-v2 for every local member and the homogeneous ceremony completes. +func TestJoinDKGIfEligible_AnchorAtCutoverRunsSecurityV2Ceremony(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 5, + 3, + // The cutover block equals the anchor: anchor >= C selects + // security-v2 from the first cutover block onward. + func(currentBlock uint64) uint64 { return currentBlock }, + nil, + ) + + harness.runCeremonyToCompletion(t, cutoverRandomSeed(t)) + + legacy := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeLegacyTotal, + ) + securityV2 := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeSecurityV2Total, + ) + if securityV2 != float64(harness.groupSize) { + t.Errorf( + "expected [%d] security-v2 permits, got [%f]", + harness.groupSize, + securityV2, + ) + } + if legacy != 0 { + t.Errorf("expected no legacy permits, got [%f]", legacy) + } +} + +// TestJoinDKGIfEligible_QuiescedGateRefusesParticipation proves a quiescing +// gate refuses every local member synchronously: no member goroutine starts, +// no protocol traffic is sent, and no result can appear. +func TestJoinDKGIfEligible_QuiescedGateRefusesParticipation(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 5, + 3, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + harness.gate.Quiesce(fmt.Errorf("test shutdown")) + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + refusals := harness.gateMetrics.counter( + clientinfo.MetricParticipationRefusalsTotal, + ) + if refusals != float64(harness.groupSize) { + t.Errorf( + "expected [%d] gate refusals, got [%f]", + harness.groupSize, + refusals, + ) + } + ceremonyRefusals := harness.gateMetrics.counter( + clientinfo.ParticipationRefusalMetricName( + string(participation.BeaconDKG), + ), + ) + if ceremonyRefusals != float64(harness.groupSize) { + t.Errorf( + "expected [%d] beacon DKG refusals, got [%f]", + harness.groupSize, + ceremonyRefusals, + ) + } + if modes := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeLegacyTotal, + ) + harness.gateMetrics.counter( + clientinfo.MetricParticipationModeSecurityV2Total, + ); modes != 0 { + t.Errorf("expected no permits to be issued, got [%f]", modes) + } + if result, _ := harness.localChain.GetLastDKGResult(); result != nil { + t.Error("expected no DKG result with a quiesced gate") + } + if active := harness.gate.State().ActiveCeremonies; active != 0 { + t.Errorf("expected no active ceremonies, got [%d]", active) + } +} + +// TestJoinDKGIfEligible_NilGateFailsClosed proves a node without a gate +// refuses DKG participation instead of selecting a protocol mode implicitly. +func TestJoinDKGIfEligible_NilGateFailsClosed(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 5, + 3, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + harness.node.participationGate = nil + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + if result, _ := harness.localChain.GetLastDKGResult(); result != nil { + t.Error("expected no DKG result without a participation gate") + } +} + +// signingOverrideChain shares the local chain's state and clock but signs +// with a different operator key, so externally driven members hold their own +// group seats. +type signingOverrideChain struct { + beaconchain.Interface + signer chain.Signing +} + +func (c *signingOverrideChain) Signing() chain.Signing { return c.signer } + +// TestJoinDKGIfEligible_LegacyAnchorInteroperatesWithLegacyPeers is the +// discriminating proof that the node derives the ceremony bundle from the +// permit rather than pinning one mode: the node controls two seats through +// the gate at a pre-cutover anchor, while three seats run standalone members +// with an explicitly legacy bundle — the pre-cutover peer behavior. The +// honest threshold of four is reachable only if the node's members actually +// speak legacy; a node wrongly selecting security-v2 would split the group +// into cohorts of two and three, neither reaching the threshold, and no +// result could be published. +func TestJoinDKGIfEligible_LegacyAnchorInteroperatesWithLegacyPeers(t *testing.T) { + groupSize := 5 + honestThreshold := 4 + + externalPrivateKey, externalPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + externalSigner := local_v1.NewSigner(externalPrivateKey) + externalAddress, err := externalSigner.PublicKeyToAddress(externalPublicKey) + if err != nil { + t.Fatal(err) + } + + var selectedOperators chain.Addresses + harness := newCutoverNodeHarness( + t, + groupSize, + honestThreshold, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + func(nodeAddress chain.Address) chain.Addresses { + selectedOperators = chain.Addresses{ + nodeAddress, + nodeAddress, + externalAddress, + externalAddress, + externalAddress, + } + return selectedOperators + }, + ) + + seed := cutoverRandomSeed(t) + + externalChain := &signingOverrideChain{ + Interface: harness.localChain, + signer: externalSigner, + } + externalProvider := netLocal.ConnectWithKey(externalPublicKey) + externalChannel, err := externalProvider.BroadcastChannelFor( + fmt.Sprintf("%s-%s", ProtocolName, seed.Text(16)), + ) + if err != nil { + t.Fatal(err) + } + membershipValidator := group.NewMembershipValidator( + logger, + selectedOperators, + externalSigner, + ) + + // The standalone legacy peers run through their own always-legacy gate — + // the pre-cutover peer behavior — so their execution path carries a permit + // context and commit guard exactly like a production member. + externalBlockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + externalMetrics := newCutoverGateMetrics() + externalGate, err := participation.NewGate( + context.Background(), + participation.Schedule{}, + externalBlockCounter, + externalMetrics, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(externalGate.Close) + + externalErrors := make(chan error, 3) + var externalWait sync.WaitGroup + for _, memberIndex := range []group.MemberIndex{3, 4, 5} { + externalPermit, err := externalGate.Begin( + participation.BeaconDKG, + harness.anchorBlock, + ) + if err != nil { + t.Fatal(err) + } + + externalWait.Add(1) + go func(memberIndex group.MemberIndex) { + defer externalWait.Done() + defer externalPermit.Close() + _, _, err := dkg.ExecuteDKG( + externalPermit.Context(), + logger, + seed, + memberIndex, + harness.anchorBlock, + externalChain, + externalChannel, + membershipValidator, + selectedOperators, + compatibility.Legacy(), + externalPermit, + ) + if err != nil { + externalErrors <- fmt.Errorf( + "external member [%v]: %w", + memberIndex, + err, + ) + } + }(memberIndex) + } + + harness.runCeremonyToCompletion(t, seed) + + externalWait.Wait() + close(externalErrors) + for err := range externalErrors { + t.Errorf("external legacy member failed: [%v]", err) + } + + legacy := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeLegacyTotal, + ) + securityV2 := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeSecurityV2Total, + ) + if legacy != 2 { + t.Errorf("expected [2] legacy permits, got [%f]", legacy) + } + if securityV2 != 0 { + t.Errorf("expected no security-v2 permits, got [%f]", securityV2) + } +} + +// TestJoinDKGIfEligible_LegacyPermitCompletesAfterCutover proves a permit +// pinned from a pre-cutover anchor survives the cutover block and completes in +// legacy mode: the cutover falls in the middle of the ceremony, the process +// state transitions to open_security_v2, yet every member finishes with its +// legacy permit and the completion commits are accepted and counted as +// legacy completions after the cutover. +func TestJoinDKGIfEligible_LegacyPermitCompletesAfterCutover(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 5, + 3, + // The cutover block falls inside the ceremony: the DKG protocol takes + // tens of blocks beyond the GJKR phase alone, so block anchor+30 is + // crossed while the ceremony is still running. + func(currentBlock uint64) uint64 { return currentBlock + 30 }, + nil, + ) + + harness.runCeremonyToCompletion(t, cutoverRandomSeed(t)) + + legacy := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeLegacyTotal, + ) + securityV2 := harness.gateMetrics.counter( + clientinfo.MetricParticipationModeSecurityV2Total, + ) + if legacy != float64(harness.groupSize) { + t.Errorf( + "expected [%d] legacy permits, got [%f]", + harness.groupSize, + legacy, + ) + } + if securityV2 != 0 { + t.Errorf("expected no security-v2 permits, got [%f]", securityV2) + } + + // The ceremony must genuinely have completed at or after the cutover + // block: the process state already derives open_security_v2 while every + // signer activation still committed under its legacy permit. + if state := harness.gate.State(); state.State != participation.StateOpenSecurityV2 { + t.Errorf( + "expected the process state [%s] after the cutover, got [%s]", + participation.StateOpenSecurityV2, + state.State, + ) + } + completions := harness.gateMetrics.counter( + clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal, + ) + if completions < float64(harness.groupSize) { + t.Errorf( + "expected at least [%d] legacy completions after the cutover "+ + "(one signer activation per member), got [%f]", + harness.groupSize, + completions, + ) + } + + // Every member's accepted signer was activated normally. + if got := harness.registryPersistence.savesContaining("/membership_"); got != harness.groupSize { + t.Errorf( + "expected [%d] active membership saves, got [%d]", + harness.groupSize, + got, + ) + } + if got := harness.quarantinePersistence.savesContaining("/membership_"); got != 0 { + t.Errorf("expected no quarantined memberships, got [%d]", got) + } +} + +// TestJoinDKGIfEligible_ForcedShutdownAfterKeyGenerationQuarantinesSigner +// proves the forced-quiescence path after share generation: the gate is +// force-closed inside the result publication window, when the group key +// material already exists but no on-chain publication was observed. Every +// member's orphaned signer must be preserved in the quarantine namespace, no +// active membership may be written, and nothing may reach the chain. +func TestJoinDKGIfEligible_ForcedShutdownAfterKeyGenerationQuarantinesSigner(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + blockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + // The trigger fires inside the result publication signing window: after + // the last GJKR protocol block, before the earliest possible submission. + trigger, err := blockCounter.BlockHeightWaiter( + harness.anchorBlock + gjkr.ProtocolBlocks() + 2, + ) + if err != nil { + t.Fatal(err) + } + + shutdownDone := make(chan struct{}) + go func() { + defer close(shutdownDone) + <-trigger + harness.gate.Quiesce(fmt.Errorf("test shutdown")) + harness.gate.Close() + }() + + seed := cutoverRandomSeed(t) + harness.node.JoinDKGIfEligible(seed, harness.anchorBlock) + + <-shutdownDone + harness.waitForPermitRelease(t) + + if result, _ := harness.localChain.GetLastDKGResult(); result != nil { + t.Error("expected no DKG result after the forced shutdown") + } + if got := harness.quarantinePersistence.savesContaining("/membership_"); got != harness.groupSize { + t.Errorf( + "expected [%d] quarantined memberships, got [%d]", + harness.groupSize, + got, + ) + } + if got := harness.quarantinePersistence.savesContaining("/metadata_"); got != harness.groupSize { + t.Errorf( + "expected [%d] quarantine metadata records, got [%d]", + harness.groupSize, + got, + ) + } + expectedSeedHash := sha256.Sum256(seed.Bytes()) + for _, data := range harness.quarantinePersistence.savedDataContaining( + "/metadata_", + ) { + metadata := ®istry.QuarantinedSignerMetadata{} + if err := json.Unmarshal(data, metadata); err != nil { + t.Fatal(err) + } + if metadata.SeedHash != hex.EncodeToString(expectedSeedHash[:]) { + t.Errorf( + "unexpected quarantine seed hash [%s]", + metadata.SeedHash, + ) + } + } + if got := harness.registryPersistence.savesContaining("/membership_"); got != 0 { + t.Errorf( + "expected no active membership saves, got [%d]", + got, + ) + } + forcedAborts := harness.gateMetrics.counter( + clientinfo.MetricParticipationQuiesceForcedAbortsTotal, + ) + if forcedAborts != float64(harness.groupSize) { + t.Errorf( + "expected [%d] forced aborts, got [%f]", + harness.groupSize, + forcedAborts, + ) + } + if got := harness.gateMetrics.counter( + clientinfo. + MetricParticipationBeaconQuarantinePreservationFailuresTotal, + ); got != 0 { + t.Errorf( + "successful beacon quarantine incremented preservation failures: [%v]", + got, + ) + } + if got := harness.gateMetrics.gauge( + clientinfo.MetricParticipationBeaconQuarantineIncompleteOutputs, + ); got != 0 { + t.Errorf( + "successful beacon quarantine left [%v] incomplete outputs", + got, + ) + } +} + +// TestJoinDKGIfEligible_RefusedQuarantineMembershipIsNotAQuarantinedOutcome +// proves a share the quarantine namespace refused does not end its permit as +// quarantined, while the audit metadata naming the lost share is still +// written. +// +// The terminal outcome is what the offline audit and the rollback decision +// read as "this material is preserved and accounted for". A namespace that +// took the metadata but not the key material has preserved nothing, so +// claiming the outcome on the strength of the record alone would report key +// material that no namespace holds. The metadata is still attempted because it +// is the only thing that tells the audit a share was generated and lost. +func TestJoinDKGIfEligible_RefusedQuarantineMembershipIsNotAQuarantinedOutcome( + t *testing.T, +) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + // The same forced-quiescence interruption as above, over a namespace that + // will not accept the key material in any of the forms it is offered in. + refusing := &cutoverRefusingPersistence{ + refusedMarkers: []string{"/membership_", "/handoff_"}, + } + harness.node.signerQuarantine = registry.NewQuarantine( + endedProcessLifetime(), + logger, + refusing, + ) + + blockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + trigger, err := blockCounter.BlockHeightWaiter( + harness.anchorBlock + gjkr.ProtocolBlocks() + 2, + ) + if err != nil { + t.Fatal(err) + } + + shutdownDone := make(chan struct{}) + go func() { + defer close(shutdownDone) + <-trigger + harness.gate.Quiesce(fmt.Errorf("test shutdown")) + harness.gate.Close() + }() + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + <-shutdownDone + harness.waitForPermitRelease(t) + + if got := refusing.savesContaining("/membership_"); got != 0 { + t.Errorf("expected no membership to be accepted, got [%d]", got) + } + if got := refusing.savesContaining("/metadata_"); got != harness.groupSize { + t.Errorf( + "expected [%d] quarantine metadata records naming the lost "+ + "shares, got [%d]", + harness.groupSize, + got, + ) + } + + for _, record := range harness.gate.State().RecentTerminalOutcomes { + if record.Outcome == participation.TerminalOutcomeQuarantined { + t.Errorf( + "a share the namespace refused was reported as quarantined "+ + "[%v]", + record, + ) + } + } +} + +// TestJoinDKGIfEligible_RefusedQuarantineMetadataIsNotAQuarantinedOutcome +// proves the other incomplete pair: preserved key material with no audit record +// explaining it does not end its permit as quarantined either. +// +// The share is on disk here, so nothing was lost — but the terminal outcome is +// what the offline audit and the rollback decision read as "this material is +// preserved and accounted for", and the accounting is exactly what is missing. +// The mode, canonical anchor, ceremony, seat, and refused operation that would +// let the audit match the share against the chain all live in the metadata that +// the namespace would not take. Claiming the outcome on the strength of the key +// material alone would call the inventory complete while it holds a quarantine +// nothing explains. +func TestJoinDKGIfEligible_RefusedQuarantineMetadataIsNotAQuarantinedOutcome( + t *testing.T, +) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + // The same forced-quiescence interruption as above, over a namespace that + // takes the key material but no record explaining it. + refusing := &cutoverRefusingPersistence{ + refusedMarkers: []string{"/metadata_", "/handoff_"}, + } + harness.node.signerQuarantine = registry.NewQuarantine( + endedProcessLifetime(), + logger, + refusing, + ) + + blockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + trigger, err := blockCounter.BlockHeightWaiter( + harness.anchorBlock + gjkr.ProtocolBlocks() + 2, + ) + if err != nil { + t.Fatal(err) + } + + shutdownDone := make(chan struct{}) + go func() { + defer close(shutdownDone) + <-trigger + harness.gate.Quiesce(fmt.Errorf("test shutdown")) + harness.gate.Close() + }() + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + <-shutdownDone + harness.waitForPermitRelease(t) + + // The key material still has to be preserved. Refusing the outcome is about + // what a rollback can conclude, not a reason to stop writing the share. + if got := refusing.savesContaining( + "/membership_", + ); got != harness.groupSize { + t.Errorf( + "expected [%d] preserved memberships, got [%d]", + harness.groupSize, + got, + ) + } + if got := refusing.savesContaining("/metadata_"); got != 0 { + t.Errorf("expected no metadata to be accepted, got [%d]", got) + } + + for _, record := range harness.gate.State().RecentTerminalOutcomes { + if record.Outcome == participation.TerminalOutcomeQuarantined { + t.Errorf( + "a share with no audit record explaining it was reported as "+ + "quarantined [%v]", + record, + ) + } + } +} + +// TestJoinDKGIfEligible_RefusedMembershipStillPreservesTheOutputWhole proves a +// namespace that will not take the key material's own record does not cost this +// node its shares: the combined handoff carries every interrupted output, the +// permits end quarantined, and the node keeps taking work. +// +// This is the state a beacon node can least afford to lose. The group whose +// share was generated here may already have an accepted result, and a member +// that cannot produce its share leaves that group permanently short of it. The +// membership record is only where preservation prefers to put the material — +// when that name is refused, the output still has a name it can land under, and +// what lands carries the mode, anchor, ceremony, and refused operation the +// offline audit reconciles against the chain. +func TestJoinDKGIfEligible_RefusedMembershipStillPreservesTheOutputWhole( + t *testing.T, +) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + // Only the key material's own record is refused. The rest of the namespace + // works, which is what leaves the output somewhere to go. + refusing := &cutoverRefusingPersistence{ + refusedMarkers: []string{"/membership_"}, + } + harness.node.signerQuarantine = registry.NewQuarantine( + endedProcessLifetime(), + logger, + refusing, + ) + + blockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + trigger, err := blockCounter.BlockHeightWaiter( + harness.anchorBlock + gjkr.ProtocolBlocks() + 2, + ) + if err != nil { + t.Fatal(err) + } + + clockFailed := make(chan struct{}) + go func() { + defer close(clockFailed) + <-trigger + harness.gateClock.failing.Store(true) + }() + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + <-clockFailed + harness.waitForPermitRelease(t) + + if got := refusing.savesContaining("/membership_"); got != 0 { + t.Errorf("expected no membership to be accepted, got [%d]", got) + } + if got := refusing.savesContaining("/handoff_"); got != harness.groupSize { + t.Fatalf( + "expected [%d] outputs preserved whole, got [%d]", + harness.groupSize, + got, + ) + } + + // Every preserved output has to read back as a share with its explanation, + // which is what makes the permit resolvable rather than a finding. + for _, preserved := range refusing.savedDataContaining("/handoff_") { + handoff, err := registry.DecodeQuarantinedSignerHandoff(preserved) + if err != nil { + t.Fatalf("a preserved output cannot be read back: [%v]", err) + } + + membership := ®istry.Membership{} + if err := membership.Unmarshal(handoff.Membership); err != nil { + t.Fatalf( + "a preserved output carries key material that cannot be read "+ + "back: [%v]", + err, + ) + } + if handoff.Metadata.Ceremony != string(participation.BeaconDKG) { + t.Errorf( + "a preserved output names ceremony [%s]", + handoff.Metadata.Ceremony, + ) + } + if handoff.Metadata.CanonicalStartBlock != harness.anchorBlock { + t.Errorf( + "a preserved output names canonical anchor [%d], expected [%d]", + handoff.Metadata.CanonicalStartBlock, + harness.anchorBlock, + ) + } + } + + quarantined := 0 + for _, record := range harness.gate.State().RecentTerminalOutcomes { + if record.Outcome == participation.TerminalOutcomeQuarantined { + quarantined++ + } + } + if quarantined != harness.groupSize { + t.Errorf( + "expected [%d] outputs reported as quarantined, got [%d]", + harness.groupSize, + quarantined, + ) + } + + // Nothing was lost, so there is no inventory gap for the node to stop on. + testutils.AssertBoolsEqual( + t, + "the gate quiesced over an output the namespace holds whole", + false, + harness.gate.State().Quiescing, + ) +} + +// TestBlockOnIncompleteQuarantine_QuiescesOnEitherMissingHalf proves the node +// stops taking new ceremonies for either incomplete pair, not only for the lost +// share. +// +// A preserved share whose audit record did not land is on disk but unexplained, +// and a rollback reconciles namespaces against the chain using precisely the +// fields that record carries. Continuing to take work in that state builds more +// state on a host whose inventory is already known to be incomplete — the same +// reason the lost-share case blocks — so both halves have to reach the gate. +func TestBlockOnIncompleteQuarantine_QuiescesOnEitherMissingHalf(t *testing.T) { + tests := map[string]registry.QuarantineState{ + "the key material reached no namespace": { + MembershipPersisted: false, + MetadataPersisted: true, + }, + "the audit record reached no namespace": { + MembershipPersisted: true, + MetadataPersisted: false, + }, + } + + for testName, state := range tests { + t.Run(testName, func(t *testing.T) { + localChain := local_v1.Connect(5, 3) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: currentBlock + 100_000}, + blockCounter, + newCutoverGateMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + node := &node{participationGate: gate} + + testutils.AssertBoolsEqual( + t, + "the gate issues permits before the incomplete pair", + true, + gate.State().Allowed, + ) + + node.blockOnIncompleteQuarantine( + logger, + group.MemberIndex(1), + state, + fmt.Errorf("namespace refused the write"), + ) + + testutils.AssertBoolsEqual( + t, + "the gate quiesced on the incomplete pair", + true, + gate.State().Quiescing, + ) + + // A quiescing gate refuses before it looks at the anchor at all, + // so the refusal is read by sentinel rather than by the fact that + // one happened. The identity is a well-formed one, since a + // malformed one is refused before quiescence is ever consulted. + seedHash := sha256.Sum256([]byte("beacon-dkg-seed")) + if _, err := gate.Begin( + participation.BeaconDKG, + currentBlock, + participation.PermitIdentity{ + WorkID: hex.EncodeToString(seedHash[:]), + PermitID: "1", + OperatedMembers: participation.MemberIndexes{1}, + }, + ); !errors.Is(err, participation.ErrQuiescing) { + t.Errorf( + "a node holding an incomplete quarantine must refuse new "+ + "ceremonies, got [%v]", + err, + ) + } + }) + } +} + +// TestJoinDKGIfEligible_LostShareQuiescesTheNode proves a beacon share that +// reached no namespace stops this node from starting new ceremonies. +// +// The chain clock fails after key generation, so the gate cancels the permits +// and every generated share goes to the quarantine namespace — which refuses it. +// The share existed only in the goroutine that generated it, and nothing an +// operator or the offline audit can read accounts for it. A beacon group whose +// member cannot produce its share is permanently short of it, and the rollback +// audit reconciles namespaces against the chain — it cannot reconcile a share +// nobody wrote down. Taking on further work after that builds more state on a +// host whose inventory is already known to be incomplete. +// +// The clock failure is what interrupts the ceremony here rather than a +// quiescence, so the gate is not already quiescing when the loss happens: the +// transition proves the node's own response to the loss. +func TestJoinDKGIfEligible_LostShareQuiescesTheNode(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + refusing := &cutoverRefusingPersistence{ + refusedMarkers: []string{"/membership_", "/handoff_"}, + } + lifetime, cancelLifetime := context.WithCancel(context.Background()) + defer cancelLifetime() + harness.node.signerQuarantine = registry.NewQuarantine( + lifetime, + logger, + refusing, + ) + + blockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + // The trigger fires inside the result publication signing window: after the + // last GJKR protocol block, before the earliest possible submission. + trigger, err := blockCounter.BlockHeightWaiter( + harness.anchorBlock + gjkr.ProtocolBlocks() + 2, + ) + if err != nil { + t.Fatal(err) + } + + testutils.AssertBoolsEqual( + t, + "the gate is quiescing before the ceremony", + false, + harness.gate.State().Quiescing, + ) + + clockFailed := make(chan struct{}) + go func() { + defer close(clockFailed) + <-trigger + harness.gateClock.failing.Store(true) + }() + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + <-clockFailed + + // Wait for every preserving member to exhaust the write-grace rounds while + // the process lifetime remains live. The metric update channel avoids + // polling sleeps and the values are rechecked after every unrelated gate + // gauge update. + signalDeadline := time.NewTimer(10 * time.Second) + defer signalDeadline.Stop() + for { + failures := harness.gateMetrics.counter( + clientinfo. + MetricParticipationBeaconQuarantinePreservationFailuresTotal, + ) + incomplete := harness.gateMetrics.gauge( + clientinfo.MetricParticipationBeaconQuarantineIncompleteOutputs, + ) + if failures == float64(harness.groupSize) && + incomplete == float64(harness.groupSize) { + break + } + + select { + case <-harness.gateMetrics.updates: + case <-signalDeadline.C: + t.Fatalf( + "beacon quarantine signal while live = failures [%v], "+ + "incomplete [%v]; expected [%d] of each", + failures, + incomplete, + harness.groupSize, + ) + } + } + + if lifetime.Err() != nil { + t.Fatal("the process lifetime ended before the signal was inspected") + } + if active := harness.gate.State().ActiveCeremonies; active == 0 { + t.Error( + "the incomplete signal appeared only after all preserving permits " + + "had ended", + ) + } + + cancelLifetime() + harness.waitForPermitRelease(t) + + if got := refusing.savesContaining("/membership_"); got != 0 { + t.Errorf("expected no membership to be accepted, got [%d]", got) + } + if got := harness.gateMetrics.counter( + clientinfo. + MetricParticipationBeaconQuarantinePreservationFailuresTotal, + ); got != float64(harness.groupSize) { + t.Errorf( + "beacon quarantine-preservation failures = [%v], expected [%d]", + got, + harness.groupSize, + ) + } + if got := harness.gateMetrics.gauge( + clientinfo.MetricParticipationBeaconQuarantineIncompleteOutputs, + ); got != float64(harness.groupSize) { + t.Errorf( + "beacon incomplete quarantine outputs = [%v], expected [%d]", + got, + harness.groupSize, + ) + } + testutils.AssertBoolsEqual( + t, + "the gate quiesced on the lost shares", + true, + harness.gate.State().Quiescing, + ) + + // A quiescing gate refuses before it looks at the clock or the anchor, which + // is why the refusal has to be read by sentinel rather than by the fact that + // one happened. + if _, err := harness.gate.Begin( + participation.BeaconDKG, + harness.anchorBlock, + beaconDKGPermitIdentity(big.NewInt(7), group.MemberIndex(1)), + ); !errors.Is(err, participation.ErrQuiescing) { + t.Errorf( + "a node holding a lost share must refuse new ceremonies, got [%v]", + err, + ) + } +} + +// endedProcessLifetime is a process lifetime that has already ended. +// +// A quarantine store keeps trying to write the output it is holding for as long +// as its process lives, because the key material cannot be generated again. A +// test driving a namespace that never accepts the write has to supply that +// ending itself, and an already-ended lifetime is the shortest honest one: the +// store makes a single pass and reports what the namespace took. +func endedProcessLifetime() context.Context { + lifetime, end := context.WithCancel(context.Background()) + end() + + return lifetime +} + +// cutoverRefusingPersistence is a namespace that refuses one record name while +// recording its neighbours, as a disk namespace does when a single file cannot +// be written. +type cutoverRefusingPersistence struct { + cutoverRecordingPersistence + + // refusedMarkers name the records this namespace will not accept. They are + // a list because a preserved output is offered under more than one name: + // refusing the record pair and refusing the output are different + // namespaces, and only the second one costs the node a share. + refusedMarkers []string +} + +func (p *cutoverRefusingPersistence) Save( + data []byte, + directory string, + name string, +) error { + for _, marker := range p.refusedMarkers { + if strings.Contains(name, marker) { + return fmt.Errorf("cannot write [%s]", name) + } + } + + return p.cutoverRecordingPersistence.Save(data, directory, name) +} + +// TestJoinDKGIfEligible_ClockFailureAfterKeyGenerationQuarantinesSigner proves +// the chain-clock-failure path after share generation: the gate's synchronous +// clock reads start failing inside the result publication window. The commit +// fence and the clock supervisor fail closed, the permits are canceled with +// the clock sentinel, and every member's orphaned signer is preserved in the +// quarantine namespace without any on-chain submission. +func TestJoinDKGIfEligible_ClockFailureAfterKeyGenerationQuarantinesSigner(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + blockCounter, err := harness.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + // The trigger fires inside the result publication signing window: after + // the last GJKR protocol block, before the earliest possible submission. + trigger, err := blockCounter.BlockHeightWaiter( + harness.anchorBlock + gjkr.ProtocolBlocks() + 2, + ) + if err != nil { + t.Fatal(err) + } + + clockFailed := make(chan struct{}) + go func() { + defer close(clockFailed) + <-trigger + harness.gateClock.failing.Store(true) + }() + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + <-clockFailed + harness.waitForPermitRelease(t) + + if result, _ := harness.localChain.GetLastDKGResult(); result != nil { + t.Error("expected no DKG result after the clock failure") + } + if got := harness.quarantinePersistence.savesContaining("/membership_"); got != harness.groupSize { + t.Errorf( + "expected [%d] quarantined memberships, got [%d]", + harness.groupSize, + got, + ) + } + if got := harness.registryPersistence.savesContaining("/membership_"); got != 0 { + t.Errorf( + "expected no active membership saves, got [%d]", + got, + ) + } + clockAborts := harness.gateMetrics.counter( + clientinfo.MetricParticipationClockAbortsTotal, + ) + if clockAborts != float64(harness.groupSize) { + t.Errorf( + "expected [%d] clock aborts, got [%f]", + harness.groupSize, + clockAborts, + ) + } +} + +// TestJoinDKGIfEligible_GateCancellationDuringKeyGenerationAbortsCleanly +// proves cancellation reaches a running ceremony before key material exists: +// the gate is force-closed right after the members start, every member aborts +// as a gate decision — not an ordinary DKG failure — and nothing is +// quarantined, registered, or submitted. +func TestJoinDKGIfEligible_GateCancellationDuringKeyGenerationAbortsCleanly(t *testing.T) { + harness := newCutoverNodeHarness( + t, + 2, + 2, + func(currentBlock uint64) uint64 { return currentBlock + 100_000 }, + nil, + ) + + harness.node.JoinDKGIfEligible(cutoverRandomSeed(t), harness.anchorBlock) + + // The members are inside GJKR now: no group key material exists yet. + harness.gate.Quiesce(fmt.Errorf("test shutdown")) + harness.gate.Close() + + harness.waitForPermitRelease(t) + + if result, _ := harness.localChain.GetLastDKGResult(); result != nil { + t.Error("expected no DKG result after the cancellation") + } + if got := harness.quarantinePersistence.savesContaining("/membership_"); got != 0 { + t.Errorf( + "expected no quarantined memberships before key generation, "+ + "got [%d]", + got, + ) + } + if got := harness.registryPersistence.savesContaining("/membership_"); got != 0 { + t.Errorf("expected no active membership saves, got [%d]", got) + } + forcedAborts := harness.gateMetrics.counter( + clientinfo.MetricParticipationQuiesceForcedAbortsTotal, + ) + if forcedAborts != float64(harness.groupSize) { + t.Errorf( + "expected [%d] forced aborts, got [%f]", + harness.groupSize, + forcedAborts, + ) + } +} + +// TestMonitorRelayEntry_LegacyTimeoutReportSuppressedAfterCutover proves the +// timeout-report penalty fence: a monitor holding a legacy permit whose +// timeout block falls at or after the cutover block must not report the +// timeout. The technical grace for pre-cutover work must never create new +// penalty state after the cutover. +func TestMonitorRelayEntry_LegacyTimeoutReportSuppressedAfterCutover(t *testing.T) { + localChain := local_v1.Connect(5, 3) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + // The relay request anchors below the cutover block, but its timeout + // block — request plus the relay entry timeout — falls after it. + gateMetrics := newCutoverGateMetrics() + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: currentBlock + 5}, + blockCounter, + gateMetrics, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + node := &node{ + beaconChain: localChain, + participationGate: gate, + } + + monitorDone := make(chan struct{}) + go func() { + defer close(monitorDone) + node.MonitorRelayEntry(monitoredPreviousEntry, currentBlock) + }() + + timeoutBlock := currentBlock + + localChain.GetConfig().RelayEntryTimeout + if err := blockCounter.WaitForBlockHeight(timeoutBlock + 2); err != nil { + t.Fatal(err) + } + + select { + case <-monitorDone: + case <-time.After(30 * time.Second): + t.Fatal("the monitor did not return after the timeout block") + } + + if reports := localChain.GetRelayEntryTimeoutReports(); len(reports) != 0 { + t.Errorf( + "expected no timeout reports after the cutover, got [%v]", + reports, + ) + } + refusals := gateMetrics.counter( + clientinfo.MetricParticipationCommitRefusalsTotal, + ) + if refusals != 1 { + t.Errorf("expected [1] commit refusal, got [%f]", refusals) + } +} + +// TestMonitorRelayEntry_TimeoutReportedBelowCutover proves the monitor still +// files the timeout report while both the anchor and the timeout block stay +// below the cutover block: the penalty fence suppresses only post-cutover +// legacy penalties, not normal pre-cutover operation. +func TestMonitorRelayEntry_TimeoutReportedBelowCutover(t *testing.T) { + localChain := local_v1.Connect(5, 3) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + gateMetrics := newCutoverGateMetrics() + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: currentBlock + 100_000}, + blockCounter, + gateMetrics, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + node := &node{ + beaconChain: localChain, + participationGate: gate, + } + + monitorDone := make(chan struct{}) + go func() { + defer close(monitorDone) + node.MonitorRelayEntry(monitoredPreviousEntry, currentBlock) + }() + + timeoutBlock := currentBlock + + localChain.GetConfig().RelayEntryTimeout + if err := blockCounter.WaitForBlockHeight(timeoutBlock + 2); err != nil { + t.Fatal(err) + } + + select { + case <-monitorDone: + case <-time.After(30 * time.Second): + t.Fatal("the monitor did not return after the timeout block") + } + + if reports := localChain.GetRelayEntryTimeoutReports(); len(reports) != 1 { + t.Errorf( + "expected exactly one timeout report below the cutover, got [%v]", + reports, + ) + } +} + +// cutoverStubForwarder is a controllable net.Forwarder for forwarding +// lifecycle tests. +type cutoverStubForwarder struct { + closeOnce sync.Once + done chan struct{} +} + +func newCutoverStubForwarder() *cutoverStubForwarder { + return &cutoverStubForwarder{done: make(chan struct{})} +} + +func (f *cutoverStubForwarder) Close() { + f.closeOnce.Do(func() { close(f.done) }) +} + +func (f *cutoverStubForwarder) Done() <-chan struct{} { return f.done } + +func (f *cutoverStubForwarder) closed() bool { + select { + case <-f.done: + return true + default: + return false + } +} + +// cutoverForwardingProvider delegates everything to the wrapped provider but +// hands out a controllable forwarder handle. +type cutoverForwardingProvider struct { + net.Provider + + forwarder *cutoverStubForwarder +} + +func (p *cutoverForwardingProvider) BroadcastChannelForwarderFor(string) ( + net.Forwarder, + error, +) { + return p.forwarder, nil +} + +// TestForwardSignatureShares_GateCancellationClosesForwarder proves the +// forwarding permit owns the relay's lifecycle: the forwarding runs under a +// permit, and when the gate force-cancels it the forwarder handle is closed +// and the permit released. +func TestForwardSignatureShares_GateCancellationClosesForwarder(t *testing.T) { + localChain := local_v1.Connect(5, 3) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + gateMetrics := newCutoverGateMetrics() + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: currentBlock + 100_000}, + blockCounter, + gateMetrics, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + stubForwarder := newCutoverStubForwarder() + node := &node{ + beaconChain: localChain, + netProvider: &cutoverForwardingProvider{ + Provider: netLocal.Connect(), + forwarder: stubForwarder, + }, + participationGate: gate, + } + + groupPublicKeyBytes := new(bn256.G2).ScalarBaseMult(big.NewInt(1)).Marshal() + node.ForwardSignatureShares(groupPublicKeyBytes, currentBlock) + + if active := gate.State().ActiveCeremonies; active != 1 { + t.Fatalf("expected one active forwarding permit, got [%d]", active) + } + + gate.Quiesce(fmt.Errorf("test shutdown")) + gate.Close() + + deadline := time.Now().Add(10 * time.Second) + for time.Now().Before(deadline) { + if stubForwarder.closed() && gate.State().ActiveCeremonies == 0 { + break + } + time.Sleep(10 * time.Millisecond) + } + if !stubForwarder.closed() { + t.Error("expected the forwarder to be closed on gate cancellation") + } + if active := gate.State().ActiveCeremonies; active != 0 { + t.Errorf("expected the forwarding permit released, got [%d]", active) + } +} + +// TestForwardSignatureShares_ForwarderEndClosesPermit proves the reverse +// lifecycle direction: when the relay ends on its own — TTL expiry or +// provider shutdown — the forwarding permit is released without gate action. +func TestForwardSignatureShares_ForwarderEndClosesPermit(t *testing.T) { + localChain := local_v1.Connect(5, 3) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + gateMetrics := newCutoverGateMetrics() + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: currentBlock + 100_000}, + blockCounter, + gateMetrics, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + stubForwarder := newCutoverStubForwarder() + node := &node{ + beaconChain: localChain, + netProvider: &cutoverForwardingProvider{ + Provider: netLocal.Connect(), + forwarder: stubForwarder, + }, + participationGate: gate, + } + + groupPublicKeyBytes := new(bn256.G2).ScalarBaseMult(big.NewInt(1)).Marshal() + node.ForwardSignatureShares(groupPublicKeyBytes, currentBlock) + + if active := gate.State().ActiveCeremonies; active != 1 { + t.Fatalf("expected one active forwarding permit, got [%d]", active) + } + + // The relay ends naturally. + stubForwarder.Close() + + deadline := time.Now().Add(10 * time.Second) + for time.Now().Before(deadline) { + if gate.State().ActiveCeremonies == 0 { + break + } + time.Sleep(10 * time.Millisecond) + } + if active := gate.State().ActiveCeremonies; active != 0 { + t.Errorf("expected the forwarding permit released, got [%d]", active) + } +} diff --git a/pkg/beacon/node_test.go b/pkg/beacon/node_test.go index f204ee5042..49cd5d1a04 100644 --- a/pkg/beacon/node_test.go +++ b/pkg/beacon/node_test.go @@ -1,21 +1,59 @@ package beacon import ( + "context" "fmt" "math/big" "testing" + beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) var relayEntryTimeout = uint64(15) +// monitoredPreviousEntry stands in for the previous entry the relay request +// under monitoring is signing over. The monitor carries it so it can tell an +// accepted timeout report from a late delivery; these tests only need it to be +// a request the monitor can name. +var monitoredPreviousEntry = []byte("monitored-request-previous-entry") + +// newMonitorTestNode builds the minimal node a relay entry monitoring test +// needs: the local chain plus a real participation gate with the +// developer-only disabled schedule, in which timeout reports stay allowed. +func newMonitorTestNode( + t *testing.T, + localChain beaconchain.Interface, +) *node { + t.Helper() + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{}, + blockCounter, + newCutoverGateMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + return &node{ + beaconChain: localChain, + participationGate: gate, + } +} + func TestMonitorRelayEntryOnChain_EntrySubmitted(t *testing.T) { localChain := local_v1.Connect(5, 3) - node := &node{ - beaconChain: localChain, - } + node := newMonitorTestNode(t, localChain) blockCounter, err := node.beaconChain.BlockCounter() if err != nil { @@ -27,7 +65,7 @@ func TestMonitorRelayEntryOnChain_EntrySubmitted(t *testing.T) { t.Fatal(err) } - go node.MonitorRelayEntry(startBlockHeight) + go node.MonitorRelayEntry(monitoredPreviousEntry, startBlockHeight) // the window to get a relay entry is from currentBlock to (currentBlock+relayEntryTimeout) // we subtract arbitarly 5 blocks to be within this window. Ex. 0 + 15 - 5 @@ -65,9 +103,7 @@ func TestMonitorRelayEntryOnChain_EntrySubmitted(t *testing.T) { func TestMonitorRelayEntryOnChain_EntryNotSubmitted(t *testing.T) { localChain := local_v1.Connect(5, 3) - node := &node{ - beaconChain: localChain, - } + node := newMonitorTestNode(t, localChain) blockCounter, err := node.beaconChain.BlockCounter() if err != nil { @@ -79,7 +115,7 @@ func TestMonitorRelayEntryOnChain_EntryNotSubmitted(t *testing.T) { t.Fatal(err) } - go node.MonitorRelayEntry(startBlockHeight) + go node.MonitorRelayEntry(monitoredPreviousEntry, startBlockHeight) relayEntryTimeoutFromStart := startBlockHeight + relayEntryTimeout diff --git a/pkg/beacon/participation.go b/pkg/beacon/participation.go new file mode 100644 index 0000000000..e60d20fbd4 --- /dev/null +++ b/pkg/beacon/participation.go @@ -0,0 +1,69 @@ +package beacon + +import ( + "fmt" + "math" + + beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" + dkgResult "github.com/keep-network/keep-core/pkg/beacon/dkg/result" + "github.com/keep-network/keep-core/pkg/beacon/gjkr" +) + +// MaximumLegacyCompletionBlocks returns the maximum number of Ethereum blocks +// that any already-started random beacon protocol work may legitimately need +// to reach its natural completion: the larger of the full DKG duration — GJKR +// protocol states, pre-publication result signing, and the worst-case +// publication loop over all group members — and the on-chain relay entry +// timeout. +// +// The bound sizes cutover-rehearsal timing, local straggler-roster retention, +// graceful rollback quiescence, and alerts about unexpectedly long legacy +// overlap after the protocol cutover block. It is deliberately not an +// activation height and must never gate new work; each protocol's existing +// validity context remains the hard end of any in-flight grace behavior. +// +// The configuration is chain-supplied at runtime, so a nil config, a +// non-positive group size, and arithmetic overflow are rejected instead of +// silently producing a wrapped-around retention or quiescence deadline. +func MaximumLegacyCompletionBlocks(config *beaconchain.Config) (uint64, error) { + if config == nil { + return 0, fmt.Errorf( + "cannot derive the completion bound: beacon chain config is nil", + ) + } + if config.GroupSize <= 0 { + return 0, fmt.Errorf( + "cannot derive the completion bound: beacon group size [%d] "+ + "must be positive", + config.GroupSize, + ) + } + + groupSize := uint64(config.GroupSize) + if config.ResultPublicationBlockStep != 0 && + groupSize > math.MaxUint64/config.ResultPublicationBlockStep { + return 0, fmt.Errorf( + "cannot derive the completion bound: publication loop of group "+ + "size [%d] times publication block step [%d] overflows", + config.GroupSize, + config.ResultPublicationBlockStep, + ) + } + publicationBlocks := groupSize * config.ResultPublicationBlockStep + + fixedBlocks := gjkr.ProtocolBlocks() + dkgResult.PrePublicationBlocks() + if publicationBlocks > math.MaxUint64-fixedBlocks { + return 0, fmt.Errorf( + "cannot derive the completion bound: fixed DKG duration [%d] "+ + "plus publication loop [%d] blocks overflows", + fixedBlocks, + publicationBlocks, + ) + } + dkgBlocks := fixedBlocks + publicationBlocks + + if config.RelayEntryTimeout > dkgBlocks { + return config.RelayEntryTimeout, nil + } + return dkgBlocks, nil +} diff --git a/pkg/beacon/participation_outcome_test.go b/pkg/beacon/participation_outcome_test.go new file mode 100644 index 0000000000..9c593a2a72 --- /dev/null +++ b/pkg/beacon/participation_outcome_test.go @@ -0,0 +1,684 @@ +package beacon + +import ( + "bytes" + "context" + "encoding/hex" + "math/big" + "strconv" + "sync" + "testing" + + bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + + "github.com/keep-network/keep-core/pkg/altbn128" + "github.com/keep-network/keep-core/pkg/bls" + + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/beacon/dkg" + "github.com/keep-network/keep-core/pkg/beacon/event" + "github.com/keep-network/keep-core/pkg/beacon/registry" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// recordingPermit is a minimal participation.Permit that captures the terminal +// dispositions its ceremony owner authors, so a test can assert what reaches +// the rollback journal without running a gate. +type recordingPermit struct { + ctx context.Context + cancel context.CancelCauseFunc + ceremony participation.Ceremony + workID string + + mu sync.Mutex + outcomes []recordedOutcome +} + +type recordedOutcome struct { + outcome participation.TerminalOutcome + evidence participation.TerminalEvidence +} + +// testRelayRequestStartBlock is the relay request every relay permit in this +// file is issued for. Relay evidence names the request it answers and the gate +// holds it to the permit's own request, so the two have to agree on one block. +const testRelayRequestStartBlock = uint64(7_654_321) + +func newRecordingPermit(ceremony participation.Ceremony) *recordingPermit { + ctx, cancel := context.WithCancelCause(context.Background()) + + // Beacon relay work is the one ceremony whose work identity is parsed + // rather than only compared, so its permits carry the identity the node + // would really issue. + workID := "test-work" + if ceremony == participation.BeaconRelaySigning || + ceremony == participation.BeaconTimeoutReport { + workID = participation.BeaconRelayWorkID(testRelayRequestStartBlock) + } + + return &recordingPermit{ + ctx: ctx, + cancel: cancel, + ceremony: ceremony, + workID: workID, + } +} + +// newRecordingRelayPermit issues a relay-flavored permit for one named relay +// request, so a test can vary the request its evidence has to answer. +func newRecordingRelayPermit( + ceremony participation.Ceremony, + requestStartBlock uint64, +) *recordingPermit { + permit := newRecordingPermit(ceremony) + permit.workID = participation.BeaconRelayWorkID(requestStartBlock) + return permit +} + +func (rp *recordingPermit) Context() context.Context { return rp.ctx } + +func (rp *recordingPermit) Ceremony() participation.Ceremony { return rp.ceremony } + +func (rp *recordingPermit) CanonicalStartBlock() uint64 { return 1 } + +func (rp *recordingPermit) Mode() participation.ProtocolMode { + return participation.ModeSecurityV2 +} + +func (rp *recordingPermit) WorkID() string { return rp.workID } + +func (rp *recordingPermit) PermitID() string { return "1" } + +func (rp *recordingPermit) CheckCommit(string, participation.CommitClass) error { + return nil +} + +func (rp *recordingPermit) RecordTerminalOutcome( + outcome participation.TerminalOutcome, + evidence participation.TerminalEvidence, +) error { + rp.mu.Lock() + defer rp.mu.Unlock() + + rp.outcomes = append( + rp.outcomes, + recordedOutcome{outcome: outcome, evidence: evidence}, + ) + + return nil +} + +func (rp *recordingPermit) Close() { rp.cancel(participation.ErrPermitClosed) } + +func (rp *recordingPermit) recorded() []recordedOutcome { + rp.mu.Lock() + defer rp.mu.Unlock() + + return append([]recordedOutcome(nil), rp.outcomes...) +} + +// assertRecordedTerminalOutcome checks that the ceremony owner authored exactly +// one terminal disposition of the expected shape and that the live gate's own +// validator accepts it for that ceremony. An outcome the node writes but the +// gate rejects never reaches the rollback audit at all. +func assertRecordedTerminalOutcome( + t *testing.T, + permit *recordingPermit, + expectedOutcome participation.TerminalOutcome, + expectedKind participation.TerminalEvidenceKind, +) participation.TerminalEvidence { + t.Helper() + + recorded := permit.recorded() + if len(recorded) != 1 { + t.Fatalf( + "expected exactly one terminal outcome, got [%d]", + len(recorded), + ) + } + + if recorded[0].outcome != expectedOutcome { + t.Errorf( + "unexpected terminal outcome\nexpected: [%s]\nactual: [%s]", + expectedOutcome, + recorded[0].outcome, + ) + } + + if recorded[0].evidence.Kind != expectedKind { + t.Errorf( + "unexpected terminal evidence kind\nexpected: [%s]\nactual: [%s]", + expectedKind, + recorded[0].evidence.Kind, + ) + } + + if err := participation.ValidateTerminalOutcome( + permit.Ceremony(), + permit.WorkID(), + recorded[0].outcome, + recorded[0].evidence, + ); err != nil { + t.Errorf( + "the gate rejects the node-authored outcome for ceremony [%s]: [%v]", + permit.Ceremony(), + err, + ) + } + + return recorded[0].evidence +} + +// localMemberships builds the group memberships a node operates, carrying only +// the member indexes the relay transcript is read against. +func localMemberships( + memberIndexes ...group.MemberIndex, +) []*registry.Membership { + memberships := make([]*registry.Membership, 0, len(memberIndexes)) + for _, memberIndex := range memberIndexes { + memberships = append(memberships, ®istry.Membership{ + Signer: dkg.NewThresholdSigner(memberIndex, nil, nil, nil, nil), + }) + } + + return memberships +} + +// TestBeaconDKGTranscriptContribution covers the transcript a completed beacon +// DKG publishes: the members the key material was generated with, and this +// node's own seat among them. A completion alone cannot carry that — every +// member of a finished DKG writes the same word and names the same group key +// whatever population produced it — so the group key this record names would +// otherwise be indistinguishable from a share one party generated and persisted +// on its own. +func TestBeaconDKGTranscriptContribution(t *testing.T) { + groupPublicKey := hex.EncodeToString( + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(big.NewInt(42)), + }.Compress(), + ) + + t.Run("operating member", func(t *testing.T) { + signer := dkg.NewThresholdSigner(3, nil, nil, nil, nil) + operating := participation.MemberIndexes{1, 3, 4} + + contribution := beaconDKGTranscriptContribution(signer, operating) + + expected := &participation.TranscriptContribution{ + IncorporatedMembers: operating, + LocalMembers: participation.MemberIndexes{3}, + } + if !contribution.Equal(expected) { + t.Fatalf( + "unexpected transcript contribution\n"+ + "expected: [%+v]\nactual: [%+v]", + expected, + contribution, + ) + } + + if err := participation.ValidateTerminalOutcome( + participation.BeaconDKG, + "test-work", + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedBeaconSigner, + Reference: groupPublicKey, + MembershipIndex: signer.MemberID(), + Contribution: contribution, + }, + ); err != nil { + t.Errorf( + "the gate rejects the node-authored beacon DKG outcome: [%v]", + err, + ) + } + }) + + // A signer persisted from a ceremony whose accepted result excluded it is + // incoherent rather than something to write down, so the seat is not claimed + // and the record fails closed at the gate: the permit is left unresolved and + // the offline barrier blocks on it, which is the safe reading of key material + // no transcript accounts for. + t.Run("member outside the operating set", func(t *testing.T) { + signer := dkg.NewThresholdSigner(5, nil, nil, nil, nil) + operating := participation.MemberIndexes{1, 3, 4} + + contribution := beaconDKGTranscriptContribution(signer, operating) + + if len(contribution.LocalMembers) != 0 { + t.Errorf( + "expected no local membership, got [%v]", + contribution.LocalMembers, + ) + } + + if err := participation.ValidateTerminalOutcome( + participation.BeaconDKG, + "test-work", + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedBeaconSigner, + Reference: groupPublicKey, + MembershipIndex: signer.MemberID(), + Contribution: contribution, + }, + ); err == nil { + t.Error( + "the gate accepted a persisted membership the transcript " + + "does not account for", + ) + } + }) +} + +// TestRecordRelayEntryTerminalOutcome covers one relay signing membership's +// disposition. A relay entry is deterministic for a given previous entry, so +// the recovered entry is the ceremony's durable result whichever member +// published it, and it is named together with the group and previous entry +// that make it verifiable rather than merely asserted. +func TestRecordRelayEntryTerminalOutcome(t *testing.T) { + groupSecret := big.NewInt(42) + + groupPublicKey := altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(groupSecret), + }.Compress() + previousEntryPoint := altbn128.G1HashToPoint([]byte("previous-entry")) + previousEntry := previousEntryPoint.Marshal() + relayEntry := bls.SignG1(groupSecret, previousEntryPoint).Marshal() + + t.Run("no threshold reached", func(t *testing.T) { + permit := newRecordingPermit(participation.BeaconRelaySigning) + + recordRelayEntryTerminalOutcome( + &testutils.MockLogger{}, + permit, + testRelayRequestStartBlock, + groupPublicKey, + previousEntry, + nil, + nil, + nil, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidenceNoThreshold, + ) + + if evidence.Reference != "" { + t.Errorf( + "expected no evidence reference, got [%s]", + evidence.Reference, + ) + } + }) + + // The memberships whose authenticated shares were combined into the entry, + // two of which this node does not operate. + incorporated := participation.MemberIndexes{1, 3, 4} + + t.Run("entry recovered", func(t *testing.T) { + permit := newRecordingPermit(participation.BeaconRelaySigning) + + recordRelayEntryTerminalOutcome( + &testutils.MockLogger{}, + permit, + testRelayRequestStartBlock, + groupPublicKey, + previousEntry, + relayEntry, + incorporated, + localMemberships(3, 5), + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ) + + expectedReference, err := participation.BeaconRelayEntryReference( + testRelayRequestStartBlock, + groupPublicKey, + previousEntry, + relayEntry, + ) + if err != nil { + t.Fatal(err) + } + if evidence.Reference != expectedReference { + t.Errorf( + "unexpected evidence reference\nexpected: [%s]\nactual: [%s]", + expectedReference, + evidence.Reference, + ) + } + + // The decimal request start block and three 64-byte components in + // hex, separated by three colons. + expectedLength := len(strconv.FormatUint( + testRelayRequestStartBlock, 10, + )) + 3*128 + 3 + if len(evidence.Reference) != expectedLength { + t.Errorf( + "expected a %d character reference, got [%d] characters", + expectedLength, + len(evidence.Reference), + ) + } + + // The transcript behind the entry, which the reference cannot carry: an + // entry is deterministic for a given previous entry, so its identity + // reads the same whether several parties supplied the shares it was + // recovered from or one party held every seat. Membership 5 is this + // node's and contributed no share to this entry, so it is not part of + // the population, and memberships 1 and 4 are seats some other node + // supplied — which is the only part of the transcript this record can + // attribute elsewhere. + expectedContribution := &participation.TranscriptContribution{ + IncorporatedMembers: incorporated, + LocalMembers: participation.MemberIndexes{3}, + } + if !evidence.Contribution.Equal(expectedContribution) { + t.Errorf( + "unexpected transcript contribution\n"+ + "expected: [%+v]\nactual: [%+v]", + expectedContribution, + evidence.Contribution, + ) + } + }) + + // A result the node cannot name verifiably must not reach the journal as + // an unverifiable one. Leaving the permit unresolved blocks the offline + // barrier, which is the safe reading of a result nobody can check. + for name, test := range map[string]struct { + previousEntry []byte + relayEntry []byte + }{ + "entry that is not a full-width point": { + previousEntry: previousEntry, + relayEntry: []byte{0x01, 0x02, 0x03, 0x04}, + }, + "previous entry that is not a curve point": { + previousEntry: bytes.Repeat([]byte{0xff}, 64), + relayEntry: relayEntry, + }, + } { + t.Run(name, func(t *testing.T) { + permit := newRecordingPermit(participation.BeaconRelaySigning) + + recordRelayEntryTerminalOutcome( + &testutils.MockLogger{}, + permit, + testRelayRequestStartBlock, + groupPublicKey, + test.previousEntry, + test.relayEntry, + participation.MemberIndexes{1, 3, 4}, + localMemberships(3), + ) + + if recorded := permit.recorded(); len(recorded) != 0 { + t.Errorf( + "expected an unnameable entry to record no outcome, "+ + "got [%+v]", + recorded, + ) + } + }) + } +} + +// TestRecordRelayEntryTerminalOutcome_ReferenceVerifies asserts the recorded +// reference is not merely well formed but actually checks out: the entry it +// names verifies as the group's threshold signature over the previous entry it +// names. That pairing is the whole reason the reference carries points rather +// than a digest, so a record that could not be verified would defeat it. +func TestRecordRelayEntryTerminalOutcome_ReferenceVerifies(t *testing.T) { + groupSecret := big.NewInt(42) + previousEntryPoint := altbn128.G1HashToPoint([]byte("previous-entry")) + + permit := newRecordingPermit(participation.BeaconRelaySigning) + + recordRelayEntryTerminalOutcome( + &testutils.MockLogger{}, + permit, + testRelayRequestStartBlock, + altbn128.G2Point{ + G2: new(bn256.G2).ScalarBaseMult(groupSecret), + }.Compress(), + previousEntryPoint.Marshal(), + bls.SignG1(groupSecret, previousEntryPoint).Marshal(), + participation.MemberIndexes{2}, + localMemberships(2), + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ) + + _, groupPublicKeyBytes, previousEntryBytes, entryBytes, err := + participation.ParseBeaconRelayEntryReference(evidence.Reference) + if err != nil { + t.Fatal(err) + } + + publicKey, err := altbn128.DecompressToG2(groupPublicKeyBytes) + if err != nil { + t.Fatal(err) + } + previousEntry := new(bn256.G1) + if _, err := previousEntry.Unmarshal(previousEntryBytes); err != nil { + t.Fatal(err) + } + entry := new(bn256.G1) + if _, err := entry.Unmarshal(entryBytes); err != nil { + t.Fatal(err) + } + + if !bls.VerifyG1(publicKey, previousEntry, entry) { + t.Error( + "the recorded relay entry does not verify as the named group's " + + "signature over the named previous entry", + ) + } +} + +// beaconTimeoutSettlement builds the beacon's own record of a terminated relay +// request, as the chain handle resolves it from canonical logs. +func beaconTimeoutSettlement( + requestBlock uint64, + requestID int64, + terminatedGroupID uint64, +) *event.RelayEntryTimeoutSettlement { + return &event.RelayEntryTimeoutSettlement{ + RequestID: big.NewInt(requestID), + TerminatedGroupID: terminatedGroupID, + RequestBlockNumber: requestBlock, + RequestPreviousEntry: []byte("previous-entry"), + BlockNumber: requestBlock + 64, + ContractAddress: "0xbeac0n", + } +} + +// TestRecordRelayTimeoutTerminalOutcome covers the relay entry monitor. The +// monitor exists only to file the penalty report, and only the settlement the +// beacon itself recorded is a durable result; a report the node merely handed +// to a provider created no penalty state it can account for. +func TestRecordRelayTimeoutTerminalOutcome(t *testing.T) { + const relayRequestBlock = uint64(100) + + t.Run("no settlement the beacon recorded", func(t *testing.T) { + permit := newRecordingRelayPermit( + participation.BeaconTimeoutReport, + relayRequestBlock, + ) + + recordRelayTimeoutTerminalOutcome( + &testutils.MockLogger{}, + permit, + relayRequestBlock, + nil, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidenceNoThreshold, + ) + + if evidence.Reference != "" { + t.Errorf( + "expected no evidence reference, got [%s]", + evidence.Reference, + ) + } + }) + + t.Run("a settlement the beacon recorded", func(t *testing.T) { + permit := newRecordingRelayPermit( + participation.BeaconTimeoutReport, + relayRequestBlock, + ) + + recordRelayTimeoutTerminalOutcome( + &testutils.MockLogger{}, + permit, + relayRequestBlock, + beaconTimeoutSettlement(relayRequestBlock, 7, 3), + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceEthereumTransaction, + ) + + // The identity names the authenticated log the audit joins to: the + // beacon's own request identifier and the group it terminated. + expected, err := participation.BeaconRelayTimeoutSettlementReference( + relayRequestBlock, + big.NewInt(7), + 3, + ) + if err != nil { + t.Fatal(err) + } + if evidence.Reference != expected { + t.Errorf( + "unexpected evidence reference\nexpected: [%s]\nactual: [%s]", + expected, + evidence.Reference, + ) + } + }) + + // A settlement whose identity cannot be rendered is one the offline audit + // could not join to any log, so it must not clear the rollback barrier. + t.Run("a settlement naming no request identifier", func(t *testing.T) { + permit := newRecordingRelayPermit( + participation.BeaconTimeoutReport, + relayRequestBlock, + ) + + settlement := beaconTimeoutSettlement(relayRequestBlock, 7, 3) + settlement.RequestID = nil + + recordRelayTimeoutTerminalOutcome( + &testutils.MockLogger{}, + permit, + relayRequestBlock, + settlement, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidenceNoThreshold, + ) + + if evidence.Reference != "" { + t.Errorf( + "expected no evidence reference, got [%s]", + evidence.Reference, + ) + } + }) + + t.Run("distinct requests produce distinct references", func(t *testing.T) { + references := make([]string, 0, 2) + for i, requestBlock := range []uint64{100, 200} { + permit := newRecordingRelayPermit( + participation.BeaconTimeoutReport, + requestBlock, + ) + + recordRelayTimeoutTerminalOutcome( + &testutils.MockLogger{}, + permit, + requestBlock, + beaconTimeoutSettlement(requestBlock, int64(i+1), 3), + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceEthereumTransaction, + ) + references = append(references, evidence.Reference) + } + + if references[0] == references[1] { + t.Errorf( + "two relay requests produced the same evidence reference [%s]", + references[0], + ) + } + }) + + // A settlement the beacon recorded against one request must not settle the + // permit a node holds for another. The penalty is real either way, so the + // component that keeps it on its own permit is the request start block the + // identity leads with. + t.Run("a settlement recorded for another request", func(t *testing.T) { + permit := newRecordingRelayPermit( + participation.BeaconTimeoutReport, + relayRequestBlock, + ) + + recordRelayTimeoutTerminalOutcome( + &testutils.MockLogger{}, + permit, + relayRequestBlock+1, + beaconTimeoutSettlement(relayRequestBlock+1, 7, 3), + ) + + recorded := permit.recorded() + if len(recorded) != 1 { + t.Fatalf("expected one terminal outcome, got [%d]", len(recorded)) + } + if err := participation.ValidateTerminalOutcome( + permit.Ceremony(), + permit.WorkID(), + recorded[0].outcome, + recorded[0].evidence, + ); err == nil { + t.Error( + "a timeout settlement naming another request settled this permit", + ) + } + }) +} diff --git a/pkg/beacon/participation_test.go b/pkg/beacon/participation_test.go new file mode 100644 index 0000000000..30914d705c --- /dev/null +++ b/pkg/beacon/participation_test.go @@ -0,0 +1,123 @@ +package beacon + +import ( + "math" + "testing" + + beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" + dkgResult "github.com/keep-network/keep-core/pkg/beacon/dkg/result" + "github.com/keep-network/keep-core/pkg/beacon/gjkr" + "github.com/keep-network/keep-core/pkg/chain/ethereum" +) + +// TestMaximumLegacyCompletionBlocks pins the derived in-flight completion +// bound against the configuration the production Ethereum beacon adapter +// actually supplies, so adapter drift fails this test, not only a changed +// literal. GetConfig reads no receiver state today; if that ever changes, +// this test fails loudly and the bound must be re-anchored deliberately. +func TestMaximumLegacyCompletionBlocks(t *testing.T) { + config := (ðereum.BeaconChain{}).GetConfig() + + maximum, err := MaximumLegacyCompletionBlocks(config) + if err != nil { + t.Fatalf("unexpected completion bound error: [%v]", err) + } + if maximum != 136 { + t.Errorf( + "expected maximum legacy completion bound [136], got [%d]", + maximum, + ) + } + + // A configuration with a dominant relay entry timeout must return it. + timeoutDominant := &beaconchain.Config{ + GroupSize: 64, + ResultPublicationBlockStep: 1, + RelayEntryTimeout: 500, + } + maximum, err = MaximumLegacyCompletionBlocks(timeoutDominant) + if err != nil { + t.Fatalf("unexpected completion bound error: [%v]", err) + } + if maximum != 500 { + t.Errorf( + "expected the relay entry timeout [500] to dominate, got [%d]", + maximum, + ) + } +} + +// TestMaximumLegacyCompletionBlocks_Validation proves the bound rejects a nil +// config, a non-positive group size, and arithmetic overflow instead of +// deriving a wrapped-around retention or quiescence deadline from them. +func TestMaximumLegacyCompletionBlocks_Validation(t *testing.T) { + if _, err := MaximumLegacyCompletionBlocks(nil); err == nil { + t.Error("expected a nil config rejection") + } + + invalid := map[string]*beaconchain.Config{ + "zero group size": { + GroupSize: 0, + ResultPublicationBlockStep: 1, + RelayEntryTimeout: 64, + }, + "negative group size": { + GroupSize: -1, + ResultPublicationBlockStep: 1, + RelayEntryTimeout: 64, + }, + "publication loop multiplication overflow": { + GroupSize: 2, + ResultPublicationBlockStep: math.MaxUint64/2 + 1, + RelayEntryTimeout: 64, + }, + "completion bound addition overflow": { + GroupSize: 1, + ResultPublicationBlockStep: math.MaxUint64 - 10, + RelayEntryTimeout: 64, + }, + } + for name, config := range invalid { + if _, err := MaximumLegacyCompletionBlocks(config); err == nil { + t.Errorf("expected a rejection for %s", name) + } + } +} + +// TestMaximumLegacyCompletionBlocksConstituents is a drift test: it fails when +// a GJKR or result-publication protocol constant changes without the +// completion bound — and everything derived from it, such as roster retention +// and rollback quiescence deadlines — being deliberately re-reviewed. +func TestMaximumLegacyCompletionBlocksConstituents(t *testing.T) { + if blocks := gjkr.ProtocolBlocks(); blocks != 66 { + t.Errorf( + "GJKR protocol duration changed: expected [66] blocks, got [%d]; "+ + "re-review the maximum legacy completion bound", + blocks, + ) + } + if blocks := dkgResult.PrePublicationBlocks(); blocks != 6 { + t.Errorf( + "result pre-publication duration changed: expected [6] blocks, "+ + "got [%d]; re-review the maximum legacy completion bound", + blocks, + ) + } + + // The production adapter inputs themselves are constituents: a changed + // adapter configuration must re-trip the bound review even if the formula + // is untouched. + config := (ðereum.BeaconChain{}).GetConfig() + if config.GroupSize != 64 || + config.ResultPublicationBlockStep != 1 || + config.RelayEntryTimeout != 64 { + t.Errorf( + "Ethereum beacon adapter configuration changed: got group size "+ + "[%d], publication step [%d], relay entry timeout [%d]; "+ + "re-review the maximum legacy completion bound", + config.GroupSize, + config.ResultPublicationBlockStep, + config.RelayEntryTimeout, + ) + } +} diff --git a/pkg/beacon/registry/groups.go b/pkg/beacon/registry/groups.go index b601e6d54b..0f738a2120 100644 --- a/pkg/beacon/registry/groups.go +++ b/pkg/beacon/registry/groups.go @@ -76,6 +76,33 @@ func (g *Groups) RegisterGroup( return nil } +// SaveAcceptedGroup persists the membership durably without activating it in +// the in-memory group cache. It preserves a signer whose result was accepted +// on chain but whose local activation the participation gate refused — during +// quiescence or after a clock failure: dropping an accepted share would +// permanently reduce its group, while activating it would start participation +// the gate no longer allows. The membership loads as active on the next +// process start, when the gate re-derives the process state. +func (g *Groups) SaveAcceptedGroup( + signer *dkg.ThresholdSigner, + channelName string, +) error { + g.mutex.Lock() + defer g.mutex.Unlock() + + membership := &Membership{ + Signer: signer, + ChannelName: channelName, + } + + err := g.storage.save(membership) + if err != nil { + return fmt.Errorf("could not persist membership to the storage: [%v]", err) + } + + return nil +} + // GetGroup gets a group by a groupPublicKey func (g *Groups) GetGroup(groupPublicKey []byte) []*Membership { g.mutex.Lock() diff --git a/pkg/beacon/registry/quarantine.go b/pkg/beacon/registry/quarantine.go new file mode 100644 index 0000000000..adbdd2b9a2 --- /dev/null +++ b/pkg/beacon/registry/quarantine.go @@ -0,0 +1,530 @@ +package registry + +import ( + "context" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/ipfs/go-log" + + "github.com/keep-network/keep-common/pkg/persistence" +) + +// QuarantineSchemaVersion versions the quarantined-signer metadata document +// for the offline state-audit tooling. +const QuarantineSchemaVersion uint32 = 1 + +// QuarantineHandoffSchemaVersion versions the combined handoff document for the +// offline state-audit tooling. +const QuarantineHandoffSchemaVersion uint32 = 1 + +// QuarantinedSignerHandoff carries one quarantined signer output whole: the key +// material and the audit record that explains it, in a single document written +// with a single save. +// +// The membership and metadata records preservation prefers are two independent +// writes, and a namespace that takes one but refuses the other leaves the output +// split. One of those halves cannot be split off harmlessly: a refused +// membership write leaves an audit record describing a share that reached no +// disk, and the share is the half no ceremony can generate again — for a beacon +// group whose result may already be accepted on chain, it is also a member the +// group's usable threshold permanently loses. This document is the form that +// cannot reach a reader in halves — a document a crash left half written fails +// the encrypted handle's authentication rather than decoding as the part that +// got through — and it is written under a name of its own, so a name the +// namespace refuses does not decide whether the output survives. +type QuarantinedSignerHandoff struct { + SchemaVersion uint32 `json:"schema_version"` + Metadata QuarantinedSignerMetadata `json:"metadata"` + // Membership is the marshaled membership, byte for byte what the membership + // record holds, so a reader decodes either form the same way. + Membership []byte `json:"membership"` +} + +// DecodeQuarantinedSignerHandoff reads a combined handoff record back into the +// halves it carries, for a later process and for the offline state audit. +// +// A document naming a schema this binary does not know is refused rather than +// read past. The handoff is the only account of an output preserved this way, +// so a reader that guessed at unknown fields would be inventing the evidence a +// rollback decision is made on. +func DecodeQuarantinedSignerHandoff( + recordBytes []byte, +) (*QuarantinedSignerHandoff, error) { + handoff := &QuarantinedSignerHandoff{} + if err := json.Unmarshal(recordBytes, handoff); err != nil { + return nil, fmt.Errorf( + "could not decode the quarantine handoff: [%v]", + err, + ) + } + + if handoff.SchemaVersion != QuarantineHandoffSchemaVersion { + return nil, fmt.Errorf( + "quarantine handoff has schema version [%d], expected [%d]", + handoff.SchemaVersion, + QuarantineHandoffSchemaVersion, + ) + } + + if len(handoff.Membership) == 0 { + return nil, fmt.Errorf( + "quarantine handoff carries no key material", + ) + } + + return handoff, nil +} + +// QuarantinedSignerMetadata describes one quarantined signer output for the +// offline state audit, without any private material: the key share itself +// stays only inside the encrypted membership record it accompanies. +type QuarantinedSignerMetadata struct { + SchemaVersion uint32 `json:"schema_version"` + ReleaseEpoch string `json:"release_epoch"` + ProtocolMode string `json:"protocol_mode"` + CutoverBlock uint64 `json:"cutover_block"` + CanonicalStartBlock uint64 `json:"canonical_start_block"` + Ceremony string `json:"ceremony"` + SeedHash string `json:"seed_hash"` + MemberIndex uint8 `json:"member_index"` + GroupPublicKey string `json:"group_public_key"` + FailedOperation string `json:"failed_operation"` + LastObservedBlock uint64 `json:"last_observed_block"` + PreservedAt time.Time `json:"preserved_at"` +} + +// Quarantine preserves signer outputs whose completion the participation gate +// interrupted — clock failure, forced quiescence, or a refused commit fence — +// before an accepted on-chain publication was observed. The handle MUST be +// rooted in a dedicated protected namespace that no release's active-group +// scan reads: quarantined records use the same membership encoding as active +// ones, so placing them beside active membership files would make a prior +// binary load them as active signers, which is not rollback-safe. Quarantined +// material is recovery evidence for the offline state audit; it is never +// activated by the running process. +type Quarantine struct { + logger log.StandardLogger + handle persistence.ProtectedHandle + + // lifetime bounds how long a preservation keeps trying to write the output + // it is holding. It is the process lifetime rather than the ceremony's: + // the ceremony context is normally already canceled by the very refusal + // that sent the share here, and until the process itself is going away the + // generated key material is still this node's to write down. It is held on + // the store because the choke points that preserve run several call levels + // below the startup that knows the process context. + lifetime context.Context + + // graceAttempts, retryDelay, and maxRetryDelay shape the retry, and wait + // pauses between rounds unless the lifetime ends first. They are fields so + // a test does not have to spend the real delays. + graceAttempts int + retryDelay time.Duration + maxRetryDelay time.Duration + wait func(context.Context, time.Duration) bool +} + +// quarantineGraceAttempts bounds how many rounds a preservation makes before +// the node is told it is holding key material the namespace does not have, and +// quarantineRetryDelay and quarantineMaxRetryDelay bound the wait between +// rounds. +// +// The grace budget is not a deadline. A refused write is often transient — a +// namespace being remounted, a disk an operator is draining — so the first +// rounds pass without disturbing the fleet; what follows is a node that stops +// taking new work while it keeps trying, not a node that gives the share up. +// The material on this path cannot be generated again, so the retry ends only +// with the process, and the backoff grows to keep a namespace that is down for +// an operator's whole repair from being hammered. +const ( + quarantineGraceAttempts = 3 + quarantineRetryDelay = 100 * time.Millisecond + quarantineMaxRetryDelay = 30 * time.Second +) + +// NewQuarantine creates a quarantine store over the given protected handle, +// preserving outputs for as long as the given process lifetime lasts. +func NewQuarantine( + lifetime context.Context, + logger log.StandardLogger, + handle persistence.ProtectedHandle, +) *Quarantine { + return &Quarantine{ + logger: logger, + handle: handle, + lifetime: lifetime, + graceAttempts: quarantineGraceAttempts, + retryDelay: quarantineRetryDelay, + maxRetryDelay: quarantineMaxRetryDelay, + wait: waitWithinLifetime, + } +} + +// waitWithinLifetime pauses between preservation rounds, reporting whether the +// process is still around to make another one. +func waitWithinLifetime(lifetime context.Context, delay time.Duration) bool { + if lifetime == nil { + time.Sleep(delay) + return true + } + + // Checked before the wait so a lifetime that has already ended stops the + // retry rather than racing the timer for it. + if lifetime.Err() != nil { + return false + } + + timer := time.NewTimer(delay) + defer timer.Stop() + + select { + case <-lifetime.Done(): + return false + case <-timer.C: + return true + } +} + +// QuarantineState reports which halves of a preserved output reached the +// namespace. A caller cannot infer this from the error alone, and the two +// halves mean different things: the membership is the key material a rollback +// has to account for, while the metadata is the audit record explaining it. +// Reporting the wrong one is how an operator log, a published count, and the +// offline audit come to disagree about the same directory. +type QuarantineState struct { + // MembershipPersisted reports whether the key material reached the + // namespace. + MembershipPersisted bool + // MetadataPersisted reports whether the audit record reached the + // namespace. + MetadataPersisted bool + // HandoffPersisted reports whether the namespace holds the combined record + // carrying both halves at once. It is written only when the pair could not + // be completed, and once it lands the output is whole whatever the pair is + // still missing. + HandoffPersisted bool +} + +// KeyMaterialPersisted reports whether the namespace holds the generated share +// in either of the forms preservation writes it in. This is what the count of +// preserved material follows: a share on disk is material a rollback has to +// account for however it got written down. +func (s QuarantineState) KeyMaterialPersisted() bool { + return s.MembershipPersisted || s.HandoffPersisted +} + +// Complete reports whether the namespace holds the whole output — the key +// material and the audit record explaining it. The pair says so when both its +// halves landed, and the combined record says so on its own, since it carries +// both. +func (s QuarantineState) Complete() bool { + return (s.MembershipPersisted && s.MetadataPersisted) || s.HandoffPersisted +} + +// Preserve durably saves the membership and its audit metadata under the +// quarantine namespace. It keeps ownership of the generated output until both +// halves are durable, retrying for as long as the process lives, and returns +// early only when the process is going away with a half still missing. +// +// Both records are attempted in every round and what actually landed is +// returned beside the error, because the two halves mean different things and a +// caller cannot infer either from the error alone. A membership without +// metadata is unexplained key material; metadata without a membership is a +// share that was lost. What must not happen is the node reporting a state the +// namespace contradicts. +// +// The membership is attempted first so that a process killed between the two +// writes leaves the key material behind rather than only the note describing +// it: an unexplained share is recoverable, a lost one is not. +// +// A round that cannot complete the pair falls back on the combined handoff +// record, which carries both halves in one write under a name of its own. It is +// what keeps a namespace refusing one particular record from costing the node a +// share it can never generate again, and once it lands the output is whole +// however little of the pair the namespace took. +// +// notifyIncomplete is called once, after graceAttempts rounds have left a half +// unwritten, with what the namespace holds so far. It exists so the node can +// stop taking new work while it is still holding an output no namespace fully +// has — not to end the attempt, which continues behind it until the pair is +// durable or the process ends. +func (q *Quarantine) Preserve( + membership *Membership, + metadata QuarantinedSignerMetadata, + notifyIncomplete func(QuarantineState, error), +) (QuarantineState, error) { + var state QuarantineState + + membershipBytes, err := membership.Marshal() + if err != nil { + return state, fmt.Errorf( + "could not marshal the quarantined membership: [%v]", + err, + ) + } + + metadata.SchemaVersion = QuarantineSchemaVersion + metadata.GroupPublicKey = hex.EncodeToString( + membership.Signer.GroupPublicKeyBytesCompressed(), + ) + metadata.MemberIndex = uint8(membership.Signer.MemberID()) + metadata.PreservedAt = time.Now().UTC() + + metadataBytes, err := json.Marshal(metadata) + if err != nil { + return state, fmt.Errorf( + "could not marshal the quarantine metadata: [%v]", + err, + ) + } + + handoffBytes, err := json.Marshal(QuarantinedSignerHandoff{ + SchemaVersion: QuarantineHandoffSchemaVersion, + Metadata: metadata, + Membership: membershipBytes, + }) + if err != nil { + return state, fmt.Errorf( + "could not marshal the quarantine handoff: [%v]", + err, + ) + } + + directory := metadata.GroupPublicKey + memberSuffix := fmt.Sprint(membership.Signer.MemberID()) + + // One line names the output and what the namespace holds of it, so the + // operator record and the namespace cannot drift apart. An incomplete pair + // is a finding the offline audit will raise, so it reads as an error rather + // than like an ordinary quarantine. + report := func(rounds int, complete bool) { + logQuarantine := q.logger.Warnf + if !complete { + logQuarantine = q.logger.Errorf + } + logQuarantine( + "quarantined a beacon signer output [group=0x%v] [member=%v] "+ + "[mode=%s] [canonicalStartBlock=%d] [failedOperation=%s] "+ + "[lastObservedBlock=%d] [keyMaterialPreserved=%v] "+ + "[auditMetadataPreserved=%v] [preservedAsOneRecord=%v] "+ + "[rounds=%d]", + metadata.GroupPublicKey, + membership.Signer.MemberID(), + metadata.ProtocolMode, + metadata.CanonicalStartBlock, + metadata.FailedOperation, + metadata.LastObservedBlock, + state.KeyMaterialPersisted(), + state.MetadataPersisted || state.HandoffPersisted, + state.HandoffPersisted, + rounds, + ) + } + + rounds, lastErr := q.persistOutput( + &state, + directory, + memberSuffix, + membershipBytes, + metadataBytes, + handoffBytes, + notifyIncomplete, + ) + if lastErr == nil { + report(rounds, true) + return state, nil + } + + report(rounds, false) + + return state, fmt.Errorf( + "could not preserve the quarantined beacon signer output in %d rounds "+ + "before the process ended [keyMaterialPreserved=%v] "+ + "[auditMetadataPreserved=%v]: %w", + rounds, + state.KeyMaterialPersisted(), + state.MetadataPersisted || state.HandoffPersisted, + lastErr, + ) +} + +// persistOutput writes whichever records of a preserved output the namespace +// has not taken yet, round after round, until it holds the whole output or the +// process ends. It reports how many rounds were spent and the last round's +// failure, which is nil exactly when the output is durable. +// +// The preferred form is the pair — a membership record beside its metadata — +// because it is the layout the active namespace uses and the one every reader +// already understands. A round that cannot complete the pair falls back on the +// combined handoff record, which carries both halves under a name of its own, so +// no namespace that refuses one particular record can leave key material with +// nowhere to go. A landed handoff ends the attempt, since there is nothing left +// the namespace does not hold. +// +// A record counts as landed on the namespace's word that it took the write, not +// on a reader's. The disk persistence behind this handle creates the file, +// writes the document, and syncs it, with no temporary record renamed into +// place, so a write a crash interrupts leaves a truncated document behind — and +// confirming each write by enumerating the namespace would put a share this node +// is still holding behind a directory listing that may never return, which is +// the more expensive way to lose it. What a torn write leaves is caught on the +// way out instead: the document fails the encrypted handle's authentication, so +// the offline audit reads it as an unreadable record and blocks on it rather +// than any reader taking it for a preserved output. +// +// The state is updated in place as each record lands so that a caller reading it +// after an interrupted preservation sees what the namespace actually has, and +// so a record that succeeded is never rewritten by a later round. +func (q *Quarantine) persistOutput( + state *QuarantineState, + directory string, + memberSuffix string, + membershipBytes []byte, + metadataBytes []byte, + handoffBytes []byte, + notifyIncomplete func(QuarantineState, error), +) (int, error) { + graceAttempts := q.graceAttempts + if graceAttempts < 1 { + graceAttempts = 1 + } + wait := q.wait + if wait == nil { + wait = waitWithinLifetime + } + delay := q.retryDelay + + notified := false + + // announcedLostMaterial remembers that the operator record says this share + // reached no namespace. It is what makes a later write worth a line of its + // own: until one is written, the standing account of this output is an error + // saying the material is only in memory, over a namespace that now holds it. + announcedLostMaterial := false + + var lastErr error + + // announceRecoveredMaterial takes back an operator record saying this share + // reached no namespace, once one of the records carrying it lands. + announceRecoveredMaterial := func(round int) { + if !announcedLostMaterial { + return + } + announcedLostMaterial = false + + q.logger.Warnf( + "the quarantine namespace took the beacon key material it had "+ + "been refusing [group=0x%s] [member=%s] [round=%d]; the share "+ + "this node reported as only in memory is on disk", + directory, + memberSuffix, + round, + ) + } + + for round := 1; ; round++ { + var roundErrs []error + + if !state.MembershipPersisted { + if err := q.handle.Save( + membershipBytes, + directory, + "/membership_"+memberSuffix, + ); err != nil { + roundErrs = append(roundErrs, fmt.Errorf( + "could not persist the quarantined membership: [%v]", + err, + )) + } else { + state.MembershipPersisted = true + + announceRecoveredMaterial(round) + } + } + + if !state.MetadataPersisted { + if err := q.handle.Save( + metadataBytes, + directory, + "/metadata_"+memberSuffix, + ); err != nil { + roundErrs = append(roundErrs, fmt.Errorf( + "could not persist the quarantine metadata: [%v]", + err, + )) + } else { + state.MetadataPersisted = true + } + } + + // The pair is what this round could not finish, so the output is + // offered whole under a name of its own. A namespace refusing one + // particular record — a leftover file nothing can overwrite, a name an + // operator's repair left behind — still has somewhere to put a share + // that cannot be generated a second time. + // + // When the half that did land was the membership, the namespace ends up + // holding the material twice. That is the cheaper mistake: both copies + // are the same encrypted bytes under the same handle, readers count the + // seat once, and the alternative is choosing which refusals are worth + // leaving an output incomplete for. + if len(roundErrs) > 0 && !state.HandoffPersisted { + if err := q.handle.Save( + handoffBytes, + directory, + "/handoff_"+memberSuffix, + ); err != nil { + roundErrs = append(roundErrs, fmt.Errorf( + "could not persist the quarantine handoff: [%v]", + err, + )) + } else { + state.HandoffPersisted = true + + q.logger.Warnf( + "preserved a beacon signer output as a single handoff "+ + "record [group=0x%s] [member=%s] [round=%d]; the "+ + "namespace would not take the record pair, and the key "+ + "material and its audit record are held together "+ + "instead", + directory, + memberSuffix, + round, + ) + + announceRecoveredMaterial(round) + } + } + + if state.Complete() { + return round, nil + } + + lastErr = errors.Join(roundErrs...) + + // The node is told once the grace rounds are spent, so a namespace that + // clears on its own does not take the node out of the fleet, and one + // that does not stops it from building further state it cannot account + // for. Preservation does not end here: the share is still in hand and + // the retry keeps running behind the notification. + if !notified && round >= graceAttempts { + notified = true + announcedLostMaterial = !state.KeyMaterialPersisted() + if notifyIncomplete != nil { + notifyIncomplete(*state, lastErr) + } + } + + if !wait(q.lifetime, delay) { + return round, lastErr + } + + if delay *= 2; delay > q.maxRetryDelay { + delay = q.maxRetryDelay + } + } +} diff --git a/pkg/beacon/registry/quarantine_test.go b/pkg/beacon/registry/quarantine_test.go new file mode 100644 index 0000000000..0f3e59dcb3 --- /dev/null +++ b/pkg/beacon/registry/quarantine_test.go @@ -0,0 +1,658 @@ +package registry + +import ( + "context" + "encoding/hex" + "fmt" + "math" + "reflect" + "strings" + "testing" + "time" + + "github.com/keep-network/keep-core/internal/testutils" + + "github.com/keep-network/keep-common/pkg/persistence" +) + +// TestQuarantine_Preserve_AttemptsBothRecordsAndReportsWhatLanded proves no +// record of a quarantined output is skipped because another failed, and that +// the caller is told which of them the namespace actually holds. +// +// The records mean different things — the membership is the key material a +// rollback has to account for, the metadata is the record explaining it, the +// handoff is both at once — and the error alone cannot say which are on disk. A +// caller that guesses is how the operator log and the offline audit come to +// describe the same directory differently. +// +// A name the namespace refuses does not decide the outcome: a round that cannot +// complete the pair offers the output whole under a name of its own, so only a +// namespace refusing everything leaves a share with nowhere to go. +func TestQuarantine_Preserve_AttemptsBothRecordsAndReportsWhatLanded( + t *testing.T, +) { + membership := &Membership{Signer: signer1, ChannelName: channelName1} + + tests := map[string]struct { + refusedNamePrefix string + expectedSaved []string + membershipPersisted bool + metadataPersisted bool + handoffPersisted bool + }{ + "both records land": { + refusedNamePrefix: "/nothing_is_refused", + expectedSaved: []string{"/membership_1", "/metadata_1"}, + membershipPersisted: true, + metadataPersisted: true, + }, + "the metadata is refused": { + refusedNamePrefix: "/metadata_", + expectedSaved: []string{"/membership_1", "/handoff_1"}, + membershipPersisted: true, + metadataPersisted: false, + handoffPersisted: true, + }, + "the membership is refused": { + refusedNamePrefix: "/membership_", + expectedSaved: []string{"/metadata_1", "/handoff_1"}, + membershipPersisted: false, + metadataPersisted: true, + handoffPersisted: true, + }, + "the namespace refuses every record": { + refusedNamePrefix: "/", + expectedSaved: nil, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + handle := &unwritableRecordHandle{ + refusedNamePrefixes: []string{test.refusedNamePrefix}, + } + // One round, then the process is taken to have ended: this asks + // what a single pass writes and reports, not how long the retry + // behind it lasts. + quarantine := newTestQuarantine(handle, 1) + + state, err := quarantine.Preserve( + membership, + QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + Ceremony: "beacon_dkg", + }, + nil, + ) + + expectedComplete := test.handoffPersisted || + (test.membershipPersisted && test.metadataPersisted) + if expectedComplete && err != nil { + t.Fatalf("expected no error, got [%v]", err) + } + if !expectedComplete && err == nil { + t.Fatal("expected a preservation error") + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + test.membershipPersisted, + state.MembershipPersisted, + ) + testutils.AssertBoolsEqual( + t, + "metadata persisted", + test.metadataPersisted, + state.MetadataPersisted, + ) + testutils.AssertBoolsEqual( + t, + "handoff persisted", + test.handoffPersisted, + state.HandoffPersisted, + ) + + if !reflect.DeepEqual(handle.savedNames, test.expectedSaved) { + t.Errorf( + "namespace holds %v, expected %v", + handle.savedNames, + test.expectedSaved, + ) + } + }) + } +} + +// flakyRecordHandle refuses the first refusals writes of the named record and +// accepts every write after that, counting the attempts made on it. It stands +// for a namespace that is unwritable for a while — a mount being restored, a +// disk an operator is draining — which is the condition a bounded write budget +// would report as permanently lost key material. +type flakyRecordHandle struct { + unwritableRecordHandle + + // namePrefixes name the records this namespace refuses while it is + // unwritable. The first of them is the one whose attempts are counted: + // preservation writes it once per round for as long as it has not landed, + // so its attempt number is the round number. + namePrefixes []string + refusals int + + attempts int +} + +func (h *flakyRecordHandle) Save( + data []byte, + directory string, + name string, +) error { + refused := false + for _, prefix := range h.namePrefixes { + if strings.HasPrefix(name, prefix) { + refused = true + break + } + } + if !refused { + return h.unwritableRecordHandle.Save(data, directory, name) + } + + if strings.HasPrefix(name, h.namePrefixes[0]) { + h.attempts++ + } + if h.attempts <= h.refusals { + return fmt.Errorf("cannot write [%s] yet", name) + } + + return h.unwritableRecordHandle.Save(data, directory, name) +} + +// newTestQuarantine builds a quarantine store that retries exactly the way the +// production one does but ends its process lifetime after the given number of +// rounds instead of spending the real waits between them. +// +// Production preservation stops only with the process, because the key material +// it is holding cannot be generated again and no elapsed time makes discarding +// it safe. A test driving a namespace that never accepts the write therefore has +// to supply the ending itself: the round count is where this process is taken to +// have gone away. +func newTestQuarantine( + handle persistence.ProtectedHandle, + roundsBeforeShutdown int, +) *Quarantine { + quarantine := NewQuarantine( + context.Background(), + &testutils.MockLogger{}, + handle, + ) + + rounds := 0 + quarantine.wait = func(context.Context, time.Duration) bool { + rounds++ + return rounds < roundsBeforeShutdown + } + + return quarantine +} + +// TestQuarantine_Preserve_KeepsTryingThroughAProlongedRefusal proves key +// material is not declared lost because a namespace stayed unwritable for a +// while: preservation keeps the share in hand across far more refusals than a +// passing fault produces, and still writes it when the namespace comes back. +// +// A beacon share is worse to lose than most: the group it belongs to may already +// have an accepted result, so a member that cannot produce its share permanently +// reduces that group's usable threshold. The conditions that refuse a write are +// the ones an operator repairs, and a fixed attempt budget turns the length of +// that repair into the difference between a preserved share and a lost one. +func TestQuarantine_Preserve_KeepsTryingThroughAProlongedRefusal(t *testing.T) { + const refusals = quarantineGraceAttempts * 8 + + handle := &flakyRecordHandle{ + unwritableRecordHandle: unwritableRecordHandle{ + refusedNamePrefixes: []string{"/nothing_is_refused"}, + }, + // The handoff is refused for as long as the membership, so what is being + // held across the repair is the key material itself and not a record + // that already put it somewhere. + namePrefixes: []string{"/membership_", "/handoff_"}, + refusals: refusals, + } + + // The notification the node acts on fires once and does not end the + // attempt, so it is counted rather than allowed to stand in for a result. + notifications := 0 + + quarantine := newTestQuarantine(handle, refusals*2) + + operatorLog := &warningCapture{} + quarantine.logger = operatorLog + + state, err := quarantine.Preserve( + &Membership{Signer: signer1, ChannelName: channelName1}, + QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + Ceremony: "beacon_dkg", + }, + func(QuarantineState, error) { notifications++ }, + ) + if err != nil { + t.Fatalf( + "expected the retried write to preserve the share, got [%v]", + err, + ) + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + true, + state.MembershipPersisted, + ) + testutils.AssertIntsEqual( + t, + "membership write attempts", + refusals+1, + handle.attempts, + ) + testutils.AssertIntsEqual(t, "block notifications", 1, notifications) + + if !reflect.DeepEqual( + handle.savedNames, + []string{"/metadata_1", "/membership_1"}, + ) { + t.Errorf( + "namespace holds %v, expected both halves", + handle.savedNames, + ) + } + + // The node was told, and the operator record says, that this share reached + // no namespace. Leaving that as the last word would have an operator repair + // a loss the namespace had already stopped being. + if logged := operatorLog.joined(); !strings.Contains( + logged, + "took the beacon key material it had been refusing", + ) || !strings.Contains( + logged, + fmt.Sprintf("[round=%d]", refusals+1), + ) { + t.Errorf( + "the operator record must say which round the namespace took the "+ + "material in, got [%s]", + logged, + ) + } +} + +// TestQuarantine_Preserve_GivesUpOnlyWhenTheProcessEnds proves the retry has no +// deadline of its own: a namespace that never accepts the write is attempted +// every round until the process itself goes away, and the error says that is +// what ended it. +// +// The node is told once, well before that, that it is holding key material the +// namespace does not have — but being told is not the same as being finished, +// and preservation carries on behind the notification. +func TestQuarantine_Preserve_GivesUpOnlyWhenTheProcessEnds(t *testing.T) { + const roundsBeforeShutdown = quarantineGraceAttempts * 9 + + handle := &flakyRecordHandle{ + unwritableRecordHandle: unwritableRecordHandle{ + refusedNamePrefixes: []string{"/nothing_is_refused"}, + }, + namePrefixes: []string{"/membership_", "/handoff_"}, + refusals: math.MaxInt32, + } + + notifications := 0 + + state, err := newTestQuarantine(handle, roundsBeforeShutdown).Preserve( + &Membership{Signer: signer1, ChannelName: channelName1}, + QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + Ceremony: "beacon_dkg", + }, + func(QuarantineState, error) { notifications++ }, + ) + if err == nil { + t.Fatal("expected a preservation error") + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + false, + state.MembershipPersisted, + ) + testutils.AssertBoolsEqual( + t, + "metadata persisted", + true, + state.MetadataPersisted, + ) + testutils.AssertBoolsEqual( + t, + "handoff persisted", + false, + state.HandoffPersisted, + ) + testutils.AssertIntsEqual( + t, + "membership write attempts", + roundsBeforeShutdown, + handle.attempts, + ) + testutils.AssertIntsEqual(t, "block notifications", 1, notifications) + + if want := fmt.Sprintf( + "in %d rounds before the process ended", + roundsBeforeShutdown, + ); !strings.Contains(err.Error(), want) { + t.Errorf( + "the error must say the process ending is what stopped the "+ + "retry, got [%v]", + err, + ) + } +} + +// TestQuarantine_Preserve_WritesOnlyTheHalfTheNamespaceLacks proves a round does +// not rewrite a half that already landed. The retry exists for the missing +// record, and rewriting the preserved one would keep touching key material the +// namespace has already accepted. +func TestQuarantine_Preserve_WritesOnlyTheHalfTheNamespaceLacks(t *testing.T) { + handle := &flakyRecordHandle{ + unwritableRecordHandle: unwritableRecordHandle{ + refusedNamePrefixes: []string{"/nothing_is_refused"}, + }, + // The handoff is refused for as long as the metadata, so what the retry + // is waiting on is the pair itself. + namePrefixes: []string{"/metadata_", "/handoff_"}, + refusals: quarantineGraceAttempts + 2, + } + + if _, err := newTestQuarantine(handle, 50).Preserve( + &Membership{Signer: signer1, ChannelName: channelName1}, + QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + Ceremony: "beacon_dkg", + }, + nil, + ); err != nil { + t.Fatalf( + "expected the retried write to preserve the share, got [%v]", + err, + ) + } + + if !reflect.DeepEqual( + handle.savedNames, + []string{"/membership_1", "/metadata_1"}, + ) { + t.Errorf( + "namespace holds %v, expected each half written exactly once", + handle.savedNames, + ) + } +} + +// TestQuarantine_Preserve_RecoversAWholeOutputTheNamespaceWouldNotPair proves +// what a later process recovers when the namespace refuses the key material's +// own record for good: the combined handoff carries the share and everything +// that explains it, and both read back. +// +// This is the state that used to cost a node a share. The membership record is +// where preservation prefers to put the material, the metadata beside it is only +// the explanation, and a namespace that took the second while refusing the first +// left a note about a share that reached no disk. A beacon share is the worse +// one to lose: its group may already have an accepted result, and a member that +// cannot produce its share permanently reduces that group's usable threshold. +// +// The handoff is one write carrying both, so the output survives under a name +// the namespace will take, and what comes next can read back the material, the +// group and seat it belongs to, and the mode, canonical anchor, ceremony, and +// refused operation the offline audit reconciles against the chain. +func TestQuarantine_Preserve_RecoversAWholeOutputTheNamespaceWouldNotPair( + t *testing.T, +) { + // The combined record is taken only well after the grace rounds are spent, + // so the node has already been told it is holding a share nothing has by the + // time the namespace comes back. + const handoffTakenAtRound = quarantineGraceAttempts * 2 + + handle := &latchedHandoffHandle{handoffTakenAtRound: handoffTakenAtRound} + + state, err := newTestQuarantine(handle, handoffTakenAtRound+1).Preserve( + &Membership{Signer: signer1, ChannelName: channelName1}, + QuarantinedSignerMetadata{ + ReleaseEpoch: "security_v2_cutover", + ProtocolMode: "security_v2", + CanonicalStartBlock: 4321, + Ceremony: "beacon_dkg", + FailedOperation: "beacon_dkg_result_publication", + }, + nil, + ) + if err != nil { + t.Fatalf("expected the output to be preserved whole, got [%v]", err) + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + false, + state.MembershipPersisted, + ) + testutils.AssertBoolsEqual( + t, + "handoff persisted", + true, + state.HandoffPersisted, + ) + testutils.AssertBoolsEqual( + t, + "the output is preserved whole", + true, + state.Complete(), + ) + + if !reflect.DeepEqual( + handle.savedNames, + []string{"/metadata_1", "/handoff_1"}, + ) { + t.Fatalf( + "namespace holds %v, expected the output preserved whole", + handle.savedNames, + ) + } + + // Counting a record is not the same as being able to use it. The next + // process reads the preserved output the way the offline audit does, and has + // to find the group and seat the material belongs to. + handoff, err := DecodeQuarantinedSignerHandoff( + handle.savedContent["/handoff_1"], + ) + if err != nil { + t.Fatalf("the preserved output cannot be read back: [%v]", err) + } + + preserved := &Membership{} + if err := preserved.Unmarshal(handoff.Membership); err != nil { + t.Fatalf("the preserved key material cannot be read back: [%v]", err) + } + + if got, expected := preserved.Signer.MemberID(), + signer1.MemberID(); got != expected { + t.Errorf( + "preserved material was generated for seat [%v], expected [%v]", + got, + expected, + ) + } + if got, expected := preserved.Signer.GroupPublicKeyBytesCompressed(), + signer1.GroupPublicKeyBytesCompressed(); !reflect.DeepEqual( + got, + expected, + ) { + t.Errorf( + "preserved material belongs to group [%x], expected [%x]", + got, + expected, + ) + } + + // The fields the offline audit matches against the chain travel with the + // material, so a share recovered this way is reconcilable rather than just + // countable. + metadata := handoff.Metadata + testutils.AssertStringsEqual( + t, + "protocol mode the preserved output was generated under", + "security_v2", + metadata.ProtocolMode, + ) + testutils.AssertUintsEqual( + t, + "canonical anchor the preserved output was generated under", + 4321, + metadata.CanonicalStartBlock, + ) + testutils.AssertStringsEqual( + t, + "ceremony the preserved output was generated in", + "beacon_dkg", + metadata.Ceremony, + ) + testutils.AssertStringsEqual( + t, + "operation that was refused", + "beacon_dkg_result_publication", + metadata.FailedOperation, + ) + testutils.AssertStringsEqual( + t, + "group the preserved output belongs to", + hex.EncodeToString(signer1.GroupPublicKeyBytesCompressed()), + metadata.GroupPublicKey, + ) +} + +// latchedHandoffHandle refuses the record carrying key material for good and +// takes the combined handoff only from the given round, as a namespace does when +// one particular file cannot be written and the rest of the directory is +// part-way through an operator's repair. +type latchedHandoffHandle struct { + unwritableRecordHandle + + // handoffTakenAtRound is the round from which the combined record is + // accepted. The membership is attempted once per round for as long as it has + // not landed, and it never lands here, so its attempt count is the round + // number. + handoffTakenAtRound int + + membershipAttempts int +} + +func (h *latchedHandoffHandle) Save( + data []byte, + directory string, + name string, +) error { + if strings.HasPrefix(name, "/membership_") { + h.membershipAttempts++ + return fmt.Errorf("cannot write [%s]", name) + } + + if strings.HasPrefix(name, "/handoff_") && + h.membershipAttempts < h.handoffTakenAtRound { + return fmt.Errorf("cannot write [%s] yet", name) + } + + return h.unwritableRecordHandle.Save(data, directory, name) +} + +// warningCapture records the warning lines a preservation emits, so a test can +// hold the operator's account of a preserved output to what the namespace +// actually holds. That account is the only description of a quarantine an +// operator reads at the time it matters. +type warningCapture struct { + testutils.MockLogger + + warnings []string +} + +func (c *warningCapture) Warnf(format string, args ...interface{}) { + c.warnings = append(c.warnings, fmt.Sprintf(format, args...)) +} + +func (c *warningCapture) joined() string { + return strings.Join(c.warnings, "\n") +} + +// unwritableRecordHandle is a protected namespace that refuses the record names +// it is given while writing their neighbours, as a disk namespace does when +// particular files cannot be written. +// +// The names are a list because a preserved output is offered under more than +// one of them: refusing the record pair and refusing the output are different +// namespaces, and only the second one costs the node a share. +type unwritableRecordHandle struct { + // refusedNamePrefixes name the records this namespace will not accept. + refusedNamePrefixes []string + + savedNames []string + + // savedContent keeps what each accepted record holds, so a test can read a + // preserved record back the way a later process would rather than only + // observe that a write happened. + savedContent map[string][]byte +} + +func (h *unwritableRecordHandle) refuses(name string) bool { + for _, prefix := range h.refusedNamePrefixes { + if strings.HasPrefix(name, prefix) { + return true + } + } + + return false +} + +func (h *unwritableRecordHandle) Save( + data []byte, + directory string, + name string, +) error { + if h.refuses(name) { + return fmt.Errorf("cannot write [%s]", name) + } + + h.savedNames = append(h.savedNames, name) + + if h.savedContent == nil { + h.savedContent = make(map[string][]byte) + } + h.savedContent[name] = append([]byte(nil), data...) + + return nil +} + +func (h *unwritableRecordHandle) Snapshot( + data []byte, + directory string, + name string, +) error { + panic("not implemented") +} + +func (h *unwritableRecordHandle) ReadAll() ( + <-chan persistence.DataDescriptor, + <-chan error, +) { + panic("not implemented") +} + +func (h *unwritableRecordHandle) Archive(directory string) error { + panic("not implemented") +} diff --git a/pkg/beacon/relay_timeout_settlement_test.go b/pkg/beacon/relay_timeout_settlement_test.go new file mode 100644 index 0000000000..1d5075de4b --- /dev/null +++ b/pkg/beacon/relay_timeout_settlement_test.go @@ -0,0 +1,739 @@ +package beacon + +import ( + "context" + "errors" + "math" + "math/big" + "sync" + "testing" + + "github.com/keep-network/keep-core/internal/testutils" + beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" + "github.com/keep-network/keep-core/pkg/beacon/event" + "github.com/keep-network/keep-core/pkg/chain" +) + +// reportedPreviousEntry is the previous entry the relay request under test is +// signing over. relayedPreviousEntry is what the beacon's previous entry +// becomes once that request is answered — a distinct value, because the relay +// advances only by a delivered entry. +var ( + reportedPreviousEntry = []byte("previous-entry-of-the-reported-request") + relayedPreviousEntry = []byte("previous-entry-after-a-delivered-entry") +) + +// errUnscriptedSettlement is what the scripted chain answers when a test did +// not script the settlement lookup. Reaching it means the reconciliation +// consulted state the test did not intend it to, which must not read as a +// deliberate outcome. +var errUnscriptedSettlement = errors.New("settlement lookup not scripted") + +// relayTimeoutLookup records one settlement lookup the reconciliation made. +type relayTimeoutLookup struct { + requestBlockNumber uint64 + previousEntry string +} + +// relayStateChain answers only the settlement lookup the timeout report +// reconciliation makes. Every other method of the beacon interface is left to +// the embedded nil interface, so a reconciliation that reached for the relay +// slot — the state whose readings cannot tell an accepted report from a chain +// view taken before the request existed — would panic instead of silently +// deciding on it. +type relayStateChain struct { + beaconchain.Interface + + mutex sync.Mutex + // reads counts settlement lookups, so a scripted chain can change its + // answer as the reconciliation polls. + reads int + // askedFor records what each lookup asked about, so a test can hold the + // reconciliation to the request its permit was issued for. + askedFor []relayTimeoutLookup + + // settlement answers one lookup. A nil settlement with a nil error is the + // chain saying it holds no record of the request being terminated, which + // covers a report not yet mined and a chain a reorg left the request out + // of alike. + settlement func(read int) (*event.RelayEntryTimeoutSettlement, error) + + // entries is the delivery channel the reconciliation reads. onRead lets a + // scripted chain push a relay entry onto it at a chosen point in the poll, + // which is how the race between a late delivery and the settlement lookup + // is driven deterministically. + entries chan *event.RelayEntrySubmitted + onRead func(read int, entries chan *event.RelayEntrySubmitted) +} + +func (c *relayStateChain) RelayEntryTimeoutSettlement( + requestBlockNumber uint64, + requestPreviousEntry []byte, +) (*event.RelayEntryTimeoutSettlement, error) { + c.mutex.Lock() + defer c.mutex.Unlock() + + c.reads++ + c.askedFor = append(c.askedFor, relayTimeoutLookup{ + requestBlockNumber: requestBlockNumber, + previousEntry: string(requestPreviousEntry), + }) + if c.onRead != nil { + c.onRead(c.reads, c.entries) + } + if c.settlement == nil { + return nil, errUnscriptedSettlement + } + return c.settlement(c.reads) +} + +// terminatedRequest builds the beacon's own record that the reported request +// was terminated by an accepted timeout report. +func terminatedRequest( + requestBlockNumber uint64, +) *event.RelayEntryTimeoutSettlement { + return &event.RelayEntryTimeoutSettlement{ + RequestID: big.NewInt(11), + TerminatedGroupID: 4, + RequestBlockNumber: requestBlockNumber, + RequestPreviousEntry: reportedPreviousEntry, + BlockNumber: requestBlockNumber + 64, + ContractAddress: "0xbeac0n", + } +} + +// noSettlement scripts a beacon that holds no record of the reported request +// being terminated, however often it is asked. +func noSettlement(int) (*event.RelayEntryTimeoutSettlement, error) { + return nil, nil +} + +// alwaysSettled scripts a beacon that answers every lookup with the same +// record. +func alwaysSettled( + settlement *event.RelayEntryTimeoutSettlement, +) func(int) (*event.RelayEntryTimeoutSettlement, error) { + return func(int) (*event.RelayEntryTimeoutSettlement, error) { + return settlement, nil + } +} + +// settledFrom scripts a beacon that records the termination only from the +// given lookup onwards, which is how a report that mines some blocks after the +// submitting call returned is driven deterministically. +func settledFrom( + read int, + settlement *event.RelayEntryTimeoutSettlement, +) func(int) (*event.RelayEntryTimeoutSettlement, error) { + return func(current int) (*event.RelayEntryTimeoutSettlement, error) { + if current < read { + return nil, nil + } + return settlement, nil + } +} + +// countingBlockCounter advances one block per wait, so a bounded resolution +// loop terminates deterministically without any real timing. +type countingBlockCounter struct { + chain.BlockCounter + + mutex sync.Mutex + height uint64 + waits int + blockFn func(height uint64) (uint64, error) +} + +func (b *countingBlockCounter) CurrentBlock() (uint64, error) { + b.mutex.Lock() + defer b.mutex.Unlock() + + if b.blockFn != nil { + return b.blockFn(b.height) + } + return b.height, nil +} + +func (b *countingBlockCounter) WaitForBlockHeight(blockNumber uint64) error { + b.mutex.Lock() + defer b.mutex.Unlock() + + b.waits++ + if blockNumber > b.height { + b.height = blockNumber + } + return nil +} + +// submittedEntries builds the relay entry delivery channel the reconciliation +// reads, pre-loaded with the deliveries the monitor's live subscription would +// have handed it. +func submittedEntries(blockNumbers ...uint64) chan *event.RelayEntrySubmitted { + entries := make(chan *event.RelayEntrySubmitted, len(blockNumbers)+1) + for _, blockNumber := range blockNumbers { + entries <- &event.RelayEntrySubmitted{BlockNumber: blockNumber} + } + return entries +} + +// TestRelayTimeoutReportSettled asserts a filed relay entry timeout report is +// claimed as a penalty only when the beacon holds its own record that the +// reported request was terminated. +// +// The submitting call returns once a provider accepts the transaction, which +// says nothing about whether it mined, reverted, or was dropped. Nor does a +// quiet subscription say anything: the event carrying a delivery arrives +// strictly after the state read that would have observed its effect can return, +// so an empty channel means "not yet", never "never". +// +// The reading that settles the report is the beacon's settlement record and +// nothing else. That record is resolved from canonical chain state on every +// lookup, so it can neither be manufactured by an ordering between this node's +// reads and its event deliveries, nor outlive a reorg that removed the state it +// rests on. Every other reading leaves the rollback barrier in place, which is +// what it exists for. +func TestRelayTimeoutReportSettled(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + // A settlement the beacon holds for a neighbouring request. It is a real + // penalty, just not this permit's. + otherRequest := terminatedRequest(relayRequestBlock + 1) + + // A settlement over a previous entry the reported request was not signing + // over, which is a different request made in the same block. + otherPreviousEntry := terminatedRequest(relayRequestBlock) + otherPreviousEntry.RequestPreviousEntry = relayedPreviousEntry + + // A settlement with nothing to join an authenticated log by. + anonymousRequest := terminatedRequest(relayRequestBlock) + anonymousRequest.RequestID = nil + + tests := map[string]struct { + chain *relayStateChain + entries chan *event.RelayEntrySubmitted + expectSettled bool + expectNoLookup bool + }{ + // The beacon terminated the reported request. Nothing but an accepted + // timeout report does that. + "the beacon recorded the reported request as terminated": { + chain: &relayStateChain{ + settlement: alwaysSettled(terminatedRequest(relayRequestBlock)), + }, + expectSettled: true, + }, + // The report mines a few blocks after the call returned, which is the + // ordinary case the bounded wait exists for. + "the beacon records the termination a few blocks later": { + chain: &relayStateChain{ + settlement: settledFrom( + 4, + terminatedRequest(relayRequestBlock), + ), + }, + expectSettled: true, + }, + // The consequential case: the provider took the transaction and the + // beacon never recorded a termination, so no penalty exists to claim. + // This is also every chain view a reorg removed the request from — the + // lookup is resolved afresh each time, so a view that no longer holds + // the request holds no record for this node to claim either. + "the beacon holds no record of the request being terminated": { + chain: &relayStateChain{settlement: noSettlement}, + }, + // The group delivered late. There is no penalty to claim, and the + // delivery ends the reconciliation before the beacon is asked again. + "an entry was delivered before the beacon was asked": { + chain: &relayStateChain{settlement: noSettlement}, + entries: submittedEntries(relayRequestBlock + 20), + expectNoLookup: true, + }, + // The same race one observation later: the delivery reaches this node + // while the beacon is being asked, and the answer it gives is still no + // record. + "an entry is delivered while the beacon is being asked": { + chain: &relayStateChain{ + settlement: noSettlement, + onRead: func( + read int, + entries chan *event.RelayEntrySubmitted, + ) { + if read == 2 { + entries <- &event.RelayEntrySubmitted{ + BlockNumber: relayRequestBlock + 20, + } + } + }, + }, + }, + "the beacon cannot be asked for its settlement record": { + chain: &relayStateChain{ + settlement: func(int) ( + *event.RelayEntryTimeoutSettlement, + error, + ) { + return nil, errors.New("not implemented") + }, + }, + }, + // A chain that keeps no relay request lifecycle at all answers every + // lookup with an error, which is not a penalty. + "the chain exposes no settlement records": { + chain: &relayStateChain{}, + }, + // The three ways a record can fail to answer the reported request. Each + // one is a settlement that exists, so refusing it is what keeps a real + // penalty on the permit that earned it. + "the beacon answers with another request's termination": { + chain: &relayStateChain{settlement: alwaysSettled(otherRequest)}, + }, + "the beacon answers with a termination over another previous entry": { + chain: &relayStateChain{ + settlement: alwaysSettled(otherPreviousEntry), + }, + }, + "the beacon answers with a termination naming no request identifier": { + chain: &relayStateChain{ + settlement: alwaysSettled(anonymousRequest), + }, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + entries := test.entries + if entries == nil { + entries = submittedEntries() + } + test.chain.entries = entries + + node := &node{beaconChain: test.chain} + + settlement := node.relayTimeoutReportSettled( + context.Background(), + &testutils.MockLogger{}, + &countingBlockCounter{height: 5_000}, + relayRequestBlock, + reportedPreviousEntry, + entries, + ) + + if settled := settlement != nil; settled != test.expectSettled { + t.Errorf( + "unexpected settlement\nexpected: [%t]\nactual: [%t]", + test.expectSettled, + settled, + ) + } + + if test.expectNoLookup { + if len(test.chain.askedFor) != 0 { + t.Errorf( + "the reconciliation asked the beacon [%d] times "+ + "after seeing a delivered entry", + len(test.chain.askedFor), + ) + } + return + } + + // Whatever the outcome, every lookup names the request the permit + // was issued for. A lookup that drifted onto another request could + // answer with a penalty this permit never earned. + for _, lookup := range test.chain.askedFor { + if lookup.requestBlockNumber != relayRequestBlock || + lookup.previousEntry != string(reportedPreviousEntry) { + t.Errorf( + "the reconciliation asked about the request of block "+ + "[%d] over previous entry [%s]", + lookup.requestBlockNumber, + lookup.previousEntry, + ) + } + } + }) + } +} + +// TestRelayTimeoutReportSettled_DecidesWithoutEventDelivery is the ordering +// guarantee stated as a test: the settled reading is taken from canonical chain +// state alone, so no delay on this node's relay entry callback can turn a +// delivery into a penalty, and no delay can withhold a penalty the beacon +// recorded. +// +// Both subtests run with the subscription channel empty for the entire +// reconciliation — the harshest form of "the callback has not arrived yet" — +// and assert the outcome the chain state alone dictates. The channel is then +// handed an event afterwards to confirm nothing in the window consumed or +// depended on it. +func TestRelayTimeoutReportSettled_DecidesWithoutEventDelivery(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + tests := map[string]struct { + chain *relayStateChain + expectSettled bool + }{ + // A late entry answered the request and the event announcing it reaches + // this node only after the reconciliation has returned. The beacon + // recorded no termination, so there is nothing to claim. + "a request answered by a delivery no event has announced yet": { + chain: &relayStateChain{settlement: noSettlement}, + }, + // The mirror image: an accepted report is claimed on the beacon's own + // record, with no event ever delivered to help. + "a terminated request with no event ever delivered": { + chain: &relayStateChain{ + settlement: alwaysSettled(terminatedRequest(relayRequestBlock)), + }, + expectSettled: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + entries := submittedEntries() + test.chain.entries = entries + + node := &node{beaconChain: test.chain} + + settlement := node.relayTimeoutReportSettled( + context.Background(), + &testutils.MockLogger{}, + &countingBlockCounter{height: 5_000}, + relayRequestBlock, + reportedPreviousEntry, + entries, + ) + + if settled := settlement != nil; settled != test.expectSettled { + t.Errorf( + "unexpected settlement\nexpected: [%t]\nactual: [%t]", + test.expectSettled, + settled, + ) + } + if test.chain.reads == 0 { + t.Error("the reconciliation never asked the beacon") + } + + // The callback the production subscription would have run, now + // that the reconciliation is over. Nothing should have been + // waiting on it and nothing should have consumed it. + entries <- &event.RelayEntrySubmitted{ + BlockNumber: relayRequestBlock + 20, + } + if len(entries) != 1 { + t.Errorf( + "the reconciliation consumed [%d] deliveries it was "+ + "never handed", + len(entries)-1, + ) + } + }) + } +} + +// TestRelayTimeoutReportSettled_WithoutPreviousEntryClaimsNothing asserts a +// monitor that does not know what previous entry its request was signing over +// refuses the claim outright. The beacon identifies the terminated request by +// that entry, so there is no request to ask about at all. +func TestRelayTimeoutReportSettled_WithoutPreviousEntryClaimsNothing(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + for _, previousEntry := range [][]byte{nil, {}} { + chain := &relayStateChain{ + settlement: alwaysSettled(terminatedRequest(relayRequestBlock)), + } + node := &node{beaconChain: chain} + + if node.relayTimeoutReportSettled( + context.Background(), + &testutils.MockLogger{}, + &countingBlockCounter{height: 5_000}, + relayRequestBlock, + previousEntry, + submittedEntries(), + ) != nil { + t.Errorf( + "a report on a request with previous entry [%v] was claimed "+ + "as settled", + previousEntry, + ) + } + if chain.reads != 0 { + t.Errorf( + "the reconciliation asked the beacon [%d] times without the "+ + "previous entry that identifies the request", + chain.reads, + ) + } + } +} + +// TestRelayTimeoutReportSettled_IsBounded asserts the reconciliation gives up +// rather than following a beacon that never records the termination. An +// unbounded wait would hold the monitor's goroutine and its permit open past +// the quiescence transition that has to capture it. +// +// It also pins that the decision is re-derived from the chain on every polled +// block rather than remembered between them. A reconciliation that carried an +// answer forward would be deciding on process-local history, which no reorg can +// take back. +func TestRelayTimeoutReportSettled_IsBounded(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + blockCounter := &countingBlockCounter{height: 5_000} + chain := &relayStateChain{settlement: noSettlement} + node := &node{beaconChain: chain} + + if node.relayTimeoutReportSettled( + context.Background(), + &testutils.MockLogger{}, + blockCounter, + relayRequestBlock, + reportedPreviousEntry, + submittedEntries(), + ) != nil { + t.Fatal("a request the beacon never terminated was claimed as settled") + } + + if blockCounter.waits > relayEntryTimeoutReportResolutionBlocks { + t.Errorf( + "the reconciliation waited [%d] blocks, past its bound of [%d]", + blockCounter.waits, + relayEntryTimeoutReportResolutionBlocks, + ) + } + if chain.reads != blockCounter.waits+1 { + t.Errorf( + "the reconciliation made [%d] settlement lookups over [%d] "+ + "polled blocks; the decision must be re-derived from the "+ + "chain on every one of them", + chain.reads, + blockCounter.waits+1, + ) + } +} + +// TestRelayTimeoutReportSettled_CanceledPermitClaimsNothing asserts a monitor +// whose permit the release gate closed mid-resolution does not claim a penalty +// it never saw recorded. +func TestRelayTimeoutReportSettled_CanceledPermitClaimsNothing(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + ctx, cancel := context.WithCancelCause(context.Background()) + + blockCounter := &countingBlockCounter{height: 5_000} + blockCounter.blockFn = func(height uint64) (uint64, error) { + cancel(errors.New("quiescence")) + return height, nil + } + + node := &node{beaconChain: &relayStateChain{settlement: noSettlement}} + + if node.relayTimeoutReportSettled( + ctx, + &testutils.MockLogger{}, + blockCounter, + relayRequestBlock, + reportedPreviousEntry, + submittedEntries(), + ) != nil { + t.Error("a canceled resolution claimed the report as settled") + } +} + +// TestRelayTimeoutReportSettled_UnrepresentableBoundClaimsNothing asserts a +// resolution bound the block range cannot represent is rejected rather than +// clamped, and that the rejection reaches the beacon not at all. +// +// Clamping to the top of the range would silently widen the window the monitor +// holds its permit open for; running the loop against a wrapped bound would +// close it before the beacon could answer while still reading as a bounded +// observation. Refusing outright reaches the honest disposition — no penalty +// claimed — without either pretence. +func TestRelayTimeoutReportSettled_UnrepresentableBoundClaimsNothing(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + chain := &relayStateChain{ + settlement: alwaysSettled(terminatedRequest(relayRequestBlock)), + } + node := &node{beaconChain: chain} + + if node.relayTimeoutReportSettled( + context.Background(), + &testutils.MockLogger{}, + &countingBlockCounter{height: math.MaxUint64 - 1}, + relayRequestBlock, + reportedPreviousEntry, + submittedEntries(), + ) != nil { + t.Error( + "a report whose resolution bound is not representable was " + + "claimed as settled", + ) + } + if chain.reads != 0 { + t.Errorf( + "the reconciliation asked the beacon [%d] times without a "+ + "representable bound", + chain.reads, + ) + } +} + +// TestRelayTimeoutReportSettled_ResolvesAtTopOfBlockRange asserts the highest +// height that still admits a representable bound resolves normally, so the +// rejection above is the overflow itself and not an off-by-one that gives up a +// block early. +func TestRelayTimeoutReportSettled_ResolvesAtTopOfBlockRange(t *testing.T) { + const relayRequestBlock = uint64(1_000) + + chain := &relayStateChain{ + settlement: settledFrom(2, terminatedRequest(relayRequestBlock)), + } + node := &node{beaconChain: chain} + + if node.relayTimeoutReportSettled( + context.Background(), + &testutils.MockLogger{}, + &countingBlockCounter{ + height: math.MaxUint64 - relayEntryTimeoutReportResolutionBlocks, + }, + relayRequestBlock, + reportedPreviousEntry, + submittedEntries(), + ) == nil { + t.Error( + "a report confirmed at the top of the block range was not " + + "claimed as settled", + ) + } + if chain.reads < 2 { + t.Errorf( + "the reconciliation gave up after [%d] reads at the top of the "+ + "block range", + chain.reads, + ) + } +} + +// TestRelayEntryTimeoutBlock asserts the block a group runs out of time at is +// rejected when the sum overflows instead of wrapping to a block already +// passed. A wrapped timeout block fires its waiter at once, so the monitor +// would file a penalty against a group that has had no chance to deliver. +func TestRelayEntryTimeoutBlock(t *testing.T) { + tests := map[string]struct { + relayRequestBlock uint64 + relayEntryTimeout uint64 + expectedBlock uint64 + expectedError bool + }{ + "an ordinary request": { + relayRequestBlock: 1_000, + relayEntryTimeout: 100, + expectedBlock: 1_100, + }, + "the highest request block that still admits the timeout": { + relayRequestBlock: math.MaxUint64 - 100, + relayEntryTimeout: 100, + expectedBlock: math.MaxUint64, + }, + "a request block one past what the timeout admits": { + relayRequestBlock: math.MaxUint64 - 99, + relayEntryTimeout: 100, + expectedError: true, + }, + "a timeout that overflows the block range on its own": { + relayRequestBlock: 1, + relayEntryTimeout: math.MaxUint64, + expectedError: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + timeoutBlock, err := relayEntryTimeoutBlock( + test.relayRequestBlock, + test.relayEntryTimeout, + ) + + if test.expectedError { + if !errors.Is(err, errRelayEntryDeadlineOverflow) { + t.Errorf( + "expected an overflow rejection\nactual: [%v]", + err, + ) + } + return + } + + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if timeoutBlock != test.expectedBlock { + t.Errorf( + "unexpected timeout block\nexpected: [%v]\nactual: [%v]", + test.expectedBlock, + timeoutBlock, + ) + } + }) + } +} + +// TestRelayEntryTimeoutReportResolutionDeadline asserts the resolution bound is +// rejected on overflow rather than clamped to the top of the block range. +func TestRelayEntryTimeoutReportResolutionDeadline(t *testing.T) { + tests := map[string]struct { + currentBlock uint64 + expectedDeadline uint64 + expectedError bool + }{ + "an ordinary height": { + currentBlock: 5_000, + expectedDeadline: 5_000 + relayEntryTimeoutReportResolutionBlocks, + }, + "the highest height that still admits the bound": { + currentBlock: math.MaxUint64 - relayEntryTimeoutReportResolutionBlocks, + expectedDeadline: math.MaxUint64, + }, + "one height past what the bound admits": { + currentBlock: math.MaxUint64 - relayEntryTimeoutReportResolutionBlocks + 1, + expectedError: true, + }, + "the top of the block range": { + currentBlock: math.MaxUint64, + expectedError: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + deadline, err := relayEntryTimeoutReportResolutionDeadline( + test.currentBlock, + ) + + if test.expectedError { + if !errors.Is(err, errRelayEntryDeadlineOverflow) { + t.Errorf( + "expected an overflow rejection\nactual: [%v]", + err, + ) + } + return + } + + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if deadline != test.expectedDeadline { + t.Errorf( + "unexpected deadline\nexpected: [%v]\nactual: [%v]", + test.expectedDeadline, + deadline, + ) + } + }) + } +} diff --git a/pkg/bitcoin/block.go b/pkg/bitcoin/block.go index ba54a45ffe..a963d82820 100644 --- a/pkg/bitcoin/block.go +++ b/pkg/bitcoin/block.go @@ -128,6 +128,15 @@ func (bh *BlockHeader) Difficulty() *big.Int { target := bh.Target() + // A malformed or zero-mantissa `Bits` field (e.g. 0x03000000) makes + // Target() return zero. Guard against it, as dividing by a zero target + // would panic. A zero or negative target yields zero difficulty, which is + // the safe, non-panicking result: such a header contributes no difficulty + // and gracefully fails downstream proof-difficulty checks. + if target.Sign() <= 0 { + return big.NewInt(0) + } + difficulty := new(big.Int) difficulty.Div(maxTarget, target) diff --git a/pkg/bitcoin/block_test.go b/pkg/bitcoin/block_test.go index 5ed5a8a851..39692276a5 100644 --- a/pkg/bitcoin/block_test.go +++ b/pkg/bitcoin/block_test.go @@ -226,3 +226,67 @@ func TestBlockHeaderDifficulty_LowestDifficulty(t *testing.T) { actualDifficulty, ) } + +func TestBlockHeaderDifficulty_ZeroTarget(t *testing.T) { + // A malformed `Bits` field with a zero mantissa (here 0x03000000) makes + // Target() return zero. Difficulty() must not panic on the division and + // must instead report zero difficulty. + defer func() { + if r := recover(); r != nil { + t.Fatalf("Difficulty() panicked on a zero target: %v", r) + } + }() + + blockHeader := BlockHeader{ + Bits: 0x03000000, + } + + actualDifficulty := blockHeader.Difficulty() + expectedDifficulty := big.NewInt(0) + + testutils.AssertBigIntsEqual( + t, + "difficulty", + expectedDifficulty, + actualDifficulty, + ) +} + +// TestBlockHeaderDifficulty_NonDIFF1RoundsToOne documents that integer +// difficulty is not a unique identifier of a header's target. The exact DIFF1 +// bits (0x1d00ffff) and a harder non-DIFF1 encoding (0x1d00aaaa) both round to +// integer difficulty 1, yet decode to different targets. Callers that must +// match the Bridge's exact minimum-difficulty target (BitcoinTx) therefore have +// to compare decoded targets, not Difficulty() == 1. +func TestBlockHeaderDifficulty_NonDIFF1RoundsToOne(t *testing.T) { + diff1Header := BlockHeader{Bits: 0x1d00ffff} + nonDiff1Header := BlockHeader{Bits: 0x1d00aaaa} + + // Both integer difficulties round down to 1. + one := big.NewInt(1) + testutils.AssertBigIntsEqual( + t, + "exact DIFF1 difficulty", + one, + diff1Header.Difficulty(), + ) + testutils.AssertBigIntsEqual( + t, + "non-DIFF1 difficulty", + one, + nonDiff1Header.Difficulty(), + ) + + // The decoded targets are not equal; the non-DIFF1 target is harder + // (numerically smaller) than the exact DIFF1 target. + diff1Target := diff1Header.Target() + nonDiff1Target := nonDiff1Header.Target() + if nonDiff1Target.Cmp(diff1Target) >= 0 { + t.Fatalf( + "expected non-DIFF1 target [%v] to be harder (smaller) than the "+ + "exact DIFF1 target [%v]", + nonDiff1Target, + diff1Target, + ) + } +} diff --git a/pkg/bitcoin/electrum/electrum.go b/pkg/bitcoin/electrum/electrum.go index e670646e4a..242ad7cf10 100644 --- a/pkg/bitcoin/electrum/electrum.go +++ b/pkg/bitcoin/electrum/electrum.go @@ -123,6 +123,22 @@ func (c *Connection) GetTransaction( return nil, fmt.Errorf("failed to convert transaction: [%w]", err) } + // Verify the server returned the transaction we actually asked for. The + // Electrum backend is untrusted: a malicious or MITM server could return a + // different (e.g. shorter) transaction for the requested hash, which would + // otherwise propagate downstream and trigger out-of-range panics when its + // outputs/inputs are indexed by an outpoint taken from another transaction. + // Transaction.Hash is the txid (non-witness double-SHA-256), which is what + // the request is keyed on. + if returnedHash := result.Hash(); returnedHash != transactionHash { + return nil, fmt.Errorf( + "electrum server returned transaction with hash [%s] "+ + "but [%s] was requested", + returnedHash.Hex(bitcoin.ReversedByteOrder), + txID, + ) + } + return result, nil } diff --git a/pkg/bitcoin/electrum/electrum_integration_test.go b/pkg/bitcoin/electrum/electrum_integration_test.go index 3b61743d83..6ce29e58b2 100644 --- a/pkg/bitcoin/electrum/electrum_integration_test.go +++ b/pkg/bitcoin/electrum/electrum_integration_test.go @@ -42,6 +42,22 @@ type testConfig struct { network bitcoin.Network } +// syncTestConfigNetworks applies defaults not already set inline on each test +// config. Network is now set explicitly on every literal below (see the +// package comment on testConfigs), so this is now a defensive no-op for +// Network; it still supplies the ConnectRetryTimeout default so integration +// tests fail fast against dead public endpoints instead of waiting out the +// 1m production default. +func syncTestConfigNetworks() { + for key, tc := range testConfigs { + tc.clientConfig.Network = tc.network + if tc.clientConfig.ConnectRetryTimeout == 0 { + tc.clientConfig.ConnectRetryTimeout = 15 * time.Second + } + testConfigs[key] = tc + } +} + // Servers details were taken from a public Electrum servers list published // at https://1209k.com/bitcoin-eye/ele.php?chain=tbtc. // @@ -80,15 +96,6 @@ var testConfigs = map[string]testConfig{ }, network: bitcoin.Testnet, }, - "fulcrum tcp": { - clientConfig: electrum.Config{ - URL: "tcp://testnet.aranguren.org:51001", - Network: bitcoin.Testnet, - RequestTimeout: requestTimeout * 2, - RequestRetryTimeout: requestRetryTimeout * 2, - }, - network: bitcoin.Testnet, - }, } var invalidTxID bitcoin.Hash @@ -156,6 +163,8 @@ func init() { if err != nil { panic(err) } + + syncTestConfigNetworks() } func TestConnect_Integration(t *testing.T) { @@ -643,6 +652,9 @@ func newTestConnection(t *testing.T, config electrum.Config) (bitcoin.Chain, con ctx, cancelCtx := context.WithCancel(context.Background()) electrum, err := electrum.Connect(ctx, config) if err != nil { + if shouldSkipElectrumIntegrationError(err) { + t.Skipf("skipping due to unavailable electrum endpoint: %v", err) + } t.Fatal(err) } @@ -781,5 +793,6 @@ func shouldSkipElectrumIntegrationError(err error) bool { return strings.Contains(msg, "request timeout") || strings.Contains(msg, "retry timeout") || - strings.Contains(msg, "enough information") + strings.Contains(msg, "enough information") || + strings.Contains(msg, "connection refused") } diff --git a/pkg/bitcoin/fuzz_test.go b/pkg/bitcoin/fuzz_test.go new file mode 100644 index 0000000000..2eeb33c5ff --- /dev/null +++ b/pkg/bitcoin/fuzz_test.go @@ -0,0 +1,140 @@ +package bitcoin + +import ( + "bytes" + "encoding/hex" + "testing" +) + +// Native coverage-guided fuzz targets for the pure deserializers that run on +// untrusted data fetched from external sources (an Electrum server). The +// invariant for every one of them is the same: arbitrary bytes must never +// cause a panic. Malformed input must be rejected with an error, not crash the +// process. Seeds include the valid examples used by the table-driven tests plus +// a few known malformed shapes; the fuzzer mutates from there. +// +// Seeds are decoded with the file-local fhex helper rather than the package's +// test-only decodeString: the OSS-Fuzz / ClusterFuzzLite native-fuzzing shim +// compiles each target from a generated non-test file, so a target may only +// reference symbols defined in this file or in non-test package code. +// +// Run locally with, e.g.: +// +// go test ./pkg/bitcoin/ -run=^$ -fuzz=FuzzNewScriptFromVarLenData -fuzztime=60s +// +// Crashers are persisted under testdata/fuzz// and become permanent +// regression cases on the next normal `go test` run. + +// fhex decodes a hex string seed. It is intentionally defined in this file (not +// shared with other _test.go files) so the fuzz targets remain compilable by +// the native-fuzzing shim. Seeds are compile-time constants, so a decode error +// is a programming mistake and fails the target loudly rather than silently +// degrading the seed corpus to nil. +func fhex(f *testing.F, s string) []byte { + b, err := hex.DecodeString(s) + if err != nil { + f.Fatalf("invalid hex seed %q: %v", s, err) + } + return b +} + +// FuzzNewScriptFromVarLenData fuzzes the variable-length script parser. Beyond +// "never panics", it asserts a round-trip property: any byte slice that parses +// successfully must serialize back to exactly the input via ToVarLenData (the +// CompactSizeUint length prefix is canonical, so this must hold). +func FuzzNewScriptFromVarLenData(f *testing.F) { + f.Add(fhex(f, "1600148db50eb52063ea9d98b3eac91489a90f738986f6")) // valid + f.Add(fhex(f, "16")) // missing script body + f.Add(fhex(f, "00148db50eb52063ea9d98b3eac91489a90f738986f6")) // missing length prefix + f.Add([]byte(nil)) // empty + f.Add([]byte{0xfd}) // truncated multi-byte CompactSizeUint + f.Add([]byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}) // huge declared length + + f.Fuzz(func(t *testing.T, data []byte) { + script, err := NewScriptFromVarLenData(data) + if err != nil { + // Malformed input rejected cleanly: the expected outcome. + return + } + + // On success the parsed script must round-trip back to the input. + roundTripped, err := script.ToVarLenData() + if err != nil { + t.Fatalf("ToVarLenData failed on a successfully parsed script: %v", err) + } + if !bytes.Equal(roundTripped, data) { + t.Fatalf( + "round-trip mismatch\n input: %x\n got: %x", + data, + roundTripped, + ) + } + }) +} + +// FuzzTransactionDeserialize fuzzes the transaction deserializer, the entry +// point for untrusted transaction bytes returned by an Electrum server. It must +// never panic on arbitrary input; an error return is the correct rejection. +// +// It also asserts a canonical fixed-point property: for any successfully +// parsed transaction, our own Serialize output must re-parse and serialize to +// identical bytes. Byte-identity with the INPUT deliberately is not asserted: +// Deserialize reads from the front of the buffer and accepts (ignores) +// trailing bytes, so non-canonical inputs can parse successfully — but +// everything downstream (Hash, the SPV proofs) operates on our serialization, +// which must be stable. +func FuzzTransactionDeserialize(f *testing.F) { + // A complete, valid standard (non-witness) serialized transaction. + f.Add(fhex(f, + "01000000036896f9abcac13ce6bd2b80d125bedf997ff6330e999f2f60"+ + "5ea15ea542f2eaf80000000000ffffffffed0ae94da996c6f3b89dfe967675d"+ + "4808251db93e81022ae9e038d06f92efed400000000c948304502210092327d"+ + "dff69a2b8c7ae787c5d590a2f14586089e6339e942d56e82aa42052cd902204"+ + "c0d1700ba1ac617da27fee032a57937c9607f0187199ed3c46954df845643d7"+ + "012103989d253b17a6a0f41838b84ff0d20e8898f9d7b1a98f2564da4cc29dc"+ + "f8581d94c5c14934b98637ca318a4d6e7ca6ffd1690b8e77df6377508f9f0c9"+ + "0d000395237576a9148db50eb52063ea9d98b3eac91489a90f738986f68763a"+ + "c6776a914e257eccafbc07c381642ce6e7e55120fb077fbed8804e0250162b1"+ + "75ac68ffffffffe37f552fc23fa0032bfd00c8eef5f5c22bf85fe4c6e735857"+ + "719ff8a4ff66eb80000000000ffffffff0180ed0000000000001600148db50e"+ + "b52063ea9d98b3eac91489a90f738986f600000000", + )) + f.Add([]byte(nil)) // empty + f.Add([]byte{0x01, 0x00, 0x00, 0x00}) // version only, truncated + f.Add([]byte{0x01, 0x00, 0x00, 0x00, 0xff}) // version + oversized input count + + f.Fuzz(func(t *testing.T, data []byte) { + var tx Transaction + // Must not panic on arbitrary input; an error return is acceptable. + if err := tx.Deserialize(data); err != nil { + return + } + + // Zero-input transactions are consensus-invalid but parseable from + // the witness encoding; their standard re-serialization starts with + // a 0x00 input count that collides with the segwit marker byte and + // cannot re-parse. That is a wire-format ambiguity, not a serializer + // defect, so the fixed-point property only applies to transactions + // with inputs. + if len(tx.Inputs) == 0 { + return + } + + serialized := tx.Serialize() + var reparsed Transaction + if err := reparsed.Deserialize(serialized); err != nil { + t.Fatalf( + "own serialization does not re-parse: %v\n serialized: %x", + err, + serialized, + ) + } + if reserialized := reparsed.Serialize(); !bytes.Equal(reserialized, serialized) { + t.Fatalf( + "serialization is not a fixed point\n first: %x\n second: %x", + serialized, + reserialized, + ) + } + }) +} diff --git a/pkg/bitcoin/testdata/fuzz/FuzzTransactionDeserialize/73d4d7631f5b3691 b/pkg/bitcoin/testdata/fuzz/FuzzTransactionDeserialize/73d4d7631f5b3691 new file mode 100644 index 0000000000..af840aa847 --- /dev/null +++ b/pkg/bitcoin/testdata/fuzz/FuzzTransactionDeserialize/73d4d7631f5b3691 @@ -0,0 +1,2 @@ +go test fuzz v1 +[]byte("0000\x00\x01\x00\x000000") diff --git a/pkg/bitcoin/testdata/fuzz/FuzzTransactionDeserialize/ed2e47109acbcdc8 b/pkg/bitcoin/testdata/fuzz/FuzzTransactionDeserialize/ed2e47109acbcdc8 new file mode 100644 index 0000000000..e1f46da656 --- /dev/null +++ b/pkg/bitcoin/testdata/fuzz/FuzzTransactionDeserialize/ed2e47109acbcdc8 @@ -0,0 +1,2 @@ +go test fuzz v1 +[]byte("0000\x03000000000000000000000000000000000000!0000000000000000000000000000000000000000000000000000000000000000000000000\b000000000000000000000000000000000000000000000000\x000000\x0000000") diff --git a/pkg/bitcoin/transaction.go b/pkg/bitcoin/transaction.go index d70c28f37b..2993a783df 100644 --- a/pkg/bitcoin/transaction.go +++ b/pkg/bitcoin/transaction.go @@ -3,6 +3,7 @@ package bitcoin import ( "bytes" "encoding/binary" + "fmt" "github.com/btcsuite/btcd/wire" ) @@ -183,6 +184,46 @@ func (t *Transaction) WitnessHash() Hash { return ComputeHash(t.Serialize(Witness)) } +// OutputAt returns the transaction output at the given zero-based index. It +// returns an error if the index is out of range instead of panicking. +// +// Prefer this over indexing Outputs directly whenever the index originates +// from untrusted or separately-fetched data (e.g. an outpoint from one +// transaction used to index the outputs of another transaction fetched from +// an Electrum backend). A backend that returns a valid-but-shorter transaction +// for a requested hash would otherwise trigger an index-out-of-range panic +// and crash the process. +func (t *Transaction) OutputAt(index uint32) (*TransactionOutput, error) { + if index >= uint32(len(t.Outputs)) { + return nil, fmt.Errorf( + "output index [%d] is out of range for transaction [%s] "+ + "that has [%d] output(s)", + index, + t.Hash().Hex(ReversedByteOrder), + len(t.Outputs), + ) + } + + return t.Outputs[index], nil //nolint:gocritic // accessor body: this IS the bounds-checked access, guarded by the len() check above +} + +// InputAt returns the transaction input at the given zero-based index. It +// returns an error if the index is out of range instead of panicking. See +// OutputAt for the rationale on untrusted/separately-fetched data. +func (t *Transaction) InputAt(index uint32) (*TransactionInput, error) { + if index >= uint32(len(t.Inputs)) { + return nil, fmt.Errorf( + "input index [%d] is out of range for transaction [%s] "+ + "that has [%d] input(s)", + index, + t.Hash().Hex(ReversedByteOrder), + len(t.Inputs), + ) + } + + return t.Inputs[index], nil //nolint:gocritic // accessor body: this IS the bounds-checked access, guarded by the len() check above +} + // TransactionOutpoint represents a Bitcoin transaction outpoint. // For reference, see: // https://developer.bitcoin.org/reference/transactions.html#outpoint-the-specific-part-of-a-specific-output diff --git a/pkg/bitcoin/transaction_bounds_test.go b/pkg/bitcoin/transaction_bounds_test.go new file mode 100644 index 0000000000..105d8fac8c --- /dev/null +++ b/pkg/bitcoin/transaction_bounds_test.go @@ -0,0 +1,77 @@ +package bitcoin + +import "testing" + +// These tests cover the bounds-checked accessors that guard against +// out-of-range panics when an index originates from untrusted or +// separately-fetched transaction data. See the security audit OOB cluster +// (F-002/003/004/006/007/012). + +func TestTransaction_OutputAt(t *testing.T) { + transaction := &Transaction{ + Outputs: []*TransactionOutput{ + {Value: 100, PublicKeyScript: []byte{0x01}}, + {Value: 200, PublicKeyScript: []byte{0x02}}, + }, + } + + for _, index := range []uint32{0, 1} { + output, err := transaction.OutputAt(index) + if err != nil { + t.Fatalf("unexpected error for in-range index [%d]: [%v]", index, err) + } + if output != transaction.Outputs[index] { + t.Errorf("OutputAt(%d) returned the wrong output", index) + } + } + + // Out-of-range indices must return an error, never panic. + for _, index := range []uint32{2, 3, 1 << 31} { + output, err := transaction.OutputAt(index) + if err == nil { + t.Errorf("expected an out-of-range error for index [%d], got nil", index) + } + if output != nil { + t.Errorf("expected a nil output for out-of-range index [%d]", index) + } + } +} + +func TestTransaction_OutputAt_NoOutputs(t *testing.T) { + transaction := &Transaction{} + if _, err := transaction.OutputAt(0); err == nil { + t.Error("expected an error indexing a transaction that has no outputs") + } +} + +func TestTransaction_InputAt(t *testing.T) { + transaction := &Transaction{ + Inputs: []*TransactionInput{ + {Outpoint: &TransactionOutpoint{OutputIndex: 0}}, + }, + } + + input, err := transaction.InputAt(0) + if err != nil { + t.Fatalf("unexpected error for in-range index: [%v]", err) + } + if input != transaction.Inputs[0] { + t.Error("InputAt(0) returned the wrong input") + } + + for _, index := range []uint32{1, 5} { + if _, err := transaction.InputAt(index); err == nil { + t.Errorf("expected an out-of-range error for index [%d], got nil", index) + } + } +} + +// TestTransaction_InputAt_NoInputs covers the F-012 case directly: indexing +// Inputs[0] on a zero-input transaction (a malicious Electrum backend can +// decode a segwit-flagged zero-input tx) must return an error, not panic. +func TestTransaction_InputAt_NoInputs(t *testing.T) { + transaction := &Transaction{} + if _, err := transaction.InputAt(0); err == nil { + t.Error("expected an error indexing a transaction that has no inputs") + } +} diff --git a/pkg/bitcoin/transaction_builder.go b/pkg/bitcoin/transaction_builder.go index 06e595d65d..8e7c622c63 100644 --- a/pkg/bitcoin/transaction_builder.go +++ b/pkg/bitcoin/transaction_builder.go @@ -148,18 +148,12 @@ func (tb *TransactionBuilder) getScript( ) } - outputIndex := utxo.Outpoint.OutputIndex - if outputIndex >= uint32(len(transaction.Outputs)) { - return nil, fmt.Errorf( - "output index [%d] out of range for transaction [%s] "+ - "with [%d] outputs", - outputIndex, - hash.Hex(InternalByteOrder), - len(transaction.Outputs), - ) + output, err := transaction.OutputAt(utxo.Outpoint.OutputIndex) + if err != nil { + return nil, err } - return transaction.Outputs[outputIndex].PublicKeyScript, nil + return output.PublicKeyScript, nil } // AddOutput adds a new transaction's output. diff --git a/pkg/bitcoin/transaction_builder_test.go b/pkg/bitcoin/transaction_builder_test.go index 96adf8dede..49a658e375 100644 --- a/pkg/bitcoin/transaction_builder_test.go +++ b/pkg/bitcoin/transaction_builder_test.go @@ -135,7 +135,7 @@ func TestTransactionBuilder_AddInputReturnsErrorForOutOfRangeOutputIndex( if err == nil { t.Fatal("expected out-of-range output index error") } - if !strings.Contains(err.Error(), "output index [3] out of range") { + if !strings.Contains(err.Error(), "output index [3] is out of range") { t.Fatalf("unexpected error: [%v]", err) } } diff --git a/pkg/chain/ethereum/beacon.go b/pkg/chain/ethereum/beacon.go index 6f37143e62..17bdebd5b1 100644 --- a/pkg/chain/ethereum/beacon.go +++ b/pkg/chain/ethereum/beacon.go @@ -1,9 +1,11 @@ package ethereum import ( + "bytes" "fmt" "math/big" + "github.com/ethereum/go-ethereum/core/types" "github.com/ethereum/go-ethereum/crypto" beaconchain "github.com/keep-network/keep-core/pkg/beacon/chain" "github.com/keep-network/keep-core/pkg/beacon/event" @@ -12,6 +14,7 @@ import ( "github.com/ethereum/go-ethereum/common" "github.com/keep-network/keep-common/pkg/chain/ethereum" "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/chain/ethereum/beacon/gen/abi" "github.com/keep-network/keep-core/pkg/chain/ethereum/beacon/gen/contract" "github.com/keep-network/keep-core/pkg/operator" ) @@ -29,6 +32,12 @@ type BeaconChain struct { randomBeacon *contract.RandomBeacon sortitionPool *contract.BeaconSortitionPool + + // randomBeaconAddress is the address the RandomBeacon handle was resolved + // to. Canonical chain records read back from that contract carry it, so a + // record can name the deployment it came from rather than leaving a reader + // to assume one. + randomBeaconAddress common.Address } // newBeaconChain construct a new instance of the beacon-specific Ethereum @@ -91,9 +100,10 @@ func newBeaconChain( } return &BeaconChain{ - baseChain: baseChain, - randomBeacon: randomBeacon, - sortitionPool: sortitionPool, + baseChain: baseChain, + randomBeacon: randomBeacon, + sortitionPool: sortitionPool, + randomBeaconAddress: randomBeaconAddress, }, nil } @@ -429,3 +439,511 @@ func (bc *BeaconChain) CurrentRequestPreviousEntry() ([]byte, error) { func (bc *BeaconChain) CurrentRequestGroupPublicKey() ([]byte, error) { return nil, errNotImplemented } + +// soleCanonicalLog returns the index of the one log in a filtered range that +// the caller's predicate accepts, or -1 when none does, and reports separately +// whether more than one did. +// +// More than one match is reported rather than resolved. Nothing in the logs +// says which one a caller meant, and picking either would make a penalty rest +// on an ordering the chain never promised. +func soleCanonicalLog(count int, matches func(int) bool) (int, bool) { + index := -1 + for i := 0; i < count; i++ { + if !matches(i) { + continue + } + if index >= 0 { + return -1, true + } + index = i + } + + return index, false +} + +// relayEntryLogs is the chain view a relay entry timeout settlement is read +// from: the head that view is pinned to, the hash each height on it commits +// to, and the three log sets the settlement is composed of. +// +// The settlement is a penalty claim over three separate reads, so what the +// reads are allowed to disagree about is part of the contract rather than a +// property of whichever backend happens to serve them. Naming the view lets a +// backend that moves between the reads be exercised. +type relayEntryLogs interface { + // CurrentBlock returns the head of the view. + CurrentBlock() (uint64, error) + + // BlockHashByNumber returns the hash the view holds at the given height. + BlockHashByNumber(blockNumber uint64) ([32]byte, error) + + // RelayEntryRequests returns the relay entry requests logged in the given + // inclusive block range. + RelayEntryRequests( + startBlock, endBlock uint64, + ) ([]*abi.RandomBeaconRelayEntryRequested, error) + + // RelayEntrySubmissions returns the entry submissions logged for the given + // request in the given inclusive block range. + RelayEntrySubmissions( + startBlock, endBlock uint64, + requestID *big.Int, + ) ([]*abi.RandomBeaconRelayEntrySubmitted, error) + + // RelayEntryTimeouts returns the entry timeouts logged for the given + // request in the given inclusive block range. + RelayEntryTimeouts( + startBlock, endBlock uint64, + requestID *big.Int, + ) ([]*abi.RandomBeaconRelayEntryTimedOut, error) +} + +// beaconRelayEntryLogs reads relay entry logs off the RandomBeacon deployment +// this node is attached to. +type beaconRelayEntryLogs struct { + chain *BeaconChain +} + +func (brel beaconRelayEntryLogs) CurrentBlock() (uint64, error) { + return brel.chain.blockCounter.CurrentBlock() +} + +func (brel beaconRelayEntryLogs) BlockHashByNumber(blockNumber uint64) ( + [32]byte, + error, +) { + return brel.chain.GetBlockHashByNumber(blockNumber) +} + +func (brel beaconRelayEntryLogs) RelayEntryRequests( + startBlock, endBlock uint64, +) ([]*abi.RandomBeaconRelayEntryRequested, error) { + return brel.chain.randomBeacon.PastRelayEntryRequestedEvents( + startBlock, + &endBlock, + nil, + ) +} + +func (brel beaconRelayEntryLogs) RelayEntrySubmissions( + startBlock, endBlock uint64, + requestID *big.Int, +) ([]*abi.RandomBeaconRelayEntrySubmitted, error) { + return brel.chain.randomBeacon.PastRelayEntrySubmittedEvents( + startBlock, + &endBlock, + []*big.Int{requestID}, + ) +} + +func (brel beaconRelayEntryLogs) RelayEntryTimeouts( + startBlock, endBlock uint64, + requestID *big.Int, +) ([]*abi.RandomBeaconRelayEntryTimedOut, error) { + return brel.chain.randomBeacon.PastRelayEntryTimedOutEvents( + startBlock, + &endBlock, + []*big.Int{requestID}, + ) +} + +// RelayEntryTimeoutSettlement reads the RandomBeacon's own record that the +// relay request made at the given block, over the given previous entry, was +// terminated by an accepted timeout report. +// +// The record is assembled from canonical logs on every call and nothing is +// carried between calls. The request is identified by the RelayEntryRequested +// log at the named block that signs over the named previous entry, so a chain +// view that does not hold that log yields no settlement rather than one built +// on a request the view never had — which is what makes a reorg that removed +// the request take the penalty claim with it. +// +// A request the beacon answered is refused before the timeout logs are read. +// A delivered entry and a timeout are mutually exclusive endings, so a chain +// reporting both is one this node must not choose between. +func (bc *BeaconChain) RelayEntryTimeoutSettlement( + requestBlockNumber uint64, + requestPreviousEntry []byte, +) (*event.RelayEntryTimeoutSettlement, error) { + return resolveRelayEntryTimeoutSettlement( + beaconRelayEntryLogs{chain: bc}, + bc.randomBeaconAddress.String(), + requestBlockNumber, + requestPreviousEntry, + ) +} + +// resolveRelayEntryTimeoutSettlement reads one relay request, its submissions +// and its timeouts out of a single pinned chain view, and composes the +// settlement from them. +// +// The three log sets are read as one snapshot rather than three independent +// latest-head queries. Every read is bounded by the same end block, every log +// kept is checked to sit on the branch that end block descends from, and the +// hash of that end block is confirmed unchanged once the reads are done. A +// backend that reorgs part-way through therefore fails the resolution instead +// of returning a settlement that pairs a request on one branch with the +// termination of the same request ID on another — a pairing no canonical view +// of the chain ever held, and one that would claim a penalty on that basis. +func resolveRelayEntryTimeoutSettlement( + logs relayEntryLogs, + contractAddress string, + requestBlockNumber uint64, + requestPreviousEntry []byte, +) (*event.RelayEntryTimeoutSettlement, error) { + if len(requestPreviousEntry) == 0 { + return nil, fmt.Errorf( + "cannot resolve a relay entry timeout settlement without the " + + "previous entry the request was signing over", + ) + } + + snapshot, err := openRelayEntryLogSnapshot(logs) + if err != nil { + return nil, err + } + + // A view whose head has not reached the request cannot hold the request, + // let alone its termination. There is nothing to settle yet and nothing to + // read a settlement from; the caller reads again against a later view. + if snapshot.endBlock < requestBlockNumber { + return nil, nil + } + + requests, err := logs.RelayEntryRequests( + requestBlockNumber, + requestBlockNumber, + ) + if err != nil { + return nil, fmt.Errorf( + "failed to read the relay entry requests of block [%v]: [%w]", + requestBlockNumber, + err, + ) + } + + requestIndex, ambiguous, err := snapshot.soleHeldLog( + len(requests), + func(i int) types.Log { return requests[i].Raw }, + func(i int) bool { + return bytes.Equal(requests[i].PreviousEntry, requestPreviousEntry) + }, + ) + if err != nil { + return nil, err + } + if ambiguous { + return nil, fmt.Errorf( + "block [%v] holds more than one relay entry request over the "+ + "named previous entry; the request a timeout settlement would "+ + "terminate is ambiguous", + requestBlockNumber, + ) + } + if requestIndex < 0 { + return nil, nil + } + request := requests[requestIndex] + + submissions, err := logs.RelayEntrySubmissions( + requestBlockNumber, + snapshot.endBlock, + request.RequestId, + ) + if err != nil { + return nil, fmt.Errorf( + "failed to read the relay entry submissions of request [%s]: [%w]", + request.RequestId, + err, + ) + } + + timeouts, err := logs.RelayEntryTimeouts( + requestBlockNumber, + snapshot.endBlock, + request.RequestId, + ) + if err != nil { + return nil, fmt.Errorf( + "failed to read the relay entry timeouts of request [%s]: [%w]", + request.RequestId, + err, + ) + } + + heldSubmissions, err := snapshot.heldSubmissions(submissions) + if err != nil { + return nil, err + } + + heldTimeouts, err := snapshot.heldTimeouts(timeouts) + if err != nil { + return nil, err + } + + // Everything the settlement rests on has now been read and bound to the + // branch this snapshot was opened on. Confirming that branch is still the + // one the view holds is what rules out a composition assembled across a + // reorg that happened mid-read. + if err := snapshot.confirm(); err != nil { + return nil, err + } + + return relayEntryTimeoutSettlement( + request, + heldSubmissions, + heldTimeouts, + contractAddress, + ) +} + +// relayEntryLogSnapshot is one pinned chain view: an end block every read is +// bounded by, and the hash that block was seen with. That hash commits to the +// block's whole ancestry, so it is what a log read below the end block is +// checked against and what the view is confirmed to still hold afterwards. +type relayEntryLogSnapshot struct { + logs relayEntryLogs + endBlock uint64 + endBlockHash [32]byte + + // blockHashes caches the hash the view holds at each height a log was + // checked at, so a range holding many logs costs one lookup per block + // rather than one per log. + blockHashes map[uint64][32]byte +} + +// openRelayEntryLogSnapshot pins a view to the head the backend reports and +// the hash it holds there. +func openRelayEntryLogSnapshot(logs relayEntryLogs) ( + *relayEntryLogSnapshot, + error, +) { + endBlock, err := logs.CurrentBlock() + if err != nil { + return nil, fmt.Errorf( + "failed to read the head a relay entry timeout settlement would "+ + "be resolved against: [%w]", + err, + ) + } + + endBlockHash, err := logs.BlockHashByNumber(endBlock) + if err != nil { + return nil, fmt.Errorf( + "failed to read the hash of block [%v] a relay entry timeout "+ + "settlement would be resolved against: [%w]", + endBlock, + err, + ) + } + + return &relayEntryLogSnapshot{ + logs: logs, + endBlock: endBlock, + endBlockHash: endBlockHash, + blockHashes: make(map[uint64][32]byte), + }, nil +} + +// holds reports whether the given log sits on the branch this snapshot was +// opened on. +// +// A log is held when the view still reports its block as the canonical one at +// that height and the backend has not flagged it as removed. A log the view +// no longer holds belongs to an abandoned branch: it is not evidence of +// anything the chain currently says happened, and composing it with logs that +// are held is exactly the cross-branch reading the snapshot exists to reject. +// +// A hash that cannot be read is an error rather than a "not held", because +// dropping a log on a failed lookup silently moves the composition towards +// claiming a penalty. +func (rels *relayEntryLogSnapshot) holds(log types.Log) (bool, error) { + if log.Removed { + return false, nil + } + + if log.BlockNumber > rels.endBlock { + return false, fmt.Errorf( + "the backend returned a log from block [%v], past the block [%v] "+ + "the read was bounded by", + log.BlockNumber, + rels.endBlock, + ) + } + + blockHash, cached := rels.blockHashes[log.BlockNumber] + if !cached { + if log.BlockNumber == rels.endBlock { + blockHash = rels.endBlockHash + } else { + var err error + blockHash, err = rels.logs.BlockHashByNumber(log.BlockNumber) + if err != nil { + return false, fmt.Errorf( + "failed to read the hash the chain view holds at block "+ + "[%v], where a relay entry log the settlement would "+ + "be composed from was mined: [%w]", + log.BlockNumber, + err, + ) + } + } + rels.blockHashes[log.BlockNumber] = blockHash + } + + return common.Hash(blockHash) == log.BlockHash, nil +} + +// soleHeldLog returns the index of the one log the snapshot holds and the +// caller's predicate accepts, or -1 when none does, and reports separately +// whether more than one did. +// +// More than one match is reported rather than resolved. Nothing in the logs +// says which one a caller meant, and picking either would make a penalty rest +// on an ordering the chain never promised. +func (rels *relayEntryLogSnapshot) soleHeldLog( + count int, + raw func(int) types.Log, + matches func(int) bool, +) (int, bool, error) { + var lookupErr error + + index, ambiguous := soleCanonicalLog(count, func(i int) bool { + if lookupErr != nil || !matches(i) { + return false + } + + held, err := rels.holds(raw(i)) + if err != nil { + lookupErr = err + return false + } + + return held + }) + if lookupErr != nil { + return -1, false, lookupErr + } + + return index, ambiguous, nil +} + +// heldSubmissions drops the entry submissions this snapshot's branch does not +// hold. +func (rels *relayEntryLogSnapshot) heldSubmissions( + submissions []*abi.RandomBeaconRelayEntrySubmitted, +) ([]*abi.RandomBeaconRelayEntrySubmitted, error) { + held := make([]*abi.RandomBeaconRelayEntrySubmitted, 0, len(submissions)) + for _, submission := range submissions { + onBranch, err := rels.holds(submission.Raw) + if err != nil { + return nil, err + } + if onBranch { + held = append(held, submission) + } + } + + return held, nil +} + +// heldTimeouts drops the entry timeouts this snapshot's branch does not hold. +func (rels *relayEntryLogSnapshot) heldTimeouts( + timeouts []*abi.RandomBeaconRelayEntryTimedOut, +) ([]*abi.RandomBeaconRelayEntryTimedOut, error) { + held := make([]*abi.RandomBeaconRelayEntryTimedOut, 0, len(timeouts)) + for _, timeout := range timeouts { + onBranch, err := rels.holds(timeout.Raw) + if err != nil { + return nil, err + } + if onBranch { + held = append(held, timeout) + } + } + + return held, nil +} + +// confirm re-reads the hash of the end block and fails when the view no longer +// holds the branch the snapshot was opened on. +// +// The logs were read one call at a time; this is what makes the composition of +// them answer a single view of the chain. Without it a reorg landing between +// two of the reads would go unnoticed, and the two halves of a settlement +// could come from branches that never coexisted. +func (rels *relayEntryLogSnapshot) confirm() error { + endBlockHash, err := rels.logs.BlockHashByNumber(rels.endBlock) + if err != nil { + return fmt.Errorf( + "failed to re-read the hash of block [%v] the relay entry "+ + "timeout settlement was resolved against: [%w]", + rels.endBlock, + err, + ) + } + + if endBlockHash != rels.endBlockHash { + return fmt.Errorf( + "the chain view moved off block [%v] hash [%s] while the relay "+ + "entry timeout settlement was being read; the logs it would "+ + "be composed from do not answer one view of the chain", + rels.endBlock, + common.Hash(rels.endBlockHash), + ) + } + + return nil +} + +// relayEntryTimeoutSettlement decides, from the canonical logs of one relay +// request, whether that request was terminated by an accepted timeout report. +// +// The decision is separated from the reads because it is the part that has to +// be right: a request the beacon answered ends the claim before the timeout +// logs matter, since a delivered entry and a timeout are mutually exclusive +// endings and a chain reporting both is one this node must not choose between. +// A submission belonging to an abandoned branch is not such an answer, and +// neither is a termination — both are skipped, so a reorg takes the state it +// removed with it rather than leaving a stale reading behind. +func relayEntryTimeoutSettlement( + request *abi.RandomBeaconRelayEntryRequested, + submissions []*abi.RandomBeaconRelayEntrySubmitted, + timeouts []*abi.RandomBeaconRelayEntryTimedOut, + contractAddress string, +) (*event.RelayEntryTimeoutSettlement, error) { + for _, submission := range submissions { + if !submission.Raw.Removed { + return nil, nil + } + } + + timeoutIndex, ambiguous := soleCanonicalLog( + len(timeouts), + func(i int) bool { return !timeouts[i].Raw.Removed }, + ) + if ambiguous { + return nil, fmt.Errorf( + "request [%s] was terminated more than once; the settlement to "+ + "record is ambiguous", + request.RequestId, + ) + } + if timeoutIndex < 0 { + return nil, nil + } + timeout := timeouts[timeoutIndex] + + previousEntry := make([]byte, len(request.PreviousEntry)) + copy(previousEntry, request.PreviousEntry) + + return &event.RelayEntryTimeoutSettlement{ + RequestID: new(big.Int).Set(request.RequestId), + TerminatedGroupID: timeout.TerminatedGroupId, + RequestBlockNumber: request.Raw.BlockNumber, + RequestPreviousEntry: previousEntry, + BlockNumber: timeout.Raw.BlockNumber, + TransactionHash: timeout.Raw.TxHash, + ContractAddress: contractAddress, + }, nil +} diff --git a/pkg/chain/ethereum/beacon_test.go b/pkg/chain/ethereum/beacon_test.go new file mode 100644 index 0000000000..7c4d24c3f1 --- /dev/null +++ b/pkg/chain/ethereum/beacon_test.go @@ -0,0 +1,838 @@ +package ethereum + +import ( + "bytes" + "fmt" + "math/big" + "reflect" + "testing" + + "github.com/ethereum/go-ethereum/common" + "github.com/ethereum/go-ethereum/core/types" + + "github.com/keep-network/keep-core/pkg/chain/ethereum/beacon/gen/abi" +) + +// TestSoleCanonicalLog_SkipsOrphanedBranches asserts the selection a relay +// entry timeout settlement is assembled from ignores logs the backend marked as +// removed and refuses to choose between two live matches. +// +// The removed flag is what a reorg leaves behind on a log that no longer +// belongs to the canonical chain. A settlement built on one would claim a +// penalty over state the chain has already abandoned, which is precisely the +// claim this lookup exists to prevent. +func TestSoleCanonicalLog_SkipsOrphanedBranches(t *testing.T) { + previousEntry := []byte("previous-entry-of-the-reported-request") + otherPreviousEntry := []byte("previous-entry-after-a-delivered-entry") + + requestLog := func( + removed bool, + entry []byte, + ) *abi.RandomBeaconRelayEntryRequested { + return &abi.RandomBeaconRelayEntryRequested{ + PreviousEntry: entry, + Raw: types.Log{Removed: removed}, + } + } + + tests := map[string]struct { + requests []*abi.RandomBeaconRelayEntryRequested + expectedIndex int + expectedAmbiguous bool + }{ + "no requests at all": { + requests: nil, + expectedIndex: -1, + }, + "one live request over the named previous entry": { + requests: []*abi.RandomBeaconRelayEntryRequested{ + requestLog(false, previousEntry), + }, + expectedIndex: 0, + }, + // The reorg case: the only log naming the request belongs to an + // orphaned branch, so the canonical chain holds no such request. + "the only matching request was reorged out": { + requests: []*abi.RandomBeaconRelayEntryRequested{ + requestLog(true, previousEntry), + }, + expectedIndex: -1, + }, + // The same request re-mined on the new branch after the orphaned one. + // The live log is the canonical answer and the removed one must not + // make the pair look ambiguous. + "a request re-mined after being reorged out": { + requests: []*abi.RandomBeaconRelayEntryRequested{ + requestLog(true, previousEntry), + requestLog(false, previousEntry), + }, + expectedIndex: 1, + }, + "another request in the same block": { + requests: []*abi.RandomBeaconRelayEntryRequested{ + requestLog(false, otherPreviousEntry), + requestLog(false, previousEntry), + }, + expectedIndex: 1, + }, + "two live requests over the named previous entry": { + requests: []*abi.RandomBeaconRelayEntryRequested{ + requestLog(false, previousEntry), + requestLog(false, previousEntry), + }, + expectedIndex: -1, + expectedAmbiguous: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + requests := test.requests + + index, ambiguous := soleCanonicalLog( + len(requests), + func(i int) bool { + return !requests[i].Raw.Removed && + bytes.Equal(requests[i].PreviousEntry, previousEntry) + }, + ) + + if index != test.expectedIndex { + t.Errorf( + "unexpected log index\nexpected: [%d]\nactual: [%d]", + test.expectedIndex, + index, + ) + } + if ambiguous != test.expectedAmbiguous { + t.Errorf( + "unexpected ambiguity\nexpected: [%t]\nactual: [%t]", + test.expectedAmbiguous, + ambiguous, + ) + } + }) + } +} + +// TestRelayEntryTimeoutSettlement asserts the reading a filed timeout report is +// claimed as a penalty on. +// +// The monitor's permit closes as completed on whatever this returns, and that +// record clears the rollback barrier the penalty exists to hold. So the +// endings a request can actually have are enumerated here rather than left to +// the one path a passing case exercises: the report was accepted, the report +// never landed, the group delivered late, the delivery itself was reorged away, +// and the termination was. +func TestRelayEntryTimeoutSettlement(t *testing.T) { + const contractAddress = "0x1111111111111111111111111111111111111111" + const terminatedGroupID = uint64(4) + const requestBlock = uint64(1_000) + const timeoutBlock = uint64(1_100) + + requestID := big.NewInt(77) + previousEntry := []byte("previous-entry-of-the-reported-request") + timeoutTransaction := common.HexToHash("0xabc") + + request := &abi.RandomBeaconRelayEntryRequested{ + RequestId: requestID, + GroupId: 1, + PreviousEntry: previousEntry, + Raw: types.Log{BlockNumber: requestBlock}, + } + + submission := func(removed bool) *abi.RandomBeaconRelayEntrySubmitted { + return &abi.RandomBeaconRelayEntrySubmitted{ + RequestId: requestID, + Entry: []byte("the group's answer"), + Raw: types.Log{BlockNumber: timeoutBlock, Removed: removed}, + } + } + timeout := func( + removed bool, + groupID uint64, + ) *abi.RandomBeaconRelayEntryTimedOut { + return &abi.RandomBeaconRelayEntryTimedOut{ + RequestId: requestID, + TerminatedGroupId: groupID, + Raw: types.Log{ + BlockNumber: timeoutBlock, + TxHash: timeoutTransaction, + Removed: removed, + }, + } + } + + tests := map[string]struct { + submissions []*abi.RandomBeaconRelayEntrySubmitted + timeouts []*abi.RandomBeaconRelayEntryTimedOut + expectSettlement bool + expectError bool + expectedGroupID uint64 + }{ + "the report was accepted": { + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{timeout(false, terminatedGroupID)}, + expectSettlement: true, + expectedGroupID: terminatedGroupID, + }, + // The report reverted, was dropped, or lost the race to another + // reporter: the beacon is exactly as it was and no penalty was earned. + "no termination was recorded": {}, + // The group answered after the report was filed. A delivered entry and + // a timeout are mutually exclusive endings, so the delivery ends the + // claim before the timeout logs are even weighed. + "the group delivered late": { + submissions: []*abi.RandomBeaconRelayEntrySubmitted{submission(false)}, + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{timeout(false, terminatedGroupID)}, + }, + "the group delivered and no termination followed": { + submissions: []*abi.RandomBeaconRelayEntrySubmitted{submission(false)}, + }, + // The branch carrying the delivery was abandoned, so it answers + // nothing and the live termination stands. + "the delivery was reorged out": { + submissions: []*abi.RandomBeaconRelayEntrySubmitted{submission(true)}, + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{timeout(false, terminatedGroupID)}, + expectSettlement: true, + expectedGroupID: terminatedGroupID, + }, + // The mirror case: the termination is the log the chain abandoned, so + // the penalty went with it. + "the termination was reorged out": { + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{timeout(true, terminatedGroupID)}, + }, + "a termination re-mined after being reorged out": { + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{ + timeout(true, terminatedGroupID+1), + timeout(false, terminatedGroupID), + }, + expectSettlement: true, + expectedGroupID: terminatedGroupID, + }, + // Nothing says which group the settlement should name, and picking + // either would rest a penalty on an ordering the chain never promised. + "two live terminations": { + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{ + timeout(false, terminatedGroupID), + timeout(false, terminatedGroupID+1), + }, + expectError: true, + }, + "a live delivery among reorged ones": { + submissions: []*abi.RandomBeaconRelayEntrySubmitted{ + submission(true), + submission(false), + }, + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{timeout(false, terminatedGroupID)}, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + settlement, err := relayEntryTimeoutSettlement( + request, + test.submissions, + test.timeouts, + contractAddress, + ) + + if test.expectError { + if err == nil { + t.Fatal("expected an ambiguous reading to be refused") + } + if settlement != nil { + t.Error("a refused reading must claim no settlement") + } + return + } + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + if !test.expectSettlement { + if settlement != nil { + t.Fatalf( + "expected no penalty to be claimed, got group [%d]", + settlement.TerminatedGroupID, + ) + } + return + } + if settlement == nil { + t.Fatal("expected an accepted report to be claimed as settled") + } + + if settlement.RequestID.Cmp(requestID) != 0 { + t.Errorf( + "unexpected request identifier\nexpected: [%s]\nactual: [%s]", + requestID, + settlement.RequestID, + ) + } + if settlement.TerminatedGroupID != test.expectedGroupID { + t.Errorf( + "unexpected terminated group\nexpected: [%d]\nactual: [%d]", + test.expectedGroupID, + settlement.TerminatedGroupID, + ) + } + if settlement.RequestBlockNumber != requestBlock { + t.Errorf( + "unexpected request block\nexpected: [%d]\nactual: [%d]", + requestBlock, + settlement.RequestBlockNumber, + ) + } + if !bytes.Equal(settlement.RequestPreviousEntry, previousEntry) { + t.Error("the settlement names another request's previous entry") + } + if settlement.BlockNumber != timeoutBlock { + t.Errorf( + "unexpected settlement block\nexpected: [%d]\nactual: [%d]", + timeoutBlock, + settlement.BlockNumber, + ) + } + if settlement.TransactionHash != timeoutTransaction { + t.Error("the settlement names another transaction") + } + if settlement.ContractAddress != contractAddress { + t.Errorf( + "unexpected contract\nexpected: [%s]\nactual: [%s]", + contractAddress, + settlement.ContractAddress, + ) + } + }) + } +} + +// TestRelayEntryTimeoutSettlement_DoesNotAliasTheRequestLog asserts the +// settlement carries its own copy of the previous entry the request signs over. +// +// The record outlives the log slice the caller read it from, and a settlement +// that aliased that memory would name whatever a later read wrote over it. +func TestRelayEntryTimeoutSettlement_DoesNotAliasTheRequestLog(t *testing.T) { + previousEntry := []byte("previous-entry-of-the-reported-request") + request := &abi.RandomBeaconRelayEntryRequested{ + RequestId: big.NewInt(77), + PreviousEntry: previousEntry, + Raw: types.Log{BlockNumber: 1_000}, + } + + settlement, err := relayEntryTimeoutSettlement( + request, + nil, + []*abi.RandomBeaconRelayEntryTimedOut{ + { + RequestId: request.RequestId, + TerminatedGroupId: 4, + Raw: types.Log{BlockNumber: 1_100}, + }, + }, + "0x1111111111111111111111111111111111111111", + ) + if err != nil { + t.Fatal(err) + } + if settlement == nil { + t.Fatal("expected an accepted report to be claimed as settled") + } + + for i := range previousEntry { + previousEntry[i] = 0 + } + if bytes.Equal(settlement.RequestPreviousEntry, previousEntry) { + t.Error("the settlement aliases the request log's previous entry") + } + + request.RequestId.SetInt64(0) + if settlement.RequestID.Sign() == 0 { + t.Error("the settlement aliases the request log's identifier") + } +} + +// relayEntryChainView is one branch of the chain as a backend would serve it: +// the head it reports, the hash it holds at each height, and the logs it +// returns for each of the three reads a settlement is composed from. +type relayEntryChainView struct { + head uint64 + blockHashes map[uint64]common.Hash + requests []*abi.RandomBeaconRelayEntryRequested + submissions []*abi.RandomBeaconRelayEntrySubmitted + timeouts []*abi.RandomBeaconRelayEntryTimedOut +} + +// relayEntryLogRead records the bounds one read was made with, so a test can +// assert the three reads were bounded by one view instead of each running +// against whatever the head happened to be. +type relayEntryLogRead struct { + name string + startBlock uint64 + endBlock uint64 + requestID *big.Int +} + +// scriptedRelayEntryLogs is a backend whose chain view moves from one branch to +// another part-way through the reads a settlement is composed from. +// +// A reorg is not something a settlement can be tested against by handing it +// pre-composed log slices: the whole question is what happens when the reads +// disagree about which branch they answer, and that only exists between the +// calls. Switching the served branch at a named read reproduces it exactly. +type scriptedRelayEntryLogs struct { + before relayEntryChainView + after relayEntryChainView + + // reorgBefore names the read the view moves to the second branch at: one + // of "requests", "submissions", "timeouts", or "confirm" for the re-read + // of the end block once the log reads are done. + reorgBefore string + + // blockHashErrors holds the heights the backend cannot answer a hash for. + blockHashErrors map[uint64]error + + reorged bool + timeoutsRead bool + pinnedHead uint64 + reads []relayEntryLogRead +} + +func (srel *scriptedRelayEntryLogs) view() *relayEntryChainView { + if srel.reorged { + return &srel.after + } + + return &srel.before +} + +func (srel *scriptedRelayEntryLogs) reorgAt(read string) { + if srel.reorgBefore == read { + srel.reorged = true + } +} + +func (srel *scriptedRelayEntryLogs) CurrentBlock() (uint64, error) { + srel.pinnedHead = srel.view().head + return srel.pinnedHead, nil +} + +func (srel *scriptedRelayEntryLogs) BlockHashByNumber(blockNumber uint64) ( + [32]byte, + error, +) { + // The confirming read is the one that asks for the pinned head again after + // all the logs are in hand. + if srel.timeoutsRead && blockNumber == srel.pinnedHead { + srel.reorgAt("confirm") + } + + if err, failing := srel.blockHashErrors[blockNumber]; failing { + return [32]byte{}, err + } + + blockHash, held := srel.view().blockHashes[blockNumber] + if !held { + return [32]byte{}, fmt.Errorf("the view holds no block [%v]", blockNumber) + } + + return blockHash, nil +} + +func (srel *scriptedRelayEntryLogs) RelayEntryRequests( + startBlock, endBlock uint64, +) ([]*abi.RandomBeaconRelayEntryRequested, error) { + srel.reorgAt("requests") + srel.reads = append( + srel.reads, + relayEntryLogRead{"requests", startBlock, endBlock, nil}, + ) + + return srel.view().requests, nil +} + +func (srel *scriptedRelayEntryLogs) RelayEntrySubmissions( + startBlock, endBlock uint64, + requestID *big.Int, +) ([]*abi.RandomBeaconRelayEntrySubmitted, error) { + srel.reorgAt("submissions") + srel.reads = append( + srel.reads, + relayEntryLogRead{"submissions", startBlock, endBlock, requestID}, + ) + + return srel.view().submissions, nil +} + +func (srel *scriptedRelayEntryLogs) RelayEntryTimeouts( + startBlock, endBlock uint64, + requestID *big.Int, +) ([]*abi.RandomBeaconRelayEntryTimedOut, error) { + srel.reorgAt("timeouts") + srel.reads = append( + srel.reads, + relayEntryLogRead{"timeouts", startBlock, endBlock, requestID}, + ) + srel.timeoutsRead = true + + return srel.view().timeouts, nil +} + +// TestResolveRelayEntryTimeoutSettlement asserts the three reads a relay entry +// timeout settlement is composed from answer one view of the chain. +// +// The request, the entry submissions and the terminations are three separate +// backend calls, and a reorg between any two of them can hand back a request +// from one branch and the same request ID's termination from another. That +// pairing was never held by any canonical view of the chain, and a penalty +// claimed on it would be a penalty for something that did not happen. So each +// seam a branch can move at is exercised here, along with the readings that +// must survive one: a log the view has abandoned closes nothing, and a delivery +// whose block cannot be checked stops the claim rather than being dropped from +// it. +func TestResolveRelayEntryTimeoutSettlement(t *testing.T) { + const contractAddress = "0x1111111111111111111111111111111111111111" + const requestBlock = uint64(1_000) + const submissionBlock = uint64(1_050) + const timeoutBlock = uint64(1_100) + const head = uint64(2_000) + const groupOnFirstBranch = uint64(4) + const groupOnSecondBranch = uint64(9) + + requestID := big.NewInt(77) + previousEntry := []byte("previous-entry-of-the-reported-request") + + // Both branches carry the same request ID at the same heights. That is what + // makes a cross-branch composition look coherent to everything except the + // branch check itself. + branchHash := func(branch string, blockNumber uint64) common.Hash { + return common.HexToHash(fmt.Sprintf("0x%s%d", branch, blockNumber)) + } + branchHashes := func(branch string) map[uint64]common.Hash { + return map[uint64]common.Hash{ + requestBlock: branchHash(branch, requestBlock), + submissionBlock: branchHash(branch, submissionBlock), + timeoutBlock: branchHash(branch, timeoutBlock), + head: branchHash(branch, head), + } + } + request := func(branch string) *abi.RandomBeaconRelayEntryRequested { + return &abi.RandomBeaconRelayEntryRequested{ + RequestId: new(big.Int).Set(requestID), + GroupId: 1, + PreviousEntry: previousEntry, + Raw: types.Log{ + BlockNumber: requestBlock, + BlockHash: branchHash(branch, requestBlock), + }, + } + } + submission := func(branch string) *abi.RandomBeaconRelayEntrySubmitted { + return &abi.RandomBeaconRelayEntrySubmitted{ + RequestId: new(big.Int).Set(requestID), + Entry: []byte("the group's answer"), + Raw: types.Log{ + BlockNumber: submissionBlock, + BlockHash: branchHash(branch, submissionBlock), + }, + } + } + timeout := func( + branch string, + groupID uint64, + ) *abi.RandomBeaconRelayEntryTimedOut { + return &abi.RandomBeaconRelayEntryTimedOut{ + RequestId: new(big.Int).Set(requestID), + TerminatedGroupId: groupID, + Raw: types.Log{ + BlockNumber: timeoutBlock, + BlockHash: branchHash(branch, timeoutBlock), + }, + } + } + + // The branch the request was made on, terminated by an accepted report. + firstBranch := func() relayEntryChainView { + return relayEntryChainView{ + head: head, + blockHashes: branchHashes("a"), + requests: []*abi.RandomBeaconRelayEntryRequested{ + request("a"), + }, + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{ + timeout("a", groupOnFirstBranch), + }, + } + } + // The branch that replaces it: the same request, terminated against another + // group. Composed with the first branch's request it reads as a settlement. + secondBranch := func() relayEntryChainView { + return relayEntryChainView{ + head: head, + blockHashes: branchHashes("b"), + requests: []*abi.RandomBeaconRelayEntryRequested{ + request("b"), + }, + timeouts: []*abi.RandomBeaconRelayEntryTimedOut{ + timeout("b", groupOnSecondBranch), + }, + } + } + + tests := map[string]struct { + logs *scriptedRelayEntryLogs + expectSettlement bool + expectError bool + expectedGroupID uint64 + expectedReads []string + }{ + "one view holds the request and its termination": { + logs: &scriptedRelayEntryLogs{ + before: firstBranch(), + }, + expectSettlement: true, + expectedGroupID: groupOnFirstBranch, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + // The reorg lands between the request read and the termination read, so + // the composition would pair a request this view no longer holds with a + // termination the request read never saw. + "the view moves between the request and the termination reads": { + logs: &scriptedRelayEntryLogs{ + before: firstBranch(), + after: secondBranch(), + reorgBefore: "timeouts", + }, + expectError: true, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + "the view moves between the request and the delivery reads": { + logs: &scriptedRelayEntryLogs{ + before: firstBranch(), + after: func() relayEntryChainView { + view := secondBranch() + view.submissions = []*abi.RandomBeaconRelayEntrySubmitted{ + submission("b"), + } + return view + }(), + reorgBefore: "submissions", + }, + expectError: true, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + // Every read answered the first branch, but the branch was gone by the + // time they were composed. The settlement would be a reading of a chain + // this node no longer follows. + "the view moves once the reads are done": { + logs: &scriptedRelayEntryLogs{ + before: firstBranch(), + after: secondBranch(), + reorgBefore: "confirm", + }, + expectError: true, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + // A stable view that hands back a termination mined on a branch it has + // abandoned. The report was not accepted on the chain this node + // follows, so it closes nothing. + "a termination from an abandoned branch": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + view.timeouts = []*abi.RandomBeaconRelayEntryTimedOut{ + timeout("b", groupOnSecondBranch), + } + return &scriptedRelayEntryLogs{before: view} + }(), + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + // The mirror case, and the one that must not be over-corrected: a + // delivery the view has abandoned answers nothing, so the termination + // this view does hold still stands. + "a delivery from an abandoned branch": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + view.submissions = []*abi.RandomBeaconRelayEntrySubmitted{ + submission("b"), + } + return &scriptedRelayEntryLogs{before: view} + }(), + expectSettlement: true, + expectedGroupID: groupOnFirstBranch, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + "a delivery this view holds": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + view.submissions = []*abi.RandomBeaconRelayEntrySubmitted{ + submission("a"), + } + return &scriptedRelayEntryLogs{before: view} + }(), + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + // The request itself is the log the view abandoned. There is no request + // to terminate, so the reads stop before the termination is weighed. + "the request is from an abandoned branch": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + view.requests = []*abi.RandomBeaconRelayEntryRequested{ + request("b"), + } + return &scriptedRelayEntryLogs{before: view} + }(), + expectedReads: []string{"requests"}, + }, + // A view behind the request cannot hold the request, let alone its + // termination. Nothing is read and nothing is claimed. + "the head has not reached the request": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + view.head = requestBlock - 1 + view.blockHashes[requestBlock-1] = branchHash("a", requestBlock-1) + return &scriptedRelayEntryLogs{before: view} + }(), + expectedReads: nil, + }, + // A log past the bound the read was given is a backend answering a + // different question than the one asked, not a later block to accept. + "a termination past the block the read was bounded by": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + beyond := timeout("a", groupOnFirstBranch) + beyond.Raw.BlockNumber = head + 1 + view.timeouts = []*abi.RandomBeaconRelayEntryTimedOut{beyond} + return &scriptedRelayEntryLogs{before: view} + }(), + expectError: true, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + // The delivery cannot be placed on a branch, so whether the group + // answered is unknown. Dropping it would turn an unreadable block into + // a penalty. + "the block a delivery was mined in cannot be read": { + logs: func() *scriptedRelayEntryLogs { + view := firstBranch() + view.submissions = []*abi.RandomBeaconRelayEntrySubmitted{ + submission("a"), + } + return &scriptedRelayEntryLogs{ + before: view, + blockHashErrors: map[uint64]error{ + submissionBlock: fmt.Errorf("no such block"), + }, + } + }(), + expectError: true, + expectedReads: []string{"requests", "submissions", "timeouts"}, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + logs := test.logs + + settlement, err := resolveRelayEntryTimeoutSettlement( + logs, + contractAddress, + requestBlock, + previousEntry, + ) + + var readNames []string + for _, read := range logs.reads { + readNames = append(readNames, read.name) + + if read.startBlock != requestBlock { + t.Errorf( + "the %s read starts at block [%d], not at the request "+ + "block [%d]", + read.name, + read.startBlock, + requestBlock, + ) + } + + // The request read is pinned to the one block the request was + // made in; the two that follow it are bounded by the head the + // view was pinned to, so all three answer one snapshot. + expectedEnd := logs.pinnedHead + if read.name == "requests" { + expectedEnd = requestBlock + } + if read.endBlock != expectedEnd { + t.Errorf( + "the %s read is bounded by block [%d], not by the "+ + "pinned block [%d]", + read.name, + read.endBlock, + expectedEnd, + ) + } + + if read.requestID != nil && read.requestID.Cmp(requestID) != 0 { + t.Errorf( + "the %s read filters on request [%s], not on the "+ + "request that was found [%s]", + read.name, + read.requestID, + requestID, + ) + } + } + if !reflect.DeepEqual(readNames, test.expectedReads) { + t.Errorf( + "unexpected reads\nexpected: %v\nactual: %v", + test.expectedReads, + readNames, + ) + } + + if test.expectError { + if err == nil { + t.Fatal( + "expected a reading that does not answer one view of " + + "the chain to be refused", + ) + } + if settlement != nil { + t.Error("a refused reading must claim no settlement") + } + return + } + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + if !test.expectSettlement { + if settlement != nil { + t.Fatalf( + "expected no penalty to be claimed, got group [%d]", + settlement.TerminatedGroupID, + ) + } + return + } + if settlement == nil { + t.Fatal("expected an accepted report to be claimed as settled") + } + if settlement.TerminatedGroupID != test.expectedGroupID { + t.Errorf( + "unexpected terminated group\nexpected: [%d]\nactual: [%d]", + test.expectedGroupID, + settlement.TerminatedGroupID, + ) + } + if settlement.RequestBlockNumber != requestBlock { + t.Errorf( + "unexpected request block\nexpected: [%d]\nactual: [%d]", + requestBlock, + settlement.RequestBlockNumber, + ) + } + if !bytes.Equal(settlement.RequestPreviousEntry, previousEntry) { + t.Error("the settlement names another request's previous entry") + } + }) + } +} diff --git a/pkg/chain/ethereum/ethereum.go b/pkg/chain/ethereum/ethereum.go index d0f9657cf0..d358f42abc 100644 --- a/pkg/chain/ethereum/ethereum.go +++ b/pkg/chain/ethereum/ethereum.go @@ -495,6 +495,20 @@ func (bc *baseChain) AverageBlockTime() time.Duration { return 12 * time.Second } +// ChainID is the chain id the connected Ethereum endpoint reported when this +// handle was constructed, checked there against the configured network. +// +// It is the chain this node actually reached rather than the one an operator +// or a deployment record claims it reached, which is what makes it usable as +// evidence: a cutover block is only meaningful together with the chain it +// counts on, and a release decision taken over a record whose chain identity +// was copied from its own inputs has verified nothing about where the blocks +// came from. A copy is returned because the value is shared with every +// transaction this handle submits. +func (bc *baseChain) ChainID() *big.Int { + return new(big.Int).Set(bc.chainID) +} + // wrapClientAddons wraps the client instance with add-ons like logging, rate // limiting and so on. func wrapClientAddons( diff --git a/pkg/chain/ethereum/tbtc.go b/pkg/chain/ethereum/tbtc.go index 779e184dc0..83eafb0a11 100644 --- a/pkg/chain/ethereum/tbtc.go +++ b/pkg/chain/ethereum/tbtc.go @@ -1320,23 +1320,11 @@ func (tc *TbtcChain) PastRedemptionRequestedEvents( convertedEvents := make([]*tbtc.RedemptionRequestedEvent, 0) for _, event := range events { - redeemerOutputScript, err := bitcoin.NewScriptFromVarLenData( - event.RedeemerOutputScript, - ) + convertedEvent, err := convertRedemptionRequestedEvent(event) if err != nil { return nil, err } - convertedEvent := &tbtc.RedemptionRequestedEvent{ - WalletPublicKeyHash: event.WalletPubKeyHash, - RedeemerOutputScript: redeemerOutputScript, - Redeemer: chain.Address(event.Redeemer.Hex()), - RequestedAmount: event.RequestedAmount, - TreasuryFee: event.TreasuryFee, - TxMaxFee: event.TreasuryFee, - BlockNumber: event.Raw.BlockNumber, - } - convertedEvents = append(convertedEvents, convertedEvent) } @@ -1350,6 +1338,36 @@ func (tc *TbtcChain) PastRedemptionRequestedEvents( return convertedEvents, err } +// convertRedemptionRequestedEvent converts a raw on-chain RedemptionRequested +// event into the internal tbtc.RedemptionRequestedEvent. Extracted from +// PastRedemptionRequestedEvents so the field mapping is unit-testable without +// a simulated chain backend. +func convertRedemptionRequestedEvent( + event *tbtcabi.BridgeRedemptionRequested, +) (*tbtc.RedemptionRequestedEvent, error) { + redeemerOutputScript, err := bitcoin.NewScriptFromVarLenData( + event.RedeemerOutputScript, + ) + if err != nil { + return nil, err + } + + return &tbtc.RedemptionRequestedEvent{ + WalletPublicKeyHash: event.WalletPubKeyHash, + RedeemerOutputScript: redeemerOutputScript, + Redeemer: chain.Address(event.Redeemer.Hex()), + RequestedAmount: event.RequestedAmount, + TreasuryFee: event.TreasuryFee, + // Previously mapped from event.TreasuryFee by mistake (a copy-paste + // defect). TxMaxFee is a distinct fee bound and must come from the + // event's TxMaxFee field. Latent at the time of the fix (no consumer + // read the event-path TxMaxFee), corrected to prevent a future + // fund-relevant fee-bound bug. + TxMaxFee: event.TxMaxFee, + BlockNumber: event.Raw.BlockNumber, + }, nil +} + func (tc *TbtcChain) GetDepositRequest( fundingTxHash bitcoin.Hash, fundingOutputIndex uint32, diff --git a/pkg/chain/ethereum/tbtc_redemption_event_property_test.go b/pkg/chain/ethereum/tbtc_redemption_event_property_test.go new file mode 100644 index 0000000000..78a8923b8a --- /dev/null +++ b/pkg/chain/ethereum/tbtc_redemption_event_property_test.go @@ -0,0 +1,91 @@ +package ethereum + +import ( + "bytes" + "testing" + + "github.com/ethereum/go-ethereum/common" + "pgregory.net/rapid" + + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/chain" + tbtcabi "github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen/abi" +) + +// Property-based (adapter) coverage for security-audit finding F-014: the +// RedemptionRequested event conversion must map each scalar field from its own +// source field. The original defect mapped TxMaxFee from event.TreasuryFee. +// +// TestConvertRedemptionRequestedEvent (the table test) pins one distinct-fee +// case; this property generalizes it: for arbitrary field values the converted +// event must reproduce every source field exactly. rapid will readily generate +// inputs where TreasuryFee != TxMaxFee, so any reintroduced cross-wiring of the +// two (or of any other scalar) is caught. +func TestRapidConvertRedemptionRequestedEventFieldMapping(t *testing.T) { + rapid.Check(t, func(t *rapid.T) { + requestedAmount := rapid.Uint64().Draw(t, "requestedAmount") + treasuryFee := rapid.Uint64().Draw(t, "treasuryFee") + txMaxFee := rapid.Uint64().Draw(t, "txMaxFee") + blockNumber := rapid.Uint64().Draw(t, "blockNumber") + redeemerBytes := rapid.SliceOfN(rapid.Byte(), 20, 20).Draw(t, "redeemer") + + // Draw raw script bytes and wrap them in the var-len encoding the + // converter parses, so the script round-trips through + // NewScriptFromVarLenData rather than failing on malformed input. + scriptBytes := rapid.SliceOfN(rapid.Byte(), 0, 64).Draw(t, "script") + varLenScript, err := bitcoin.Script(scriptBytes).ToVarLenData() + if err != nil { + t.Fatalf("cannot var-len encode generated script: %v", err) + } + + event := &tbtcabi.BridgeRedemptionRequested{ + WalletPubKeyHash: [20]byte{0x01, 0x02, 0x03}, + RedeemerOutputScript: varLenScript, + Redeemer: common.BytesToAddress(redeemerBytes), + RequestedAmount: requestedAmount, + TreasuryFee: treasuryFee, + TxMaxFee: txMaxFee, + } + event.Raw.BlockNumber = blockNumber + + got, err := convertRedemptionRequestedEvent(event) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + + if got.RequestedAmount != requestedAmount { + t.Fatalf("RequestedAmount: got %d, want %d", got.RequestedAmount, requestedAmount) + } + if got.TreasuryFee != treasuryFee { + t.Fatalf("TreasuryFee: got %d, want %d", got.TreasuryFee, treasuryFee) + } + // The F-014 invariant: TxMaxFee must come from event.TxMaxFee, never + // from event.TreasuryFee. + if got.TxMaxFee != txMaxFee { + t.Fatalf( + "TxMaxFee: got %d, want %d (must map from event.TxMaxFee, not TreasuryFee=%d)", + got.TxMaxFee, txMaxFee, treasuryFee, + ) + } + if got.BlockNumber != blockNumber { + t.Fatalf("BlockNumber: got %d, want %d", got.BlockNumber, blockNumber) + } + if got.WalletPublicKeyHash != event.WalletPubKeyHash { + t.Fatalf("WalletPublicKeyHash mismatch") + } + if got.Redeemer != chain.Address(event.Redeemer.Hex()) { + t.Fatalf( + "Redeemer: got %s, want %s", + got.Redeemer, event.Redeemer.Hex(), + ) + } + // The converted script must be the decoded payload of the var-len + // data, i.e. exactly the raw bytes that were wrapped above. + if !bytes.Equal(got.RedeemerOutputScript, scriptBytes) { + t.Fatalf( + "RedeemerOutputScript: got %x, want %x", + got.RedeemerOutputScript, scriptBytes, + ) + } + }) +} diff --git a/pkg/chain/ethereum/tbtc_redemption_event_test.go b/pkg/chain/ethereum/tbtc_redemption_event_test.go new file mode 100644 index 0000000000..e4917eea98 --- /dev/null +++ b/pkg/chain/ethereum/tbtc_redemption_event_test.go @@ -0,0 +1,59 @@ +package ethereum + +import ( + "testing" + + "github.com/ethereum/go-ethereum/common" + + tbtcabi "github.com/keep-network/keep-core/pkg/chain/ethereum/tbtc/gen/abi" +) + +// TestConvertRedemptionRequestedEvent is regression coverage for the +// security-audit finding F-014: the RedemptionRequested event conversion +// mapped TxMaxFee from event.TreasuryFee (a copy-paste defect). TreasuryFee +// and TxMaxFee are distinct fee bounds and must each map from their own +// source field. +// +// The test uses deliberately distinct TreasuryFee and TxMaxFee values so the +// previous (buggy) mapping returns the wrong TxMaxFee and the test fails +// against the unpatched code. +func TestConvertRedemptionRequestedEvent(t *testing.T) { + event := &tbtcabi.BridgeRedemptionRequested{ + WalletPubKeyHash: [20]byte{0x01, 0x02, 0x03}, + RedeemerOutputScript: []byte{0x01, 0xaa}, // var-len: 1-byte script + Redeemer: common.HexToAddress("0x1111111111111111111111111111111111111111"), + RequestedAmount: 1000, + TreasuryFee: 100, + TxMaxFee: 7, // intentionally distinct from TreasuryFee + } + + converted, err := convertRedemptionRequestedEvent(event) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + if converted.TreasuryFee != event.TreasuryFee { + t.Errorf( + "wrong TreasuryFee\nexpected: %v\nactual: %v", + event.TreasuryFee, + converted.TreasuryFee, + ) + } + + if converted.TxMaxFee != event.TxMaxFee { + t.Errorf( + "wrong TxMaxFee (must map from event.TxMaxFee, not event.TreasuryFee)"+ + "\nexpected: %v\nactual: %v", + event.TxMaxFee, + converted.TxMaxFee, + ) + } + + if converted.RequestedAmount != event.RequestedAmount { + t.Errorf( + "wrong RequestedAmount\nexpected: %v\nactual: %v", + event.RequestedAmount, + converted.RequestedAmount, + ) + } +} diff --git a/pkg/chain/local_v1/local.go b/pkg/chain/local_v1/local.go index 6715634ab2..cbc76e8c53 100644 --- a/pkg/chain/local_v1/local.go +++ b/pkg/chain/local_v1/local.go @@ -44,6 +44,15 @@ type localChain struct { dkgStartedHandlers map[int]func(submission *event.DKGStarted) resultSubmissionHandlers map[int]func(submission *event.DKGResultSubmission) + // resultSubmissionRegisteredSignal, when non-nil, receives an empty value + // after each DKG result submission handler is installed via + // OnDKGResultSubmitted. It is test-only instrumentation that lets a test + // deterministically wait until a member has installed its result submission + // subscription before triggering a competing submission, eliminating the + // race between a result submission and a concurrent subscription setup. + // Access is guarded by handlerMutex. + resultSubmissionRegisteredSignal chan<- struct{} + simulatedHeight uint64 blockCounter chain.BlockCounter @@ -80,6 +89,9 @@ func (c *localChain) SubmitRelayEntry(newEntry []byte) error { } c.handlerMutex.Lock() + // Record the last submitted entry under the same lock that guards it in + // GetLastRelayEntry so concurrent submissions/reads do not race. + c.lastSubmittedRelayEntry = newEntry for _, handler := range c.relayEntryHandlers { go func(handler func(entry *event.RelayEntrySubmitted), entry *event.RelayEntrySubmitted) { handler(entry) @@ -87,8 +99,6 @@ func (c *localChain) SubmitRelayEntry(newEntry []byte) error { } c.handlerMutex.Unlock() - c.lastSubmittedRelayEntry = newEntry - return nil } @@ -110,6 +120,9 @@ func (c *localChain) OnRelayEntrySubmitted( } func (c *localChain) GetLastRelayEntry() []byte { + c.handlerMutex.Lock() + defer c.handlerMutex.Unlock() + return c.lastSubmittedRelayEntry } @@ -236,6 +249,9 @@ func (c *localChain) IsStaleGroup(groupPublicKey []byte) (bool, error) { } func (c *localChain) IsGroupRegistered(groupPublicKey []byte) (bool, error) { + c.handlerMutex.Lock() + defer c.handlerMutex.Unlock() + for _, group := range c.groups { if bytes.Equal(group.groupPublicKey, groupPublicKey) { return true, nil @@ -274,9 +290,6 @@ func (c *localChain) SubmitDKGResult( groupPublicKey: resultToPublish.GroupPublicKey, registrationBlockHeight: currentBlock, } - c.groups = append(c.groups, myGroup) - c.lastSubmittedDKGResult = resultToPublish - c.lastSubmittedDKGResultSignatures = signatures groupRegistrationEvent := &event.GroupRegistration{ GroupPublicKey: resultToPublish.GroupPublicKey[:], @@ -284,6 +297,14 @@ func (c *localChain) SubmitDKGResult( } c.handlerMutex.Lock() + // Register the group and record the last submitted result under the same + // lock that guards these fields in IsGroupRegistered, IsStaleGroup, and + // GetLastDKGResult. Concurrent DKG result publications by multiple members + // would otherwise race on the groups slice. + c.groups = append(c.groups, myGroup) + c.lastSubmittedDKGResult = resultToPublish + c.lastSubmittedDKGResultSignatures = signatures + for _, handler := range c.resultSubmissionHandlers { go func(handler func(*event.DKGResultSubmission), dkgResultPublication *event.DKGResultSubmission) { handler(dkgResultPublicationEvent) @@ -326,6 +347,16 @@ func (c *localChain) OnDKGResultSubmitted( handlerID := GenerateHandlerID() c.resultSubmissionHandlers[handlerID] = handler + // Notify any test synchronization listener that a result submission handler + // has been installed. The send is non-blocking so chain operation is never + // blocked; a sufficiently buffered listener channel guarantees delivery. + if c.resultSubmissionRegisteredSignal != nil { + select { + case c.resultSubmissionRegisteredSignal <- struct{}{}: + default: + } + } + return subscription.NewEventSubscription(func() { c.handlerMutex.Lock() defer c.handlerMutex.Unlock() @@ -334,10 +365,35 @@ func (c *localChain) OnDKGResultSubmitted( }) } +// SetResultSubmissionRegisteredSignal installs a test-only signal channel that +// receives an empty value after each DKG result submission handler is registered +// via OnDKGResultSubmitted. It lets tests synchronize on subscription +// installation without relying on timing, for example to guarantee a member has +// installed its subscription before a competing member submits a result. The +// provided channel should be buffered so signals are never dropped; pass nil to +// disable notifications. +func (c *localChain) SetResultSubmissionRegisteredSignal(signal chan<- struct{}) { + c.handlerMutex.Lock() + defer c.handlerMutex.Unlock() + + c.resultSubmissionRegisteredSignal = signal +} + func (c *localChain) GetLastDKGResult() ( *beaconchain.DKGResult, map[beaconchain.GroupMemberIndex][]byte, ) { + c.handlerMutex.Lock() + defer c.handlerMutex.Unlock() + + // Read these fields under the same lock SubmitDKGResult holds while writing + // them. The deferred unlock runs only after the return values are evaluated, + // so the field reads happen inside the critical section and establish the + // happens-before edge the race detector requires. SubmitDKGResult only ever + // reassigns lastSubmittedDKGResult and lastSubmittedDKGResultSignatures (it + // never mutates the pointed-to result or the signatures map in place), so the + // references returned here remain a stable snapshot after the lock is + // released. return c.lastSubmittedDKGResult, c.lastSubmittedDKGResultSignatures } @@ -354,24 +410,49 @@ func (c *localChain) ReportRelayEntryTimeout() error { return nil } +// errNoRelayRequestState reports that this chain keeps no relay request +// lifecycle. It is returned rather than panicked because the relay entry +// timeout monitor reconciles a filed report against these reads on every run; +// an error tells the monitor it cannot confirm the report and leaves the +// penalty unclaimed, which is the honest reading. Answering "no request is in +// progress" would be vacuously true here and would let a local run claim a +// penalty no chain ever confirmed. +var errNoRelayRequestState = fmt.Errorf( + "the local chain keeps no relay request state", +) + func (c *localChain) IsEntryInProgress() (bool, error) { - panic("not implemented") + return false, errNoRelayRequestState } func (c *localChain) CurrentRequestStartBlock() (*big.Int, error) { - panic("not implemented") + return nil, errNoRelayRequestState } func (c *localChain) CurrentRequestPreviousEntry() ([]byte, error) { - panic("not implemented") + return nil, errNoRelayRequestState } func (c *localChain) CurrentRequestGroupPublicKey() ([]byte, error) { panic("not implemented") } +func (c *localChain) RelayEntryTimeoutSettlement( + uint64, + []byte, +) (*event.RelayEntryTimeoutSettlement, error) { + return nil, errNoRelayRequestState +} + func (c *localChain) GetRelayEntryTimeoutReports() []uint64 { - return c.relayEntryTimeoutReports + c.relayEntryTimeoutReportsMutex.Lock() + defer c.relayEntryTimeoutReportsMutex.Unlock() + + // Return a snapshot copy so callers can read the reports without racing a + // concurrent ReportRelayEntryTimeout append. + reports := make([]uint64, len(c.relayEntryTimeoutReports)) + copy(reports, c.relayEntryTimeoutReports) + return reports } // CalculateDKGResultHash calculates a 256-bit hash of the DKG result. diff --git a/pkg/chain/local_v1/local_test.go b/pkg/chain/local_v1/local_test.go index f061f9a6e8..0228cf840c 100644 --- a/pkg/chain/local_v1/local_test.go +++ b/pkg/chain/local_v1/local_test.go @@ -5,6 +5,7 @@ import ( "fmt" "math/big" "reflect" + "sync" "testing" "time" @@ -276,12 +277,20 @@ func TestWatchBlocks(t *testing.T) { watcher1ReceivedCount := 0 watcher2ReceivedCount := 0 + // The watcher channels are closed once their context is cancelled, so each + // consumer goroutine exits its range loop then. Closing a done channel on + // exit gives the main goroutine a happens-before edge to read the counters + // without racing the increments. + watcher1Done := make(chan struct{}) + watcher2Done := make(chan struct{}) go func() { + defer close(watcher1Done) for range watcher1 { watcher1ReceivedCount++ } }() go func() { + defer close(watcher2Done) for range watcher2 { watcher2ReceivedCount++ } @@ -292,6 +301,10 @@ func TestWatchBlocks(t *testing.T) { time.Sleep(600 * time.Millisecond) cancel2() + // Wait for both consumers to drain and exit before reading their counters. + <-watcher1Done + <-watcher2Done + if watcher1ReceivedCount != 1 { t.Errorf("watcher 1 should receive [1] block, has [%v]", watcher1ReceivedCount) } @@ -314,13 +327,19 @@ func TestWatchBlocksNonBlocking(t *testing.T) { watcher := blockCounter.WatchBlocks(ctx) // does read blocks var receivedCount uint64 + // The watcher channel is closed once the context is cancelled, so the + // consumer goroutine exits its range loop then. Closing done on exit gives + // the main goroutine a happens-before edge to read receivedCount. + done := make(chan struct{}) go func() { + defer close(done) for range watcher { receivedCount++ } }() <-ctx.Done() + <-done if receivedCount != 2 { t.Errorf("watcher should receive [2] blocks, has [%v]", receivedCount) @@ -588,6 +607,150 @@ func TestLocalSubmitDKGResultWithSignatures(t *testing.T) { } } +// TestGetLastDKGResultConcurrentAccess pins the synchronization contract between +// SubmitDKGResult, which reassigns lastSubmittedDKGResult and +// lastSubmittedDKGResultSignatures under handlerMutex, and GetLastDKGResult, +// which must read those same fields under the same lock. A lock-free getter is a +// data race against the concurrent writers even though the functional assertions +// below still pass, so this test is only meaningful under `go test -race`. +// +// The goroutines are coordinated so the race is exercised reliably rather than +// by luck of the scheduler: +// - every reader enters a read loop and reports readiness only after it has +// executed GetLastDKGResult at least once; +// - the writers stay blocked until all readers have reported readiness, so no +// field reassignment can begin before the readers are already reading; +// - the readers keep reading until writesComplete is closed, which happens +// strictly after every writer has returned. +// +// Consequently each reader keeps looping over GetLastDKGResult across the whole +// window in which the writers reassign the shared fields, and those reads carry +// no happens-before edge ordering them against the writes — the condition the +// race detector needs to flag a lock-free getter. Keeping the readers live and +// reading for the entire write window (rather than doing a fixed number of reads +// that could all land before the first write) also keeps the racy read and write +// temporally adjacent, which is what makes -race report the race deterministically +// instead of missing it once the writer's shadow state is evicted. The +// coordination uses only WaitGroups and channel closes: no sleeps, timeouts, or +// fixed read counts. +func TestGetLastDKGResultConcurrentAccess(t *testing.T) { + groupSize := 10 + honestThreshold := 4 + + chainHandle := Connect(groupSize, honestThreshold) + + const submitters = 8 + const readers = 8 + + // A threshold-satisfying signature set reused by every submitter; the map is + // never mutated in place, matching SubmitDKGResult's reassign-only contract. + signatures := map[beaconchain.GroupMemberIndex][]byte{ + 1: {101}, + 2: {102}, + 3: {103}, + 4: {104}, + } + + // writersStart gates the writers. It is closed only after readersReady + // reports that every reader is already inside its read loop, so no writer can + // reassign lastSubmittedDKGResult / lastSubmittedDKGResultSignatures until the + // readers are actively reading those same fields. + writersStart := make(chan struct{}) + // writesComplete is closed only after every writer has returned from + // SubmitDKGResult. Readers keep calling GetLastDKGResult until they observe it + // closed, so they stay live for the entire window in which the writers + // reassign the shared fields. This removes the false-negative window of a + // fixed read count, where every read could complete before the first write + // landed and the race detector would observe no overlapping accesses. + writesComplete := make(chan struct{}) + + // readersReady lets the main goroutine wait until every reader has entered + // its read loop (and taken at least one read) before releasing the writers. + // readersWg tracks reader shutdown; writersWg tracks writer completion. + var readersReady sync.WaitGroup + var readersWg sync.WaitGroup + var writersWg sync.WaitGroup + readersReady.Add(readers) + readersWg.Add(readers) + writersWg.Add(submitters) + + for i := 0; i < readers; i++ { + go func() { + defer readersWg.Done() + reported := false + for { + // Read the shared fields, then touch the returned snapshot so + // the read cannot be optimized away. + result, sigs := chainHandle.GetLastDKGResult() + if result != nil { + _ = len(result.GroupPublicKey) + _ = len(sigs) + } + + // Report readiness once, after the first read has executed, so + // the main goroutine only releases the writers once every reader + // is provably already reading GetLastDKGResult. + if !reported { + reported = true + readersReady.Done() + } + + // Keep reading until every writer has returned. This keeps the + // reader live for the whole write phase with no sleep, timeout, + // or fixed read count. + select { + case <-writesComplete: + return + default: + } + } + }() + } + + for i := 0; i < submitters; i++ { + go func(index int) { + defer writersWg.Done() + <-writersStart + memberIndex := beaconchain.GroupMemberIndex(uint8(index%groupSize) + 1) + result := &beaconchain.DKGResult{ + GroupPublicKey: []byte{byte(index)}, + } + if err := chainHandle.SubmitDKGResult( + memberIndex, + result, + signatures, + ); err != nil { + t.Errorf("unexpected error submitting DKG result: [%v]", err) + } + }(i) + } + + // Barrier: wait until every reader is inside its read loop, then release the + // writers. Because the readers are already reading and only stop once + // writesComplete is closed (strictly after every writer returns), each reader + // keeps calling GetLastDKGResult throughout the writers' reassignment window, + // with no happens-before edge ordering those reads against the writes. + readersReady.Wait() + close(writersStart) + writersWg.Wait() + close(writesComplete) + readersWg.Wait() + + // After every submission completes, the getter must return one of the + // submitted results together with its signatures. + result, sigs := chainHandle.GetLastDKGResult() + if result == nil { + t.Fatal("expected a last DKG result after concurrent submissions") + } + if len(sigs) != len(signatures) { + t.Fatalf( + "unexpected signatures count\nexpected: [%v]\nactual: [%v]", + len(signatures), + len(sigs), + ) + } +} + func TestCalculateDKGResultHash(t *testing.T) { localChain := &localChain{} diff --git a/pkg/clientinfo/clientinfo_test.go b/pkg/clientinfo/clientinfo_test.go new file mode 100644 index 0000000000..05b4562edd --- /dev/null +++ b/pkg/clientinfo/clientinfo_test.go @@ -0,0 +1,60 @@ +package clientinfo + +import ( + "context" + "testing" + + keepclientinfo "github.com/keep-network/keep-common/pkg/clientinfo" +) + +func TestInitialize_PortZeroDisablesServer(t *testing.T) { + registry, isConfigured := Initialize(context.Background(), 0) + + if isConfigured { + t.Fatal("expected port 0 to disable the client info server") + } + + if registry != nil { + t.Fatal("expected no registry when client info server is disabled") + } +} + +func TestInitialize_NonZeroPortEnablesServer(t *testing.T) { + registry, isConfigured := Initialize(context.Background(), 9601) + + if !isConfigured { + t.Fatal("expected non-zero port to enable the client info server") + } + + if registry == nil { + t.Fatal("expected a registry when client info server is enabled") + } +} + +// TestRegisterMetricClientInfo_RegistersArtifactIdentity proves the identity +// metric is registered under the exact exported name: a subsequent +// registration attempt for client_info must be refused as a duplicate. The +// version, revision, and protocol-epoch labels travel in that single +// registration. +func TestRegisterMetricClientInfo_RegistersArtifactIdentity(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + registry := &Registry{keepclientinfo.NewRegistry(), ctx} + + registry.RegisterMetricClientInfo( + "v2.2.0", + "33808cba", + "security_v2_cutover", + ) + + if _, err := registry.NewMetricInfo( + ClientInfoMetricName, + []keepclientinfo.Label{keepclientinfo.NewLabel("version", "other")}, + ); err == nil { + t.Fatal( + "expected the client_info metric to already be registered with " + + "the artifact identity labels", + ) + } +} diff --git a/pkg/clientinfo/cutover_metrics_test.go b/pkg/clientinfo/cutover_metrics_test.go new file mode 100644 index 0000000000..8c62c17c64 --- /dev/null +++ b/pkg/clientinfo/cutover_metrics_test.go @@ -0,0 +1,516 @@ +package clientinfo + +import ( + "context" + "fmt" + "os" + "path/filepath" + "reflect" + "regexp" + "sort" + "strings" + "testing" + + keepclientinfo "github.com/keep-network/keep-common/pkg/clientinfo" +) + +// TestCutoverMetrics_ExactExportedNames pins the exact metric names exposed on +// the /metrics endpoint. The performance registry prepends the "performance_" +// application prefix (see ObserveApplicationSource), so the exported name is +// "performance_" + the internal constant. This guards against the regression +// where the internal constant itself carried a "performance_" prefix and the +// metric was exposed as performance_performance_* (or, being unregistered, not +// at all). +func TestCutoverMetrics_ExactExportedNames(t *testing.T) { + cases := []struct { + internal string + exported string + }{ + {MetricAnnouncerSessionIDMismatchTotal, "performance_announcer_session_id_mismatch_total"}, + {MetricAnnouncerCrossFormatPeerTotal, "performance_announcer_cross_format_peer_total"}, + {MetricAnnouncerLegacyPeersCurrent, "performance_announcer_legacy_peers_current"}, + {MetricAnnouncerLegacyPeerOldestAgeBlocks, "performance_announcer_legacy_peer_oldest_age_blocks"}, + {MetricAnnouncerLegacyPeerRosterRevision, "performance_announcer_legacy_peer_roster_revision"}, + {MetricAnnouncerLegacyPeerAdditionsTotal, "performance_announcer_legacy_peer_additions_total"}, + {MetricAnnouncerLegacyPeerEvictionsTotal, "performance_announcer_legacy_peer_evictions_total"}, + {MetricParticipationTBTCQuarantinePreservationFailuresTotal, "performance_participation_tbtc_quarantine_preservation_failures_total"}, + {MetricParticipationBeaconQuarantinePreservationFailuresTotal, "performance_participation_beacon_quarantine_preservation_failures_total"}, + {MetricParticipationTBTCQuarantineIncompleteOutputs, "performance_participation_tbtc_quarantine_incomplete_outputs"}, + {MetricParticipationBeaconQuarantineIncompleteOutputs, "performance_participation_beacon_quarantine_incomplete_outputs"}, + } + + for _, c := range cases { + got := fmt.Sprintf("performance_%s", c.internal) + if got != c.exported { + t.Errorf( + "internal metric %q exposes as %q, want %q", + c.internal, + got, + c.exported, + ) + } + } +} + +// TestCutoverMetrics_RegisteredAtZeroAndRecordable proves that the seven cutover +// observability metrics are registered at zero by registerAllMetrics (so they +// appear on /metrics before any event) and that they record through the +// production PerformanceMetrics recorder used by the tBTC announcer wiring and +// the node-local cutover roster. +func TestCutoverMetrics_RegisteredAtZeroAndRecordable(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + registry := &Registry{keepclientinfo.NewRegistry(), ctx} + pm := NewPerformanceMetrics(ctx, registry) + + counters := []string{ + MetricAnnouncerSessionIDMismatchTotal, + MetricAnnouncerCrossFormatPeerTotal, + MetricAnnouncerLegacyPeerAdditionsTotal, + MetricAnnouncerLegacyPeerEvictionsTotal, + } + gauges := []string{ + MetricAnnouncerLegacyPeersCurrent, + MetricAnnouncerLegacyPeerOldestAgeBlocks, + MetricAnnouncerLegacyPeerRosterRevision, + } + + for _, name := range counters { + if got := pm.GetCounterValue(name); got != 0 { + t.Errorf("counter %q should be registered at zero, got %v", name, got) + } + } + for _, name := range gauges { + if got := pm.GetGaugeValue(name); got != 0 { + t.Errorf("gauge %q should be registered at zero, got %v", name, got) + } + } + + // Record like the production announcer/roster path does. + pm.IncrementCounter(MetricAnnouncerSessionIDMismatchTotal, 1) + pm.IncrementCounter(MetricAnnouncerSessionIDMismatchTotal, 1) + pm.IncrementCounter(MetricAnnouncerCrossFormatPeerTotal, 1) + pm.SetGauge(MetricAnnouncerLegacyPeersCurrent, 3) + pm.SetGauge(MetricAnnouncerLegacyPeerRosterRevision, 7) + + if got := pm.GetCounterValue(MetricAnnouncerSessionIDMismatchTotal); got != 2 { + t.Errorf("mismatch counter = %v, want 2", got) + } + if got := pm.GetCounterValue(MetricAnnouncerCrossFormatPeerTotal); got != 1 { + t.Errorf("cross-format counter = %v, want 1", got) + } + if got := pm.GetGaugeValue(MetricAnnouncerLegacyPeersCurrent); got != 3 { + t.Errorf("legacy peers gauge = %v, want 3", got) + } + if got := pm.GetGaugeValue(MetricAnnouncerLegacyPeerRosterRevision); got != 7 { + t.Errorf("roster revision gauge = %v, want 7", got) + } +} + +// participationMetricFamily is the observability contract of the cutover gate: +// every series the rehearsal's evidence steps snapshot by name. It is the +// pre-image of the PARTICIPATION_METRICS list the exact-image rehearsal reads +// off a running node, so a series missing from the production registration +// fails here rather than at rehearsal time. +var participationMetricFamily = []string{ + MetricParticipationGateState, + MetricParticipationCurrentBlock, + MetricParticipationCutoverBlock, + MetricParticipationAllowed, + MetricParticipationActiveCeremonies, + MetricParticipationActiveLegacyCeremonies, + MetricParticipationActiveSecurityV2Ceremonies, + MetricParticipationModeLegacyTotal, + MetricParticipationModeSecurityV2Total, + MetricParticipationLegacyCompletionsAfterCutoverTotal, + MetricParticipationRefusalsTotal, + MetricParticipationCommitRefusalsTotal, + MetricParticipationClockErrorsTotal, + MetricParticipationClockAbortsTotal, + MetricParticipationQuiesceTotal, + MetricParticipationQuiesceForcedAbortsTotal, + MetricParticipationTBTCQuarantinePreservationFailuresTotal, + MetricParticipationBeaconQuarantinePreservationFailuresTotal, + MetricParticipationTBTCQuarantineIncompleteOutputs, + MetricParticipationBeaconQuarantineIncompleteOutputs, + MetricParticipationQuarantinedTBTCSigners, +} + +const participationMetricsSectionHeading = "=== Protocol Participation Gate Metrics" + +var participationMetricHeadingPattern = regexp.MustCompile( + "(?m)^==== `performance_(participation_[^`]+)`[[:space:]]*$", +) + +type participationMetricsDocumentationDiff struct { + missing []string + unexpected []string + duplicate []string +} + +// compareParticipationMetricsDocumentation compares an in-memory metrics +// reference with the fixed participation family. Missing metrics are scoped to +// the canonical participation section, while unknown and duplicated +// participation headings are detected across the whole document. This reports +// a known metric moved under another section as missing and an invented metric +// anywhere in the reference as unexpected. +func compareParticipationMetricsDocumentation( + document string, +) (participationMetricsDocumentationDiff, error) { + sectionStart := strings.Index(document, participationMetricsSectionHeading) + if sectionStart < 0 { + return participationMetricsDocumentationDiff{}, fmt.Errorf( + "metrics reference has no %q section", + participationMetricsSectionHeading, + ) + } + + section := document[sectionStart+len(participationMetricsSectionHeading):] + if nextSection := strings.Index(section, "\n=== "); nextSection >= 0 { + section = section[:nextSection] + } + + documentedInSection := make(map[string]int) + for _, match := range participationMetricHeadingPattern.FindAllStringSubmatch( + section, + -1, + ) { + documentedInSection[match[1]]++ + } + + documentedEverywhere := make(map[string]int) + for _, match := range participationMetricHeadingPattern.FindAllStringSubmatch( + document, + -1, + ) { + documentedEverywhere[match[1]]++ + } + + expected := make(map[string]struct{}, len(participationMetricFamily)+1) + for _, metric := range participationMetricFamily { + expected[metric] = struct{}{} + } + expected["participation_refusals__total"] = struct{}{} + + result := participationMetricsDocumentationDiff{} + for metric := range expected { + if documentedInSection[metric] == 0 { + result.missing = append(result.missing, "performance_"+metric) + } + } + + for metric, count := range documentedEverywhere { + if _, ok := expected[metric]; !ok { + result.unexpected = append( + result.unexpected, + "performance_"+metric, + ) + } + if count > 1 { + result.duplicate = append( + result.duplicate, + "performance_"+metric, + ) + } + } + + sort.Strings(result.missing) + sort.Strings(result.unexpected) + sort.Strings(result.duplicate) + + return result, nil +} + +// TestParticipationMetrics_DocumentationMatchesFamily keeps the canonical +// metrics reference and the fixed participation family in lockstep in both +// directions. A code metric with no heading leaves operators without its +// semantics; a participation heading with no code metric promises a series no +// node can publish. +// +// Per-ceremony refusals are generated from a closed roster and intentionally +// share one documentation template, so that template joins the fixed family +// for this comparison. +func TestParticipationMetrics_DocumentationMatchesFamily(t *testing.T) { + documentPath := filepath.Join("..", "..", "docs", "performance-metrics.adoc") + document, err := os.ReadFile(documentPath) + if err != nil { + t.Fatalf("cannot read participation metrics reference: %v", err) + } + + diff, err := compareParticipationMetricsDocumentation(string(document)) + if err != nil { + t.Fatal(err) + } + if len(diff.missing) > 0 { + t.Errorf( + "participation metrics missing from reference: %v", + diff.missing, + ) + } + if len(diff.unexpected) > 0 { + t.Errorf( + "reference documents unknown participation metrics: %v", + diff.unexpected, + ) + } + if len(diff.duplicate) > 0 { + t.Errorf( + "reference duplicates participation metric headings: %v", + diff.duplicate, + ) + } +} + +// TestParticipationMetrics_DocumentationComparisonDetectsDrift proves each +// direction of the documentation comparison is load-bearing. The missing case +// moves a real heading below the next level-three section, exercising the +// section truncation rather than simply deleting text. The unexpected case +// places an invented participation heading in that later section, proving +// headings outside the canonical section are not invisible to drift checks. +func TestParticipationMetrics_DocumentationComparisonDetectsDrift( + t *testing.T, +) { + canonical := append([]string(nil), participationMetricFamily...) + canonical = append( + canonical, + "participation_refusals__total", + ) + sort.Strings(canonical) + + render := func(sectionMetrics, laterMetrics []string) string { + var result strings.Builder + result.WriteString("= Metrics\n\n") + result.WriteString(participationMetricsSectionHeading) + result.WriteString("\n") + for _, metric := range sectionMetrics { + fmt.Fprintf( + &result, + "\n==== `performance_%s`\n*Type*: Gauge\n", + metric, + ) + } + result.WriteString("\n=== Another Metrics Section\n") + for _, metric := range laterMetrics { + fmt.Fprintf( + &result, + "\n==== `performance_%s`\n*Type*: Gauge\n", + metric, + ) + } + return result.String() + } + + movedMetric := canonical[0] + duplicatedMetric := canonical[1] + cases := map[string]struct { + document string + expected participationMetricsDocumentationDiff + }{ + "known heading moved beyond the section": { + document: render(canonical[1:], []string{movedMetric}), + expected: participationMetricsDocumentationDiff{ + missing: []string{"performance_" + movedMetric}, + }, + }, + "unknown heading in another section": { + document: render( + canonical, + []string{"participation_invented_total"}, + ), + expected: participationMetricsDocumentationDiff{ + unexpected: []string{ + "performance_participation_invented_total", + }, + }, + }, + "duplicated heading": { + document: render( + append(canonical, duplicatedMetric), + nil, + ), + expected: participationMetricsDocumentationDiff{ + duplicate: []string{"performance_" + duplicatedMetric}, + }, + }, + } + + for name, test := range cases { + t.Run(name, func(t *testing.T) { + actual, err := compareParticipationMetricsDocumentation( + test.document, + ) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(test.expected, actual) { + t.Errorf( + "unexpected documentation diff\nexpected: %#v\nactual: %#v", + test.expected, + actual, + ) + } + }) + } +} + +// TestParticipationMetrics_RegisteredWithTheExposingRegistry proves every +// participation series is registered with the client-info registry that backs +// /metrics, not merely readable back through the recorder. +// +// The distinction is the whole point. GetGaugeValue and GetCounterValue read +// the recorder's own maps, and SetGauge inserts into those maps for a name it +// was never asked to register — so a series omitted from registerAllMetrics +// reads back correctly through the recorder while being absent from the +// exposition entirely. Membership is asserted by re-registering the exported +// name: the registry refuses a name it already holds, which is exactly the set +// the exposition enumerates. +func TestParticipationMetrics_RegisteredWithTheExposingRegistry(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + registry := &Registry{keepclientinfo.NewRegistry(), ctx} + NewPerformanceMetrics(ctx, registry) + + for _, name := range participationMetricFamily { + exported := fmt.Sprintf("performance_%s", name) + if _, err := registry.NewMetricGauge(exported); err == nil { + t.Errorf( + "metric %q is not registered with the registry, so it is "+ + "absent from /metrics until something records it", + exported, + ) + } + } + + // A name nothing registered must register cleanly, otherwise the loop + // above would pass against any registry that refuses everything. + if _, err := registry.NewMetricGauge( + "performance_participation_absent_control", + ); err != nil { + t.Fatalf( + "an unregistered name must be registrable, otherwise the "+ + "membership assertion above is vacuous: %v", + err, + ) + } +} + +// TestParticipationMetrics_QuarantinedSignersRegisteredAtZero proves the +// quarantined-signer gauge is pre-registered at zero rather than created on +// first use by SetGauge, and that it then records through the production +// recorder the tBTC quarantine reporter holds. +// +// Pre-registration is what makes an empty quarantine distinguishable from an +// unreported one: a node that never quarantines anything must still publish a +// zero, or a rollback decision cannot tell "nothing preserved" from "nothing +// said". +func TestParticipationMetrics_QuarantinedSignersRegisteredAtZero(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + registry := &Registry{keepclientinfo.NewRegistry(), ctx} + pm := NewPerformanceMetrics(ctx, registry) + + pm.gaugesMutex.RLock() + _, preRegistered := pm.gauges[MetricParticipationQuarantinedTBTCSigners] + pm.gaugesMutex.RUnlock() + + if !preRegistered { + t.Fatal( + "the quarantined-signer gauge must be registered before any " + + "quarantine occurs, not created by the first SetGauge", + ) + } + + if got := pm.GetGaugeValue( + MetricParticipationQuarantinedTBTCSigners, + ); got != 0 { + t.Errorf("quarantined-signer gauge = %v at startup, want 0", got) + } + + pm.SetGauge(MetricParticipationQuarantinedTBTCSigners, 2) + + if got := pm.GetGaugeValue( + MetricParticipationQuarantinedTBTCSigners, + ); got != 2 { + t.Errorf("quarantined-signer gauge = %v after update, want 2", got) + } +} + +// TestParticipationMetrics_QuarantinePreservationFailuresRegisteredAtZero +// proves both protocol-specific failure counters are present before a +// quarantine attempt. Zero must be a published reading rather than an absent +// series: the rollback gate distinguishes "no preservation failed" from "this +// node never reported whether preservation failed". +func TestParticipationMetrics_QuarantinePreservationFailuresRegisteredAtZero( + t *testing.T, +) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + registry := &Registry{keepclientinfo.NewRegistry(), ctx} + pm := NewPerformanceMetrics(ctx, registry) + + for _, name := range []string{ + MetricParticipationTBTCQuarantinePreservationFailuresTotal, + MetricParticipationBeaconQuarantinePreservationFailuresTotal, + } { + if got := pm.GetCounterValue(name); got != 0 { + t.Errorf("quarantine-preservation counter %q = %v at startup, want 0", name, got) + } + + pm.IncrementCounter(name, 1) + if got := pm.GetCounterValue(name); got != 1 { + t.Errorf("quarantine-preservation counter %q = %v after increment, want 1", name, got) + } + } +} + +// TestParticipationMetrics_QuarantineIncompleteOutputsRegisteredAtZero proves +// both live incomplete-output gauges are present before any quarantine attempt. +// A rollback sampler must be able to distinguish a running node reporting zero +// incomplete outputs from one that never exposed the signal. +func TestParticipationMetrics_QuarantineIncompleteOutputsRegisteredAtZero( + t *testing.T, +) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + registry := &Registry{keepclientinfo.NewRegistry(), ctx} + pm := NewPerformanceMetrics(ctx, registry) + + for _, name := range []string{ + MetricParticipationTBTCQuarantineIncompleteOutputs, + MetricParticipationBeaconQuarantineIncompleteOutputs, + } { + pm.gaugesMutex.RLock() + _, preRegistered := pm.gauges[name] + pm.gaugesMutex.RUnlock() + if !preRegistered { + t.Errorf( + "quarantine-incomplete gauge %q must be registered before "+ + "preservation starts", + name, + ) + continue + } + + if got := pm.GetGaugeValue(name); got != 0 { + t.Errorf( + "quarantine-incomplete gauge %q = %v at startup, want 0", + name, + got, + ) + } + + pm.SetGauge(name, 2) + if got := pm.GetGaugeValue(name); got != 2 { + t.Errorf( + "quarantine-incomplete gauge %q = %v after update, want 2", + name, + got, + ) + } + } +} diff --git a/pkg/clientinfo/metrics.go b/pkg/clientinfo/metrics.go index 91af36bcd8..8401e01379 100644 --- a/pkg/clientinfo/metrics.go +++ b/pkg/clientinfo/metrics.go @@ -144,12 +144,21 @@ func (r *Registry) ObserveApplicationSource( } } -// RegisterMetricClientInfo registers static client information labels for metrics. -func (r *Registry) RegisterMetricClientInfo(version string) { +// RegisterMetricClientInfo registers the static artifact-identity labels of +// the client_info metric: the release version, the exact source revision, and +// the compiled protocol epoch. Fleet tooling reconciles these against the +// expected release identity, so all three travel together. +func (r *Registry) RegisterMetricClientInfo( + version string, + revision string, + protocolEpoch string, +) { _, err := r.NewMetricInfo( ClientInfoMetricName, []clientinfo.Label{ clientinfo.NewLabel("version", version), + clientinfo.NewLabel("revision", revision), + clientinfo.NewLabel("protocol_epoch", protocolEpoch), }, ) if err != nil { diff --git a/pkg/clientinfo/metrics_test.go b/pkg/clientinfo/metrics_test.go index 18769a8e0d..c05d47a64d 100644 --- a/pkg/clientinfo/metrics_test.go +++ b/pkg/clientinfo/metrics_test.go @@ -54,7 +54,9 @@ func (m *mockProvider) CreateTransportIdentifier( ) (net.TransportIdentifier, error) { return nil, nil } -func (m *mockProvider) BroadcastChannelForwarderFor(string) {} +func (m *mockProvider) BroadcastChannelForwarderFor(string) (net.Forwarder, error) { + return net.NoopForwarder(), nil +} // TestObserveConnectedWellknownPeersCount_Callable verifies that the renamed // function exists on the Registry type and can be called without panicking. diff --git a/pkg/clientinfo/performance.go b/pkg/clientinfo/performance.go index d48c1c6b4d..193348e4c7 100644 --- a/pkg/clientinfo/performance.go +++ b/pkg/clientinfo/performance.go @@ -140,6 +140,22 @@ func (pm *PerformanceMetrics) registerAllMetrics() { MetricFirewallRejectionsTotal, MetricFirewallOnChainChecksTotal, MetricWalletDispatcherRejectedTotal, + MetricAnnouncerSessionIDMismatchTotal, + MetricAnnouncerCrossFormatPeerTotal, + MetricAnnouncerLegacyPeerAdditionsTotal, + MetricAnnouncerLegacyPeerEvictionsTotal, + MetricParticipationModeLegacyTotal, + MetricParticipationModeSecurityV2Total, + MetricParticipationLegacyCompletionsAfterCutoverTotal, + MetricParticipationRefusalsTotal, + MetricParticipationCommitRefusalsTotal, + MetricParticipationClockErrorsTotal, + MetricParticipationClockAbortsTotal, + MetricParticipationQuiesceTotal, + MetricParticipationQuiesceForcedAbortsTotal, + MetricParticipationTBTCQuarantinePreservationFailuresTotal, + MetricParticipationBeaconQuarantinePreservationFailuresTotal, + MetricHeartbeatPenaltySuppressedTotal, } // Register per-reason network join failure counters @@ -147,6 +163,11 @@ func (pm *PerformanceMetrics) registerAllMetrics() { counters = append(counters, NetworkJoinFailureMetricName(reason)) } + // Register per-ceremony participation refusal counters + for _, ceremony := range GetAllParticipationCeremonies() { + counters = append(counters, ParticipationRefusalMetricName(ceremony)) + } + // First, initialize all counters in the map pm.countersMutex.Lock() for _, name := range counters { @@ -307,6 +328,19 @@ func (pm *PerformanceMetrics) registerAllMetrics() { MetricCPULoadPercent, MetricRAMUtilizationPercent, MetricSwapUtilizationPercent, + MetricAnnouncerLegacyPeersCurrent, + MetricAnnouncerLegacyPeerOldestAgeBlocks, + MetricAnnouncerLegacyPeerRosterRevision, + MetricParticipationGateState, + MetricParticipationCurrentBlock, + MetricParticipationCutoverBlock, + MetricParticipationAllowed, + MetricParticipationActiveCeremonies, + MetricParticipationActiveLegacyCeremonies, + MetricParticipationActiveSecurityV2Ceremonies, + MetricParticipationTBTCQuarantineIncompleteOutputs, + MetricParticipationBeaconQuarantineIncompleteOutputs, + MetricParticipationQuarantinedTBTCSigners, } // First, initialize all gauges in the map @@ -692,6 +726,76 @@ const ( MetricCPULoadPercent = "cpu_load_percent" MetricRAMUtilizationPercent = "ram_utilization_percent" MetricSwapUtilizationPercent = "swap_utilization_percent" + + // Cutover observability Metrics + // + // These are the internal (unprefixed) names; they are exposed with the + // application prefix as performance_announcer_* by ObserveApplicationSource. + // They back the announcer session-ID mismatch observer and the node-local + // cutover peer roster used to identify operators that remain on the legacy + // release across a coordinated security-v2 cutover. + MetricAnnouncerSessionIDMismatchTotal = "announcer_session_id_mismatch_total" + MetricAnnouncerCrossFormatPeerTotal = "announcer_cross_format_peer_total" + MetricAnnouncerLegacyPeersCurrent = "announcer_legacy_peers_current" + MetricAnnouncerLegacyPeerOldestAgeBlocks = "announcer_legacy_peer_oldest_age_blocks" + MetricAnnouncerLegacyPeerRosterRevision = "announcer_legacy_peer_roster_revision" + MetricAnnouncerLegacyPeerAdditionsTotal = "announcer_legacy_peer_additions_total" + MetricAnnouncerLegacyPeerEvictionsTotal = "announcer_legacy_peer_evictions_total" + + // Protocol participation gate Metrics + // + // These back the chain-clocked cutover gate: the process participation + // state, the resolved cutover block, per-mode permit activity, and the + // refusal/abort/suppression evidence required for the cutover go/no-go + // and rollback decisions. Per-ceremony refusal counters are generated + // with ParticipationRefusalMetricName. + MetricParticipationGateState = "participation_gate_state" + MetricParticipationCurrentBlock = "participation_current_block" + MetricParticipationCutoverBlock = "participation_cutover_block" + MetricParticipationAllowed = "participation_allowed" + MetricParticipationActiveCeremonies = "participation_active_ceremonies" + MetricParticipationActiveLegacyCeremonies = "participation_active_legacy_ceremonies" + MetricParticipationActiveSecurityV2Ceremonies = "participation_active_security_v2_ceremonies" + MetricParticipationModeLegacyTotal = "participation_mode_legacy_total" + MetricParticipationModeSecurityV2Total = "participation_mode_security_v2_total" + MetricParticipationLegacyCompletionsAfterCutoverTotal = "participation_legacy_completions_after_cutover_total" + MetricParticipationRefusalsTotal = "participation_refusals_total" + MetricParticipationCommitRefusalsTotal = "participation_commit_refusals_total" + MetricParticipationClockErrorsTotal = "participation_clock_errors_total" + MetricParticipationClockAbortsTotal = "participation_clock_aborts_total" + MetricParticipationQuiesceTotal = "participation_quiesce_total" + MetricParticipationQuiesceForcedAbortsTotal = "participation_quiesce_forced_aborts_total" + + // The quarantine-preservation failure counters increment as soon as an + // output remains incomplete after the write-grace rounds, while the process + // is still alive and retrying. An attempt that returns incomplete before + // that observer can run is counted on return. They remain cumulative if a + // later retry completes; the paired live gauges below distinguish recovered + // history from an output still incomplete. They are protocol-specific fixed + // counters so fleet and rollback evidence can identify which recovery path + // exhausted its grace without introducing per-output labels. + MetricParticipationTBTCQuarantinePreservationFailuresTotal = "participation_tbtc_quarantine_preservation_failures_total" + MetricParticipationBeaconQuarantinePreservationFailuresTotal = "participation_beacon_quarantine_preservation_failures_total" + MetricHeartbeatPenaltySuppressedTotal = "heartbeat_penalty_suppressed_total" + + // The quarantine-incomplete gauges report outputs the running process is + // still holding while their protected namespace lacks either key material + // or the audit record explaining it. A live retry raises them at the same + // grace-exhaustion transition as the counters above, and they return to zero + // only after the full output becomes durable. A total-write refusal + // therefore remains visible for the entire live retry instead of surfacing + // only during teardown. + MetricParticipationTBTCQuarantineIncompleteOutputs = "participation_tbtc_quarantine_incomplete_outputs" + MetricParticipationBeaconQuarantineIncompleteOutputs = "participation_beacon_quarantine_incomplete_outputs" + + // MetricParticipationQuarantinedTBTCSigners counts the tBTC signer outputs + // held in the protected quarantine namespace that this process has not + // activated in its wallet cache. It belongs to the participation family + // because a quarantined output is a gate refusal that outlived the ceremony, + // but only tBTC can report it: the namespace and the wallet cache it is + // compared against are both owned there, so it is published from + // pkg/tbtc rather than by the gate itself. + MetricParticipationQuarantinedTBTCSigners = "participation_quarantined_tbtc_signers" ) // Network join request failure reasons. These are the low-cardinality @@ -753,3 +857,30 @@ func GetAllWalletActionTypes() []string { "moved_funds_sweep", } } + +// ParticipationRefusalMetricName generates the per-ceremony refusal counter +// name for the protocol participation gate. ceremony should be one of the +// GetAllParticipationCeremonies values. +// Format: participation_refusals_{ceremony}_total +// Example: participation_refusals_tbtc_dkg_total +func ParticipationRefusalMetricName(ceremony string) string { + return fmt.Sprintf("participation_refusals_%s_total", ceremony) +} + +// GetAllParticipationCeremonies returns the fixed set of gated protocol +// ceremonies whose per-ceremony refusal counters should be tracked. It must +// stay in lockstep with the participation package's ceremony constants; a +// drift test there asserts the two lists are identical. +func GetAllParticipationCeremonies() []string { + return []string{ + "tbtc_dkg", + "tbtc_wallet_coordination", + "tbtc_signing", + "tbtc_heartbeat", + "tbtc_inactivity_claim", + "beacon_dkg", + "beacon_relay_signing", + "beacon_relay_forwarding", + "beacon_timeout_report", + } +} diff --git a/pkg/clientinfo/performance_test.go b/pkg/clientinfo/performance_test.go index 5ebf253288..6b20d9dea4 100644 --- a/pkg/clientinfo/performance_test.go +++ b/pkg/clientinfo/performance_test.go @@ -336,6 +336,33 @@ func TestMetricsInitialization(t *testing.T) { } } + // The SPV maintainer redemption-proof counters must be registered at zero + // and increment through IncrementCounter, so the maintainer's proof path is + // scrapeable at performance_redemption_proof_submissions_*. + redemptionProofCounters := []string{ + MetricRedemptionProofSubmissionsTotal, + MetricRedemptionProofSubmissionsSuccessTotal, + MetricRedemptionProofSubmissionsFailedTotal, + } + + for _, counterName := range redemptionProofCounters { + if value := pm.GetCounterValue(counterName); value != 0 { + t.Errorf( + "Counter %s should start at 0, got %v", + counterName, + value, + ) + } + pm.IncrementCounter(counterName, 1) + if value := pm.GetCounterValue(counterName); value != 1 { + t.Errorf( + "Counter %s should be 1 after increment, got %v", + counterName, + value, + ) + } + } + // Test gauges gauges := []string{ MetricCPUUtilization, diff --git a/pkg/crypto/ephemeral/full_ecdh_test.go b/pkg/crypto/ephemeral/full_ecdh_test.go index c73c643978..792e5a153d 100644 --- a/pkg/crypto/ephemeral/full_ecdh_test.go +++ b/pkg/crypto/ephemeral/full_ecdh_test.go @@ -24,10 +24,10 @@ func TestFullEcdh(t *testing.T) { // // player 1: - symmetricKey1 := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey) + symmetricKey1 := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, nil) // player 2: - symmetricKey2 := keyPair2.PrivateKey.Ecdh(keyPair1.PublicKey) + symmetricKey2 := keyPair2.PrivateKey.Ecdh(keyPair1.PublicKey, nil) // // players use symmetric key for encryption/decryption diff --git a/pkg/crypto/ephemeral/symmetric_key.go b/pkg/crypto/ephemeral/symmetric_key.go index 75fba04baf..42618d0a8a 100644 --- a/pkg/crypto/ephemeral/symmetric_key.go +++ b/pkg/crypto/ephemeral/symmetric_key.go @@ -2,9 +2,11 @@ package ephemeral import ( "crypto/sha256" + "io" "github.com/btcsuite/btcd/btcec" "github.com/keep-network/keep-common/pkg/encryption" + "golang.org/x/crypto/hkdf" ) // SymmetricEcdhKey is an ephemeral Elliptic Curve key created with @@ -13,10 +15,42 @@ type SymmetricEcdhKey struct { box encryption.Box } -// Ecdh performs Elliptic Curve Diffie-Hellman operation between public and -// private key. The returned value is `SymmetricEcdhKey` that can be used -// for encryption and decryption. -func (pk *PrivateKey) Ecdh(publicKey *PublicKey) *SymmetricEcdhKey { +// Ecdh performs Elliptic Curve Diffie-Hellman between the private key and +// publicKey, then derives a 32-byte symmetric key via HKDF-SHA256. The info +// parameter provides domain separation: callers should pass a label encoding +// the protocol name and the canonical (sorted) peer-pair IDs so that keys +// derived for different protocols or peer pairs are cryptographically +// independent. +// +// This is the hardened security-v2 derivation. A ceremony participates with +// exactly one of Ecdh or EcdhLegacy for its entire lifetime, selected +// explicitly from the ceremony's pinned protocol mode — never from a global +// toggle or the current chain height. +func (pk *PrivateKey) Ecdh(publicKey *PublicKey, info []byte) *SymmetricEcdhKey { + shared := btcec.GenerateSharedSecret( + (*btcec.PrivateKey)(pk), + (*btcec.PublicKey)(publicKey), + ) + + kdf := hkdf.New(sha256.New, shared, nil, info) + var key [32]byte + if _, err := io.ReadFull(kdf, key[:]); err != nil { + // HKDF over a fixed-size output cannot fail in practice. + panic("ephemeral.Ecdh: HKDF derivation failed: " + err.Error()) + } + + return &SymmetricEcdhKey{ + box: encryption.NewBox(key), + } +} + +// EcdhLegacy performs Elliptic Curve Diffie-Hellman between the private key +// and publicKey and derives the symmetric key as a direct SHA-256 of the +// shared secret, byte-for-byte as the pre-hardening production releases do. +// It exists solely so a ceremony pinned to the legacy protocol mode can +// interoperate with peers running the prior release during the coordinated +// cutover; ceremonies pinned to security-v2 use Ecdh. +func (pk *PrivateKey) EcdhLegacy(publicKey *PublicKey) *SymmetricEcdhKey { shared := btcec.GenerateSharedSecret( (*btcec.PrivateKey)(pk), (*btcec.PublicKey)(publicKey), diff --git a/pkg/crypto/ephemeral/symmetric_key_test.go b/pkg/crypto/ephemeral/symmetric_key_test.go index 4a61ec3524..b0d1429b3a 100644 --- a/pkg/crypto/ephemeral/symmetric_key_test.go +++ b/pkg/crypto/ephemeral/symmetric_key_test.go @@ -1,9 +1,15 @@ package ephemeral import ( + "crypto/sha256" "fmt" + "io" "reflect" "testing" + + "github.com/btcsuite/btcd/btcec" + "github.com/keep-network/keep-common/pkg/encryption" + "golang.org/x/crypto/hkdf" ) func TestEncryptDecrypt(t *testing.T) { @@ -86,6 +92,61 @@ func TestGracefullyHandleBrokenCipher(t *testing.T) { } } +// TestEcdhInfoDomainSeparation verifies that different info values produce +// different keys even for the same ECDH shared secret. +func TestEcdhInfoDomainSeparation(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + keyA := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, []byte("protocol-a")) + keyB := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, []byte("protocol-b")) + + msgA, err := keyA.Encrypt([]byte("hello")) + if err != nil { + t.Fatal(err) + } + // keyB must not decrypt a message encrypted with keyA. + if _, err := keyB.Decrypt(msgA); err == nil { + t.Fatal("different info values produced the same key") + } +} + +// TestEcdhSymmetry verifies that both sides of ECDH with the same info derive +// the same key (ECDH is commutative and HKDF is deterministic). +func TestEcdhSymmetry(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + info := []byte("symmetry-test") + key1 := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, info) + key2 := keyPair2.PrivateKey.Ecdh(keyPair1.PublicKey, info) + + msg := []byte("message") + encrypted, err := key1.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + decrypted, err := key2.Decrypt(encrypted) + if err != nil { + t.Fatalf("symmetric ECDH keys do not match: %v", err) + } + if string(decrypted) != string(msg) { + t.Fatalf("expected %q, got %q", msg, decrypted) + } +} + func newEcdhSymmetricKey() (*SymmetricEcdhKey, error) { keyPair1, err := GenerateKeyPair() if err != nil { @@ -97,5 +158,195 @@ func newEcdhSymmetricKey() (*SymmetricEcdhKey, error) { return nil, err } - return keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey), nil + return keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, []byte("test")), nil +} + +// TestEcdhNilInfoDiffersFromLabeled documents that passing nil info produces a +// key that is cryptographically distinct from any labeled derivation. This +// prevents a regression where nil and a real label converge to the same key. +func TestEcdhNilInfoDiffersFromLabeled(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + keyNil := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, nil) + keyLabeled := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, []byte("some-protocol")) + + msg := []byte("probe") + encrypted, err := keyNil.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + if _, err := keyLabeled.Decrypt(encrypted); err == nil { + t.Fatal("nil info and labeled info produced the same HKDF key") + } +} + +// TestEcdhLegacyMatchesPreHardeningDerivation proves EcdhLegacy derives the +// exact pre-hardening key: a box keyed with the direct SHA-256 of the ECDH +// shared secret, computed here independently, must interoperate with the +// EcdhLegacy box in both directions. +func TestEcdhLegacyMatchesPreHardeningDerivation(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + legacyKey := keyPair1.PrivateKey.EcdhLegacy(keyPair2.PublicKey) + + shared := btcec.GenerateSharedSecret( + (*btcec.PrivateKey)(keyPair2.PrivateKey), + (*btcec.PublicKey)(keyPair1.PublicKey), + ) + referenceKey := &SymmetricEcdhKey{ + box: encryption.NewBox(sha256.Sum256(shared)), + } + + msg := []byte("legacy reference interop") + encrypted, err := legacyKey.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + decrypted, err := referenceKey.Decrypt(encrypted) + if err != nil { + t.Fatalf("reference sha256(shared) key cannot decrypt: [%v]", err) + } + if string(decrypted) != string(msg) { + t.Fatalf("expected %q, got %q", msg, decrypted) + } + + encrypted, err = referenceKey.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + decrypted, err = legacyKey.Decrypt(encrypted) + if err != nil { + t.Fatalf("EcdhLegacy cannot decrypt the reference key: [%v]", err) + } + if string(decrypted) != string(msg) { + t.Fatalf("expected %q, got %q", msg, decrypted) + } +} + +// TestEcdhMatchesHKDFDerivation proves the hardened Ecdh derives the exact +// HKDF-SHA256 key for a protocol/peer info label, computed here independently. +func TestEcdhMatchesHKDFDerivation(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + info := []byte("protocol-label-peer-1-2") + hardenedKey := keyPair1.PrivateKey.Ecdh(keyPair2.PublicKey, info) + + shared := btcec.GenerateSharedSecret( + (*btcec.PrivateKey)(keyPair2.PrivateKey), + (*btcec.PublicKey)(keyPair1.PublicKey), + ) + kdf := hkdf.New(sha256.New, shared, nil, info) + var key [32]byte + if _, err := io.ReadFull(kdf, key[:]); err != nil { + t.Fatal(err) + } + referenceKey := &SymmetricEcdhKey{box: encryption.NewBox(key)} + + msg := []byte("hardened reference interop") + encrypted, err := hardenedKey.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + decrypted, err := referenceKey.Decrypt(encrypted) + if err != nil { + t.Fatalf("reference HKDF key cannot decrypt: [%v]", err) + } + if string(decrypted) != string(msg) { + t.Fatalf("expected %q, got %q", msg, decrypted) + } +} + +// TestEcdhLegacySymmetry proves both sides of the legacy derivation reach the +// same key, exactly as prior-release peers do. +func TestEcdhLegacySymmetry(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + key1 := keyPair1.PrivateKey.EcdhLegacy(keyPair2.PublicKey) + key2 := keyPair2.PrivateKey.EcdhLegacy(keyPair1.PublicKey) + + msg := []byte("legacy homogeneous message") + encrypted, err := key1.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + decrypted, err := key2.Decrypt(encrypted) + if err != nil { + t.Fatalf("legacy symmetric ECDH keys do not match: [%v]", err) + } + if string(decrypted) != string(msg) { + t.Fatalf("expected %q, got %q", msg, decrypted) + } +} + +// TestEcdhCrossModeDecryptionFails proves the two derivations are +// cryptographically disjoint: a legacy key must not decrypt a security-v2 +// ciphertext and the reverse must fail with an error, without producing +// plaintext and without panicking. +func TestEcdhCrossModeDecryptionFails(t *testing.T) { + keyPair1, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + keyPair2, err := GenerateKeyPair() + if err != nil { + t.Fatal(err) + } + + legacyKey := keyPair1.PrivateKey.EcdhLegacy(keyPair2.PublicKey) + hardenedKey := keyPair2.PrivateKey.Ecdh( + keyPair1.PublicKey, + []byte("protocol-label"), + ) + + msg := []byte("cross-mode probe") + + hardenedCiphertext, err := hardenedKey.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + if plaintext, err := legacyKey.Decrypt(hardenedCiphertext); err == nil { + t.Fatalf( + "legacy key decrypted a security-v2 ciphertext: %q", + plaintext, + ) + } + + legacyCiphertext, err := legacyKey.Encrypt(msg) + if err != nil { + t.Fatal(err) + } + if plaintext, err := hardenedKey.Decrypt(legacyCiphertext); err == nil { + t.Fatalf( + "security-v2 key decrypted a legacy ciphertext: %q", + plaintext, + ) + } } diff --git a/pkg/generator/scheduler_test.go b/pkg/generator/scheduler_test.go index 178cfc5dd9..04f93bc923 100644 --- a/pkg/generator/scheduler_test.go +++ b/pkg/generator/scheduler_test.go @@ -3,6 +3,8 @@ package generator import ( "context" "math/big" + "sync" + "sync/atomic" "testing" "time" @@ -11,6 +13,33 @@ import ( var one = big.NewInt(1) +// safeCounter is a goroutine-safe counter used by the scheduler tests. The +// scheduler runs worker functions in their own goroutines while the test's main +// goroutine reads the accumulated results, so both the increment and the +// snapshot read must be synchronized to avoid a data race. +type safeCounter struct { + mu sync.Mutex + value *big.Int +} + +func newSafeCounter() *safeCounter { + return &safeCounter{value: big.NewInt(0)} +} + +func (c *safeCounter) increment() { + c.mu.Lock() + defer c.mu.Unlock() + c.value.Add(c.value, one) +} + +// snapshot returns an independent copy of the current value that the caller can +// compare without holding the lock. +func (c *safeCounter) snapshot() *big.Int { + c.mu.Lock() + defer c.mu.Unlock() + return new(big.Int).Set(c.value) +} + // TestComputeStop tests the situation when two new worker functions are added // to a scheduler in a working state. The test ensures the worker functions // starts doing their work. Then, the scheduler is stopped and the test ensures @@ -18,22 +47,22 @@ var one = big.NewInt(1) func TestComputeStop(t *testing.T) { scheduler := new(Scheduler) - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) // give some time to perform computations time.Sleep(10 * time.Millisecond) // ensure computations started - testutils.AssertBigIntNonZero(t, "computation result", number1) - testutils.AssertBigIntNonZero(t, "computation result", number2) + testutils.AssertBigIntNonZero(t, "computation result", number1.snapshot()) + testutils.AssertBigIntNonZero(t, "computation result", number2.snapshot()) // send the stop signal and give some time to stop computations scheduler.stop() @@ -41,8 +70,8 @@ func TestComputeStop(t *testing.T) { // at this point, all computations should be stopped, capture the current // result - result1 := new(big.Int).Set(number1) - result2 := new(big.Int).Set(number2) + result1 := number1.snapshot() + result2 := number2.snapshot() // wait some time and ensure computations stopped time.Sleep(20 * time.Millisecond) @@ -50,13 +79,13 @@ func TestComputeStop(t *testing.T) { t, "computation result after stop signal", result1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsEqual( t, "computation result after stop signal", result2, - number2, + number2.snapshot(), ) } @@ -66,18 +95,20 @@ func TestComputeStop(t *testing.T) { func TestComputeStopContext(t *testing.T) { scheduler := new(Scheduler) - cancelled1 := false - cancelled2 := false + // cancelled1/cancelled2 are written by the worker goroutines and read by the + // main goroutine, so they are accessed atomically. + var cancelled1 atomic.Bool + var cancelled2 atomic.Bool scheduler.compute(func(ctx context.Context) { // this simulates a long-running task <-ctx.Done() - cancelled1 = true + cancelled1.Store(true) }) scheduler.compute(func(ctx context.Context) { // this simulates a long-running task <-ctx.Done() - cancelled2 = true + cancelled2.Store(true) }) // give some time to perform computations @@ -88,10 +119,10 @@ func TestComputeStopContext(t *testing.T) { time.Sleep(100 * time.Millisecond) // ensure context got cancelled - if !cancelled1 { + if !cancelled1.Load() { t.Errorf("expected context to be cancelled") } - if !cancelled2 { + if !cancelled2.Load() { t.Errorf("expected context to be cancelled") } } @@ -104,14 +135,14 @@ func TestComputeStopResume(t *testing.T) { scheduler := new(Scheduler) defer scheduler.stop() - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) // send the stop signal and give some time to stop computations @@ -120,8 +151,8 @@ func TestComputeStopResume(t *testing.T) { // at this point, all computations should be stopped, capture the current // result - intermediateResult1 := new(big.Int).Set(number1) - intermediateResult2 := new(big.Int).Set(number2) + intermediateResult1 := number1.snapshot() + intermediateResult2 := number2.snapshot() // send the resume signal and give some time to resume computations scheduler.resume() @@ -132,13 +163,13 @@ func TestComputeStopResume(t *testing.T) { t, "computation results after resume signal", intermediateResult1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsNotEqual( t, "computation results after resume signal", intermediateResult2, - number2, + number2.snapshot(), ) } @@ -149,14 +180,14 @@ func TestComputeStopResume(t *testing.T) { func TestComputeStopResumeStop(t *testing.T) { scheduler := new(Scheduler) - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) scheduler.stop() @@ -165,8 +196,8 @@ func TestComputeStopResumeStop(t *testing.T) { // at this point, all computations should be stopped, capture the current // result - result1 := new(big.Int).Set(number1) - result2 := new(big.Int).Set(number2) + result1 := number1.snapshot() + result2 := number2.snapshot() // wait some time and ensure computations stopped time.Sleep(20 * time.Millisecond) @@ -174,13 +205,13 @@ func TestComputeStopResumeStop(t *testing.T) { t, "computation result after stop signal", result1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsEqual( t, "computation result after stop signal", result2, - number2, + number2.snapshot(), ) } @@ -194,19 +225,19 @@ func TestStopComputeResume(t *testing.T) { scheduler.stop() - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) // assert computations have not started - the scheduler is stopped - testutils.AssertBigIntsEqual(t, "computation result", big.NewInt(0), number1) - testutils.AssertBigIntsEqual(t, "computation result", big.NewInt(0), number2) + testutils.AssertBigIntsEqual(t, "computation result", big.NewInt(0), number1.snapshot()) + testutils.AssertBigIntsEqual(t, "computation result", big.NewInt(0), number2.snapshot()) scheduler.resume() // give some time to perform computations; @@ -215,8 +246,8 @@ func TestStopComputeResume(t *testing.T) { time.Sleep(250 * time.Millisecond) // ensure computations started - testutils.AssertBigIntNonZero(t, "computation result", number1) - testutils.AssertBigIntNonZero(t, "computation result", number2) + testutils.AssertBigIntNonZero(t, "computation result", number1.snapshot()) + testutils.AssertBigIntNonZero(t, "computation result", number2.snapshot()) } // TestCheckProtocols_NoProtocols ensures the execution of checkProtocols @@ -225,14 +256,14 @@ func TestCheckProtocols_NoProtocols(t *testing.T) { scheduler := new(Scheduler) defer scheduler.stop() - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) // give some time to perform computations @@ -245,21 +276,21 @@ func TestCheckProtocols_NoProtocols(t *testing.T) { // there are no protocols executed, nothing can stop the scheduler; // ensure the computations are performed - intermediateResult1 := new(big.Int).Set(number1) - intermediateResult2 := new(big.Int).Set(number2) + intermediateResult1 := number1.snapshot() + intermediateResult2 := number2.snapshot() time.Sleep(20 * time.Millisecond) testutils.AssertBigIntsNotEqual( t, "computation result after stop signal", intermediateResult1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsNotEqual( t, "computation result after stop signal", intermediateResult2, - number2, + number2.snapshot(), ) } @@ -270,14 +301,14 @@ func TestCheckProtocols_ProtocolNotExecuting(t *testing.T) { scheduler := new(Scheduler) defer scheduler.stop() - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) protocol1 := &mockProtocol{} @@ -295,21 +326,21 @@ func TestCheckProtocols_ProtocolNotExecuting(t *testing.T) { // there are two protocols but they are not executing; // ensure the computations are performed - intermediateResult1 := new(big.Int).Set(number1) - intermediateResult2 := new(big.Int).Set(number2) + intermediateResult1 := number1.snapshot() + intermediateResult2 := number2.snapshot() time.Sleep(20 * time.Millisecond) testutils.AssertBigIntsNotEqual( t, "computation result after stop signal", intermediateResult1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsNotEqual( t, "computation result after stop signal", intermediateResult2, - number2, + number2.snapshot(), ) } @@ -319,14 +350,14 @@ func TestCheckProtocols_ProtocolExecuting(t *testing.T) { scheduler := new(Scheduler) defer scheduler.stop() - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) protocol1 := &mockProtocol{} @@ -345,21 +376,21 @@ func TestCheckProtocols_ProtocolExecuting(t *testing.T) { // there are two protocols and the second one is executing // ensure the computations are stopped - intermediateResult1 := new(big.Int).Set(number1) - intermediateResult2 := new(big.Int).Set(number2) + intermediateResult1 := number1.snapshot() + intermediateResult2 := number2.snapshot() time.Sleep(20 * time.Millisecond) testutils.AssertBigIntsEqual( t, "computation result after stop signal", intermediateResult1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsEqual( t, "computation result after stop signal", intermediateResult2, - number2, + number2.snapshot(), ) } @@ -370,14 +401,14 @@ func TestCheckProtocols_ProtocolFinishedExecution(t *testing.T) { scheduler := new(Scheduler) defer scheduler.stop() - number1 := big.NewInt(0) - number2 := big.NewInt(0) + number1 := newSafeCounter() + number2 := newSafeCounter() scheduler.compute(func(context.Context) { - number1.Add(number1, one) + number1.increment() }) scheduler.compute(func(context.Context) { - number2.Add(number2, one) + number2.increment() }) protocol1 := &mockProtocol{} @@ -402,21 +433,21 @@ func TestCheckProtocols_ProtocolFinishedExecution(t *testing.T) { // there are two protocols, the second one was executing, but it has // finished; ensure the computations are resumed - intermediateResult1 := new(big.Int).Set(number1) - intermediateResult2 := new(big.Int).Set(number2) + intermediateResult1 := number1.snapshot() + intermediateResult2 := number2.snapshot() time.Sleep(20 * time.Millisecond) testutils.AssertBigIntsNotEqual( t, "computation result after stop signal", intermediateResult1, - number1, + number1.snapshot(), ) testutils.AssertBigIntsNotEqual( t, "computation result after stop signal", intermediateResult2, - number2, + number2.snapshot(), ) } diff --git a/pkg/internal/byzantine/strategy.go b/pkg/internal/byzantine/strategy.go new file mode 100644 index 0000000000..4dae2500de --- /dev/null +++ b/pkg/internal/byzantine/strategy.go @@ -0,0 +1,118 @@ +// Package byzantine provides a small library of named, composable +// interception.Strategy constructors for simulating malicious-operator +// behavior in deterministic protocol tests (Tier-2 Byzantine simulation). +// +// Each constructor targets a single group member by its protocol-level +// MemberIndex and passes every other member's traffic through untouched, so a +// scenario reads as "member N does X". The strategies are protocol-agnostic: +// they act on net.TaggedMarshaler and a caller-supplied match predicate, so the +// same library serves DKG today and threshold signing once those harnesses +// exist. +// +// Scope and boundary are inherited from interception.Strategy: these act on the +// wire, after a sender serialized and encrypted its message. They can withhold, +// duplicate, and corrupt/replace a message, but cannot forge a chosen +// inconsistent-but-individually-valid share (that needs a malicious member, not +// channel interception). See the interception.Strategy contract for the precise +// wire-level boundary. +package byzantine + +import ( + "github.com/keep-network/keep-core/pkg/internal/interception" + "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// MatchAll is the nil predicate: it matches every message. Pass it (or nil) +// where a constructor accepts a match predicate to act on all of a member's +// messages regardless of type. +var MatchAll func(net.TaggedMarshaler) bool = nil + +// matches reports whether the predicate selects the message. A nil predicate +// matches everything. +func matches(match func(net.TaggedMarshaler) bool, m net.TaggedMarshaler) bool { + return match == nil || match(m) +} + +// targeted builds a Strategy that applies action to messages from member that +// satisfy match, and passes every other message (other senders, or +// non-matching types from this member) through unchanged. action receives the +// outbound message and returns the set actually delivered. +func targeted( + member group.MemberIndex, + match func(net.TaggedMarshaler) bool, + action func(out interception.Outbound) []net.TaggedMarshaler, +) interception.Strategy { + return func(out interception.Outbound) []net.TaggedMarshaler { + if out.Sender == member && matches(match, out.Message) { + return action(out) + } + return interception.PassThrough(out) + } +} + +// Inactive drops every message from member, modelling a member that is silent +// for the whole protocol. Equivalent to Withhold(member, MatchAll). +func Inactive(member group.MemberIndex) interception.Strategy { + return Withhold(member, MatchAll) +} + +// Withhold drops messages from member that satisfy match (e.g. a single phase's +// message type), passing all others through. Models selective withholding at a +// specific protocol step. A nil match withholds every message from the member. +func Withhold( + member group.MemberIndex, + match func(net.TaggedMarshaler) bool, +) interception.Strategy { + return targeted(member, match, func(interception.Outbound) []net.TaggedMarshaler { + return nil // empty set -> dropped + }) +} + +// Flood delivers `copies` instances of every message from member that satisfies +// match. Each copy is sent independently and receives its own transport +// sequence number, so receivers do not treat them as retransmissions: the +// protocol's own per-sender deduplication is what must absorb the flood. +// copies <= 1 is a no-op pass-through (one delivery). +// +// The copies share the same underlying message pointer. That is safe for +// duplication, but do NOT combine Flood with an in-place mutation - mutating +// one copy mutates them all. To duplicate-then-corrupt, return distinct cloned +// messages from a custom Strategy instead. +func Flood( + member group.MemberIndex, + copies int, + match func(net.TaggedMarshaler) bool, +) interception.Strategy { + return targeted(member, match, func(out interception.Outbound) []net.TaggedMarshaler { + if copies < 1 { + return []net.TaggedMarshaler{out.Message} + } + flooded := make([]net.TaggedMarshaler, copies) + for i := range flooded { + flooded[i] = out.Message + } + return flooded + }) +} + +// Corrupt replaces messages from member that satisfy match with +// transform(message), modelling a malformed-but-typed message. If transform +// returns nil the message is dropped. transform may mutate and return the +// message in place (e.g. PeerSharesMessage.RemoveShares): the local transport +// marshals each outbound message synchronously at Send, so the mutation is +// captured for this delivery and other members' independently-marshaled sends +// are unaffected. This matches the existing GJKR disqualification tests. +func Corrupt( + member group.MemberIndex, + match func(net.TaggedMarshaler) bool, + transform func(net.TaggedMarshaler) net.TaggedMarshaler, +) interception.Strategy { + return targeted(member, match, func(out interception.Outbound) []net.TaggedMarshaler { + replaced := transform(out.Message) + if replaced == nil { + return nil + } + return []net.TaggedMarshaler{replaced} + }) +} diff --git a/pkg/internal/byzantine/strategy_test.go b/pkg/internal/byzantine/strategy_test.go new file mode 100644 index 0000000000..f289f91979 --- /dev/null +++ b/pkg/internal/byzantine/strategy_test.go @@ -0,0 +1,101 @@ +package byzantine_test + +import ( + "testing" + + "github.com/keep-network/keep-core/pkg/internal/byzantine" + "github.com/keep-network/keep-core/pkg/internal/interception" + "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// msg is a minimal sender-attributed TaggedMarshaler for exercising the +// strategy constructors without running a protocol. +type msg struct { + kind string + sender group.MemberIndex +} + +func (m *msg) Type() string { return m.kind } +func (m *msg) Marshal() ([]byte, error) { return []byte(m.kind), nil } +func (m *msg) Unmarshal(b []byte) error { m.kind = string(b); return nil } +func (m *msg) SenderID() group.MemberIndex { return m.sender } + +// apply runs a strategy against a message attributed to sender and returns the +// delivered set, mirroring what interception's channel extracts. +func apply(s interception.Strategy, sender group.MemberIndex, m net.TaggedMarshaler) []net.TaggedMarshaler { + return s(interception.Outbound{Sender: sender, Message: m}) +} + +func TestInactiveDropsOnlyTargetMember(t *testing.T) { + s := byzantine.Inactive(group.MemberIndex(3)) + + if got := apply(s, 3, &msg{"any", 3}); len(got) != 0 { + t.Errorf("member 3 message: delivered %d; want 0 (dropped)", len(got)) + } + if got := apply(s, 2, &msg{"any", 2}); len(got) != 1 { + t.Errorf("member 2 message: delivered %d; want 1 (passed through)", len(got)) + } +} + +func TestWithholdMatchesTypeAndMember(t *testing.T) { + isPhase1 := func(m net.TaggedMarshaler) bool { return m.Type() == "phase1" } + s := byzantine.Withhold(group.MemberIndex(3), isPhase1) + + // Targeted member, matching type -> dropped. + if got := apply(s, 3, &msg{"phase1", 3}); len(got) != 0 { + t.Errorf("member 3 phase1: delivered %d; want 0", len(got)) + } + // Targeted member, non-matching type -> passes. + if got := apply(s, 3, &msg{"phase2", 3}); len(got) != 1 { + t.Errorf("member 3 phase2: delivered %d; want 1", len(got)) + } + // Other member, matching type -> passes. + if got := apply(s, 5, &msg{"phase1", 5}); len(got) != 1 { + t.Errorf("member 5 phase1: delivered %d; want 1", len(got)) + } +} + +func TestFloodDuplicatesTargetMember(t *testing.T) { + s := byzantine.Flood(group.MemberIndex(3), 4, byzantine.MatchAll) + + if got := apply(s, 3, &msg{"any", 3}); len(got) != 4 { + t.Errorf("member 3 flood: delivered %d; want 4", len(got)) + } + if got := apply(s, 2, &msg{"any", 2}); len(got) != 1 { + t.Errorf("member 2 (untargeted): delivered %d; want 1", len(got)) + } + + // copies < 1 is a single pass-through, never a drop. + noop := byzantine.Flood(group.MemberIndex(3), 0, byzantine.MatchAll) + if got := apply(noop, 3, &msg{"any", 3}); len(got) != 1 { + t.Errorf("flood copies=0: delivered %d; want 1", len(got)) + } +} + +func TestCorruptReplacesAndCanDrop(t *testing.T) { + corrupted := &msg{"corrupted", 3} + replace := byzantine.Corrupt( + group.MemberIndex(3), + byzantine.MatchAll, + func(net.TaggedMarshaler) net.TaggedMarshaler { return corrupted }, + ) + got := apply(replace, 3, &msg{"original", 3}) + if len(got) != 1 || got[0] != corrupted { + t.Errorf("corrupt: got %v; want the corrupted replacement", got) + } + // Untargeted member is untouched. + if got := apply(replace, 1, &msg{"original", 1}); len(got) != 1 || got[0].Type() != "original" { + t.Errorf("corrupt leaked onto member 1: %v", got) + } + + // transform returning nil drops the message. + drop := byzantine.Corrupt( + group.MemberIndex(3), + byzantine.MatchAll, + func(net.TaggedMarshaler) net.TaggedMarshaler { return nil }, + ) + if got := apply(drop, 3, &msg{"original", 3}); len(got) != 0 { + t.Errorf("corrupt->nil: delivered %d; want 0 (dropped)", len(got)) + } +} diff --git a/pkg/internal/dkgtest/assertions.go b/pkg/internal/dkgtest/assertions.go index 1a53928e87..dd99390cfa 100644 --- a/pkg/internal/dkgtest/assertions.go +++ b/pkg/internal/dkgtest/assertions.go @@ -1,6 +1,7 @@ package dkgtest import ( + "strings" "testing" "github.com/keep-network/keep-core/internal/testutils" @@ -16,6 +17,51 @@ func AssertDkgResultPublished(t *testing.T, testResult *Result) { } } +// AssertNoDkgResultPublished checks that no DKG result reached the chain: the +// fail-closed outcome of a ceremony that must not complete. +func AssertNoDkgResultPublished(t *testing.T, testResult *Result) { + if testResult.dkgResult != nil { + t.Fatal("expected no dkg result to be published") + } +} + +// reconstructionGuardMarker is a stable substring of the F-008 defensive guard's +// Error message (gjkr/protocol.go ComputeGroupPublicKeyShares). Its appearance +// means the reconstructed-share branch found peerSharesS missing an entry for an +// operating member - i.e. the gap F-008 posits actually occurred at runtime and +// was absorbed by the guard (upstream, without the guard, this is the crash). +const reconstructionGuardMarker = "missing revealed share" + +// reconstructionGapErrors returns the captured Errorf messages that match the +// F-008 guard marker. Pure (no *testing.T) so the detection logic is unit +// testable independently of a full DKG run. +func reconstructionGapErrors(testResult *Result) []string { + var hits []string + for _, msg := range testResult.loggedErrors { + if strings.Contains(msg, reconstructionGuardMarker) { + hits = append(hits, msg) + } + } + return hits +} + +// AssertNoReconstructionGap fails if the F-008 reconstruction guard fired during +// the run. A passing assertion is the execution-verified evidence that the +// reconstructed-share branch found peerSharesS fully populated - corroborating +// the reachability analysis that the gap does not occur under real execution. +// This is distinct from the unit-level guard regression +// (gjkr.TestComputeGroupPublicKeyShares_MissingRevealedShare), which forces the +// gap artificially to check the guard; here we check the gap never forms. +func AssertNoReconstructionGap(t *testing.T, testResult *Result) { + for _, msg := range reconstructionGapErrors(testResult) { + t.Errorf( + "F-008 reconstruction guard fired - a peerSharesS gap occurred "+ + "at runtime (would crash upstream): %q", + msg, + ) + } +} + // AssertSuccessfulSignersCount checks the number of successful signers. It does // not check which particular signers were successful. func AssertSuccessfulSignersCount( diff --git a/pkg/internal/dkgtest/capturing_logger.go b/pkg/internal/dkgtest/capturing_logger.go new file mode 100644 index 0000000000..ee05fb4614 --- /dev/null +++ b/pkg/internal/dkgtest/capturing_logger.go @@ -0,0 +1,46 @@ +package dkgtest + +import ( + "fmt" + "sync" + + "github.com/keep-network/keep-core/internal/testutils" +) + +// capturingLogger is a thread-safe log.StandardLogger that records Errorf +// messages emitted during a DKG run, discarding every other level via the +// embedded MockLogger. Byzantine scenarios use it to assert on protocol-internal +// diagnostics that are otherwise invisible (MockLogger drops them) - notably the +// F-008 reconstruction guard's "missing revealed share" Error. Whether that +// Error appears is what distinguishes "no reconstruction gap occurred" from "a +// gap occurred but the defensive guard absorbed it"; without capturing it, a +// non-crashing run cannot tell the two apart. +// +// One instance is shared by every member goroutine in a run, so the mutex is +// load-bearing: members log concurrently. +type capturingLogger struct { + *testutils.MockLogger + mu sync.Mutex + errorf []string +} + +func newCapturingLogger() *capturingLogger { + return &capturingLogger{MockLogger: &testutils.MockLogger{}} +} + +// Errorf overrides the embedded no-op to record the formatted message. +func (l *capturingLogger) Errorf(format string, args ...interface{}) { + l.mu.Lock() + defer l.mu.Unlock() + l.errorf = append(l.errorf, fmt.Sprintf(format, args...)) +} + +// snapshot returns a copy of the captured Errorf messages. Call after all +// member goroutines have finished (no concurrent writers). +func (l *capturingLogger) snapshot() []string { + l.mu.Lock() + defer l.mu.Unlock() + out := make([]string, len(l.errorf)) + copy(out, l.errorf) + return out +} diff --git a/pkg/internal/dkgtest/capturing_logger_test.go b/pkg/internal/dkgtest/capturing_logger_test.go new file mode 100644 index 0000000000..f4c37c4cd3 --- /dev/null +++ b/pkg/internal/dkgtest/capturing_logger_test.go @@ -0,0 +1,44 @@ +package dkgtest + +import "testing" + +// TestCapturingLoggerAndGapDetection proves the F-008 corroboration machinery +// has teeth: the capturing logger records Errorf (and discards other levels), +// and reconstructionGapErrors matches the real guard-message format while +// rejecting unrelated errors. Without this, AssertNoReconstructionGap could be +// vacuously green if the plumbing silently dropped messages. +func TestCapturingLoggerAndGapDetection(t *testing.T) { + l := newCapturingLogger() + + // Non-Errorf levels are discarded (inherited MockLogger no-ops). + l.Infof("info %d", 1) + l.Warnf("warn %d", 2) + if got := len(l.snapshot()); got != 0 { + t.Fatalf("non-error levels should be discarded; captured %d", got) + } + + // Errorf is captured, formatted. This mirrors the guard message emitted by + // gjkr.ComputeGroupPublicKeyShares (protocol.go); keep the marker in sync if + // that log is reworded. + l.Errorf( + "[member:%v] missing revealed share for operating member [%v] from "+ + "misbehaved member [%v]; skipping term (unexpected per DKG invariants)", + 1, 2, 3, + ) + captured := l.snapshot() + if len(captured) != 1 { + t.Fatalf("expected 1 captured Errorf, got %d: %v", len(captured), captured) + } + + // Positive: the guard message is detected as a reconstruction gap. + withGap := &Result{loggedErrors: captured} + if hits := reconstructionGapErrors(withGap); len(hits) != 1 { + t.Errorf("expected the guard message to be detected; got %d hits", len(hits)) + } + + // Negative: an unrelated error is not a false positive. + noGap := &Result{loggedErrors: []string{"[member:1] some unrelated error"}} + if hits := reconstructionGapErrors(noGap); len(hits) != 0 { + t.Errorf("unrelated error must not be flagged as a gap; got %v", hits) + } +} diff --git a/pkg/internal/dkgtest/determinism_probe_test.go b/pkg/internal/dkgtest/determinism_probe_test.go new file mode 100644 index 0000000000..a4b42e20fb --- /dev/null +++ b/pkg/internal/dkgtest/determinism_probe_test.go @@ -0,0 +1,190 @@ +package dkgtest + +// Determinism probe for Tier-2 (Byzantine deterministic-simulation testing) +// work-package 0. Before building a Byzantine-strategy sweep on top of +// dkgtest.RunTest, we must know whether an honest run produces a STABLE VERDICT +// across repetitions. DST only yields trustworthy pass/fail signals if the +// honest baseline is stable; an unstable baseline means every "failure" is +// ambiguous. +// +// What this probe does and does NOT measure: +// +// - It does NOT check value-identity (identical group public key bytes). +// GJKR draws every polynomial coefficient from crypto/rand +// (pkg/beacon/gjkr/protocol.go:265), so the group public key differs on +// every run BY DESIGN, even at a fixed seed (the `seed` arg only becomes a +// session/channel id, not protocol randomness). A seeded DKG would be a +// vulnerability, not a feature. Value-level reproducers are therefore out +// of scope and would require an injected RNG seam. +// +// - It DOES check verdict stability: across N honest runs at a fixed seed, +// does every run reach the same structural end-state (all members succeed, +// zero misbehaving, zero failures, a valid group public key agreed by all +// signers)? +// +// Every run is bucketed into one of three outcomes and the DISTRIBUTION is the +// result (we do not t.Fatal inside the loop): +// +// (a) clean - full success, all structural invariants hold +// (b) timeout-miss - result.dkgResult == nil: the async OnDKGResultSubmitted +// handler missed the 5s wall-clock window in +// executeDKG (line ~190). This is a HARNESS wall-clock +// artifact, amplified by -race and load, NOT protocol +// nondeterminism. RunTest returns a nil error on this path, +// so it must be detected via the nil dkgResult, not err. +// (c) instability - a run that published a result but with a non-clean +// verdict (misbehaving members, member failures, or +// disagreeing/invalid public key). ONLY (c) blocks DST. +// +// Run it explicitly (it is skipped in normal `go test ./...`): +// +// DETERMINISM_PROBE=1 go test ./pkg/internal/dkgtest/ -run TestDeterminismProbe -v -timeout 60m +// DETERMINISM_PROBE=1 DETERMINISM_PROBE_N=200 go test -race ./pkg/internal/dkgtest/ -run TestDeterminismProbe -v -timeout 120m +// +// Compare the (b) timeout-miss rate with and without -race: a large shift +// confirms the timeout (not the protocol) is the nondeterminism source. + +import ( + "encoding/hex" + "math/big" + "os" + "strconv" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/altbn128" + "github.com/keep-network/keep-core/pkg/net" +) + +func TestDeterminismProbe(t *testing.T) { + if os.Getenv("DETERMINISM_PROBE") == "" { + t.Skip("set DETERMINISM_PROBE=1 to run the Tier-2 work-package-0 determinism probe") + } + + const ( + groupSize = 10 + honestThreshold = 6 + ) + + n := 100 + if v := os.Getenv("DETERMINISM_PROBE_N"); v != "" { + parsed, err := strconv.Atoi(v) + if err != nil || parsed < 1 { + t.Fatalf("invalid DETERMINISM_PROBE_N=%q: %v", v, err) + } + n = parsed + } + + // Fixed seed for every iteration: the whole point is to vary nothing the + // caller controls and observe what the protocol/harness still varies. + seed := big.NewInt(0x5EED) + + // Honest interceptor: identity, no message modification or dropping. This is + // the baseline the Byzantine sweep will perturb. + honest := func(msg net.TaggedMarshaler) net.TaggedMarshaler { return msg } + + var ( + clean int + timeoutMiss int + runError int + instability int + ) + // Track distinct group public keys to confirm value-nondeterminism is real + // (we expect ~all distinct), and instability detail for the failing bucket. + pubKeys := make(map[string]struct{}) + var instabilityDetail []string + + t.Logf("probe: %d honest runs, groupSize=%d, honestThreshold=%d, fixed seed=0x%x", + n, groupSize, honestThreshold, seed) + + start := time.Now() + for i := 0; i < n; i++ { + result, err := RunTest(groupSize, honestThreshold, seed, honest) + + switch { + case err != nil: + runError++ + instabilityDetail = append(instabilityDetail, + "run "+strconv.Itoa(i)+": RunTest error: "+err.Error()) + + case result.dkgResult == nil: + // Bucket (b): wall-clock timeout-miss. Harness artifact, not a + // protocol-determinism finding. + timeoutMiss++ + + default: + // Result published; classify the verdict. + successCount := len(result.signers) + failures := len(result.memberFailures) + misbehaved := len(result.dkgResult.Misbehaved) + + pkValid := true + if _, derr := altbn128.DecompressToG2(result.dkgResult.GroupPublicKey); derr != nil { + pkValid = false + } + pubKeys[hex.EncodeToString(result.dkgResult.GroupPublicKey)] = struct{}{} + + // All successful signers must agree on the published group key. + agreed := true + for _, s := range result.signers { + if hex.EncodeToString(s.GroupPublicKeyBytes()) != + hex.EncodeToString(result.dkgResult.GroupPublicKey) { + agreed = false + break + } + } + + isClean := successCount == groupSize && + failures == 0 && + misbehaved == 0 && + pkValid && + agreed + + if isClean { + clean++ + } else { + instability++ + instabilityDetail = append(instabilityDetail, + "run "+strconv.Itoa(i)+": signers="+strconv.Itoa(successCount)+ + " failures="+strconv.Itoa(failures)+ + " misbehaved="+strconv.Itoa(misbehaved)+ + " pkValid="+strconv.FormatBool(pkValid)+ + " agreed="+strconv.FormatBool(agreed)) + } + } + } + elapsed := time.Since(start) + + t.Logf("=== determinism probe distribution (n=%d, %s, %.2fs/run avg) ===", + n, elapsed.Round(time.Second), elapsed.Seconds()/float64(n)) + t.Logf(" (a) clean success : %d", clean) + t.Logf(" (b) timeout-miss : %d (harness wall-clock artifact; not a protocol finding)", timeoutMiss) + t.Logf(" RunTest error : %d", runError) + t.Logf(" (c) verdict instability: %d (BLOCKS DST if > 0)", instability) + t.Logf(" distinct group pubkeys: %d / %d published (expected ~all distinct: crypto/rand)", + len(pubKeys), clean+instability) + + for _, d := range instabilityDetail { + t.Logf(" ! %s", d) + } + + // The gate: only genuine verdict instability (c) or hard errors fail the + // probe. Timeout-misses (b) are reported but do not fail; they characterize + // the harness wall-clock margin, not protocol determinism. + if instability > 0 || runError > 0 { + t.Errorf("honest baseline is NOT verdict-stable: %d instability + %d errors over %d runs; "+ + "DST verdicts would be ambiguous until this is pinned down", instability, runError, n) + } + + // A handful of timeout-misses characterize the wall-clock margin and are + // expected. A MAJORITY of runs failing to publish is no longer a plausible + // margin artifact: it is a liveness / non-delivery signal that would + // otherwise pass silently in bucket (b). Gate on a rate budget so systematic + // non-publication fails the probe instead of hiding as "harness noise". + const timeoutMissBudget = 0.5 // fraction of runs + if float64(timeoutMiss) > timeoutMissBudget*float64(n) { + t.Errorf("timeout-miss rate %d/%d exceeds %.0f%% budget: this is no longer a plausible "+ + "wall-clock artifact but a liveness/non-delivery signal that must be investigated "+ + "(re-run with and without -race to confirm)", timeoutMiss, n, timeoutMissBudget*100) + } +} diff --git a/pkg/internal/dkgtest/dkgtest.go b/pkg/internal/dkgtest/dkgtest.go index 1db67bd27e..51ec1e0532 100644 --- a/pkg/internal/dkgtest/dkgtest.go +++ b/pkg/internal/dkgtest/dkgtest.go @@ -22,10 +22,13 @@ import ( dkgResult "github.com/keep-network/keep-core/pkg/beacon/dkg/result" "github.com/keep-network/keep-core/pkg/beacon/event" "github.com/keep-network/keep-core/pkg/beacon/gjkr" + "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/internal/interception" netLocal "github.com/keep-network/keep-core/pkg/net/local" "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) // Result of a DKG test execution. @@ -34,6 +37,11 @@ type Result struct { dkgResultSignatures map[group.MemberIndex][]byte signers []*dkg.ThresholdSigner memberFailures []error + // loggedErrors holds every Errorf message emitted by the member goroutines + // during the run (captured via capturingLogger). It lets Byzantine + // scenarios assert on protocol-internal diagnostics - e.g. the F-008 + // reconstruction guard - that MockLogger would otherwise discard. + loggedErrors []string } // GetSigners returns all signers created from DKG protocol execution. @@ -43,6 +51,13 @@ func (r *Result) GetSigners() []*dkg.ThresholdSigner { return r.signers } +// LoggedErrors returns the Errorf messages emitted by the member goroutines +// during the run, in capture order. Used by assertions that check whether a +// specific protocol-internal error path was hit. +func (r *Result) LoggedErrors() []string { + return r.loggedErrors +} + // RandomSeed generates a random DKG seed value. It is important to do not // reuse the same seed value between integration tests run in parallel. // Broadcast channel name contains a seed to avoid mixing up channel messages @@ -63,15 +78,78 @@ func RunTest( honestThreshold int, seed *big.Int, rules interception.Rules, +) (*Result, error) { + return RunTestWithStrategy( + groupSize, + honestThreshold, + seed, + interception.FromRules(rules), + ) +} + +// RunTestWithModes executes the full DKG roundtrip test like RunTest, but +// selects each member's compatibility strategy bundle through the given +// selector instead of pinning security-v2 for every member. Homogeneous +// legacy and mixed-mode cutover scenarios use it to prove per-mode protocol +// behavior on the production execution path. +func RunTestWithModes( + groupSize int, + honestThreshold int, + seed *big.Int, + rules interception.Rules, + strategiesForMember func(group.MemberIndex) compatibility.Strategies, +) (*Result, error) { + return runTest( + groupSize, + honestThreshold, + seed, + interception.FromRules(rules), + strategiesForMember, + ) +} + +// RunTestWithStrategy executes the full DKG roundtrip test like RunTest, but +// applies an interception.Strategy instead of the legacy modify-or-drop Rules. +// A Strategy can additionally attribute each message to its sender, duplicate +// it, or inject new messages - the building blocks for Byzantine-operator +// simulation scenarios. RunTest is the special case +// RunTestWithStrategy(..., interception.FromRules(rules)). +// +// The harness pins security-v2 strategies for every member: it exercises the +// hardened protocol behavior end to end. Per-mode cutover coverage selects +// explicit bundles through RunTestWithModes instead. +func RunTestWithStrategy( + groupSize int, + honestThreshold int, + seed *big.Int, + strategy interception.Strategy, +) (*Result, error) { + return runTest( + groupSize, + honestThreshold, + seed, + strategy, + func(group.MemberIndex) compatibility.Strategies { + return compatibility.SecurityV2() + }, + ) +} + +func runTest( + groupSize int, + honestThreshold int, + seed *big.Int, + strategy interception.Strategy, + strategiesForMember func(group.MemberIndex) compatibility.Strategies, ) (*Result, error) { operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair(local_v1.DefaultCurve) if err != nil { return nil, err } - network := interception.NewNetwork( + network := interception.NewNetworkWithStrategy( netLocal.ConnectWithKey(operatorPublicKey), - rules, + strategy, ) localChain := local_v1.ConnectWithKey( @@ -99,6 +177,7 @@ func RunTest( localChain.GetLastDKGResult, network, selectedOperators, + strategiesForMember, ) } @@ -111,6 +190,7 @@ func executeDKG( ), network interception.Network, selectedOperators []chain.Address, + strategiesForMember func(group.MemberIndex) compatibility.Strategies, ) (*Result, error) { beaconConfig := beaconChain.GetConfig() @@ -149,6 +229,23 @@ func executeDKG( // make sure all members are up. startBlockHeight := currentBlockHeight + 3 + // The harness runs every member through a real participation gate with the + // developer-only disabled schedule: each member holds a permit whose + // context and commit fence follow the production execution path, while the + // cryptographic behavior stays selected by the explicit per-member strategy + // bundle. The anchor is the current height because the deliberately future + // execution start block is not a canonical chain anchor. + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + return nil, fmt.Errorf("cannot construct the test gate: [%v]", err) + } + defer gate.Close() + gjkr.RegisterUnmarshallers(broadcastChannel) dkgResult.RegisterUnmarshallers(broadcastChannel) @@ -158,11 +255,29 @@ func executeDKG( beaconChain.Signing(), ) + // One capturing logger shared by all member goroutines, so member-level + // Errorf diagnostics (e.g. the F-008 reconstruction guard) survive the run + // and can be asserted on. Thread-safe; snapshotted after wg.Wait(). + memberLogger := newCapturingLogger() + for i := 0; i < beaconConfig.GroupSize; i++ { memberIndex := group.MemberIndex(i + 1) // capture for goroutine + + permit, err := gate.Begin(participation.BeaconDKG, currentBlockHeight) + if err != nil { + return nil, fmt.Errorf( + "cannot begin the ceremony for member [%v]: [%v]", + memberIndex, + err, + ) + } + go func() { - signer, err := dkg.ExecuteDKG( - &testutils.MockLogger{}, + defer permit.Close() + + signer, _, err := dkg.ExecuteDKG( + permit.Context(), + memberLogger, seed, memberIndex, startBlockHeight, @@ -170,6 +285,8 @@ func executeDKG( broadcastChannel, membershipValidator, selectedOperators, + strategiesForMember(memberIndex), + permit, ) if signer != nil { signersMutex.Lock() @@ -198,19 +315,19 @@ func executeDKG( // result was published to the chain, let's fetch it dkgResult, dkgResultSignatures := lastDKGResultGetter() return &Result{ - dkgResult, - dkgResultSignatures, - signers, - memberFailures, + dkgResult: dkgResult, + dkgResultSignatures: dkgResultSignatures, + signers: signers, + memberFailures: memberFailures, + loggedErrors: memberLogger.snapshot(), }, nil case <-ctx.Done(): // no result published to the chain return &Result{ - nil, - nil, - signers, - memberFailures, + signers: signers, + memberFailures: memberFailures, + loggedErrors: memberLogger.snapshot(), }, nil } } diff --git a/pkg/internal/entrytest/assertions.go b/pkg/internal/entrytest/assertions.go index 1a1cdae08a..f05af9ea6e 100644 --- a/pkg/internal/entrytest/assertions.go +++ b/pkg/internal/entrytest/assertions.go @@ -1,6 +1,11 @@ package entrytest -import "testing" +import ( + "slices" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) // AssertEntryPublished checks if relay entry has been published to the chain. // It does not inspect the entry. @@ -50,3 +55,65 @@ func AssertSignerFailuresCount( ) } } + +// AssertIncorporatedPopulations checks the transcript each signer that +// recovered the entry reported behind it: the memberships whose authenticated +// signature shares it combined. +// +// That population is the whole of what distinguishes a threshold entry several +// parties produced from one a single party recovered among its own kind — a +// relay entry is deterministic for a given previous entry, so every member of a +// finished round names the same result whoever supplied the shares — and it is +// what the release evidence records as the parties to the ceremony. It is +// checked here rather than only where it is written, because only a real round +// establishes that the population comes out of the shares that were actually +// combined. +func AssertIncorporatedPopulations( + t *testing.T, + testResult *Result, + threshold int, + groupSize int, +) { + if len(testResult.populations) == 0 { + t.Fatal("expected at least one signer to report a transcript") + } + + for memberIndex, population := range testResult.populations { + if len(population) != threshold { + t.Errorf( + "signer [%v] combined [%d] memberships into the entry, "+ + "expected the honest threshold of [%d]", + memberIndex, + len(population), + threshold, + ) + } + + if !slices.Contains(population, memberIndex) { + t.Errorf( + "signer [%v] left its own share out of the transcript it "+ + "reported: [%v]", + memberIndex, + population, + ) + } + + // One population must have exactly one rendering, or two members' + // records of the same round would not compare equal and a reader could + // be shown one seat twice. + previous := group.MemberIndex(0) + for _, seat := range population { + if seat <= previous || int(seat) > groupSize { + t.Errorf( + "signer [%v] reported [%v], which is not an ascending set "+ + "of memberships of a group of [%d]", + memberIndex, + population, + groupSize, + ) + break + } + previous = seat + } + } +} diff --git a/pkg/internal/entrytest/entrytest.go b/pkg/internal/entrytest/entrytest.go index 336971b7ef..c425ee17f2 100644 --- a/pkg/internal/entrytest/entrytest.go +++ b/pkg/internal/entrytest/entrytest.go @@ -18,8 +18,11 @@ import ( bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/internal/interception" "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/beacon/dkg" "github.com/keep-network/keep-core/pkg/beacon/entry" @@ -32,6 +35,10 @@ import ( type Result struct { entry []byte signerFailures []error + // populations is the transcript each signer reported behind the entry it + // recovered: the memberships whose authenticated shares it combined. A + // signer that recovered no entry is absent. + populations map[group.MemberIndex]participation.MemberIndexes } // EntryValue returns the value of relay entry from the result as G1 or @@ -125,6 +132,7 @@ func executeSigning( var signerFailuresMutex sync.Mutex var signerFailures []error + populations := make(map[group.MemberIndex]participation.MemberIndexes) var wg sync.WaitGroup wg.Add(len(signers)) @@ -138,11 +146,42 @@ func executeSigning( // make sure all signers are ready startBlockHeight := currentBlockHeight + 3 + // The harness runs every signer through a real participation gate with the + // developer-only disabled schedule: each signer holds a permit whose + // context and commit fence follow the production execution path. The + // anchor is the current height because the deliberately future execution + // start block is not a canonical chain anchor. + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{}, + blockCounter, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + return nil, fmt.Errorf("cannot construct the test gate: [%v]", err) + } + defer gate.Close() + entry.RegisterUnmarshallers(broadcastChannel) for _, signer := range signers { - go func(signer *dkg.ThresholdSigner) { - err := entry.SignAndSubmit( + permit, err := gate.Begin( + participation.BeaconRelaySigning, + currentBlockHeight, + ) + if err != nil { + return nil, fmt.Errorf( + "cannot begin the ceremony for signer [%v]: [%v]", + signer.MemberID(), + err, + ) + } + + go func(signer *dkg.ThresholdSigner, permit participation.Permit) { + defer permit.Close() + + recovered, incorporated, err := entry.SignAndSubmit( + permit.Context(), &testutils.MockLogger{}, blockCounter, broadcastChannel, @@ -151,7 +190,13 @@ func executeSigning( threshold, signer, startBlockHeight, + permit, ) + if len(recovered) != 0 { + signerFailuresMutex.Lock() + populations[signer.MemberID()] = incorporated + signerFailuresMutex.Unlock() + } if err != nil { fmt.Printf("[signer:%v %v] failed with: [%v]\n", signer.MemberID(), previousEntry, err) signerFailuresMutex.Lock() @@ -159,7 +204,7 @@ func executeSigning( signerFailuresMutex.Unlock() } wg.Done() - }(signer) + }(signer, permit) } wg.Wait() @@ -175,6 +220,7 @@ func executeSigning( return &Result{ entry, signerFailures, + populations, }, nil case <-ctx.Done(): @@ -182,6 +228,7 @@ func executeSigning( return &Result{ nil, signerFailures, + populations, }, nil } } diff --git a/pkg/internal/interception/interception.go b/pkg/internal/interception/interception.go index ba52b25465..c93e507850 100644 --- a/pkg/internal/interception/interception.go +++ b/pkg/internal/interception/interception.go @@ -2,39 +2,138 @@ package interception import ( "context" + "sync" "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/group" ) -// Rules defines the rules of intercepting network messages. Messages can be -// returned unmodified, they may be modified on the fly and they can be dropped -// by returning nil. +// Rules defines the legacy modify-or-drop interception contract. A message can +// be returned unmodified, modified on the fly, or dropped by returning nil. +// +// Rules cannot attribute a message to a sender, duplicate it, or inject new +// messages. New Byzantine scenarios should use Strategy; Rules is retained so +// existing callers keep working and is adapted onto Strategy via FromRules. type Rules = func(msg net.TaggedMarshaler) net.TaggedMarshaler -// Network is the local test network implementation capable of -// intercepting network messages and modifying/dropping them based on rules -// passed to the network. +// Outbound is an intercepted outbound message together with its protocol-level +// sender index, extracted from the message payload. Sender is 0 (an invalid +// group.MemberIndex) when the message does not expose a SenderID - i.e. it is +// not a per-member protocol message and cannot be attributed to a group member +// (e.g. a chain-result submission). Strategies that target a specific sender +// should treat Sender == 0 as "not attributable" and leave such messages alone. +type Outbound struct { + Sender group.MemberIndex + Message net.TaggedMarshaler +} + +// Strategy is the Byzantine fault model applied to every outbound message of a +// simulated run. It decides what a single Send becomes on the wire, returning +// the set of messages actually delivered: +// +// - nil / empty slice -> the message is dropped. Models sender inactivity +// or selective withholding. The delegate is never called, so no +// retransmission is scheduled for the dropped message. +// - exactly one message -> pass-through (return out.Message unchanged) or a +// content mutation (return a modified / corrupted message). +// - more than one message -> duplication or injection. Models flooding and +// extra-message attacks. Each message is sent independently and receives +// its own sequence number, so receivers do not deduplicate the copies away. +// +// Invocation contract: Strategy is called EXACTLY ONCE per Send, under a lock +// held by the interceptor. Because all members of a simulated group share a +// single channel and Send concurrently, that lock serializes strategy +// invocations - so a single Strategy value may carry mutable state across calls +// (e.g. "go inactive after phase 2", "flood the next N messages") without +// additional synchronization. The lock does not impose a deterministic message +// ORDER (goroutine scheduling still varies which Send arrives first); it only +// guarantees the strategy never runs concurrently with itself. This matches the +// strategy-level (not byte-level) reproducibility established in Tier-2 +// work-package 0. +// +// The serialization is PER CHANNEL: each BroadcastChannelFor call mints a fresh +// channel with its own lock (see BroadcastChannelFor). A stateful Strategy is +// therefore safe only on the supported topology - one channel per run, shared +// by the whole group. Do not hand the same stateful Strategy value to more than +// one channel of a network; their independent locks would not serialize it. +// +// Boundary - what a Strategy CANNOT do, by construction: it observes a message +// after the sender has serialized and encrypted it. For GJKR peer shares it can +// corrupt or drop the encrypted per-receiver ciphertext (provoking a decryption +// failure -> accusation -> disqualification / recovery, which exercises the +// contested F-008 reconstructed-share path), and it can withhold a message +// entirely. It CANNOT forge a chosen inconsistent-but-individually-valid share, +// because the pairwise i-j symmetric key never appears on the wire. Modeling a +// member that emits internally inconsistent but individually valid values +// requires a malicious gjkr.Member implementation, not channel interception. +type Strategy = func(out Outbound) []net.TaggedMarshaler + +// PassThrough is the identity Strategy: every message is delivered unmodified. +// It is the honest baseline a Byzantine sweep perturbs. +func PassThrough(out Outbound) []net.TaggedMarshaler { + return []net.TaggedMarshaler{out.Message} +} + +// FromRules adapts a legacy modify-or-drop Rules function to a Strategy. A nil +// Rules result becomes an empty (drop) action set; any other result becomes a +// single pass-through / mutated message. +func FromRules(rules Rules) Strategy { + return func(out Outbound) []net.TaggedMarshaler { + altered := rules(out.Message) + if altered == nil { + return nil + } + return []net.TaggedMarshaler{altered} + } +} + +// senderAware is implemented by every per-member protocol message (all GJKR +// message types expose SenderID). The interceptor uses it to attribute an +// outbound message to the group member that produced it, without depending on +// the protocol packages. +type senderAware interface { + SenderID() group.MemberIndex +} + +// Network is the local test network implementation capable of intercepting +// network messages and modifying, dropping, duplicating, or injecting them +// based on a Strategy. type Network interface { BroadcastChannelFor(name string) (net.BroadcastChannel, error) } -// NewNetwork creates a new instance of Network interface implementation with -// message filtering rules passed as a parameter. +// NewNetwork creates a Network applying the legacy modify-or-drop Rules to every +// outbound message. Retained for existing callers; new Byzantine scenarios +// should use NewNetworkWithStrategy. func NewNetwork( provider net.Provider, rules Rules, +) Network { + return NewNetworkWithStrategy(provider, FromRules(rules)) +} + +// NewNetworkWithStrategy creates a Network applying the given Byzantine Strategy +// to every outbound message. +func NewNetworkWithStrategy( + provider net.Provider, + strategy Strategy, ) Network { return &network{ provider: provider, - rules: rules, + strategy: strategy, } } type network struct { provider net.Provider - rules Rules + strategy Strategy } +// BroadcastChannelFor returns a new intercepting channel each call, each with +// its own strategyMutex. The network's Strategy is shared across them, so a +// stateful Strategy is only race-free when a run uses a single channel (the +// supported topology - see the Strategy contract). The current harness +// (dkgtest.RunTestWithStrategy) calls this exactly once per run. func (n *network) BroadcastChannelFor(name string) (net.BroadcastChannel, error) { delegate, err := n.provider.BroadcastChannelFor(name) if err != nil { @@ -42,14 +141,20 @@ func (n *network) BroadcastChannelFor(name string) (net.BroadcastChannel, error) } return &channel{ - delegate, - n.rules, + delegate: delegate, + strategy: n.strategy, }, nil } type channel struct { delegate net.BroadcastChannel - rules Rules + strategy Strategy + + // strategyMutex serializes Strategy invocations. All members of a simulated + // group share one channel and Send concurrently; serializing the strategy + // decision lets a stateful Strategy run without data races. It is held only + // across the strategy call, not across delivery. + strategyMutex sync.Mutex } func (c *channel) Name() string { @@ -61,13 +166,31 @@ func (c *channel) Send( m net.TaggedMarshaler, retransmissionStrategy ...net.RetransmissionStrategy, ) error { - altered := c.rules(m) - if altered == nil { - // drop the message - return nil + out := Outbound{Message: m} + if sa, ok := m.(senderAware); ok { + out.Sender = sa.SenderID() + } + + c.strategyMutex.Lock() + messages := c.strategy(out) // invoked exactly once per Send + c.strategyMutex.Unlock() + + // An empty result drops the message: the delegate is never called, so no + // retransmission is scheduled for it. + for _, message := range messages { + if message == nil { + continue + } + // Each delegate.Send assigns a fresh sequence number, so duplicated or + // injected copies are delivered as distinct messages instead of being + // deduplicated by receivers. The caller's retransmission strategy is + // forwarded (the previous wrapper silently dropped it). + if err := c.delegate.Send(ctx, message, retransmissionStrategy...); err != nil { + return err + } } - return c.delegate.Send(ctx, c.rules(m)) + return nil } func (c *channel) Recv(ctx context.Context, handler func(m net.Message)) { diff --git a/pkg/internal/interception/strategy_test.go b/pkg/internal/interception/strategy_test.go new file mode 100644 index 0000000000..6dd8feb468 --- /dev/null +++ b/pkg/internal/interception/strategy_test.go @@ -0,0 +1,176 @@ +package interception + +import ( + "context" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/net" + netLocal "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// senderTestMessage is a TaggedMarshaler that also exposes a protocol-level +// SenderID, mimicking the GJKR message types the interceptor attributes. +type senderTestMessage struct { + payload string + sender group.MemberIndex +} + +func (m *senderTestMessage) Type() string { return "sender_test_message" } +func (m *senderTestMessage) Marshal() ([]byte, error) { return []byte(m.payload), nil } +func (m *senderTestMessage) Unmarshal(b []byte) error { m.payload = string(b); return nil } +func (m *senderTestMessage) SenderID() group.MemberIndex { + return m.sender +} + +// TestStrategyInvokedExactlyOncePerSend is the regression guard for the +// double-invocation bug in the previous wrapper (it called rules(m) twice per +// Send). A stateful Byzantine strategy must see each Send exactly once. +func TestStrategyInvokedExactlyOncePerSend(t *testing.T) { + var calls int32 + strategy := func(out Outbound) []net.TaggedMarshaler { + atomic.AddInt32(&calls, 1) + return []net.TaggedMarshaler{out.Message} + } + + channel := newStrategyTestChannel(t, strategy) + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { return &testMessage{} }) + ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second) + defer cancel() + + if err := channel.Send(ctx, &testMessage{"hello"}); err != nil { + t.Fatal(err) + } + + if got := atomic.LoadInt32(&calls); got != 1 { + t.Errorf("strategy invoked %d times per Send; want exactly 1", got) + } +} + +// TestStrategyExtractsSender confirms the interceptor attributes an outbound +// message to its protocol-level sender, and reports 0 for non-attributable +// messages. +func TestStrategyExtractsSender(t *testing.T) { + var seen group.MemberIndex + strategy := func(out Outbound) []net.TaggedMarshaler { + seen = out.Sender + return []net.TaggedMarshaler{out.Message} + } + + channel := newStrategyTestChannel(t, strategy) + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { return &senderTestMessage{} }) + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { return &testMessage{} }) + ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second) + defer cancel() + + if err := channel.Send(ctx, &senderTestMessage{"from-7", group.MemberIndex(7)}); err != nil { + t.Fatal(err) + } + if seen != group.MemberIndex(7) { + t.Errorf("extracted sender = %d; want 7", seen) + } + + if err := channel.Send(ctx, &testMessage{"no-sender"}); err != nil { + t.Fatal(err) + } + if seen != group.MemberIndex(0) { + t.Errorf("sender for a non-attributable message = %d; want 0", seen) + } +} + +// TestStrategyDuplicateDelivers confirms a strategy returning N copies results +// in N distinct messages at the receiver (distinct seqnos, not deduped away), +// and that an empty result drops the message entirely. +func TestStrategyDuplicateDelivers(t *testing.T) { + tests := map[string]struct { + strategy Strategy + wantCount int + }{ + "pass-through": { + strategy: PassThrough, + wantCount: 1, + }, + "drop": { + strategy: func(Outbound) []net.TaggedMarshaler { return nil }, + wantCount: 0, + }, + "triplicate (flood)": { + strategy: func(out Outbound) []net.TaggedMarshaler { + return []net.TaggedMarshaler{out.Message, out.Message, out.Message} + }, + wantCount: 3, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + channel := newStrategyTestChannel(t, test.strategy) + + ctx, cancel := context.WithTimeout(context.Background(), 750*time.Millisecond) + defer cancel() + + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { return &testMessage{} }) + + var received int32 + channel.Recv(ctx, func(net.Message) { atomic.AddInt32(&received, 1) }) + + if err := channel.Send(ctx, &testMessage{"flood-me"}); err != nil { + t.Fatal(err) + } + + <-ctx.Done() // let retransmissions settle; dedup keeps the count stable + if got := int(atomic.LoadInt32(&received)); got != test.wantCount { + t.Errorf("received %d distinct messages; want %d", got, test.wantCount) + } + }) + } +} + +// TestStrategyConcurrentStatefulNoRace drives many concurrent Sends through a +// stateful strategy (as the shared-channel group does) to confirm the +// interceptor's lock lets a Strategy carry state without a data race. Run with +// -race for the assertion to have teeth. +func TestStrategyConcurrentStatefulNoRace(t *testing.T) { + const sends = 100 + + // A deliberately non-atomic counter: correctness here depends entirely on + // the interceptor serializing strategy invocations. + count := 0 + strategy := func(out Outbound) []net.TaggedMarshaler { + count++ + return []net.TaggedMarshaler{out.Message} + } + + channel := newStrategyTestChannel(t, strategy) + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + + var wg sync.WaitGroup + wg.Add(sends) + for i := 0; i < sends; i++ { + go func() { + defer wg.Done() + _ = channel.Send(ctx, &testMessage{"concurrent"}) + }() + } + wg.Wait() + + if count != sends { + t.Errorf("stateful strategy counted %d invocations; want %d", count, sends) + } +} + +func newStrategyTestChannel(t *testing.T, strategy Strategy) net.BroadcastChannel { + t.Helper() + // t.Name() is unique per (sub)test, isolating this channel from others in + // the process-global local broadcast registry (keyed by name). + channel, err := NewNetworkWithStrategy(netLocal.Connect(), strategy). + BroadcastChannelFor(t.Name()) + if err != nil { + t.Fatal(err) + } + return channel +} diff --git a/pkg/internal/signingtest/assertions.go b/pkg/internal/signingtest/assertions.go new file mode 100644 index 0000000000..8c58dd48a4 --- /dev/null +++ b/pkg/internal/signingtest/assertions.go @@ -0,0 +1,82 @@ +package signingtest + +import ( + "crypto/ecdsa" + "math/big" + "testing" +) + +// AssertSignatureGenerated checks how many members produced a signature. +func AssertSignatureGenerated(t *testing.T, result *Result, expectedCount int) { + if len(result.signatures) != expectedCount { + t.Errorf( + "unexpected number of produced signatures\nexpected: [%v]\nactual: [%v]", + expectedCount, + len(result.signatures), + ) + } +} + +// AssertMemberFailuresCount checks how many members failed to complete. +func AssertMemberFailuresCount(t *testing.T, result *Result, expectedCount int) { + if len(result.memberFailures) != expectedCount { + t.Errorf( + "unexpected number of member failures\nexpected: [%v]\nactual: [%v]\nerrors: %v", + expectedCount, + len(result.memberFailures), + result.memberFailures, + ) + } +} + +// AssertSameSignature checks that every member that completed produced the +// identical signature - the core agreement invariant of threshold signing. +func AssertSameSignature(t *testing.T, result *Result) { + if len(result.signatures) < 2 { + return + } + first := result.signatures[0] + for i, sig := range result.signatures[1:] { + if !first.Equals(sig) { + t.Errorf( + "signatures disagree: member-result[0] != member-result[%d]\n[0]: %s\n[%d]: %s", + i+1, first, i+1, sig, + ) + } + } +} + +// AssertNoDivergentSignatures is the safety invariant for Byzantine scenarios: +// regardless of how many members complete (a disrupted signing session may +// produce zero), no two members may ever output DIFFERENT signatures. A +// Byzantine participant may cause a denial of service, but must never split the +// group onto conflicting signatures. +func AssertNoDivergentSignatures(t *testing.T, result *Result) { + for i := 1; i < len(result.signatures); i++ { + if !result.signatures[0].Equals(result.signatures[i]) { + t.Errorf( + "SAFETY VIOLATION: divergent signatures produced\n[0]: %s\n[%d]: %s", + result.signatures[0], i, result.signatures[i], + ) + } + } +} + +// AssertValidSignature checks that every produced signature verifies against +// the group public key for the signed message. publicKey is the ECDSA group key +// the fixture shares correspond to (see GroupPublicKey). +func AssertValidSignature( + t *testing.T, + result *Result, + publicKey *ecdsa.PublicKey, + message *big.Int, +) { + for i, sig := range result.signatures { + if !ecdsa.Verify(publicKey, message.Bytes(), sig.R, sig.S) { + t.Errorf( + "signature %d does not verify against the group public key: %s", + i, sig, + ) + } + } +} diff --git a/pkg/internal/signingtest/signingtest.go b/pkg/internal/signingtest/signingtest.go new file mode 100644 index 0000000000..6babca891d --- /dev/null +++ b/pkg/internal/signingtest/signingtest.go @@ -0,0 +1,203 @@ +// Package signingtest provides a full-roundtrip tECDSA signing test engine, +// the signing analogue of dkgtest. All members run signing.Execute against a +// single shared local broadcast channel; an optional interception.Strategy +// lets a test inject Byzantine behavior (drop / mutate / duplicate / inject). +// +// It is the first whole-signing-protocol harness in the tree - the per-round +// unit tests in pkg/tecdsa/signing stress phases individually and the file's +// own TODO asks for an integration test of the whole protocol. Member private +// key shares come from the committed tECDSA fixtures +// (tecdsatest.LoadPrivateKeyShareTestFixtures), so no expensive tECDSA DKG runs +// per test. +// +// Unlike dkgtest (block-driven SyncMachine, ~37.5s/run), signing uses the +// message-driven AsyncMachine, so an honest roundtrip completes in seconds. +package signingtest + +import ( + "fmt" + "math/big" + "sync" + + "github.com/keep-network/keep-core/internal/testutils" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/internal/interception" + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + netLocal "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/tecdsa" + "github.com/keep-network/keep-core/pkg/tecdsa/signing" + + "context" + "time" +) + +// maxFixtures is the number of committed private-key-share fixtures +// (private_key_share_data_0..4.json), and thus the maximum group size. +const maxFixtures = 5 + +// Result of a signing test execution. signatures holds the signature produced +// by each member that completed; memberFailures holds the error from each +// member that did not. +type Result struct { + signatures []*tecdsa.Signature + memberFailures []error +} + +// GetSignatures returns the signatures produced by members that completed the +// protocol. Order is nondeterministic (members complete concurrently). +func (r *Result) GetSignatures() []*tecdsa.Signature { + return r.signatures +} + +// GetMemberFailures returns the errors from members that did not complete. +func (r *Result) GetMemberFailures() []error { + return r.memberFailures +} + +// RunTest executes the full tECDSA signing protocol for the given message over +// a group of groupSize members (loaded from key-share fixtures), applying the +// provided interception.Strategy to the shared broadcast channel. Pass +// interception.PassThrough for an honest run. Uses a 60s execution bound; an +// honest run finishes in seconds and never approaches it. +func RunTest( + message *big.Int, + groupSize int, + dishonestThreshold int, + strategy interception.Strategy, +) (*Result, error) { + return RunTestWithTimeout(message, groupSize, dishonestThreshold, 60*time.Second, strategy) +} + +// RunTestWithTimeout is RunTest with an explicit execution bound. Use a short +// timeout for Byzantine scenarios where a withheld or corrupted message leaves +// peers waiting (they cannot complete and block until the bound fires); the +// timeout bounds how long that denial-of-service takes to observe. +func RunTestWithTimeout( + message *big.Int, + groupSize int, + dishonestThreshold int, + timeout time.Duration, + strategy interception.Strategy, +) (*Result, error) { + if groupSize < 1 || groupSize > maxFixtures { + return nil, fmt.Errorf( + "groupSize %d out of range [1,%d] (available key-share fixtures)", + groupSize, maxFixtures, + ) + } + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(groupSize) + if err != nil { + return nil, fmt.Errorf("failed to load key-share fixtures: [%v]", err) + } + + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair(local_v1.DefaultCurve) + if err != nil { + return nil, err + } + + network := interception.NewNetworkWithStrategy( + netLocal.ConnectWithKey(operatorPublicKey), + strategy, + ) + + // The local chain is used only for its Signing() (public-key-to-address + // conversion) so the membership validator can be built. signing.Execute + // itself takes no chain - it is driven entirely by the broadcast channel. + localChain := local_v1.ConnectWithKey( + groupSize, + groupSize-dishonestThreshold, + operatorPrivateKey, + ) + + address, err := localChain.Signing().PublicKeyToAddress(operatorPublicKey) + if err != nil { + return nil, fmt.Errorf( + "cannot convert operator public key to chain address: [%v]", + err, + ) + } + + selectedOperators := make([]chain.Address, groupSize) + for i := range selectedOperators { + selectedOperators[i] = address + } + + broadcastChannel, err := network.BroadcastChannelFor( + fmt.Sprintf("signing-test-%v", message), + ) + if err != nil { + return nil, err + } + signing.RegisterUnmarshallers(broadcastChannel) + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + selectedOperators, + localChain.Signing(), + ) + + // Prefix with a fixed label so the session ID always clears tss-lib's + // 16-byte minimum-length floor (hardened in #8), regardless of how small + // the test message's hex encoding is. The message hex still keeps the ID + // unique per signed message, and all members derive the same value. + sessionID := "signingtest-session-" + message.Text(16) + + ctx, cancel := context.WithTimeout(context.Background(), timeout) + defer cancel() + + var mutex sync.Mutex + var signatures []*tecdsa.Signature + var memberFailures []error + + var wg sync.WaitGroup + wg.Add(groupSize) + for i := 0; i < groupSize; i++ { + memberIndex := group.MemberIndex(i + 1) + privateKeyShare := tecdsa.NewPrivateKeyShare(testData[i]) + go func() { + defer wg.Done() + result, err := signing.Execute( + ctx, + &testutils.MockLogger{}, + message, + sessionID, + memberIndex, + privateKeyShare, + groupSize, + dishonestThreshold, + []group.MemberIndex{}, // no statically-excluded members + broadcastChannel, + membershipValidator, + compatibility.SecurityV2(), + ) + + mutex.Lock() + defer mutex.Unlock() + if result != nil { + signatures = append(signatures, result.Signature) + } + if err != nil { + memberFailures = append(memberFailures, err) + } + }() + } + wg.Wait() + + return &Result{signatures: signatures, memberFailures: memberFailures}, nil +} + +// GroupPublicKey returns the ECDSA public key the fixture key shares correspond +// to, for verifying produced signatures. It loads the first fixture only. +func GroupPublicKey() (*tecdsa.PrivateKeyShare, error) { + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + return nil, err + } + return tecdsa.NewPrivateKeyShare(testData[0]), nil +} diff --git a/pkg/maintainer/btcdiff/bitcoin_chain_test.go b/pkg/maintainer/btcdiff/bitcoin_chain_test.go index 348502e8be..5fbc9a6ec7 100644 --- a/pkg/maintainer/btcdiff/bitcoin_chain_test.go +++ b/pkg/maintainer/btcdiff/bitcoin_chain_test.go @@ -2,6 +2,7 @@ package btcdiff import ( "fmt" + "sync" "github.com/keep-network/keep-core/pkg/bitcoin" ) @@ -10,7 +11,11 @@ var errNoBlocksSet = fmt.Errorf("blockchain does not contain any blocks") // localBitcoinChain represents a local Bitcoin chain. type localBitcoinChain struct { - blockHeaders map[uint]*bitcoin.BlockHeader + // blockHeadersMutex guards blockHeaders. The maintainer reads it from its + // proving goroutine (GetLatestBlockHeight, GetBlockHeader) while the test's + // main goroutine replaces it via SetBlockHeaders. + blockHeadersMutex sync.Mutex + blockHeaders map[uint]*bitcoin.BlockHeader } // GetTransaction gets the transaction with the given transaction hash. @@ -45,6 +50,9 @@ func (lbc *localBitcoinChain) BroadcastTransaction( // GetLatestBlockHeight gets the height of the latest block (tip). If the // latest block was not determined, this function returns an error. func (lbc *localBitcoinChain) GetLatestBlockHeight() (uint, error) { + lbc.blockHeadersMutex.Lock() + defer lbc.blockHeadersMutex.Unlock() + blockchainTip := uint(0) for blockHeaderHeight := range lbc.blockHeaders { if blockHeaderHeight > blockchainTip { @@ -65,6 +73,9 @@ func (lbc *localBitcoinChain) GetLatestBlockHeight() (uint, error) { func (lbc *localBitcoinChain) GetBlockHeader( blockNumber uint, ) (*bitcoin.BlockHeader, error) { + lbc.blockHeadersMutex.Lock() + defer lbc.blockHeadersMutex.Unlock() + blockHeader, found := lbc.blockHeaders[blockNumber] if !found { return nil, fmt.Errorf( @@ -118,6 +129,9 @@ func (lbc *localBitcoinChain) GetMempoolUtxosForPublicKeyHash( func (lbc *localBitcoinChain) SetBlockHeaders( blockHeaders map[uint]*bitcoin.BlockHeader, ) { + lbc.blockHeadersMutex.Lock() + defer lbc.blockHeadersMutex.Unlock() + lbc.blockHeaders = blockHeaders } diff --git a/pkg/maintainer/btcdiff/chain_test.go b/pkg/maintainer/btcdiff/chain_test.go index 691a50f328..887bc0e277 100644 --- a/pkg/maintainer/btcdiff/chain_test.go +++ b/pkg/maintainer/btcdiff/chain_test.go @@ -2,6 +2,7 @@ package btcdiff import ( "math/big" + "sync" "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/chain" @@ -18,6 +19,12 @@ type RetargetEvent struct { type localBitcoinDifficultyChain struct { operatorPrivateKey *operator.PrivateKey + // mutex guards the mutable fields below. The maintainer runs its proving + // loop in a separate goroutine that reads this state (CurrentEpoch, Ready, + // Retarget, ...) while the test's main goroutine mutates it, so every + // accessor must synchronize. + mutex sync.Mutex + currentEpoch uint64 proofLength uint64 @@ -31,6 +38,9 @@ type localBitcoinDifficultyChain struct { // Ready checks whether the relay is active (i.e. genesis has been performed). func (lbdc *localBitcoinDifficultyChain) Ready() (bool, error) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + return lbdc.ready, nil } @@ -40,6 +50,9 @@ func (lbdc *localBitcoinDifficultyChain) Ready() (bool, error) { func (lbdc *localBitcoinDifficultyChain) IsAuthorized( address chain.Address, ) (bool, error) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + return lbdc.authorizedOperators[address], nil } @@ -50,6 +63,9 @@ func (lbdc *localBitcoinDifficultyChain) IsAuthorized( func (lbdc *localBitcoinDifficultyChain) IsAuthorizedForRefund( address chain.Address, ) (bool, error) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + return lbdc.authorizedForRefundOperators[address], nil } @@ -63,6 +79,9 @@ func (lbdc *localBitcoinDifficultyChain) Signing() chain.Signing { func (lbdc *localBitcoinDifficultyChain) Retarget( headers []*bitcoin.BlockHeader, ) error { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + // For simplicity, store block header bits instead of their difficulty // targets. retargetEvent := &RetargetEvent{ @@ -82,6 +101,9 @@ func (lbdc *localBitcoinDifficultyChain) Retarget( func (lbdc *localBitcoinDifficultyChain) RetargetWithRefund( headers []*bitcoin.BlockHeader, ) error { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + // For simplicity, store block header bits instead of their difficulty // targets. retargetEvent := &RetargetEvent{ @@ -103,12 +125,18 @@ func (lbdc *localBitcoinDifficultyChain) RetargetWithRefund( // retargets along the way have been legitimate, this equals the height of // the block starting the most recent epoch, divided by 2016. func (lbdc *localBitcoinDifficultyChain) CurrentEpoch() (uint64, error) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + return lbdc.currentEpoch, nil } // ProofLength returns the number of blocks required for each side of a // retarget proof. func (lbdc *localBitcoinDifficultyChain) ProofLength() (uint64, error) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + return lbdc.proofLength, nil } @@ -122,6 +150,9 @@ func (lbdc *localBitcoinDifficultyChain) GetCurrentAndPrevEpochDifficulty() ( // SetReady sets chain's status as either ready or not. func (lbdc *localBitcoinDifficultyChain) SetReady(ready bool) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + lbdc.ready = ready } @@ -131,6 +162,9 @@ func (lbdc *localBitcoinDifficultyChain) SetAuthorizedOperator( operatorAddress chain.Address, authorized bool, ) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + lbdc.authorizedOperators[operatorAddress] = authorized } @@ -140,27 +174,50 @@ func (lbdc *localBitcoinDifficultyChain) SetAuthorizedForRefundOperator( operatorAddress chain.Address, authorized bool, ) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + lbdc.authorizedForRefundOperators[operatorAddress] = authorized } // SetCurrentEpoch sets the current proven epoch in the chain. func (lbdc *localBitcoinDifficultyChain) SetCurrentEpoch(currentEpoch uint64) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + lbdc.currentEpoch = currentEpoch } // SetProofLength sets the proof length needed for a retarget. func (lbdc *localBitcoinDifficultyChain) SetProofLength(proofLength uint64) { + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + lbdc.proofLength = proofLength } // RetargetEvents returns all invocations of the Retarget method. func (lbdc *localBitcoinDifficultyChain) RetargetEvents() []*RetargetEvent { - return lbdc.retargetEvents + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + + // Return a snapshot so callers can iterate without racing a concurrent + // Retarget append. + events := make([]*RetargetEvent, len(lbdc.retargetEvents)) + copy(events, lbdc.retargetEvents) + return events } // RetargetWithRefundEvents returns all invocations of the Retarget method. func (lbdc *localBitcoinDifficultyChain) RetargetWithRefundEvents() []*RetargetEvent { - return lbdc.retargetWithRefundEvents + lbdc.mutex.Lock() + defer lbdc.mutex.Unlock() + + // Return a snapshot so callers can iterate without racing a concurrent + // RetargetWithRefund append. + events := make([]*RetargetEvent, len(lbdc.retargetWithRefundEvents)) + copy(events, lbdc.retargetWithRefundEvents) + return events } // connectLocalBitcoinDifficultyChain connects to the local Bitcoin difficulty diff --git a/pkg/maintainer/maintainer.go b/pkg/maintainer/maintainer.go index ea2fbfabcf..8d89eb651f 100644 --- a/pkg/maintainer/maintainer.go +++ b/pkg/maintainer/maintainer.go @@ -46,9 +46,17 @@ func Initialize( ) } + // The blast radius of a panic here is this dedicated `keep-client + // maintainer` process, which hosts the co-resident SPV and + // Bitcoin-difficulty maintainers - not the separate `keep-client start` + // beacon/tBTC operator. The SPV maintainer recovers panics per iteration + // and restarts after a backoff (see spvMaintainer.runMaintainSpv), so a + // panic in a single SPV iteration no longer terminates this process. + // // TODO: Allow for launching multiple maintainers here. Every flag // indicating a maintainer task should launch a separate maintainer. - // Notice that panic on one maintainer goroutine will crush the whole - // program. Consider cancelling all maintainers if one maintainer - // cannot ba launched due to a configuration error. + // A panic in a maintainer without its own recovery boundary still + // terminates this process; extend per-iteration recovery to the + // other maintainers. Consider cancelling all maintainers if one + // maintainer cannot be launched due to a configuration error. } diff --git a/pkg/maintainer/spv/bitcoin_chain_test.go b/pkg/maintainer/spv/bitcoin_chain_test.go index 266128a94d..3545282622 100644 --- a/pkg/maintainer/spv/bitcoin_chain_test.go +++ b/pkg/maintainer/spv/bitcoin_chain_test.go @@ -250,3 +250,72 @@ func (lbc *localBitcoinChain) addTransactionConfirmations( return nil } + +// countingHeaderGetter wraps a header getter (typically +// localBitcoinChain.GetBlockHeader) with a per-height call counter and optional +// per-height failure injection. Tests use it to assert how many times the +// backend is hit and that the header cache does not cache failures. +type countingHeaderGetter struct { + inner func(uint) (*bitcoin.BlockHeader, error) + mutex sync.Mutex + calls map[uint]int + failuresLeft map[uint]int +} + +func newCountingHeaderGetter( + inner func(uint) (*bitcoin.BlockHeader, error), +) *countingHeaderGetter { + return &countingHeaderGetter{ + inner: inner, + calls: make(map[uint]int), + failuresLeft: make(map[uint]int), + } +} + +// get records the call and either injects a pending failure for the height or +// delegates to the wrapped getter. +func (c *countingHeaderGetter) get(blockHeight uint) ( + *bitcoin.BlockHeader, + error, +) { + c.mutex.Lock() + c.calls[blockHeight]++ + if c.failuresLeft[blockHeight] > 0 { + c.failuresLeft[blockHeight]-- + c.mutex.Unlock() + return nil, fmt.Errorf( + "injected header failure at height [%d]", + blockHeight, + ) + } + c.mutex.Unlock() + + return c.inner(blockHeight) +} + +// failNext makes the next `times` calls for the given height return an error +// before the wrapped getter is consulted. +func (c *countingHeaderGetter) failNext(blockHeight uint, times int) { + c.mutex.Lock() + defer c.mutex.Unlock() + c.failuresLeft[blockHeight] = times +} + +// callsAt returns the number of get calls recorded for the given height. +func (c *countingHeaderGetter) callsAt(blockHeight uint) int { + c.mutex.Lock() + defer c.mutex.Unlock() + return c.calls[blockHeight] +} + +// totalCalls returns the total number of get calls across all heights. +func (c *countingHeaderGetter) totalCalls() int { + c.mutex.Lock() + defer c.mutex.Unlock() + + total := 0 + for _, n := range c.calls { + total += n + } + return total +} diff --git a/pkg/maintainer/spv/control_loop_test.go b/pkg/maintainer/spv/control_loop_test.go new file mode 100644 index 0000000000..bf15edbc9e --- /dev/null +++ b/pkg/maintainer/spv/control_loop_test.go @@ -0,0 +1,114 @@ +package spv + +import ( + "context" + "errors" + "strings" + "sync/atomic" + "testing" + "time" +) + +// TestRunMaintainSpvRecoversPanic asserts that a panic inside a maintainer +// iteration is recovered and surfaced as an error instead of escaping the +// goroutine. +func TestRunMaintainSpvRecoversPanic(t *testing.T) { + sm := &spvMaintainer{} + + err := sm.runMaintainSpv( + context.Background(), + func(context.Context) error { + panic("sentinel panic") + }, + ) + + if err == nil { + t.Fatal("expected a non-nil error from a recovered panic") + } + if !strings.Contains(err.Error(), "sentinel panic") { + t.Fatalf("expected the error to mention the panic value, got [%v]", err) + } +} + +// TestRunMaintainSpvPassesThroughError asserts that an ordinary error returned +// by the iteration is preserved unchanged by the recovery wrapper. +func TestRunMaintainSpvPassesThroughError(t *testing.T) { + sentinel := errors.New("ordinary maintainer error") + + sm := &spvMaintainer{} + + err := sm.runMaintainSpv( + context.Background(), + func(context.Context) error { + return sentinel + }, + ) + + if !errors.Is(err, sentinel) { + t.Fatalf("expected the sentinel error to pass through, got [%v]", err) + } +} + +// TestRunControlLoopRestartsAfterRecoveredPanic asserts that the control loop +// recovers a panicking iteration, waits the restart backoff, and invokes the +// iteration again, then exits promptly on context cancellation. Synchronization +// uses channels rather than sleeps to stay deterministic. +func TestRunControlLoopRestartsAfterRecoveredPanic(t *testing.T) { + sm := &spvMaintainer{ + config: Config{RestartBackoffTime: time.Millisecond}, + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Buffered generously so a late iteration after cancellation can never block + // the loop goroutine on send. + invocations := make(chan int, 8) + var count int32 + + iteration := func(context.Context) error { + n := atomic.AddInt32(&count, 1) + invocations <- int(n) + if n == 1 { + panic("first-iteration panic") + } + // Later invocations block until the context is cancelled, mimicking the + // real maintainSpv steady state. + <-ctx.Done() + return ctx.Err() + } + + done := make(chan struct{}) + go func() { + sm.runControlLoop(ctx, iteration) + close(done) + }() + + // First invocation panics and is recovered. + select { + case n := <-invocations: + if n != 1 { + t.Fatalf("expected first invocation, got [%d]", n) + } + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for the first invocation") + } + + // After the restart backoff, the loop invokes the iteration again. + select { + case n := <-invocations: + if n != 2 { + t.Fatalf("expected a second invocation after restart, got [%d]", n) + } + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for the restart invocation") + } + + // Cancellation exits the loop promptly and does not spin into a restart. + cancel() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for the control loop to exit") + } +} diff --git a/pkg/maintainer/spv/deposit_sweep.go b/pkg/maintainer/spv/deposit_sweep.go index f8405e1576..43a5e88dc8 100644 --- a/pkg/maintainer/spv/deposit_sweep.go +++ b/pkg/maintainer/spv/deposit_sweep.go @@ -153,8 +153,18 @@ func parseDepositSweepTransactionInputs( ) } - publicKeyScript := previousTransaction.Outputs[outpointIndex].PublicKeyScript - value := previousTransaction.Outputs[outpointIndex].Value + // Bounds-checked output access (OOB hardening) combined with the + // btcd script-type classification isolated in pkg/bitcoin (#4165). + previousOutput, err := previousTransaction.OutputAt(outpointIndex) + if err != nil { + return bitcoin.UnspentTransactionOutput{}, common.Address{}, fmt.Errorf( + "failed to read previous transaction output: [%v]", + err, + ) + } + + publicKeyScript := previousOutput.PublicKeyScript + value := previousOutput.Value scriptType := bitcoin.GetScriptType(publicKeyScript) if scriptType == bitcoin.P2PKHScript || diff --git a/pkg/maintainer/spv/header_cache_test.go b/pkg/maintainer/spv/header_cache_test.go new file mode 100644 index 0000000000..57d4bb3991 --- /dev/null +++ b/pkg/maintainer/spv/header_cache_test.go @@ -0,0 +1,380 @@ +package spv + +import ( + "context" + "errors" + "math/big" + "sync/atomic" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/tbtc" +) + +func TestBlockHeaderCache(t *testing.T) { + header := func(h uint) *bitcoin.BlockHeader { + return &bitcoin.BlockHeader{Bits: uint32(0x1d000000 + h)} + } + + t.Run("repeated successful lookup hits the backend once", func(t *testing.T) { + getter := newCountingHeaderGetter( + func(h uint) (*bitcoin.BlockHeader, error) { + return header(h), nil + }, + ) + cache := newBlockHeaderCache(getter.get) + + first, err := cache.getBlockHeader(100) + if err != nil { + t.Fatal(err) + } + second, err := cache.getBlockHeader(100) + if err != nil { + t.Fatal(err) + } + + if got := getter.totalCalls(); got != 1 { + t.Fatalf("expected 1 backend call, got [%d]", got) + } + if first != second { + t.Fatal("expected the same cached header value on repeated lookup") + } + }) + + t.Run("distinct heights hit the backend once each", func(t *testing.T) { + getter := newCountingHeaderGetter( + func(h uint) (*bitcoin.BlockHeader, error) { + return header(h), nil + }, + ) + cache := newBlockHeaderCache(getter.get) + + if _, err := cache.getBlockHeader(100); err != nil { + t.Fatal(err) + } + if _, err := cache.getBlockHeader(101); err != nil { + t.Fatal(err) + } + if _, err := cache.getBlockHeader(100); err != nil { + t.Fatal(err) + } + + if got := getter.totalCalls(); got != 2 { + t.Fatalf( + "expected 2 backend calls for 2 distinct heights, got [%d]", + got, + ) + } + if got := getter.callsAt(100); got != 1 { + t.Fatalf("expected height 100 fetched once, got [%d]", got) + } + if got := getter.callsAt(101); got != 1 { + t.Fatalf("expected height 101 fetched once, got [%d]", got) + } + }) + + t.Run("errors are not cached", func(t *testing.T) { + getter := newCountingHeaderGetter( + func(h uint) (*bitcoin.BlockHeader, error) { + return header(h), nil + }, + ) + getter.failNext(100, 1) + cache := newBlockHeaderCache(getter.get) + + if _, err := cache.getBlockHeader(100); err == nil { + t.Fatal("expected an error on the first lookup") + } + got, err := cache.getBlockHeader(100) + if err != nil { + t.Fatalf("expected success on retry, got [%v]", err) + } + if got == nil { + t.Fatal("expected a header on retry") + } + + if calls := getter.callsAt(100); calls != 2 { + t.Fatalf( + "expected 2 backend calls (error not cached), got [%d]", + calls, + ) + } + }) +} + +// TestGetProofInfoUsesPassHeaderCache proves that a single pass-scoped cache +// shared across transactions with overlapping proof windows fetches each +// distinct height from the backend exactly once, and that a fresh cache on the +// next pass refetches. It also pins that the cache imposes no proof-length cap: +// walks proceed unchanged, only their backend reads are deduplicated. +func TestGetProofInfoUsesPassHeaderCache(t *testing.T) { + const proofStart = 790270 + + txA, err := bitcoin.NewHashFromString( + "44c568bc0eac07a2a9c2b46829be5b5d46e7d00e17bfb613f506a75ccf86a473", + bitcoin.InternalByteOrder, + ) + if err != nil { + t.Fatal(err) + } + txB, err := bitcoin.NewHashFromString( + "1111111111111111111111111111111111111111111111111111111111111111", + bitcoin.InternalByteOrder, + ) + if err != nil { + t.Fatal(err) + } + + localChain := newLocalChain() + localChain.setTxProofDifficultyFactor(big.NewInt(6)) + localChain.setCurrentEpoch(392) + localChain.setCurrentAndPrevEpochDifficulty(big.NewInt(32), big.NewInt(16)) + + btcChain := newLocalBitcoinChain() + // 20 headers of difficulty 32, so both transactions bind to the current + // epoch and each needs 6 headers (6*32 = 192). + if err := populateBlockHeaders( + btcChain, + proofStart, + proofStart+19, + func(uint) *big.Int { return big.NewInt(32) }, + ); err != nil { + t.Fatal(err) + } + // latestBlockHeight = proofStart+19 = 790289. + // txA: 20 confirmations -> walks 790270..790275. + // txB: 18 confirmations -> walks 790272..790277 (overlaps 790272..790275). + // Distinct heights across both walks: 790270..790277 = 8. + btcChain.addTransactionConfirmations(txA, 20) + btcChain.addTransactionConfirmations(txB, 18) + + getter := newCountingHeaderGetter(btcChain.GetBlockHeader) + + proveWith := func(cache *blockHeaderCache, txHash bitcoin.Hash) { + withinRange, _, required, err := getProofInfo( + txHash, + btcChain, + localChain, + localChain, + cache, + ) + if err != nil { + t.Fatal(err) + } + if !withinRange { + t.Fatal("expected transaction proof within relay range") + } + if required != 6 { + t.Fatalf("expected required confirmations 6, got [%d]", required) + } + } + + // One pass: both transactions share a single cache. The backend is hit once + // per distinct height across the two overlapping walks. + passCache := newBlockHeaderCache(getter.get) + proveWith(passCache, txA) + proveWith(passCache, txB) + + if got := getter.totalCalls(); got != 8 { + t.Fatalf( + "expected 8 backend calls for 8 distinct heights in one pass, "+ + "got [%d]", + got, + ) + } + for h := uint(proofStart); h <= proofStart+7; h++ { + if got := getter.callsAt(h); got != 1 { + t.Fatalf( + "expected height [%d] fetched once in the pass, got [%d]", + h, + got, + ) + } + } + + // A new pass uses a fresh cache and refetches the overlapping heights. + nextPassCache := newBlockHeaderCache(getter.get) + proveWith(nextPassCache, txA) + + if got := getter.totalCalls(); got != 14 { + t.Fatalf( + "expected 14 total backend calls after a second-pass refetch, "+ + "got [%d]", + got, + ) + } + if got := getter.callsAt(proofStart); got != 2 { + t.Fatalf( + "expected height [%d] fetched once per pass (2 total), got [%d]", + uint(proofStart), + got, + ) + } + if got := getter.callsAt(proofStart + 7); got != 1 { + t.Fatalf( + "expected height [%d] fetched only in the first pass, got [%d]", + uint(proofStart+7), + got, + ) + } +} + +// countingBitcoinChain wraps a localBitcoinChain and counts GetBlockHeader +// backend fetches. maintainSpv builds its per-pass cache from +// sm.btcChain.GetBlockHeader, so wiring this as sm.btcChain lets a test count +// exactly the backend header fetches that cache makes through the real +// production pass structure. All other bitcoin.Chain methods are promoted from +// the embedded localBitcoinChain. +type countingBitcoinChain struct { + *localBitcoinChain + getter *countingHeaderGetter +} + +func (c *countingBitcoinChain) GetBlockHeader( + blockHeight uint, +) (*bitcoin.BlockHeader, error) { + return c.getter.get(blockHeight) +} + +// TestMaintainSpvSharesHeaderCacheAcrossProofTypes proves, by driving the real +// maintainSpv, that the single pass-scoped cache it creates above the proofTypes +// loop (spv.go: newBlockHeaderCache before `for action, v := range +// sm.proofTypes`) is shared across every proof type in a pass. Unlike a test +// that hand-assembles the shared cache, this fails if production ever moves the +// cache construction inside the loop (a per-proof-type cache would refetch the +// overlapping heights). It also asserts the next pass builds a fresh cache and +// refetches, so height-keyed entries never survive across passes. +func TestMaintainSpvSharesHeaderCacheAcrossProofTypes(t *testing.T) { + const proofStart = 790270 + + // Two transactions with distinct hashes and overlapping proof windows, each + // surfaced by a different proof type. + depositSweepTx := &bitcoin.Transaction{Version: 1} + redemptionTx := &bitcoin.Transaction{Version: 2} + if depositSweepTx.Hash() == redemptionTx.Hash() { + t.Fatal("expected the two fixtures to have distinct hashes") + } + + localChain := newLocalChain() + localChain.setTxProofDifficultyFactor(big.NewInt(6)) + localChain.setCurrentEpoch(392) + localChain.setCurrentAndPrevEpochDifficulty(big.NewInt(32), big.NewInt(16)) + + btcChain := newLocalBitcoinChain() + if err := populateBlockHeaders( + btcChain, + proofStart, + proofStart+19, + func(uint) *big.Int { return big.NewInt(32) }, + ); err != nil { + t.Fatal(err) + } + // latestBlockHeight = 790289. depositSweepTx: 20 confirmations -> walks + // 790270..790275; redemptionTx: 18 confirmations -> walks 790272..790277. + // Distinct heights across both proof types: 790270..790277 = 8. + btcChain.addTransactionConfirmations(depositSweepTx.Hash(), 20) + btcChain.addTransactionConfirmations(redemptionTx.Hash(), 18) + + counting := &countingBitcoinChain{ + localBitcoinChain: btcChain, + getter: newCountingHeaderGetter(btcChain.GetBlockHeader), + } + + noopSubmitter := func(bitcoin.Hash, uint, bitcoin.Chain, Chain) error { + return nil + } + + sm := &spvMaintainer{ + // A long idle backoff guarantees exactly one pass runs per maintainSpv + // call: after the pass, the post-pass select observes the cancelled + // context and returns instead of starting another pass. + config: Config{ + HistoryDepth: 100, + TransactionLimit: 10, + IdleBackoffTime: time.Hour, + }, + spvChain: localChain, + btcDiffChain: localChain, + btcChain: counting, + } + + // runPass drives one real maintainSpv pass. It ends the pass deterministically + // by cancelling the context once both proof types' getters have run (the + // getter is the first call proveTransactions makes). proveTransactions is not + // ctx-aware, so both proof types still process fully - fetching their header + // windows through the single per-pass cache - and only maintainSpv's post-pass + // select observes the cancellation. + runPass := func() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + var getterCalls int32 + mkGetter := func(tx *bitcoin.Transaction) unprovenTransactionsGetter { + return func( + uint64, + int, + bitcoin.Chain, + Chain, + ) ([]*bitcoin.Transaction, error) { + if atomic.AddInt32(&getterCalls, 1) == 2 { + cancel() + } + return []*bitcoin.Transaction{tx}, nil + } + } + + sm.proofTypes = map[tbtc.WalletActionType]proofType{ + tbtc.ActionDepositSweep: { + unprovenTransactionsGetter: mkGetter(depositSweepTx), + transactionProofSubmitter: noopSubmitter, + }, + tbtc.ActionRedemption: { + unprovenTransactionsGetter: mkGetter(redemptionTx), + transactionProofSubmitter: noopSubmitter, + }, + } + + if err := sm.maintainSpv(ctx); !errors.Is(err, context.Canceled) { + t.Fatalf( + "expected maintainSpv to stop with context.Canceled, got [%v]", + err, + ) + } + } + + // One real maintainSpv pass creates a single cache above the proofTypes loop, + // so the 8 distinct heights across the two overlapping proof-type walks are + // fetched from the backend exactly once. + runPass() + + if got := counting.getter.totalCalls(); got != 8 { + t.Fatalf( + "expected 8 backend header fetches shared across proof types in one "+ + "maintainSpv pass, got [%d] (12 would mean the cache is created "+ + "per proof type instead of once per pass)", + got, + ) + } + for h := uint(proofStart); h <= proofStart+7; h++ { + if got := counting.getter.callsAt(h); got != 1 { + t.Fatalf( + "expected height [%d] fetched once in the pass, got [%d]", + h, + got, + ) + } + } + + // A second maintainSpv pass builds a fresh cache and refetches the shared + // heights, so height-keyed entries never survive across passes (reorg safety). + runPass() + + if got := counting.getter.totalCalls(); got != 16 { + t.Fatalf( + "expected 16 total backend header fetches after a second maintainSpv "+ + "pass refetch, got [%d]", + got, + ) + } +} diff --git a/pkg/maintainer/spv/moved_funds_sweep.go b/pkg/maintainer/spv/moved_funds_sweep.go index 417a1f5347..1befc22031 100644 --- a/pkg/maintainer/spv/moved_funds_sweep.go +++ b/pkg/maintainer/spv/moved_funds_sweep.go @@ -117,7 +117,13 @@ func parseMovedFundsSweepTransactionInputs( } // Get the specific output spent by the moved funds sweep transaction. - spentOutput := inputTx.Outputs[input.Outpoint.OutputIndex] + spentOutput, err := inputTx.OutputAt(input.Outpoint.OutputIndex) + if err != nil { + return bitcoin.UnspentTransactionOutput{}, fmt.Errorf( + "failed to read spent output: [%v]", + err, + ) + } // Build the main UTXO object based on available data. mainUtxo := bitcoin.UnspentTransactionOutput{ diff --git a/pkg/maintainer/spv/moving_funds.go b/pkg/maintainer/spv/moving_funds.go index 81d1e13e51..036f5d73c8 100644 --- a/pkg/maintainer/spv/moving_funds.go +++ b/pkg/maintainer/spv/moving_funds.go @@ -104,7 +104,13 @@ func parseMovingFundsTransactionInput( } // Get the specific output spent by the moving funds transaction. - spentOutput := inputTx.Outputs[input.Outpoint.OutputIndex] + spentOutput, err := inputTx.OutputAt(input.Outpoint.OutputIndex) + if err != nil { + return bitcoin.UnspentTransactionOutput{}, [20]byte{}, fmt.Errorf( + "failed to read spent output: [%v]", + err, + ) + } // Build the main UTXO object based on available data. mainUtxo := bitcoin.UnspentTransactionOutput{ diff --git a/pkg/maintainer/spv/oob_regression_test.go b/pkg/maintainer/spv/oob_regression_test.go new file mode 100644 index 0000000000..6a99a54d60 --- /dev/null +++ b/pkg/maintainer/spv/oob_regression_test.go @@ -0,0 +1,169 @@ +package spv + +import ( + "testing" + + "github.com/keep-network/keep-core/pkg/bitcoin" +) + +// These tests are regression coverage for the security-audit OOB cluster +// (F-003/004/006/007): an output index taken from a candidate transaction's +// input outpoint is used to index a separately-fetched previous transaction's +// Outputs slice. A malicious or MITM Electrum backend can return a +// valid-but-shorter transaction for the requested hash, so the index can be +// out of range. Before the fix each site panicked (index out of range), and a +// panic on the SPV-maintainer goroutine crashes the whole client. After the +// fix each site returns an error instead. +// +// Each test wires the shared localBitcoinChain mock to return a previous +// transaction with a single output, then references it from a candidate +// transaction with an out-of-range output index, and asserts an error rather +// than a panic. + +// oobPreviousTransaction is a minimal previous transaction with exactly one +// output (index 0 is the only valid index). +func oobPreviousTransaction() *bitcoin.Transaction { + return &bitcoin.Transaction{ + Version: 1, + Inputs: []*bitcoin.TransactionInput{ + { + Outpoint: &bitcoin.TransactionOutpoint{ + TransactionHash: bitcoin.Hash{}, + OutputIndex: 0, + }, + SignatureScript: []byte{0x00}, + }, + }, + Outputs: []*bitcoin.TransactionOutput{ + {Value: 1000, PublicKeyScript: []byte{0x00, 0x14, 0x01}}, + }, + Locktime: 0, + } +} + +const oobOutOfRangeIndex = uint32(5) // the previous tx has only 1 output + +func TestIsInputCurrentWalletsMainUTXO_OutOfRangeIndex(t *testing.T) { + btcChain := newLocalBitcoinChain() + localChain := newLocalChain() + + prevTx := oobPreviousTransaction() + if err := btcChain.BroadcastTransaction(prevTx); err != nil { + t.Fatal(err) + } + + walletPublicKeyHash := [20]byte{} + + _, err := isInputCurrentWalletsMainUTXO( + prevTx.Hash(), + oobOutOfRangeIndex, + walletPublicKeyHash, + btcChain, + localChain, + ) + if err == nil { + t.Fatal("expected an out-of-range error, got nil (the unguarded code panics here)") + } +} + +func TestParseDepositSweepTransactionInputs_OutOfRangeIndex(t *testing.T) { + btcChain := newLocalBitcoinChain() + localChain := newLocalChain() + + prevTx := oobPreviousTransaction() + if err := btcChain.BroadcastTransaction(prevTx); err != nil { + t.Fatal(err) + } + + // A deposit sweep transaction must have exactly one output. + candidate := &bitcoin.Transaction{ + Version: 1, + Inputs: []*bitcoin.TransactionInput{ + { + Outpoint: &bitcoin.TransactionOutpoint{ + TransactionHash: prevTx.Hash(), + OutputIndex: oobOutOfRangeIndex, + }, + SignatureScript: []byte{0x00}, + }, + }, + Outputs: []*bitcoin.TransactionOutput{ + {Value: 900, PublicKeyScript: []byte{0x00, 0x14, 0x02}}, + }, + } + + _, _, err := parseDepositSweepTransactionInputs(btcChain, localChain, candidate) + if err == nil { + t.Fatal("expected an out-of-range error, got nil (the unguarded code panics here)") + } +} + +func TestParseMovingFundsTransactionInput_OutOfRangeIndex(t *testing.T) { + btcChain := newLocalBitcoinChain() + + prevTx := oobPreviousTransaction() + if err := btcChain.BroadcastTransaction(prevTx); err != nil { + t.Fatal(err) + } + + // A moving funds transaction must have exactly one input. + candidate := &bitcoin.Transaction{ + Version: 1, + Inputs: []*bitcoin.TransactionInput{ + { + Outpoint: &bitcoin.TransactionOutpoint{ + TransactionHash: prevTx.Hash(), + OutputIndex: oobOutOfRangeIndex, + }, + SignatureScript: []byte{0x00}, + }, + }, + Outputs: []*bitcoin.TransactionOutput{ + {Value: 900, PublicKeyScript: []byte{0x00, 0x14, 0x03}}, + }, + } + + _, _, err := parseMovingFundsTransactionInput(btcChain, candidate) + if err == nil { + t.Fatal("expected an out-of-range error, got nil (the unguarded code panics here)") + } +} + +func TestParseMovedFundsSweepTransactionInputs_OutOfRangeIndex(t *testing.T) { + btcChain := newLocalBitcoinChain() + + prevTx := oobPreviousTransaction() + if err := btcChain.BroadcastTransaction(prevTx); err != nil { + t.Fatal(err) + } + + // A moved funds sweep transaction with two inputs uses Inputs[1] (the + // wallet's main UTXO) for the output lookup. + candidate := &bitcoin.Transaction{ + Version: 1, + Inputs: []*bitcoin.TransactionInput{ + { + Outpoint: &bitcoin.TransactionOutpoint{ + TransactionHash: bitcoin.Hash{}, + OutputIndex: 0, + }, + SignatureScript: []byte{0x00}, + }, + { + Outpoint: &bitcoin.TransactionOutpoint{ + TransactionHash: prevTx.Hash(), + OutputIndex: oobOutOfRangeIndex, + }, + SignatureScript: []byte{0x00}, + }, + }, + Outputs: []*bitcoin.TransactionOutput{ + {Value: 900, PublicKeyScript: []byte{0x00, 0x14, 0x04}}, + }, + } + + _, err := parseMovedFundsSweepTransactionInputs(btcChain, candidate) + if err == nil { + t.Fatal("expected an out-of-range error, got nil (the unguarded code panics here)") + } +} diff --git a/pkg/maintainer/spv/redemptions.go b/pkg/maintainer/spv/redemptions.go index e504860f81..421bc5bf3b 100644 --- a/pkg/maintainer/spv/redemptions.go +++ b/pkg/maintainer/spv/redemptions.go @@ -137,8 +137,17 @@ func parseRedemptionTransactionInput( ) } - // Get the specific output spent by the redemption transaction. - spentOutput := inputTx.Outputs[input.Outpoint.OutputIndex] + // Get the specific output spent by the redemption transaction. The + // input transaction is fetched from the Bitcoin node, so its output + // count is untrusted; use the bounds-checked accessor to avoid an + // out-of-range panic on a short or malformed node response. + spentOutput, err := inputTx.OutputAt(input.Outpoint.OutputIndex) + if err != nil { + return bitcoin.UnspentTransactionOutput{}, [20]byte{}, fmt.Errorf( + "cannot get spent output: [%v]", + err, + ) + } // Build the main UTXO object based on available data. mainUtxo := bitcoin.UnspentTransactionOutput{ diff --git a/pkg/maintainer/spv/redemptions_metrics_test.go b/pkg/maintainer/spv/redemptions_metrics_test.go new file mode 100644 index 0000000000..033406c8cb --- /dev/null +++ b/pkg/maintainer/spv/redemptions_metrics_test.go @@ -0,0 +1,153 @@ +package spv + +import ( + "encoding/hex" + "fmt" + "sync" + "testing" + + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/clientinfo" +) + +// fakeMetricsRecorder records counter increments so tests can assert that the +// SPV redemption-proof metrics fire through the production recorder path. +type fakeMetricsRecorder struct { + mutex sync.Mutex + counters map[string]float64 +} + +func newFakeMetricsRecorder() *fakeMetricsRecorder { + return &fakeMetricsRecorder{counters: make(map[string]float64)} +} + +func (f *fakeMetricsRecorder) IncrementCounter(name string, value float64) { + f.mutex.Lock() + defer f.mutex.Unlock() + f.counters[name] += value +} + +func (f *fakeMetricsRecorder) value(name string) float64 { + f.mutex.Lock() + defer f.mutex.Unlock() + return f.counters[name] +} + +// TestSubmitRedemptionProofRecordsMetrics installs a fake recorder through the +// production SetMetricsRecorder/getGlobalMetricsRecorder path and asserts that a +// successful submission records total+success while a failing submission records +// total+failed. +func TestSubmitRedemptionProofRecordsMetrics(t *testing.T) { + bytesFromHex := func(str string) []byte { + value, err := hex.DecodeString(str) + if err != nil { + t.Fatal(err) + } + return value + } + + txFromHex := func(str string) *bitcoin.Transaction { + transaction := new(bitcoin.Transaction) + if err := transaction.Deserialize(bytesFromHex(str)); err != nil { + t.Fatal(err) + } + return transaction + } + + requiredConfirmations := uint(6) + + // The same arbitrary redemption transaction and its input used by + // TestSubmitRedemptionProof. + redemptionTransaction := txFromHex("0100000000010189a128bbd1fd4626f752aa9036a118b2f4b2363ef409f5b527c69d048214d3130000000000ffffffff039ef9e92e0000000016001403b74d6893ad46dfdd01b9e0e3b3385f4fce2d1e6eed10000000000017a91486884e6be1525dab5ae0b451bd2c72cee67dcf4187791411000000000017a914538e4cc700d6510c8cae5e8b688d65276771e6088702483045022100b2e7fc655e0ddadbfef49201fb5f7046a40b36848c08f17ef2e4483bffb7a29e022024616909a96f8c901572d6a9e19d29d6aee6a835b409d4383a463fe1b338a2940121028ed84936be6a9f594a2dcc636d4bebf132713da3ce4dac5c61afbf8bbb47d6f700000000") + redemptionInputTransaction := txFromHex("01000000000101db7aad9f51cffa7cebf5a3b41dc3552e1151d2550d8919a8e13d6bb00e046d5b0000000000ffffffff0333fc0b2f0000000016001403b74d6893ad46dfdd01b9e0e3b3385f4fce2d1e182612000000000017a914538e4cc700d6510c8cae5e8b688d65276771e60887aa9f10000000000017a91486884e6be1525dab5ae0b451bd2c72cee67dcf418702483045022100dded6eeacf49830de6f6b590a56f9b8ba3c2fda0b24e7f51884226a5ee78b5c2022024b1fbf3406716c9f9c5bfe241cfc0766af8209ecf8eb5f3318b407fd41c59ec0121028ed84936be6a9f594a2dcc636d4bebf132713da3ce4dac5c61afbf8bbb47d6f700000000") + + proof := &bitcoin.SpvProof{ + MerkleProof: []byte{0x01}, + TxIndexInBlock: 2, + BitcoinHeaders: []byte{0x03}, + } + + t.Run("successful submission records total and success", func(t *testing.T) { + recorder := newFakeMetricsRecorder() + SetMetricsRecorder(recorder) + defer SetMetricsRecorder(nil) + + btcChain := newLocalBitcoinChain() + spvChain := newLocalChain() + if err := btcChain.BroadcastTransaction(redemptionTransaction); err != nil { + t.Fatal(err) + } + if err := btcChain.BroadcastTransaction(redemptionInputTransaction); err != nil { + t.Fatal(err) + } + + assembler := func( + bitcoin.Hash, + uint, + bitcoin.Chain, + ) (*bitcoin.Transaction, *bitcoin.SpvProof, error) { + return redemptionTransaction, proof, nil + } + + err := submitRedemptionProof( + redemptionTransaction.Hash(), + requiredConfirmations, + btcChain, + spvChain, + assembler, + getGlobalMetricsRecorder(), + ) + if err != nil { + t.Fatal(err) + } + + if got := recorder.value(clientinfo.MetricRedemptionProofSubmissionsTotal); got != 1 { + t.Errorf("expected total 1, got %v", got) + } + if got := recorder.value(clientinfo.MetricRedemptionProofSubmissionsSuccessTotal); got != 1 { + t.Errorf("expected success 1, got %v", got) + } + if got := recorder.value(clientinfo.MetricRedemptionProofSubmissionsFailedTotal); got != 0 { + t.Errorf("expected failed 0, got %v", got) + } + }) + + t.Run("assembler error records total and failed", func(t *testing.T) { + recorder := newFakeMetricsRecorder() + SetMetricsRecorder(recorder) + defer SetMetricsRecorder(nil) + + btcChain := newLocalBitcoinChain() + spvChain := newLocalChain() + + assembler := func( + bitcoin.Hash, + uint, + bitcoin.Chain, + ) (*bitcoin.Transaction, *bitcoin.SpvProof, error) { + return nil, nil, fmt.Errorf("assembler failure") + } + + err := submitRedemptionProof( + redemptionTransaction.Hash(), + requiredConfirmations, + btcChain, + spvChain, + assembler, + getGlobalMetricsRecorder(), + ) + if err == nil { + t.Fatal("expected an error from the failing assembler") + } + + if got := recorder.value(clientinfo.MetricRedemptionProofSubmissionsTotal); got != 1 { + t.Errorf("expected total 1, got %v", got) + } + if got := recorder.value(clientinfo.MetricRedemptionProofSubmissionsFailedTotal); got != 1 { + t.Errorf("expected failed 1, got %v", got) + } + if got := recorder.value(clientinfo.MetricRedemptionProofSubmissionsSuccessTotal); got != 0 { + t.Errorf("expected success 0, got %v", got) + } + }) +} diff --git a/pkg/maintainer/spv/spv.go b/pkg/maintainer/spv/spv.go index 20bd84bd3c..aab04f8a07 100644 --- a/pkg/maintainer/spv/spv.go +++ b/pkg/maintainer/spv/spv.go @@ -6,9 +6,12 @@ import ( "encoding/hex" "fmt" "math/big" + "runtime/debug" "sync" "time" + "github.com/btcsuite/btcd/blockchain" + "github.com/keep-network/keep-core/pkg/tbtc" "github.com/ipfs/go-log/v2" @@ -19,13 +22,13 @@ import ( var logger = log.Logger("keep-maintainer-spv") -// The length of the Bitcoin difficulty epoch in blocks. -const difficultyEpochLength = 2016 - -// The maximum number of block headers allowed in a single SPV proof. Bounds -// the forward walk over headers when computing required confirmations -// (relevant on testnet4 where long runs of minimum-difficulty blocks occur). -const maxProofHeaders = 144 +// minDifficultyTarget is the decoded Bitcoin minimum-difficulty (DIFF1) target. +// It matches the Bridge's minimum-difficulty target (BTCUtils.DIFF1_TARGET) +// used by BitcoinTx.determineRequestedDifficulty. Decoded targets, not integer +// difficulties, must be compared: multiple compact-bits encodings round to +// integer difficulty 1 while decoding to different targets, so only a header +// whose decoded target equals this exact value is a skippable DIFF1 header. +var minDifficultyTarget = blockchain.CompactToBig(0x1d00ffff) func Initialize( ctx context.Context, @@ -39,6 +42,7 @@ func Initialize( spvChain: spvChain, btcDiffChain: btcDiffChain, btcChain: btcChain, + proofTypes: proofTypes, } go spvMaintainer.startControlLoop(ctx) @@ -72,12 +76,26 @@ func getMetricsRecorder() interface { return globalMetricsRecorder } -// proofTypes holds the information about proof types supported by the -// SPV maintainer. -var proofTypes = map[tbtc.WalletActionType]struct { +// MetricsRecorder returns the metrics recorder currently wired into the SPV +// maintainer, or nil when none is set. It is the exported read counterpart to +// SetMetricsRecorder and lets the maintainer startup path assert that the +// recorder was actually wired, without duplicating the wiring in tests. +func MetricsRecorder() interface { + IncrementCounter(name string, value float64) +} { + return getMetricsRecorder() +} + +// proofType bundles the unproven-transactions source and the proof submitter +// for a single SPV proof type. +type proofType struct { unprovenTransactionsGetter unprovenTransactionsGetter transactionProofSubmitter transactionProofSubmitter -}{ +} + +// proofTypes holds the information about proof types supported by the +// SPV maintainer. +var proofTypes = map[tbtc.WalletActionType]proofType{ tbtc.ActionDepositSweep: { unprovenTransactionsGetter: getUnprovenDepositSweepTransactions, transactionProofSubmitter: SubmitDepositSweepProof, @@ -101,9 +119,26 @@ type spvMaintainer struct { spvChain Chain btcDiffChain btcdiff.Chain btcChain bitcoin.Chain + // proofTypes are the proof types processed in each maintainSpv pass. It + // defaults to the package-level proofTypes map and is a field so tests can + // drive a real pass with controlled proof types. + proofTypes map[tbtc.WalletActionType]proofType } func (sm *spvMaintainer) startControlLoop(ctx context.Context) { + sm.runControlLoop(ctx, sm.maintainSpv) +} + +// runControlLoop repeatedly runs the given maintainer iteration, backing off by +// RestartBackoffTime between runs and exiting when the context is cancelled. +// Each iteration runs under runMaintainSpv's panic-recovery boundary, so a +// panic in one iteration is logged and converted into a restart rather than +// crashing the dedicated maintainer process. The iteration function is a +// parameter so the loop's restart behavior can be exercised in tests. +func (sm *spvMaintainer) runControlLoop( + ctx context.Context, + iteration func(context.Context) error, +) { logger.Info("starting SPV maintainer") defer func() { @@ -111,7 +146,7 @@ func (sm *spvMaintainer) startControlLoop(ctx context.Context) { }() for { - err := sm.maintainSpv(ctx) + err := sm.runMaintainSpv(ctx, iteration) if err != nil { logger.Errorf( "error while maintaining SPV: [%v]; restarting maintainer", @@ -127,14 +162,51 @@ func (sm *spvMaintainer) startControlLoop(ctx context.Context) { } } +// runMaintainSpv runs a single maintainer iteration under a panic-recovery +// boundary. A panic inside the iteration is recovered, its value and a full Go +// stack trace are logged at error level, and it is converted into a non-nil +// error so the caller can follow the ordinary error/restart path instead of +// letting the panic terminate the dedicated maintainer process (which also runs +// the co-resident Bitcoin-difficulty maintainer). This is residual containment +// only; it does not replace the source-level bounds checks in the SPV +// maintainer. Go runtime fatal errors are not recoverable and are intentionally +// not handled here. The error return is named so the deferred recovery can set +// it. +func (sm *spvMaintainer) runMaintainSpv( + ctx context.Context, + iteration func(context.Context) error, +) (err error) { + defer func() { + if r := recover(); r != nil { + logger.Errorf( + "recovered from panic in SPV maintainer: [%v]\n%s", + r, + debug.Stack(), + ) + err = fmt.Errorf("recovered from SPV maintainer panic: [%v]", r) + } + }() + + return iteration(ctx) +} + func (sm *spvMaintainer) maintainSpv(ctx context.Context) error { for { - for action, v := range proofTypes { + // Create one header cache per proof-task pass. Transactions with + // overlapping proof windows - across all proof types processed in this + // pass - reuse the same cached headers instead of repeatedly fetching + // them from the Bitcoin backend. The cache is discarded before the idle + // backoff and rebuilt on the next pass, so height-keyed entries never + // survive a reorg between passes. + headerCache := newBlockHeaderCache(sm.btcChain.GetBlockHeader) + + for action, v := range sm.proofTypes { logger.Infof("starting [%s] proof task execution...", action) if err := sm.proveTransactions( v.unprovenTransactionsGetter, v.transactionProofSubmitter, + headerCache, ); err != nil { return fmt.Errorf( "error while proving [%s] transactions: [%v]", @@ -186,6 +258,7 @@ type transactionProofSubmitter func( func (sm *spvMaintainer) proveTransactions( unprovenTransactionsGetter unprovenTransactionsGetter, transactionProofSubmitter transactionProofSubmitter, + headerCache *blockHeaderCache, ) error { transactions, err := unprovenTransactionsGetter( sm.config.HistoryDepth, @@ -213,6 +286,7 @@ func (sm *spvMaintainer) proveTransactions( sm.btcChain, sm.spvChain, sm.btcDiffChain, + headerCache, ) if err != nil { return fmt.Errorf("failed to get proof info: [%v]", err) @@ -277,16 +351,16 @@ func isInputCurrentWalletsMainUTXO( if err != nil { return false, fmt.Errorf("failed to get previous transaction: [%v]", err) } - if fundingOutputIndex >= uint32(len(previousTransaction.Outputs)) { + fundingOutput, err := previousTransaction.OutputAt(fundingOutputIndex) + if err != nil { return false, fmt.Errorf( - "funding output index [%d] out of range for transaction [%s] "+ - "with [%d] outputs", + "funding output index [%d] invalid for transaction [%s]: [%v]", fundingOutputIndex, fundingTxHash.String(), - len(previousTransaction.Outputs), + err, ) } - fundingOutputValue := previousTransaction.Outputs[fundingOutputIndex].Value + fundingOutputValue := fundingOutput.Value // Assume the input is the main UTXO and calculate hash. mainUtxoHash := spvChain.ComputeMainUtxoHash(&bitcoin.UnspentTransactionOutput{ @@ -306,15 +380,68 @@ func isInputCurrentWalletsMainUTXO( return bytes.Equal(mainUtxoHash[:], wallet.MainUtxoHash[:]), nil } +// blockHeaderCache memoizes successful GetBlockHeader lookups by Bitcoin block +// height for the lifetime of a single maintainSpv proof-task pass. Transactions +// with overlapping proof windows - possibly across different proof types in the +// same pass - otherwise re-walk and re-fetch the same headers from the Bitcoin +// backend. Only successful results are cached, so a transient backend failure +// is retried on a later call or pass. The cache is created fresh each pass and +// discarded before the idle backoff, which bounds memory and keeps height-keyed +// entries from surviving a reorg between passes. Access is currently +// single-threaded (proof types are processed sequentially); the mutex makes the +// at-most-one-fetch-per-height guarantee hold if that is ever parallelized. +type blockHeaderCache struct { + getter func(blockHeight uint) (*bitcoin.BlockHeader, error) + mutex sync.Mutex + headers map[uint]*bitcoin.BlockHeader +} + +// newBlockHeaderCache returns a blockHeaderCache backed by the given header +// getter, typically bitcoin.Chain.GetBlockHeader. +func newBlockHeaderCache( + getter func(blockHeight uint) (*bitcoin.BlockHeader, error), +) *blockHeaderCache { + return &blockHeaderCache{ + getter: getter, + headers: make(map[uint]*bitcoin.BlockHeader), + } +} + +// getBlockHeader returns the header at the given height, fetching it from the +// backend on the first request and serving the cached value afterwards. Errors +// are not cached. +func (c *blockHeaderCache) getBlockHeader(blockHeight uint) ( + *bitcoin.BlockHeader, + error, +) { + c.mutex.Lock() + defer c.mutex.Unlock() + + if header, exists := c.headers[blockHeight]; exists { + return header, nil + } + + header, err := c.getter(blockHeight) + if err != nil { + return nil, err + } + + c.headers[blockHeight] = header + return header, nil +} + // getProofInfo returns information about the SPV proof. It includes the // information whether the transaction proof range is within the previous and // current difficulty epochs as seen by the relay, the accumulated number of -// confirmations and the required number of confirmations. +// confirmations and the required number of confirmations. Block headers are +// read through the provided pass-scoped headerCache; tip, confirmation, and +// difficulty data come directly from the chains. func getProofInfo( transactionHash bitcoin.Hash, btcChain bitcoin.Chain, spvChain Chain, btcDiffChain btcdiff.Chain, + headerCache *blockHeaderCache, ) ( bool, uint, uint, error, ) { @@ -375,13 +502,6 @@ func getProofInfo( headerCount := uint(0) for { - if headerCount >= maxProofHeaders { - // Could not find a decisive header or accumulate enough - // difficulty within a sane number of headers. Skip the - // transaction; it may become provable later. - return false, 0, 0, nil - } - blockHeight := proofStartBlock + uint64(headerCount) if blockHeight > uint64(latestBlockHeight) { // Not enough mined blocks yet to assemble the proof. Report the @@ -390,7 +510,7 @@ func getProofInfo( return true, accumulatedConfirmations, headerCount + 1, nil } - header, err := btcChain.GetBlockHeader(uint(blockHeight)) + header, err := headerCache.getBlockHeader(uint(blockHeight)) if err != nil { return false, 0, 0, fmt.Errorf( "failed to get block header at height [%v]: [%v]", @@ -399,13 +519,26 @@ func getProofInfo( ) } + // Compare decoded targets, not integer difficulties, when identifying a + // minimum-difficulty header (see minDifficultyTarget). Reject a + // non-positive target before calling Difficulty(), which would divide by + // a zero target. + headerTarget := header.Target() + if headerTarget.Sign() <= 0 { + return false, 0, 0, fmt.Errorf( + "invalid target [%v] for block header at height [%v]", + headerTarget, + blockHeight, + ) + } + headerDiff := header.Difficulty() headerCount++ observedDiff.Add(observedDiff, headerDiff) if requestedDiff == nil { // Still looking for the decisive header. - if skipMinDifficulty && headerDiff.Cmp(one) == 0 { + if skipMinDifficulty && headerTarget.Cmp(minDifficultyTarget) == 0 { continue } diff --git a/pkg/maintainer/spv/spv_test.go b/pkg/maintainer/spv/spv_test.go index 088c619883..59665bfb5e 100644 --- a/pkg/maintainer/spv/spv_test.go +++ b/pkg/maintainer/spv/spv_test.go @@ -27,7 +27,9 @@ func TestGetProofInfo(t *testing.T) { currentEpochDifficulty *big.Int previousEpochDifficulty *big.Int headerDifficultyAt func(uint) *big.Int + headerAt func(uint) *bitcoin.BlockHeader headersFrom, headersTo uint + expectedError string expectedIsProofWithinRelayRange bool expectedAccumulatedConfirmations uint expectedRequiredConfirmations uint @@ -147,20 +149,86 @@ func TestGetProofInfo(t *testing.T) { expectedAccumulatedConfirmations: 0, expectedRequiredConfirmations: 0, }, - // A run of minimum-difficulty headers longer than maxProofHeaders - // never reaches a decisive header. - "minimum difficulty run exceeds header bound": { - transactionConfirmations: 150, + // This header's target is harder than the exact DIFF1 target, but its + // integer difficulty rounds down to one. The Bridge does not skip it; + // it treats it as the decisive header and rejects it because it matches + // neither relay epoch difficulty. The maintainer must do the same. + "non-DIFF1 target rounding to difficulty one is not skipped": { + transactionConfirmations: 1, currentEpochDifficulty: diff(32), previousEpochDifficulty: diff(16), - headerDifficultyAt: func(uint) *big.Int { return diff(1) }, - headersFrom: proofStart, - headersTo: proofStart + 149, + headerAt: func(uint) *bitcoin.BlockHeader { + return &bitcoin.BlockHeader{Bits: 0x1d00aaaa} + }, + headersFrom: proofStart, + headersTo: proofStart, expectedIsProofWithinRelayRange: false, expectedAccumulatedConfirmations: 0, expectedRequiredConfirmations: 0, }, + // Compact bits can decode to a zero target. Reject it before calling + // BlockHeader.Difficulty, which would otherwise divide by zero. + "zero target is rejected": { + transactionConfirmations: 1, + currentEpochDifficulty: diff(32), + previousEpochDifficulty: diff(16), + headerAt: func(uint) *bitcoin.BlockHeader { + return &bitcoin.BlockHeader{Bits: 0} + }, + headersFrom: proofStart, + headersTo: proofStart, + expectedError: "invalid target [0] for block header at height [790270]", + + expectedIsProofWithinRelayRange: false, + expectedAccumulatedConfirmations: 0, + expectedRequiredConfirmations: 0, + }, + // Long testnet4 runs of minimum-difficulty headers must not prevent a + // proof from reaching a later decisive header. The Bridge consumes the + // full header chain, so the maintainer must do the same. After 144 DIFF1 + // headers, the previous-epoch difficulty 16 header binds the requested + // difficulty and brings the observed total above 6*16=96. + "decisive header follows long minimum difficulty run": { + transactionConfirmations: 145, + currentEpochDifficulty: diff(32), + previousEpochDifficulty: diff(16), + headerDifficultyAt: func(h uint) *big.Int { + if h < proofStart+144 { + return diff(1) + } + return diff(16) + }, + headersFrom: proofStart, + headersTo: proofStart + 144, + + expectedIsProofWithinRelayRange: true, + expectedAccumulatedConfirmations: 145, + expectedRequiredConfirmations: 145, + }, + // A minimum-difficulty run far longer than the removed 144-header cap + // (and longer than the 145-header case above) must still reach its + // decisive header. This pins the absence of any disguised replacement + // cap rather than only the former boundary. After 200 DIFF1 headers, the + // previous-epoch difficulty 16 header binds the requested difficulty and + // the observed total (200 + 16) already exceeds 6*16=96. + "decisive header follows very long minimum difficulty run": { + transactionConfirmations: 201, + currentEpochDifficulty: diff(32), + previousEpochDifficulty: diff(16), + headerDifficultyAt: func(h uint) *big.Int { + if h < proofStart+200 { + return diff(1) + } + return diff(16) + }, + headersFrom: proofStart, + headersTo: proofStart + 200, + + expectedIsProofWithinRelayRange: true, + expectedAccumulatedConfirmations: 201, + expectedRequiredConfirmations: 201, + }, // The chain tip is reached before enough difficulty is accumulated. // The reported requirement is one header more than currently exists, // so the caller waits for more confirmations. @@ -191,13 +259,21 @@ func TestGetProofInfo(t *testing.T) { localChain := newLocalChain() btcChain := newLocalBitcoinChain() - if err := populateBlockHeaders( - btcChain, - test.headersFrom, - test.headersTo, - test.headerDifficultyAt, - ); err != nil { - t.Fatal(err) + if test.headerAt != nil { + for h := test.headersFrom; h <= test.headersTo; h++ { + if err := btcChain.addBlockHeader(h, test.headerAt(h)); err != nil { + t.Fatal(err) + } + } + } else { + if err := populateBlockHeaders( + btcChain, + test.headersFrom, + test.headersTo, + test.headerDifficultyAt, + ); err != nil { + t.Fatal(err) + } } btcChain.addTransactionConfirmations( transactionHash, @@ -220,8 +296,20 @@ func TestGetProofInfo(t *testing.T) { btcChain, localChain, localChain, + newBlockHeaderCache(btcChain.GetBlockHeader), ) - if err != nil { + if test.expectedError != "" { + if err == nil { + t.Fatalf("expected error containing [%v]", test.expectedError) + } + if !strings.Contains(err.Error(), test.expectedError) { + t.Fatalf( + "unexpected error\nexpected to contain: [%v]\nactual: [%v]", + test.expectedError, + err, + ) + } + } else if err != nil { t.Fatal(err) } @@ -359,7 +447,7 @@ func TestIsInputCurrentWalletsMainUTXO_OutOfRangeFundingOutput(t *testing.T) { if err == nil { t.Fatal("expected out-of-range funding output error") } - if !strings.Contains(err.Error(), "funding output index [2] out of range") { + if !strings.Contains(err.Error(), "out of range") { t.Fatalf("unexpected error: [%v]", err) } } diff --git a/pkg/monitoring/cutoverroster/alerts.go b/pkg/monitoring/cutoverroster/alerts.go new file mode 100644 index 0000000000..7ec2f932ca --- /dev/null +++ b/pkg/monitoring/cutoverroster/alerts.go @@ -0,0 +1,126 @@ +package cutoverroster + +import ( + "fmt" + "strings" +) + +// AlertRule is a single Prometheus alerting rule for the fleet collector. +type AlertRule struct { + Alert string + Expr string + For string + Labels map[string]string + Annotations map[string]string +} + +// CutoverRosterJob is the Prometheus scrape job that collects the fleet metrics +// (infrastructure/kube/keep-prd/monitoring/prometheus/config/config.yaml). The +// collector-down alert keys on it: if the collector dies, every +// performance_cutover_* series vanishes, so a value-threshold alert on those +// series would itself evaluate absent and never fire. An up/absent() alert on the +// scrape target catches that hole. +const CutoverRosterJob = "cutover-roster" + +// AlertRules returns the required fleet-readiness alerts. All fire only after two +// consecutive one-minute evaluations and are routed to the Release and Operator +// Coordination teams via routing labels. +func AlertRules() []AlertRule { + routing := func(severity string) map[string]string { + return map[string]string{ + "severity": severity, + "team": "release", + "route_to": "release,operator-coordination", + } + } + + return []AlertRule{ + { + Alert: "CutoverBlockingOperatorsPresent", + Expr: fmt.Sprintf("%s > 0", MetricFleetBlockingOperators), + // Two consecutive one-minute evaluations. + For: "2m", + Labels: routing("critical"), + Annotations: map[string]string{ + "summary": "Cutover-eligible operators remain in a blocking status.", + "description": "One or more authoritative operators are not exact-R1 " + + "or independently quarantined. Cutover readiness is not met.", + }, + }, + { + Alert: "CutoverRosterIncomplete", + Expr: fmt.Sprintf( + "%s > 0 or %s > 0 or %s > 0", + MetricFleetBlockingOperators, + MetricReportersStale, + MetricInventoryUnreconciled, + ), + For: "2m", + Labels: routing("warning"), + Annotations: map[string]string{ + "summary": "Cutover fleet roster is incomplete.", + "description": "Blocking operators, stale reporters, or unreconciled " + + "inventory are present. The go/no-go completeness criteria are not met.", + }, + }, + { + // A dead or unscraped collector makes every performance_cutover_* series + // vanish, so the two alerts above would evaluate absent and never fire. + // This alert fires when the collector scrape target is down (up == 0) OR + // has disappeared entirely from the scrape config (absent), so a missing + // collector can never leave both roster alerts silently absent. + Alert: "CutoverRosterCollectorDown", + Expr: fmt.Sprintf( + "up{job=%q} == 0 or absent(up{job=%q})", + CutoverRosterJob, CutoverRosterJob, + ), + For: "2m", + Labels: routing("critical"), + Annotations: map[string]string{ + "summary": "Cutover-roster collector scrape target is down or absent.", + "description": "Prometheus cannot scrape the cutover-roster collector " + + "(job cutover-roster): the target is down or has disappeared from the " + + "scrape config. Every performance_cutover_* series is therefore stale " + + "or absent, so the other roster alerts can be silently absent. Cutover " + + "readiness cannot be evaluated until the collector is restored.", + }, + }, + } +} + +// RenderAlertRulesYAML renders the alert rules as a Prometheus rule-file group. +func RenderAlertRulesYAML() string { + var b strings.Builder + b.WriteString("groups:\n") + b.WriteString(" - name: cutover-roster\n") + b.WriteString(" rules:\n") + for _, rule := range AlertRules() { + fmt.Fprintf(&b, " - alert: %s\n", rule.Alert) + fmt.Fprintf(&b, " expr: %s\n", rule.Expr) + fmt.Fprintf(&b, " for: %s\n", rule.For) + b.WriteString(" labels:\n") + for _, key := range sortedKeys(rule.Labels) { + fmt.Fprintf(&b, " %s: %q\n", key, rule.Labels[key]) + } + b.WriteString(" annotations:\n") + for _, key := range sortedKeys(rule.Annotations) { + fmt.Fprintf(&b, " %s: %q\n", key, rule.Annotations[key]) + } + } + return b.String() +} + +func sortedKeys(m map[string]string) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + // Small maps; simple insertion sort keeps output deterministic without a + // sort import churn. + for i := 1; i < len(keys); i++ { + for j := i; j > 0 && keys[j-1] > keys[j]; j-- { + keys[j-1], keys[j] = keys[j], keys[j-1] + } + } + return keys +} diff --git a/pkg/monitoring/cutoverroster/alerts_test.go b/pkg/monitoring/cutoverroster/alerts_test.go new file mode 100644 index 0000000000..231b9051cf --- /dev/null +++ b/pkg/monitoring/cutoverroster/alerts_test.go @@ -0,0 +1,83 @@ +package cutoverroster + +import ( + "strings" + "testing" +) + +func TestAlertRules_NamesForAndRouting(t *testing.T) { + rules := AlertRules() + + byName := map[string]AlertRule{} + for _, rule := range rules { + byName[rule.Alert] = rule + } + + for _, name := range []string{ + "CutoverBlockingOperatorsPresent", + "CutoverRosterIncomplete", + } { + rule, ok := byName[name] + if !ok { + t.Fatalf("expected alert %q to be defined", name) + } + // Two consecutive one-minute evaluations. + if rule.For != "2m" { + t.Errorf("alert %q: expected for=2m, got %q", name, rule.For) + } + // Routed to Release and Operator Coordination. + route := rule.Labels["route_to"] + if !strings.Contains(route, "release") || + !strings.Contains(route, "operator-coordination") { + t.Errorf("alert %q: expected routing to release and operator-coordination, got %q", name, route) + } + if rule.Expr == "" { + t.Errorf("alert %q: expected a non-empty expression", name) + } + } +} + +// TestAlertRules_CollectorDownAlert proves the up/absent() alert exists so a dead +// collector — which makes every performance_cutover_* series vanish — cannot leave +// both roster alerts silently absent. +func TestAlertRules_CollectorDownAlert(t *testing.T) { + var found bool + for _, r := range AlertRules() { + if r.Alert != "CutoverRosterCollectorDown" { + continue + } + found = true + if !strings.Contains(r.Expr, `up{job="cutover-roster"}`) { + t.Errorf("collector-down alert must key on the cutover-roster scrape job: %q", r.Expr) + } + if !strings.Contains(r.Expr, "absent(") { + t.Errorf("collector-down alert must use absent() so a vanished target fires: %q", r.Expr) + } + if r.For != "2m" { + t.Errorf("collector-down alert for=%q, want 2m", r.For) + } + if route := r.Labels["route_to"]; !strings.Contains(route, "release") { + t.Errorf("collector-down alert must route to release, got %q", route) + } + } + if !found { + t.Fatal("expected a CutoverRosterCollectorDown alert to be defined") + } +} + +func TestRenderAlertRulesYAML(t *testing.T) { + yaml := RenderAlertRulesYAML() + + for _, want := range []string{ + "groups:", + "name: cutover-roster", + "alert: CutoverBlockingOperatorsPresent", + "alert: CutoverRosterIncomplete", + MetricFleetBlockingOperators, + "for: 2m", + } { + if !strings.Contains(yaml, want) { + t.Errorf("rendered rules missing %q:\n%s", want, yaml) + } + } +} diff --git a/pkg/monitoring/cutoverroster/api.go b/pkg/monitoring/cutoverroster/api.go new file mode 100644 index 0000000000..057db00429 --- /dev/null +++ b/pkg/monitoring/cutoverroster/api.go @@ -0,0 +1,258 @@ +package cutoverroster + +import ( + "context" + "encoding/json" + "fmt" + "net" + "net/http" + "strings" + "time" + + "github.com/prometheus/client_golang/prometheus/promhttp" +) + +// readinessPath is the single authoritative readiness endpoint. +const readinessPath = "/api/v1/cutover-readiness" + +// healthzPath is an unauthenticated liveness/readiness endpoint for the process +// itself. It is deliberately served OUTSIDE the CIDR allowlist because Kubernetes +// kubelet probes originate from the node IP, which is not on the monitoring pod +// network and is not loopback — an allowlisted probe would return 403 and leave +// the pod permanently unready. It exposes no fleet data (only "ok"), so serving +// it openly is safe; the authoritative readiness data and /metrics stay behind +// the allowlist. +const healthzPath = "/healthz" + +// CIDRAllowlist is the monitoring-network trust boundary for the readiness API. +// When configured, only clients whose source IP is loopback or within one of the +// allowed networks are served; every other client is denied. It is a defensive +// application-level control that complements (does not replace) network-level +// firewalling. +type CIDRAllowlist struct { + nets []*net.IPNet +} + +// ParseCIDRAllowlist parses a comma-separated list of CIDR networks. An empty +// string returns a nil allowlist, meaning "no application-level boundary +// configured" (the caller's loopback bind default is then the only mitigation). +func ParseCIDRAllowlist(csv string) (*CIDRAllowlist, error) { + csv = strings.TrimSpace(csv) + if csv == "" { + return nil, nil + } + var nets []*net.IPNet + for _, part := range strings.Split(csv, ",") { + part = strings.TrimSpace(part) + if part == "" { + continue + } + _, ipNet, err := net.ParseCIDR(part) + if err != nil { + return nil, fmt.Errorf("invalid CIDR %q: %w", part, err) + } + nets = append(nets, ipNet) + } + if len(nets) == 0 { + return nil, nil + } + return &CIDRAllowlist{nets: nets}, nil +} + +// Allowed reports whether a request from remoteAddr (host or host:port) is within +// the trust boundary. Loopback is always allowed so a local operator/health check +// works; every other source must fall within an allowed network. +func (a *CIDRAllowlist) Allowed(remoteAddr string) bool { + host, _, err := net.SplitHostPort(remoteAddr) + if err != nil { + host = remoteAddr + } + ip := net.ParseIP(strings.TrimSpace(host)) + if ip == nil { + return false + } + if ip.IsLoopback() { + return true + } + for _, n := range a.nets { + if n.Contains(ip) { + return true + } + } + return false +} + +// bindIsLoopbackOnly reports whether addr binds only the loopback interface. An +// empty host, "0.0.0.0", "::", or a hostname it cannot classify are treated as +// non-loopback (routable) so the safe default is to demand an allowlist. +func bindIsLoopbackOnly(addr string) bool { + host, _, err := net.SplitHostPort(addr) + if err != nil { + host = addr + } + host = strings.TrimSpace(host) + if host == "" { + // No host = all interfaces. + return false + } + if host == "localhost" { + return true + } + ip := net.ParseIP(host) + if ip == nil { + // A hostname we cannot resolve to an IP here; do not assume it is loopback. + return false + } + return ip.IsLoopback() +} + +// healthzHandler serves the unauthenticated liveness/readiness endpoint. It +// always returns 200 with a tiny body once the HTTP server is accepting +// connections, which is exactly what a kubelet probe needs to mark the pod ready +// so Prometheus and Grafana can reach it. It intentionally reflects only that the +// process is serving, not fleet readiness, and exposes no fleet data. +func healthzHandler() http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet && r.Method != http.MethodHead { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte("ok\n")) + }) +} + +// withAllowlist wraps next so a request from outside the monitoring trust +// boundary is denied with 403 before reaching the readiness data. A nil +// allowlist means no application-level boundary is configured and next is served +// unchanged (the server's loopback bind default is then the mitigation). +func withAllowlist(allowlist *CIDRAllowlist, next http.Handler) http.Handler { + if allowlist == nil { + return next + } + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if !allowlist.Allowed(r.RemoteAddr) { + http.Error(w, "forbidden: not on the monitoring network", http.StatusForbidden) + return + } + next.ServeHTTP(w, r) + }) +} + +// snapshotSource is the minimal collector view the API needs. +type snapshotSource interface { + Snapshot() FleetSnapshot +} + +// NewHandler builds the HTTP handler exposing the deterministic readiness +// endpoint and, when a Prometheus registry is supplied, a /metrics endpoint. +// The TrustedReportTarget inventory field is never serialized (it is +// `json:"-"`), so it cannot leak through the API. +func NewHandler(source snapshotSource, metrics *PrometheusMetrics) http.Handler { + mux := http.NewServeMux() + + mux.HandleFunc(readinessPath, func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + + snapshot := source.Snapshot() + + w.Header().Set("Content-Type", "application/json") + encoder := json.NewEncoder(w) + encoder.SetIndent("", " ") + if err := encoder.Encode(snapshot); err != nil { + http.Error(w, "cannot encode snapshot", http.StatusInternalServerError) + return + } + }) + + if metrics != nil { + mux.Handle("/metrics", promhttp.HandlerFor( + metrics.Registry(), + promhttp.HandlerOpts{}, + )) + } + + return mux +} + +// Server serves the readiness API. It MUST be bound only to a monitoring +// network address; the readiness data is authoritative but not public. +type Server struct { + httpServer *http.Server + listener net.Listener +} + +// NewServer binds a TCP listener on addr and prepares an HTTP server for the +// readiness API. Bind addr to the monitoring interface only. When allowlist is +// non-nil, it enforces the monitoring-network trust boundary: only loopback and +// allowed-CIDR clients are served, everything else is denied with 403. +// +// A non-loopback bind with no allowlist is refused: exposing the authoritative +// readiness data on a routable interface without an application-level trust +// boundary is a misconfiguration, so it fails closed at startup rather than +// silently serving every client. +func NewServer( + addr string, + source snapshotSource, + metrics *PrometheusMetrics, + allowlist *CIDRAllowlist, +) (*Server, error) { + if allowlist == nil && !bindIsLoopbackOnly(addr) { + return nil, fmt.Errorf( + "refusing to bind the readiness API to non-loopback address [%s] without "+ + "an allowlist; set --allowedCIDRs to define the monitoring trust boundary "+ + "or bind to loopback", + addr, + ) + } + + listener, err := net.Listen("tcp", addr) + if err != nil { + return nil, fmt.Errorf("cannot bind cutover-roster API on [%s]: %w", addr, err) + } + + return &Server{ + httpServer: &http.Server{ + Handler: serverHandler(allowlist, source, metrics), + ReadHeaderTimeout: 10 * time.Second, + }, + listener: listener, + }, nil +} + +// serverHandler builds the top-level HTTP handler. /healthz is routed OUTSIDE the +// CIDR allowlist so kubelet liveness/readiness probes from the node IP succeed, +// while the authoritative readiness data and /metrics stay behind the allowlist. +func serverHandler( + allowlist *CIDRAllowlist, + source snapshotSource, + metrics *PrometheusMetrics, +) http.Handler { + top := http.NewServeMux() + top.Handle(healthzPath, healthzHandler()) + top.Handle("/", withAllowlist(allowlist, NewHandler(source, metrics))) + return top +} + +// Addr returns the actual bound address (useful when addr requested port 0). +func (s *Server) Addr() string { + return s.listener.Addr().String() +} + +// Serve blocks serving requests until the server is closed. +func (s *Server) Serve() error { + err := s.httpServer.Serve(s.listener) + if err == http.ErrServerClosed { + return nil + } + return err +} + +// Close gracefully shuts the server down. +func (s *Server) Close(ctx context.Context) error { + return s.httpServer.Shutdown(ctx) +} diff --git a/pkg/monitoring/cutoverroster/api_test.go b/pkg/monitoring/cutoverroster/api_test.go new file mode 100644 index 0000000000..53a1af9d8d --- /dev/null +++ b/pkg/monitoring/cutoverroster/api_test.go @@ -0,0 +1,153 @@ +package cutoverroster + +import ( + "context" + "net/http" + "net/http/httptest" + "testing" +) + +// TestCIDRAllowlist_EnforcesMonitoringBoundary proves the monitoring-network +// trust boundary: with an allowlist configured, a request from an untrusted +// source IP is denied with 403, while loopback and an allowed-CIDR client are +// served. A nil allowlist serves everyone (no application-level boundary). +func TestCIDRAllowlist_EnforcesMonitoringBoundary(t *testing.T) { + allowlist, err := ParseCIDRAllowlist("10.1.0.0/16") + if err != nil { + t.Fatalf("parse allowlist: %v", err) + } + if allowlist == nil { + t.Fatal("expected a non-nil allowlist") + } + + // The wrapped handler just writes 200 so we can observe allow vs deny. + inner := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + }) + handler := withAllowlist(allowlist, inner) + + for _, tt := range []struct { + name string + remoteAddr string + wantStatus int + }{ + {"untrusted public IP denied", "203.0.113.7:5555", http.StatusForbidden}, + {"outside allowed CIDR denied", "10.2.0.4:5555", http.StatusForbidden}, + {"allowed CIDR served", "10.1.2.3:5555", http.StatusOK}, + {"loopback always served", "127.0.0.1:5555", http.StatusOK}, + } { + t.Run(tt.name, func(t *testing.T) { + req := httptest.NewRequest(http.MethodGet, readinessPath, nil) + req.RemoteAddr = tt.remoteAddr + rec := httptest.NewRecorder() + handler.ServeHTTP(rec, req) + if rec.Code != tt.wantStatus { + t.Errorf("remote %s: got %d, want %d", tt.remoteAddr, rec.Code, tt.wantStatus) + } + }) + } + + // A nil allowlist imposes no application-level boundary. + served := withAllowlist(nil, inner) + req := httptest.NewRequest(http.MethodGet, readinessPath, nil) + req.RemoteAddr = "203.0.113.7:5555" + rec := httptest.NewRecorder() + served.ServeHTTP(rec, req) + if rec.Code != http.StatusOK { + t.Errorf("nil allowlist must serve everyone, got %d", rec.Code) + } +} + +// stubSnapshotSource is a minimal snapshotSource for handler routing tests. +type stubSnapshotSource struct{} + +func (stubSnapshotSource) Snapshot() FleetSnapshot { return FleetSnapshot{} } + +// TestHealthzServedOutsideAllowlist proves the kubelet-probe fix: /healthz is +// served to any source (including an IP outside the monitoring pod CIDR, which is +// where kubelet probes originate), while the authoritative readiness data stays +// behind the allowlist and is denied to that same untrusted source. Without this, +// an allowlisted probe from the node IP would 403 and keep the pod permanently +// unready. +func TestHealthzServedOutsideAllowlist(t *testing.T) { + allowlist, err := ParseCIDRAllowlist("10.1.0.0/16") + if err != nil { + t.Fatalf("parse allowlist: %v", err) + } + handler := serverHandler(allowlist, stubSnapshotSource{}, nil) + + // A kubelet-style probe from a node IP outside the allowlisted pod CIDR. + const nodeIP = "192.168.1.10:41234" + + // /healthz must be served regardless of source IP. + healthReq := httptest.NewRequest(http.MethodGet, healthzPath, nil) + healthReq.RemoteAddr = nodeIP + healthRec := httptest.NewRecorder() + handler.ServeHTTP(healthRec, healthReq) + if healthRec.Code != http.StatusOK { + t.Errorf("/healthz from a node IP must be served, got %d", healthRec.Code) + } + + // The authoritative readiness data from that same untrusted source is denied. + dataReq := httptest.NewRequest(http.MethodGet, readinessPath, nil) + dataReq.RemoteAddr = nodeIP + dataRec := httptest.NewRecorder() + handler.ServeHTTP(dataRec, dataReq) + if dataRec.Code != http.StatusForbidden { + t.Errorf("readiness data from an untrusted source must be denied, got %d", dataRec.Code) + } + + // The readiness data from an allowed-CIDR source is served. + okReq := httptest.NewRequest(http.MethodGet, readinessPath, nil) + okReq.RemoteAddr = "10.1.2.3:5555" + okRec := httptest.NewRecorder() + handler.ServeHTTP(okRec, okReq) + if okRec.Code != http.StatusOK { + t.Errorf("readiness data from an allowed CIDR must be served, got %d", okRec.Code) + } +} + +// TestParseCIDRAllowlist_Validation proves an empty allowlist parses to nil and +// an invalid CIDR is rejected. +func TestParseCIDRAllowlist_Validation(t *testing.T) { + if a, err := ParseCIDRAllowlist(" "); err != nil || a != nil { + t.Errorf("empty allowlist must parse to (nil, nil), got (%v, %v)", a, err) + } + if _, err := ParseCIDRAllowlist("not-a-cidr"); err == nil { + t.Error("expected an error for an invalid CIDR") + } +} + +// TestNewServer_RequiresAllowlistForNonLoopbackBind proves a non-loopback bind +// without an allowlist is refused at startup (fail closed), while a loopback bind +// or a non-loopback bind with an allowlist is accepted. +func TestNewServer_RequiresAllowlistForNonLoopbackBind(t *testing.T) { + allowlist, err := ParseCIDRAllowlist("10.0.0.0/8") + if err != nil { + t.Fatalf("parse allowlist: %v", err) + } + + // Non-loopback bind, no allowlist: refused. + if s, err := NewServer("0.0.0.0:0", nil, nil, nil); err == nil { + t.Error("a non-loopback bind without an allowlist must be refused") + if s != nil { + _ = s.Close(context.Background()) + } + } + + // Loopback bind, no allowlist: accepted (loopback is the mitigation). + loopback, err := NewServer("127.0.0.1:0", nil, nil, nil) + if err != nil { + t.Errorf("a loopback bind without an allowlist must be accepted: %v", err) + } else { + _ = loopback.Close(context.Background()) + } + + // Non-loopback bind WITH an allowlist: accepted. + guarded, err := NewServer("0.0.0.0:0", nil, nil, allowlist) + if err != nil { + t.Errorf("a non-loopback bind with an allowlist must be accepted: %v", err) + } else { + _ = guarded.Close(context.Background()) + } +} diff --git a/pkg/monitoring/cutoverroster/collector.go b/pkg/monitoring/cutoverroster/collector.go new file mode 100644 index 0000000000..e863b7e605 --- /dev/null +++ b/pkg/monitoring/cutoverroster/collector.go @@ -0,0 +1,1328 @@ +package cutoverroster + +import ( + "context" + "fmt" + "sort" + "strings" + "sync" + "time" + + "github.com/ipfs/go-log/v2" +) + +var logger = log.Logger("keep-cutover-roster") + +// QuarantineVerifier independently verifies that a quarantine/removal evidence +// reference is real before the collector accepts it. A note, unreachable +// endpoint, or self-report must not verify. When no verifier is configured, the +// collector fails closed and accepts no quarantine evidence. +type QuarantineVerifier interface { + // Verify reports whether evidenceRef is independently verified network or + // eligibility quarantine/removal evidence for the given instance. + Verify(instanceID, operatorAddress, evidenceRef string) bool +} + +// IdentityVerifier independently confirms the operator→staking-provider identity +// asserted by the authoritative inventory against the on-chain WalletRegistry +// mapping, at a block no earlier than the observation. It is the authoritative +// join required by the spec so a forged or stale inventory staking-provider claim +// cannot contribute to a resolved status. When no verifier is configured the +// collector cannot confirm identity on chain; the command layer logs that gap +// explicitly rather than silently trusting the inventory. +type IdentityVerifier interface { + // OperatorStakingProviderAtBlock returns the canonical (lowercase 0x + 40 hex) + // staking-provider address the WalletRegistry maps the operator to at the given + // block. A zero/empty return means the operator is not registered. block 0 + // means "latest". It honors ctx so a canceled collection/shutdown context + // aborts the lookup promptly. + OperatorStakingProviderAtBlock( + ctx context.Context, operatorAddress string, block uint64, + ) (string, error) +} + +// MetricsSink is the metrics interface the collector needs. The fleet-level +// gauges are label-less; the operator-level gauges carry +// {operator_address, staking_provider, status} labels. +type MetricsSink interface { + // SetGauge sets a label-less fleet gauge. + SetGauge(name string, value float64) + // SetOperatorGauge sets a per-operator labeled gauge. + SetOperatorGauge(name, operatorAddress, stakingProvider, status string, value float64) + // ResetOperatorGauges clears all per-operator labeled gauge series before a + // cycle re-emits them, so stale label sets do not linger. + ResetOperatorGauges() +} + +// instanceClass is the per-instance reconciliation classification. +type instanceClass uint8 + +const ( + classExactConfirmed instanceClass = iota + classOfflineUnknown + classNonCutoverRevision +) + +// Collector reconciles the authoritative eligible inventory, per-instance +// attestations, and node-local legacy sightings into a per-operator fleet +// status. It persists central state transactionally and refreshes metrics. +type Collector struct { + config CollectorConfig + store *Store + metrics MetricsSink + clock func() time.Time + verifier QuarantineVerifier + identity IdentityVerifier + + // serviceDiscoveryConfigured records whether the command wired a production + // service-discovery feed. Completeness requires it when + // config.RequireServiceDiscovery is set, so a collector run without discovery + // can never certify readiness. + serviceDiscoveryConfigured bool + + // mu guards the mutable central state (operators/instances) and + // lastSnapshot against concurrent Collect and HTTP Snapshot access. + mu sync.RWMutex + operators map[string]*operatorRecord + instances map[string]*instanceRecord + lastSnapshot FleetSnapshot +} + +// SetServiceDiscoveryConfigured records whether the production service-discovery +// feed is wired. When config.RequireServiceDiscovery is set, completeness is +// blocked until this is true, so a missing discovery feed blocks readiness +// rather than silently degrading to an inventory-only view. +func (c *Collector) SetServiceDiscoveryConfigured(configured bool) { + c.mu.Lock() + defer c.mu.Unlock() + c.serviceDiscoveryConfigured = configured +} + +// SetQuarantineVerifier installs the independent quarantine-evidence verifier. +// Until one is set, the collector accepts no quarantine evidence (fail closed). +func (c *Collector) SetQuarantineVerifier(verifier QuarantineVerifier) { + c.mu.Lock() + defer c.mu.Unlock() + c.verifier = verifier +} + +// SetIdentityVerifier installs the on-chain operator→staking-provider identity +// verifier. When set, every eligible operator's inventory staking-provider claim +// must match the WalletRegistry mapping at the current block or the operator is +// treated as an inventory-reconciliation fault (fail closed): it cannot resolve. +func (c *Collector) SetIdentityVerifier(identity IdentityVerifier) { + c.mu.Lock() + defer c.mu.Unlock() + c.identity = identity +} + +// NewCollector constructs a collector, loading any persisted central state from +// the store so it survives process restarts. +func NewCollector( + config CollectorConfig, + store *Store, + metrics MetricsSink, +) (*Collector, error) { + return newCollectorWithClock(config, store, metrics, time.Now) +} + +func newCollectorWithClock( + config CollectorConfig, + store *Store, + metrics MetricsSink, + clock func() time.Time, +) (*Collector, error) { + if store == nil { + return nil, fmt.Errorf("store is required") + } + if metrics == nil { + return nil, fmt.Errorf("metrics sink is required") + } + if config.MissedThreshold == 0 { + return nil, fmt.Errorf("missed threshold must be non-zero") + } + if config.SuccessThreshold == 0 { + return nil, fmt.Errorf("success threshold must be non-zero") + } + // A nonpositive collection interval would panic time.NewTicker in the + // command's collection loop; reject it at construction so the invariant is + // enforced regardless of the consumer. + if config.CollectionInterval <= 0 { + return nil, fmt.Errorf("collection interval must be positive") + } + + operators, err := store.LoadOperators() + if err != nil { + return nil, fmt.Errorf("cannot load operators: %w", err) + } + instances, err := store.LoadInstances() + if err != nil { + return nil, fmt.Errorf("cannot load instances: %w", err) + } + + return &Collector{ + config: config, + store: store, + metrics: metrics, + clock: clock, + operators: operators, + instances: instances, + }, nil +} + +// Collect runs one collection cycle with a background context. It is retained +// for callers (and tests) that do not thread a cancellation context; production +// uses CollectContext so identity-verification RPCs honor shutdown. +func (c *Collector) Collect( + inventory []InventoryInstance, + reports map[string]InstanceReport, + sightings []LegacySighting, + currentBlock uint64, +) (FleetSnapshot, error) { + return c.CollectContext(context.Background(), inventory, reports, sightings, currentBlock) +} + +// CollectContext runs one collection cycle. reports maps instance ID to the +// report obtained this cycle; a missing key means the instance was not +// reachable. sightings are post-cutover node-local legacy sightings aggregated +// this cycle. It updates and persists central state, refreshes metrics, emits +// logs, and returns the resulting snapshot. +// +// On-chain identity verification runs BEFORE the central-state lock is taken, so +// a degraded WalletRegistry RPC endpoint can never block readiness snapshots (or +// concurrent HTTP readers) for the duration of the whole per-operator RPC +// sweep. The verifier is captured under a short read lock; its results are then +// applied inside the write lock. +func (c *Collector) CollectContext( + ctx context.Context, + inventory []InventoryInstance, + reports map[string]InstanceReport, + sightings []LegacySighting, + currentBlock uint64, +) (FleetSnapshot, error) { + now := c.clock() + + // Phase 1 — no lock held. Verify each eligible operator's inventory + // staking-provider claim against the on-chain WalletRegistry via network RPCs. + // This is pure with respect to central state (it only reads the inventory + // argument and the captured verifier), so holding the lock across it would + // needlessly serialize readers behind a slow endpoint. + c.mu.RLock() + identity := c.identity + c.mu.RUnlock() + claims := eligibleStakingClaims(inventory) + identityFailed := verifyOperatorIdentities(ctx, identity, claims, currentBlock) + + c.mu.Lock() + defer c.mu.Unlock() + + // Reset the per-cycle transient flags on every known instance so a stale value + // from a prior cycle never leaks into this cycle's reporter count or + // discovery-disappearance classification. + for _, inst := range c.instances { + inst.ReportedThisCycle = false + inst.DisappearedFromDiscovery = false + } + + eligibleByOperator := map[string][]InventoryInstance{} + stakingProviderByOperator := map[string]string{} + // contradicted records operators whose eligible instances asserted more than + // one distinct staking provider in this cycle. A cross-instance contradiction + // means the inventory disagrees with itself about the operator's identity, so + // the operator must not resolve (the last assignment must not silently win). + contradicted := map[string]bool{} + // seenInstanceIDs records which instances were present and eligible in the + // current inventory, so the reconciliation step can detect instances that + // have disappeared from service discovery since an earlier cycle. + seenInstanceIDs := map[string]bool{} + // seenNetworkIDs records which per-instance network identities have already + // been claimed this cycle. A network ID is a globally unique libp2p identity, + // so two eligible instances asserting the same one cannot be two distinct + // nodes; the duplicate is rejected so a single responding node cannot certify + // more than one same-operator inventory instance. + seenNetworkIDs := map[string]bool{} + totalInstances := len(inventory) + unreconciled := 0 + stale := 0 + reconciledEligible := 0 + + for _, rawInv := range inventory { + inv := rawInv + inv.OperatorAddress = normalizeAddress(inv.OperatorAddress) + + if !inv.CeremonyEligible { + continue + } + + // Reject a malformed, duplicated, or internally-contradictory + // authoritative inventory entry as an inventory-reconciliation fault + // before it can contribute to a resolved status. A missing instance + // identity or a non-canonical operator address cannot be joined or + // tracked; a duplicate instance ID within one cycle would let one entry + // silently overwrite another; a blank required inventory field (staking + // provider, expected revision/epoch/digest) leaves the entry unable to + // prove what "current" is for that instance; a per-instance expected + // identity that contradicts the collector's configured expected release + // means the inventory disagrees with itself. Any of these forces readiness + // closed (unreconciled > 0) rather than silently contributing to success. + if inv.InstanceID == "" || !isCanonicalAddress(inv.OperatorAddress) { + unreconciled++ + continue + } + if seenInstanceIDs[inv.InstanceID] { + unreconciled++ + continue + } + if inv.StakingProvider == "" || + inv.ExpectedRevision == "" || + inv.ExpectedEpoch == "" || + inv.ExpectedImageDigest == "" { + unreconciled++ + continue + } + // The staking provider is an on-chain identity: it must be a canonical + // address and must not be the zero address (an unregistered/blank + // operator). A non-address or zero staking provider cannot be joined to the + // WalletRegistry mapping and must not contribute to a resolved status. + normalizedStakingProvider := normalizeAddress(inv.StakingProvider) + if !isCanonicalAddress(normalizedStakingProvider) || + isZeroAddress(normalizedStakingProvider) { + unreconciled++ + continue + } + // The expected image digest must be a full, immutable content digest + // (sha256:<64 hex>). An abbreviated or malformed digest cannot pin the + // exact runtime image, so it must not certify what "current" is. + if !isValidImageDigest(inv.ExpectedImageDigest) { + unreconciled++ + continue + } + if c.inventoryExpectationContradicts(inv) { + unreconciled++ + continue + } + // The per-instance network ID is the identity that keeps distinct + // same-operator instances from collapsing onto one responding node. When + // on-chain identity or service-discovery verification is required it is a + // mandatory inventory field and must be unique across the cycle: a missing + // or duplicated network ID means a single node could stand in for more than + // one instance, so it is an inventory-reconciliation fault (fail closed). + if c.networkIdentityRequired() { + networkID := strings.TrimSpace(inv.NetworkID) + if networkID == "" || seenNetworkIDs[networkID] { + unreconciled++ + continue + } + seenNetworkIDs[networkID] = true + } + + reconciledEligible++ + seenInstanceIDs[inv.InstanceID] = true + eligibleByOperator[inv.OperatorAddress] = append( + eligibleByOperator[inv.OperatorAddress], inv, + ) + // Record the operator's staking provider, rejecting a cross-instance + // contradiction rather than letting the last assignment silently win. The + // first canonical claim is retained; a differing later claim marks the + // operator contradicted so it cannot resolve this cycle. + if existing, ok := stakingProviderByOperator[inv.OperatorAddress]; ok { + if existing != normalizedStakingProvider { + contradicted[inv.OperatorAddress] = true + unreconciled++ + } + } else { + stakingProviderByOperator[inv.OperatorAddress] = normalizedStakingProvider + } + + record := c.instanceForInventory(inv) + + // Record the per-instance authoritative inventory expectations for + // auditability so a reader (or a restarted collector) can see exactly what + // this instance was expected to report, not only whether it reported. + record.CeremonyEligible = inv.CeremonyEligible + record.StakingProvider = inv.StakingProvider + record.ExpectedRevision = inv.ExpectedRevision + record.ExpectedEpoch = inv.ExpectedEpoch + record.ExpectedImageDigest = inv.ExpectedImageDigest + + // Quarantine evidence is accepted only when independently verified; a + // bare reference, absent a verifier, never quarantines (fail closed). A + // verified-quarantined instance is intentionally removed, so it is + // excluded from the stale and unreconciled counts below. + record.QuarantineRef = inv.QuarantineEvidenceRef + record.HasQuarantine = inv.QuarantineEvidenceRef != "" && + c.verifier != nil && + c.verifier.Verify( + inv.InstanceID, inv.OperatorAddress, inv.QuarantineEvidenceRef, + ) + + report, reported := reports[inv.InstanceID] + + // Reconciliation rule 2: an eligible instance present in the authoritative + // inventory but absent from the production service-discovery target set has + // disappeared from discovery. It is offline_unknown for this cycle — its + // report (if any) is not accepted (the stale/missed accounting below then + // applies) — unless it is independently quarantined. + if inv.DisappearedFromDiscovery { + record.DisappearedFromDiscovery = true + reported = false + } + + // A missing trusted report target is an inventory-reconciliation failure + // (unless the instance is quarantined and thus not expected to report). + if inv.TrustedReportTarget == "" { + reported = false + if !record.HasQuarantine { + unreconciled++ + } + } + + // Reject an attestation whose identity, freshness, or reporter revision + // cannot be validated, rather than silently accepting it. A report must + // self-identify with the same instance ID and operator address as the + // authoritative inventory entry it answers for: a missing or a mismatched + // identity is an inventory-reconciliation failure, so a report that does + // not name itself cannot stand in for the trusted instance (the reporter + // deliberately does not fabricate these fields from inventory). A + // stale/replayed attestation is merely a missed collection. + unreconciledFault := false + if reported { + normalizedReportOperator := normalizeAddress(report.OperatorAddress) + switch { + case report.InstanceID != inv.InstanceID: + reported, unreconciledFault = false, true + case normalizedReportOperator != inv.OperatorAddress: + reported, unreconciledFault = false, true + case report.AttestedAt.IsZero(): + // Missing attestation time cannot prove freshness. + reported, unreconciledFault = false, true + case report.AttestedAt.After(now): + // A future attestation time is invalid evidence; accepting it + // would also poison the monotonic freshness guard below. + reported, unreconciledFault = false, true + case report.ReporterRevision == 0: + // Missing reporter revision. + reported, unreconciledFault = false, true + case record.LatestReport != nil && + !report.AttestedAt.After(record.LatestReport.AttestedAt): + // Stale or replayed attestation (not newer than the last one). + reported = false + case report.ReporterRevision < record.LastReporterRevision: + // Reporter-revision downgrade. + reported = false + } + } + if unreconciledFault { + unreconciled++ + } + + if reported { + r := report + r.OperatorAddress = inv.OperatorAddress + record.LatestReport = &r + record.LastReporterRevision = report.ReporterRevision + record.ReportedThisCycle = true + record.ConsecutiveMissed = 0 + if c.reportIsExact(report) { + record.ConsecutiveExact++ + } else { + record.ConsecutiveExact = 0 + } + } else { + if !record.HasQuarantine { + stale++ + } + record.ConsecutiveMissed++ + record.ConsecutiveExact = 0 + } + } + + // Fold in this cycle's post-cutover legacy sightings. A sighting before the + // cutover block, or after the current block, is not valid post-cutover + // straggler evidence and is ignored. + freshLegacy := map[string]bool{} + for _, sighting := range sightings { + operator := normalizeAddress(sighting.OperatorAddress) + // A non-canonical operator address cannot be joined to an operator and + // must not create straggler evidence. + if !isCanonicalAddress(operator) { + continue + } + // A sighting before the cutover block, or after the current block, is not + // valid post-cutover straggler evidence. + if sighting.Block < c.config.CutoverBlock { + continue + } + if currentBlock > 0 && sighting.Block > currentBlock { + continue + } + // Reject a zero or future observation timestamp outright rather than + // admitting the sighting while skipping LastLegacyAt. Admitting it would + // create an observed_legacy status yet leave LastLegacyAt at zero, which + // makes the "every report newer than the last legacy observation" + // resolution proof pass trivially — weakening the required post-sighting + // resolution evidence. Genuine node-local sightings always carry a real + // clock timestamp, so this only rejects malformed input. + if sighting.ObservedAt.IsZero() || sighting.ObservedAt.After(now) { + continue + } + op := c.operatorForAddress(operator, stakingProviderByOperator) + freshLegacy[operator] = true + if sighting.Block > op.LastLegacyBlock { + op.LastLegacyBlock = sighting.Block + } + if sighting.ObservedAt.After(op.LastLegacyAt) { + op.LastLegacyAt = sighting.ObservedAt + } + } + + // Group every known instance record by operator so instances that were + // present in an earlier cycle but have since disappeared from the current + // inventory are still reconciled rather than silently dropped. + instancesByOperator := map[string][]*instanceRecord{} + for _, inst := range c.instances { + instancesByOperator[inst.OperatorAddress] = append( + instancesByOperator[inst.OperatorAddress], inst, + ) + } + + // Fold the pre-lock on-chain identity verification into the unreconciled + // count: every eligible operator whose asserted staking-provider identity + // could not be confirmed against the WalletRegistry is an + // inventory-reconciliation fault (fail closed). The verification itself ran + // before the lock (Phase 1) so a slow RPC never blocked readers. + for op := range eligibleByOperator { + if identityFailed[op] { + unreconciled++ + } + } + + // Reconcile every operator with eligible instances this cycle, every operator + // with a fresh legacy sighting, AND every operator that still has persisted + // state (instance records or an operator record) even if it vanished entirely + // from this cycle's inventory and sightings. Reconciling the vanished set is + // the fail-closed safety property: a previously resolved_current operator whose + // instances all disappear must reopen as offline_unknown, not silently stay + // resolved (and later be purged) — removal from inventory never resolves + // central state. + toReconcile := map[string]bool{} + for op := range eligibleByOperator { + toReconcile[op] = true + } + for op := range freshLegacy { + toReconcile[op] = true + } + for op := range instancesByOperator { + toReconcile[op] = true + } + for op := range c.operators { + toReconcile[op] = true + } + + for operatorAddress := range toReconcile { + op := c.operatorForAddress(operatorAddress, stakingProviderByOperator) + if provider, ok := stakingProviderByOperator[operatorAddress]; ok { + op.StakingProvider = provider + } + + instanceRecords := c.reconcileDisappearedInstances( + instancesByOperator[operatorAddress], + seenInstanceIDs, + ) + + previousStatus := op.Status + status, reason := c.reconcileOperatorStatus( + instanceRecords, + freshLegacy[operatorAddress], + op.LastLegacyAt, + ) + + // A failed on-chain identity verification is fail-closed: the operator's + // asserted staking-provider identity could not be confirmed against the + // WalletRegistry, so it must not resolve regardless of what it reports. + if identityFailed[operatorAddress] && status == FleetResolvedCurrent { + status = FleetOfflineUnknown + reason = "on-chain operator→staking-provider identity unverified" + } + // A cross-instance staking-provider contradiction is likewise fail-closed: + // the inventory disagrees with itself about who this operator is, so it + // cannot resolve until the inventory is made self-consistent. + if contradicted[operatorAddress] && status == FleetResolvedCurrent { + status = FleetOfflineUnknown + reason = "contradictory staking-provider claims across instances" + } + + if op.FirstSeenBlock == 0 { + op.FirstSeenBlock = currentBlock + } + op.LastSeenBlock = currentBlock + op.Status = status + op.Reason = reason + + if status == FleetResolvedCurrent { + // Refresh the resolution timestamp every cycle the operator stays + // resolved. Because every persisted operator is now reconciled each + // cycle (a vanished resolved operator reopens as offline_unknown rather + // than staying resolved), an actively-resolved operator's record never + // ages out — its resolution is continuously re-confirmed. purgeResolved + // remains a defensive backstop for any resolved record that stops being + // reconciled; it never removes an operator that vanished, since such an + // operator is no longer resolved. + op.ResolvedAt = now + if previousStatus != FleetResolvedCurrent { + logger.Infof( + "cutover operator resolved [operator=%s] "+ + "[stakingProvider=%s] [resolution=%s] [currentBlock=%d]", + op.OperatorAddress, + op.StakingProvider, + status, + currentBlock, + ) + } + } else { + // Reopened or still blocking: it is no longer resolved. + op.ResolvedAt = time.Time{} + } + } + + c.purgeResolved(now) + + if err := c.store.Save(c.operators, c.instances); err != nil { + // Persisting the reconciled central state failed. Fail readiness closed + // for this cycle — supersede any earlier "complete=true" snapshot and its + // zero watched gauges with an incomplete one carrying a nonzero + // unreconciled signal — before surfacing the error to the caller. Leaving + // the prior snapshot served would keep certifying readiness after a failed + // write, exactly the false-ready signal an unreadable input already guards + // against. + c.publishFailClosed(now, currentBlock) + return FleetSnapshot{}, fmt.Errorf("cannot persist central state: %w", err) + } + + snapshot := c.buildSnapshot(now, currentBlock) + snapshot.Complete = c.isComplete( + snapshot, reconciledEligible, stale, unreconciled, currentBlock, + ) + + // Guarantee that an incomplete readiness determination always surfaces as a + // nonzero blocking/stale/unreconciled signal, so the CutoverRosterIncomplete + // alert fires even when readiness cannot be established at all — an empty + // inventory, unset expected identity, a zero cutover block, or a stale chain + // clock would otherwise leave every watched gauge at zero and hide the fault. + if !snapshot.Complete && + len(snapshot.Blocking) == 0 && stale == 0 && unreconciled == 0 { + unreconciled = 1 + } + + snapshot.Inventory = FleetInventoryCounts{ + TotalInstances: totalInstances, + EligibleInstances: reconciledEligible, + ReportersStale: stale, + Unreconciled: unreconciled, + } + c.lastSnapshot = snapshot + + c.updateMetrics(snapshot, stale, unreconciled) + c.logCycle(snapshot) + + return snapshot, nil +} + +// RecordInputUnavailable records a collection cycle in which an authoritative +// input — the ceremony-eligible inventory or the legacy-sightings evidence — +// could not be read. It fails readiness closed for the cycle: the published +// snapshot is forced incomplete with a nonzero inventory-unreconciled signal, and +// the fleet metrics are refreshed so the CutoverRosterIncomplete alert fires. A +// stale "complete=true" snapshot from an earlier cycle must never keep certifying +// readiness while the authoritative denominator is missing. +// +// Persisted operator/instance history is deliberately left untouched: a transient +// input blip is not evidence that any operator converged, disappeared, or missed +// a collection, so it must not advance a missed counter, purge a resolved record, +// or otherwise mutate central state. +func (c *Collector) RecordInputUnavailable(currentBlock uint64) FleetSnapshot { + c.mu.Lock() + defer c.mu.Unlock() + + return c.publishFailClosed(c.clock(), currentBlock) +} + +// publishFailClosed forces the served snapshot incomplete with a nonzero +// inventory-unreconciled signal and refreshes the fleet metrics and cycle log, so +// a stale "complete=true" snapshot from an earlier cycle can never keep certifying +// readiness after a cycle that could not be completed — whether an authoritative +// input was unreadable or persisting the reconciled central state failed. Any +// persisted blocking operators stay visible via buildSnapshot; forcing +// unreconciled to at least one guarantees a nonzero watched gauge (so the +// CutoverRosterIncomplete alert fires) even when nothing was blocking. +// +// It does not itself mutate persisted operator/instance history. The caller MUST +// hold c.mu. +func (c *Collector) publishFailClosed(now time.Time, currentBlock uint64) FleetSnapshot { + snapshot := c.buildSnapshot(now, currentBlock) + snapshot.Complete = false + + const stale = 0 + const unreconciled = 1 + snapshot.Inventory = FleetInventoryCounts{ + TotalInstances: len(c.instances), + EligibleInstances: 0, + ReportersStale: stale, + Unreconciled: unreconciled, + } + + c.lastSnapshot = snapshot + c.updateMetrics(snapshot, stale, unreconciled) + c.logCycle(snapshot) + + return snapshot +} + +// networkIdentityRequired reports whether the per-instance network ID is a +// mandatory, unique inventory field this run. It is required whenever the +// authoritative trust chain is enforced — on-chain identity verification or +// production service-discovery reconciliation — because both rely on the network +// ID to bind one responding node to exactly one inventory instance. A +// developer/test collector with neither requirement leaves it optional so +// narrowly-scoped fixtures need not carry it. +func (c *Collector) networkIdentityRequired() bool { + return c.config.RequireIdentityVerification || c.config.RequireServiceDiscovery +} + +// isComplete fails closed: readiness is "complete" only with a nonempty +// reconciled authoritative inventory, a fresh current block, fully specified +// expected artifact identity and chain ID, and zero blocking/stale/unreconciled. +// An empty or missing inventory, an unavailable chain clock, or an unset +// expected identity can never produce complete=true. +func (c *Collector) isComplete( + snapshot FleetSnapshot, + reconciledEligible, stale, unreconciled int, + currentBlock uint64, +) bool { + if reconciledEligible == 0 { + return false + } + if currentBlock == 0 { + return false + } + // A zero cutover block is placeholder metadata: readiness cannot be + // certified for a go/no-go until a real cutover block C is supplied. + if c.config.CutoverBlock == 0 { + return false + } + if c.config.ExpectedRevision == "" || + c.config.ExpectedEpoch == "" || + c.config.ChainID == "" || + !isValidImageDigest(c.config.ExpectedImageDigest) { + return false + } + // Identity verification against the WalletRegistry is a mandatory part of the + // authoritative trust chain: without an installed verifier the collector would + // certify trusted-file staking-provider assertions on their own. A missing + // verifier blocks readiness rather than degrading to trusting the inventory. + if c.config.RequireIdentityVerification && c.identity == nil { + return false + } + // Reconciliation against production service discovery is likewise mandatory: + // without it a single responding target could stand in for several inventory + // instances of one operator. A missing discovery feed blocks readiness. + if c.config.RequireServiceDiscovery && !c.serviceDiscoveryConfigured { + return false + } + return len(snapshot.Blocking) == 0 && stale == 0 && unreconciled == 0 +} + +// normalizeAddress normalizes an operator/staking address for identity joins: +// trimmed and lowercased. It is lenient — a value that is not a 0x-prefixed hex +// address is returned lowercased rather than dropped — so inventory, reports, +// and sightings that refer to one operator with different casing deduplicate to +// a single record. Canonical-form enforcement is a separate step +// (isCanonicalAddress); normalization alone must not reject, so that a +// case-different but otherwise valid address still deduplicates correctly. +func normalizeAddress(address string) string { + return strings.ToLower(strings.TrimSpace(address)) +} + +// isCanonicalAddress reports whether s is a canonical operator address: lowercase +// "0x" followed by exactly 40 hexadecimal characters. Identity joins require a +// canonical address so a malformed or truncated identity cannot contribute to a +// resolved status. +func isCanonicalAddress(s string) bool { + if len(s) != 42 || s[0] != '0' || s[1] != 'x' { + return false + } + for _, c := range s[2:] { + if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) { + return false + } + } + return true +} + +// eligibleStakingClaims extracts each eligible operator's canonical +// staking-provider claim from the inventory, applying the same identity-relevant +// validation the reconciliation loop uses so on-chain verification runs over the +// same operator set. It is pure with respect to central state, so it can run +// before the collector lock is taken. An operator with a cross-instance +// staking-provider contradiction is excluded: it cannot resolve regardless of +// the on-chain answer, and there is no single claim to verify. +func eligibleStakingClaims(inventory []InventoryInstance) map[string]string { + claims := map[string]string{} + seenInstances := map[string]bool{} + contradicted := map[string]bool{} + for _, raw := range inventory { + operator := normalizeAddress(raw.OperatorAddress) + if !raw.CeremonyEligible { + continue + } + if raw.InstanceID == "" || !isCanonicalAddress(operator) { + continue + } + if seenInstances[raw.InstanceID] { + continue + } + seenInstances[raw.InstanceID] = true + provider := normalizeAddress(raw.StakingProvider) + if !isCanonicalAddress(provider) || isZeroAddress(provider) { + continue + } + if existing, ok := claims[operator]; ok { + if existing != provider { + contradicted[operator] = true + } + continue + } + claims[operator] = provider + } + for operator := range contradicted { + delete(claims, operator) + } + return claims +} + +// verifyOperatorIdentities verifies each eligible operator's inventory +// staking-provider claim against the on-chain WalletRegistry mapping. A mismatch +// or a lookup failure marks the operator failed (fail closed). When no verifier +// is configured it returns an empty set: the command layer is responsible for +// surfacing the unverified-identity gap, and completeness is blocked separately +// when identity verification is required. It performs only network I/O and holds +// no lock, so it MUST run outside c.mu. +func verifyOperatorIdentities( + ctx context.Context, + identity IdentityVerifier, + claims map[string]string, + currentBlock uint64, +) map[string]bool { + failed := map[string]bool{} + if identity == nil { + return failed + } + for operatorAddress, claimedProvider := range claims { + onChain, err := identity.OperatorStakingProviderAtBlock( + ctx, operatorAddress, currentBlock, + ) + if err != nil { + logger.Errorf( + "cannot verify operator identity on chain [operator=%s]: %v", + operatorAddress, err, + ) + failed[operatorAddress] = true + continue + } + if normalizeAddress(onChain) != normalizeAddress(claimedProvider) { + logger.Errorf( + "operator staking-provider identity mismatch [operator=%s]: "+ + "inventory claim does not match the on-chain WalletRegistry mapping", + operatorAddress, + ) + failed[operatorAddress] = true + } + } + return failed +} + +// zeroAddress is the all-zero Ethereum address, returned by the WalletRegistry +// for an unregistered operator and never a valid staking-provider identity. +const zeroAddress = "0x0000000000000000000000000000000000000000" + +// isZeroAddress reports whether s normalizes to the all-zero address. +func isZeroAddress(s string) bool { + return normalizeAddress(s) == zeroAddress +} + +// isValidImageDigest reports whether s is a full, immutable content digest of the +// form sha256:<64 lowercase hex>. An abbreviated or mutable-tag digest is +// rejected so it cannot only partially pin the exact runtime image. +func isValidImageDigest(s string) bool { + const prefix = "sha256:" + if !strings.HasPrefix(s, prefix) { + return false + } + hexPart := s[len(prefix):] + if len(hexPart) != 64 { + return false + } + for _, ch := range hexPart { + if !((ch >= '0' && ch <= '9') || (ch >= 'a' && ch <= 'f')) { + return false + } + } + return true +} + +// inventoryExpectationContradicts reports whether an authoritative inventory +// entry's own expected release identity contradicts the collector's configured +// expected release. A per-instance expected revision, epoch, or image digest +// that is set but differs from the configured value means the inventory is +// internally inconsistent about what the cutover release is for that instance; +// the entry is treated as an inventory-reconciliation fault so it cannot +// contribute to a resolved status. +func (c *Collector) inventoryExpectationContradicts(inv InventoryInstance) bool { + if inv.ExpectedRevision != "" && + inv.ExpectedRevision != c.config.ExpectedRevision { + return true + } + if inv.ExpectedEpoch != "" && + inv.ExpectedEpoch != c.config.ExpectedEpoch { + return true + } + if inv.ExpectedImageDigest != "" && + inv.ExpectedImageDigest != c.config.ExpectedImageDigest { + return true + } + return false +} + +// reconcileOperatorStatus applies the six reconciliation rules to one operator's +// eligible instance records and returns its status and a human-readable reason. +func (c *Collector) reconcileOperatorStatus( + instances []*instanceRecord, + freshLegacyThisCycle bool, + lastLegacyAt time.Time, +) (FleetStatus, string) { + // Rule 3: a valid post-cutover legacy sighting outranks other blocking + // statuses and reopens/refreshes the operator. + if freshLegacyThisCycle { + return FleetObservedLegacy, "post-cutover legacy wire sighting" + } + + if len(instances) == 0 { + return FleetOfflineUnknown, "no eligible instances reporting" + } + + nonQuarantinedBlocking := 0 + anyNonCutover := false + anyQuarantined := false + allExactConfirmed := true + + for _, inst := range instances { + class := c.classifyInstance(inst) + if class != classExactConfirmed { + allExactConfirmed = false + } + if inst.HasQuarantine { + anyQuarantined = true + continue + } + switch class { + case classExactConfirmed: + // current + case classNonCutoverRevision: + nonQuarantinedBlocking++ + anyNonCutover = true + default: + nonQuarantinedBlocking++ + } + } + + if nonQuarantinedBlocking == 0 { + if allExactConfirmed { + // Rule 4: resolved only if every report is newer than the last + // legacy observation. + if c.allReportsNewerThan(instances, lastLegacyAt) { + return FleetResolvedCurrent, "all instances report exact cutover release" + } + return FleetOfflineUnknown, + "exact reports not yet newer than last legacy observation" + } + // Rule 5: every otherwise-blocking instance is quarantined. + if anyQuarantined { + return FleetQuarantined, "all blocking instances independently quarantined" + } + return FleetOfflineUnknown, "awaiting confirmation" + } + + // Rule 1/2: blocking. noncutover_revision is reported ahead of a bare + // offline/unknown because it is a confirmed stale binary. + if anyNonCutover { + return FleetNonCutoverRevision, "instance reporting a non-cutover revision/epoch/digest" + } + return FleetOfflineUnknown, "instance offline or unconfirmed" +} + +func (c *Collector) classifyInstance(inst *instanceRecord) instanceClass { + // Disappearance from the production service-discovery target set is + // offline_unknown immediately (rule 2): offline is never ready. + if inst.DisappearedFromDiscovery { + return classOfflineUnknown + } + if inst.ConsecutiveMissed >= c.config.MissedThreshold { + return classOfflineUnknown + } + if inst.LatestReport == nil { + return classOfflineUnknown + } + if !c.reportIsExact(*inst.LatestReport) { + return classNonCutoverRevision + } + if inst.ConsecutiveExact >= c.config.SuccessThreshold { + return classExactConfirmed + } + return classOfflineUnknown +} + +func (c *Collector) allReportsNewerThan( + instances []*instanceRecord, + reference time.Time, +) bool { + if reference.IsZero() { + return true + } + for _, inst := range instances { + if inst.LatestReport == nil { + return false + } + if !inst.LatestReport.AttestedAt.After(reference) { + return false + } + } + return true +} + +func (c *Collector) reportIsExact(report InstanceReport) bool { + return report.Revision == c.config.ExpectedRevision && + report.Epoch == c.config.ExpectedEpoch && + report.ImageDigest == c.config.ExpectedImageDigest +} + +func (c *Collector) instanceForInventory(inv InventoryInstance) *instanceRecord { + record, ok := c.instances[inv.InstanceID] + if !ok { + record = &instanceRecord{ + InstanceID: inv.InstanceID, + OperatorAddress: inv.OperatorAddress, + } + c.instances[inv.InstanceID] = record + } + record.OperatorAddress = inv.OperatorAddress + return record +} + +func (c *Collector) operatorForAddress( + address string, + stakingProviders map[string]string, +) *operatorRecord { + record, ok := c.operators[address] + if !ok { + record = &operatorRecord{ + OperatorAddress: address, + StakingProvider: stakingProviders[address], + Status: FleetOfflineUnknown, + } + c.operators[address] = record + } + return record +} + +// reconcileDisappearedInstances returns every known instance record for an +// operator, applying a missed-collection update to any instance absent from the +// current inventory (disappeared from service discovery). A disappeared instance +// loses its exact-confirmation streak and is re-checked for verified quarantine +// using its last-known evidence reference, so removal never silently resolves +// central state: the operator stays blocking until the instance either reappears +// with fresh exact reports or is independently quarantined. +func (c *Collector) reconcileDisappearedInstances( + records []*instanceRecord, + seenInstanceIDs map[string]bool, +) []*instanceRecord { + for _, inst := range records { + if seenInstanceIDs[inst.InstanceID] { + continue + } + inst.ConsecutiveMissed++ + inst.ConsecutiveExact = 0 + inst.HasQuarantine = inst.QuarantineRef != "" && + c.verifier != nil && + c.verifier.Verify(inst.InstanceID, inst.OperatorAddress, inst.QuarantineRef) + } + return records +} + +// purgeResolved removes resolved operator records older than the retention +// window. Unresolved (blocking/quarantined) history is never purged. +func (c *Collector) purgeResolved(now time.Time) { + for address, op := range c.operators { + if op.Status != FleetResolvedCurrent { + continue + } + if op.ResolvedAt.IsZero() { + continue + } + if now.Sub(op.ResolvedAt) > ResolvedRetention { + delete(c.operators, address) + // Drop the resolved operator's instance records too. + for instanceID, inst := range c.instances { + if inst.OperatorAddress == address { + delete(c.instances, instanceID) + } + } + } + } +} + +func (c *Collector) buildSnapshot(now time.Time, currentBlock uint64) FleetSnapshot { + var blocking, quarantined, resolved []FleetOperatorEntry + + for _, op := range c.operators { + entry := c.operatorEntry(op) + switch { + case op.Status == FleetQuarantined: + quarantined = append(quarantined, entry) + case op.Status == FleetResolvedCurrent: + resolved = append(resolved, entry) + case op.Status.IsBlocking(): + blocking = append(blocking, entry) + } + } + + sortEntries(blocking) + sortEntries(quarantined) + sortEntries(resolved) + + // Complete is decided by the caller's fail-closed isComplete; never derive + // it from an empty blocking list alone, which would pass an empty inventory. + return FleetSnapshot{ + SchemaVersion: FleetSnapshotSchemaVersion, + GeneratedAt: now, + CurrentBlock: currentBlock, + CutoverBlock: c.config.CutoverBlock, + Complete: false, + ExpectedRevision: c.config.ExpectedRevision, + ExpectedEpoch: c.config.ExpectedEpoch, + ExpectedDigest: c.config.ExpectedImageDigest, + Blocking: blocking, + Quarantined: quarantined, + RecentlyResolved: resolved, + } +} + +func (c *Collector) operatorEntry(op *operatorRecord) FleetOperatorEntry { + var records []*instanceRecord + for _, inst := range c.instances { + if inst.OperatorAddress == op.OperatorAddress { + records = append(records, inst) + } + } + sort.Slice(records, func(i, j int) bool { + return records[i].InstanceID < records[j].InstanceID + }) + + instances := make([]InstanceReport, 0, len(records)) + statuses := make([]FleetInstanceStatus, 0, len(records)) + for _, inst := range records { + if inst.LatestReport != nil { + instances = append(instances, *inst.LatestReport) + } else { + // Offline / never-reported authoritative instance: surface its + // identity so the audit trail lists every instance the operator + // owns, not only those that produced a report this window. + instances = append(instances, InstanceReport{ + InstanceID: inst.InstanceID, + OperatorAddress: inst.OperatorAddress, + }) + } + statuses = append(statuses, c.instanceStatus(inst)) + } + + return FleetOperatorEntry{ + OperatorAddress: op.OperatorAddress, + StakingProvider: op.StakingProvider, + Status: op.Status, + Instances: instances, + InstanceStatuses: statuses, + FirstSeenBlock: op.FirstSeenBlock, + LastSeenBlock: op.LastSeenBlock, + Reason: op.Reason, + } +} + +// instanceStatus builds the per-instance reconciliation detail for the snapshot, +// pairing the instance's observed identity with its reconciliation class and a +// short human-readable reason. +func (c *Collector) instanceStatus(inst *instanceRecord) FleetInstanceStatus { + class := c.classifyInstance(inst) + status := FleetInstanceStatus{ + InstanceID: inst.InstanceID, + OperatorAddress: inst.OperatorAddress, + Class: instanceClassString(class), + Reason: c.instanceReason(inst, class), + Reported: inst.LatestReport != nil, + ReportedThisCycle: inst.ReportedThisCycle, + CeremonyEligible: inst.CeremonyEligible, + StakingProvider: inst.StakingProvider, + ExpectedRevision: inst.ExpectedRevision, + ExpectedEpoch: inst.ExpectedEpoch, + ExpectedImageDigest: inst.ExpectedImageDigest, + ConsecutiveExact: inst.ConsecutiveExact, + ConsecutiveMissed: inst.ConsecutiveMissed, + Quarantined: inst.HasQuarantine, + QuarantineRef: inst.QuarantineRef, + } + if inst.LatestReport != nil { + status.ObservedRevision = inst.LatestReport.Revision + status.ObservedEpoch = inst.LatestReport.Epoch + status.ObservedDigest = inst.LatestReport.ImageDigest + status.ReporterRevision = inst.LatestReport.ReporterRevision + status.AttestedAt = inst.LatestReport.AttestedAt + } + return status +} + +func instanceClassString(class instanceClass) string { + switch class { + case classExactConfirmed: + return "exact_confirmed" + case classNonCutoverRevision: + return "noncutover_revision" + default: + return "offline_unknown" + } +} + +func (c *Collector) instanceReason(inst *instanceRecord, class instanceClass) string { + if inst.HasQuarantine { + return "independently verified quarantine/removal" + } + switch class { + case classExactConfirmed: + return "exact cutover release confirmed" + case classNonCutoverRevision: + return "reporting a non-cutover revision/epoch/digest" + default: + if inst.LatestReport == nil { + return "no accepted report" + } + if inst.ConsecutiveMissed >= c.config.MissedThreshold { + return "missed consecutive collections" + } + return "awaiting consecutive exact confirmations" + } +} + +func sortEntries(entries []FleetOperatorEntry) { + sort.Slice(entries, func(i, j int) bool { + return entries[i].OperatorAddress < entries[j].OperatorAddress + }) +} + +func (c *Collector) updateMetrics(snapshot FleetSnapshot, stale, unreconciled int) { + blockingOperators := len(snapshot.Blocking) + observedLegacy := 0 + for _, op := range snapshot.Blocking { + if op.Status == FleetObservedLegacy { + observedLegacy++ + } + } + + c.metrics.SetGauge(MetricFleetBlockingOperators, float64(blockingOperators)) + c.metrics.SetGauge(MetricFleetObservedLegacy, float64(observedLegacy)) + c.metrics.SetGauge(MetricReportersStale, float64(stale)) + c.metrics.SetGauge(MetricInventoryUnreconciled, float64(unreconciled)) + + c.metrics.ResetOperatorGauges() + emit := func(entry FleetOperatorEntry) { + status := string(entry.Status) + c.metrics.SetOperatorGauge( + MetricOperatorInfo, entry.OperatorAddress, entry.StakingProvider, status, 1, + ) + c.metrics.SetOperatorGauge( + MetricOperatorFirstSeenBlock, entry.OperatorAddress, entry.StakingProvider, status, + float64(entry.FirstSeenBlock), + ) + c.metrics.SetOperatorGauge( + MetricOperatorLastSeenBlock, entry.OperatorAddress, entry.StakingProvider, status, + float64(entry.LastSeenBlock), + ) + } + for _, entry := range snapshot.Blocking { + emit(entry) + } + for _, entry := range snapshot.Quarantined { + emit(entry) + } + for _, entry := range snapshot.RecentlyResolved { + emit(entry) + } +} + +func (c *Collector) logCycle(snapshot FleetSnapshot) { + noncutover, observedLegacy, offlineUnknown := 0, 0, 0 + for _, op := range snapshot.Blocking { + switch op.Status { + case FleetNonCutoverRevision: + noncutover++ + case FleetObservedLegacy: + observedLegacy++ + case FleetOfflineUnknown: + offlineUnknown++ + } + } + + logger.Infof( + "cutover readiness fleet snapshot [currentBlock=%d] [cutoverBlock=%d] "+ + "[complete=%t] [blockingOperators=%d] [noncutoverRevision=%d] "+ + "[observedLegacy=%d] [offlineUnknown=%d] [quarantined=%d]", + snapshot.CurrentBlock, + snapshot.CutoverBlock, + snapshot.Complete, + len(snapshot.Blocking), + noncutover, + observedLegacy, + offlineUnknown, + len(snapshot.Quarantined), + ) + + for _, op := range snapshot.Blocking { + // reporters is the number of instances that produced an accepted report in + // THIS collection cycle (not "ever reported"), which is distinct from the + // total instance count (the latter includes offline/never-reported and + // disappeared authoritative instances, plus historical reporters that did + // not report this cycle). + reporters := 0 + for _, st := range op.InstanceStatuses { + if st.ReportedThisCycle { + reporters++ + } + } + logger.Infof( + "cutover operator unresolved [operator=%s] [stakingProvider=%s] "+ + "[status=%s] [firstSeenBlock=%d] [lastSeenBlock=%d] "+ + "[reporters=%d] [instances=%d]", + op.OperatorAddress, + op.StakingProvider, + op.Status, + op.FirstSeenBlock, + op.LastSeenBlock, + reporters, + len(op.Instances), + ) + } +} + +// Snapshot returns the most recently computed fleet snapshot. It is safe to call +// concurrently with Collect. +func (c *Collector) Snapshot() FleetSnapshot { + c.mu.RLock() + defer c.mu.RUnlock() + return c.lastSnapshot +} diff --git a/pkg/monitoring/cutoverroster/collector_disappearance_test.go b/pkg/monitoring/cutoverroster/collector_disappearance_test.go new file mode 100644 index 0000000000..a1d824382d --- /dev/null +++ b/pkg/monitoring/cutoverroster/collector_disappearance_test.go @@ -0,0 +1,274 @@ +package cutoverroster + +import ( + "path/filepath" + "testing" + "time" +) + +// TestCollector_DisappearedInstanceKeepsOperatorBlocking proves reconciliation +// rule 6: removal of an instance from the authoritative inventory never resolves +// central state. An operator that is blocking because of one instance must stay +// blocking when that instance simply disappears, even if its other instances are +// exact-confirmed. +func TestCollector_DisappearedInstanceKeepsOperatorBlocking(t *testing.T) { + tc := newTestCollector(t) + + both := []InventoryInstance{ + eligibleInstance("i1", "op1"), + eligibleInstance("i2", "op1"), + } + + // Cycles 1-3: both instances eligible; only i1 reports exact. i2 never + // reports and is offline_unknown, so op1 is blocking throughout while i1 + // accrues its exact-confirmation streak. + block := uint64(1001) + for cycle := 1; cycle <= 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(both, reports, nil, block) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status == FleetResolvedCurrent { + t.Fatalf("operator resolved while i2 was offline at cycle %d", cycle) + } + tc.now = tc.now.Add(time.Minute) + block++ + } + + // Cycle 4: i2 disappears from the inventory. i1 reports exact and is now + // exact-confirmed; without disappearance handling the operator would falsely + // resolve. i2's unverified removal must keep op1 blocking. + onlyI1 := []InventoryInstance{eligibleInstance("i1", "op1")} + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(onlyI1, reports, nil, block) + if err != nil { + t.Fatal(err) + } + status, ok := operatorStatus(snap, "op1") + if !ok { + t.Fatal("op1 missing from snapshot") + } + if status == FleetResolvedCurrent { + t.Fatalf("disappeared instance i2 must keep op1 blocking, got %s", status) + } + if !status.IsBlocking() { + t.Fatalf("expected op1 blocking after i2 disappeared, got %s", status) + } + if snap.Complete { + t.Error("snapshot must not be complete while op1 is blocking") + } +} + +// TestCollector_WholeOperatorDisappearanceStaysBlocking proves the fail-closed +// safety property for a whole-operator disappearance alongside a still-resolved +// operator: an operator that is blocking when every one of its instances vanishes +// from the authoritative inventory remains blocking (its removal is not treated +// as convergence), while a separate operator that keeps reporting exact resolves +// independently. The readiness determination must not become complete while the +// vanished operator is still blocking. +func TestCollector_WholeOperatorDisappearanceStaysBlocking(t *testing.T) { + tc := newTestCollector(t) + + both := []InventoryInstance{ + eligibleInstance("i1", "opBlocking"), + eligibleInstance("i2", "opResolving"), + } + + // Cycles 1-3: opBlocking never reports (offline_unknown, blocking) while + // opResolving reports exact and accrues its confirmation streak. + block := uint64(1001) + for cycle := 1; cycle <= 3; cycle++ { + reports := map[string]InstanceReport{ + "i2": exactReport("i2", "opResolving", tc.now), + } + if _, err := tc.collector.Collect(both, reports, nil, block); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + block++ + } + + // Cycle 4: opBlocking's only instance disappears from the inventory entirely. + // opResolving remains and reports exact, so it resolves; opBlocking must not + // silently drop out of the blocking set just because it vanished. + onlyResolving := []InventoryInstance{eligibleInstance("i2", "opResolving")} + reports := map[string]InstanceReport{ + "i2": exactReport("i2", "opResolving", tc.now), + } + snap, err := tc.collector.Collect(onlyResolving, reports, nil, block) + if err != nil { + t.Fatal(err) + } + + blockingStatus, ok := operatorStatus(snap, "opBlocking") + if !ok { + t.Fatal("vanished blocking operator must be retained in the snapshot") + } + if !blockingStatus.IsBlocking() { + t.Fatalf( + "vanished operator must stay blocking, got %s", blockingStatus, + ) + } + if resolvingStatus, _ := operatorStatus(snap, "opResolving"); resolvingStatus != FleetResolvedCurrent { + t.Fatalf( + "still-present exact operator must resolve independently, got %s", + resolvingStatus, + ) + } + if snap.Complete { + t.Error("readiness must not be complete while the vanished operator blocks") + } +} + +// TestNewCollector_RejectsNonPositiveCollectionInterval proves the collection +// interval is validated at construction, so a zero or negative interval cannot +// reach time.NewTicker and panic the collection loop. +func TestNewCollector_RejectsNonPositiveCollectionInterval(t *testing.T) { + store, err := OpenStore(filepath.Join(t.TempDir(), "roster.db")) + if err != nil { + t.Fatal(err) + } + defer func() { _ = store.Close() }() + + for _, interval := range []time.Duration{0, -time.Second} { + cfg := testConfig() + cfg.CollectionInterval = interval + if _, err := NewCollector(cfg, store, newFakeSink()); err == nil { + t.Fatalf("expected error for collection interval %s", interval) + } + } +} + +// TestCollector_ZeroCutoverBlockNotComplete proves readiness cannot be certified +// while the cutover block C is the placeholder zero, even when every instance is +// exact-confirmed. +func TestCollector_ZeroCutoverBlockNotComplete(t *testing.T) { + store, err := OpenStore(filepath.Join(t.TempDir(), "roster.db")) + if err != nil { + t.Fatal(err) + } + defer func() { _ = store.Close() }() + + cfg := testConfig() + cfg.CutoverBlock = 0 + now := fleetBaseTime + collector, err := newCollectorWithClock( + cfg, store, newFakeSink(), func() time.Time { return now }, + ) + if err != nil { + t.Fatal(err) + } + + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + block := uint64(10) + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", now)} + snap, err = collector.Collect(inventory, reports, nil, block) + if err != nil { + t.Fatal(err) + } + now = now.Add(time.Minute) + block++ + } + if snap.Complete { + t.Fatal("snapshot must not be complete with a zero cutover block") + } +} + +// TestCollector_FutureAttestationRejected proves a report timestamped in the +// future is treated as an invalid attestation: it is an unreconciled fault and +// does not count toward resolution. +func TestCollector_FutureAttestationRejected(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + future := tc.now.Add(time.Hour) + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", future)} + snap, err := tc.collector.Collect(inventory, reports, nil, 1001) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status == FleetResolvedCurrent { + t.Fatal("future-dated attestation must not count toward resolution") + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Error("future attestation should raise the unreconciled gauge") + } +} + +// TestCollector_IncompleteRaisesWatchedGauge proves that any incomplete readiness +// determination raises at least one of the blocking/stale/unreconciled gauges, so +// the CutoverRosterIncomplete alert fires instead of the fault staying silent. +func TestCollector_IncompleteRaisesWatchedGauge(t *testing.T) { + tc := newTestCollector(t) + + // Empty inventory: readiness cannot be established at all. + snap, err := tc.collector.Collect(nil, nil, nil, 2000) + if err != nil { + t.Fatal(err) + } + if snap.Complete { + t.Fatal("empty inventory must not be complete") + } + if tc.sink.gauge(MetricFleetBlockingOperators) == 0 && + tc.sink.gauge(MetricReportersStale) == 0 && + tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Error("incomplete readiness must raise at least one watched gauge") + } + if snap.Inventory.Unreconciled == 0 { + t.Error("snapshot inventory should record the readiness fault") + } +} + +// TestCollector_SnapshotInventoryCountsAndInstanceStatuses proves the snapshot +// exposes inventory totals and per-instance reconciliation detail, and that the +// reporter count is distinct from the total instance count. +func TestCollector_SnapshotInventoryCountsAndInstanceStatuses(t *testing.T) { + tc := newTestCollector(t) + + inventory := []InventoryInstance{ + eligibleInstance("i1", "op1"), + eligibleInstance("i2", "op1"), + {InstanceID: "i3", OperatorAddress: opAddr("op2"), CeremonyEligible: false}, + } + // i1 reports exact; i2 never reports, so op1 is blocking (i2 offline). + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inventory, reports, nil, 1001) + if err != nil { + t.Fatal(err) + } + + if snap.Inventory.TotalInstances != 3 { + t.Errorf("total instances = %d, want 3", snap.Inventory.TotalInstances) + } + if snap.Inventory.EligibleInstances != 2 { + t.Errorf("eligible instances = %d, want 2", snap.Inventory.EligibleInstances) + } + + var op1 *FleetOperatorEntry + for i := range snap.Blocking { + if snap.Blocking[i].OperatorAddress == opAddr("op1") { + op1 = &snap.Blocking[i] + } + } + if op1 == nil { + t.Fatal("op1 not blocking") + } + if len(op1.InstanceStatuses) != 2 { + t.Fatalf("expected 2 instance statuses, got %d", len(op1.InstanceStatuses)) + } + reported := 0 + for _, st := range op1.InstanceStatuses { + if st.Reported { + reported++ + } + } + if reported != 1 { + t.Errorf( + "expected exactly 1 reporting instance of %d, got %d", + len(op1.InstanceStatuses), reported, + ) + } +} diff --git a/pkg/monitoring/cutoverroster/collector_hardening_test.go b/pkg/monitoring/cutoverroster/collector_hardening_test.go new file mode 100644 index 0000000000..946a0b24f2 --- /dev/null +++ b/pkg/monitoring/cutoverroster/collector_hardening_test.go @@ -0,0 +1,561 @@ +package cutoverroster + +import ( + "encoding/json" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +func newTestCollectorConfig(t *testing.T, cfg CollectorConfig) *testCollector { + t.Helper() + store, err := OpenStore(filepath.Join(t.TempDir(), "roster.db")) + if err != nil { + t.Fatalf("cannot open store: %v", err) + } + tc := &testCollector{store: store, sink: newFakeSink(), now: fleetBaseTime} + collector, err := newCollectorWithClock( + cfg, store, tc.sink, func() time.Time { return tc.now }, + ) + if err != nil { + t.Fatalf("cannot construct collector: %v", err) + } + collector.SetQuarantineVerifier(testQuarantineVerifier()) + // Match newTestCollectorAtPath: satisfy the mandatory trust-chain completeness + // requirements so configuration-specific tests still exercise completeness. + collector.SetIdentityVerifier(derivedIdentityVerifier{}) + collector.SetServiceDiscoveryConfigured(true) + tc.collector = collector + t.Cleanup(func() { _ = store.Close() }) + return tc +} + +func resolveOperator(t *testing.T, tc *testCollector, inv []InventoryInstance, instanceID, operatorAddr string, block uint64) FleetSnapshot { + t.Helper() + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{ + instanceID: exactReport(instanceID, operatorAddr, tc.now), + } + var err error + snap, err = tc.collector.Collect(inv, reports, nil, block) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + return snap +} + +// TestInventoryInput_TargetIngestedButNeverSerialized proves the report target +// is accepted on input (via the explicit JSON key) yet never serialized back out +// of an InventoryInstance. +func TestInventoryInput_TargetIngestedButNeverSerialized(t *testing.T) { + raw := `[{ + "instance_id": "i1", + "operator_address": "0xabc", + "ceremony_eligible": true, + "trusted_report_target": "https://reports.example/i1" + }]` + + var inputs []InventoryInstanceInput + if err := json.Unmarshal([]byte(raw), &inputs); err != nil { + t.Fatalf("cannot decode inventory input: %v", err) + } + if len(inputs) != 1 { + t.Fatalf("expected 1 input, got %d", len(inputs)) + } + + inv := inputs[0].ToInventoryInstance() + if inv.TrustedReportTarget != "https://reports.example/i1" { + t.Errorf("target not ingested: %q", inv.TrustedReportTarget) + } + + // The in-memory InventoryInstance must never serialize the target. + out, err := json.Marshal(inv) + if err != nil { + t.Fatalf("cannot marshal inventory instance: %v", err) + } + if strings.Contains(string(out), "reports.example") || + strings.Contains(string(out), "trusted_report_target") { + t.Errorf("InventoryInstance leaked the trusted report target: %s", out) + } +} + +// TestCollector_EmptyInventoryNotComplete proves the fail-closed default: an +// empty/missing authoritative inventory can never be complete. +func TestCollector_EmptyInventoryNotComplete(t *testing.T) { + tc := newTestCollector(t) + snap, err := tc.collector.Collect(nil, nil, nil, 2000) + if err != nil { + t.Fatal(err) + } + if snap.Complete { + t.Errorf("empty inventory must never be complete") + } +} + +// TestCollector_FreshBlockRequiredForComplete proves that a resolved fleet is +// still not complete without a fresh (nonzero) current block. +func TestCollector_FreshBlockRequiredForComplete(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Resolve at a fresh block: complete. + snap := resolveOperator(t, tc, inv, "i1", "op1", 1000) + if !snap.Complete { + t.Fatalf("expected complete with a fresh block, got not complete") + } + + // One more cycle with currentBlock 0 (chain clock unavailable): not complete. + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inv, reports, nil, 0) + if err != nil { + t.Fatal(err) + } + if snap.Complete { + t.Errorf("must not be complete when the current block is zero") + } +} + +// TestCollector_ExpectedIdentityRequiredForComplete proves that an unset +// expected artifact identity or chain ID can never produce complete=true. +func TestCollector_ExpectedIdentityRequiredForComplete(t *testing.T) { + cfg := testConfig() + cfg.ExpectedImageDigest = "" // missing expected artifact identity + tc := newTestCollectorConfig(t, cfg) + + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + // The instance reports the expected revision/epoch but the collector's own + // expected digest is unset, so readiness must fail closed. + snap := resolveOperator(t, tc, inv, "i1", "op1", 1000) + if snap.Complete { + t.Errorf("must not be complete with an unset expected image digest") + } +} + +// TestCollector_PreCutoverSightingIgnored proves a legacy sighting before the +// cutover block is not valid post-cutover straggler evidence. +func TestCollector_PreCutoverSightingIgnored(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + resolveOperator(t, tc, inv, "i1", "op1", 1000) + + // CutoverBlock is 1000; a sighting at block 900 is pre-cutover. + sightings := []LegacySighting{ + {OperatorAddress: opAddr("op1"), Block: 900, ObservedAt: tc.now}, + } + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inv, reports, sightings, 1100) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status == FleetObservedLegacy { + t.Errorf("pre-cutover sighting must not produce observed_legacy") + } + if tc.sink.gauge(MetricFleetObservedLegacy) != 0 { + t.Errorf("pre-cutover sighting must not increment observed-legacy gauge") + } +} + +// TestCollector_FutureSightingIgnored proves a sighting past the current block is +// ignored. +func TestCollector_FutureSightingIgnored(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + resolveOperator(t, tc, inv, "i1", "op1", 1000) + + sightings := []LegacySighting{ + {OperatorAddress: opAddr("op1"), Block: 5000, ObservedAt: tc.now}, + } + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inv, reports, sightings, 1100) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status == FleetObservedLegacy { + t.Errorf("future sighting must not produce observed_legacy") + } +} + +// TestCollector_AddressNormalization proves inventory and sightings that refer to +// one operator with different casing deduplicate to a single operator record. +func TestCollector_AddressNormalization(t *testing.T) { + tc := newTestCollector(t) + + inv := []InventoryInstance{eligibleInstance("i1", "0xABCdef0000000000000000000000000000000001")} + // The node-local sighting is already lowercase; it must map to the same + // operator, not a second one. + sightings := []LegacySighting{ + {OperatorAddress: "0xabcdef0000000000000000000000000000000001", Block: 1100, ObservedAt: tc.now}, + } + snap, err := tc.collector.Collect(inv, map[string]InstanceReport{}, sightings, 1100) + if err != nil { + t.Fatal(err) + } + + total := len(snap.Blocking) + len(snap.Quarantined) + len(snap.RecentlyResolved) + if total != 1 { + t.Fatalf("expected exactly 1 deduplicated operator, got %d: %+v", total, snap) + } + if status, _ := operatorStatus(snap, "0xABCDEF0000000000000000000000000000000001"); status != FleetObservedLegacy { + t.Errorf("expected the case-different sighting to attach to the same operator") + } +} + +// TestCollector_ReplayedReportRejected proves an attestation that is not newer +// than the last accepted one is treated as a missed collection, not accepted. +func TestCollector_ReplayedReportRejected(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + report := exactReport("i1", "op1", tc.now) + // First cycle accepts. + if _, err := tc.collector.Collect(inv, map[string]InstanceReport{"i1": report}, nil, 1000); err != nil { + t.Fatal(err) + } + // Replay the identical attestation (same AttestedAt, same ReporterRevision) + // twice more. It must not advance ConsecutiveExact; the instance instead + // accrues missed collections and becomes offline. + var snap FleetSnapshot + for cycle := 0; cycle < 2; cycle++ { + var err error + snap, err = tc.collector.Collect(inv, map[string]InstanceReport{"i1": report}, nil, 1000) + if err != nil { + t.Fatal(err) + } + } + if status, _ := operatorStatus(snap, "op1"); status != FleetOfflineUnknown { + t.Errorf("replayed reports must not resolve; got %s", status) + } +} + +// TestCollector_MissingAttestationTimeRejected proves a report without an +// attestation time is rejected and counted as an inventory-reconciliation +// failure, not accepted. +func TestCollector_MissingAttestationTimeRejected(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + report := exactReport("i1", "op1", tc.now) + report.AttestedAt = time.Time{} // missing + + snap, err := tc.collector.Collect(inv, map[string]InstanceReport{"i1": report}, nil, 1000) + if err != nil { + t.Fatal(err) + } + if snap.Complete { + t.Errorf("a report missing its attestation time must not yield completeness") + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("a missing attestation time must count as unreconciled") + } +} + +// TestCollector_ConcurrentCollectAndSnapshot exercises the collector's mutex: +// concurrent Collect writes and HTTP-style Snapshot reads must be race-free. +func TestCollector_ConcurrentCollectAndSnapshot(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + stop := make(chan struct{}) + var wg sync.WaitGroup + wg.Add(1) + go func() { + defer wg.Done() + for { + select { + case <-stop: + return + default: + _ = tc.collector.Snapshot() + } + } + }() + + for i := 0; i < 50; i++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + if _, err := tc.collector.Collect(inv, reports, nil, uint64(1000+i)); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + close(stop) + wg.Wait() +} + +// TestCollector_MalformedIdentityRejected proves an eligible inventory entry with +// a missing instance or operator identity is rejected as an inventory- +// reconciliation fault: it cannot be reconciled or counted as an eligible +// instance, and it forces readiness closed. +func TestCollector_MalformedIdentityRejected(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{ + eligibleInstance("", "op1"), // missing instance ID + eligibleInstance("i2", ""), // missing operator address + } + snap, err := tc.collector.Collect(inv, nil, nil, 1000) + if err != nil { + t.Fatal(err) + } + if snap.Complete { + t.Errorf("a malformed inventory identity must not yield completeness") + } + if snap.Inventory.EligibleInstances != 0 { + t.Errorf( + "malformed entries must not count as reconciled eligible; got %d", + snap.Inventory.EligibleInstances, + ) + } + if snap.Inventory.Unreconciled < 2 { + t.Errorf( + "both malformed entries must count as unreconciled; got %d", + snap.Inventory.Unreconciled, + ) + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("a malformed identity must raise the unreconciled gauge") + } +} + +// TestCollector_MissingReportIdentityRejected proves a report that does not +// self-identify — an empty instance ID or operator address — is rejected as an +// unreconciled fault and cannot resolve the operator, matching the reporter's +// documented contract that missing identity is rejected rather than fabricated +// from inventory. +func TestCollector_MissingReportIdentityRejected(t *testing.T) { + for _, tt := range []struct { + name string + mutate func(*InstanceReport) + }{ + {"empty instance id", func(r *InstanceReport) { r.InstanceID = "" }}, + {"empty operator address", func(r *InstanceReport) { r.OperatorAddress = "" }}, + } { + t.Run(tt.name, func(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + report := exactReport("i1", "op1", tc.now) + tt.mutate(&report) + var err error + snap, err = tc.collector.Collect( + inv, map[string]InstanceReport{"i1": report}, nil, 1001, + ) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + if status, _ := operatorStatus(snap, "op1"); status == FleetResolvedCurrent { + t.Errorf("a report without a self-identity must not resolve the operator") + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("a report without a self-identity must count as unreconciled") + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("a missing report identity must raise the unreconciled gauge") + } + }) + } +} + +// TestCollector_DuplicateInstanceIDRejected proves a duplicate instance ID within +// one inventory cycle is flagged: the first entry is reconciled, the duplicate +// cannot silently overwrite it or create a second operator record, and readiness +// fails closed. +func TestCollector_DuplicateInstanceIDRejected(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{ + eligibleInstance("i1", "op1"), + eligibleInstance("i1", "op2"), // duplicate instance ID, different operator + } + snap, err := tc.collector.Collect( + inv, + map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)}, + nil, + 1000, + ) + if err != nil { + t.Fatal(err) + } + if snap.Complete { + t.Errorf("a duplicate instance ID must not yield completeness") + } + // Exactly one entry is reconciled eligible (the first i1); the duplicate is a + // fault, not a second eligible instance. + if snap.Inventory.EligibleInstances != 1 { + t.Errorf( + "expected 1 reconciled eligible instance, got %d", + snap.Inventory.EligibleInstances, + ) + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("the duplicate instance ID must count as unreconciled") + } + if _, ok := operatorStatus(snap, "op2"); ok { + t.Errorf("the duplicate entry must not create a second operator record") + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("the duplicate instance ID must raise the unreconciled gauge") + } +} + +// TestCollector_ContradictoryInstanceExpectationRejected proves an eligible +// inventory entry whose own expected release identity contradicts the collector's +// configured expected release is rejected as an inventory-reconciliation fault +// and cannot resolve the operator even with otherwise-exact reports. +func TestCollector_ContradictoryInstanceExpectationRejected(t *testing.T) { + tc := newTestCollector(t) + contradictory := eligibleInstance("i1", "op1") + contradictory.ExpectedRevision = "some-other-revision" // contradicts config + inv := []InventoryInstance{contradictory} + + // Even three exact reports (matching the collector config) must not resolve + // the operator, because the inventory disagrees with the config about what the + // cutover release is for this instance. + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inv, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if snap.Complete { + t.Errorf("a contradictory per-instance expectation must not yield completeness") + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("a contradictory per-instance expectation must count as unreconciled") + } + if status, _ := operatorStatus(snap, "op1"); status == FleetResolvedCurrent { + t.Errorf("a contradictory per-instance expectation must not resolve the operator") + } +} + +// TestCollector_RecordInputUnavailableFailsClosed proves that once an +// authoritative input becomes unreadable, a previously-certified "complete=true" +// readiness snapshot is superseded by an incomplete one carrying a nonzero +// inventory-unreconciled signal, and that persisted operator history is left +// intact so a transient input blip neither resolves nor ages any operator. +func TestCollector_RecordInputUnavailableFailsClosed(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Drive the operator to resolved_current across three exact collections so the + // last published snapshot is genuinely complete with zero watched gauges. + var snap FleetSnapshot + for cycle := 1; cycle <= 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inventory, reports, nil, uint64(1000+cycle)) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if !snap.Complete { + t.Fatalf("precondition failed: expected a complete snapshot after three exact reports") + } + if tc.sink.gauge(MetricInventoryUnreconciled) != 0 { + t.Fatalf("precondition failed: expected zero unreconciled before the input failure") + } + + // The next cycle cannot read its authoritative input. + failed := tc.collector.RecordInputUnavailable(1004) + + if failed.Complete { + t.Errorf("an unavailable authoritative input must not certify readiness") + } + if failed.Inventory.Unreconciled == 0 { + t.Errorf("an unavailable authoritative input must surface as unreconciled") + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("expected a nonzero unreconciled gauge after the input failure") + } + if failed.CurrentBlock != 1004 { + t.Errorf("expected the failed snapshot to be stamped with the read block, got %d", failed.CurrentBlock) + } + + // The most recently served snapshot must be the incomplete one, not the stale + // complete snapshot from the prior cycle. + served := tc.collector.Snapshot() + if served.Complete { + t.Errorf("the served snapshot must be incomplete while the input is unavailable") + } + + // Persisted history is untouched: op1 is still recorded as resolved_current, it + // was neither aged out nor reopened by the transient input failure. + if status, ok := operatorStatus(served, "op1"); !ok || status != FleetResolvedCurrent { + t.Errorf("expected op1 to remain resolved_current after the input failure, got %s (present=%t)", status, ok) + } +} + +// TestCollector_PersistenceFailureFailsClosed proves that a collection cycle whose +// central-state write fails also fails readiness closed: even though the inputs +// were fully readable and the reconciliation succeeded, a persistence error must +// supersede a previously-certified "complete=true" snapshot with an incomplete one +// carrying a nonzero inventory-unreconciled signal, rather than leaving the stale +// complete snapshot and its zero gauges served. This covers the Collect error path, +// complementing the unreadable-input path above. +func TestCollector_PersistenceFailureFailsClosed(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Drive the operator to resolved_current across three exact collections so the + // last published snapshot is genuinely complete with zero watched gauges. + var snap FleetSnapshot + for cycle := 1; cycle <= 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inventory, reports, nil, uint64(1000+cycle)) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if !snap.Complete { + t.Fatalf("precondition failed: expected a complete snapshot after three exact reports") + } + if tc.sink.gauge(MetricInventoryUnreconciled) != 0 { + t.Fatalf("precondition failed: expected zero unreconciled before the persistence failure") + } + + // Force the next cycle's central-state write to fail by closing the bbolt store + // out from under the collector. The inputs are still perfectly readable. + if err := tc.store.Close(); err != nil { + t.Fatalf("cannot close store to induce a persistence failure: %v", err) + } + + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + _, err := tc.collector.Collect(inventory, reports, nil, 1004) + if err == nil { + t.Fatalf("expected Collect to return an error when persistence fails") + } + + // The most recently served snapshot must be the incomplete one, not the stale + // complete snapshot from the prior cycle. + served := tc.collector.Snapshot() + if served.Complete { + t.Errorf("the served snapshot must be incomplete after a persistence failure") + } + if served.Inventory.Unreconciled == 0 { + t.Errorf("a persistence failure must surface as unreconciled") + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("expected a nonzero unreconciled gauge after the persistence failure") + } + + // Persisted history is untouched: op1 remains resolved_current in the served + // snapshot; the failed write neither aged it out nor reopened it. + if status, ok := operatorStatus(served, "op1"); !ok || status != FleetResolvedCurrent { + t.Errorf("expected op1 to remain resolved_current after the persistence failure, got %s (present=%t)", status, ok) + } +} diff --git a/pkg/monitoring/cutoverroster/collector_test.go b/pkg/monitoring/cutoverroster/collector_test.go new file mode 100644 index 0000000000..46daf99ef1 --- /dev/null +++ b/pkg/monitoring/cutoverroster/collector_test.go @@ -0,0 +1,766 @@ +package cutoverroster + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "net/http" + "net/http/httptest" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +// opAddr maps a symbolic test operator name to a canonical (lowercase 0x + 40 +// hex) address so tests can keep using readable names while the collector +// enforces canonical addresses. An empty name maps to the empty string (so the +// malformed-identity tests still exercise a missing address), and a value that +// already looks like a 0x address is passed through lowercased. +func opAddr(name string) string { + if name == "" { + return "" + } + lower := strings.ToLower(strings.TrimSpace(name)) + if strings.HasPrefix(lower, "0x") && len(lower) == 42 { + return lower + } + sum := sha256.Sum256([]byte(name)) + return "0x" + hex.EncodeToString(sum[:20]) +} + +const ( + testRevision = "abc123def456" + // testDigest is a full, immutable content digest (sha256:<64 hex>). The + // collector now rejects an abbreviated digest, so the fixture uses a complete + // one (this happens to be the sha256 of the empty string). + testDigest = "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" +) + +// spForOperator derives a canonical (lowercase 0x + 40 hex) staking-provider +// address deterministically from an operator address, so test inventory carries +// a canonical staking provider (the collector now rejects a non-address one) and +// the default identity verifier can independently reproduce the expected mapping. +func spForOperator(operatorAddress string) string { + sum := sha256.Sum256([]byte("sp:" + normalizeAddress(operatorAddress))) + return "0x" + hex.EncodeToString(sum[:20]) +} + +// derivedIdentityVerifier is the default test identity verifier. It returns the +// same canonical staking provider spForOperator derives, so an operator whose +// inventory claim matches verifies successfully while a mismatched or non-derived +// claim fails — a real verification, not an echo of the inventory claim. +type derivedIdentityVerifier struct{} + +func (derivedIdentityVerifier) OperatorStakingProviderAtBlock( + _ context.Context, operatorAddress string, _ uint64, +) (string, error) { + op := normalizeAddress(operatorAddress) + if !isCanonicalAddress(op) { + return "", errNoIdentity + } + return spForOperator(op), nil +} + +var fleetBaseTime = time.Date(2026, 7, 24, 12, 0, 0, 0, time.UTC) + +// fakeSink is a recording MetricsSink. +type fakeSink struct { + mu sync.Mutex + gauges map[string]float64 + operatorGauges map[string]float64 +} + +func newFakeSink() *fakeSink { + return &fakeSink{ + gauges: map[string]float64{}, + operatorGauges: map[string]float64{}, + } +} + +func (s *fakeSink) SetGauge(name string, value float64) { + s.mu.Lock() + defer s.mu.Unlock() + s.gauges[name] = value +} + +func (s *fakeSink) SetOperatorGauge(name, addr, provider, status string, value float64) { + s.mu.Lock() + defer s.mu.Unlock() + s.operatorGauges[name+"|"+addr+"|"+provider+"|"+status] = value +} + +func (s *fakeSink) ResetOperatorGauges() { + s.mu.Lock() + defer s.mu.Unlock() + s.operatorGauges = map[string]float64{} +} + +func (s *fakeSink) gauge(name string) float64 { + s.mu.Lock() + defer s.mu.Unlock() + return s.gauges[name] +} + +func testConfig() CollectorConfig { + return CollectorConfig{ + ExpectedRevision: testRevision, + ExpectedEpoch: ExpectedEpochSecurityV2Cutover, + ExpectedImageDigest: testDigest, + CutoverBlock: 1000, + ChainID: "1", + CollectionInterval: time.Minute, + MissedThreshold: 2, + SuccessThreshold: 3, + // Production readiness requires the full trust chain. The test collector + // (newTestCollectorAtPath) installs the default identity verifier and marks + // discovery configured, so completeness is reachable while these stay on. + RequireServiceDiscovery: true, + RequireIdentityVerification: true, + } +} + +func eligibleInstance(instanceID, operatorName string) InventoryInstance { + op := opAddr(operatorName) + return InventoryInstance{ + InstanceID: instanceID, + OperatorAddress: op, + StakingProvider: spForOperator(op), + // A unique per-instance network ID: the collector now requires one (and + // requires it to be distinct) whenever the trust chain is enforced, so two + // instances cannot collapse onto one responding node. Derived from the + // instance ID so every fixture instance is automatically distinct. + NetworkID: "net-" + instanceID, + CeremonyEligible: true, + ExpectedRevision: testRevision, + ExpectedEpoch: ExpectedEpochSecurityV2Cutover, + ExpectedImageDigest: testDigest, + TrustedReportTarget: "https://reports.example/" + instanceID, + } +} + +// reporterRevisionFor derives a nonzero, monotonically non-decreasing reporter +// revision from the attestation time so the collector's replay/downgrade guard +// accepts a genuinely advancing report while rejecting a stale replay. +func reporterRevisionFor(at time.Time) uint64 { + return uint64(at.Unix()) +} + +func exactReport(instanceID, operatorName string, at time.Time) InstanceReport { + return InstanceReport{ + InstanceID: instanceID, + OperatorAddress: opAddr(operatorName), + NetworkID: "net-" + instanceID, + Revision: testRevision, + Epoch: ExpectedEpochSecurityV2Cutover, + ImageDigest: testDigest, + AttestedAt: at, + ReporterRevision: reporterRevisionFor(at), + } +} + +func staleReport(instanceID, operatorName string, at time.Time) InstanceReport { + return InstanceReport{ + InstanceID: instanceID, + OperatorAddress: opAddr(operatorName), + NetworkID: "net-" + instanceID, + Revision: "old-revision", + Epoch: ExpectedEpochSecurityV2Cutover, + ImageDigest: testDigest, + AttestedAt: at, + ReporterRevision: reporterRevisionFor(at), + } +} + +// testQuarantineVerifier verifies the specific evidence references used by the +// tests. It is installed on every test collector so verified quarantine survives +// a restart; any reference not listed here fails verification (fail closed). +func testQuarantineVerifier() QuarantineVerifier { + return NewAllowlistQuarantineVerifier([]VerifiedQuarantineEntry{ + {InstanceID: "i1", OperatorAddress: opAddr("op1"), EvidenceRef: "evidence://verified/op1"}, + {InstanceID: "i-quar", OperatorAddress: opAddr("opQuarantined"), EvidenceRef: "evidence://verified/opQuarantined"}, + }) +} + +type testCollector struct { + collector *Collector + store *Store + sink *fakeSink + now time.Time +} + +func newTestCollector(t *testing.T) *testCollector { + t.Helper() + return newTestCollectorAtPath(t, filepath.Join(t.TempDir(), "roster.db")) +} + +func newTestCollectorAtPath(t *testing.T, path string) *testCollector { + t.Helper() + store, err := OpenStore(path) + if err != nil { + t.Fatalf("cannot open store: %v", err) + } + tc := &testCollector{store: store, sink: newFakeSink(), now: fleetBaseTime} + collector, err := newCollectorWithClock( + testConfig(), + store, + tc.sink, + func() time.Time { return tc.now }, + ) + if err != nil { + t.Fatalf("cannot construct collector: %v", err) + } + collector.SetQuarantineVerifier(testQuarantineVerifier()) + // Install the default identity verifier and mark discovery configured so the + // mandatory-trust-chain completeness requirements are satisfied. Tests that + // exercise identity mismatch install their own verifier over this default. + collector.SetIdentityVerifier(derivedIdentityVerifier{}) + collector.SetServiceDiscoveryConfigured(true) + tc.collector = collector + t.Cleanup(func() { _ = store.Close() }) + return tc +} + +func operatorStatus(snapshot FleetSnapshot, addr string) (FleetStatus, bool) { + // Operator addresses are canonicalized on ingestion, so map the query (a + // symbolic test name or a raw address) through the same mapping. + want := opAddr(addr) + for _, group := range [][]FleetOperatorEntry{ + snapshot.Blocking, snapshot.Quarantined, snapshot.RecentlyResolved, + } { + for _, e := range group { + if e.OperatorAddress == want { + return e.Status, true + } + } + } + return "", false +} + +func TestCollector_ThreeExactReportsResolve(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + for cycle := 1; cycle <= 2; cycle++ { + reports := map[string]InstanceReport{ + "i1": exactReport("i1", "op1", tc.now), + } + snap, err := tc.collector.Collect(inventory, reports, nil, uint64(1000+cycle)) + if err != nil { + t.Fatal(err) + } + status, _ := operatorStatus(snap, "op1") + if status == FleetResolvedCurrent { + t.Fatalf("operator resolved too early at cycle %d", cycle) + } + tc.now = tc.now.Add(time.Minute) + } + + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inventory, reports, nil, 1003) + if err != nil { + t.Fatal(err) + } + status, _ := operatorStatus(snap, "op1") + if status != FleetResolvedCurrent { + t.Fatalf("expected resolved_current after three exact reports, got %s", status) + } + if !snap.Complete { + t.Errorf("expected snapshot to be complete") + } + if tc.sink.gauge(MetricFleetBlockingOperators) != 0 { + t.Errorf("expected zero blocking operators") + } +} + +func TestCollector_PreCutoverExactWithoutMismatch(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + var snap FleetSnapshot + // currentBlock 500 is before the cutover block 1000; exact reports still + // resolve, and no mismatch/legacy status is fabricated. + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inventory, reports, nil, 500) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + status, _ := operatorStatus(snap, "op1") + if status != FleetResolvedCurrent { + t.Fatalf("expected resolved_current pre-cutover, got %s", status) + } + if tc.sink.gauge(MetricFleetObservedLegacy) != 0 { + t.Errorf("expected no observed-legacy operators pre-cutover") + } +} + +func TestCollector_TwoMissedReportsOffline(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Establish resolution first. + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + if _, err := tc.collector.Collect(inventory, reports, nil, 1000); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + // Two consecutive missed collections -> offline_unknown and blocking. + var snap FleetSnapshot + for cycle := 0; cycle < 2; cycle++ { + var err error + snap, err = tc.collector.Collect(inventory, map[string]InstanceReport{}, nil, 1010) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + status, _ := operatorStatus(snap, "op1") + if status != FleetOfflineUnknown { + t.Fatalf("expected offline_unknown after two missed reports, got %s", status) + } + if snap.Complete { + t.Errorf("snapshot must not be complete when an operator is offline") + } +} + +func TestCollector_NonCutoverRevisionBlocks(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + // An instance reporting a stale revision is noncutover_revision, before or + // after the cutover block. + reports := map[string]InstanceReport{"i1": staleReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inventory, reports, nil, 1100) + if err != nil { + t.Fatal(err) + } + status, _ := operatorStatus(snap, "op1") + if status != FleetNonCutoverRevision { + t.Fatalf("expected noncutover_revision, got %s", status) + } + if snap.Complete { + t.Errorf("snapshot must not be complete with a noncutover instance") + } +} + +func TestCollector_PostCutoverLegacyReopens(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + if _, err := tc.collector.Collect(inventory, reports, nil, 1000); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + // A fresh post-cutover legacy sighting reopens the operator. + sightings := []LegacySighting{ + {OperatorAddress: opAddr("op1"), Block: 1100, ObservedAt: tc.now}, + } + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inventory, reports, sightings, 1100) + if err != nil { + t.Fatal(err) + } + status, _ := operatorStatus(snap, "op1") + if status != FleetObservedLegacy { + t.Fatalf("expected observed_legacy after a post-cutover sighting, got %s", status) + } + if tc.sink.gauge(MetricFleetObservedLegacy) != 1 { + t.Errorf("expected observed-legacy gauge to be 1") + } +} + +func TestCollector_SameOperatorMultiInstanceBlocking(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{ + eligibleInstance("i1", "op1"), + eligibleInstance("i2", "op1"), + } + + // i1 reports exact three times; i2 never reports (offline). + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inventory, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + status, _ := operatorStatus(snap, "op1") + if status != FleetOfflineUnknown { + t.Fatalf("expected the operator to remain blocking while one instance is offline, got %s", status) + } +} + +func TestCollector_VerifiedQuarantineOnly(t *testing.T) { + tc := newTestCollector(t) + + // A single blocking (never-reporting) instance without quarantine evidence + // keeps the operator blocking. + blocking := eligibleInstance("i1", "op1") + var snap FleetSnapshot + for cycle := 0; cycle < 2; cycle++ { + var err error + snap, err = tc.collector.Collect( + []InventoryInstance{blocking}, map[string]InstanceReport{}, nil, 1000, + ) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if status, _ := operatorStatus(snap, "op1"); status != FleetOfflineUnknown { + t.Fatalf("expected offline_unknown without quarantine evidence, got %s", status) + } + + // Adding independently verified quarantine evidence flips it to quarantined. + quarantined := blocking + quarantined.QuarantineEvidenceRef = "evidence://verified/op1" + snap, err := tc.collector.Collect( + []InventoryInstance{quarantined}, map[string]InstanceReport{}, nil, 1000, + ) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status != FleetQuarantined { + t.Fatalf("expected quarantined with verified evidence, got %s", status) + } + if !snap.Complete { + t.Errorf("a fully quarantined fleet has no blocking operators and is complete") + } +} + +// TestCollector_UnverifiedQuarantineStaysBlocking is the negative quarantine +// case: an evidence reference that is not independently verified (absent from +// the verifier allowlist) does not quarantine the operator; it stays blocking. +func TestCollector_UnverifiedQuarantineStaysBlocking(t *testing.T) { + tc := newTestCollector(t) + + unverified := eligibleInstance("i1", "op1") + // A plausible-looking but not independently-verified reference. + unverified.QuarantineEvidenceRef = "evidence://unverified/op1" + + var snap FleetSnapshot + for cycle := 0; cycle < 2; cycle++ { + var err error + snap, err = tc.collector.Collect( + []InventoryInstance{unverified}, map[string]InstanceReport{}, nil, 1000, + ) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + if status, _ := operatorStatus(snap, "op1"); status != FleetOfflineUnknown { + t.Fatalf("unverified quarantine evidence must not quarantine; got %s", status) + } + if snap.Complete { + t.Errorf("snapshot must not be complete with an unverified, blocking operator") + } +} + +func TestCollector_DistinctStatesSurviveRestart(t *testing.T) { + path := filepath.Join(t.TempDir(), "roster.db") + tc := newTestCollectorAtPath(t, path) + + inventory := []InventoryInstance{ + eligibleInstance("i-res", "opResolved"), + eligibleInstance("i-off", "opOffline"), + func() InventoryInstance { + inv := eligibleInstance("i-quar", "opQuarantined") + inv.QuarantineEvidenceRef = "evidence://verified/opQuarantined" + return inv + }(), + } + + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{ + "i-res": exactReport("i-res", "opResolved", tc.now), + // i-off and i-quar never report. + } + if _, err := tc.collector.Collect(inventory, reports, nil, 1000); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + assertStates := func(t *testing.T, snap FleetSnapshot) { + t.Helper() + checks := map[string]FleetStatus{ + "opResolved": FleetResolvedCurrent, + "opOffline": FleetOfflineUnknown, + "opQuarantined": FleetQuarantined, + } + for addr, want := range checks { + got, ok := operatorStatus(snap, addr) + if !ok { + t.Errorf("operator %s missing from snapshot", addr) + continue + } + if got != want { + t.Errorf("operator %s: got %s, want %s", addr, got, want) + } + } + } + assertStates(t, tc.collector.Snapshot()) + + // Restart the collector against the same bbolt file. + if err := tc.store.Close(); err != nil { + t.Fatal(err) + } + reopened := newTestCollectorAtPath(t, path) + reopened.now = tc.now + + // One more cycle with the same inputs must preserve the distinct states. + reports := map[string]InstanceReport{ + "i-res": exactReport("i-res", "opResolved", reopened.now), + } + snap, err := reopened.collector.Collect(inventory, reports, nil, 1001) + if err != nil { + t.Fatal(err) + } + assertStates(t, snap) +} + +// TestCollector_ResolvedWhollyVanishedReopensOfflineNotPurged is the fail-closed +// regression test for reconciliation rule 6: a resolved_current operator whose +// every instance disappears entirely from the authoritative inventory (and whose +// sightings are also absent) must REOPEN as offline_unknown, not silently stay +// resolved and later be purged. Removal from inventory never resolves central +// state, so its unresolved history is then retained indefinitely. +func TestCollector_ResolvedWhollyVanishedReopensOfflineNotPurged(t *testing.T) { + tc := newTestCollector(t) + resolvedInv := []InventoryInstance{eligibleInstance("ir", "opR")} + + // Resolve opR across three exact collections while it is present. + for cycle := 0; cycle < 3; cycle++ { + r := map[string]InstanceReport{"ir": exactReport("ir", "opR", tc.now)} + if _, err := tc.collector.Collect(resolvedInv, r, nil, 1000); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if status, ok := operatorStatus(tc.collector.Snapshot(), "opR"); !ok || status != FleetResolvedCurrent { + t.Fatalf("precondition: opR must be resolved before it vanishes, got %s (present=%t)", status, ok) + } + + // opR wholly vanishes from the authoritative inventory and current sightings. + tc.now = tc.now.Add(time.Minute) + snap, err := tc.collector.Collect(nil, nil, nil, 1005) + if err != nil { + t.Fatal(err) + } + status, ok := operatorStatus(snap, "opR") + if !ok { + t.Fatal("a wholly-vanished resolved operator must be retained, not dropped") + } + if status == FleetResolvedCurrent { + t.Fatalf("a wholly-vanished resolved operator must reopen, not stay resolved_current") + } + if status != FleetOfflineUnknown { + t.Fatalf("expected the vanished operator to reopen as offline_unknown, got %s", status) + } + if snap.Complete { + t.Error("readiness must not be complete while a reopened operator blocks") + } + + // Advance far past the resolved-retention window and reconcile again with the + // operator still absent: it must NOT be purged, because it is now blocking + // (unresolved) and unresolved history is retained indefinitely. + tc.now = tc.now.Add(ResolvedRetention + time.Hour) + snap, err = tc.collector.Collect(nil, nil, nil, 2000) + if err != nil { + t.Fatal(err) + } + if status, ok := operatorStatus(snap, "opR"); !ok || status != FleetOfflineUnknown { + t.Errorf( + "a reopened (blocking) operator must be retained indefinitely, not purged; got %s (present=%t)", + status, ok, + ) + } +} + +// TestCollector_ReportsAcceptedImmediatelyAfterRestart is the regression for the +// restart-durable reporter revision (net-new finding 1). The collector persists +// the highest accepted reporter revision as a high-water mark and rejects +// anything below it. If the reporter revision were a process-local counter that +// reset on restart, every post-restart report would sit below the persisted +// high-water mark and be rejected for as many cycles as the previous process had +// run, silently reopening a resolved operator. Because the reporter revision is +// derived from the (monotonic wall-clock) attestation timestamp, a report taken +// after a restart still exceeds the persisted high-water mark and is accepted +// immediately, keeping the operator resolved. +func TestCollector_ReportsAcceptedImmediatelyAfterRestart(t *testing.T) { + path := filepath.Join(t.TempDir(), "roster.db") + tc := newTestCollectorAtPath(t, path) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Run many pre-restart cycles so the persisted reporter-revision high-water + // mark is large (mirroring a long-running previous process). + for cycle := 0; cycle < 20; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + if _, err := tc.collector.Collect(inv, reports, nil, 1000); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if status, _ := operatorStatus(tc.collector.Snapshot(), "op1"); status != FleetResolvedCurrent { + t.Fatalf("precondition: op1 must be resolved before restart, got %s", status) + } + + // Restart the collector against the same bbolt file (a fresh process resets + // any in-memory reporter-revision counter). + if err := tc.store.Close(); err != nil { + t.Fatal(err) + } + reopened := newTestCollectorAtPath(t, path) + reopened.now = tc.now.Add(time.Minute) + + // A single exact report immediately after restart must be ACCEPTED (its + // timestamp-derived reporter revision exceeds the persisted high-water mark), + // keeping the operator resolved. A rejected report would drop the operator's + // exact-confirmation streak and reopen it as offline_unknown. + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", reopened.now)} + snap, err := reopened.collector.Collect(inv, reports, nil, 1001) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status != FleetResolvedCurrent { + t.Fatalf( + "a report taken after restart must be accepted immediately, keeping op1 "+ + "resolved; got %s (a reset reporter revision would reject it)", status, + ) + } +} + +func TestCollector_BlockingNeverPurged(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Never reports -> offline_unknown (blocking). + for cycle := 0; cycle < 2; cycle++ { + if _, err := tc.collector.Collect(inventory, map[string]InstanceReport{}, nil, 1000); err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + // Advance far past the retention window; blocking history is never purged. + tc.now = tc.now.Add(ResolvedRetention * 3) + snap, err := tc.collector.Collect(inventory, map[string]InstanceReport{}, nil, 5000) + if err != nil { + t.Fatal(err) + } + if status, ok := operatorStatus(snap, "op1"); !ok || status != FleetOfflineUnknown { + t.Fatalf("blocking operator must be retained indefinitely; got ok=%v status=%s", ok, status) + } +} + +func TestCollector_ReadinessAPIDeterministicAndDenies(t *testing.T) { + tc := newTestCollector(t) + inventory := []InventoryInstance{ + eligibleInstance("i2", "opB"), + eligibleInstance("i1", "opA"), + } + if _, err := tc.collector.Collect(inventory, map[string]InstanceReport{}, nil, 1000); err != nil { + t.Fatal(err) + } + + handler := NewHandler(tc.collector, nil) + + // GET returns JSON with sorted, deterministic content and no TrustedReportTarget. + req := httptest.NewRequest(http.MethodGet, readinessPath, nil) + rec := httptest.NewRecorder() + handler.ServeHTTP(rec, req) + if rec.Code != http.StatusOK { + t.Fatalf("expected 200, got %d", rec.Code) + } + var snap FleetSnapshot + if err := json.Unmarshal(rec.Body.Bytes(), &snap); err != nil { + t.Fatalf("cannot decode readiness response: %v", err) + } + if len(snap.Blocking) != 2 { + t.Fatalf("expected 2 blocking operators, got %d", len(snap.Blocking)) + } + // Blocking operators are sorted deterministically by canonical operator + // address, and both expected operators are present. + if snap.Blocking[0].OperatorAddress >= snap.Blocking[1].OperatorAddress { + t.Errorf("blocking operators are not sorted deterministically: %+v", snap.Blocking) + } + present := map[string]bool{ + snap.Blocking[0].OperatorAddress: true, + snap.Blocking[1].OperatorAddress: true, + } + if !present[opAddr("opA")] || !present[opAddr("opB")] { + t.Errorf("expected both opA and opB in the blocking set, got %+v", snap.Blocking) + } + if bytes.Contains(rec.Body.Bytes(), []byte("reports.example")) { + t.Errorf("TrustedReportTarget must never be exposed in the API") + } + + // Non-GET is denied. + postReq := httptest.NewRequest(http.MethodPost, readinessPath, nil) + postRec := httptest.NewRecorder() + handler.ServeHTTP(postRec, postReq) + if postRec.Code != http.StatusMethodNotAllowed { + t.Errorf("expected 405 for POST, got %d", postRec.Code) + } + + // Unknown paths are denied. + unknownReq := httptest.NewRequest(http.MethodGet, "/secret", nil) + unknownRec := httptest.NewRecorder() + handler.ServeHTTP(unknownRec, unknownReq) + if unknownRec.Code != http.StatusNotFound { + t.Errorf("expected 404 for unknown path, got %d", unknownRec.Code) + } +} + +func TestServer_BindsToConfiguredAddress(t *testing.T) { + tc := newTestCollector(t) + if _, err := tc.collector.Collect(nil, nil, nil, 1000); err != nil { + t.Fatal(err) + } + + server, err := NewServer("127.0.0.1:0", tc.collector, nil, nil) + if err != nil { + t.Fatalf("cannot start server: %v", err) + } + go func() { _ = server.Serve() }() + t.Cleanup(func() { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + _ = server.Close(ctx) + }) + + if got := server.Addr(); got == "" { + t.Fatal("expected a bound address") + } + + resp, err := http.Get("http://" + server.Addr() + readinessPath) + if err != nil { + t.Fatalf("cannot reach readiness endpoint: %v", err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Errorf("expected 200 from bound server, got %d", resp.StatusCode) + } +} diff --git a/pkg/monitoring/cutoverroster/collector_trustchain_test.go b/pkg/monitoring/cutoverroster/collector_trustchain_test.go new file mode 100644 index 0000000000..9e8dd19274 --- /dev/null +++ b/pkg/monitoring/cutoverroster/collector_trustchain_test.go @@ -0,0 +1,273 @@ +package cutoverroster + +import ( + "path/filepath" + "testing" + "time" +) + +// newBareCollector builds a collector with no identity verifier installed and no +// service-discovery feed marked, so a test can exercise exactly which parts of +// the mandatory trust chain gate completeness. +func newBareCollector(t *testing.T, cfg CollectorConfig) *testCollector { + t.Helper() + store, err := OpenStore(filepath.Join(t.TempDir(), "roster.db")) + if err != nil { + t.Fatalf("cannot open store: %v", err) + } + tc := &testCollector{store: store, sink: newFakeSink(), now: fleetBaseTime} + collector, err := newCollectorWithClock( + cfg, store, tc.sink, func() time.Time { return tc.now }, + ) + if err != nil { + t.Fatalf("cannot construct collector: %v", err) + } + collector.SetQuarantineVerifier(testQuarantineVerifier()) + tc.collector = collector + t.Cleanup(func() { _ = store.Close() }) + return tc +} + +// TestCollector_IdentityVerifierRequiredForComplete proves an installed on-chain +// identity verifier is mandatory for completeness: without it, an otherwise fully +// resolved fleet is not complete (a missing WalletRegistry verification blocks +// readiness rather than certifying inventory identity assertions on their own). +func TestCollector_IdentityVerifierRequiredForComplete(t *testing.T) { + cfg := testConfig() // RequireIdentityVerification + RequireServiceDiscovery on + tc := newBareCollector(t, cfg) + tc.collector.SetServiceDiscoveryConfigured(true) // isolate the identity requirement + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + // Three exact reports with NO identity verifier installed: the operator + // resolves but readiness must not be complete. + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inv, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if snap.Complete { + t.Fatal("readiness must not be complete without an installed identity verifier") + } + + // Installing the verifier (which confirms the derived staking provider) makes + // the same fleet complete. + tc.collector.SetIdentityVerifier(derivedIdentityVerifier{}) + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inv, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + if !snap.Complete { + t.Fatal("readiness must be complete once identity verification is configured") + } +} + +// TestCollector_ServiceDiscoveryRequiredForComplete proves a wired +// service-discovery feed is mandatory for completeness: without it, an otherwise +// fully resolved fleet is not complete (a missing discovery feed blocks readiness +// rather than degrading to an inventory-only view). +func TestCollector_ServiceDiscoveryRequiredForComplete(t *testing.T) { + cfg := testConfig() + tc := newBareCollector(t, cfg) + tc.collector.SetIdentityVerifier(derivedIdentityVerifier{}) // isolate the discovery requirement + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inv, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if snap.Complete { + t.Fatal("readiness must not be complete without a wired service-discovery feed") + } + + tc.collector.SetServiceDiscoveryConfigured(true) + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + snap, err := tc.collector.Collect(inv, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + if !snap.Complete { + t.Fatal("readiness must be complete once service discovery is configured") + } +} + +// TestCollector_NonCanonicalOrZeroStakingProviderRejected proves the staking +// provider must be a canonical, non-zero address: a symbolic or zero-address +// staking provider is an inventory-reconciliation fault that blocks readiness. +func TestCollector_NonCanonicalOrZeroStakingProviderRejected(t *testing.T) { + for _, bad := range []string{ + "sp-op1", // symbolic, not an address + "0x1234", // too short + zeroAddress, + } { + t.Run(bad, func(t *testing.T) { + tc := newTestCollector(t) + inv := eligibleInstance("i1", "op1") + inv.StakingProvider = bad + snap, err := tc.collector.Collect([]InventoryInstance{inv}, nil, nil, 1000) + if err != nil { + t.Fatal(err) + } + if snap.Inventory.EligibleInstances != 0 { + t.Errorf("staking provider %q must not count as reconciled eligible", bad) + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("staking provider %q must count as unreconciled", bad) + } + if snap.Complete { + t.Errorf("staking provider %q must not yield completeness", bad) + } + }) + } +} + +// TestCollector_MalformedImageDigestRejected proves the expected image digest must +// be a full sha256:<64 hex> content digest: an abbreviated digest cannot pin the +// exact runtime image and is an inventory-reconciliation fault. +func TestCollector_MalformedImageDigestRejected(t *testing.T) { + for _, bad := range []string{ + "sha256:deadbeefcafe", // abbreviated + "latest", // mutable tag + "sha256:" + shortHex(63), // one hex short + "sha256:" + shortHex(64) + "0", // one hex too long + "md5:" + shortHex(64), // wrong algorithm + } { + t.Run(bad, func(t *testing.T) { + tc := newTestCollector(t) + inv := eligibleInstance("i1", "op1") + inv.ExpectedImageDigest = bad + snap, err := tc.collector.Collect([]InventoryInstance{inv}, nil, nil, 1000) + if err != nil { + t.Fatal(err) + } + if snap.Inventory.EligibleInstances != 0 { + t.Errorf("digest %q must not count as reconciled eligible", bad) + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("digest %q must count as unreconciled", bad) + } + }) + } +} + +func shortHex(n int) string { + const hexDigits = "0123456789abcdef" + b := make([]byte, n) + for i := range b { + b[i] = hexDigits[i%len(hexDigits)] + } + return string(b) +} + +// TestCollector_ContradictoryStakingProvidersAcrossInstances proves that when two +// eligible instances of one operator assert different (canonical) staking +// providers, the contradiction is a fail-closed inventory fault: the last claim +// does not silently win, the operator cannot resolve, and readiness fails closed. +func TestCollector_ContradictoryStakingProvidersAcrossInstances(t *testing.T) { + tc := newTestCollector(t) + + i1 := eligibleInstance("i1", "op1") + i2 := eligibleInstance("i2", "op1") + // Two distinct, individually-canonical staking providers for the same operator. + i1.StakingProvider = "0x1111111111111111111111111111111111111111" + i2.StakingProvider = "0x2222222222222222222222222222222222222222" + inv := []InventoryInstance{i1, i2} + + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + reports := map[string]InstanceReport{ + "i1": exactReport("i1", "op1", tc.now), + "i2": exactReport("i2", "op1", tc.now), + } + var err error + snap, err = tc.collector.Collect(inv, reports, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + + if status, _ := operatorStatus(snap, "op1"); status == FleetResolvedCurrent { + t.Fatalf("a cross-instance staking-provider contradiction must not resolve, got %s", status) + } + if snap.Inventory.Unreconciled == 0 { + t.Error("a cross-instance staking-provider contradiction must count as unreconciled") + } + if snap.Complete { + t.Error("a cross-instance staking-provider contradiction must not yield completeness") + } +} + +// TestCollector_PurgeResolvedAfter30Days restores independent coverage of the +// 30-day resolved-purge mechanism (collector.go purgeResolved). It is a +// white-box test of the mechanism rather than an end-to-end reconciliation +// scenario, because the two semantics genuinely conflict: the fail-closed +// reopening of a vanished resolved operator (reconciliation rule 6, covered by +// TestCollector_ResolvedWhollyVanishedReopensOfflineNotPurged) deliberately keeps +// a departed operator retained-and-blocking rather than resolved, so under normal +// reconciliation an actively resolved operator is continuously re-confirmed and a +// departed one reopens — neither ages out. purgeResolved therefore remains a +// bounded-store backstop for a resolved record that is no longer being +// re-confirmed, and this test pins that backstop: a resolved record older than +// the retention window is purged (with its instances), while a fresh resolved +// record and any blocking record are retained. +func TestCollector_PurgeResolvedAfter30Days(t *testing.T) { + tc := newTestCollector(t) + c := tc.collector + now := fleetBaseTime + + const ( + staleResolved = "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + freshResolved = "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + staleBlocking = "0xcccccccccccccccccccccccccccccccccccccccc" + ) + + c.mu.Lock() + c.operators[staleResolved] = &operatorRecord{ + OperatorAddress: staleResolved, + Status: FleetResolvedCurrent, + ResolvedAt: now.Add(-(ResolvedRetention + time.Hour)), + } + c.instances["i-stale"] = &instanceRecord{ + InstanceID: "i-stale", OperatorAddress: staleResolved, + } + c.operators[freshResolved] = &operatorRecord{ + OperatorAddress: freshResolved, + Status: FleetResolvedCurrent, + ResolvedAt: now, + } + // A blocking record with an ancient timestamp must never be purged: unresolved + // history is retained indefinitely. + c.operators[staleBlocking] = &operatorRecord{ + OperatorAddress: staleBlocking, + Status: FleetOfflineUnknown, + ResolvedAt: now.Add(-(ResolvedRetention * 3)), + } + + c.purgeResolved(now) + + if _, ok := c.operators[staleResolved]; ok { + t.Error("a resolved record older than the retention window must be purged") + } + if _, ok := c.instances["i-stale"]; ok { + t.Error("a purged operator's instance records must be dropped too") + } + if _, ok := c.operators[freshResolved]; !ok { + t.Error("a freshly resolved record must be retained") + } + if _, ok := c.operators[staleBlocking]; !ok { + t.Error("a blocking record must be retained indefinitely, never purged") + } + c.mu.Unlock() +} diff --git a/pkg/monitoring/cutoverroster/collector_validation_test.go b/pkg/monitoring/cutoverroster/collector_validation_test.go new file mode 100644 index 0000000000..270a108bad --- /dev/null +++ b/pkg/monitoring/cutoverroster/collector_validation_test.go @@ -0,0 +1,218 @@ +package cutoverroster + +import ( + "context" + "testing" + "time" +) + +// TestCollector_NonCanonicalOperatorAddressRejected proves an eligible inventory +// entry whose operator address is not a canonical 0x + 40 hex address is rejected +// as an inventory-reconciliation fault: it is not counted as reconciled eligible +// and forces readiness closed. +func TestCollector_NonCanonicalOperatorAddressRejected(t *testing.T) { + tc := newTestCollector(t) + + for _, bad := range []string{ + "op1", // symbolic, not an address + "0x1234", // too short + "deadbeef", // missing 0x + "0xZZZZef0000000000000000000000000000000001", // non-hex + } { + inv := eligibleInstance("i1", "unused") + inv.OperatorAddress = bad + snap, err := tc.collector.Collect([]InventoryInstance{inv}, nil, nil, 1000) + if err != nil { + t.Fatal(err) + } + if snap.Inventory.EligibleInstances != 0 { + t.Errorf("address %q must not count as reconciled eligible", bad) + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("address %q must count as unreconciled", bad) + } + if snap.Complete { + t.Errorf("a non-canonical operator address must not yield completeness (%q)", bad) + } + } +} + +// TestCollector_BlankRequiredInventoryFieldsRejected proves a blank required +// inventory field (staking provider, expected revision/epoch/digest) is an +// inventory-reconciliation fault: the entry cannot prove what "current" is for the +// instance, so readiness fails closed. +func TestCollector_BlankRequiredInventoryFieldsRejected(t *testing.T) { + tc := newTestCollector(t) + + for _, mut := range []struct { + name string + mutate func(*InventoryInstance) + }{ + {"blank staking provider", func(i *InventoryInstance) { i.StakingProvider = "" }}, + {"blank expected revision", func(i *InventoryInstance) { i.ExpectedRevision = "" }}, + {"blank expected epoch", func(i *InventoryInstance) { i.ExpectedEpoch = "" }}, + {"blank expected digest", func(i *InventoryInstance) { i.ExpectedImageDigest = "" }}, + } { + t.Run(mut.name, func(t *testing.T) { + inv := eligibleInstance("i1", "op1") + mut.mutate(&inv) + snap, err := tc.collector.Collect([]InventoryInstance{inv}, nil, nil, 1000) + if err != nil { + t.Fatal(err) + } + if snap.Inventory.EligibleInstances != 0 { + t.Errorf("%s must not count as reconciled eligible", mut.name) + } + if snap.Inventory.Unreconciled == 0 { + t.Errorf("%s must count as unreconciled", mut.name) + } + }) + } +} + +// TestCollector_ZeroAndFutureSightingTimestampRejected proves a post-cutover +// sighting whose observation timestamp is zero or in the future is rejected +// outright — it does not create observed_legacy evidence — rather than being +// admitted while leaving LastLegacyAt unset (which would weaken the post-sighting +// resolution proof). +func TestCollector_ZeroAndFutureSightingTimestampRejected(t *testing.T) { + for _, tt := range []struct { + name string + observedAt time.Time + }{ + {"zero timestamp", time.Time{}}, + {"future timestamp", fleetBaseTime.Add(time.Hour)}, + } { + t.Run(tt.name, func(t *testing.T) { + tc := newTestCollector(t) + inv := []InventoryInstance{eligibleInstance("i1", "op1")} + + // A valid post-cutover block (>= cutover 1000, <= current 1100) but an + // invalid timestamp. + sightings := []LegacySighting{ + {OperatorAddress: opAddr("op1"), Block: 1050, ObservedAt: tt.observedAt}, + } + snap, err := tc.collector.Collect(inv, nil, sightings, 1100) + if err != nil { + t.Fatal(err) + } + if status, _ := operatorStatus(snap, "op1"); status == FleetObservedLegacy { + t.Errorf("%s sighting must not create observed_legacy", tt.name) + } + if tc.sink.gauge(MetricFleetObservedLegacy) != 0 { + t.Errorf("%s sighting must not increment the observed-legacy gauge", tt.name) + } + }) + } +} + +// fakeIdentityVerifier maps operator addresses to their (test-asserted) on-chain +// staking provider. A missing operator returns an error. +type fakeIdentityVerifier struct { + providers map[string]string +} + +func (f *fakeIdentityVerifier) OperatorStakingProviderAtBlock( + _ context.Context, operatorAddress string, _ uint64, +) (string, error) { + if p, ok := f.providers[normalizeAddress(operatorAddress)]; ok { + return p, nil + } + return "", errNoIdentity +} + +var errNoIdentity = &identityError{} + +type identityError struct{} + +func (*identityError) Error() string { return "no on-chain identity for operator" } + +// TestCollector_IdentityVerificationGatesResolution proves that with an on-chain +// identity verifier configured, an operator whose inventory staking-provider claim +// matches the WalletRegistry mapping can resolve, while a mismatch (or a lookup +// failure) blocks resolution and raises the unreconciled signal (fail closed). +func TestCollector_IdentityVerificationGatesResolution(t *testing.T) { + provider := "0x1111111111111111111111111111111111111111" + otherProvider := "0x2222222222222222222222222222222222222222" + + newInv := func(claim string) []InventoryInstance { + inv := eligibleInstance("i1", "op1") + inv.StakingProvider = claim + return []InventoryInstance{inv} + } + + t.Run("matching identity resolves", func(t *testing.T) { + tc := newTestCollector(t) + tc.collector.SetIdentityVerifier(&fakeIdentityVerifier{ + providers: map[string]string{opAddr("op1"): provider}, + }) + inv := newInv(provider) + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + r := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inv, r, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if status, _ := operatorStatus(snap, "op1"); status != FleetResolvedCurrent { + t.Fatalf("matching on-chain identity must allow resolution, got %s", status) + } + }) + + t.Run("mismatching identity blocks resolution", func(t *testing.T) { + tc := newTestCollector(t) + tc.collector.SetIdentityVerifier(&fakeIdentityVerifier{ + providers: map[string]string{opAddr("op1"): otherProvider}, + }) + inv := newInv(provider) // inventory claims `provider`, chain says `otherProvider` + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + r := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var err error + snap, err = tc.collector.Collect(inv, r, nil, 1000) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if status, _ := operatorStatus(snap, "op1"); status == FleetResolvedCurrent { + t.Fatalf("an on-chain identity mismatch must block resolution, got %s", status) + } + if tc.sink.gauge(MetricInventoryUnreconciled) == 0 { + t.Errorf("an identity mismatch must raise the unreconciled gauge") + } + }) +} + +// TestCollector_DisappearedFromDiscoveryOffline proves that an eligible instance +// flagged as absent from the production service-discovery target set is +// offline_unknown for the cycle even if it produced an otherwise-exact report: +// disappearance from discovery is never ready. +func TestCollector_DisappearedFromDiscoveryOffline(t *testing.T) { + tc := newTestCollector(t) + + inv := eligibleInstance("i1", "op1") + inv.DisappearedFromDiscovery = true + + // The instance still supplies an exact report, but it has vanished from + // discovery, so it must be treated as offline. + reports := map[string]InstanceReport{"i1": exactReport("i1", "op1", tc.now)} + var snap FleetSnapshot + for cycle := 0; cycle < 3; cycle++ { + var err error + snap, err = tc.collector.Collect([]InventoryInstance{inv}, reports, nil, 1100) + if err != nil { + t.Fatal(err) + } + tc.now = tc.now.Add(time.Minute) + } + if status, _ := operatorStatus(snap, "op1"); status != FleetOfflineUnknown { + t.Fatalf("an instance absent from service discovery must be offline_unknown, got %s", status) + } + if snap.Complete { + t.Error("readiness must not be complete while an instance has disappeared from discovery") + } +} diff --git a/pkg/monitoring/cutoverroster/identity.go b/pkg/monitoring/cutoverroster/identity.go new file mode 100644 index 0000000000..6d979608df --- /dev/null +++ b/pkg/monitoring/cutoverroster/identity.go @@ -0,0 +1,165 @@ +package cutoverroster + +import ( + "bytes" + "context" + "encoding/hex" + "encoding/json" + "fmt" + "net/http" + "strings" + "time" + + "github.com/ethereum/go-ethereum/crypto" +) + +// operatorToStakingProviderSelector is the 4-byte function selector for the +// WalletRegistry view function operatorToStakingProvider(address). It is computed +// from the canonical signature so it stays correct without a hardcoded literal. +var operatorToStakingProviderSelector = crypto.Keccak256( + []byte("operatorToStakingProvider(address)"), +)[:4] + +// EthCallIdentityVerifier verifies operator→staking-provider identity against the +// on-chain WalletRegistry contract using a read-only eth_call. It implements +// IdentityVerifier. It intentionally uses a raw eth_call rather than the full +// generated binding: the mapping is a plain view function, so no account key, +// nonce manager, or mining infrastructure is required for a read-only monitor. +type EthCallIdentityVerifier struct { + rpcURL string + contractAddress string + client *http.Client +} + +// NewEthCallIdentityVerifier constructs a verifier for the WalletRegistry at +// contractAddress, reached over the given Ethereum JSON-RPC URL. Both must be +// non-empty and the contract address canonical. +func NewEthCallIdentityVerifier( + rpcURL, contractAddress string, + client *http.Client, +) (*EthCallIdentityVerifier, error) { + if strings.TrimSpace(rpcURL) == "" { + return nil, fmt.Errorf("ethereum RPC URL is required") + } + contractAddress = normalizeAddress(contractAddress) + if !isCanonicalAddress(contractAddress) { + return nil, fmt.Errorf("wallet registry address must be a canonical 0x address") + } + if client == nil { + client = &http.Client{Timeout: 10 * time.Second} + } + return &EthCallIdentityVerifier{ + rpcURL: rpcURL, + contractAddress: contractAddress, + client: client, + }, nil +} + +// OperatorStakingProviderAtBlock reads WalletRegistry.operatorToStakingProvider +// for operatorAddress at the given block (0 = latest) and returns the canonical +// staking-provider address. A zero address means the operator is not registered. +// The RPC honors ctx, so a canceled collection/shutdown context aborts the call +// promptly rather than blocking for the full fixed timeout. +func (v *EthCallIdentityVerifier) OperatorStakingProviderAtBlock( + ctx context.Context, + operatorAddress string, + block uint64, +) (string, error) { + operatorAddress = normalizeAddress(operatorAddress) + if !isCanonicalAddress(operatorAddress) { + return "", fmt.Errorf("operator address is not canonical") + } + + // calldata = selector ++ left-padded 32-byte operator address. + callData := make([]byte, 0, 4+32) + callData = append(callData, operatorToStakingProviderSelector...) + addrBytes, err := hex.DecodeString(operatorAddress[2:]) + if err != nil { + return "", fmt.Errorf("cannot decode operator address") + } + padded := make([]byte, 32) + copy(padded[32-len(addrBytes):], addrBytes) + callData = append(callData, padded...) + + blockTag := "latest" + if block > 0 { + blockTag = fmt.Sprintf("0x%x", block) + } + + result, err := v.ethCall(ctx, "0x"+hex.EncodeToString(callData), blockTag) + if err != nil { + return "", err + } + return decodeAddressResult(result) +} + +// ethCall performs a single eth_call and returns the hex "result" string. The +// per-call timeout is bounded to 10s but derives from ctx, so cancellation of +// the passed-in context takes effect immediately. +func (v *EthCallIdentityVerifier) ethCall(ctx context.Context, data, blockTag string) (string, error) { + payload := map[string]interface{}{ + "jsonrpc": "2.0", + "id": 1, + "method": "eth_call", + "params": []interface{}{ + map[string]string{"to": v.contractAddress, "data": data}, + blockTag, + }, + } + body, err := json.Marshal(payload) + if err != nil { + return "", err + } + + ctx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + + // #nosec G107 -- the RPC URL is operator-supplied monitoring configuration. + req, err := http.NewRequestWithContext(ctx, http.MethodPost, v.rpcURL, bytes.NewReader(body)) + if err != nil { + return "", err + } + req.Header.Set("Content-Type", "application/json") + + resp, err := v.client.Do(req) + if err != nil { + // Sanitize: the raw transport error embeds the RPC URL (host), which must + // not appear in logs. + return "", fmt.Errorf("ethereum RPC request failed") + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("ethereum RPC returned status %d", resp.StatusCode) + } + + var rpcResp struct { + Result string `json:"result"` + Error *struct { + Message string `json:"message"` + } `json:"error"` + } + if err := json.NewDecoder(resp.Body).Decode(&rpcResp); err != nil { + return "", err + } + if rpcResp.Error != nil { + return "", fmt.Errorf("ethereum RPC error: %s", rpcResp.Error.Message) + } + return rpcResp.Result, nil +} + +// decodeAddressResult decodes a 32-byte ABI-encoded address return value (the +// low 20 bytes) into a canonical 0x address. +func decodeAddressResult(result string) (string, error) { + result = strings.TrimPrefix(strings.TrimSpace(result), "0x") + if len(result) < 64 { + return "", fmt.Errorf("unexpected eth_call result length") + } + // The ABI encodes an address right-aligned in a 32-byte word: the address is + // the last 40 hex characters of the first 64-hex-character word. + word := result[:64] + addr := "0x" + strings.ToLower(word[24:]) + if !isCanonicalAddress(addr) { + return "", fmt.Errorf("eth_call did not return a valid address") + } + return addr, nil +} diff --git a/pkg/monitoring/cutoverroster/metrics.go b/pkg/monitoring/cutoverroster/metrics.go new file mode 100644 index 0000000000..7c1cecf1c8 --- /dev/null +++ b/pkg/monitoring/cutoverroster/metrics.go @@ -0,0 +1,81 @@ +package cutoverroster + +import ( + "github.com/prometheus/client_golang/prometheus" +) + +// operatorLabels are the label names on the per-operator gauges. +var operatorLabels = []string{"operator_address", "staking_provider", "status"} + +// PrometheusMetrics is a Prometheus-backed MetricsSink for the fleet collector. +type PrometheusMetrics struct { + registry *prometheus.Registry + + fleetGauges map[string]prometheus.Gauge + operatorGauges map[string]*prometheus.GaugeVec +} + +// NewPrometheusMetrics constructs and registers all fleet and operator metrics +// in a dedicated registry. +func NewPrometheusMetrics() *PrometheusMetrics { + registry := prometheus.NewRegistry() + + fleetGauges := map[string]prometheus.Gauge{} + for name, help := range map[string]string{ + MetricFleetBlockingOperators: "Distinct nonquarantined operators in any blocking status", + MetricFleetObservedLegacy: "Operators with retained post-cutover legacy wire evidence", + MetricReportersStale: "Eligible instances without a fresh accepted report", + MetricInventoryUnreconciled: "Identity/target/inventory reconciliation failures", + } { + gauge := prometheus.NewGauge(prometheus.GaugeOpts{Name: name, Help: help}) + registry.MustRegister(gauge) + fleetGauges[name] = gauge + } + + operatorGauges := map[string]*prometheus.GaugeVec{} + for name, help := range map[string]string{ + MetricOperatorInfo: "Bounded central-inventory operator status", + MetricOperatorFirstSeenBlock: "First relevant evidence block for the operator", + MetricOperatorLastSeenBlock: "Last wire/report evidence block for the operator", + } { + vec := prometheus.NewGaugeVec(prometheus.GaugeOpts{Name: name, Help: help}, operatorLabels) + registry.MustRegister(vec) + operatorGauges[name] = vec + } + + return &PrometheusMetrics{ + registry: registry, + fleetGauges: fleetGauges, + operatorGauges: operatorGauges, + } +} + +// Registry returns the underlying Prometheus registry for exposition. +func (m *PrometheusMetrics) Registry() *prometheus.Registry { + return m.registry +} + +// SetGauge implements MetricsSink for the label-less fleet gauges. +func (m *PrometheusMetrics) SetGauge(name string, value float64) { + if gauge, ok := m.fleetGauges[name]; ok { + gauge.Set(value) + } +} + +// SetOperatorGauge implements MetricsSink for the per-operator labeled gauges. +func (m *PrometheusMetrics) SetOperatorGauge( + name, operatorAddress, stakingProvider, status string, + value float64, +) { + if vec, ok := m.operatorGauges[name]; ok { + vec.WithLabelValues(operatorAddress, stakingProvider, status).Set(value) + } +} + +// ResetOperatorGauges clears every per-operator labeled series so stale label +// sets do not linger between cycles. +func (m *PrometheusMetrics) ResetOperatorGauges() { + for _, vec := range m.operatorGauges { + vec.Reset() + } +} diff --git a/pkg/monitoring/cutoverroster/production_integration_test.go b/pkg/monitoring/cutoverroster/production_integration_test.go new file mode 100644 index 0000000000..fcfba87314 --- /dev/null +++ b/pkg/monitoring/cutoverroster/production_integration_test.go @@ -0,0 +1,540 @@ +package cutoverroster + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +// TestParseServiceDiscovery proves the Prometheus file_sd target file (keep-sd.json) +// is parsed into an operator→scrape-URL map keyed by the __meta_chain_address +// label, skipping rows without a canonical chain address. +func TestParseServiceDiscovery(t *testing.T) { + op := "0xabcdef0000000000000000000000000000000001" + raw := fmt.Sprintf(`[ + {"targets": ["10.0.0.5:9601"], "labels": {"__meta_chain_address": "%s", "__meta_network_id": "1"}}, + {"targets": ["10.0.0.6:9601"], "labels": {"__meta_chain_address": "not-an-address"}}, + {"targets": [], "labels": {"__meta_chain_address": "0x1111111111111111111111111111111111111111"}} + ]`, strings.ToUpper(op)) + + sd, err := ParseServiceDiscovery([]byte(raw)) + if err != nil { + t.Fatalf("parse: %v", err) + } + if sd.Len() != 1 { + t.Fatalf("expected 1 usable discovery entry, got %d", sd.Len()) + } + if !sd.Has(op) { + t.Errorf("expected operator %s present (case-insensitive)", op) + } + // Discovery is keyed by network ID (the per-instance disambiguator); the + // usable entry carries network_id "1". + if got := sd.MetricsURLForInstance(op, "1"); got != "http://10.0.0.5:9601/metrics" { + t.Errorf("metrics URL = %q", got) + } + if got := sd.DiagnosticsURLForInstance(op, "1"); got != "http://10.0.0.5:9601/diagnostics" { + t.Errorf("diagnostics URL = %q", got) + } + // A different operator claiming the same network ID does not match. + if got := sd.MetricsURLForInstance("0x1111111111111111111111111111111111111111", "1"); got != "" { + t.Errorf("network id must not resolve for a mismatched operator: %q", got) + } + // A missing network ID yields no per-instance target even for a present operator. + if got := sd.MetricsURLForInstance(op, ""); got != "" { + t.Errorf("empty network id must not resolve a target: %q", got) + } +} + +// TestServiceDiscovery_MultipleInstancesPerOperatorStayDistinct proves two +// instances of one operator resolve to distinct discovered targets rather than +// collapsing onto a single operator-level URL. +func TestServiceDiscovery_MultipleInstancesPerOperatorStayDistinct(t *testing.T) { + op := "0xabcdef0000000000000000000000000000000001" + raw := fmt.Sprintf(`[ + {"targets": ["10.0.0.5:9601"], "labels": {"__meta_chain_address": "%s", "__meta_network_id": "net-a"}}, + {"targets": ["10.0.0.6:9601"], "labels": {"__meta_chain_address": "%s", "__meta_network_id": "net-b"}} + ]`, op, op) + + sd, err := ParseServiceDiscovery([]byte(raw)) + if err != nil { + t.Fatalf("parse: %v", err) + } + if got := sd.MetricsURLForInstance(op, "net-a"); got != "http://10.0.0.5:9601/metrics" { + t.Errorf("instance net-a URL = %q", got) + } + if got := sd.MetricsURLForInstance(op, "net-b"); got != "http://10.0.0.6:9601/metrics" { + t.Errorf("instance net-b URL = %q", got) + } + if sd.Len() != 1 { + t.Errorf("two instances of one operator are still one operator, got Len %d", sd.Len()) + } +} + +// TestReconcileWithDiscovery proves an eligible instance whose exact +// (operator, networkID) identity is absent from service discovery is flagged +// DisappearedFromDiscovery, while a discovered instance is not — and, critically, +// that a SECOND instance of a discovered operator whose own network ID is not in +// discovery is still flagged. Per-instance keying is what keeps distinct +// same-operator instances from collapsing on a sibling's presence. +func TestReconcileWithDiscovery(t *testing.T) { + present := "0xabcdef0000000000000000000000000000000001" + absent := "0xabcdef0000000000000000000000000000000002" + raw := fmt.Sprintf( + `[{"targets": ["h:9601"], "labels": {"__meta_chain_address": "%s", "__meta_network_id": "net-1"}}]`, + present, + ) + sd, err := ParseServiceDiscovery([]byte(raw)) + if err != nil { + t.Fatal(err) + } + + inventory := []InventoryInstance{ + {InstanceID: "i1", OperatorAddress: present, NetworkID: "net-1", CeremonyEligible: true}, + {InstanceID: "i2", OperatorAddress: absent, NetworkID: "net-2", CeremonyEligible: true}, + // A second instance of the discovered operator whose own network ID is NOT + // in discovery: it must still be flagged, because a sibling instance's + // presence does not cover a distinct network identity. + {InstanceID: "i3", OperatorAddress: present, NetworkID: "net-UNKNOWN", CeremonyEligible: true}, + } + out := ReconcileWithDiscovery(inventory, sd) + if out[0].DisappearedFromDiscovery { + t.Error("discovered instance (operator+networkID) must not be flagged disappeared") + } + if !out[1].DisappearedFromDiscovery { + t.Error("operator absent from discovery must be flagged disappeared") + } + if !out[2].DisappearedFromDiscovery { + t.Error("a same-operator instance whose network ID is not discovered must be flagged disappeared") + } + + // A nil discovery feed leaves the inventory untouched. + untouched := ReconcileWithDiscovery( + []InventoryInstance{{InstanceID: "i1", OperatorAddress: absent, NetworkID: "net-2", CeremonyEligible: true}}, + nil, + ) + if untouched[0].DisappearedFromDiscovery { + t.Error("nil discovery feed must not flag anything") + } +} + +// TestParseClientInfoLabels proves the client_info metric labels are extracted +// from a Prometheus text exposition and that longer names do not falsely match. +func TestParseClientInfoLabels(t *testing.T) { + text := `# HELP client_info Client info +# TYPE client_info gauge +client_info_extra{version="x"} 1 +client_info{version="v2.0.0",revision="abc123",protocol_epoch="security_v2_cutover"} 1 +` + labels := parseClientInfoLabels(text) + if labels == nil { + t.Fatal("expected client_info labels") + } + if labels["version"] != "v2.0.0" || labels["revision"] != "abc123" || + labels["protocol_epoch"] != "security_v2_cutover" { + t.Errorf("unexpected labels: %+v", labels) + } + + if parseClientInfoLabels("performance_signing_operations_total 0\n") != nil { + t.Error("absent client_info must return nil") + } +} + +// TestMetricsReportSource_Fetch proves the adapter builds a report from the +// node's real /metrics and /diagnostics endpoints, taking the revision from +// diagnostics when the client_info metric carries only the version (the current +// build), and folding in the externally-attested digest and epoch. +func TestMetricsReportSource_Fetch(t *testing.T) { + const operator = "0xabcdef0000000000000000000000000000000001" + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/metrics": + // Current build: client_info carries only version. + _, _ = io.WriteString(w, "client_info{version=\"v2.0.0\"} 1\n") + case "/diagnostics": + // The node self-attests its identity (chain address + network id). + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "client_info": map[string]string{ + "version": "v2.0.0", + "revision": "abc123def456", + "chain_address": operator, + "network_id": "net-1", + }, + }) + default: + http.NotFound(w, r) + } + })) + defer srv.Close() + + source := NewMetricsReportSource(srv.Client(), &MapAttestationSource{ + Digests: map[string]string{"i1": "sha256:deadbeef"}, + Epochs: map[string]string{"i1": ExpectedEpochSecurityV2Cutover}, + }) + // Control the clock so the reporter revision (derived from the attestation + // timestamp) is deterministic across the two scrapes. + scrapeAt := fleetBaseTime + source.clock = func() time.Time { return scrapeAt } + + inv := InventoryInstance{ + InstanceID: "i1", + OperatorAddress: operator, + NetworkID: "net-1", + TrustedReportTarget: srv.URL, + } + report, err := source.Fetch(context.Background(), inv) + if err != nil { + t.Fatalf("fetch: %v", err) + } + if report.Revision != "abc123def456" { + t.Errorf("revision from diagnostics = %q", report.Revision) + } + // The operator address comes from the node's own attestation, not inventory. + if report.OperatorAddress != operator { + t.Errorf("operator address from diagnostics = %q", report.OperatorAddress) + } + if report.Epoch != ExpectedEpochSecurityV2Cutover { + t.Errorf("epoch from attestation = %q", report.Epoch) + } + if report.ImageDigest != "sha256:deadbeef" { + t.Errorf("digest from attestation = %q", report.ImageDigest) + } + if report.ReporterRevision == 0 { + t.Error("reporter revision must advance from zero") + } + if report.AttestedAt.IsZero() { + t.Error("attested time must be stamped") + } + + // A later scrape advances the reporter revision (monotonic with the clock). + scrapeAt = scrapeAt.Add(time.Minute) + report2, err := source.Fetch(context.Background(), inv) + if err != nil { + t.Fatal(err) + } + if report2.ReporterRevision <= report.ReporterRevision { + t.Errorf("reporter revision must be monotonic: %d then %d", report.ReporterRevision, report2.ReporterRevision) + } +} + +// TestMetricsReportSource_ReporterRevisionSurvivesRestart proves the reporter +// revision is durable across a collector/reporter restart: a fresh +// MetricsReportSource (a "restart", losing any in-process counter) still produces +// a revision strictly greater than the one persisted before the restart, so the +// collector's high-water-mark guard keeps accepting reports immediately rather +// than rejecting them for as many cycles as the previous process had run. +func TestMetricsReportSource_ReporterRevisionSurvivesRestart(t *testing.T) { + const operator = "0xabcdef0000000000000000000000000000000001" + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/metrics": + _, _ = io.WriteString(w, "client_info{version=\"v2.0.0\"} 1\n") + case "/diagnostics": + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "client_info": map[string]string{ + "revision": "abc123def456", "chain_address": operator, "network_id": "net-1", + }, + }) + default: + http.NotFound(w, r) + } + })) + defer srv.Close() + + inv := InventoryInstance{ + InstanceID: "i1", OperatorAddress: operator, NetworkID: "net-1", TrustedReportTarget: srv.URL, + } + + // The pre-restart source runs many cycles, driving any process-local counter + // high. Its persisted high-water mark is the last revision it produced. + before := NewMetricsReportSource(srv.Client(), nil) + at := fleetBaseTime + before.clock = func() time.Time { return at } + var highWater uint64 + for i := 0; i < 500; i++ { + at = at.Add(time.Second) + r, err := before.Fetch(context.Background(), inv) + if err != nil { + t.Fatal(err) + } + highWater = r.ReporterRevision + } + + // The restarted source has no memory of the counter. Its first report at a + // later wall-clock time must still exceed the persisted high-water mark. + after := NewMetricsReportSource(srv.Client(), nil) + restartAt := at.Add(time.Second) + after.clock = func() time.Time { return restartAt } + r, err := after.Fetch(context.Background(), inv) + if err != nil { + t.Fatal(err) + } + if r.ReporterRevision <= highWater { + t.Errorf( + "reporter revision must survive restart: got %d, high-water %d", + r.ReporterRevision, highWater, + ) + } +} + +// TestMetricsReportSource_RejectsIdentityMismatch proves the responding node's +// self-attested identity is validated: a node whose diagnostics chain address (or +// network id) does not match the inventory the target answers for is rejected +// rather than accepted with an inventory-copied identity. +func TestMetricsReportSource_RejectsIdentityMismatch(t *testing.T) { + const inventoryOperator = "0xabcdef0000000000000000000000000000000001" + newSource := func(chainAddr, networkID string) (*MetricsReportSource, InventoryInstance) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/metrics": + _, _ = io.WriteString(w, "client_info{version=\"v2.0.0\"} 1\n") + case "/diagnostics": + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "client_info": map[string]string{ + "revision": "abc123def456", "chain_address": chainAddr, "network_id": networkID, + }, + }) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + return NewMetricsReportSource(srv.Client(), nil), InventoryInstance{ + InstanceID: "i1", OperatorAddress: inventoryOperator, + NetworkID: "net-1", TrustedReportTarget: srv.URL, + } + } + + t.Run("operator mismatch rejected", func(t *testing.T) { + src, inv := newSource("0x2222222222222222222222222222222222222222", "net-1") + if _, err := src.Fetch(context.Background(), inv); err == nil { + t.Error("a foreign chain address must be rejected") + } + }) + t.Run("network id mismatch rejected", func(t *testing.T) { + src, inv := newSource(inventoryOperator, "net-OTHER") + if _, err := src.Fetch(context.Background(), inv); err == nil { + t.Error("a mismatched network id must be rejected") + } + }) + t.Run("matching identity accepted", func(t *testing.T) { + src, inv := newSource(inventoryOperator, "net-1") + if _, err := src.Fetch(context.Background(), inv); err != nil { + t.Errorf("a matching self-attested identity must be accepted: %v", err) + } + }) +} + +// TestEthCallIdentityVerifier proves the verifier ABI-encodes the +// operatorToStakingProvider(address) call and decodes the returned address. +func TestEthCallIdentityVerifier(t *testing.T) { + operator := "0xabcdef0000000000000000000000000000000001" + stakingProvider := "0x1111111111111111111111111111111111111111" + + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var req struct { + Params []json.RawMessage `json:"params"` + } + if err := json.NewDecoder(r.Body).Decode(&req); err != nil { + t.Errorf("decode rpc: %v", err) + } + // The call object must target the contract and carry the selector. + var call struct { + To string `json:"to"` + Data string `json:"data"` + } + _ = json.Unmarshal(req.Params[0], &call) + if !strings.HasPrefix(call.Data, "0x") || len(call.Data) != 2+8+64 { + t.Errorf("unexpected calldata length: %q", call.Data) + } + // Return the staking provider right-aligned in a 32-byte word. + padded := "000000000000000000000000" + strings.TrimPrefix(stakingProvider, "0x") + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "jsonrpc": "2.0", "id": 1, "result": "0x" + padded, + }) + })) + defer srv.Close() + + verifier, err := NewEthCallIdentityVerifier( + srv.URL, "0x2222222222222222222222222222222222222222", srv.Client(), + ) + if err != nil { + t.Fatalf("construct: %v", err) + } + got, err := verifier.OperatorStakingProviderAtBlock(context.Background(), operator, 12345) + if err != nil { + t.Fatalf("verify: %v", err) + } + if got != stakingProvider { + t.Errorf("staking provider = %q, want %q", got, stakingProvider) + } + + // A non-canonical contract address is rejected at construction. + if _, err := NewEthCallIdentityVerifier(srv.URL, "not-an-address", srv.Client()); err == nil { + t.Error("expected rejection of a non-canonical contract address") + } +} + +// newExactAttestingNode is a fake keep node that self-attests exactly ONE +// identity — the given (operator chain address, network id) — and reports the +// exact cutover revision/epoch. It is the single physical responder used to prove +// one node cannot certify a second same-operator instance. +func newExactAttestingNode(t *testing.T, operator, networkID string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case metricsPath: + _, _ = io.WriteString(w, + `client_info{version="v2.0.0",revision="`+testRevision+ + `",protocol_epoch="`+ExpectedEpochSecurityV2Cutover+`"} 1`+"\n") + case diagnosticsPath: + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "client_info": map[string]string{ + "version": "v2.0.0", "revision": testRevision, + "chain_address": operator, "network_id": networkID, + }, + }) + default: + http.NotFound(w, r) + } + })) + t.Cleanup(srv.Close) + return srv +} + +// fetchAllReports mirrors the command's pollReports over the REAL +// MetricsReportSource: every eligible instance with a target is fetched, and a +// fetch error (e.g. an identity mismatch) simply omits that instance, exactly as +// production treats a missed collection. +func fetchAllReports( + source *MetricsReportSource, inventory []InventoryInstance, +) map[string]InstanceReport { + reports := map[string]InstanceReport{} + for _, in := range inventory { + if !in.CeremonyEligible || in.TrustedReportTarget == "" { + continue + } + r, err := source.Fetch(context.Background(), in) + if err != nil { + continue + } + reports[in.InstanceID] = r + } + return reports +} + +// TestEndToEnd_OneNodeCannotCertifyTwoSameOperatorInstances is the fix-round-3 +// regression guard for the per-instance trust-chain bypass. It drives the real +// metrics fetch path (as the command's pollReports does) into the collector and +// proves a single responding node can never satisfy two distinct same-operator +// inventory instances — the exact collapse the earlier round left reachable. +func TestEndToEnd_OneNodeCannotCertifyTwoSameOperatorInstances(t *testing.T) { + // run drives `cycles` full collection cycles over the real fetch path, keeping + // the report clock and the collector clock in lockstep so each cycle's + // attestation is strictly newer than the last. + run := func( + t *testing.T, tc *testCollector, source *MetricsReportSource, + inv []InventoryInstance, cycles int, + ) FleetSnapshot { + source.clock = func() time.Time { return tc.now } + var snap FleetSnapshot + for i := 0; i < cycles; i++ { + tc.now = tc.now.Add(time.Minute) + reports := fetchAllReports(source, inv) + var err error + snap, err = tc.collector.Collect(inv, reports, nil, 2000) + if err != nil { + t.Fatalf("collect: %v", err) + } + } + return snap + } + + twoTargets := func(node string, id1, net1, id2, net2 string) []InventoryInstance { + a := eligibleInstance(id1, "op1") + a.NetworkID = net1 + a.TrustedReportTarget = node + b := eligibleInstance(id2, "op1") + b.NetworkID = net2 + b.TrustedReportTarget = node + return []InventoryInstance{a, b} + } + + // The exact original bypass: two same-operator inventory entries with EMPTY + // network ids and the same explicit target, both answered by one node. On the + // pre-fix code both were certified; now neither can be. + t.Run("empty network ids answered by one node are not certified", func(t *testing.T) { + node := newExactAttestingNode(t, opAddr("op1"), "net-1") + source := NewMetricsReportSource(node.Client(), &MapAttestationSource{ + Digests: map[string]string{"i1": testDigest, "i2": testDigest}, + }) + inv := twoTargets(node.URL, "i1", "", "i2", "") + + tc := newTestCollector(t) + snap := run(t, tc, source, inv, 4) + + if status, ok := operatorStatus(snap, "op1"); ok && status == FleetResolvedCurrent { + t.Fatal("two empty-network-id instances answered by one node must not certify the operator") + } + if snap.Complete { + t.Fatal("readiness must not be complete when instances lack a per-instance network id") + } + if snap.Inventory.Unreconciled == 0 { + t.Fatal("empty per-instance network ids must count as an inventory-reconciliation fault") + } + }) + + // Even with well-formed distinct network ids, one node (attesting net-1) + // cannot cover a second same-operator instance declared as net-2: the metrics + // adapter rejects the mismatched fetch, so that instance stays offline and the + // operator never resolves. + t.Run("distinct network ids: one node cannot cover the second instance", func(t *testing.T) { + node := newExactAttestingNode(t, opAddr("op1"), "net-1") + source := NewMetricsReportSource(node.Client(), &MapAttestationSource{ + Digests: map[string]string{"i1": testDigest, "i2": testDigest}, + }) + inv := twoTargets(node.URL, "i1", "net-1", "i2", "net-2") + + tc := newTestCollector(t) + snap := run(t, tc, source, inv, 4) + + status, ok := operatorStatus(snap, "op1") + if !ok || status == FleetResolvedCurrent { + t.Fatalf( + "one node answering net-1 must not certify an operator whose second "+ + "instance is net-2; got ok=%v status=%v", ok, status, + ) + } + if snap.Complete { + t.Fatal("readiness must not be complete while the second same-operator instance is uncovered") + } + }) + + // Positive control: a single legitimate instance whose own node attests the + // matching identity still resolves through the exact same fetch path, so the + // fix does not block real convergence. + t.Run("a legitimate single node-attested instance still resolves", func(t *testing.T) { + node := newExactAttestingNode(t, opAddr("op1"), "net-1") + source := NewMetricsReportSource(node.Client(), &MapAttestationSource{ + Digests: map[string]string{"i1": testDigest}, + }) + i1 := eligibleInstance("i1", "op1") + i1.NetworkID = "net-1" + i1.TrustedReportTarget = node.URL + + tc := newTestCollector(t) + snap := run(t, tc, source, []InventoryInstance{i1}, 4) + + if status, ok := operatorStatus(snap, "op1"); !ok || status != FleetResolvedCurrent { + t.Fatalf("a legitimate node-attested instance must resolve; got ok=%v status=%v", ok, status) + } + if !snap.Complete { + t.Fatal("a fully resolved single-instance fleet must be complete") + } + }) +} diff --git a/pkg/monitoring/cutoverroster/quarantine.go b/pkg/monitoring/cutoverroster/quarantine.go new file mode 100644 index 0000000000..7fe888f98c --- /dev/null +++ b/pkg/monitoring/cutoverroster/quarantine.go @@ -0,0 +1,54 @@ +package cutoverroster + +import "strings" + +// VerifiedQuarantineEntry is one independently-verified quarantine/removal +// evidence record. It is a separate trusted input from the authoritative +// inventory, so an operator self-report placed in the inventory cannot fabricate +// quarantine on its own — the evidence reference must also appear here, having +// been independently verified out of band. +type VerifiedQuarantineEntry struct { + InstanceID string `json:"instance_id"` + OperatorAddress string `json:"operator_address"` + EvidenceRef string `json:"evidence_ref"` +} + +// AllowlistQuarantineVerifier verifies quarantine evidence against a fixed set +// of independently-verified entries. It satisfies QuarantineVerifier. +type AllowlistQuarantineVerifier struct { + verified map[string]struct{} +} + +// NewAllowlistQuarantineVerifier builds a verifier from the given +// independently-verified entries. +func NewAllowlistQuarantineVerifier( + entries []VerifiedQuarantineEntry, +) *AllowlistQuarantineVerifier { + verified := make(map[string]struct{}, len(entries)) + for _, e := range entries { + if strings.TrimSpace(e.EvidenceRef) == "" { + continue + } + verified[quarantineKey(e.InstanceID, e.OperatorAddress, e.EvidenceRef)] = struct{}{} + } + return &AllowlistQuarantineVerifier{verified: verified} +} + +// Verify reports whether the (instance, operator, evidence) triple is present in +// the independently-verified allowlist. An empty evidence reference never +// verifies. +func (v *AllowlistQuarantineVerifier) Verify( + instanceID, operatorAddress, evidenceRef string, +) bool { + if strings.TrimSpace(evidenceRef) == "" { + return false + } + _, ok := v.verified[quarantineKey(instanceID, operatorAddress, evidenceRef)] + return ok +} + +func quarantineKey(instanceID, operatorAddress, evidenceRef string) string { + return strings.ToLower(strings.TrimSpace(instanceID)) + "|" + + normalizeAddress(operatorAddress) + "|" + + strings.TrimSpace(evidenceRef) +} diff --git a/pkg/monitoring/cutoverroster/reportadapter.go b/pkg/monitoring/cutoverroster/reportadapter.go new file mode 100644 index 0000000000..4881d4fb79 --- /dev/null +++ b/pkg/monitoring/cutoverroster/reportadapter.go @@ -0,0 +1,310 @@ +package cutoverroster + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "time" +) + +// clientInfoMetricName is the Prometheus metric the node exposes carrying its +// build labels (pkg/clientinfo/metrics.go). In the current build it carries only +// `version`; once the cutover release adds `revision` and `protocol_epoch` to +// this metric (Part A's observability change), the adapter picks them up from the +// same place with no change here. +const clientInfoMetricName = "client_info" + +// AttestationSource supplies the externally-attested image digest and, until the +// node itself emits it, the release epoch for an instance. The running binary +// does not know its own container image digest, so the digest is always external +// inventory (per the spec, "the container digest remains external inventory +// because the binary does not know it"). A nil source attests nothing, which +// keeps a report that cannot prove its digest/epoch blocking (fail closed). +type AttestationSource interface { + // AttestedDigest returns the independently-attested image digest for the + // instance, and whether one exists. + AttestedDigest(instanceID string) (string, bool) + // AttestedEpoch returns the independently-attested release epoch for the + // instance, and whether one exists. It is consulted only when the node does + // not itself report the epoch via client_info. + AttestedEpoch(instanceID string) (string, bool) +} + +// MapAttestationSource is a fixed map-backed AttestationSource. +type MapAttestationSource struct { + Digests map[string]string + Epochs map[string]string +} + +// AttestedDigest implements AttestationSource. +func (m *MapAttestationSource) AttestedDigest(instanceID string) (string, bool) { + d, ok := m.Digests[instanceID] + return d, ok +} + +// AttestedEpoch implements AttestationSource. +func (m *MapAttestationSource) AttestedEpoch(instanceID string) (string, bool) { + e, ok := m.Epochs[instanceID] + return e, ok +} + +// MetricsReportSource builds an InstanceReport from a node's real exposed +// endpoints — /metrics (the client_info metric labels) and /diagnostics (the +// client_info JSON, which carries the exact revision) — rather than a bespoke +// JSON attestation contract. The image digest, and the release epoch until the +// node emits it, come from the independent AttestationSource. +type MetricsReportSource struct { + client *http.Client + attestation AttestationSource + clock func() time.Time +} + +// NewMetricsReportSource constructs a MetricsReportSource. A nil attestation +// source attests no digest/epoch (fail closed). A nil client uses a default with +// a 10s timeout. +func NewMetricsReportSource( + client *http.Client, + attestation AttestationSource, +) *MetricsReportSource { + if client == nil { + client = &http.Client{Timeout: 10 * time.Second} + } + return &MetricsReportSource{ + client: client, + attestation: attestation, + clock: time.Now, + } +} + +// diagnosticsPayload is the subset of the /diagnostics JSON the adapter reads. +// The /diagnostics endpoint returns a JSON object keyed by diagnostic source +// name; the "client_info" source carries the exact version and revision plus the +// node's self-attested chain address and network ID (pkg/clientinfo/diagnostics.go). +type diagnosticsPayload struct { + ClientInfo struct { + Version string `json:"version"` + Revision string `json:"revision"` + ChainAddress string `json:"chain_address"` + NetworkID string `json:"network_id"` + } `json:"client_info"` +} + +// Fetch scrapes the instance's /metrics and /diagnostics endpoints (derived from +// its trusted report base URL) and assembles an InstanceReport. The report's +// revision comes from the node's own diagnostics/metrics, its epoch from the +// node's client_info metric when present or otherwise from external attestation, +// and its image digest from external attestation. A missing endpoint or an +// unparseable body is an error (treated by the collector as a missed collection). +// +// The responding node's identity is validated against its OWN diagnostics +// payload rather than copied from inventory: the report's operator address is +// taken from the node's self-attested chain_address, and it must match the +// inventory operator address the target answers for. When the inventory carries +// the instance's network ID, the node's self-attested network_id must match it +// too. A mismatch means the responding target is not the trusted instance, and +// the fetch fails (a missed collection) rather than certifying a foreign report. +func (s *MetricsReportSource) Fetch( + ctx context.Context, + inv InventoryInstance, +) (InstanceReport, error) { + // InstanceID is only the inventory's label for this instance; it is NOT the + // node's proven identity. The trust binding below is (operator address, + // network ID), both taken from the responding node's own attestation and + // matched against inventory. The collector separately re-validates that the + // report's InstanceID equals the inventory instance it answers for. + report := InstanceReport{ + InstanceID: inv.InstanceID, + } + + base := strings.TrimSuffix(strings.TrimSpace(inv.TrustedReportTarget), "/") + base = strings.TrimSuffix(base, metricsPath) + base = strings.TrimSuffix(base, diagnosticsPath) + if base == "" { + return report, fmt.Errorf("no report target for instance %s", inv.InstanceID) + } + + // /metrics: confirm the node is up and read the client_info labels (version, + // and revision/protocol_epoch once the release adds them there). + metricsBody, err := s.get(ctx, base+metricsPath) + if err != nil { + return report, err + } + labels := parseClientInfoLabels(metricsBody) + if labels == nil { + return report, fmt.Errorf("client_info metric absent from %s", inv.InstanceID) + } + report.Revision = labels["revision"] + report.Epoch = labels["protocol_epoch"] + + // /diagnostics: read the exact revision (which the current build exposes here + // rather than in the client_info metric) and the node's self-attested identity. + diagBody, err := s.get(ctx, base+diagnosticsPath) + if err != nil { + return report, err + } + var diag diagnosticsPayload + if err := json.Unmarshal([]byte(diagBody), &diag); err != nil { + return report, fmt.Errorf("cannot decode diagnostics from %s: %w", inv.InstanceID, err) + } + if report.Revision == "" { + report.Revision = strings.TrimSpace(diag.ClientInfo.Revision) + } + + // Validate the responding node's self-attested identity instead of copying it + // from inventory. The chain address it reports for itself must be canonical and + // must equal the operator address the inventory says this target answers for. + observedOperator := normalizeAddress(diag.ClientInfo.ChainAddress) + if !isCanonicalAddress(observedOperator) { + return report, fmt.Errorf( + "diagnostics chain address is not a canonical operator address for %s", + inv.InstanceID, + ) + } + if observedOperator != normalizeAddress(inv.OperatorAddress) { + return report, fmt.Errorf( + "responding node operator identity mismatch for %s", inv.InstanceID, + ) + } + // The report's operator address comes from the node's own attestation. + report.OperatorAddress = observedOperator + + // Bind the report to the responding node's OWN network identity, + // unconditionally — this is the per-instance guarantee that closes the + // collapse where one responding node certifies several same-operator + // inventory instances. The inventory must pin the instance's network ID, the + // node must self-attest a network ID, and the two must match. A single node + // attests exactly one network ID, so it can satisfy at most the one inventory + // instance whose NetworkID equals it; a same-operator instance carrying a + // different (or absent) network ID is rejected here rather than certified from + // an inventory-copied identity. The network ID stored on the report is the + // node's attested value, never inventory. + expectedNetworkID := strings.TrimSpace(inv.NetworkID) + observedNetworkID := strings.TrimSpace(diag.ClientInfo.NetworkID) + if expectedNetworkID == "" { + return report, fmt.Errorf( + "inventory does not pin a network ID for %s; cannot bind report identity", + inv.InstanceID, + ) + } + if observedNetworkID == "" { + return report, fmt.Errorf( + "responding node did not attest a network ID for %s", inv.InstanceID, + ) + } + if observedNetworkID != expectedNetworkID { + return report, fmt.Errorf( + "responding node network identity mismatch for %s", inv.InstanceID, + ) + } + report.NetworkID = observedNetworkID + + // The image digest is always external attestation; the release epoch is taken + // from attestation only when the node did not report it via client_info. + if s.attestation != nil { + if digest, ok := s.attestation.AttestedDigest(inv.InstanceID); ok { + report.ImageDigest = digest + } + if report.Epoch == "" { + if epoch, ok := s.attestation.AttestedEpoch(inv.InstanceID); ok { + report.Epoch = epoch + } + } + } + + report.AttestedAt = s.clock() + // ReporterRevision is derived from the attestation timestamp (wall-clock + // nanoseconds) rather than a process-local counter, so the collector's + // replay/downgrade high-water-mark guard keeps accepting reports immediately + // after a collector restart. A resettable in-memory sequence would start below + // the persisted high-water mark and reject every report for as many cycles as + // the previous process had run. + report.ReporterRevision = uint64(report.AttestedAt.UnixNano()) + + return report, nil +} + +func (s *MetricsReportSource) get(ctx context.Context, url string) (string, error) { + // #nosec G107 -- the URL is derived from operator-supplied trusted service + // discovery / inventory for the monitoring tool. + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return "", err + } + resp, err := s.client.Do(req) + if err != nil { + // Sanitize: do not surface the raw transport error, which embeds the + // requested URL (host/IP) that must not appear in logs. + return "", fmt.Errorf("request failed") + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("unexpected status %d", resp.StatusCode) + } + buf := new(strings.Builder) + if _, err := io.Copy(buf, io.LimitReader(resp.Body, maxReportBodyBytes)); err != nil { + return "", fmt.Errorf("cannot read response body") + } + return buf.String(), nil +} + +// maxReportBodyBytes caps how much of a report endpoint response is read, so a +// misbehaving or hostile target cannot exhaust memory. +const maxReportBodyBytes = 8 << 20 // 8 MiB + +// parseClientInfoLabels extracts the label set of the client_info metric from a +// Prometheus text exposition. It returns nil if the metric line is absent. Only +// the first client_info series is read; HELP/TYPE comment lines are ignored. +func parseClientInfoLabels(promText string) map[string]string { + for _, line := range strings.Split(promText, "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + if !strings.HasPrefix(line, clientInfoMetricName) || + len(line) == len(clientInfoMetricName) { + continue + } + // The metric name must be followed by a label brace or whitespace before + // the value, so a longer name such as client_info_extra does not match. + next := line[len(clientInfoMetricName)] + if next != '{' && next != ' ' && next != '\t' { + continue + } + open := strings.IndexByte(line, '{') + if open < 0 { + // client_info with no labels. + return map[string]string{} + } + closeIdx := strings.IndexByte(line, '}') + if closeIdx < open { + continue + } + return parseMetricLabels(line[open+1 : closeIdx]) + } + return nil +} + +// parseMetricLabels parses a Prometheus label list body (the text between the +// braces) into a map. It handles simple double-quoted values without escape +// sequences, which is sufficient for the build-info labels the node emits. +func parseMetricLabels(body string) map[string]string { + labels := map[string]string{} + for _, pair := range strings.Split(body, ",") { + pair = strings.TrimSpace(pair) + if pair == "" { + continue + } + eq := strings.IndexByte(pair, '=') + if eq < 0 { + continue + } + key := strings.TrimSpace(pair[:eq]) + value := strings.TrimSpace(pair[eq+1:]) + value = strings.Trim(value, `"`) + labels[key] = value + } + return labels +} diff --git a/pkg/monitoring/cutoverroster/servicediscovery.go b/pkg/monitoring/cutoverroster/servicediscovery.go new file mode 100644 index 0000000000..b2cd983cd5 --- /dev/null +++ b/pkg/monitoring/cutoverroster/servicediscovery.go @@ -0,0 +1,204 @@ +package cutoverroster + +import ( + "encoding/json" + "fmt" + "strings" +) + +// The production Prometheus consumes a file-based service-discovery target file +// (keep-sd.json) in the standard Prometheus file_sd format and attaches the +// operator's on-chain address under the __meta_chain_address label +// (infrastructure/kube/keep-prd/monitoring/prometheus/config/config.yaml). The +// discovered targets expose /metrics over http. This parser reads exactly that +// file so the collector reconciles against the same authoritative discovery input +// Prometheus scrapes, rather than an inventory-only view. +const ( + // metaChainAddressLabel is the Prometheus meta-label carrying the operator's + // on-chain address in the keep-sd.json target file. + metaChainAddressLabel = "__meta_chain_address" + // metaNetworkIDLabel is the Prometheus meta-label carrying the node's libp2p + // network ID in the keep-sd.json target file. It is the per-instance + // disambiguator: two instances of one operator carry the same chain address + // but distinct network IDs. + metaNetworkIDLabel = "__meta_network_id" + // discoveryScheme is the scrape scheme the production Prometheus config uses + // for discovered nodes. + discoveryScheme = "http" + // metricsPath and diagnosticsPath are the endpoints a discovered node exposes. + metricsPath = "/metrics" + diagnosticsPath = "/diagnostics" +) + +// fileSDEntry is one entry of the Prometheus file_sd target file: a set of +// scrape targets plus the meta-labels attached to them. +type fileSDEntry struct { + Targets []string `json:"targets"` + Labels map[string]string `json:"labels"` +} + +// discoveredTarget is one instance's discovered scrape base URL together with +// the operator it belongs to, so a per-instance lookup can confirm the operator +// matches before handing back a target. +type discoveredTarget struct { + operator string + baseURL string +} + +// ServiceDiscovery is the parsed production service-discovery target set. It is +// keyed by node network ID (the per-instance disambiguator) so multiple +// instances of one operator resolve to distinct discovered targets rather than +// collapsing onto a single operator-level URL, and it separately records which +// operators appear anywhere in discovery for reconciliation rule 2. +type ServiceDiscovery struct { + byNetworkID map[string]discoveredTarget + operatorsPresent map[string]bool +} + +// ParseServiceDiscovery parses the Prometheus file_sd target file (keep-sd.json) +// that production Prometheus consumes. For every target it reads the +// __meta_chain_address label (the operator address), the __meta_network_id label +// (the per-instance network ID), and the target host:port, and records the +// instance's http scrape base URL keyed by network ID. Entries without a +// canonical chain-address label or without a target are skipped as unusable +// discovery rows; an entry without a network ID still marks the operator present +// (for rule 2) but yields no per-instance target. +func ParseServiceDiscovery(data []byte) (*ServiceDiscovery, error) { + var entries []fileSDEntry + if err := json.Unmarshal(data, &entries); err != nil { + return nil, fmt.Errorf("cannot decode service-discovery target file: %w", err) + } + + sd := &ServiceDiscovery{ + byNetworkID: map[string]discoveredTarget{}, + operatorsPresent: map[string]bool{}, + } + for _, entry := range entries { + operator := normalizeAddress(entry.Labels[metaChainAddressLabel]) + if !isCanonicalAddress(operator) { + continue + } + var base string + for _, target := range entry.Targets { + target = strings.TrimSpace(target) + if target != "" { + base = discoveryScheme + "://" + target + break + } + } + if base == "" { + continue + } + // The operator is present in discovery regardless of whether the entry + // carries a per-instance network ID. + sd.operatorsPresent[operator] = true + + networkID := strings.TrimSpace(entry.Labels[metaNetworkIDLabel]) + if networkID == "" { + continue + } + // First usable target wins for a given network ID: one instance maps to a + // single scrape base URL. + if _, exists := sd.byNetworkID[networkID]; !exists { + sd.byNetworkID[networkID] = discoveredTarget{operator: operator, baseURL: base} + } + } + return sd, nil +} + +// Has reports whether the operator is present anywhere in the service-discovery +// target set. +func (s *ServiceDiscovery) Has(operatorAddress string) bool { + return s.operatorsPresent[normalizeAddress(operatorAddress)] +} + +// HasInstance reports whether the specific instance identified by the full +// (operatorAddress, networkID) identity tuple is present in the service-discovery +// target set. It is the per-instance disambiguator: unlike Has (which is true for +// an operator with any discovered instance), this requires the exact network ID +// to be discovered AND to belong to the claimed operator, so a second instance of +// one operator that never appears in discovery is not covered by a sibling +// instance's presence. +func (s *ServiceDiscovery) HasInstance(operatorAddress, networkID string) bool { + networkID = strings.TrimSpace(networkID) + if networkID == "" { + return false + } + target, ok := s.byNetworkID[networkID] + if !ok { + return false + } + return target.operator == normalizeAddress(operatorAddress) +} + +// instanceBaseURL returns the discovered scrape base URL for the specific +// instance identified by (operatorAddress, networkID), or "". It requires the +// network ID (the per-instance key) and confirms the discovered target belongs +// to the claimed operator. +func (s *ServiceDiscovery) instanceBaseURL(operatorAddress, networkID string) string { + networkID = strings.TrimSpace(networkID) + if networkID == "" { + return "" + } + target, ok := s.byNetworkID[networkID] + if !ok { + return "" + } + if target.operator != normalizeAddress(operatorAddress) { + return "" + } + return target.baseURL +} + +// MetricsURLForInstance returns the discovered /metrics scrape URL for the +// specific instance identified by (operatorAddress, networkID), or "". +func (s *ServiceDiscovery) MetricsURLForInstance(operatorAddress, networkID string) string { + base := s.instanceBaseURL(operatorAddress, networkID) + if base == "" { + return "" + } + return base + metricsPath +} + +// DiagnosticsURLForInstance returns the discovered /diagnostics URL for the +// specific instance identified by (operatorAddress, networkID), or "". +func (s *ServiceDiscovery) DiagnosticsURLForInstance(operatorAddress, networkID string) string { + base := s.instanceBaseURL(operatorAddress, networkID) + if base == "" { + return "" + } + return base + diagnosticsPath +} + +// Len returns the number of operators present in service discovery. +func (s *ServiceDiscovery) Len() int { + return len(s.operatorsPresent) +} + +// ReconcileWithDiscovery annotates the authoritative inventory against the +// production service-discovery target set. An eligible instance whose exact +// (operator, networkID) identity is absent from discovery is flagged +// DisappearedFromDiscovery (reconciliation rule 2: disappearance from service +// discovery is offline_unknown). Keying by the full identity tuple — not the +// operator alone — is what keeps distinct instances of one operator from +// collapsing: a second instance that never appears in discovery is flagged even +// when a sibling instance of the same operator is discovered. It returns the +// inventory with the flags applied. A nil ServiceDiscovery leaves the inventory +// unchanged (no discovery feed configured). +func ReconcileWithDiscovery( + inventory []InventoryInstance, + sd *ServiceDiscovery, +) []InventoryInstance { + if sd == nil { + return inventory + } + for i := range inventory { + if !inventory[i].CeremonyEligible { + continue + } + if !sd.HasInstance(inventory[i].OperatorAddress, inventory[i].NetworkID) { + inventory[i].DisappearedFromDiscovery = true + } + } + return inventory +} diff --git a/pkg/monitoring/cutoverroster/store.go b/pkg/monitoring/cutoverroster/store.go new file mode 100644 index 0000000000..3376e71b08 --- /dev/null +++ b/pkg/monitoring/cutoverroster/store.go @@ -0,0 +1,239 @@ +package cutoverroster + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "time" + + bolt "go.etcd.io/bbolt" +) + +var ( + bucketOperators = []byte("operators") + bucketInstances = []byte("instances") +) + +// operatorRecord is the persisted per-operator central state. Central state is +// only advanced by resolution or verified quarantine; local eviction, reporter +// restarts, quiet counters, or service-discovery churn never resolve it. +type operatorRecord struct { + OperatorAddress string `json:"operator_address"` + StakingProvider string `json:"staking_provider"` + Status FleetStatus `json:"status"` + FirstSeenBlock uint64 `json:"first_seen_block"` + LastSeenBlock uint64 `json:"last_seen_block"` + Reason string `json:"reason"` + LastLegacyBlock uint64 `json:"last_legacy_block"` + LastLegacyAt time.Time `json:"last_legacy_at"` + ResolvedAt time.Time `json:"resolved_at"` +} + +// instanceRecord is the persisted per-instance report history and the +// authoritative inventory expectations that were last reconciled for the +// instance. The per-instance expectations (ceremony eligibility, staking +// provider, and expected revision/epoch/digest) are persisted so an audit or a +// restarted collector can see exactly what each instance was expected to report, +// not only whether it reported. +type instanceRecord struct { + InstanceID string `json:"instance_id"` + OperatorAddress string `json:"operator_address"` + LatestReport *InstanceReport `json:"latest_report,omitempty"` + ConsecutiveExact uint `json:"consecutive_exact"` + ConsecutiveMissed uint `json:"consecutive_missed"` + HasQuarantine bool `json:"has_quarantine"` + QuarantineRef string `json:"quarantine_ref,omitempty"` + // LastReporterRevision is the highest accepted InstanceReport.ReporterRevision + // for this instance. It guards against replayed or downgraded attestations. + LastReporterRevision uint64 `json:"last_reporter_revision"` + + // Per-instance authoritative inventory expectations, last observed for the + // instance. They are recorded for auditability so a reader can see the exact + // per-instance expected artifact identity rather than only the collector-wide + // configured expectation. + CeremonyEligible bool `json:"ceremony_eligible"` + StakingProvider string `json:"staking_provider,omitempty"` + ExpectedRevision string `json:"expected_revision,omitempty"` + ExpectedEpoch string `json:"expected_epoch,omitempty"` + ExpectedImageDigest string `json:"expected_image_digest,omitempty"` + + // ReportedThisCycle records whether a report from this instance was accepted + // in the most recent collection cycle. It is deliberately distinct from + // "LatestReport != nil" (which means "ever reported"): the unresolved-operator + // log and the per-instance status use this to count only instances that + // reported in the current cycle, not historical reporters. + ReportedThisCycle bool `json:"reported_this_cycle"` + + // DisappearedFromDiscovery records whether the instance was absent from the + // production service-discovery target set in the most recent cycle while still + // present in the authoritative inventory. Disappearance from service discovery + // is offline_unknown and never resolves central state. + DisappearedFromDiscovery bool `json:"disappeared_from_discovery"` +} + +// Store is the transactional bbolt persistence for the fleet collector. +type Store struct { + db *bolt.DB +} + +// OpenStore opens (creating if needed) the bbolt database at path, creating the +// parent directory and the required buckets. +func OpenStore(path string) (*Store, error) { + if path == "" { + return nil, fmt.Errorf("store path must not be empty") + } + + dir := filepath.Dir(path) + if err := os.MkdirAll(dir, 0o700); err != nil { + return nil, fmt.Errorf("cannot create store directory [%s]: %w", dir, err) + } + + // #nosec G304 -- path is the operator-supplied database location for the + // monitoring tool; it is intentionally configurable. + db, err := bolt.Open(path, 0o600, &bolt.Options{Timeout: 5 * time.Second}) + if err != nil { + return nil, fmt.Errorf("cannot open store [%s]: %w", path, err) + } + + err = db.Update(func(tx *bolt.Tx) error { + if _, err := tx.CreateBucketIfNotExists(bucketOperators); err != nil { + return err + } + if _, err := tx.CreateBucketIfNotExists(bucketInstances); err != nil { + return err + } + return nil + }) + if err != nil { + _ = db.Close() + return nil, fmt.Errorf("cannot initialize store buckets: %w", err) + } + + return &Store{db: db}, nil +} + +// LoadOperators reads all persisted operator records. +func (s *Store) LoadOperators() (map[string]*operatorRecord, error) { + operators := make(map[string]*operatorRecord) + err := s.db.View(func(tx *bolt.Tx) error { + bucket := tx.Bucket(bucketOperators) + if bucket == nil { + return nil + } + return bucket.ForEach(func(k, v []byte) error { + record := &operatorRecord{} + if err := json.Unmarshal(v, record); err != nil { + return fmt.Errorf("cannot decode operator [%s]: %w", k, err) + } + operators[string(k)] = record + return nil + }) + }) + if err != nil { + return nil, err + } + return operators, nil +} + +// LoadInstances reads all persisted instance records. +func (s *Store) LoadInstances() (map[string]*instanceRecord, error) { + instances := make(map[string]*instanceRecord) + err := s.db.View(func(tx *bolt.Tx) error { + bucket := tx.Bucket(bucketInstances) + if bucket == nil { + return nil + } + return bucket.ForEach(func(k, v []byte) error { + record := &instanceRecord{} + if err := json.Unmarshal(v, record); err != nil { + return fmt.Errorf("cannot decode instance [%s]: %w", k, err) + } + instances[string(k)] = record + return nil + }) + }) + if err != nil { + return nil, err + } + return instances, nil +} + +// Save transactionally rewrites the operator and instance buckets so that they +// exactly match the supplied maps, including deletions (used for the 30-day +// resolved purge). The entire write is a single bbolt transaction. +func (s *Store) Save( + operators map[string]*operatorRecord, + instances map[string]*instanceRecord, +) error { + return s.db.Update(func(tx *bolt.Tx) error { + if err := syncBucket(tx, bucketOperators, encodeOperators(operators)); err != nil { + return err + } + return syncBucket(tx, bucketInstances, encodeInstances(instances)) + }) +} + +// Close closes the underlying database. It is safe to call on a nil store. +func (s *Store) Close() error { + if s == nil || s.db == nil { + return nil + } + return s.db.Close() +} + +func encodeOperators(operators map[string]*operatorRecord) map[string][]byte { + encoded := make(map[string][]byte, len(operators)) + for key, record := range operators { + // Errors are impossible for these plain structs; ignore defensively. + data, _ := json.Marshal(record) + encoded[key] = data + } + return encoded +} + +func encodeInstances(instances map[string]*instanceRecord) map[string][]byte { + encoded := make(map[string][]byte, len(instances)) + for key, record := range instances { + data, _ := json.Marshal(record) + encoded[key] = data + } + return encoded +} + +// syncBucket makes the bucket contents equal to `desired`, deleting any keys +// not present in it. +func syncBucket(tx *bolt.Tx, name []byte, desired map[string][]byte) error { + bucket, err := tx.CreateBucketIfNotExists(name) + if err != nil { + return err + } + + // Delete keys that are no longer present. + var toDelete [][]byte + err = bucket.ForEach(func(k, _ []byte) error { + if _, ok := desired[string(k)]; !ok { + key := make([]byte, len(k)) + copy(key, k) + toDelete = append(toDelete, key) + } + return nil + }) + if err != nil { + return err + } + for _, key := range toDelete { + if err := bucket.Delete(key); err != nil { + return err + } + } + + // Upsert current keys. + for key, value := range desired { + if err := bucket.Put([]byte(key), value); err != nil { + return err + } + } + + return nil +} diff --git a/pkg/monitoring/cutoverroster/types.go b/pkg/monitoring/cutoverroster/types.go new file mode 100644 index 0000000000..15488cc2d8 --- /dev/null +++ b/pkg/monitoring/cutoverroster/types.go @@ -0,0 +1,273 @@ +// Package cutoverroster implements the authoritative fleet aggregation view for +// a coordinated protocol cutover. It joins node-local post-cutover legacy +// sightings and per-instance revision/epoch/digest attestations to an +// authoritative ceremony-eligible inventory, and answers "which eligible +// instance has not reported the exact cutover release?" — the primary go/no-go +// question. +// +// This package is decoupled from the cutover gate itself: the expected release +// identity (revision, epoch, image digest) and the cutover block are plain +// operator-supplied configuration. They become meaningful once the real cutover +// release ships. +package cutoverroster + +import "time" + +// FleetSnapshotSchemaVersion is the schema version of the persisted and +// API-exposed fleet snapshot. +const FleetSnapshotSchemaVersion uint32 = 1 + +// ExpectedEpochSecurityV2Cutover is the release epoch string the cutover +// artifact reports. +const ExpectedEpochSecurityV2Cutover = "security_v2_cutover" + +// FleetStatus is the reconciled per-operator cutover status. +type FleetStatus string + +const ( + // FleetObservedLegacy means a valid post-cutover node-local legacy sighting + // exists for the operator; it outranks every other blocking status. + FleetObservedLegacy FleetStatus = "observed_legacy" + // FleetNonCutoverRevision means an eligible, nonquarantined instance is + // reporting a revision, epoch, or image digest that differs from the + // expected cutover release. + FleetNonCutoverRevision FleetStatus = "noncutover_revision" + // FleetOfflineUnknown means the operator cannot be confirmed current: + // missing collections, no trusted report path, identity mismatch, or not + // yet confirmed by enough consecutive exact reports. Offline is never ready. + FleetOfflineUnknown FleetStatus = "offline_unknown" + // FleetQuarantined means every otherwise-blocking instance has independently + // verified network/eligibility quarantine or removal evidence. + FleetQuarantined FleetStatus = "quarantined" + // FleetResolvedCurrent means every authoritative eligible instance reported + // the exact cutover revision/epoch/digest in enough consecutive accepted + // collections, all newer than the last legacy observation. + FleetResolvedCurrent FleetStatus = "resolved_current" +) + +// IsBlocking reports whether the status blocks cutover readiness. +func (s FleetStatus) IsBlocking() bool { + switch s { + case FleetObservedLegacy, FleetNonCutoverRevision, FleetOfflineUnknown: + return true + default: + return false + } +} + +// InventoryInstance is one authoritative ceremony-eligible instance record. It +// is operator-supplied inventory, not a discovered scrape target. +type InventoryInstance struct { + InstanceID string `json:"instance_id"` + OperatorAddress string `json:"operator_address"` + StakingProvider string `json:"staking_provider"` + // NetworkID is the instance's libp2p network identity (the node's own + // network_id, exposed by /diagnostics client_info). It is the per-instance + // join key against production service discovery and the responding node's + // self-attested identity, so multiple instances of one operator resolve to + // distinct discovered targets rather than collapsing onto one. + NetworkID string `json:"network_id"` + CeremonyEligible bool `json:"ceremony_eligible"` + ExpectedRevision string `json:"expected_revision"` + ExpectedEpoch string `json:"expected_epoch"` + ExpectedImageDigest string `json:"expected_image_digest"` + TrustedReportTarget string `json:"-"` + QuarantineEvidenceRef string `json:"quarantine_evidence_ref,omitempty"` + + // DisappearedFromDiscovery is set by the command layer when the instance's + // operator is present in the authoritative inventory but absent from the + // production service-discovery target set for this cycle. It is in-memory only + // (never serialized) and drives reconciliation rule 2: disappearance from + // service discovery is offline_unknown and never resolves central state. + DisappearedFromDiscovery bool `json:"-"` +} + +// InventoryInstanceInput is the on-disk inventory input form. Unlike +// InventoryInstance — whose TrustedReportTarget is `json:"-"` so it is never +// serialized back out — this input form carries the trusted report target under +// an explicit JSON key so operator inventory can supply it. The collector copies +// it into the in-memory InventoryInstance, which never serializes the target. +type InventoryInstanceInput struct { + InstanceID string `json:"instance_id"` + OperatorAddress string `json:"operator_address"` + StakingProvider string `json:"staking_provider"` + NetworkID string `json:"network_id"` + CeremonyEligible bool `json:"ceremony_eligible"` + ExpectedRevision string `json:"expected_revision"` + ExpectedEpoch string `json:"expected_epoch"` + ExpectedImageDigest string `json:"expected_image_digest"` + TrustedReportTarget string `json:"trusted_report_target"` + QuarantineEvidenceRef string `json:"quarantine_evidence_ref,omitempty"` +} + +// ToInventoryInstance converts the on-disk input form to the in-memory +// InventoryInstance, carrying the trusted report target across. The in-memory +// form additionally carries DisappearedFromDiscovery, which is never sourced from +// operator input — it is computed by the command layer from the production +// service-discovery target set — so the conversion maps the shared fields +// explicitly and leaves that field at its zero value. +func (i InventoryInstanceInput) ToInventoryInstance() InventoryInstance { + return InventoryInstance{ + InstanceID: i.InstanceID, + OperatorAddress: i.OperatorAddress, + StakingProvider: i.StakingProvider, + NetworkID: i.NetworkID, + CeremonyEligible: i.CeremonyEligible, + ExpectedRevision: i.ExpectedRevision, + ExpectedEpoch: i.ExpectedEpoch, + ExpectedImageDigest: i.ExpectedImageDigest, + TrustedReportTarget: i.TrustedReportTarget, + QuarantineEvidenceRef: i.QuarantineEvidenceRef, + } +} + +// InstanceReport is one attested report obtained from an instance's trusted +// report target during a collection cycle. +type InstanceReport struct { + InstanceID string `json:"instance_id"` + OperatorAddress string `json:"operator_address"` + // NetworkID is the responding node's OWN self-attested libp2p network identity + // (from its /diagnostics client_info), NOT copied from inventory. It is the + // per-instance identity the node proves for itself, so one responding node — + // which can attest only a single network ID — cannot stand in for two distinct + // same-operator inventory instances. The metrics report source populates it and + // rejects a report whose attested network ID does not match the inventory + // instance it answers for. + NetworkID string `json:"network_id"` + Revision string `json:"revision"` + Epoch string `json:"epoch"` + ImageDigest string `json:"image_digest"` + AttestedAt time.Time `json:"attested_at"` + ReporterRevision uint64 `json:"reporter_revision"` +} + +// LegacySighting is a post-cutover node-local legacy sighting for an operator, +// aggregated from the node-local cutover peer rosters. +type LegacySighting struct { + OperatorAddress string `json:"operator_address"` + Block uint64 `json:"block"` + ObservedAt time.Time `json:"observed_at"` +} + +// FleetInstanceStatus is the per-instance reconciliation detail exposed in a +// snapshot for the dashboard's instance-level reasons and the audit trail. It +// pairs each authoritative instance's observed identity with the expected +// release identity, its reconciliation class/reason, and any independently +// verified quarantine evidence, so a reader can see exactly why an operator is +// blocking without joining separate inputs. +type FleetInstanceStatus struct { + InstanceID string `json:"instance_id"` + OperatorAddress string `json:"operator_address"` + Class string `json:"class"` + Reason string `json:"reason"` + // Reported means the instance has ever produced an accepted report. + Reported bool `json:"reported"` + // ReportedThisCycle means an accepted report was obtained in the current + // collection cycle. It is deliberately distinct from Reported so an auditor + // can tell a currently-reporting instance from a historical one. + ReportedThisCycle bool `json:"reported_this_cycle"` + // Per-instance authoritative inventory expectations, exposed so the dashboard + // and audit trail can show exactly what the instance was expected to report. + CeremonyEligible bool `json:"ceremony_eligible"` + StakingProvider string `json:"staking_provider,omitempty"` + ExpectedRevision string `json:"expected_revision,omitempty"` + ExpectedEpoch string `json:"expected_epoch,omitempty"` + ExpectedImageDigest string `json:"expected_image_digest,omitempty"` + ObservedRevision string `json:"observed_revision,omitempty"` + ObservedEpoch string `json:"observed_epoch,omitempty"` + ObservedDigest string `json:"observed_image_digest,omitempty"` + // ReporterRevision is the reporter-revision of the latest accepted report, + // exposed for auditability alongside the observed artifact identity. + ReporterRevision uint64 `json:"reporter_revision,omitempty"` + AttestedAt time.Time `json:"attested_at,omitempty"` + ConsecutiveExact uint `json:"consecutive_exact"` + ConsecutiveMissed uint `json:"consecutive_missed"` + Quarantined bool `json:"quarantined"` + QuarantineRef string `json:"quarantine_ref,omitempty"` +} + +// FleetOperatorEntry is the reconciled per-operator entry exposed in a snapshot. +// Instances carries the raw attested reports (as specified); InstanceStatuses +// adds the per-instance reconciliation detail (class, reason, expected-vs- +// observed identity, and quarantine evidence) required for the dashboard's +// instance-level reasons and the audit record. +type FleetOperatorEntry struct { + OperatorAddress string `json:"operator_address"` + StakingProvider string `json:"staking_provider"` + Status FleetStatus `json:"status"` + Instances []InstanceReport `json:"instances"` + InstanceStatuses []FleetInstanceStatus `json:"instance_statuses"` + FirstSeenBlock uint64 `json:"first_seen_block"` + LastSeenBlock uint64 `json:"last_seen_block"` + Reason string `json:"reason"` +} + +// FleetInventoryCounts summarizes the authoritative inventory reconciled this +// cycle: how many instances were supplied in total, how many were ceremony +// eligible, how many eligible instances lacked a fresh accepted report, and how +// many identity/target/inventory reconciliation faults were seen. +type FleetInventoryCounts struct { + TotalInstances int `json:"total_instances"` + EligibleInstances int `json:"eligible_instances"` + ReportersStale int `json:"reporters_stale"` + Unreconciled int `json:"unreconciled"` +} + +// FleetSnapshot is the deterministic authoritative fleet view. +type FleetSnapshot struct { + SchemaVersion uint32 `json:"schema_version"` + GeneratedAt time.Time `json:"generated_at"` + CurrentBlock uint64 `json:"current_block"` + CutoverBlock uint64 `json:"cutover_block"` + Complete bool `json:"complete"` + ExpectedRevision string `json:"expected_revision"` + ExpectedEpoch string `json:"expected_epoch"` + ExpectedDigest string `json:"expected_image_digest"` + Inventory FleetInventoryCounts `json:"inventory"` + Blocking []FleetOperatorEntry `json:"blocking"` + Quarantined []FleetOperatorEntry `json:"quarantined"` + RecentlyResolved []FleetOperatorEntry `json:"recently_resolved"` +} + +// CollectorConfig configures the fleet collector. ExpectedRevision, +// ExpectedEpoch, ExpectedImageDigest, and CutoverBlock are plain +// operator-supplied values; they become meaningful once the real cutover +// release ships. +type CollectorConfig struct { + ExpectedRevision string + ExpectedEpoch string + ExpectedImageDigest string + CutoverBlock uint64 + ChainID string + CollectionInterval time.Duration + MissedThreshold uint + SuccessThreshold uint + + // RequireServiceDiscovery makes reconciliation against the production + // service-discovery target set mandatory for completeness. When true, a + // collector that was not told service discovery is configured can never + // certify readiness — a missing discovery feed blocks readiness rather than + // silently degrading to trusting the inventory alone. + RequireServiceDiscovery bool + // RequireIdentityVerification makes an installed on-chain + // operator→staking-provider identity verifier mandatory for completeness. + // When true and no verifier is installed, readiness can never be complete — + // a missing WalletRegistry verification blocks readiness rather than + // certifying trusted-file identity assertions on their own. + RequireIdentityVerification bool +} + +// Metric names for the authoritative fleet aggregation. +const ( + MetricFleetBlockingOperators = "performance_cutover_fleet_blocking_operators" + MetricFleetObservedLegacy = "performance_cutover_fleet_observed_legacy" + MetricReportersStale = "performance_cutover_reporters_stale" + MetricInventoryUnreconciled = "performance_cutover_inventory_unreconciled" + MetricOperatorInfo = "performance_cutover_operator_info" + MetricOperatorFirstSeenBlock = "performance_cutover_operator_first_seen_block" + MetricOperatorLastSeenBlock = "performance_cutover_operator_last_seen_block" +) + +// ResolvedRetention is how long resolved operator records are retained before +// purge. Unresolved (blocking/quarantined) history is retained indefinitely. +const ResolvedRetention = 30 * 24 * time.Hour diff --git a/pkg/net/libp2p/authenticated_connection.go b/pkg/net/libp2p/authenticated_connection.go index b2f477854f..661ff43718 100644 --- a/pkg/net/libp2p/authenticated_connection.go +++ b/pkg/net/libp2p/authenticated_connection.go @@ -22,10 +22,8 @@ import ( "github.com/keep-network/keep-core/pkg/net/gen/pb" "github.com/keep-network/keep-core/pkg/net/security/handshake" + "google.golang.org/protobuf/encoding/protodelim" "google.golang.org/protobuf/proto" - // TODO: Stop using `dev` version of `google.golang.org/protobuf` once v.1.28.2 - // is published. - protodelim "google.golang.org/protobuf/dev/encoding/protodelim" ) // Enough space for a proto-encoded envelope with a message, peer.ID, and sig. diff --git a/pkg/net/libp2p/bootstrap_test.go b/pkg/net/libp2p/bootstrap_test.go index 259e442eda..8698aecfef 100644 --- a/pkg/net/libp2p/bootstrap_test.go +++ b/pkg/net/libp2p/bootstrap_test.go @@ -10,7 +10,6 @@ import ( "testing" "time" - config "github.com/ipfs/go-ipfs-config" log2 "github.com/ipfs/go-log/v2" "github.com/libp2p/go-libp2p/core/host" "github.com/libp2p/go-libp2p/core/network" @@ -28,24 +27,21 @@ func TestMultipleAddrsPerPeer(t *testing.T) { t.Fatal(err) } - addr := fmt.Sprintf("/ip4/127.0.0.1/tcp/5001/ipfs/%s", pid.String()) - bsp1, err := config.ParseBootstrapPeers([]string{addr}) + addr1, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/tcp/5001/p2p/%s", pid.String())) if err != nil { t.Fatal(err) } - - addr = fmt.Sprintf("/ip4/127.0.0.1/udp/5002/utp/ipfs/%s", pid.String()) - bsp2, err := config.ParseBootstrapPeers([]string{addr}) + addr2, err := ma.NewMultiaddr(fmt.Sprintf("/ip4/127.0.0.1/udp/5002/quic-v1/p2p/%s", pid.String())) if err != nil { t.Fatal(err) } - bsp1Addr, err := peer.AddrInfoFromP2pAddr(bsp1[0].Multiaddr()) + bsp1Addr, err := peer.AddrInfoFromP2pAddr(addr1) if err != nil { t.Fatal(err) } - bsp2Addr, err := peer.AddrInfoFromP2pAddr(bsp2[0].Multiaddr()) + bsp2Addr, err := peer.AddrInfoFromP2pAddr(addr2) if err != nil { t.Fatal(err) } diff --git a/pkg/net/libp2p/channel_manager.go b/pkg/net/libp2p/channel_manager.go index bcb10f7ffb..0cf0141584 100644 --- a/pkg/net/libp2p/channel_manager.go +++ b/pkg/net/libp2p/channel_manager.go @@ -44,7 +44,7 @@ type channelManager struct { retransmissionTicker *retransmission.Ticker forwardersMutex sync.Mutex - forwarders map[string]pubsub.RelayCancelFunc + forwarders map[string]*forwarder topicsMutex sync.Mutex topics map[string]*pubsub.Topic @@ -83,7 +83,7 @@ func newChannelManager( identity: identity, ctx: ctx, retransmissionTicker: retransmissionTicker, - forwarders: make(map[string]pubsub.RelayCancelFunc), + forwarders: make(map[string]*forwarder), topics: make(map[string]*pubsub.Topic), }, nil } @@ -178,57 +178,106 @@ func (cm *channelManager) newChannel(name string) (*channel, error) { return channel, nil } -func (cm *channelManager) newForwarder(name string, ttl time.Duration) error { - cm.forwardersMutex.Lock() - defer cm.forwardersMutex.Unlock() - - if _, ok := cm.forwarders[name]; !ok { - topic, err := cm.getTopic(name) - if err != nil { - return fmt.Errorf( - "could not get topic [%v] handle: [%v]", - name, - err, - ) - } +// forwarder is the lifecycle handle of one channel message relay. There is at +// most one live forwarder per channel name; requesting a forwarder for a name +// that already has one returns the existing handle. +type forwarder struct { + name string + relayCancel pubsub.RelayCancelFunc + manager *channelManager - cancelFn, err := topic.Relay() - if err != nil { - return fmt.Errorf( - "could not enable relay for topic [%v]: [%v]", - name, - err, - ) - } + stopOnce sync.Once + done chan struct{} +} - go func() { - ctx, cancelCtx := context.WithTimeout(cm.ctx, ttl) - defer cancelCtx() +// Close implements net.Forwarder. It is idempotent. +func (f *forwarder) Close() { + f.stop() +} - <-ctx.Done() - cm.shutdownForwarder(name) - }() +// Done implements net.Forwarder. +func (f *forwarder) Done() <-chan struct{} { + return f.done +} - cm.forwarders[name] = cancelFn - } +// stop cancels the pubsub relay, closes the done channel, and removes the +// forwarder from the manager exactly once. +func (f *forwarder) stop() { + f.stopOnce.Do(func() { + logger.Infof( + "shutting down message forwarder for channel: [%v]", + f.name, + ) - return nil + f.relayCancel() + close(f.done) + f.manager.removeForwarder(f.name, f) + }) } -func (cm *channelManager) shutdownForwarder(name string) { +func (cm *channelManager) newForwarder( + name string, + ttl time.Duration, +) (*forwarder, error) { cm.forwardersMutex.Lock() defer cm.forwardersMutex.Unlock() - logger.Infof("shutting down message forwarder for channel: [%v]", name) + if existing, ok := cm.forwarders[name]; ok { + return existing, nil + } + + topic, err := cm.getTopic(name) + if err != nil { + return nil, fmt.Errorf( + "could not get topic [%v] handle: [%v]", + name, + err, + ) + } - cancelFn, ok := cm.forwarders[name] + relayCancel, err := topic.Relay() + if err != nil { + return nil, fmt.Errorf( + "could not enable relay for topic [%v]: [%v]", + name, + err, + ) + } - if !ok { - return + newForwarder := &forwarder{ + name: name, + relayCancel: relayCancel, + manager: cm, + done: make(chan struct{}), } - cancelFn() - delete(cm.forwarders, name) + // The relay stops on its TTL, on provider shutdown through cm.ctx, or on + // an explicit Close, whichever comes first. + go func() { + ctx, cancelCtx := context.WithTimeout(cm.ctx, ttl) + defer cancelCtx() + + select { + case <-ctx.Done(): + newForwarder.stop() + case <-newForwarder.done: + } + }() + + cm.forwarders[name] = newForwarder + + return newForwarder, nil +} + +// removeForwarder drops the forwarder from the manager if it is still the one +// registered under its name. +func (cm *channelManager) removeForwarder(name string, f *forwarder) { + cm.forwardersMutex.Lock() + defer cm.forwardersMutex.Unlock() + + if cm.forwarders[name] == f { + delete(cm.forwarders, name) + } } func (cm *channelManager) getTopic(name string) (*pubsub.Topic, error) { diff --git a/pkg/net/libp2p/channel_test.go b/pkg/net/libp2p/channel_test.go index 116c5da73d..e7be61fb53 100644 --- a/pkg/net/libp2p/channel_test.go +++ b/pkg/net/libp2p/channel_test.go @@ -8,6 +8,7 @@ import ( "sort" "strings" "sync" + "sync/atomic" "testing" "time" @@ -112,12 +113,19 @@ func TestUnregisterHandler(t *testing.T) { // Handlers are fired asynchronously; wait for them time.Sleep(500 * time.Millisecond) - sort.Strings(handlersFired) - if !reflect.DeepEqual(test.handlersFired, handlersFired) { + // Read under the same mutex the handlers write under, taking a + // snapshot so the comparison cannot race a still-firing handler. + handlersFiredMutex.Lock() + firedSnapshot := make([]string, len(handlersFired)) + copy(firedSnapshot, handlersFired) + handlersFiredMutex.Unlock() + + sort.Strings(firedSnapshot) + if !reflect.DeepEqual(test.handlersFired, firedSnapshot) { t.Errorf( "Unexpected handlers fired\nExpected: %v\nActual: %v\n", test.handlersFired, - handlersFired, + firedSnapshot, ) } }) @@ -129,13 +137,13 @@ func TestUnregisterWhenHandling(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - receivedCount := 0 - stopAt := 90 + // receivedCount is written by the Recv handler goroutine and read by the + // main goroutine, so it is accessed atomically to avoid a data race. + var receivedCount atomic.Int64 + stopAt := int64(90) channel.Recv(ctx, func(msg net.Message) { - receivedCount++ - - if receivedCount == stopAt { + if receivedCount.Add(1) == stopAt { cancel() } }) @@ -148,8 +156,8 @@ func TestUnregisterWhenHandling(t *testing.T) { time.Sleep(500 * time.Millisecond) - if receivedCount != stopAt { - t.Fatalf("unexpected number of received messages: [%v]", receivedCount) + if final := receivedCount.Load(); final != stopAt { + t.Fatalf("unexpected number of received messages: [%v]", final) } } @@ -159,10 +167,12 @@ func TestUnregisterWhenHandlingBlocked(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - receivedCount := 0 + // receivedCount is written by the Recv handler goroutine and read by the + // main goroutine, so it is accessed atomically to avoid a data race. + var receivedCount atomic.Int64 channel.Recv(ctx, func(msg net.Message) { - receivedCount++ + receivedCount.Add(1) receiver <- msg // there is no receiver, this call will block }) @@ -174,10 +184,15 @@ func TestUnregisterWhenHandlingBlocked(t *testing.T) { cancel() time.Sleep(100 * time.Millisecond) - if receivedCount != 1 { + if final := receivedCount.Load(); final != 1 { t.Fatalf("expected just one Recv call") } - if len(channel.messageHandlers) != 0 { + // removeHandler mutates messageHandlers under messageHandlersMutex from the + // handler lifecycle goroutine, so read its length under the same lock. + channel.messageHandlersMutex.Lock() + remainingHandlers := len(channel.messageHandlers) + channel.messageHandlersMutex.Unlock() + if remainingHandlers != 0 { t.Fatalf("expected the handler to be unregistered") } } diff --git a/pkg/net/libp2p/fuzz_test.go b/pkg/net/libp2p/fuzz_test.go new file mode 100644 index 0000000000..46c5e8a8a8 --- /dev/null +++ b/pkg/net/libp2p/fuzz_test.go @@ -0,0 +1,42 @@ +package libp2p + +// This fuzz target exercises identity.Unmarshal, which parses the +// message.Sender bytes of broadcast-channel envelopes. The parse runs +// BEFORE sender verification in processContainerMessage, so the input +// is fully attacker-controlled: proto unmarshaling, public-key +// unmarshaling, and peer-ID derivation all see raw peer bytes. The +// invariant under test is that Unmarshal never panics on arbitrary +// input: malformed bytes must return an error, not crash the process. + +import ( + "crypto/rand" + "testing" + + libp2pcrypto "github.com/libp2p/go-libp2p/core/crypto" +) + +func FuzzIdentityUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x0a, 0x01}) + + // A well-formed identity as a seed so coverage starts past the + // proto envelope and into key/peer-ID parsing. + privateKey, _, err := libp2pcrypto.GenerateSecp256k1Key(rand.Reader) + if err != nil { + f.Fatal(err) + } + validIdentity, err := createIdentity(privateKey) + if err != nil { + f.Fatal(err) + } + validBytes, err := validIdentity.Marshal() + if err != nil { + f.Fatal(err) + } + f.Add(validBytes) + + f.Fuzz(func(t *testing.T, data []byte) { + // Must never panic on arbitrary input; an error return is correct. + _ = (&identity{}).Unmarshal(data) + }) +} diff --git a/pkg/net/libp2p/libp2p.go b/pkg/net/libp2p/libp2p.go index 0d8339df86..177a91be8b 100644 --- a/pkg/net/libp2p/libp2p.go +++ b/pkg/net/libp2p/libp2p.go @@ -19,8 +19,6 @@ import ( dstore "github.com/ipfs/go-datastore" dssync "github.com/ipfs/go-datastore/sync" - //lint:ignore SA1019 package deprecated, but we rely on its interface - addrutil "github.com/libp2p/go-addr-util" "github.com/libp2p/go-libp2p" dht "github.com/libp2p/go-libp2p-kad-dht" libp2pcrypto "github.com/libp2p/go-libp2p/core/crypto" @@ -34,6 +32,7 @@ import ( "github.com/libp2p/go-libp2p/p2p/protocol/ping" ma "github.com/multiformats/go-multiaddr" + manet "github.com/multiformats/go-multiaddr/net" ) var logger = log.Logger("keep-libp2p") @@ -132,21 +131,24 @@ func (p *provider) CreateTransportIdentifier(operatorPublicKey *operator.PublicK return peer.IDFromPublicKey(networkPublicKey) } -func (p *provider) BroadcastChannelForwarderFor(name string) { +func (p *provider) BroadcastChannelForwarderFor(name string) (net.Forwarder, error) { if p.disseminationTime == 0 { - return + return net.NoopForwarder(), nil } logger.Infof("starting message forwarder for channel [%v]", name) timeout := time.Duration(p.disseminationTime) * time.Second - if err := p.broadcastChannelManager.newForwarder(name, timeout); err != nil { - logger.Warnf( - "could not create message forwarder for channel [%v]: [%v]", + forwarder, err := p.broadcastChannelManager.newForwarder(name, timeout) + if err != nil { + return nil, fmt.Errorf( + "could not create message forwarder for channel [%v]: [%w]", name, err, ) } + + return forwarder, nil } type connectionManager struct { @@ -480,12 +482,15 @@ func discoverAndListen( } func getListenAddrs(port int) ([]ma.Multiaddr, error) { - ia, err := addrutil.InterfaceAddresses() + maddrs, err := manet.InterfaceMultiaddrs() if err != nil { return nil, err } - addrs := make([]ma.Multiaddr, 0) - for _, addr := range ia { + addrs := make([]ma.Multiaddr, 0, len(maddrs)) + for _, addr := range maddrs { + if manet.IsIP6LinkLocal(addr) { + continue + } portAddr, err := ma.NewMultiaddr(fmt.Sprintf("/tcp/%d", port)) if err != nil { return nil, err diff --git a/pkg/net/local/broadcast_channel_test.go b/pkg/net/local/broadcast_channel_test.go index 33290c4aa4..89d30e6adf 100644 --- a/pkg/net/local/broadcast_channel_test.go +++ b/pkg/net/local/broadcast_channel_test.go @@ -5,6 +5,7 @@ import ( "reflect" "sort" "sync" + "sync/atomic" "testing" "time" @@ -115,12 +116,19 @@ func TestUnregisterHandler(t *testing.T) { // Handlers are fired asynchronously; wait for them time.Sleep(500 * time.Millisecond) - sort.Strings(handlersFired) - if !reflect.DeepEqual(test.handlersFired, handlersFired) { + // Read under the same mutex the handlers write under, taking a + // snapshot so the comparison cannot race a still-firing handler. + handlersFiredMutex.Lock() + firedSnapshot := make([]string, len(handlersFired)) + copy(firedSnapshot, handlersFired) + handlersFiredMutex.Unlock() + + sort.Strings(firedSnapshot) + if !reflect.DeepEqual(test.handlersFired, firedSnapshot) { t.Errorf( "Unexpected handlers fired\nExpected: %v\nActual: %v\n", test.handlersFired, - handlersFired, + firedSnapshot, ) } }) @@ -135,13 +143,13 @@ func TestUnregisterWhenHandling(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) - receivedCount := 0 - stopAt := 90 + // receivedCount is written by the Recv handler goroutine and read by the + // main goroutine, so it is accessed atomically to avoid a data race. + var receivedCount atomic.Int64 + stopAt := int64(90) channel.Recv(ctx, func(msg net.Message) { - receivedCount++ - - if receivedCount == stopAt { + if receivedCount.Add(1) == stopAt { cancel() } }) @@ -154,8 +162,8 @@ func TestUnregisterWhenHandling(t *testing.T) { time.Sleep(500 * time.Millisecond) - if receivedCount != stopAt { - t.Fatalf("received more than expected: [%v]", receivedCount) + if final := receivedCount.Load(); final != stopAt { + t.Fatalf("received more than expected: [%v]", final) } } func TestSendAndDeliver(t *testing.T) { diff --git a/pkg/net/local/local.go b/pkg/net/local/local.go index 50be939943..48f271f205 100644 --- a/pkg/net/local/local.go +++ b/pkg/net/local/local.go @@ -55,8 +55,12 @@ func (lp *localProvider) CreateTransportIdentifier( return createLocalIdentifier(operatorPublicKey) } -func (lp *localProvider) BroadcastChannelForwarderFor(name string) { - //no-op +func (lp *localProvider) BroadcastChannelForwarderFor(name string) ( + net.Forwarder, + error, +) { + // The local provider does no relaying; the handle is already done. + return net.NoopForwarder(), nil } // Connect returns a local instance of a net provider that does not go over the diff --git a/pkg/net/net.go b/pkg/net/net.go index cc728d73c3..e25e144315 100644 --- a/pkg/net/net.go +++ b/pkg/net/net.go @@ -71,8 +71,40 @@ type Provider interface { operatorPublicKey *operator.PublicKey, ) (TransportIdentifier, error) - // BroadcastChannelForwarderFor creates a message relay for given channel name. - BroadcastChannelForwarderFor(name string) + // BroadcastChannelForwarderFor creates a message relay for given channel + // name and returns its lifecycle handle. Implementations that run no + // relay — a disabled dissemination time or a provider with no relaying — + // return an already-done no-op handle and no error. + BroadcastChannelForwarderFor(name string) (Forwarder, error) +} + +// Forwarder is the lifecycle handle of a broadcast channel message relay. The +// relay stops on its TTL, on provider shutdown, or on an explicit Close; +// whichever comes first closes the Done channel. +type Forwarder interface { + // Close stops the forwarder. It is idempotent. + Close() + // Done returns a channel that is closed when the forwarder stopped. + Done() <-chan struct{} +} + +// noopForwarderDone is the shared already-closed Done channel of every no-op +// forwarder. +var noopForwarderDone = func() chan struct{} { + done := make(chan struct{}) + close(done) + return done +}() + +type noopForwarder struct{} + +func (noopForwarder) Close() {} +func (noopForwarder) Done() <-chan struct{} { return noopForwarderDone } + +// NoopForwarder returns an already-done Forwarder for providers and +// configurations that run no message relay. +func NoopForwarder() Forwarder { + return noopForwarder{} } // ConnectionManager is an interface which exposes peers a client is connected diff --git a/pkg/net/retransmission/retransmission_test.go b/pkg/net/retransmission/retransmission_test.go index c9a1664793..047e659fe4 100644 --- a/pkg/net/retransmission/retransmission_test.go +++ b/pkg/net/retransmission/retransmission_test.go @@ -32,8 +32,18 @@ func TestRetransmitExpectedNumberOfTimes(t *testing.T) { <-ctx.Done() - if atomic.LoadUint64(&retransmissionsCount) != 10 { - t.Errorf("expected [10] retransmissions, has [%v]", retransmissionsCount) + // Each retransmission runs in its own goroutine spawned by the tick + // handler, so the last one may still be in flight when the context is + // done. Wait for the expected count before asserting on the final value. + deadline := time.Now().Add(5 * time.Second) + for atomic.LoadUint64(&retransmissionsCount) < 10 && + time.Now().Before(deadline) { + time.Sleep(time.Millisecond) + } + + got := atomic.LoadUint64(&retransmissionsCount) + if got != 10 { + t.Errorf("expected [10] retransmissions, has [%v]", got) } } diff --git a/pkg/net/retransmission/ticker.go b/pkg/net/retransmission/ticker.go index a9e3e8e802..1b4209ed34 100644 --- a/pkg/net/retransmission/ticker.go +++ b/pkg/net/retransmission/ticker.go @@ -75,9 +75,9 @@ func (t *Ticker) start() { t.handlersMutex.Unlock() } - for ctx := range t.handlers { - delete(t.handlers, ctx) - } + t.handlersMutex.Lock() + clear(t.handlers) + t.handlersMutex.Unlock() } func (t *Ticker) onTick(ctx context.Context, fn func()) { diff --git a/pkg/net/retransmission/ticker_test.go b/pkg/net/retransmission/ticker_test.go index b41a8e61e9..f150517831 100644 --- a/pkg/net/retransmission/ticker_test.go +++ b/pkg/net/retransmission/ticker_test.go @@ -2,6 +2,7 @@ package retransmission import ( "context" + "sync/atomic" "testing" "time" ) @@ -13,15 +14,15 @@ func TestOnTick(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - tickCount := 0 - ticker.onTick(ctx, func() { tickCount++ }) + var tickCount uint64 + ticker.onTick(ctx, func() { atomic.AddUint64(&tickCount, 1) }) ticks <- 1 ticks <- 2 - time.Sleep(10 * time.Millisecond) + waitForCounter(t, &tickCount, 2) - if tickCount != 2 { - t.Errorf("expected [2] executions of handler, had [%v]", tickCount) + if got := atomic.LoadUint64(&tickCount); got != 2 { + t.Errorf("expected [2] executions of handler, had [%v]", got) } } @@ -32,20 +33,21 @@ func TestOnTickSameContext(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - tickCount1 := 0 - tickCount2 := 0 - ticker.onTick(ctx, func() { tickCount1++ }) - ticker.onTick(ctx, func() { tickCount2++ }) + var tickCount1 uint64 + var tickCount2 uint64 + ticker.onTick(ctx, func() { atomic.AddUint64(&tickCount1, 1) }) + ticker.onTick(ctx, func() { atomic.AddUint64(&tickCount2, 1) }) ticks <- 1 ticks <- 2 - time.Sleep(10 * time.Millisecond) + waitForCounter(t, &tickCount1, 2) + waitForCounter(t, &tickCount2, 2) - if tickCount1 != 2 { - t.Errorf("expected [2] executions of handler, had [%v]", tickCount1) + if got := atomic.LoadUint64(&tickCount1); got != 2 { + t.Errorf("expected [2] executions of handler, had [%v]", got) } - if tickCount2 != 2 { - t.Errorf("expected [2] executions of handler, had [%v]", tickCount2) + if got := atomic.LoadUint64(&tickCount2); got != 2 { + t.Errorf("expected [2] executions of handler, had [%v]", got) } } @@ -55,13 +57,15 @@ func TestOnTickTimeTicker(t *testing.T) { ticker := NewTimeTicker(ctx, 10*time.Millisecond) - tickCount := 0 - ticker.onTick(ctx, func() { tickCount++ }) + var tickCount uint64 + ticker.onTick(ctx, func() { atomic.AddUint64(&tickCount, 1) }) <-ctx.Done() - if tickCount != 10 { - t.Errorf("expected [10] executions of handler, had [%v]", tickCount) + waitForCounter(t, &tickCount, 10) + + if got := atomic.LoadUint64(&tickCount); got != 10 { + t.Errorf("expected [10] executions of handler, had [%v]", got) } } @@ -74,11 +78,11 @@ func TestUnregisterHandler(t *testing.T) { ctx2, cancel2 := context.WithTimeout(context.Background(), 100*time.Millisecond) defer cancel2() - tickCount1 := 0 - ticker.onTick(ctx1, func() { tickCount1++ }) + var tickCount1 uint64 + ticker.onTick(ctx1, func() { atomic.AddUint64(&tickCount1, 1) }) - tickCount2 := 0 - ticker.onTick(ctx2, func() { tickCount2++ }) + var tickCount2 uint64 + ticker.onTick(ctx2, func() { atomic.AddUint64(&tickCount2, 1) }) ticks <- 1 ticks <- 2 @@ -86,13 +90,13 @@ func TestUnregisterHandler(t *testing.T) { ticks <- 3 <-ctx2.Done() ticks <- 4 - time.Sleep(10 * time.Millisecond) + waitForCounter(t, &tickCount2, 3) - if tickCount1 != 2 { - t.Errorf("expected [2] executions of the first handler, had [%v]", tickCount1) + if got := atomic.LoadUint64(&tickCount1); got != 2 { + t.Errorf("expected [2] executions of the first handler, had [%v]", got) } - if tickCount2 != 3 { - t.Errorf("expected [3] executions of the second handler, had [%v]", tickCount2) + if got := atomic.LoadUint64(&tickCount2); got != 3 { + t.Errorf("expected [3] executions of the second handler, had [%v]", got) } } @@ -103,21 +107,23 @@ func TestUnregisterHandlerSameContext(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond) defer cancel() - tickCount1 := 0 - ticker.onTick(ctx, func() { tickCount1++ }) + var tickCount1 uint64 + ticker.onTick(ctx, func() { atomic.AddUint64(&tickCount1, 1) }) - tickCount2 := 0 - ticker.onTick(ctx, func() { tickCount2++ }) + var tickCount2 uint64 + ticker.onTick(ctx, func() { atomic.AddUint64(&tickCount2, 1) }) ticks <- 1 ticks <- 2 + waitForCounter(t, &tickCount1, 2) + waitForCounter(t, &tickCount2, 2) <-ctx.Done() - if tickCount1 != 2 { - t.Errorf("expected [2] executions of the first handler, had [%v]", tickCount1) + if got := atomic.LoadUint64(&tickCount1); got != 2 { + t.Errorf("expected [2] executions of the first handler, had [%v]", got) } - if tickCount2 != 2 { - t.Errorf("expected [2] executions of the second handler, had [%v]", tickCount2) + if got := atomic.LoadUint64(&tickCount2); got != 2 { + t.Errorf("expected [2] executions of the second handler, had [%v]", got) } } @@ -131,14 +137,8 @@ func TestCloseTicker(t *testing.T) { ticker.onTick(ctx, func() {}) close(ticks) - time.Sleep(10 * time.Millisecond) - if len(ticker.handlers) != 0 { - t.Errorf( - "all handlers should be unregistered, still has [%v]", - len(ticker.handlers), - ) - } + waitForHandlersUnregistered(t, ticker) } func TestCloseTimeTicker(t *testing.T) { @@ -151,12 +151,44 @@ func TestCloseTimeTicker(t *testing.T) { <-ctx.Done() - time.Sleep(10 * time.Millisecond) + waitForHandlersUnregistered(t, ticker) +} + +// waitForCounter blocks until the atomic counter reaches at least the expected +// value, failing the test on timeout. Handlers run in the ticker's goroutine, +// so the test must await the counter rather than sleep and read it without +// synchronization. +func waitForCounter(t *testing.T, counter *uint64, expected uint64) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + if atomic.LoadUint64(counter) >= expected { + return + } + time.Sleep(time.Millisecond) + } + t.Fatalf( + "timed out waiting for counter to reach [%v], has [%v]", + expected, + atomic.LoadUint64(counter), + ) +} - if len(ticker.handlers) != 0 { - t.Errorf( - "all handlers should be unregistered, still has [%v]", - len(ticker.handlers), - ) +// waitForHandlersUnregistered blocks until the ticker has no onTick handlers +// registered. The shutdown cleanup in the ticker's start goroutine runs +// asynchronously after the ticks channel closes, so the postcondition must be +// awaited rather than asserted after a fixed sleep. +func waitForHandlersUnregistered(t *testing.T, ticker *Ticker) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + ticker.handlersMutex.Lock() + remaining := len(ticker.handlers) + ticker.handlersMutex.Unlock() + if remaining == 0 { + return + } + time.Sleep(time.Millisecond) } + t.Fatal("timed out waiting for handlers to be unregistered") } diff --git a/pkg/net/security/handshake/fuzz_test.go b/pkg/net/security/handshake/fuzz_test.go new file mode 100644 index 0000000000..22fea79ba9 --- /dev/null +++ b/pkg/net/security/handshake/fuzz_test.go @@ -0,0 +1,35 @@ +package handshake + +// These fuzz targets exercise the handshake message unmarshalers, which parse +// bytes received from untrusted peers during the connection handshake. The +// invariant under test is that Unmarshal never panics on arbitrary input: +// malformed bytes must return an error, not crash the process. + +import "testing" + +func FuzzAct1MessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + // Must never panic on arbitrary input; an error return is correct. + _ = (&Act1Message{}).Unmarshal(data) + }) +} + +func FuzzAct2MessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + // Must never panic on arbitrary input; an error return is correct. + _ = (&Act2Message{}).Unmarshal(data) + }) +} + +func FuzzAct3MessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + // Must never panic on arbitrary input; an error return is correct. + _ = (&Act3Message{}).Unmarshal(data) + }) +} diff --git a/pkg/net/watchtower/watchtower_test.go b/pkg/net/watchtower/watchtower_test.go index 3a6719f48e..36c53ac44e 100644 --- a/pkg/net/watchtower/watchtower_test.go +++ b/pkg/net/watchtower/watchtower_test.go @@ -3,6 +3,7 @@ package watchtower import ( "context" "fmt" + "sync" "testing" "time" @@ -68,10 +69,17 @@ func newMockFirewall() *mockFirewall { } type mockFirewall struct { - meetsCriteria map[uint64]bool + // meetsCriteria is read by the Guard's asynchronous checkFirewallRules + // goroutine (via Validate) while the test updates it (via updatePeer), so + // access is guarded by a mutex. + meetsCriteriaMutex sync.Mutex + meetsCriteria map[uint64]bool } func (mf *mockFirewall) Validate(remotePeerPublicKey *operator.PublicKey) error { + mf.meetsCriteriaMutex.Lock() + defer mf.meetsCriteriaMutex.Unlock() + if !mf.meetsCriteria[remotePeerPublicKey.X.Uint64()] { return fmt.Errorf("remote peer does not meet firewall criteria") } @@ -82,6 +90,9 @@ func (mf *mockFirewall) updatePeer( remotePeerOperatorPublicKey *operator.PublicKey, meetsCriteria bool, ) { + mf.meetsCriteriaMutex.Lock() + defer mf.meetsCriteriaMutex.Unlock() + x := remotePeerOperatorPublicKey.X.Uint64() mf.meetsCriteria[x] = meetsCriteria } diff --git a/pkg/protocol/announcer/announcer.go b/pkg/protocol/announcer/announcer.go index 54860aa0ee..93bb46d76a 100644 --- a/pkg/protocol/announcer/announcer.go +++ b/pkg/protocol/announcer/announcer.go @@ -7,6 +7,7 @@ import ( "context" "fmt" "sort" + "strings" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/announcer/gen/pb" @@ -63,12 +64,156 @@ func (am *announcementMessage) Type() string { return "protocol_announcer/announcement_message" } +// SessionIDFormat classifies the wire format of an announcement session ID +// without exposing the raw identifier. It is used to distinguish legacy peers +// from hardened (security-v2) peers during a coordinated cutover. +type SessionIDFormat uint8 + +const ( + // SessionIDFormatUnknown denotes a session ID that matches neither the + // legacy nor a hardened format. + SessionIDFormatUnknown SessionIDFormat = iota + // SessionIDFormatLegacy denotes the pre-hardening form: one lowercase + // hexadecimal seed/message component and one unsigned decimal attempt + // component, separated by a single hyphen (e.g. "abc123-4"). + SessionIDFormatLegacy + // SessionIDFormatHardenedDKG denotes the hardened tECDSA DKG form + // "dkg--<16 hex digits>". + SessionIDFormatHardenedDKG + // SessionIDFormatHardenedSigning denotes the hardened tECDSA signing form + // "signing--<16 hex digits>-<16 hex digits>". + SessionIDFormatHardenedSigning +) + +// String returns a stable, log-safe label for the session ID format. +func (f SessionIDFormat) String() string { + switch f { + case SessionIDFormatLegacy: + return "legacy" + case SessionIDFormatHardenedDKG: + return "hardened_dkg" + case SessionIDFormatHardenedSigning: + return "hardened_signing" + default: + return "unknown" + } +} + +// IsHardened reports whether the format is one of the hardened (security-v2) +// forms. +func (f SessionIDFormat) IsHardened() bool { + return f == SessionIDFormatHardenedDKG || f == SessionIDFormatHardenedSigning +} + +// ClassifySessionIDFormat classifies a session ID into one of the known formats +// without retaining or exposing the raw identifier. The classification is +// purely structural and mirrors the exact formats produced by the tBTC DKG and +// signing loops: +// +// - hardened DKG: "dkg--<16 hex digits>" +// - hardened signing: "signing--<16 hex digits>-<16 hex digits>" +// - legacy: "-" +// - otherwise: unknown +func ClassifySessionIDFormat(sessionID string) SessionIDFormat { + parts := strings.Split(sessionID, "-") + + switch { + case len(parts) == 3 && + parts[0] == "dkg" && + isLowerHex(parts[1]) && + isFixedWidthLowerHex(parts[2], 16): + return SessionIDFormatHardenedDKG + case len(parts) == 4 && + parts[0] == "signing" && + isLowerHex(parts[1]) && + isFixedWidthLowerHex(parts[2], 16) && + isFixedWidthLowerHex(parts[3], 16): + return SessionIDFormatHardenedSigning + case len(parts) == 2 && + isLowerHex(parts[0]) && + isDecimal(parts[1]): + return SessionIDFormatLegacy + default: + return SessionIDFormatUnknown + } +} + +// IsCrossFormatMismatch reports whether two session ID formats represent a +// legacy-versus-hardened mismatch (as opposed to two differing formats on the +// same side of the cutover). Only cross-format mismatches indicate a peer on +// the opposite side of the cutover. +func IsCrossFormatMismatch(a, b SessionIDFormat) bool { + return (a == SessionIDFormatLegacy && b.IsHardened()) || + (b == SessionIDFormatLegacy && a.IsHardened()) +} + +// isLowerHex reports whether s is a non-empty string of lowercase hexadecimal +// digits. +func isLowerHex(s string) bool { + if len(s) == 0 { + return false + } + for _, c := range s { + if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) { + return false + } + } + return true +} + +// isFixedWidthLowerHex reports whether s is exactly width lowercase hexadecimal +// digits. +func isFixedWidthLowerHex(s string, width int) bool { + return len(s) == width && isLowerHex(s) +} + +// isDecimal reports whether s is a non-empty string of decimal digits. +func isDecimal(s string) bool { + if len(s) == 0 { + return false + } + for _, c := range s { + if c < '0' || c > '9' { + return false + } + } + return true +} + +// SessionMismatchObserver is invoked once per membership-valid, protocol-matched +// sender per Announce call whose announced session ID differs from the local +// session ID. It receives only the protocol ID, the sender's group member +// index, and the classified expected/observed formats — never the raw session +// IDs — so it is safe to log or aggregate. expectedFormat is the format of the +// local node's own session ID; observedFormat is the sender's. +type SessionMismatchObserver func( + protocolID string, + sender group.MemberIndex, + expectedFormat SessionIDFormat, + observedFormat SessionIDFormat, +) + +// Option configures optional Announcer behavior. It is source-compatible: the +// existing three-argument New calls continue to compile unchanged. +type Option func(*Announcer) + +// WithSessionMismatchObserver installs an observer that is invoked when a +// membership-valid, protocol-matched announcement carries a session ID that +// differs from the local session ID. The observer is called at most once per +// sender per Announce call. A nil observer is equivalent to not setting one. +func WithSessionMismatchObserver(observer SessionMismatchObserver) Option { + return func(a *Announcer) { + a.sessionMismatchObserver = observer + } +} + // Announcer is an implementation of the protocol announcer that performs the // readiness announcement over the provided broadcast channel. type Announcer struct { - protocolID string - broadcastChannel net.BroadcastChannel - membershipValidator *group.MembershipValidator + protocolID string + broadcastChannel net.BroadcastChannel + membershipValidator *group.MembershipValidator + sessionMismatchObserver SessionMismatchObserver } // RegisterUnmarshaller initializes the given broadcast channel to be able to @@ -87,12 +232,19 @@ func New( protocolID string, broadcastChannel net.BroadcastChannel, membershipValidator *group.MembershipValidator, + options ...Option, ) *Announcer { - return &Announcer{ + announcer := &Announcer{ protocolID: protocolID, broadcastChannel: broadcastChannel, membershipValidator: membershipValidator, } + + for _, option := range options { + option(announcer) + } + + return announcer } // Announce sends the member's readiness announcement for the given protocol @@ -127,6 +279,10 @@ func (a *Announcer) Announce( // Mark itself as ready. readyMembersIndexesSet[memberIndex] = true + // Tracks senders already reported to the session mismatch observer during + // this Announce call so each mismatching sender is counted at most once. + mismatchObservedSenders := make(map[group.MemberIndex]bool) + loop: for { select { @@ -152,6 +308,23 @@ loop: } if announcement.sessionID != sessionID { + // The sender is a valid group member announcing for this + // protocol but with a different session ID. During a + // coordinated cutover this is how a peer on the opposite side + // of the boundary is observed. Report it to the optional + // observer, once per sender per call, before discarding the + // announcement. Only structural formats are exposed, never the + // raw session IDs. + if a.sessionMismatchObserver != nil && + !mismatchObservedSenders[announcement.senderID] { + mismatchObservedSenders[announcement.senderID] = true + a.sessionMismatchObserver( + announcement.protocolID, + announcement.senderID, + ClassifySessionIDFormat(sessionID), + ClassifySessionIDFormat(announcement.sessionID), + ) + } continue } diff --git a/pkg/protocol/announcer/announcer_test.go b/pkg/protocol/announcer/announcer_test.go index b96b69f8da..b20a21fdd2 100644 --- a/pkg/protocol/announcer/announcer_test.go +++ b/pkg/protocol/announcer/announcer_test.go @@ -4,6 +4,7 @@ import ( "context" "math/big" "reflect" + "strings" "sync" "testing" @@ -13,6 +14,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/chain/local_v1" "github.com/keep-network/keep-core/pkg/internal/pbutils" + "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/net/local" "github.com/keep-network/keep-core/pkg/operator" "github.com/keep-network/keep-core/pkg/protocol/group" @@ -228,6 +230,516 @@ func TestAnnouncer(t *testing.T) { } } +func TestClassifySessionIDFormat(t *testing.T) { + tests := map[string]struct { + sessionID string + expected SessionIDFormat + }{ + "hardened dkg": { + sessionID: "dkg-abc123-0000000000000001", + expected: SessionIDFormatHardenedDKG, + }, + "hardened dkg with hex attempt": { + sessionID: "dkg-0-000000000000000a", + expected: SessionIDFormatHardenedDKG, + }, + "hardened signing": { + sessionID: "signing-deadbeef-0000000000000010-0000000000000002", + expected: SessionIDFormatHardenedSigning, + }, + "legacy dkg/signing": { + sessionID: "abc123-5", + expected: SessionIDFormatLegacy, + }, + "legacy zero seed zero attempt": { + sessionID: "0-0", + expected: SessionIDFormatLegacy, + }, + "legacy long hex": { + sessionID: "deadbeef42", + expected: SessionIDFormatUnknown, // single token, no separator + }, + "legacy hex and decimal": { + sessionID: "deadbeef-42", + expected: SessionIDFormatLegacy, + }, + "dkg wrong attempt width": { + sessionID: "dkg-abc-123", + expected: SessionIDFormatUnknown, + }, + "dkg too long attempt": { + sessionID: "dkg-abc123-00000000000000001", + expected: SessionIDFormatUnknown, + }, + "dkg non-hex attempt": { + sessionID: "dkg-abc-000000000000000g", + expected: SessionIDFormatUnknown, + }, + "dkg uppercase seed": { + sessionID: "dkg-ABC-0000000000000001", + expected: SessionIDFormatUnknown, + }, + "signing too few parts": { + sessionID: "signing-abc-0000000000000001", + expected: SessionIDFormatUnknown, + }, + "signing non-hex last part": { + sessionID: "signing-abc-0000000000000001-000000000000000g", + expected: SessionIDFormatUnknown, + }, + "legacy non-decimal attempt": { + sessionID: "abc-xyz", + expected: SessionIDFormatUnknown, + }, + "single token": { + sessionID: "abc123", + expected: SessionIDFormatUnknown, + }, + "empty": { + sessionID: "", + expected: SessionIDFormatUnknown, + }, + "too many parts": { + sessionID: "a-b-c-d-e", + expected: SessionIDFormatUnknown, + }, + "hex prefix rejected": { + sessionID: "0xabc-5", + expected: SessionIDFormatUnknown, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + actual := ClassifySessionIDFormat(test.sessionID) + if actual != test.expected { + t.Errorf( + "unexpected format for [%s]\nexpected: %v\nactual: %v", + test.sessionID, + test.expected, + actual, + ) + } + }) + } +} + +func TestIsCrossFormatMismatch(t *testing.T) { + tests := map[string]struct { + a SessionIDFormat + b SessionIDFormat + expected bool + }{ + "legacy vs hardened dkg": {SessionIDFormatLegacy, SessionIDFormatHardenedDKG, true}, + "hardened dkg vs legacy": {SessionIDFormatHardenedDKG, SessionIDFormatLegacy, true}, + "legacy vs hardened signing": {SessionIDFormatLegacy, SessionIDFormatHardenedSigning, true}, + "legacy vs legacy": {SessionIDFormatLegacy, SessionIDFormatLegacy, false}, + "hardened dkg vs hardened sign": {SessionIDFormatHardenedDKG, SessionIDFormatHardenedSigning, false}, + "hardened dkg vs hardened dkg": {SessionIDFormatHardenedDKG, SessionIDFormatHardenedDKG, false}, + "legacy vs unknown": {SessionIDFormatLegacy, SessionIDFormatUnknown, false}, + "unknown vs hardened dkg": {SessionIDFormatUnknown, SessionIDFormatHardenedDKG, false}, + "unknown vs unknown": {SessionIDFormatUnknown, SessionIDFormatUnknown, false}, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + if actual := IsCrossFormatMismatch(test.a, test.b); actual != test.expected { + t.Errorf( + "unexpected cross-format result for (%v, %v): got %v, want %v", + test.a, test.b, actual, test.expected, + ) + } + }) + } +} + +type recordedMismatch struct { + protocolID string + sender group.MemberIndex + expected SessionIDFormat + observed SessionIDFormat +} + +type mismatchRecorder struct { + mu sync.Mutex + records []recordedMismatch +} + +func (r *mismatchRecorder) observer() SessionMismatchObserver { + return func( + protocolID string, + sender group.MemberIndex, + expected SessionIDFormat, + observed SessionIDFormat, + ) { + r.mu.Lock() + defer r.mu.Unlock() + r.records = append(r.records, recordedMismatch{ + protocolID: protocolID, + sender: sender, + expected: expected, + observed: observed, + }) + } +} + +func (r *mismatchRecorder) snapshot() []recordedMismatch { + r.mu.Lock() + defer r.mu.Unlock() + out := make([]recordedMismatch, len(r.records)) + copy(out, r.records) + return out +} + +// newMismatchObserverFixture builds a five-member group whose members all share +// a single operator address (so membership is valid for any in-range index) and +// returns a local network provider bound to that operator's key together with a +// matching membership validator. +func newMismatchObserverFixture(t *testing.T) (net.Provider, *group.MembershipValidator) { + t.Helper() + + const groupSize = 5 + const honestThreshold = 3 + + privateKey, publicKey, err := operator.GenerateKeyPair(local_v1.DefaultCurve) + if err != nil { + t.Fatal(err) + } + + localChain := local_v1.ConnectWithKey(groupSize, honestThreshold, privateKey) + + operatorAddress, err := localChain.Signing().PublicKeyToAddress(publicKey) + if err != nil { + t.Fatal(err) + } + + operators := make([]chain.Address, groupSize) + for i := range operators { + operators[i] = operatorAddress + } + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + operators, + localChain.Signing(), + ) + + return local.ConnectWithKey(publicKey), membershipValidator +} + +func sendAnnouncement( + t *testing.T, + ctx context.Context, + channel net.BroadcastChannel, + senderID group.MemberIndex, + protocolID string, + sessionID string, +) { + t.Helper() + + err := channel.Send(ctx, &announcementMessage{ + senderID: senderID, + protocolID: protocolID, + sessionID: sessionID, + }) + if err != nil { + t.Fatalf("cannot send crafted announcement: [%v]", err) + } +} + +// TestAnnouncer_SessionMismatchObserver verifies that the observer fires once +// per membership-valid, protocol-matched sender whose session ID differs from +// the local one, that identical session IDs do not fire it, and that the +// classified formats are correct (cross-format vs same-format). +func TestAnnouncer_SessionMismatchObserver(t *testing.T) { + const protocolID = "announcer-mismatch-observer-protocol" + const channelName = "announcer-mismatch-observer" + + provider, membershipValidator := newMismatchObserverFixture(t) + + receiverChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(receiverChannel) + + senderChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(senderChannel) + + recorder := &mismatchRecorder{} + receiver := New( + protocolID, + receiverChannel, + membershipValidator, + WithSessionMismatchObserver(recorder.observer()), + ) + + // The local node runs the hardened DKG session ID. + const receiverSessionID = "dkg-abc123-0000000000000001" + + ctx, cancel := context.WithTimeout( + context.Background(), + 10*local.RetransmissionTick, + ) + defer cancel() + + // Crafted announcements are scheduled with retransmission, so the receiver + // catches them once its Recv handler is registered inside Announce. + // member 2: legacy session ID -> cross-format mismatch + sendAnnouncement(t, ctx, senderChannel, 2, protocolID, "abc123-5") + // member 2 again with a different legacy ID -> deduplicated (same sender) + sendAnnouncement(t, ctx, senderChannel, 2, protocolID, "abc123-6") + // member 3: a different hardened DKG session ID -> same-format mismatch + sendAnnouncement(t, ctx, senderChannel, 3, protocolID, "dkg-def456-0000000000000002") + // member 4: identical session ID -> not a mismatch, must not fire + sendAnnouncement(t, ctx, senderChannel, 4, protocolID, receiverSessionID) + + if _, err := receiver.Announce(ctx, 1, receiverSessionID); err != nil { + t.Fatal(err) + } + + records := recorder.snapshot() + + // Deduplicate to unique senders (already guaranteed by the announcer, but + // assert it explicitly). + bySender := make(map[group.MemberIndex]recordedMismatch) + for _, r := range records { + if existing, ok := bySender[r.sender]; ok { + t.Errorf( + "sender %d observed more than once (per-call dedup failed): %+v and %+v", + r.sender, existing, r, + ) + } + bySender[r.sender] = r + } + + if _, ok := bySender[4]; ok { + t.Errorf("member 4 announced an identical session ID and must not be observed as a mismatch") + } + + member2, ok := bySender[2] + if !ok { + t.Fatalf("expected member 2 (legacy) to be observed as a mismatch; records: %+v", records) + } + if member2.expected != SessionIDFormatHardenedDKG || member2.observed != SessionIDFormatLegacy { + t.Errorf( + "member 2 unexpected formats: expected=%v observed=%v", + member2.expected, member2.observed, + ) + } + if !IsCrossFormatMismatch(member2.expected, member2.observed) { + t.Errorf("member 2 should be a cross-format (legacy vs hardened) mismatch") + } + + member3, ok := bySender[3] + if !ok { + t.Fatalf("expected member 3 (hardened) to be observed as a mismatch; records: %+v", records) + } + if member3.expected != SessionIDFormatHardenedDKG || member3.observed != SessionIDFormatHardenedDKG { + t.Errorf( + "member 3 unexpected formats: expected=%v observed=%v", + member3.expected, member3.observed, + ) + } + if IsCrossFormatMismatch(member3.expected, member3.observed) { + t.Errorf("member 3 should be a same-format mismatch, not cross-format") + } + + if member2.protocolID != protocolID || member3.protocolID != protocolID { + t.Errorf("observer received unexpected protocol ID") + } +} + +// TestAnnouncer_SessionMismatchObserver_Rejections verifies that the observer +// is NOT invoked for senders rejected by membership or protocol-ID validation, +// while a valid mismatching sender (positive control) IS observed — proving the +// rejections are real and not merely non-delivery. +func TestAnnouncer_SessionMismatchObserver_Rejections(t *testing.T) { + const protocolID = "announcer-mismatch-rejections-protocol" + const channelName = "announcer-mismatch-rejections" + + provider, membershipValidator := newMismatchObserverFixture(t) + + receiverChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(receiverChannel) + + senderChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(senderChannel) + + recorder := &mismatchRecorder{} + receiver := New( + protocolID, + receiverChannel, + membershipValidator, + WithSessionMismatchObserver(recorder.observer()), + ) + + const receiverSessionID = "dkg-abc123-0000000000000001" + + ctx, cancel := context.WithTimeout( + context.Background(), + 10*local.RetransmissionTick, + ) + defer cancel() + + // member index 6 is out of the 5-member group -> membership invalid. + sendAnnouncement(t, ctx, senderChannel, 6, protocolID, "abc123-5") + // wrong protocol ID -> rejected before the mismatch check. + sendAnnouncement(t, ctx, senderChannel, 3, "some-other-protocol", "abc123-5") + // positive control: valid member, matching protocol, mismatching session ID. + sendAnnouncement(t, ctx, senderChannel, 4, protocolID, "abc123-7") + + if _, err := receiver.Announce(ctx, 1, receiverSessionID); err != nil { + t.Fatal(err) + } + + records := recorder.snapshot() + + seen := make(map[group.MemberIndex]bool) + for _, r := range records { + seen[r.sender] = true + } + + if seen[6] { + t.Errorf("membership-invalid member 6 must not be observed") + } + if seen[3] { + t.Errorf("protocol-mismatched member 3 must not be observed") + } + if !seen[4] { + t.Fatalf( + "positive control member 4 must be observed; the harness delivered nothing. records: %+v", + records, + ) + } +} + +// TestAnnouncer_NilObserverCompatibility verifies that an announcer created +// without an observer (and one created with an explicit nil observer) still +// completes normally when it receives mismatching announcements. +func TestAnnouncer_NilObserverCompatibility(t *testing.T) { + const protocolID = "announcer-nil-observer-protocol" + const channelName = "announcer-nil-observer" + + provider, membershipValidator := newMismatchObserverFixture(t) + + receiverChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(receiverChannel) + + senderChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(senderChannel) + + // No observer option, plus an explicit nil observer, must both be safe. + receiver := New( + protocolID, + receiverChannel, + membershipValidator, + WithSessionMismatchObserver(nil), + ) + + const receiverSessionID = "dkg-abc123-0000000000000001" + + ctx, cancel := context.WithTimeout( + context.Background(), + 6*local.RetransmissionTick, + ) + defer cancel() + + sendAnnouncement(t, ctx, senderChannel, 2, protocolID, "abc123-5") + + readyMembers, err := receiver.Announce(ctx, 1, receiverSessionID) + if err != nil { + t.Fatal(err) + } + + // The only ready member is the receiver itself; the mismatching member 2 is + // discarded and does not appear. + if !reflect.DeepEqual(readyMembers, []group.MemberIndex{1}) { + t.Errorf("unexpected ready members: %v", readyMembers) + } +} + +// TestAnnouncer_SessionMismatchObserver_RawIDsAbsent verifies that the observer +// carries only classified formats, never the raw session ID strings. +func TestAnnouncer_SessionMismatchObserver_RawIDsAbsent(t *testing.T) { + const protocolID = "announcer-raw-ids-absent-protocol" + const channelName = "announcer-raw-ids-absent" + + provider, membershipValidator := newMismatchObserverFixture(t) + + receiverChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(receiverChannel) + + senderChannel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + RegisterUnmarshaller(senderChannel) + + // A distinctive raw seed that must never surface in the observer output. + const rawSeed = "cafebabedeadbeef" + const observedSessionID = rawSeed + "-99" + const receiverSessionID = "dkg-" + rawSeed + "-0000000000000001" + + recorder := &mismatchRecorder{} + receiver := New( + protocolID, + receiverChannel, + membershipValidator, + WithSessionMismatchObserver(recorder.observer()), + ) + + ctx, cancel := context.WithTimeout( + context.Background(), + 10*local.RetransmissionTick, + ) + defer cancel() + + sendAnnouncement(t, ctx, senderChannel, 2, protocolID, observedSessionID) + + if _, err := receiver.Announce(ctx, 1, receiverSessionID); err != nil { + t.Fatal(err) + } + + records := recorder.snapshot() + if len(records) == 0 { + t.Fatalf("expected the mismatch to be observed") + } + + for _, r := range records { + // The only strings the observer carries are the protocol ID and the + // format Stringers; none may contain the raw seed component. + fields := []string{ + r.protocolID, + r.expected.String(), + r.observed.String(), + } + for _, f := range fields { + if strings.Contains(f, rawSeed) { + t.Errorf("observer leaked raw session ID material in %q", f) + } + } + } +} + func TestUnreadyMembers(t *testing.T) { tests := map[string]struct { readyMembers []group.MemberIndex diff --git a/pkg/protocol/announcer/fuzz_test.go b/pkg/protocol/announcer/fuzz_test.go new file mode 100644 index 0000000000..ea35fd9ea8 --- /dev/null +++ b/pkg/protocol/announcer/fuzz_test.go @@ -0,0 +1,17 @@ +package announcer + +// This fuzz target exercises the announcer announcementMessage unmarshaler, +// which parses bytes received from untrusted peers over the broadcast channel. +// The invariant under test is that Unmarshal never panics on arbitrary input: +// malformed bytes must return an error, not crash the process. + +import "testing" + +func FuzzAnnouncementMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + // Must never panic on arbitrary input; an error return is correct. + _ = (&announcementMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/protocol/compatibility/mixed_transcript_test.go b/pkg/protocol/compatibility/mixed_transcript_test.go new file mode 100644 index 0000000000..a543042f95 --- /dev/null +++ b/pkg/protocol/compatibility/mixed_transcript_test.go @@ -0,0 +1,584 @@ +package compatibility + +import ( + "encoding/hex" + "math/big" + "sort" + "sync/atomic" + "testing" + "time" + + tsslibcommon "github.com/bnb-chain/tss-lib/common" + tsslibkeygen "github.com/bnb-chain/tss-lib/ecdsa/keygen" + tsslibsigning "github.com/bnb-chain/tss-lib/ecdsa/signing" + "github.com/bnb-chain/tss-lib/tss" + + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/tecdsa" +) + +// mixedTranscriptSettleTimeout bounds a ceremony expected never to finish. +// Failing closed can mean a party rejecting a proof or a party never receiving +// one it can accept, and the second ending produces no error to wait on. +const mixedTranscriptSettleTimeout = 60 * time.Second + +// TestMixedTranscriptSigningCeremonyFailsClosed asserts a tECDSA signing +// ceremony whose members were configured by different compatibility bundles +// produces no signature. +// +// The two bundles exist so a fleet can cross from one release to the other, and +// the crossing is only worth anything if it is all-or-nothing. A legacy party +// omits the session binding a security-v2 party's GG20 proofs are built on, so +// a ceremony that tolerated a mixture would let one member strip the binding +// off a signature the rest of the group believed carried it — a downgrade +// obtained by joining a ceremony rather than by attacking it. Nothing in the +// bundle-selection tests says anything about that: they check which transcript +// each bundle configures, not what happens when two of them meet. +// +// The property belongs to the pinned tss-lib revision rather than to this +// repository, which is exactly why it is asserted here. `go.mod` resolves one +// revision, this test drives that revision through the bundles that own its +// transcript configuration, and a replacement revision that quietly tolerated a +// mixture would fail here rather than in a rehearsal. +// +// The homogeneous cases are not decoration. A harness that never completes any +// ceremony would pass every mixed case for the wrong reason, so the same driver +// has to produce a real signature when the transcripts agree. +func TestMixedTranscriptSigningCeremonyFailsClosed(t *testing.T) { + tests := map[string]struct { + modeAt func(index int) participation.ProtocolMode + expectSignature bool + }{ + "every member on the security-v2 transcript": { + modeAt: func(int) participation.ProtocolMode { + return participation.ModeSecurityV2 + }, + expectSignature: true, + }, + "every member on the legacy transcript": { + modeAt: func(int) participation.ProtocolMode { + return participation.ModeLegacy + }, + expectSignature: true, + }, + // The downgrade attempt: one member that has not crossed joins a + // ceremony the rest are running on the hardened transcript. + "one legacy member among security-v2 members": { + modeAt: func(index int) participation.ProtocolMode { + if index == 0 { + return participation.ModeLegacy + } + return participation.ModeSecurityV2 + }, + }, + // The mirror: the member that has crossed must refuse to sign with a + // group that has not. + "one security-v2 member among legacy members": { + modeAt: func(index int) participation.ProtocolMode { + if index == 0 { + return participation.ModeSecurityV2 + } + return participation.ModeLegacy + }, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + signatures, members, refusals, terminal := + runMixedTranscriptSigningCeremony(t, test.modeAt) + + assertMixedTranscriptOutcome( + t, + "signature", + test.expectSignature, + signatures, + members, + refusals, + terminal, + ) + }) + } +} + +// assertMixedTranscriptOutcome holds a ceremony to the only two endings the +// crossing allows: every member outputs, or no member does. +// +// A partial count is its own failure and neither assertion would catch it. It +// would mean some members finished a ceremony others refused, which is the +// split state the all-or-nothing crossing exists to prevent — a group holding +// key material or a signature that only part of it agrees was produced. +// +// A ceremony that never reached its terminal state decides nothing either way. +// The refusal claim is that no member *will* output, and a group still holding +// live workers has not been shown that; the count in hand is what had appeared +// by the time the drive gave up. +func assertMixedTranscriptOutcome( + t *testing.T, + output string, + expectOutput bool, + outputs int, + members int, + refusals int, + terminal bool, +) { + t.Helper() + + if !terminal { + t.Fatalf( + "the ceremony was still running after %s with %d/%d %ss and %d "+ + "refusal(s); nothing was observed to terminal completion", + mixedTranscriptSettleTimeout, + outputs, + members, + output, + refusals, + ) + } + + if expectOutput { + if outputs != members { + t.Fatalf( + "a ceremony whose members agree on the transcript produced "+ + "%d/%d %ss; %d member(s) refused", + outputs, + members, + output, + refusals, + ) + } + return + } + + if outputs != 0 { + t.Fatalf( + "a ceremony mixing proof transcripts produced %d/%d %ss", + outputs, + members, + output, + ) + } + if refusals == 0 { + t.Logf( + "the mixed ceremony produced no %s and no member reported a "+ + "refusal within the settle timeout", + output, + ) + } +} + +// TestMixedTranscriptKeygenCeremonyFailsClosed asserts a tECDSA key-generation +// ceremony whose members were configured by different compatibility bundles +// produces no key share. +// +// Keygen is the half of the crossing that leaves something behind. A refused +// signing ceremony costs an attempt; a keygen ceremony that tolerated a mixture +// would mint a wallet whose members disagree about which transcript its shares +// were proved under, and that wallet then signs for as long as it exists. The +// legacy party's Paillier and DLN proofs carry no session tag, the security-v2 +// party's carry one derived from the ceremony's SSID, and each verifies the +// other's under its own rule — so a revision that let those meet would be +// caught here rather than at the first wallet that could not be trusted. +// +// The homogeneous cases carry the same weight as in signing: a harness that +// never completed a ceremony would pass the mixed cases for the wrong reason. +func TestMixedTranscriptKeygenCeremonyFailsClosed(t *testing.T) { + tests := map[string]struct { + modeAt func(index int) participation.ProtocolMode + expectKeyShare bool + }{ + "every member on the security-v2 transcript": { + modeAt: func(int) participation.ProtocolMode { + return participation.ModeSecurityV2 + }, + expectKeyShare: true, + }, + "every member on the legacy transcript": { + modeAt: func(int) participation.ProtocolMode { + return participation.ModeLegacy + }, + expectKeyShare: true, + }, + // A member that has not crossed joining a wallet the rest are + // generating on the hardened transcript. + "one legacy member among security-v2 members": { + modeAt: func(index int) participation.ProtocolMode { + if index == 0 { + return participation.ModeLegacy + } + return participation.ModeSecurityV2 + }, + }, + // The mirror: the member that has crossed must refuse to help mint a + // wallet with a group that has not. + "one security-v2 member among legacy members": { + modeAt: func(index int) participation.ProtocolMode { + if index == 0 { + return participation.ModeSecurityV2 + } + return participation.ModeLegacy + }, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + shares, members, refusals, terminal := + runMixedTranscriptKeygenCeremony(t, test.modeAt) + + assertMixedTranscriptOutcome( + t, + "key share", + test.expectKeyShare, + shares, + members, + refusals, + terminal, + ) + }) + } +} + +// runMixedTranscriptKeygenCeremony drives a real tss-lib key-generation +// ceremony, configuring each party's transcript through the compatibility +// bundle the given mode selects. It reports how many parties produced a key +// share, how many were in the group, and how many refused. +func runMixedTranscriptKeygenCeremony( + t *testing.T, + modeAt func(index int) participation.ProtocolMode, +) (shares int, members int, refusals int, terminal bool) { + t.Helper() + + // Keygen is quadratic in the group and dominated by Paillier work, so the + // group is the smallest one that still has a member to disagree with. + const keygenGroupSize = 3 + const tssThreshold = keygenGroupSize - 1 + + // The key-share fixtures were themselves produced by a keygen ceremony and + // carry the pre-parameters each party used. Reusing them is what keeps this + // test to the protocol under examination: generating fresh safe primes here + // would add minutes of work that says nothing about proof transcripts. + fixtures, err := tecdsatest.LoadPrivateKeyShareTestFixtures(keygenGroupSize) + if err != nil { + t.Fatalf("failed to load key-share fixtures: [%v]", err) + } + + partyIDs := make(tss.UnSortedPartyIDs, len(fixtures)) + for i := range fixtures { + moniker := big.NewInt(int64(i + 1)).String() + partyIDs[i] = tss.NewPartyID( + moniker, + moniker, + big.NewInt(int64(i+1)), + ) + } + sortedPartyIDs := tss.SortPartyIDs(partyIDs) + peerContext := tss.NewPeerContext(sortedPartyIDs) + + errChan := make(chan *tss.Error, len(sortedPartyIDs)*len(sortedPartyIDs)) + outgoingChan := make( + chan tss.Message, + len(sortedPartyIDs)*len(sortedPartyIDs), + ) + resultChan := make( + chan tsslibkeygen.LocalPartySaveData, + len(sortedPartyIDs), + ) + + parties := make([]tss.Party, 0, len(sortedPartyIDs)) + for i := range sortedPartyIDs { + strategies, err := StrategiesFor(modeAt(i)) + if err != nil { + t.Fatalf("failed to resolve the compatibility bundle: [%v]", err) + } + + parameters := tss.NewParameters( + tecdsa.Curve, + peerContext, + sortedPartyIDs[i], + len(sortedPartyIDs), + tssThreshold, + ) + // The session ID is the ceremony's, not the member's: a mixed group + // disagrees about the transcript, never about which ceremony it is in. + if err := strategies.ConfigureTSSParameters( + parameters, + mixedTranscriptDKGSessionID, + ); err != nil { + t.Fatalf("failed to configure the TSS parameters: [%v]", err) + } + + parties = append(parties, tsslibkeygen.NewLocalParty( + parameters, + outgoingChan, + resultChan, + fixtures[i].LocalPreParams, + )) + } + + shares, refusals, terminal = driveTSSCeremony( + parties, + outgoingChan, + resultChan, + errChan, + ) + return shares, len(parties), refusals, terminal +} + +// mixedTranscriptDKGSessionID is a ceremony session identifier long enough to +// clear the security-v2 bundle's proof-binding floor. +const mixedTranscriptDKGSessionID = "dkg-64757a1f-0000000000000001" + +// runMixedTranscriptSigningCeremony drives a real tss-lib signing ceremony over +// the tECDSA key-share fixtures, configuring each party's transcript through the +// compatibility bundle the given mode selects. It reports how many parties +// produced a signature, how many were in the group, and how many refused. +func runMixedTranscriptSigningCeremony( + t *testing.T, + modeAt func(index int) participation.ProtocolMode, +) (signatures int, members int, refusals int, terminal bool) { + t.Helper() + + // The fixtures are a 3-of-5 group, so three members carry the signing + // threshold. Fewer parties is a faster ceremony and the same transcript. + const signingGroupSize = 3 + const tssThreshold = signingGroupSize - 1 + + shares, err := tecdsatest.LoadPrivateKeyShareTestFixtures(signingGroupSize) + if err != nil { + t.Fatalf("failed to load key-share fixtures: [%v]", err) + } + + // tss-lib identifies a party by its key share's identifier and expects the + // peer context sorted by it, so the shares are ordered the same way before + // they are handed to the parties that hold them. + sort.Slice(shares, func(i, j int) bool { + return shares[i].ShareID.Cmp(shares[j].ShareID) < 0 + }) + partyIDs := make(tss.UnSortedPartyIDs, len(shares)) + for i, share := range shares { + moniker := big.NewInt(int64(i + 1)).String() + partyIDs[i] = tss.NewPartyID(moniker, moniker, share.ShareID) + } + sortedPartyIDs := tss.SortPartyIDs(partyIDs) + peerContext := tss.NewPeerContext(sortedPartyIDs) + + messageBytes, err := hex.DecodeString( + "00f163ee51bcaeff9cdff5e0e3c1a646abd19885fffbab0b3b4236e0cf95c9f5", + ) + if err != nil { + t.Fatal(err) + } + message := new(big.Int).SetBytes(messageBytes) + + // A refusing party can be reported once per peer that keeps delivering to + // it, so the channels are sized past one per party to keep the routing + // goroutines from blocking once the ceremony is decided. + errChan := make(chan *tss.Error, len(sortedPartyIDs)*len(sortedPartyIDs)) + outgoingChan := make( + chan tss.Message, + len(sortedPartyIDs)*len(sortedPartyIDs), + ) + resultChan := make(chan tsslibcommon.SignatureData, len(sortedPartyIDs)) + + parties := make([]tss.Party, 0, len(sortedPartyIDs)) + for i := range sortedPartyIDs { + strategies, err := StrategiesFor(modeAt(i)) + if err != nil { + t.Fatalf("failed to resolve the compatibility bundle: [%v]", err) + } + + parameters := tss.NewParameters( + tecdsa.Curve, + peerContext, + sortedPartyIDs[i], + len(sortedPartyIDs), + tssThreshold, + ) + // The session ID is the ceremony's, not the member's: a mixed group + // disagrees about the transcript, never about which ceremony it is in. + if err := strategies.ConfigureTSSParameters( + parameters, + mixedTranscriptSessionID, + ); err != nil { + t.Fatalf("failed to configure the TSS parameters: [%v]", err) + } + + fullBytesLen := (tecdsa.Curve.Params().N.BitLen() + 7) / 8 + parties = append(parties, tsslibsigning.NewLocalParty( + message, + parameters, + shares[i], + outgoingChan, + resultChan, + fullBytesLen, + )) + } + + signatures, refusals, terminal = driveTSSCeremony( + parties, + outgoingChan, + resultChan, + errChan, + ) + return signatures, len(parties), refusals, terminal +} + +// mixedTranscriptSessionID is a ceremony session identifier long enough to +// clear the security-v2 bundle's proof-binding floor. +const mixedTranscriptSessionID = "signing-64757a1f-0000000000000001" + +// driveTSSCeremony starts every party and routes messages between them until +// the ceremony can produce nothing further. It reports how many parties output +// a result, how many refusals were seen, and whether the ceremony was watched +// all the way to that terminal state. +// +// The terminal state is the point of this function. A tss-lib party emits its +// messages and its save data from inside Start and UpdateFromBytes — every +// goroutine a round spawns is joined before the round returns — so a ceremony +// with no Start and no delivery still running, and nothing left queued to +// deliver, has no path left by which a signature or a key share could appear. +// That is a state this function can join and observe. A stretch of silence is +// not: it says the ceremony has produced nothing *yet*, which is what a slow +// round also looks like, and a caller asserting that nothing was output would +// be asserting it of a group still holding workers that could output. +// +// A refusal therefore does not end the drive either. What is under test is that +// a mixed group produces nothing, and stopping at the first error would only +// establish that one party complained — a group that reported a refusal and +// then went on to output a signature anyway would look identical. Routing +// continues, the result channel keeps being watched, and the count the caller +// asserts on is the number of outputs observed over the whole ceremony. +// +// The settle timeout bounds a party that hangs inside tss-lib rather than +// returning. It is reported as a failure to reach the terminal state rather +// than folded into the counts, because a ceremony still running proves nothing +// about what it will not produce. +// +// It is generic over the result type because keygen and signing deliver +// different save data over otherwise identical plumbing, and the property being +// asserted is about how many results appear, not what is in them. +func driveTSSCeremony[R any]( + parties []tss.Party, + outgoingChan <-chan tss.Message, + resultChan <-chan R, + errChan chan *tss.Error, +) (outputs int, refusals int, terminal bool) { + // Everything that can still put something on one of the channels below: + // each party's Start, and each delivery into a party. Only the loop below + // adds to it, and it does so before releasing the worker, so a count of + // zero read there means no party code is executing anywhere. + var running atomic.Int64 + // Coalescing and never blocking, because its only job is to wake the loop + // so it re-reads the count. A worker that could block here would be a + // worker the count never releases. + woken := make(chan struct{}, 1) + run := func(work func()) { + running.Add(1) + go func() { + defer func() { + running.Add(-1) + select { + case woken <- struct{}{}: + default: + } + }() + work() + }() + } + + // A refusing party keeps refusing every later delivery, so reports can + // outnumber the channel long after the ceremony is decided. Dropping the + // surplus is what keeps a delivery from blocking on a full channel, which + // would leave a worker the count never releases; the first report already + // carries everything the caller reads. + report := func(err *tss.Error) { + select { + case errChan <- err: + default: + } + } + + deliver := func(to tss.Party, message tss.Message) { + run(func() { + bytes, routingInfo, err := message.WireBytes() + if err != nil { + report(to.WrapError(err)) + return + } + if _, err := to.UpdateFromBytes( + bytes, + routingInfo.From, + routingInfo.IsBroadcast, + ); err != nil { + report(err) + } + }) + } + + route := func(message tss.Message) { + destinations := message.GetTo() + if destinations == nil { + for _, party := range parties { + if party.PartyID().Index == message.GetFrom().Index { + continue + } + deliver(party, message) + } + return + } + for _, destination := range destinations { + deliver(parties[destination.Index], message) + } + } + + // A party that cannot start emits nothing, so its error is the only thing + // that says the ceremony went anywhere. + for _, party := range parties { + run(func() { + if err := party.Start(); err != nil { + report(err) + } + }) + } + + settled := time.After(mixedTranscriptSettleTimeout) + + for { + if running.Load() == 0 { + // Nothing is executing, so nothing can be added to these channels + // any more and whatever is already on them is all there will ever + // be. Taking it here rather than declaring the ceremony over keeps + // a result that landed in the same instant as the last worker + // returned counted against the caller's claim. + select { + case <-errChan: + refusals++ + case <-resultChan: + outputs++ + case message := <-outgoingChan: + route(message) + default: + return outputs, refusals, true + } + continue + } + + select { + case <-errChan: + refusals++ + + case message := <-outgoingChan: + route(message) + + case <-resultChan: + outputs++ + + case <-woken: + + case <-settled: + return outputs, refusals, false + } + } +} diff --git a/pkg/protocol/compatibility/strategies.go b/pkg/protocol/compatibility/strategies.go new file mode 100644 index 0000000000..41b17f6066 --- /dev/null +++ b/pkg/protocol/compatibility/strategies.go @@ -0,0 +1,221 @@ +// Package compatibility bundles the per-ceremony cryptographic compatibility +// strategies of the coordinated protocol cutover. A ceremony participates +// with exactly one bundle — legacy or security-v2 — selected from its +// participation permit's pinned protocol mode, and every wire- and +// transcript-sensitive decision travels together inside that bundle: the +// announcement session-ID formats, the ECDH symmetric-key derivation, the +// G1 hash-to-point mapping, and the tECDSA proof-transcript configuration. +// Selecting these decisions individually is forbidden: switching only one of +// them would produce a partially legacy ceremony that interoperates with +// neither release. +// +// The bundles are stateless values and therefore immutable: nothing can +// mutate a bundle after selection, and nothing in this package reads the +// chain clock, a configuration file, or any global mode. Legacy strategies +// reproduce, byte for byte, the behavior of the pre-hardening production +// releases; security-v2 strategies reproduce the hardened behavior. +package compatibility + +import ( + "fmt" + "math/big" + + "github.com/bnb-chain/tss-lib/tss" + bn256 "github.com/ethereum/go-ethereum/crypto/bn256/cloudflare" + + "github.com/keep-network/keep-core/pkg/altbn128" + "github.com/keep-network/keep-core/pkg/crypto/ephemeral" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// minTSSSessionIDBytes mirrors the pinned tss-lib fork's minimum session-ID +// length: the fork hashes the session ID into the GG20 proof-binding nonce +// and panics below 16 bytes (128 bits), the birthday-bound minimum for +// collision resistance. The bundle validates the length up front so a +// malformed session ID surfaces as an error, not a panic inside the party. +const minTSSSessionIDBytes = 16 + +// Strategies is the immutable per-ceremony compatibility strategy bundle. All +// methods are pure functions of their inputs and the bundle's mode; a bundle +// carries no other state. +type Strategies interface { + // Mode returns the protocol mode this bundle implements. + Mode() participation.ProtocolMode + + // DKGSessionID returns the announcement/protocol session ID for one DKG + // attempt over the given seed. + DKGSessionID(seed *big.Int, attemptNumber uint) string + + // SigningSessionID returns the announcement/protocol session ID for one + // signing attempt over the given message. The legacy format carries no + // attempt start block; the parameter participates only in the security-v2 + // format. + SigningSessionID( + message *big.Int, + attemptStartBlock uint64, + attemptNumber uint, + ) string + + // ECDH derives the symmetric key for the given key pair. The info label + // provides the security-v2 protocol/peer domain separation; the legacy + // derivation has no domain separation by design and ignores it. + ECDH( + privateKey *ephemeral.PrivateKey, + publicKey *ephemeral.PublicKey, + info []byte, + ) *ephemeral.SymmetricEcdhKey + + // G1HashToPoint maps the given message onto a G1 point. + G1HashToPoint(message []byte) *bn256.G1 + + // ConfigureTSSParameters applies this bundle's tECDSA proof-transcript + // decision to the given TSS parameters before any local party is + // constructed from them. The security-v2 configuration binds the GG20 + // proof challenges to the ceremony's session ID; the legacy configuration + // selects the historical untagged proof transcript. The applied setting is + // immutable for the party's lifetime. + ConfigureTSSParameters(parameters *tss.Parameters, sessionID string) error +} + +// StrategiesFor returns the immutable strategy bundle for the given protocol +// mode. There is no default bundle: a mode that is not explicitly legacy or +// security-v2 is a programming error, because an implicit mode could silently +// produce a partially incompatible ceremony. +func StrategiesFor(mode participation.ProtocolMode) (Strategies, error) { + switch mode { + case participation.ModeLegacy: + return legacyStrategies{}, nil + case participation.ModeSecurityV2: + return securityV2Strategies{}, nil + default: + return nil, fmt.Errorf( + "no compatibility strategies for protocol mode [%v]: the mode "+ + "must be selected explicitly from a participation permit", + mode, + ) + } +} + +// Legacy returns the legacy strategy bundle. It is a deliberate, explicit +// selection — never a fallback for an unset mode. +func Legacy() Strategies { + return legacyStrategies{} +} + +// SecurityV2 returns the security-v2 strategy bundle. It is a deliberate, +// explicit selection — never a fallback for an unset mode. +func SecurityV2() Strategies { + return securityV2Strategies{} +} + +// legacyStrategies reproduces, byte for byte, the wire- and +// transcript-sensitive behavior of the pre-hardening production releases. +type legacyStrategies struct{} + +func (legacyStrategies) Mode() participation.ProtocolMode { + return participation.ModeLegacy +} + +func (legacyStrategies) DKGSessionID( + seed *big.Int, + attemptNumber uint, +) string { + return fmt.Sprintf("%v-%v", seed.Text(16), attemptNumber) +} + +func (legacyStrategies) SigningSessionID( + message *big.Int, + _ uint64, + attemptNumber uint, +) string { + return fmt.Sprintf("%v-%v", message.Text(16), attemptNumber) +} + +func (legacyStrategies) ECDH( + privateKey *ephemeral.PrivateKey, + publicKey *ephemeral.PublicKey, + _ []byte, +) *ephemeral.SymmetricEcdhKey { + return privateKey.EcdhLegacy(publicKey) +} + +func (legacyStrategies) G1HashToPoint(message []byte) *bn256.G1 { + return altbn128.G1HashToPointLegacy(message) +} + +func (legacyStrategies) ConfigureTSSParameters( + parameters *tss.Parameters, + _ string, +) error { + if parameters == nil { + return fmt.Errorf( + "cannot configure TSS parameters: no parameters provided", + ) + } + + parameters.SetProtocolMode(tss.ProtocolModeLegacy) + return nil +} + +// securityV2Strategies reproduces the hardened behavior of the security +// release. +type securityV2Strategies struct{} + +func (securityV2Strategies) Mode() participation.ProtocolMode { + return participation.ModeSecurityV2 +} + +func (securityV2Strategies) DKGSessionID( + seed *big.Int, + attemptNumber uint, +) string { + return fmt.Sprintf("dkg-%v-%016x", seed.Text(16), attemptNumber) +} + +func (securityV2Strategies) SigningSessionID( + message *big.Int, + attemptStartBlock uint64, + attemptNumber uint, +) string { + return fmt.Sprintf( + "signing-%v-%016x-%016x", + message.Text(16), + attemptStartBlock, + attemptNumber, + ) +} + +func (securityV2Strategies) ECDH( + privateKey *ephemeral.PrivateKey, + publicKey *ephemeral.PublicKey, + info []byte, +) *ephemeral.SymmetricEcdhKey { + return privateKey.Ecdh(publicKey, info) +} + +func (securityV2Strategies) G1HashToPoint(message []byte) *bn256.G1 { + return altbn128.G1HashToPoint(message) +} + +func (securityV2Strategies) ConfigureTSSParameters( + parameters *tss.Parameters, + sessionID string, +) error { + if parameters == nil { + return fmt.Errorf( + "cannot configure TSS parameters: no parameters provided", + ) + } + if len(sessionID) < minTSSSessionIDBytes { + return fmt.Errorf( + "cannot bind GG20 proof challenges to session ID of [%d] bytes: "+ + "at least [%d] bytes are required", + len(sessionID), + minTSSSessionIDBytes, + ) + } + // Bind GG20 proof challenges to the existing protocol session. + parameters.SetProtocolMode(tss.ProtocolModeSecurityV2) + parameters.SetSessionNonceBytes([]byte(sessionID)) + return nil +} diff --git a/pkg/protocol/compatibility/strategies_test.go b/pkg/protocol/compatibility/strategies_test.go new file mode 100644 index 0000000000..9d9547820a --- /dev/null +++ b/pkg/protocol/compatibility/strategies_test.go @@ -0,0 +1,320 @@ +package compatibility + +import ( + "bytes" + "math/big" + "testing" + + tsslibcommon "github.com/bnb-chain/tss-lib/common" + "github.com/bnb-chain/tss-lib/tss" + + "github.com/keep-network/keep-core/pkg/altbn128" + "github.com/keep-network/keep-core/pkg/crypto/ephemeral" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +func TestStrategiesFor(t *testing.T) { + legacy, err := StrategiesFor(participation.ModeLegacy) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if legacy.Mode() != participation.ModeLegacy { + t.Errorf("expected legacy mode, got [%s]", legacy.Mode()) + } + + securityV2, err := StrategiesFor(participation.ModeSecurityV2) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if securityV2.Mode() != participation.ModeSecurityV2 { + t.Errorf("expected security_v2 mode, got [%s]", securityV2.Mode()) + } + + // There is no default bundle: an unset or unknown mode must fail loudly + // instead of silently selecting one side of the cutover. + for _, mode := range []participation.ProtocolMode{0, 3, 255} { + if _, err := StrategiesFor(mode); err == nil { + t.Errorf("expected an error for mode [%d]", mode) + } + } +} + +// TestDKGSessionIDFixtures pins the exact DKG announcement session-ID bytes +// of both modes. The legacy form must remain byte-for-byte the pre-hardening +// production form; the security-v2 form must remain the hardened form. A +// drifted value breaks announcement matching with the corresponding release. +func TestDKGSessionIDFixtures(t *testing.T) { + seed, ok := new(big.Int).SetString("64757a1f", 16) + if !ok { + t.Fatal("could not parse the seed fixture") + } + + legacy, _ := StrategiesFor(participation.ModeLegacy) + securityV2, _ := StrategiesFor(participation.ModeSecurityV2) + + if got := legacy.DKGSessionID(seed, 1); got != "64757a1f-1" { + t.Errorf("legacy DKG session ID drifted: [%s]", got) + } + if got := legacy.DKGSessionID(seed, 12); got != "64757a1f-12" { + t.Errorf("legacy DKG session ID drifted: [%s]", got) + } + + if got := securityV2.DKGSessionID( + seed, 1, + ); got != "dkg-64757a1f-0000000000000001" { + t.Errorf("security-v2 DKG session ID drifted: [%s]", got) + } + if got := securityV2.DKGSessionID( + seed, 12, + ); got != "dkg-64757a1f-000000000000000c" { + t.Errorf("security-v2 DKG session ID drifted: [%s]", got) + } +} + +// TestSigningSessionIDFixtures pins the exact signing announcement session-ID +// bytes of both modes. The legacy form carries no attempt start block — the +// start block must not leak into it — while the security-v2 form binds both +// the start block and the attempt with fixed width. +func TestSigningSessionIDFixtures(t *testing.T) { + message, ok := new(big.Int).SetString("9f1c8e2d", 16) + if !ok { + t.Fatal("could not parse the message fixture") + } + + legacy, _ := StrategiesFor(participation.ModeLegacy) + securityV2, _ := StrategiesFor(participation.ModeSecurityV2) + + if got := legacy.SigningSessionID(message, 12345, 3); got != "9f1c8e2d-3" { + t.Errorf("legacy signing session ID drifted: [%s]", got) + } + // Different start blocks must produce the identical legacy session ID. + if got := legacy.SigningSessionID(message, 99999, 3); got != "9f1c8e2d-3" { + t.Errorf( + "legacy signing session ID depends on the start block: [%s]", + got, + ) + } + + if got := securityV2.SigningSessionID( + message, 12345, 3, + ); got != "signing-9f1c8e2d-0000000000003039-0000000000000003" { + t.Errorf("security-v2 signing session ID drifted: [%s]", got) + } +} + +// TestECDHSelection proves the bundle selects the exact derivation of its +// mode — the key agrees with the mode's direct derivation — and that the two +// modes' keys cannot decrypt each other's ciphertexts: they fail with an +// error, not a panic or a wrong plaintext. +func TestECDHSelection(t *testing.T) { + keyPair1, err := ephemeral.GenerateKeyPair() + if err != nil { + t.Fatalf("could not generate a key pair: [%v]", err) + } + keyPair2, err := ephemeral.GenerateKeyPair() + if err != nil { + t.Fatalf("could not generate a key pair: [%v]", err) + } + + legacy, _ := StrategiesFor(participation.ModeLegacy) + securityV2, _ := StrategiesFor(participation.ModeSecurityV2) + + info := []byte("protocol-label|peer-pair") + plaintext := []byte("compatibility bundle plaintext") + + // The legacy bundle key must agree with the direct legacy derivation on + // the other side of the exchange. + legacyKey := legacy.ECDH(keyPair1.PrivateKey, keyPair2.PublicKey, info) + directLegacy := keyPair2.PrivateKey.EcdhLegacy(keyPair1.PublicKey) + ciphertext, err := legacyKey.Encrypt(plaintext) + if err != nil { + t.Fatalf("could not encrypt: [%v]", err) + } + decrypted, err := directLegacy.Decrypt(ciphertext) + if err != nil { + t.Fatalf("legacy bundle key disagrees with EcdhLegacy: [%v]", err) + } + if !bytes.Equal(decrypted, plaintext) { + t.Error("legacy round trip corrupted the plaintext") + } + + // The security-v2 bundle key must agree with the direct hardened + // derivation for the same info label. + securityV2Key := securityV2.ECDH( + keyPair1.PrivateKey, keyPair2.PublicKey, info, + ) + directSecurityV2 := keyPair2.PrivateKey.Ecdh(keyPair1.PublicKey, info) + ciphertext, err = securityV2Key.Encrypt(plaintext) + if err != nil { + t.Fatalf("could not encrypt: [%v]", err) + } + decrypted, err = directSecurityV2.Decrypt(ciphertext) + if err != nil { + t.Fatalf("security-v2 bundle key disagrees with Ecdh: [%v]", err) + } + if !bytes.Equal(decrypted, plaintext) { + t.Error("security-v2 round trip corrupted the plaintext") + } + + // Cross-mode decryption must fail closed in both directions. + securityV2Ciphertext, err := securityV2Key.Encrypt(plaintext) + if err != nil { + t.Fatalf("could not encrypt: [%v]", err) + } + if _, err := legacyKey.Decrypt(securityV2Ciphertext); err == nil { + t.Error("the legacy key decrypted a security-v2 ciphertext") + } + legacyCiphertext, err := legacyKey.Encrypt(plaintext) + if err != nil { + t.Fatalf("could not encrypt: [%v]", err) + } + if _, err := securityV2Key.Decrypt(legacyCiphertext); err == nil { + t.Error("the security-v2 key decrypted a legacy ciphertext") + } +} + +// TestG1HashToPointSelection proves the bundle selects the exact +// hash-to-point mapping of its mode and that the two mappings diverge for the +// same input, so a mode mix-up cannot go unnoticed on the wire. +func TestG1HashToPointSelection(t *testing.T) { + legacy, _ := StrategiesFor(participation.ModeLegacy) + securityV2, _ := StrategiesFor(participation.ModeSecurityV2) + + for _, message := range [][]byte{ + []byte(""), + []byte("beacon group seed"), + make([]byte, 32), + } { + legacyPoint := legacy.G1HashToPoint(message).Marshal() + if !bytes.Equal( + legacyPoint, + altbn128.G1HashToPointLegacy(message).Marshal(), + ) { + t.Errorf( + "legacy bundle mapping disagrees with G1HashToPointLegacy "+ + "for input %q", + message, + ) + } + + securityV2Point := securityV2.G1HashToPoint(message).Marshal() + if !bytes.Equal( + securityV2Point, + altbn128.G1HashToPoint(message).Marshal(), + ) { + t.Errorf( + "security-v2 bundle mapping disagrees with G1HashToPoint "+ + "for input %q", + message, + ) + } + + if bytes.Equal(legacyPoint, securityV2Point) { + t.Errorf( + "legacy and security-v2 mappings coincided for input %q", + message, + ) + } + } +} + +// newTestTSSParameters builds a minimal valid TSS parameters value: two +// parties with distinct keys and threshold one, the smallest setup the pinned +// tss-lib constructor accepts. +func newTestTSSParameters() *tss.Parameters { + parties := tss.SortPartyIDs([]*tss.PartyID{ + tss.NewPartyID("1", "member-1", big.NewInt(1)), + tss.NewPartyID("2", "member-2", big.NewInt(2)), + }) + + return tss.NewParameters( + tss.S256(), + tss.NewPeerContext(parties), + parties[0], + 2, + 1, + ) +} + +func TestConfigureTSSParametersSelection(t *testing.T) { + legacy, _ := StrategiesFor(participation.ModeLegacy) + securityV2, _ := StrategiesFor(participation.ModeSecurityV2) + + sessionID := "dkg-64757a1f-0000000000000001" + + // The security-v2 bundle binds the GG20 proof challenges to the ceremony + // session exactly as the hardened release does: the session ID hashed + // into the parameters' session nonce. + parameters := newTestTSSParameters() + if err := securityV2.ConfigureTSSParameters( + parameters, + sessionID, + ); err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + expectedNonce := new(big.Int).SetBytes( + tsslibcommon.SHA512_256([]byte(sessionID)), + ) + if parameters.SessionNonce() == nil || + expectedNonce.Cmp(parameters.SessionNonce()) != 0 { + t.Errorf( + "security-v2 session nonce disagrees with the hardened "+ + "session-ID binding: [%v]", + parameters.SessionNonce(), + ) + } + if parameters.ProtocolMode() != tss.ProtocolModeSecurityV2 { + t.Errorf( + "expected security-v2 TSS mode, got [%v]", + parameters.ProtocolMode(), + ) + } + + // The legacy bundle selects the historical untagged transcript and leaves + // the security-v2 session nonce unset. + parameters = newTestTSSParameters() + if err := legacy.ConfigureTSSParameters(parameters, sessionID); err != nil { + t.Fatalf("unexpected legacy configuration error: [%v]", err) + } + if parameters.ProtocolMode() != tss.ProtocolModeLegacy { + t.Errorf( + "expected legacy TSS mode, got [%v]", + parameters.ProtocolMode(), + ) + } + if parameters.SessionNonce() != nil { + t.Errorf( + "legacy TSS configuration set a session nonce: [%v]", + parameters.SessionNonce(), + ) + } +} + +func TestConfigureTSSParametersValidation(t *testing.T) { + legacy, _ := StrategiesFor(participation.ModeLegacy) + securityV2, _ := StrategiesFor(participation.ModeSecurityV2) + + if err := legacy.ConfigureTSSParameters(nil, "64757a1f-1"); err == nil { + t.Error("expected an error for nil legacy parameters") + } + if err := securityV2.ConfigureTSSParameters( + nil, + "dkg-64757a1f-0000000000000001", + ); err == nil { + t.Error("expected an error for nil parameters") + } + + // A session ID below the fork's 16-byte proof-binding floor must surface + // as an error from the bundle, not as a panic inside the party. + parameters := newTestTSSParameters() + if err := securityV2.ConfigureTSSParameters(parameters, "1-1"); err == nil { + t.Error("expected an error for a session ID below 16 bytes") + } + if parameters.SessionNonce() != nil { + t.Error("session nonce must remain unset after a rejected session ID") + } + if parameters.ProtocolMode() != 0 { + t.Error("protocol mode must remain unset after a rejected session ID") + } +} diff --git a/pkg/protocol/compatibility/transcript_ownership_test.go b/pkg/protocol/compatibility/transcript_ownership_test.go new file mode 100644 index 0000000000..42a093d286 --- /dev/null +++ b/pkg/protocol/compatibility/transcript_ownership_test.go @@ -0,0 +1,145 @@ +package compatibility + +import ( + "fmt" + "go/ast" + "go/parser" + "go/token" + "io/fs" + "path/filepath" + "strings" + "testing" +) + +// TestTranscriptDecisionOwnership is the repository check that every +// wire- and transcript-sensitive tECDSA decision travels through the +// per-ceremony strategy bundle instead of being taken implicitly at a call +// site. It scans every production (non-test) Go file under pkg/ and cmd/ and +// fails when: +// +// - the GG20 protocol mode or proof-binding nonce (SetProtocolMode, +// SetSessionNonce, or SetSessionNonceBytes) is set anywhere outside this +// package — the bundle owns the proof-transcript configuration; +// - the ephemeral ECDH derivation (Ecdh/EcdhLegacy) is invoked anywhere +// outside this package — protocols must call the bundle's ECDH so the +// ceremony's pinned mode selects the derivation; or +// - TSS parameters are constructed outside the tECDSA member files — every +// party construction must flow through a member holding an explicit +// strategy bundle. +// +// A new legitimate call site must be added to the allowlists deliberately, +// in the same change that proves it receives an explicit strategy. +func TestTranscriptDecisionOwnership(t *testing.T) { + repoRoot, err := filepath.Abs(filepath.Join("..", "..", "..")) + if err != nil { + t.Fatal(err) + } + + thisPackageDir := filepath.Join(repoRoot, "pkg", "protocol", "compatibility") + + nonceAllowedDirs := map[string]bool{ + thisPackageDir: true, + } + ecdhAllowedDirs := map[string]bool{ + thisPackageDir: true, + // The ephemeral package defines the derivations it exposes. + filepath.Join(repoRoot, "pkg", "crypto", "ephemeral"): true, + } + tssParametersAllowedFiles := map[string]bool{ + filepath.Join(repoRoot, "pkg", "tecdsa", "dkg", "member.go"): true, + filepath.Join(repoRoot, "pkg", "tecdsa", "signing", "member.go"): true, + } + + var violations []string + + inspectFile := func(path string) error { + fileSet := token.NewFileSet() + file, err := parser.ParseFile(fileSet, path, nil, 0) + if err != nil { + return fmt.Errorf("cannot parse [%s]: [%w]", path, err) + } + + ast.Inspect(file, func(node ast.Node) bool { + call, ok := node.(*ast.CallExpr) + if !ok { + return true + } + selector, ok := call.Fun.(*ast.SelectorExpr) + if !ok { + return true + } + + position := fileSet.Position(call.Pos()) + dir := filepath.Dir(path) + + switch selector.Sel.Name { + case "SetProtocolMode", "SetSessionNonce", "SetSessionNonceBytes": + if !nonceAllowedDirs[dir] { + violations = append(violations, fmt.Sprintf( + "%s: the GG20 transcript configuration is owned by the "+ + "compatibility strategy bundle", + position, + )) + } + case "Ecdh", "EcdhLegacy": + if !ecdhAllowedDirs[dir] { + violations = append(violations, fmt.Sprintf( + "%s: the ECDH derivation is owned by the "+ + "compatibility strategy bundle", + position, + )) + } + case "NewParameters": + receiver, ok := selector.X.(*ast.Ident) + if ok && receiver.Name == "tss" && + !tssParametersAllowedFiles[path] { + violations = append(violations, fmt.Sprintf( + "%s: TSS parameters may be constructed only by "+ + "tECDSA members holding an explicit strategy "+ + "bundle", + position, + )) + } + } + + return true + }) + + return nil + } + + for _, scanRoot := range []string{ + filepath.Join(repoRoot, "pkg"), + filepath.Join(repoRoot, "cmd"), + } { + err := filepath.WalkDir( + scanRoot, + func(path string, entry fs.DirEntry, err error) error { + if err != nil { + return err + } + if entry.IsDir() { + // Generated chain bindings and fixtures never take + // protocol decisions; skipping them keeps the scan fast. + switch entry.Name() { + case "gen", "testdata": + return filepath.SkipDir + } + return nil + } + if !strings.HasSuffix(path, ".go") || + strings.HasSuffix(path, "_test.go") { + return nil + } + return inspectFile(path) + }, + ) + if err != nil { + t.Fatal(err) + } + } + + for _, violation := range violations { + t.Error(violation) + } +} diff --git a/pkg/protocol/group/group.go b/pkg/protocol/group/group.go index 84d776d2c4..2a1dbbe340 100644 --- a/pkg/protocol/group/group.go +++ b/pkg/protocol/group/group.go @@ -2,6 +2,8 @@ // and auxiliary tools that help during group-related operations. package group +import "unsafe" + // MemberIndex is an index of a member in a group. The maximum member index // value is 255. type MemberIndex = uint8 @@ -10,6 +12,15 @@ type MemberIndex = uint8 // is represented as uint8 so the maximum member index is 255. const MaxMemberIndex = 255 +// Compile-time assertion that MemberIndex fits in a single byte. The HKDF info +// labels used for ECDH session-key domain separation in gjkr / tecdsa-dkg / +// tecdsa-signing encode each peer's MemberIndex as one byte (see F-03). If +// MemberIndex is ever widened, those encoders must switch to a width- +// independent serialization (e.g. binary.BigEndian.PutUint16) in the same +// coordinated upgrade, otherwise peers whose IDs collide modulo 256 will +// silently derive identical session keys. +var _ [1]struct{} = [unsafe.Sizeof(MemberIndex(0))]struct{}{} + // Group is protocol's members group. type Group struct { // The maximum number of misbehaving participants for which it is still diff --git a/pkg/protocol/group/member_index_test.go b/pkg/protocol/group/member_index_test.go new file mode 100644 index 0000000000..e194f3a7e2 --- /dev/null +++ b/pkg/protocol/group/member_index_test.go @@ -0,0 +1,28 @@ +package group + +import ( + "testing" + "unsafe" +) + +// TestMemberIndexFitsInOneByte is a belt-and-braces runtime check on top of the +// compile-time assertion in group.go. The F-03 HKDF info encoders in gjkr, +// tecdsa-dkg, and tecdsa-signing serialize each MemberIndex as a single byte +// via `byte(id)`. If MemberIndex is ever widened, the compile-time assertion +// in this package fires first; this test exists so a reader grepping for +// "MemberIndex" finds an explicit, named justification for that invariant. +func TestMemberIndexFitsInOneByte(t *testing.T) { + if got := unsafe.Sizeof(MemberIndex(0)); got != 1 { + t.Fatalf( + "MemberIndex must be one byte for F-03 EcdhInfo encoders to be "+ + "injective; got sizeof %d. Either revert the type change or "+ + "switch all *EcdhInfo encoders to a width-independent "+ + "encoding (e.g. binary.BigEndian.PutUint16) in a coordinated "+ + "network upgrade.", + got, + ) + } + if MaxMemberIndex != 255 { + t.Fatalf("MaxMemberIndex must be 255, got %d", MaxMemberIndex) + } +} diff --git a/pkg/protocol/inactivity/fuzz_test.go b/pkg/protocol/inactivity/fuzz_test.go new file mode 100644 index 0000000000..95779c424d --- /dev/null +++ b/pkg/protocol/inactivity/fuzz_test.go @@ -0,0 +1,17 @@ +package inactivity + +// This fuzz target exercises the inactivity claimSignatureMessage unmarshaler, +// which parses bytes received from untrusted peers over the broadcast channel. +// The invariant under test is that Unmarshal never panics on arbitrary input: +// malformed bytes must return an error, not crash the process. + +import "testing" + +func FuzzClaimSignatureMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + // Must never panic on arbitrary input; an error return is correct. + _ = (&claimSignatureMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/protocol/participation/cutover_peer_roster.go b/pkg/protocol/participation/cutover_peer_roster.go new file mode 100644 index 0000000000..18bc999841 --- /dev/null +++ b/pkg/protocol/participation/cutover_peer_roster.go @@ -0,0 +1,752 @@ +package participation + +import ( + "context" + "fmt" + "math" + "sort" + "strings" + "sync" + "time" + + "github.com/ipfs/go-log/v2" + "golang.org/x/time/rate" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +var rosterLogger = log.Logger("keep-participation") + +// CutoverPeerRosterSchemaVersion is the schema version of the node-local +// cutover peer roster snapshot. It is bumped whenever the snapshot JSON shape +// changes incompatibly. +const CutoverPeerRosterSchemaVersion uint32 = 1 + +// maxSafeMetricInteger is the largest integer that a float64 (the gauge/counter +// backing type) can represent exactly. Retention values above it cannot be +// projected to metrics without precision loss and are rejected at construction. +const maxSafeMetricInteger = uint64(1) << 53 + +// The roster reports through the client-info performance registry, which adds +// the "performance_" application prefix. These are therefore the internal +// (unprefixed) names; they are exposed as performance_announcer_legacy_peer*. +// Referencing the clientinfo constants keeps a single source of truth for the +// exact exported metric names. +const ( + metricLegacyPeersCurrent = clientinfo.MetricAnnouncerLegacyPeersCurrent + metricLegacyPeerOldestAgeBlocks = clientinfo.MetricAnnouncerLegacyPeerOldestAgeBlocks + metricLegacyPeerRosterRevision = clientinfo.MetricAnnouncerLegacyPeerRosterRevision + metricLegacyPeerAdditionsTotal = clientinfo.MetricAnnouncerLegacyPeerAdditionsTotal + metricLegacyPeerEvictionsTotal = clientinfo.MetricAnnouncerLegacyPeerEvictionsTotal +) + +const ( + // rosterSweepInterval is how often the background loop reads the chain clock + // to evict stale entries and refresh metrics. + rosterSweepInterval = 30 * time.Second + // rosterSnapshotLogInterval is how often a required roster snapshot is + // logged at INFO. + rosterSnapshotLogInterval = 5 * time.Minute +) + +// CutoverPeerSighting is a single deduplicated observation of a legacy peer at a +// specific group member position within a specific protocol. It never carries +// raw session identifiers or other sensitive material. +type CutoverPeerSighting struct { + ProtocolID string `json:"protocol"` + MemberIndex group.MemberIndex `json:"member_index"` + FirstSeenBlock uint64 `json:"first_seen_block"` + LastSeenBlock uint64 `json:"last_seen_block"` + FirstSeenAt time.Time `json:"first_seen_at"` + LastSeenAt time.Time `json:"last_seen_at"` +} + +// CutoverPeerEntry is the deduplicated per-operator roster entry. All sightings +// for one operator address — across seats, protocols, and reporters — collapse +// into a single entry with multiple sightings. +type CutoverPeerEntry struct { + OperatorAddress string `json:"operator_address"` + FirstSeenBlock uint64 `json:"first_seen_block"` + LastSeenBlock uint64 `json:"last_seen_block"` + Sightings []CutoverPeerSighting `json:"sightings"` +} + +// CutoverPeerRosterSnapshot is a deterministic, point-in-time view of the +// node-local roster suitable for diagnostics exposure and evidence capture. +type CutoverPeerRosterSnapshot struct { + SchemaVersion uint32 `json:"schema_version"` + ProcessStartedAt time.Time `json:"process_started_at"` + GeneratedAt time.Time `json:"generated_at"` + CurrentBlock uint64 `json:"current_block"` + ClockAvailable bool `json:"clock_available"` + RetentionBlocks uint64 `json:"retention_blocks"` + RosterRevision uint64 `json:"roster_revision"` + Peers []CutoverPeerEntry `json:"peers"` +} + +// CutoverRosterMetricsRecorder is the minimal metrics sink the roster needs. It +// is satisfied by the client-info performance metrics registry. +type CutoverRosterMetricsRecorder interface { + IncrementCounter(name string, value float64) + SetGauge(name string, value float64) +} + +// CutoverEvidenceWindowSignal is an operator-controlled, process-local signal +// indicating that periodic cutover roster logs are being captured as +// go/no-go or rollback evidence. It controls only whether an empty roster is +// logged: it is never consulted by Gate, ObserveLegacy, or any protocol-mode +// decision. +// +// The zero value is an inactive, usable signal. +type CutoverEvidenceWindowSignal struct { + mutex sync.RWMutex + active bool + heldActive bool +} + +// NewCutoverEvidenceWindowSignal creates an inactive evidence-window signal. +func NewCutoverEvidenceWindowSignal() *CutoverEvidenceWindowSignal { + return &CutoverEvidenceWindowSignal{} +} + +// Active reports whether periodic roster logs must include an empty snapshot. +func (s *CutoverEvidenceWindowSignal) Active() bool { + s.mutex.RLock() + defer s.mutex.RUnlock() + + return s.active +} + +// SetActive opens or closes the evidence window and reports whether its state +// changed. A signal held active for rollback cannot be closed. +func (s *CutoverEvidenceWindowSignal) SetActive(active bool) bool { + s.mutex.Lock() + defer s.mutex.Unlock() + + if !active && s.heldActive { + return false + } + if s.active == active { + return false + } + + s.active = active + return true +} + +// HoldActive opens the evidence window irreversibly for this process and +// reports whether its visible state changed. The shutdown controller uses this +// before quiescence so a later operator close signal cannot suppress rollback +// evidence while the process is draining. +func (s *CutoverEvidenceWindowSignal) HoldActive() bool { + s.mutex.Lock() + defer s.mutex.Unlock() + + changed := !s.active + s.active = true + s.heldActive = true + return changed +} + +type cutoverPeerRosterOptions struct { + cutoverBlock uint64 + evidenceWindow *CutoverEvidenceWindowSignal + tickerFactory cutoverPeerRosterTickerFactory +} + +// CutoverPeerRosterOption configures immutable cutover context used by the +// node-local roster's observability contract. +type CutoverPeerRosterOption func(*cutoverPeerRosterOptions) error + +// cutoverPeerRosterTicker is the narrow cadence source consumed by the roster +// loop. Keeping the ticker behind this seam lets the production run path be +// exercised deterministically without shortening the five-minute evidence +// contract or adding sleeps to tests. +type cutoverPeerRosterTicker interface { + Ticks() <-chan time.Time + MarkProcessed() + Stop() +} + +type cutoverPeerRosterTickerFactory func(time.Duration) cutoverPeerRosterTicker + +type wallClockCutoverPeerRosterTicker struct { + ticker *time.Ticker +} + +func newWallClockCutoverPeerRosterTicker( + interval time.Duration, +) cutoverPeerRosterTicker { + return &wallClockCutoverPeerRosterTicker{ + ticker: time.NewTicker(interval), + } +} + +func (t *wallClockCutoverPeerRosterTicker) Ticks() <-chan time.Time { + return t.ticker.C +} + +func (t *wallClockCutoverPeerRosterTicker) MarkProcessed() {} + +func (t *wallClockCutoverPeerRosterTicker) Stop() { + t.ticker.Stop() +} + +// withCutoverPeerRosterTickerFactory replaces the wall-clock cadence source. +// It is intentionally package-private: production always uses real time while +// same-package tests drive the exact run-loop boundary deterministically. +func withCutoverPeerRosterTickerFactory( + factory cutoverPeerRosterTickerFactory, +) CutoverPeerRosterOption { + return func(options *cutoverPeerRosterOptions) error { + if factory == nil { + return fmt.Errorf("cutover peer roster ticker factory is required") + } + + options.tickerFactory = factory + return nil + } +} + +// WithCutoverSchedule gives the roster the exact resolved schedule supplied to +// the process participation gate. The roster does not use the schedule to +// classify or authorize observations; it carries C only so an entry log names +// the cutover boundary the observation is evidence for. +func WithCutoverSchedule(schedule Schedule) CutoverPeerRosterOption { + return func(options *cutoverPeerRosterOptions) error { + if err := validateMetricProjectable(schedule.CutoverBlock); err != nil { + return fmt.Errorf("invalid roster cutover schedule: [%w]", err) + } + + options.cutoverBlock = schedule.CutoverBlock + return nil + } +} + +// WithCutoverEvidenceWindowSignal supplies the process-local evidence-window +// signal observed by the roster's periodic logger. The signal has no effect on +// roster contents, cutover classification, or protocol authorization. +func WithCutoverEvidenceWindowSignal( + signal *CutoverEvidenceWindowSignal, +) CutoverPeerRosterOption { + return func(options *cutoverPeerRosterOptions) error { + if signal == nil { + return fmt.Errorf("cutover evidence-window signal is required") + } + + options.evidenceWindow = signal + return nil + } +} + +type sightingKey struct { + protocolID string + memberIndex group.MemberIndex +} + +type peerState struct { + operatorAddress string + firstSeenBlock uint64 + lastSeenBlock uint64 + firstSeenAt time.Time + lastSeenAt time.Time + sightings map[sightingKey]*CutoverPeerSighting +} + +// CutoverPeerRoster is a node-local, deduplicated record of post-cutover legacy +// peer sightings, keyed by normalized operator address. A later hardened +// observation never clears an entry, because it does not prove every instance +// for that operator is current. Eviction means only "not recently observed". +type CutoverPeerRoster struct { + ctx context.Context + cancel context.CancelFunc + closeOnce sync.Once + loopDone chan struct{} + ticker cutoverPeerRosterTicker + + blockCounter chain.BlockCounter + retentionBlocks uint64 + cutoverBlock uint64 + evidenceWindow *CutoverEvidenceWindowSignal + metrics CutoverRosterMetricsRecorder + clock func() time.Time + + processStartedAt time.Time + + logLimiter *rate.Limiter + + mu sync.Mutex + peers map[string]*peerState + rosterRevision uint64 + currentBlock uint64 + clockAvailable bool +} + +// NewCutoverPeerRoster constructs a roster. It rejects zero or precision-unsafe +// retention, synchronously reads the chain clock to seed the current block, +// initializes all fixed metrics to zero, and starts one context-bound sweep +// loop. The roster is intended to be constructed unconditionally, including +// when client-info diagnostics are disabled. A process with an active cutover +// schedule supplies WithCutoverSchedule using the same resolved Schedule given +// to its Gate; omitting it represents the developer-only disabled schedule. +func NewCutoverPeerRoster( + ctx context.Context, + blockCounter chain.BlockCounter, + retentionBlocks uint64, + metrics CutoverRosterMetricsRecorder, + options ...CutoverPeerRosterOption, +) (*CutoverPeerRoster, error) { + return newCutoverPeerRoster( + ctx, + blockCounter, + retentionBlocks, + metrics, + time.Now, + options..., + ) +} + +// newCutoverPeerRoster is the clock-injecting constructor. The clock is fixed +// before the background loop starts, so callers (including tests) may supply a +// deterministic clock without racing the loop. +func newCutoverPeerRoster( + ctx context.Context, + blockCounter chain.BlockCounter, + retentionBlocks uint64, + metrics CutoverRosterMetricsRecorder, + clock func() time.Time, + optionFunctions ...CutoverPeerRosterOption, +) (*CutoverPeerRoster, error) { + options := &cutoverPeerRosterOptions{ + evidenceWindow: NewCutoverEvidenceWindowSignal(), + tickerFactory: newWallClockCutoverPeerRosterTicker, + } + for _, option := range optionFunctions { + if option == nil { + return nil, fmt.Errorf("nil cutover peer roster option") + } + if err := option(options); err != nil { + return nil, fmt.Errorf( + "invalid cutover peer roster option: [%w]", + err, + ) + } + } + + if retentionBlocks == 0 { + return nil, fmt.Errorf("retention blocks must be non-zero") + } + if retentionBlocks > maxSafeMetricInteger { + return nil, fmt.Errorf( + "retention blocks [%d] exceeds the maximum precisely projectable "+ + "metric value [%d]", + retentionBlocks, + maxSafeMetricInteger, + ) + } + if blockCounter == nil { + return nil, fmt.Errorf("block counter is required") + } + if metrics == nil { + return nil, fmt.Errorf("metrics recorder is required") + } + + ticker := options.tickerFactory(rosterSweepInterval) + if ticker == nil { + return nil, fmt.Errorf("cutover peer roster ticker is required") + } + + loopCtx, cancel := context.WithCancel(ctx) + + roster := &CutoverPeerRoster{ + ctx: loopCtx, + cancel: cancel, + loopDone: make(chan struct{}), + ticker: ticker, + blockCounter: blockCounter, + retentionBlocks: retentionBlocks, + cutoverBlock: options.cutoverBlock, + evidenceWindow: options.evidenceWindow, + metrics: metrics, + clock: clock, + logLimiter: rate.NewLimiter(rate.Every(30*time.Second), 5), + peers: make(map[string]*peerState), + } + + roster.processStartedAt = roster.clock() + + // Synchronously seed the current block from the chain clock. A clock error + // here is tolerated: the roster is still constructed with the clock marked + // unavailable, so it can be built unconditionally beside the gate. + if currentBlock, err := blockCounter.CurrentBlock(); err != nil { + rosterLogger.Warnf( + "cutover peer roster could not read the chain clock at "+ + "construction: [%v]; continuing with the clock marked "+ + "unavailable", + err, + ) + roster.clockAvailable = false + } else { + roster.currentBlock = currentBlock + roster.clockAvailable = true + } + + roster.initMetrics() + + go roster.run() + + return roster, nil +} + +// initMetrics registers every fixed metric at its zero value so scrapers see a +// complete metric set from the start. +func (r *CutoverPeerRoster) initMetrics() { + r.metrics.SetGauge(metricLegacyPeersCurrent, 0) + r.metrics.SetGauge(metricLegacyPeerOldestAgeBlocks, 0) + r.metrics.SetGauge(metricLegacyPeerRosterRevision, 0) + r.metrics.IncrementCounter(metricLegacyPeerAdditionsTotal, 0) + r.metrics.IncrementCounter(metricLegacyPeerEvictionsTotal, 0) +} + +func (r *CutoverPeerRoster) run() { + defer close(r.loopDone) + defer r.ticker.Stop() + + lastSnapshotLog := r.processStartedAt + + for { + select { + case <-r.ctx.Done(): + return + case <-r.ticker.Ticks(): + r.pollAndSweep() + + now := r.clock() + if now.Sub(lastSnapshotLog) >= rosterSnapshotLogInterval { + lastSnapshotLog = now + r.logSnapshotIfRequired() + } + r.ticker.MarkProcessed() + } + } +} + +// pollAndSweep reads the chain clock and either sweeps at the new height or, on +// a clock error, retains all state and evicts nothing. +func (r *CutoverPeerRoster) pollAndSweep() { + currentBlock, err := r.blockCounter.CurrentBlock() + if err != nil { + r.markClockUnavailable() + return + } + r.Sweep(currentBlock) +} + +func (r *CutoverPeerRoster) markClockUnavailable() { + r.mu.Lock() + defer r.mu.Unlock() + r.clockAvailable = false +} + +// ObserveLegacy records a single post-cutover legacy sighting. Only genuine +// stragglers are recorded: the local permit must be security-v2, the local +// (expected) format hardened, and the observed peer format legacy. Everything +// else — including a hardened observation — is ignored. Observations are +// deduplicated by operator address and by (protocol, member index). +func (r *CutoverPeerRoster) ObserveLegacy( + protocolID string, + memberIndex group.MemberIndex, + operatorAddress chain.Address, + permitMode ProtocolMode, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, +) { + if permitMode != ModeSecurityV2 { + return + } + if !expectedFormat.IsHardened() { + return + } + if observedFormat != announcer.SessionIDFormatLegacy { + return + } + // Member indexes are 1-based group positions; index 0 is never valid. + if memberIndex == 0 { + return + } + + normalized, ok := normalizeOperatorAddress(operatorAddress) + if !ok { + return + } + + // Stamp the sighting with a synchronously read chain height rather than the + // height cached by the 30-second sweep loop. Immediately after the cutover + // block C the cached height can still lag below C; a straggler stamped below C + // would be discarded by the central fleet collector as pre-cutover evidence, + // losing a genuine post-cutover legacy sighting. The read happens outside the + // lock so a slow chain call never blocks Snapshot/Sweep. + currentBlock, clockErr := r.blockCounter.CurrentBlock() + now := r.clock() + + r.mu.Lock() + defer r.mu.Unlock() + + if clockErr != nil { + // On a clock-read failure, retain existing roster state and mint no new + // sighting. Stamping a sighting with the stale cached height risks placing + // a genuinely post-cutover observation below C, where the central fleet + // collector would discard it as pre-cutover evidence — a worse outcome than + // deferring the record until the clock recovers, when the same persistent + // straggler will be re-observed with a correct height. The block-clock rule + // is to retain state and evict/record nothing on clock failure. + r.clockAvailable = false + return + } + r.currentBlock = currentBlock + r.clockAvailable = true + block := r.currentBlock + + entry, existed := r.peers[normalized] + if !existed { + entry = &peerState{ + operatorAddress: normalized, + firstSeenBlock: block, + lastSeenBlock: block, + firstSeenAt: now, + lastSeenAt: now, + sightings: make(map[sightingKey]*CutoverPeerSighting), + } + r.peers[normalized] = entry + + r.metrics.IncrementCounter(metricLegacyPeerAdditionsTotal, 1) + + if r.logLimiter.Allow() { + rosterLogger.Infof( + "protocol legacy peer entered cutover roster "+ + "[operator=%s] [protocol=%s] [member=%d] "+ + "[firstSeenBlock=%d] [cutoverBlock=%d]", + normalized, + protocolID, + memberIndex, + block, + r.cutoverBlock, + ) + } + } else { + if block < entry.firstSeenBlock { + entry.firstSeenBlock = block + entry.firstSeenAt = now + } + if block > entry.lastSeenBlock { + entry.lastSeenBlock = block + } + entry.lastSeenAt = now + } + + key := sightingKey{protocolID: protocolID, memberIndex: memberIndex} + sighting, sightingExisted := entry.sightings[key] + if !sightingExisted { + entry.sightings[key] = &CutoverPeerSighting{ + ProtocolID: protocolID, + MemberIndex: memberIndex, + FirstSeenBlock: block, + LastSeenBlock: block, + FirstSeenAt: now, + LastSeenAt: now, + } + } else { + if block < sighting.FirstSeenBlock { + sighting.FirstSeenBlock = block + sighting.FirstSeenAt = now + } + if block > sighting.LastSeenBlock { + sighting.LastSeenBlock = block + } + sighting.LastSeenAt = now + } + + r.rosterRevision++ + r.refreshMetricsLocked() +} + +// Sweep advances the roster to the given current block, evicting any operator +// whose most recent sighting is older than the retention window, and refreshes +// the metrics. It also marks the clock available, since a concrete block was +// supplied. +func (r *CutoverPeerRoster) Sweep(currentBlock uint64) { + r.mu.Lock() + defer r.mu.Unlock() + + r.currentBlock = currentBlock + r.clockAvailable = true + + for address, entry := range r.peers { + threshold := retentionThreshold(entry.lastSeenBlock, r.retentionBlocks) + if currentBlock > threshold { + delete(r.peers, address) + r.rosterRevision++ + r.metrics.IncrementCounter(metricLegacyPeerEvictionsTotal, 1) + + if r.logLimiter.Allow() { + rosterLogger.Infof( + "protocol legacy peer evicted from local cutover roster "+ + "[operator=%s] [lastSeenBlock=%d] [currentBlock=%d] "+ + "[retentionBlocks=%d] [reason=observation_expired]", + address, + entry.lastSeenBlock, + currentBlock, + r.retentionBlocks, + ) + } + } + } + + r.refreshMetricsLocked() +} + +// Snapshot returns a deterministic point-in-time view of the roster. Peers are +// sorted by operator address; each peer's sightings are sorted by protocol then +// member index. +func (r *CutoverPeerRoster) Snapshot() CutoverPeerRosterSnapshot { + r.mu.Lock() + defer r.mu.Unlock() + + return r.snapshotLocked() +} + +func (r *CutoverPeerRoster) snapshotLocked() CutoverPeerRosterSnapshot { + peers := make([]CutoverPeerEntry, 0, len(r.peers)) + + for _, entry := range r.peers { + sightings := make([]CutoverPeerSighting, 0, len(entry.sightings)) + for _, sighting := range entry.sightings { + sightings = append(sightings, *sighting) + } + sort.Slice(sightings, func(i, j int) bool { + if sightings[i].ProtocolID != sightings[j].ProtocolID { + return sightings[i].ProtocolID < sightings[j].ProtocolID + } + return sightings[i].MemberIndex < sightings[j].MemberIndex + }) + + peers = append(peers, CutoverPeerEntry{ + OperatorAddress: entry.operatorAddress, + FirstSeenBlock: entry.firstSeenBlock, + LastSeenBlock: entry.lastSeenBlock, + Sightings: sightings, + }) + } + + sort.Slice(peers, func(i, j int) bool { + return peers[i].OperatorAddress < peers[j].OperatorAddress + }) + + return CutoverPeerRosterSnapshot{ + SchemaVersion: CutoverPeerRosterSchemaVersion, + ProcessStartedAt: r.processStartedAt, + GeneratedAt: r.clock(), + CurrentBlock: r.currentBlock, + ClockAvailable: r.clockAvailable, + RetentionBlocks: r.retentionBlocks, + RosterRevision: r.rosterRevision, + Peers: peers, + } +} + +// Close stops and joins the background sweep loop. It is idempotent. +func (r *CutoverPeerRoster) Close() { + r.closeOnce.Do(func() { + r.cancel() + <-r.loopDone + }) +} + +// refreshMetricsLocked recomputes the gauge metrics from the current roster +// state. The caller must hold r.mu. +func (r *CutoverPeerRoster) refreshMetricsLocked() { + r.metrics.SetGauge(metricLegacyPeersCurrent, float64(len(r.peers))) + r.metrics.SetGauge(metricLegacyPeerRosterRevision, float64(r.rosterRevision)) + + oldestFirstSeen, hasPeers := r.oldestFirstSeenBlockLocked() + if !hasPeers || r.currentBlock < oldestFirstSeen { + r.metrics.SetGauge(metricLegacyPeerOldestAgeBlocks, 0) + return + } + r.metrics.SetGauge( + metricLegacyPeerOldestAgeBlocks, + float64(r.currentBlock-oldestFirstSeen), + ) +} + +func (r *CutoverPeerRoster) oldestFirstSeenBlockLocked() (uint64, bool) { + oldest := uint64(math.MaxUint64) + found := false + for _, entry := range r.peers { + if entry.firstSeenBlock < oldest { + oldest = entry.firstSeenBlock + found = true + } + } + return oldest, found +} + +// logSnapshotIfRequired emits the periodic roster evidence line when a +// post-cutover legacy peer is retained or when operators have explicitly +// opened an evidence window. The latter includes an empty roster because +// "zero observed peers" is itself required go/no-go and rollback evidence. +func (r *CutoverPeerRoster) logSnapshotIfRequired() { + r.mu.Lock() + defer r.mu.Unlock() + + if len(r.peers) == 0 && !r.evidenceWindow.Active() { + return + } + + oldestFirstSeen, hasPeers := r.oldestFirstSeenBlockLocked() + if !hasPeers { + oldestFirstSeen = 0 + } + rosterLogger.Infof( + "protocol cutover peer roster snapshot [currentBlock=%d] "+ + "[clockAvailable=%t] [legacyPeers=%d] [oldestFirstSeenBlock=%d] "+ + "[rosterRevision=%d]", + r.currentBlock, + r.clockAvailable, + len(r.peers), + oldestFirstSeen, + r.rosterRevision, + ) +} + +// retentionThreshold returns lastSeenBlock+retentionBlocks, saturating at +// math.MaxUint64 so overflow never turns into a spurious early eviction. +func retentionThreshold(lastSeenBlock, retentionBlocks uint64) uint64 { + threshold := lastSeenBlock + retentionBlocks + if threshold < lastSeenBlock { + return math.MaxUint64 + } + return threshold +} + +// normalizeOperatorAddress normalizes an operator address to lowercase "0x" +// followed by exactly 40 hexadecimal characters. It returns false if the input +// is not a valid 20-byte hex address. +func normalizeOperatorAddress(address chain.Address) (string, bool) { + s := strings.ToLower(strings.TrimSpace(address.String())) + s = strings.TrimPrefix(s, "0x") + + if len(s) != 40 { + return "", false + } + for _, c := range s { + if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f')) { + return "", false + } + } + + return "0x" + s, true +} diff --git a/pkg/protocol/participation/cutover_peer_roster_ownership_test.go b/pkg/protocol/participation/cutover_peer_roster_ownership_test.go new file mode 100644 index 0000000000..a1c6e9a87b --- /dev/null +++ b/pkg/protocol/participation/cutover_peer_roster_ownership_test.go @@ -0,0 +1,238 @@ +package participation + +import ( + "go/ast" + "go/parser" + "go/token" + "path/filepath" + "testing" +) + +// TestProductionCutoverRosterUsesGateSchedule holds the node-local roster to +// the same resolved one-value schedule supplied to the participation gate. The +// roster uses C only for evidence logs, but independently decoding or omitting +// it would let a post-cutover sighting name a boundary the process did not +// actually use. +func TestProductionCutoverRosterUsesGateSchedule(t *testing.T) { + repoRoot, err := filepath.Abs(filepath.Join("..", "..", "..")) + if err != nil { + t.Fatal(err) + } + + path := filepath.Join(repoRoot, "cmd", "start.go") + fileSet := token.NewFileSet() + file, err := parser.ParseFile(fileSet, path, nil, 0) + if err != nil { + t.Fatal(err) + } + + gateConstructions := 0 + rosterConstructions := 0 + var gateSchedule string + var rosterSchedule string + ast.Inspect(file, func(node ast.Node) bool { + call, ok := node.(*ast.CallExpr) + if !ok { + return true + } + selector, ok := call.Fun.(*ast.SelectorExpr) + if !ok { + return true + } + + switch selector.Sel.Name { + case "NewGate": + gateConstructions++ + if len(call.Args) < 2 { + t.Errorf( + "%s: production gate construction has no schedule", + fileSet.Position(call.Pos()), + ) + return false + } + schedule, ok := call.Args[1].(*ast.Ident) + if !ok { + t.Errorf( + "%s: production gate schedule is not one resolved value", + fileSet.Position(call.Pos()), + ) + return false + } + gateSchedule = schedule.Name + return false + case "NewCutoverPeerRoster": + rosterConstructions++ + for _, argument := range call.Args { + optionCall, ok := argument.(*ast.CallExpr) + if !ok { + continue + } + optionSelector, ok := optionCall.Fun.(*ast.SelectorExpr) + if !ok || optionSelector.Sel.Name != "WithCutoverSchedule" { + continue + } + if len(optionCall.Args) != 1 { + continue + } + schedule, ok := optionCall.Args[0].(*ast.Ident) + if ok { + rosterSchedule = schedule.Name + return false + } + } + + t.Errorf( + "%s: production cutover roster must receive the resolved "+ + "schedule used by the gate", + fileSet.Position(call.Pos()), + ) + return false + default: + return true + } + }) + + if gateConstructions != 1 { + t.Errorf( + "expected one production gate construction, found [%d]", + gateConstructions, + ) + } + if rosterConstructions != 1 { + t.Errorf( + "expected one production cutover roster construction, found [%d]", + rosterConstructions, + ) + } + if gateSchedule == "" || rosterSchedule == "" { + return + } + if gateSchedule != rosterSchedule { + t.Errorf( + "production gate uses schedule [%s] while cutover roster uses [%s]", + gateSchedule, + rosterSchedule, + ) + } +} + +// TestProductionCutoverRosterUsesOperationalEvidenceWindow holds production +// construction to one logging-only signal shared by the roster, the operator +// SIGUSR controller, and rollback quiescence. Independently constructed +// signals would make one of those controls appear wired while it changed an +// object the periodic roster logger never reads. +func TestProductionCutoverRosterUsesOperationalEvidenceWindow(t *testing.T) { + repoRoot, err := filepath.Abs(filepath.Join("..", "..", "..")) + if err != nil { + t.Fatal(err) + } + + path := filepath.Join(repoRoot, "cmd", "start.go") + fileSet := token.NewFileSet() + file, err := parser.ParseFile(fileSet, path, nil, 0) + if err != nil { + t.Fatal(err) + } + + constructions := 0 + var ( + constructedSignal string + rosterSignal string + operatorSignal string + quiescenceSignal string + ) + + ast.Inspect(file, func(node ast.Node) bool { + switch typedNode := node.(type) { + case *ast.AssignStmt: + if len(typedNode.Lhs) != 1 || len(typedNode.Rhs) != 1 { + return true + } + + call, ok := typedNode.Rhs[0].(*ast.CallExpr) + if !ok { + return true + } + selector, ok := call.Fun.(*ast.SelectorExpr) + if !ok || + selector.Sel.Name != "NewCutoverEvidenceWindowSignal" { + return true + } + + constructions++ + if signal, ok := typedNode.Lhs[0].(*ast.Ident); ok { + constructedSignal = signal.Name + } + case *ast.CallExpr: + switch functionName(typedNode.Fun) { + case "NewCutoverPeerRoster": + for _, argument := range typedNode.Args { + optionCall, ok := argument.(*ast.CallExpr) + if !ok || + functionName(optionCall.Fun) != + "WithCutoverEvidenceWindowSignal" || + len(optionCall.Args) != 1 { + continue + } + rosterSignal = expressionIdentifier(optionCall.Args[0]) + } + case "startEvidenceWindowSignalController": + if len(typedNode.Args) >= 2 { + operatorSignal = expressionIdentifier(typedNode.Args[1]) + } + case "startSignalLifecycleController": + if len(typedNode.Args) >= 4 { + quiescenceSignal = expressionIdentifier(typedNode.Args[3]) + } + } + } + + return true + }) + + if constructions != 1 { + t.Errorf( + "expected one production evidence-window signal construction, "+ + "found [%d]", + constructions, + ) + } + for owner, signal := range map[string]string{ + "roster": rosterSignal, + "operator controller": operatorSignal, + "quiescence controller": quiescenceSignal, + } { + if signal == "" { + t.Errorf("%s does not receive an evidence-window signal", owner) + continue + } + if signal != constructedSignal { + t.Errorf( + "%s uses evidence-window signal [%s], want the production "+ + "signal [%s]", + owner, + signal, + constructedSignal, + ) + } + } +} + +func functionName(expression ast.Expr) string { + switch typedExpression := expression.(type) { + case *ast.Ident: + return typedExpression.Name + case *ast.SelectorExpr: + return typedExpression.Sel.Name + default: + return "" + } +} + +func expressionIdentifier(expression ast.Expr) string { + identifier, ok := expression.(*ast.Ident) + if !ok { + return "" + } + return identifier.Name +} diff --git a/pkg/protocol/participation/cutover_peer_roster_test.go b/pkg/protocol/participation/cutover_peer_roster_test.go new file mode 100644 index 0000000000..4aaa6d8dd9 --- /dev/null +++ b/pkg/protocol/participation/cutover_peer_roster_test.go @@ -0,0 +1,1162 @@ +package participation + +import ( + "bufio" + "context" + "encoding/json" + "fmt" + "sync" + "testing" + "time" + + log2 "github.com/ipfs/go-log/v2" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +var fixedTestTime = time.Date(2026, 7, 24, 0, 0, 0, 0, time.UTC) + +func fixedClock() func() time.Time { + return func() time.Time { return fixedTestTime } +} + +type capturedRosterLogEntry struct { + Logger string `json:"logger"` + Message string `json:"msg"` +} + +func captureRosterLogs( + t *testing.T, + fn func(), +) []capturedRosterLogEntry { + return captureRosterLogsWithObserver(t, func(<-chan capturedRosterLogEntry) { + fn() + }) +} + +func captureRosterLogsWithObserver( + t *testing.T, + fn func(<-chan capturedRosterLogEntry), +) []capturedRosterLogEntry { + t.Helper() + + const subsystem = "keep-participation" + if err := log2.SetLogLevel(subsystem, "debug"); err != nil { + t.Fatal(err) + } + + pipe := log2.NewPipeReader() + + var mutex sync.Mutex + var entries []capturedRosterLogEntry + observed := make(chan capturedRosterLogEntry, 128) + done := make(chan struct{}) + go func() { + defer close(done) + + scanner := bufio.NewScanner(pipe) + for scanner.Scan() { + var entry capturedRosterLogEntry + if err := json.Unmarshal(scanner.Bytes(), &entry); err != nil { + continue + } + if entry.Logger != subsystem { + continue + } + + mutex.Lock() + entries = append(entries, entry) + mutex.Unlock() + + observed <- entry + } + }() + + fn(observed) + + if err := pipe.Close(); err != nil { + t.Logf("could not close log pipe reader: %v", err) + } + <-done + + mutex.Lock() + defer mutex.Unlock() + return append([]capturedRosterLogEntry(nil), entries...) +} + +// fakeBlockCounter is a controllable chain.BlockCounter for tests. +type fakeBlockCounter struct { + mu sync.Mutex + block uint64 + err error +} + +func newFakeBlockCounter(block uint64) *fakeBlockCounter { + return &fakeBlockCounter{block: block} +} + +func (f *fakeBlockCounter) set(block uint64, err error) { + f.mu.Lock() + defer f.mu.Unlock() + f.block = block + f.err = err +} + +func (f *fakeBlockCounter) CurrentBlock() (uint64, error) { + f.mu.Lock() + defer f.mu.Unlock() + + return f.block, f.err +} + +func (f *fakeBlockCounter) WaitForBlockHeight(uint64) error { return nil } + +func (f *fakeBlockCounter) BlockHeightWaiter(uint64) (<-chan uint64, error) { + ch := make(chan uint64, 1) + close(ch) + return ch, nil +} + +func (f *fakeBlockCounter) WatchBlocks(ctx context.Context) <-chan uint64 { + ch := make(chan uint64) + go func() { + <-ctx.Done() + close(ch) + }() + return ch +} + +type mutableRosterClock struct { + mu sync.Mutex + now time.Time +} + +func newMutableRosterClock(now time.Time) *mutableRosterClock { + return &mutableRosterClock{now: now} +} + +func (c *mutableRosterClock) set(now time.Time) { + c.mu.Lock() + defer c.mu.Unlock() + + c.now = now +} + +func (c *mutableRosterClock) Now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + + return c.now +} + +type manualRosterTicker struct { + ticks chan time.Time + processed chan struct{} + stopOnce sync.Once + stopped chan struct{} +} + +func newManualRosterTicker() *manualRosterTicker { + return &manualRosterTicker{ + ticks: make(chan time.Time, 1), + processed: make(chan struct{}, 1), + stopped: make(chan struct{}), + } +} + +func (t *manualRosterTicker) Ticks() <-chan time.Time { + return t.ticks +} + +func (t *manualRosterTicker) MarkProcessed() { + t.processed <- struct{}{} +} + +func (t *manualRosterTicker) Stop() { + t.stopOnce.Do(func() { + close(t.stopped) + }) +} + +func (t *manualRosterTicker) tick(at time.Time) { + t.ticks <- at +} + +func newRunLoopTestRoster( + t *testing.T, + initialBlock uint64, + retention uint64, + clock *mutableRosterClock, + ticker *manualRosterTicker, + options ...CutoverPeerRosterOption, +) (*CutoverPeerRoster, *fakeBlockCounter) { + t.Helper() + + blockCounter := newFakeBlockCounter(initialBlock) + tickerOption := withCutoverPeerRosterTickerFactory( + func(interval time.Duration) cutoverPeerRosterTicker { + if interval != rosterSweepInterval { + t.Fatalf( + "unexpected roster sweep interval [%s], expected [%s]", + interval, + rosterSweepInterval, + ) + } + + return ticker + }, + ) + options = append(options, tickerOption) + + roster, err := newCutoverPeerRoster( + context.Background(), + blockCounter, + retention, + newFakeMetrics(), + clock.Now, + options..., + ) + if err != nil { + t.Fatalf("failed to construct roster: [%v]", err) + } + t.Cleanup(roster.Close) + + return roster, blockCounter +} + +func driveRosterRunLoopTick( + t *testing.T, + clock *mutableRosterClock, + ticker *manualRosterTicker, + at time.Time, +) { + t.Helper() + + clock.set(at) + ticker.tick(at) + + timeout := time.NewTimer(5 * time.Second) + defer timeout.Stop() + + select { + case <-ticker.processed: + case <-timeout.C: + t.Fatal("roster run loop did not process the injected cadence tick") + } +} + +func waitForRosterLog( + t *testing.T, + observed <-chan capturedRosterLogEntry, + expected string, +) { + t.Helper() + + timeout := time.NewTimer(5 * time.Second) + defer timeout.Stop() + + for { + select { + case entry := <-observed: + if entry.Message == expected { + return + } + case <-timeout.C: + t.Fatalf("roster run loop did not emit log [%s]", expected) + } + } +} + +// fakeMetrics is a recording CutoverRosterMetricsRecorder. +type fakeMetrics struct { + mu sync.Mutex + gauges map[string]float64 + counters map[string]float64 +} + +func newFakeMetrics() *fakeMetrics { + return &fakeMetrics{ + gauges: make(map[string]float64), + counters: make(map[string]float64), + } +} + +func (m *fakeMetrics) IncrementCounter(name string, value float64) { + m.mu.Lock() + defer m.mu.Unlock() + m.counters[name] += value +} + +func (m *fakeMetrics) SetGauge(name string, value float64) { + m.mu.Lock() + defer m.mu.Unlock() + m.gauges[name] = value +} + +func (m *fakeMetrics) gauge(name string) float64 { + m.mu.Lock() + defer m.mu.Unlock() + return m.gauges[name] +} + +func (m *fakeMetrics) counter(name string) float64 { + m.mu.Lock() + defer m.mu.Unlock() + return m.counters[name] +} + +func (m *fakeMetrics) hasGauge(name string) bool { + m.mu.Lock() + defer m.mu.Unlock() + _, ok := m.gauges[name] + return ok +} + +func (m *fakeMetrics) hasCounter(name string) bool { + m.mu.Lock() + defer m.mu.Unlock() + _, ok := m.counters[name] + return ok +} + +func newTestRoster( + t *testing.T, + initialBlock uint64, + retention uint64, +) (*CutoverPeerRoster, *fakeBlockCounter, *fakeMetrics) { + t.Helper() + bc := newFakeBlockCounter(initialBlock) + metrics := newFakeMetrics() + roster, err := newCutoverPeerRoster( + context.Background(), + bc, + retention, + metrics, + fixedClock(), + ) + if err != nil { + t.Fatalf("failed to construct roster: [%v]", err) + } + t.Cleanup(roster.Close) + return roster, bc, metrics +} + +// validAddress returns a distinct valid 20-byte hex operator address for i. +func validAddress(i int) chain.Address { + return chain.Address(fmt.Sprintf("0x%040x", i)) +} + +func observeStraggler( + r *CutoverPeerRoster, + protocolID string, + memberIndex group.MemberIndex, + address chain.Address, +) { + r.ObserveLegacy( + protocolID, + memberIndex, + address, + ModeSecurityV2, + announcer.SessionIDFormatHardenedDKG, + announcer.SessionIDFormatLegacy, + ) +} + +func TestCutoverPeerRoster_ConstructionRejectsZeroRetention(t *testing.T) { + _, err := NewCutoverPeerRoster( + context.Background(), + newFakeBlockCounter(0), + 0, + newFakeMetrics(), + ) + if err == nil { + t.Fatal("expected an error for zero retention") + } +} + +func TestCutoverPeerRoster_ConstructionRejectsOverflowRetention(t *testing.T) { + _, err := NewCutoverPeerRoster( + context.Background(), + newFakeBlockCounter(0), + maxSafeMetricInteger+1, + newFakeMetrics(), + ) + if err == nil { + t.Fatal("expected an error for precision-unsafe retention") + } +} + +func TestCutoverPeerRoster_ConstructionRejectsUnprojectableCutoverSchedule( + t *testing.T, +) { + _, err := NewCutoverPeerRoster( + context.Background(), + newFakeBlockCounter(0), + 1000, + newFakeMetrics(), + WithCutoverSchedule(Schedule{ + CutoverBlock: maxSafeMetricInteger + 1, + }), + ) + if err == nil { + t.Fatal("expected an error for a precision-unsafe cutover schedule") + } +} + +func TestCutoverPeerRoster_ConstructionRejectsNilEvidenceWindowSignal( + t *testing.T, +) { + _, err := NewCutoverPeerRoster( + context.Background(), + newFakeBlockCounter(0), + 1000, + newFakeMetrics(), + WithCutoverEvidenceWindowSignal(nil), + ) + if err == nil { + t.Fatal("expected an error for a nil evidence-window signal") + } +} + +func TestCutoverPeerRoster_ConstructionInitializesMetricsAtZero(t *testing.T) { + _, _, metrics := newTestRoster(t, 100, 1000) + + for _, name := range []string{ + metricLegacyPeersCurrent, + metricLegacyPeerOldestAgeBlocks, + metricLegacyPeerRosterRevision, + } { + if !metrics.hasGauge(name) { + t.Errorf("expected gauge %q to be registered", name) + } + if metrics.gauge(name) != 0 { + t.Errorf("expected gauge %q to be initialized to zero", name) + } + } + for _, name := range []string{ + metricLegacyPeerAdditionsTotal, + metricLegacyPeerEvictionsTotal, + } { + if !metrics.hasCounter(name) { + t.Errorf("expected counter %q to be registered", name) + } + if metrics.counter(name) != 0 { + t.Errorf("expected counter %q to be initialized to zero", name) + } + } +} + +func TestCutoverPeerRoster_ConstructionClockFailureIsTolerated(t *testing.T) { + bc := newFakeBlockCounter(0) + bc.set(0, fmt.Errorf("clock unavailable")) + + roster, err := NewCutoverPeerRoster( + context.Background(), + bc, + 1000, + newFakeMetrics(), + ) + if err != nil { + t.Fatalf("construction should tolerate a clock error, got: [%v]", err) + } + t.Cleanup(roster.Close) + + snapshot := roster.Snapshot() + if snapshot.ClockAvailable { + t.Error("expected clock to be marked unavailable after a construction clock error") + } +} + +func TestCutoverPeerRoster_EntryLogIncludesResolvedCutoverBlock(t *testing.T) { + const ( + cutoverBlock = uint64(1000) + currentBlock = uint64(1005) + ) + + expected := fmt.Sprintf( + "protocol legacy peer entered cutover roster "+ + "[operator=%s] [protocol=%s] [member=%d] "+ + "[firstSeenBlock=%d] [cutoverBlock=%d]", + validAddress(1), + "tbtc-dkg", + 3, + currentBlock, + cutoverBlock, + ) + + entries := captureRosterLogs(t, func() { + roster, err := NewCutoverPeerRoster( + context.Background(), + newFakeBlockCounter(currentBlock), + 1000, + newFakeMetrics(), + WithCutoverSchedule(Schedule{CutoverBlock: cutoverBlock}), + ) + if err != nil { + t.Fatalf("failed to construct roster: [%v]", err) + } + + observeStraggler(roster, "tbtc-dkg", 3, validAddress(1)) + roster.Close() + }) + + for _, entry := range entries { + if entry.Message == expected { + return + } + } + + t.Errorf( + "expected roster entry log [%s], got: %+v", + expected, + entries, + ) +} + +func TestCutoverPeerRoster_ObserveLegacyRecordsStraggler(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + + observeStraggler(roster, "tbtc-dkg", 3, validAddress(1)) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected 1 peer, got %d", len(snapshot.Peers)) + } + peer := snapshot.Peers[0] + if peer.OperatorAddress != "0x"+fmt.Sprintf("%040x", 1) { + t.Errorf("unexpected operator address: %s", peer.OperatorAddress) + } + if len(peer.Sightings) != 1 { + t.Fatalf("expected 1 sighting, got %d", len(peer.Sightings)) + } + if peer.Sightings[0].FirstSeenBlock != 500 || peer.Sightings[0].LastSeenBlock != 500 { + t.Errorf("unexpected sighting blocks: %+v", peer.Sightings[0]) + } +} + +func TestCutoverPeerRoster_ObserveLegacyStampsFreshBlockAtCutover(t *testing.T) { + // The roster's cached height is only refreshed by the 30-second sweep loop. + // A straggler observed the instant the chain reaches the cutover block C must + // be stamped at C, not the stale C-1 the cache still holds, or the central + // fleet collector would discard it as pre-cutover evidence. + const cutover = 1000 + roster, bc, _ := newTestRoster(t, cutover-1, 1000) + + // The chain advances to C, but no sweep has run yet: the cached height is + // still C-1. + bc.set(cutover, nil) + + observeStraggler(roster, "tbtc-dkg", 3, validAddress(1)) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected 1 peer, got %d", len(snapshot.Peers)) + } + sighting := snapshot.Peers[0].Sightings[0] + if sighting.FirstSeenBlock != cutover || sighting.LastSeenBlock != cutover { + t.Errorf( + "straggler must be stamped at the fresh height C=%d, got first=%d last=%d", + cutover, sighting.FirstSeenBlock, sighting.LastSeenBlock, + ) + } + if snapshot.Peers[0].FirstSeenBlock != cutover { + t.Errorf( + "peer first-seen must reflect the fresh height C=%d, got %d", + cutover, snapshot.Peers[0].FirstSeenBlock, + ) + } +} + +func TestCutoverPeerRoster_ObserveLegacyClockErrorRetainsStateMintsNothing(t *testing.T) { + // On a clock-read failure at observation time the roster must retain existing + // state and mint NO new sighting: stamping a sighting with the stale cached + // height risks placing a genuinely post-cutover observation below C, where the + // central fleet collector would discard it as pre-cutover evidence. An + // existing entry (recorded while the clock was healthy) is preserved unchanged. + const seeded = 900 + roster, bc, _ := newTestRoster(t, seeded, 1000) + + // First observe a straggler while the clock is healthy so there is existing + // state to preserve. + observeStraggler(roster, "p", 1, validAddress(1)) + before := roster.Snapshot() + if len(before.Peers) != 1 { + t.Fatalf("precondition: expected 1 peer recorded while healthy, got %d", len(before.Peers)) + } + + // Now a different straggler is observed at the instant the clock fails. + bc.set(0, fmt.Errorf("clock unavailable")) + observeStraggler(roster, "p", 1, validAddress(2)) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf( + "a clock error must mint no new sighting; expected the 1 existing peer, got %d", + len(snapshot.Peers), + ) + } + if snapshot.Peers[0].OperatorAddress != before.Peers[0].OperatorAddress { + t.Errorf( + "existing state must be retained unchanged on a clock error: got %s, want %s", + snapshot.Peers[0].OperatorAddress, before.Peers[0].OperatorAddress, + ) + } + if snapshot.ClockAvailable { + t.Error("expected the clock to be marked unavailable after a failed read") + } +} + +func TestCutoverPeerRoster_ObserveLegacyFiltersNonStragglers(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + + // Not security-v2 -> ignored. + roster.ObserveLegacy("p", 1, validAddress(1), ModeLegacy, + announcer.SessionIDFormatHardenedDKG, announcer.SessionIDFormatLegacy) + // Expected format not hardened -> ignored. + roster.ObserveLegacy("p", 1, validAddress(2), ModeSecurityV2, + announcer.SessionIDFormatLegacy, announcer.SessionIDFormatLegacy) + // Observed format not legacy (e.g. a hardened peer) -> ignored. + roster.ObserveLegacy("p", 1, validAddress(3), ModeSecurityV2, + announcer.SessionIDFormatHardenedDKG, announcer.SessionIDFormatHardenedDKG) + // Member index zero -> ignored. + roster.ObserveLegacy("p", 0, validAddress(4), ModeSecurityV2, + announcer.SessionIDFormatHardenedDKG, announcer.SessionIDFormatLegacy) + // Invalid operator address -> ignored. + roster.ObserveLegacy("p", 1, chain.Address("not-an-address"), ModeSecurityV2, + announcer.SessionIDFormatHardenedDKG, announcer.SessionIDFormatLegacy) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 0 { + t.Fatalf("expected no peers recorded, got %d: %+v", len(snapshot.Peers), snapshot.Peers) + } +} + +func TestCutoverPeerRoster_HardenedObservationDoesNotClear(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + + observeStraggler(roster, "p", 1, validAddress(1)) + // A later hardened observation for the same operator must not clear it. + roster.ObserveLegacy("p", 1, validAddress(1), ModeSecurityV2, + announcer.SessionIDFormatHardenedDKG, announcer.SessionIDFormatHardenedDKG) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected the legacy entry to be retained, got %d peers", len(snapshot.Peers)) + } +} + +func TestCutoverPeerRoster_DedupAcrossSeatsAndReporters(t *testing.T) { + roster, _, metrics := newTestRoster(t, 500, 1000) + + address := validAddress(7) + + // The same operator observed at multiple seats (member indexes) and via + // repeated retransmissions of the same seat. + observeStraggler(roster, "tbtc-dkg", 3, address) + observeStraggler(roster, "tbtc-dkg", 3, address) // retransmission, dedup + observeStraggler(roster, "tbtc-dkg", 5, address) // different seat + observeStraggler(roster, "tbtc-signing", 3, address) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected 1 deduplicated operator, got %d", len(snapshot.Peers)) + } + // (dkg,3), (dkg,5), (signing,3) => 3 distinct sightings. + if got := len(snapshot.Peers[0].Sightings); got != 3 { + t.Fatalf("expected 3 distinct sightings, got %d", got) + } + if metrics.counter(metricLegacyPeerAdditionsTotal) != 1 { + t.Errorf( + "expected exactly one operator addition, got %v", + metrics.counter(metricLegacyPeerAdditionsTotal), + ) + } +} + +func TestCutoverPeerRoster_AddressNormalization(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + + base := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + + // The same address in different spellings must collapse to one entry. + observeStraggler(roster, "p", 1, chain.Address("0x"+base)) + observeStraggler(roster, "p", 1, chain.Address("0X"+base)) + observeStraggler(roster, "p", 1, chain.Address(base)) + observeStraggler(roster, "p", 1, chain.Address(" 0x"+base+" ")) + // Uppercase hex must also normalize to lowercase. + upper := "0xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + observeStraggler(roster, "p", 1, chain.Address(upper)) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected 1 normalized operator, got %d", len(snapshot.Peers)) + } + if snapshot.Peers[0].OperatorAddress != "0x"+base { + t.Errorf("unexpected normalized address: %s", snapshot.Peers[0].OperatorAddress) + } +} + +func TestCutoverPeerRoster_Retention(t *testing.T) { + const initialBlock = 1000 + const retention = 100 + + roster, _, _ := newTestRoster(t, initialBlock, retention) + observeStraggler(roster, "p", 1, validAddress(1)) + + // At exactly lastSeen+retention the entry is still retained. + roster.Sweep(initialBlock + retention) + if got := len(roster.Snapshot().Peers); got != 1 { + t.Fatalf("expected entry retained at the retention boundary, got %d peers", got) + } + + // One block past the window it is evicted. + roster.Sweep(initialBlock + retention + 1) + if got := len(roster.Snapshot().Peers); got != 0 { + t.Fatalf("expected entry evicted past the retention window, got %d peers", got) + } +} + +func TestCutoverPeerRoster_ClockFailureRetainsAndEvictsNothing(t *testing.T) { + roster, bc, _ := newTestRoster(t, 1000, 100) + observeStraggler(roster, "p", 1, validAddress(1)) + + // A clock failure during a poll must retain state and evict nothing, even + // though the (unread) height would be well past the retention window. + bc.set(1_000_000, fmt.Errorf("clock unavailable")) + roster.pollAndSweep() + + snapshot := roster.Snapshot() + if snapshot.ClockAvailable { + t.Error("expected clock to be marked unavailable") + } + if len(snapshot.Peers) != 1 { + t.Fatalf("expected entry retained on clock failure, got %d peers", len(snapshot.Peers)) + } + + // When the clock recovers, normal sweeping resumes. + bc.set(1_000_000, nil) + roster.pollAndSweep() + if got := len(roster.Snapshot().Peers); got != 0 { + t.Fatalf("expected entry evicted after clock recovery, got %d peers", got) + } +} + +func TestCutoverPeerRoster_RestartStartsEmpty(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + observeStraggler(roster, "p", 1, validAddress(1)) + roster.Close() + + // The node-local roster is in-memory: a fresh process starts empty. + fresh, _, _ := newTestRoster(t, 500, 1000) + if got := len(fresh.Snapshot().Peers); got != 0 { + t.Fatalf("expected a fresh roster to be empty, got %d peers", got) + } +} + +func TestCutoverPeerRoster_47From3Versus47From47(t *testing.T) { + // 47 sightings distributed over 3 operator addresses -> 3 operators that + // together carry all 47 sightings. + rosterA, _, _ := newTestRoster(t, 500, 100000) + counts := []int{16, 16, 15} // 47 total + for opIndex, seats := range counts { + for seat := 1; seat <= seats; seat++ { + observeStraggler(rosterA, "p", group.MemberIndex(seat), validAddress(opIndex)) + } + } + snapA := rosterA.Snapshot() + if len(snapA.Peers) != 3 { + t.Fatalf("expected 3 operators, got %d", len(snapA.Peers)) + } + totalSightings := 0 + for _, p := range snapA.Peers { + totalSightings += len(p.Sightings) + } + if totalSightings != 47 { + t.Fatalf("expected 47 total sightings across 3 operators, got %d", totalSightings) + } + + // 47 sightings from 47 distinct addresses -> 47 operators. + rosterB, _, _ := newTestRoster(t, 500, 100000) + for i := 0; i < 47; i++ { + observeStraggler(rosterB, "p", 1, validAddress(1000+i)) + } + snapB := rosterB.Snapshot() + if len(snapB.Peers) != 47 { + t.Fatalf("expected 47 operators, got %d", len(snapB.Peers)) + } +} + +func TestCutoverPeerRoster_DeterministicSnapshotJSON(t *testing.T) { + build := func(order []int) []byte { + t.Helper() + roster, _, _ := newTestRoster(t, 1000, 100000) + for _, i := range order { + observeStraggler(roster, "tbtc-dkg", group.MemberIndex(i+1), validAddress(i)) + observeStraggler(roster, "tbtc-signing", group.MemberIndex(i+1), validAddress(i)) + } + data, err := json.Marshal(roster.Snapshot()) + if err != nil { + t.Fatalf("failed to marshal snapshot: [%v]", err) + } + return data + } + + forward := build([]int{0, 1, 2, 3, 4}) + shuffled := build([]int{3, 1, 4, 0, 2}) + + if string(forward) != string(shuffled) { + t.Errorf( + "snapshot JSON is not deterministic across insertion orders\nforward: %s\nshuffled: %s", + forward, + shuffled, + ) + } +} + +func TestCutoverPeerRoster_Metrics(t *testing.T) { + const initialBlock = 1000 + const retention = 100 + + roster, _, metrics := newTestRoster(t, initialBlock, retention) + + observeStraggler(roster, "p", 1, validAddress(1)) + if metrics.gauge(metricLegacyPeersCurrent) != 1 { + t.Errorf("expected peers_current=1, got %v", metrics.gauge(metricLegacyPeersCurrent)) + } + if metrics.counter(metricLegacyPeerAdditionsTotal) != 1 { + t.Errorf("expected additions_total=1, got %v", metrics.counter(metricLegacyPeerAdditionsTotal)) + } + if metrics.gauge(metricLegacyPeerRosterRevision) < 1 { + t.Errorf("expected roster_revision>=1, got %v", metrics.gauge(metricLegacyPeerRosterRevision)) + } + + // Oldest age = current block - oldest first-seen block. + roster.Sweep(initialBlock + 5) + if metrics.gauge(metricLegacyPeerOldestAgeBlocks) != 5 { + t.Errorf("expected oldest_age_blocks=5, got %v", metrics.gauge(metricLegacyPeerOldestAgeBlocks)) + } + + // Evict and confirm counters/gauges. + roster.Sweep(initialBlock + retention + 1) + if metrics.counter(metricLegacyPeerEvictionsTotal) != 1 { + t.Errorf("expected evictions_total=1, got %v", metrics.counter(metricLegacyPeerEvictionsTotal)) + } + if metrics.gauge(metricLegacyPeersCurrent) != 0 { + t.Errorf("expected peers_current=0 after eviction, got %v", metrics.gauge(metricLegacyPeersCurrent)) + } +} + +func TestCutoverPeerRoster_CloseIdempotent(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + roster.Close() + roster.Close() // must not panic or block +} + +// TestCutoverPeerRoster_ConcurrentCloseSafe races two Close calls on a real +// roster with a live background sweep loop. Each Close joins the sweep loop +// under sync.Once, so both concurrent callers must return without panicking, +// double-closing the join channel, or blocking forever on the join; a later +// Close after shutdown must remain a safe no-op. +func TestCutoverPeerRoster_ConcurrentCloseSafe(t *testing.T) { + roster, _, _ := newTestRoster(t, 500, 1000) + + firstDone := make(chan struct{}) + secondDone := make(chan struct{}) + go func() { + roster.Close() + close(firstDone) + }() + go func() { + roster.Close() + close(secondDone) + }() + + for _, done := range []<-chan struct{}{firstDone, secondDone} { + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal( + "a concurrent Close did not return; Close did not safely " + + "join the sweep loop under the double-close overlap", + ) + } + } + + thirdDone := make(chan struct{}) + go func() { + roster.Close() + close(thirdDone) + }() + select { + case <-thirdDone: + case <-time.After(5 * time.Second): + t.Fatal("a Close after shutdown blocked; Close is not idempotent") + } +} + +func TestCutoverPeerRoster_RunLogsEmptySnapshotDuringEvidenceWindowAtCadence( + t *testing.T, +) { + const currentBlock = uint64(1005) + + expected := fmt.Sprintf( + "protocol cutover peer roster snapshot [currentBlock=%d] "+ + "[clockAvailable=true] [legacyPeers=0] "+ + "[oldestFirstSeenBlock=0] [rosterRevision=0]", + currentBlock, + ) + + entries := captureRosterLogsWithObserver(t, func( + observed <-chan capturedRosterLogEntry, + ) { + evidenceWindow := NewCutoverEvidenceWindowSignal() + evidenceWindow.SetActive(true) + clock := newMutableRosterClock(fixedTestTime) + ticker := newManualRosterTicker() + roster, _ := newRunLoopTestRoster( + t, + currentBlock, + 1000, + clock, + ticker, + WithCutoverEvidenceWindowSignal(evidenceWindow), + ) + + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(rosterSnapshotLogInterval-time.Second), + ) + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(rosterSnapshotLogInterval), + ) + waitForRosterLog(t, observed, expected) + roster.Close() + + select { + case <-ticker.stopped: + default: + t.Error("roster run loop did not stop its cadence source") + } + }) + + assertRosterLogCount(t, entries, expected, 1) +} + +func TestCutoverPeerRoster_RunLogsNonemptySnapshotAfterCutover( + t *testing.T, +) { + const ( + cutoverBlock = uint64(1000) + currentBlock = cutoverBlock + 5 + ) + + expected := fmt.Sprintf( + "protocol cutover peer roster snapshot [currentBlock=%d] "+ + "[clockAvailable=true] [legacyPeers=1] "+ + "[oldestFirstSeenBlock=%d] [rosterRevision=1]", + currentBlock, + currentBlock, + ) + + entries := captureRosterLogsWithObserver(t, func( + observed <-chan capturedRosterLogEntry, + ) { + evidenceWindow := NewCutoverEvidenceWindowSignal() + clock := newMutableRosterClock(fixedTestTime) + ticker := newManualRosterTicker() + roster, _ := newRunLoopTestRoster( + t, + currentBlock, + 1000, + clock, + ticker, + WithCutoverSchedule(Schedule{CutoverBlock: cutoverBlock}), + WithCutoverEvidenceWindowSignal(evidenceWindow), + ) + + observeStraggler(roster, "tbtc-dkg", 3, validAddress(1)) + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(rosterSnapshotLogInterval), + ) + waitForRosterLog(t, observed, expected) + roster.Close() + }) + + assertRosterLogCount(t, entries, expected, 1) +} + +func TestCutoverPeerRoster_RunLogsClockUnavailableSnapshotDuringEvidenceWindow( + t *testing.T, +) { + const lastCurrentBlock = uint64(900) + + expected := fmt.Sprintf( + "protocol cutover peer roster snapshot [currentBlock=%d] "+ + "[clockAvailable=false] [legacyPeers=0] "+ + "[oldestFirstSeenBlock=0] [rosterRevision=0]", + lastCurrentBlock, + ) + + entries := captureRosterLogsWithObserver(t, func( + observed <-chan capturedRosterLogEntry, + ) { + evidenceWindow := NewCutoverEvidenceWindowSignal() + evidenceWindow.SetActive(true) + clock := newMutableRosterClock(fixedTestTime) + ticker := newManualRosterTicker() + roster, blockCounter := newRunLoopTestRoster( + t, + lastCurrentBlock, + 1000, + clock, + ticker, + WithCutoverEvidenceWindowSignal(evidenceWindow), + ) + + blockCounter.set(0, fmt.Errorf("clock unavailable")) + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(rosterSnapshotLogInterval), + ) + waitForRosterLog(t, observed, expected) + roster.Close() + }) + + assertRosterLogCount(t, entries, expected, 1) +} + +func TestCutoverPeerRoster_RunHonorsEvidenceWindowOnOffAtCadence(t *testing.T) { + const currentBlock = uint64(1005) + + expected := fmt.Sprintf( + "protocol cutover peer roster snapshot [currentBlock=%d] "+ + "[clockAvailable=true] [legacyPeers=0] "+ + "[oldestFirstSeenBlock=0] [rosterRevision=0]", + currentBlock, + ) + + entries := captureRosterLogsWithObserver(t, func( + observed <-chan capturedRosterLogEntry, + ) { + evidenceWindow := NewCutoverEvidenceWindowSignal() + clock := newMutableRosterClock(fixedTestTime) + ticker := newManualRosterTicker() + roster, _ := newRunLoopTestRoster( + t, + currentBlock, + 1000, + clock, + ticker, + WithCutoverEvidenceWindowSignal(evidenceWindow), + ) + + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(rosterSnapshotLogInterval), + ) + if !evidenceWindow.SetActive(true) { + t.Error("expected opening the evidence window to change its state") + } + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(2*rosterSnapshotLogInterval), + ) + waitForRosterLog(t, observed, expected) + if !evidenceWindow.SetActive(false) { + t.Error("expected closing the evidence window to change its state") + } + driveRosterRunLoopTick( + t, + clock, + ticker, + fixedTestTime.Add(3*rosterSnapshotLogInterval), + ) + roster.Close() + }) + + assertRosterLogCount(t, entries, expected, 1) +} + +func TestCutoverEvidenceWindowSignal_HoldActive(t *testing.T) { + evidenceWindow := NewCutoverEvidenceWindowSignal() + + if evidenceWindow.Active() { + t.Fatal("a new evidence window must be inactive") + } + if !evidenceWindow.HoldActive() { + t.Error("holding an inactive evidence window should change its state") + } + if !evidenceWindow.Active() { + t.Fatal("a held evidence window must be active") + } + if evidenceWindow.SetActive(false) { + t.Error("a held evidence window must reject a close") + } + if !evidenceWindow.Active() { + t.Fatal("a rejected close must leave the evidence window active") + } +} + +func TestCutoverEvidenceWindowSignal_ConcurrentAccess(t *testing.T) { + evidenceWindow := NewCutoverEvidenceWindowSignal() + + var workers sync.WaitGroup + for worker := 0; worker < 32; worker++ { + workers.Add(1) + go func(worker int) { + defer workers.Done() + + for iteration := 0; iteration < 100; iteration++ { + evidenceWindow.SetActive((worker+iteration)%2 == 0) + _ = evidenceWindow.Active() + } + }(worker) + } + workers.Wait() + + evidenceWindow.HoldActive() + if !evidenceWindow.Active() { + t.Fatal("the evidence window must remain active after a concurrent hold") + } +} + +func assertRosterLogCount( + t *testing.T, + entries []capturedRosterLogEntry, + expected string, + expectedCount int, +) { + t.Helper() + + actualCount := 0 + for _, entry := range entries { + if entry.Message == expected { + actualCount++ + } + } + if actualCount != expectedCount { + t.Errorf( + "expected roster log [%s] %d time(s), got %d in: %+v", + expected, + expectedCount, + actualCount, + entries, + ) + } +} diff --git a/pkg/protocol/participation/diagnostics.go b/pkg/protocol/participation/diagnostics.go new file mode 100644 index 0000000000..d842333f06 --- /dev/null +++ b/pkg/protocol/participation/diagnostics.go @@ -0,0 +1,162 @@ +package participation + +import ( + "encoding/json" + "math/big" + + "github.com/ipfs/go-log" +) + +var diagnosticsLogger = log.Logger("keep-participation") + +const ( + // DiagnosticsSourceCutoverLegacyPeers is the diagnostics object carrying + // the node-local cutover peer roster snapshot. + DiagnosticsSourceCutoverLegacyPeers = "cutover_legacy_peers" + // DiagnosticsSourceProtocolParticipation is the diagnostics object + // carrying the gate's identity and live state. + DiagnosticsSourceProtocolParticipation = "protocol_participation" +) + +// DiagnosticsRegistry is the minimal diagnostics sink the participation +// sources need. It is satisfied by the client-info registry. +type DiagnosticsRegistry interface { + RegisterDiagnosticSource(name string, source func() string) +} + +// RosterSnapshotSource surfaces the node-local cutover peer roster. +type RosterSnapshotSource interface { + Snapshot() CutoverPeerRosterSnapshot +} + +// ChainIdentitySource surfaces the chain id of the endpoint this node is +// actually connected to. +// +// It is taken from the connected endpoint rather than from the configuration +// because a cutover block means nothing without the chain it counts on: a node +// armed with the right C against the wrong chain is exactly the +// misconfiguration a readiness scrape has to be able to see, and a configured +// value would only ever agree with itself. +type ChainIdentitySource interface { + ChainID() *big.Int +} + +// RegisterDiagnosticSources registers the cutover observability contract on +// the given diagnostics registry: the node-local legacy-peer roster, and the +// gate's identity and live state. +// +// Registration happens on the production startup path, so this function is the +// single definition of both the source names and the emitted field set — a +// scrape reads exactly what is registered here. +func RegisterDiagnosticSources( + registry DiagnosticsRegistry, + gate Gate, + roster RosterSnapshotSource, + chainIdentity ChainIdentitySource, + cutoverBlockSource string, +) { + // Expose the node-local cutover peer roster snapshot as a top-level + // diagnostics object so port-enabled nodes surface which operators are + // observed on the legacy release across the cutover. + registry.RegisterDiagnosticSource( + DiagnosticsSourceCutoverLegacyPeers, + func() string { + bytes, err := json.Marshal(roster.Snapshot()) + if err != nil { + diagnosticsLogger.Errorf( + "error on serializing cutover peer roster to JSON: [%v]", + err, + ) + return "" + } + return string(bytes) + }, + ) + + // Expose the gate's identity and live state so a diagnostics scrape + // answers the readiness questions directly: which epoch this artifact is, + // which chain it clocks itself against, which cutover block it compiled or + // resolved, and what the gate is doing right now. + registry.RegisterDiagnosticSource( + DiagnosticsSourceProtocolParticipation, + func() string { + // The chain id is re-read per scrape from the connected endpoint + // so the emitted value can never outlive the handle it describes. + chainID := chainIdentity.ChainID() + if chainID == nil { + diagnosticsLogger.Errorf( + "connected chain reported no chain id; " + + "participation diagnostics cannot be composed", + ) + return "" + } + + snapshot := gate.State() + // The permits themselves, and not only how many there are. A + // count answers "this node holds two legacy ceremonies" and + // nothing more, so an observer watching work cross the cutover + // has to fall back on comparing fleet-wide totals — which any + // two unrelated ceremonies moving in step will satisfy. The + // identities let it name the permit that crossed instead. The + // fields are the same stable public identities the quiescence + // record carries; no key material or protocol input is exposed. + permits := snapshot.ActivePermits + if permits == nil { + permits = []PermitSnapshot{} + } + // What became of the permits that are no longer here. A live + // reading names the work a node is holding; without this, the + // moment that work finishes it leaves nothing behind, and an + // observer following work across the cutover has to take some + // other party's word for how it ended. These are the node's own + // records, written by each ceremony's owner, and they carry the + // same permit identities the live list does — so a permit seen + // held can be followed to the disposition its owner recorded for + // it. Public identities only, exactly as the quiescence journal + // that outlives them carries. + terminalOutcomes := snapshot.RecentTerminalOutcomes + if terminalOutcomes == nil { + terminalOutcomes = []TerminalOutcomeRecord{} + } + // Whether the account above can be followed at all. It lives in + // memory, so a reader joining a permit it saw held to the ending its + // holder recorded has to be able to tell an account that never held + // the record from one that held it and lost it — to a restart, or to + // its own bound. Those are opposite answers: the first says this node + // did not do that work, the second says nobody can say what it did, + // and a reader that cannot distinguish them attributes the work to + // whoever else was on the network. + gateInstance := snapshot.GateInstance + if gateInstance == "" { + gateInstance = "unknown" + } + bytes, err := json.Marshal(map[string]interface{}{ + "protocol_epoch": CompiledEpoch.String(), + "ethereum_chain_id": chainID.String(), + "cutover_block": snapshot.CutoverBlock, + "cutover_block_source": cutoverBlockSource, + "gate_state": snapshot.State.String(), + "current_block": snapshot.CurrentBlock, + "clock_available": snapshot.ClockAvailable, + "allowed": snapshot.Allowed, + "quiescing": snapshot.Quiescing, + "active_ceremonies": snapshot.ActiveCeremonies, + "active_legacy_ceremonies": snapshot.ActiveLegacyCeremonies, + "active_security_v2_ceremonies": snapshot.ActiveSecurityV2Ceremonies, + "active_permits": permits, + "recent_terminal_outcomes": terminalOutcomes, + "gate_instance": gateInstance, + "forgotten_terminal_outcomes": snapshot. + ForgottenTerminalOutcomes, + }) + if err != nil { + diagnosticsLogger.Errorf( + "error on serializing participation state to JSON: [%v]", + err, + ) + return "" + } + return string(bytes) + }, + ) +} diff --git a/pkg/protocol/participation/diagnostics_test.go b/pkg/protocol/participation/diagnostics_test.go new file mode 100644 index 0000000000..9c63e2b046 --- /dev/null +++ b/pkg/protocol/participation/diagnostics_test.go @@ -0,0 +1,662 @@ +package participation + +import ( + "encoding/json" + "math/big" + "reflect" + "sort" + "strings" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// recordingRegistry captures what the production registration path registers, +// so a test reads back exactly the source names and payloads a diagnostics +// scrape would. +type recordingRegistry struct { + sources map[string]func() string + order []string +} + +func newRecordingRegistry() *recordingRegistry { + return &recordingRegistry{sources: make(map[string]func() string)} +} + +func (r *recordingRegistry) RegisterDiagnosticSource( + name string, + source func() string, +) { + if _, seen := r.sources[name]; !seen { + r.order = append(r.order, name) + } + r.sources[name] = source +} + +// scrape invokes the named source the way the client-info server does and +// decodes its JSON object. +func (r *recordingRegistry) scrape( + t *testing.T, + name string, +) map[string]interface{} { + t.Helper() + + source, ok := r.sources[name] + if !ok { + t.Fatalf( + "diagnostics source [%v] was never registered; registered: %v", + name, + r.order, + ) + } + + raw := source() + if raw == "" { + t.Fatalf("diagnostics source [%v] produced an empty payload", name) + } + + var decoded map[string]interface{} + if err := json.Unmarshal([]byte(raw), &decoded); err != nil { + t.Fatalf( + "diagnostics source [%v] produced undecodable JSON [%v]: [%v]", + name, + raw, + err, + ) + } + + return decoded +} + +// stubChainIdentity stands in for the connected chain handle. It counts reads +// so a test can prove the emitted chain id comes from the connected endpoint +// on every scrape rather than from a value captured elsewhere. +type stubChainIdentity struct { + chainID *big.Int + reads int +} + +func (s *stubChainIdentity) ChainID() *big.Int { + s.reads++ + if s.chainID == nil { + return nil + } + return new(big.Int).Set(s.chainID) +} + +func TestRegisterDiagnosticSources_RegistersBothSources(t *testing.T) { + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 900, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 900, 100) + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + &stubChainIdentity{chainID: big.NewInt(1)}, + "release_baked", + ) + + registered := append([]string(nil), registry.order...) + sort.Strings(registered) + + expected := []string{ + DiagnosticsSourceCutoverLegacyPeers, + DiagnosticsSourceProtocolParticipation, + } + if !reflect.DeepEqual(registered, expected) { + t.Errorf( + "unexpected diagnostics sources\nactual: %v\nexpected: %v", + registered, + expected, + ) + } +} + +func TestRegisterDiagnosticSources_EmitsConnectedChainID(t *testing.T) { + // A chain id that no configuration default and no test schedule value + // could coincidentally produce, so the assertion can only pass if the + // emitted value was read from the connected chain handle. + const connectedChainID = 424242 + + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 900, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 900, 100) + chainIdentity := &stubChainIdentity{ + chainID: big.NewInt(connectedChainID), + } + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + chainIdentity, + "release_baked", + ) + + decoded := registry.scrape(t, DiagnosticsSourceProtocolParticipation) + + actual, ok := decoded["ethereum_chain_id"] + if !ok { + t.Fatalf( + "participation diagnostics omit the connected chain id: %v", + decoded, + ) + } + if actual != "424242" { + t.Errorf( + "unexpected chain id\nactual: %v\nexpected: %v", + actual, + "424242", + ) + } + if chainIdentity.reads == 0 { + t.Errorf("the connected chain handle was never read") + } +} + +func TestRegisterDiagnosticSources_RereadsChainIDPerScrape(t *testing.T) { + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 900, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 900, 100) + chainIdentity := &stubChainIdentity{chainID: big.NewInt(11155111)} + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + chainIdentity, + "non_mainnet_override", + ) + + registry.scrape(t, DiagnosticsSourceProtocolParticipation) + firstReads := chainIdentity.reads + + registry.scrape(t, DiagnosticsSourceProtocolParticipation) + if chainIdentity.reads <= firstReads { + t.Errorf( + "the chain id was captured once instead of read per scrape "+ + "[reads after first scrape=%v] [reads after second=%v]", + firstReads, + chainIdentity.reads, + ) + } +} + +func TestRegisterDiagnosticSources_RefusesPayloadWithoutChainID(t *testing.T) { + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 900, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 900, 100) + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + &stubChainIdentity{chainID: nil}, + "release_baked", + ) + + // A payload that silently dropped the chain id would still decode and + // would read as a healthy scrape, so the source refuses to compose one. + if payload := registry.sources[DiagnosticsSourceProtocolParticipation](); payload != "" { + t.Errorf( + "participation diagnostics were composed without a chain id: %v", + payload, + ) + } +} + +func TestRegisterDiagnosticSources_EmitsGateStateContract(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 1200, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 1200, 100) + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + &stubChainIdentity{chainID: big.NewInt(1)}, + "release_baked", + ) + + // Hold one live security-v2 permit so the active counters describe real + // gate state rather than an idle zero that any broken payload would match. + // It is identity-bound because the emitted permit list is only useful to an + // observer if it names the work each permit was issued for. + permit, err := gate.Begin( + TBTCSigning, + 1100, + PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "1", + OperatedMembers: MemberIndexes{4}, + }, + ) + if err != nil { + t.Fatalf("failed to begin a ceremony: [%v]", err) + } + defer permit.Close() + + decoded := registry.scrape(t, DiagnosticsSourceProtocolParticipation) + + snapshot := gate.State() + expected := map[string]interface{}{ + "protocol_epoch": CompiledEpoch.String(), + "ethereum_chain_id": "1", + "cutover_block": float64(snapshot.CutoverBlock), + "cutover_block_source": "release_baked", + "gate_state": snapshot.State.String(), + "current_block": float64(snapshot.CurrentBlock), + "clock_available": snapshot.ClockAvailable, + "allowed": snapshot.Allowed, + "quiescing": snapshot.Quiescing, + "active_ceremonies": float64(snapshot.ActiveCeremonies), + "active_legacy_ceremonies": float64(snapshot.ActiveLegacyCeremonies), + "active_security_v2_ceremonies": float64(snapshot.ActiveSecurityV2Ceremonies), + "active_permits": []interface{}{ + map[string]interface{}{ + "ceremony": string(TBTCSigning), + "mode": ModeSecurityV2.String(), + "canonical_start_block": float64(1100), + "work_id": strings.Repeat("a", 64), + "permit_id": "1", + "identity_bound": true, + // The seats this permit's holder operates, published while the + // permit is still live. A reader assembling who operated which + // seat on this work reads them from here rather than waiting to + // see whether the ceremony produces a result. + "operated_members": []interface{}{float64(4)}, + }, + }, + "recent_terminal_outcomes": []interface{}{}, + // What a reader needs in order to trust the account above. It lives in + // memory, so an empty one is either a node that closed no permit or a + // node that closed one and lost the record; the instance says whether two + // readings came from the same process, and the count says whether the + // account has dropped anything to its own bound. + "gate_instance": snapshot.GateInstance, + "forgotten_terminal_outcomes": float64(0), + } + + if snapshot.GateInstance == "" { + t.Error("the gate published no instance identity") + } + + if !reflect.DeepEqual(decoded, expected) { + t.Errorf( + "unexpected participation diagnostics\nactual: %v\nexpected: %v", + decoded, + expected, + ) + } + + // The held permit must be visible; a payload hard-coding zeros would + // otherwise satisfy the comparison above on an idle gate. + if snapshot.ActiveSecurityV2Ceremonies != 1 { + t.Errorf( + "unexpected active security-v2 ceremonies\n"+ + "actual: %v\nexpected: %v", + snapshot.ActiveSecurityV2Ceremonies, + 1, + ) + } + + // Guard the chain-clock field against a stale capture the same way the + // chain id is guarded: it must follow the counter the gate reads. + blockCounter.set(1300, nil) + if _, err := gate.Begin(TBTCHeartbeat, 1250); err != nil { + t.Fatalf("failed to begin a second ceremony: [%v]", err) + } + if current := registry.scrape( + t, + DiagnosticsSourceProtocolParticipation, + )["current_block"]; current != float64(1300) { + t.Errorf( + "unexpected current block\nactual: %v\nexpected: %v", + current, + float64(1300), + ) + } +} + +// TestRegisterDiagnosticSources_NamesWhatBecameOfClosedPermits asserts a scrape +// can follow a permit past the point it stops being held. +// +// The held-permit list names work while the node has it. The moment the work +// finishes, the permit leaves that list and the scrape says nothing further — +// so an observer watching work cross the cutover can see it held and then has +// to take some other party's report for how it ended. A report about a ceremony +// is not evidence about a ceremony. The emitted record is the node's own, and +// carries the same permit identity the held list does, so the two readings join. +func TestRegisterDiagnosticSources_NamesWhatBecameOfClosedPermits(t *testing.T) { + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 1200, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 1200, 100) + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + &stubChainIdentity{chainID: big.NewInt(1)}, + "release_baked", + ) + + permit, err := gate.Begin( + TBTCSigning, + 1100, + PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "1", + OperatedMembers: MemberIndexes{4}, + }, + ) + if err != nil { + t.Fatalf("failed to begin a ceremony: [%v]", err) + } + if err := permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidenceBitcoinTransaction, + Reference: "signed-transaction-hash", + Contribution: &TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 4, 9}, + LocalMembers: []group.MemberIndex{4}, + }, + }, + ); err != nil { + t.Fatalf("failed to record a terminal outcome: [%v]", err) + } + permit.Close() + + decoded := registry.scrape(t, DiagnosticsSourceProtocolParticipation) + + if held, _ := decoded["active_permits"].([]interface{}); len(held) != 0 { + t.Fatalf("a closed permit is still emitted as held: %v", held) + } + + outcomes, _ := decoded["recent_terminal_outcomes"].([]interface{}) + if len(outcomes) != 1 { + t.Fatalf( + "expected the closed permit to be accounted for, got: %v", + decoded["recent_terminal_outcomes"], + ) + } + record, _ := outcomes[0].(map[string]interface{}) + if record["outcome"] != string(TerminalOutcomeCompleted) { + t.Errorf("unexpected emitted outcome: %v", record["outcome"]) + } + + // The permit identity is the join to the held reading; a disposition that + // named no permit would say only that something somewhere finished. + emittedPermit, _ := record["permit"].(map[string]interface{}) + if emittedPermit["work_id"] != strings.Repeat("a", 64) || + emittedPermit["permit_id"] != "1" || + emittedPermit["ceremony"] != string(TBTCSigning) || + emittedPermit["mode"] != ModeSecurityV2.String() { + t.Errorf("the closed permit is not identified: %v", record["permit"]) + } + + evidence, _ := record["evidence"].(map[string]interface{}) + if evidence["kind"] != string(TerminalEvidenceBitcoinTransaction) || + evidence["reference"] != "signed-transaction-hash" { + t.Errorf("the owner's evidence is not emitted: %v", record["evidence"]) + } + + // Who was in the transcript, and which of them this node was. Without both + // lists a scrape can see that a result exists and that this node recorded + // it, and still cannot tell a ceremony several parties produced together + // from one this node produced alone — which is the entire question a + // mixed-release reading asks. + contribution, _ := evidence["contribution"].(map[string]interface{}) + if !emittedMemberIndexesEqual( + contribution["incorporated_members"], + []group.MemberIndex{1, 4, 9}, + ) || !emittedMemberIndexesEqual( + contribution["local_members"], + []group.MemberIndex{4}, + ) { + t.Errorf( + "the transcript behind the result is not emitted: %v", + evidence["contribution"], + ) + } +} + +// emittedMemberIndexesEqual reports whether a decoded JSON member-index list +// holds exactly the expected indexes in order. JSON numbers decode as float64, +// so the comparison has to go through the emitted representation rather than +// through the typed slice the gate was given. +func emittedMemberIndexesEqual( + emitted interface{}, + expected []group.MemberIndex, +) bool { + indexes, ok := emitted.([]interface{}) + if !ok || len(indexes) != len(expected) { + return false + } + for i, index := range indexes { + number, ok := index.(float64) + if !ok || group.MemberIndex(number) != expected[i] { + return false + } + } + + return true +} + +// The emitted permit list is what lets an observer name the work a node is +// holding rather than compare fleet-wide totals, and totals are precisely what +// cannot distinguish two permits on opposite sides of the cutover from any +// other pair. +func TestRegisterDiagnosticSources_NamesEachHeldPermit(t *testing.T) { + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 1200, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 1200, 100) + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + &stubChainIdentity{chainID: big.NewInt(1)}, + "release_baked", + ) + + // An idle gate emits an empty array rather than a null. A scrape that + // cannot tell "this node holds nothing" from "this build has no such + // field" would read a missing contract as an idle node. + idle := registry.scrape(t, DiagnosticsSourceProtocolParticipation) + held, ok := idle["active_permits"].([]interface{}) + if !ok || len(held) != 0 { + t.Fatalf( + "unexpected idle permit list\nactual: %#v\nexpected: []", + idle["active_permits"], + ) + } + + legacyWorkID := strings.Repeat("b", 64) + securityWorkID := strings.Repeat("c", 64) + + // One permit anchored below the cutover and one above it, held at the same + // time. Their count is what any other pair of permits would produce; only + // the identities say which side of C each one was issued on. + legacy, err := gate.Begin( + TBTCSigning, + 900, + PermitIdentity{WorkID: legacyWorkID, PermitID: "1"}, + ) + if err != nil { + t.Fatalf("failed to begin the legacy-anchored ceremony: [%v]", err) + } + defer legacy.Close() + + securityV2, err := gate.Begin( + BeaconDKG, + 1100, + PermitIdentity{ + WorkID: securityWorkID, + PermitID: "2", + OperatedMembers: MemberIndexes{2}, + }, + ) + if err != nil { + t.Fatalf("failed to begin the security-v2 ceremony: [%v]", err) + } + defer securityV2.Close() + + decoded := registry.scrape(t, DiagnosticsSourceProtocolParticipation) + emitted, ok := decoded["active_permits"].([]interface{}) + if !ok || len(emitted) != 2 { + t.Fatalf( + "unexpected permit list\nactual: %#v\nexpected: 2 permits", + decoded["active_permits"], + ) + } + + modes := make(map[string]string, len(emitted)) + anchors := make(map[string]float64, len(emitted)) + for _, entry := range emitted { + permit, ok := entry.(map[string]interface{}) + if !ok { + t.Fatalf("unexpected permit entry: %#v", entry) + } + workID, _ := permit["work_id"].(string) + modes[workID], _ = permit["mode"].(string) + anchors[workID], _ = permit["canonical_start_block"].(float64) + } + + expectedModes := map[string]string{ + legacyWorkID: ModeLegacy.String(), + securityWorkID: ModeSecurityV2.String(), + } + if !reflect.DeepEqual(modes, expectedModes) { + t.Errorf( + "unexpected permit modes\nactual: %v\nexpected: %v", + modes, + expectedModes, + ) + } + + expectedAnchors := map[string]float64{ + legacyWorkID: float64(900), + securityWorkID: float64(1100), + } + if !reflect.DeepEqual(anchors, expectedAnchors) { + t.Errorf( + "unexpected permit anchors\nactual: %v\nexpected: %v", + anchors, + expectedAnchors, + ) + } + + // A closed permit leaves the list, so an observer reading it after the + // crossing sees what is still held rather than everything ever issued. + legacy.Close() + + after := registry.scrape(t, DiagnosticsSourceProtocolParticipation) + remaining, ok := after["active_permits"].([]interface{}) + if !ok || len(remaining) != 1 { + t.Fatalf( + "unexpected permit list after close\nactual: %#v\n"+ + "expected: 1 permit", + after["active_permits"], + ) + } + if permit, ok := remaining[0].(map[string]interface{}); !ok || + permit["work_id"] != securityWorkID { + t.Errorf( + "unexpected surviving permit\nactual: %#v\nexpected: %v", + remaining[0], + securityWorkID, + ) + } +} + +func TestRegisterDiagnosticSources_EmitsRosterSnapshot(t *testing.T) { + gate, _, _ := newTestGate( + t, + Schedule{CutoverBlock: 1000}, + 900, + inertPollInterval, + ) + roster, _, _ := newTestRoster(t, 900, 100) + registry := newRecordingRegistry() + + RegisterDiagnosticSources( + registry, + gate, + roster, + &stubChainIdentity{chainID: big.NewInt(1)}, + "release_baked", + ) + + observeStraggler(roster, "protocol-1", 1, validAddress(1)) + + decoded := registry.scrape(t, DiagnosticsSourceCutoverLegacyPeers) + + peers, ok := decoded["peers"].([]interface{}) + if !ok { + t.Fatalf("roster diagnostics omit the peer list: %v", decoded) + } + if len(peers) != 1 { + t.Fatalf( + "unexpected observed peer count\nactual: %v\nexpected: %v", + len(peers), + 1, + ) + } + + // The snapshot must be re-taken per scrape; a captured one would keep + // reporting the roster as it stood at registration. + observeStraggler(roster, "protocol-2", 2, validAddress(2)) + + peers, ok = registry.scrape( + t, + DiagnosticsSourceCutoverLegacyPeers, + )["peers"].([]interface{}) + if !ok { + t.Fatalf("roster diagnostics omit the peer list on re-scrape") + } + if len(peers) != 2 { + t.Errorf( + "unexpected observed peer count after a new sighting\n"+ + "actual: %v\nexpected: %v", + len(peers), + 2, + ) + } +} diff --git a/pkg/protocol/participation/gate.go b/pkg/protocol/participation/gate.go new file mode 100644 index 0000000000..1371b1642e --- /dev/null +++ b/pkg/protocol/participation/gate.go @@ -0,0 +1,1731 @@ +package participation + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "slices" + "sort" + "sync" + "sync/atomic" + "time" + + "github.com/ipfs/go-log/v2" + "golang.org/x/time/rate" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" +) + +// Ceremony identifies a gated protocol ceremony class. The values are fixed: +// they name the per-ceremony refusal metrics and appear in logs and evidence. +type Ceremony string + +// The complete, closed set of gated ceremonies. +const ( + TBTCDKG Ceremony = "tbtc_dkg" + TBTCWalletCoordination Ceremony = "tbtc_wallet_coordination" + TBTCSigning Ceremony = "tbtc_signing" + TBTCHeartbeat Ceremony = "tbtc_heartbeat" + TBTCInactivityClaim Ceremony = "tbtc_inactivity_claim" + BeaconDKG Ceremony = "beacon_dkg" + BeaconRelaySigning Ceremony = "beacon_relay_signing" + BeaconRelayForwarding Ceremony = "beacon_relay_forwarding" + BeaconTimeoutReport Ceremony = "beacon_timeout_report" +) + +// AllCeremonies returns the fixed set of gated ceremonies in a stable order. +func AllCeremonies() []Ceremony { + return []Ceremony{ + TBTCDKG, + TBTCWalletCoordination, + TBTCSigning, + TBTCHeartbeat, + TBTCInactivityClaim, + BeaconDKG, + BeaconRelaySigning, + BeaconRelayForwarding, + BeaconTimeoutReport, + } +} + +// CommitClass distinguishes the two commit fence classes: commits that +// complete work already performed and commits that create a new penalty. +type CommitClass uint8 + +const ( + // CompletionCommit is a terminal commit of already-performed work: signer + // activation, DKG/relay result submission, or a Bitcoin broadcast. A + // legacy permit may make completion commits after the cutover block while + // its protocol validity lasts. + CompletionCommit CommitClass = iota + 1 + // PenaltyCommit creates new penalty state: a heartbeat inactivity claim or + // a beacon timeout report. Legacy permits must not create penalty commits + // at or after the cutover block, and no permit may once quiescence begins. + PenaltyCommit +) + +// String returns the canonical string form of the commit class. +func (c CommitClass) String() string { + switch c { + case CompletionCommit: + return "completion" + case PenaltyCommit: + return "penalty" + default: + return "unknown" + } +} + +// Gate refusal and fence sentinel errors. Callers distinguish a gate refusal +// from an ordinary protocol failure with errors.Is against these values. +var ( + // ErrInvalidAnchor means the supplied canonical start block is zero while + // a cutover schedule is active, or is ahead of the current chain height. + ErrInvalidAnchor = errors.New("invalid canonical ceremony anchor") + // ErrClockUnavailable means a synchronous chain-clock read failed; the + // gate refuses new work and has canceled all outstanding permits. + ErrClockUnavailable = errors.New("chain clock unavailable") + // ErrQuiescing means process quiescence began and no new permits are + // issued. + ErrQuiescing = errors.New("participation gate is quiescing") + // ErrQuiesceDeadline means the process shutdown deadline arrived before + // natural completion and the permit was force-canceled. + ErrQuiesceDeadline = errors.New("participation quiesce deadline exceeded") + // ErrResumeUnsupported means Resume was called for a ceremony class other + // than the beacon relay restart path. + ErrResumeUnsupported = errors.New( + "resume is supported only for beacon relay signing", + ) + // ErrPenaltySuppressed means a penalty commit was refused because the + // permit is legacy at or after the cutover block, or because quiescence + // began. + ErrPenaltySuppressed = errors.New("penalty commit suppressed") + // ErrCommitBeforeCutover means a security-v2 commit was attempted while + // the current chain height is below the cutover block, e.g. after a deep + // reorg. + ErrCommitBeforeCutover = errors.New( + "security-v2 commit refused below the cutover block", + ) + // ErrInvalidCommitClass means a caller supplied a commit class other than + // CompletionCommit or PenaltyCommit. Unknown classes fail closed so an + // unset or newly added value cannot bypass completion or penalty fencing. + ErrInvalidCommitClass = errors.New("invalid participation commit class") + // ErrPermitClosed means the permit was already closed by its owner. + ErrPermitClosed = errors.New("participation permit is closed") + // ErrInvalidPermitIdentity means a caller supplied an empty, malformed, + // or ambiguous chain-work/local-permit identity. + ErrInvalidPermitIdentity = errors.New( + "invalid participation permit identity", + ) + // ErrInvalidTerminalOutcome means a permit owner supplied a terminal + // disposition or evidence shape that is not valid for its ceremony. + ErrInvalidTerminalOutcome = errors.New( + "invalid participation terminal outcome", + ) + // ErrTerminalOutcomeAlreadyRecorded means a permit owner attempted to + // replace its immutable terminal disposition. + ErrTerminalOutcomeAlreadyRecorded = errors.New( + "participation terminal outcome already recorded", + ) + // ErrTerminalOutcomePersistence means the node-owned terminal journal + // could not durably record the ceremony owner's outcome. The permit may + // still close, but the offline rollback audit will fail closed. + ErrTerminalOutcomePersistence = errors.New( + "participation terminal outcome persistence failed", + ) +) + +// IsGateRefusal reports whether the error is, or wraps, one of the gate's +// sentinel errors: the work was refused, canceled, or fenced off by the +// release gate rather than failing on its own protocol terms. Callers use it +// to keep gate decisions out of ordinary failure logs and metrics; it also +// matches a permit context's cancellation cause propagated through an error +// chain. +func IsGateRefusal(err error) bool { + for _, sentinel := range []error{ + ErrInvalidAnchor, + ErrClockUnavailable, + ErrQuiescing, + ErrQuiesceDeadline, + ErrResumeUnsupported, + ErrPenaltySuppressed, + ErrCommitBeforeCutover, + ErrInvalidCommitClass, + ErrPermitClosed, + ErrInvalidPermitIdentity, + } { + if errors.Is(err, sentinel) { + return true + } + } + return false +} + +// CommitGuard is the narrow view of a Permit handed to terminal submission +// code: it can consult the commit fence immediately before an irreversible +// chain or broadcast call, but it cannot close or otherwise manage the permit. +type CommitGuard interface { + // CheckCommit is the last-moment commit fence, called immediately before + // activating newly generated key material, submitting results or claims, + // or broadcasting Bitcoin transactions. It reads a fresh chain height and + // enforces the per-mode fence rules; a returned error is a gate sentinel, + // not a normal protocol timeout. + CheckCommit(operation string, class CommitClass) error +} + +// Permit authorizes local participation in one ceremony. Its ceremony, +// canonical start block, and protocol mode are immutable for its entire +// lifetime: crossing the cutover block never cancels a permit or mutates its +// mode. A permit is counted as active until its idempotent Close. +type Permit interface { + CommitGuard + + // Context is canceled when the gate cancels the permit: on chain-clock + // failure, at the quiesce deadline, or at Close. Ceremony work must stop + // when it is done; the cancellation cause carries the gate sentinel. + Context() context.Context + // Ceremony returns the ceremony class this permit was issued for. + Ceremony() Ceremony + // CanonicalStartBlock returns the canonical chain anchor the mode was + // pinned from. + CanonicalStartBlock() uint64 + // Mode returns the immutable protocol mode of the ceremony. + Mode() ProtocolMode + // WorkID returns the immutable chain-native work identity supplied when + // the permit was issued. + WorkID() string + // PermitID returns the immutable local membership/action identity supplied + // when the permit was issued. + PermitID() string + // RecordTerminalOutcome records the real ceremony owner's final + // disposition and the durable state or explicit no-threshold condition + // behind it. If this permit was present at the quiescence transition, the + // record is written into the encrypted node-authored terminal journal. + // The first valid outcome is immutable. + RecordTerminalOutcome( + outcome TerminalOutcome, + evidence TerminalEvidence, + ) error + // Close releases the permit. It is idempotent. + Close() +} + +// Snapshot is a point-in-time observability view of the gate. +type Snapshot struct { + State State + CutoverBlock uint64 + CurrentBlock uint64 + ClockAvailable bool + Quiescing bool + Allowed bool + ActiveCeremonies uint64 + ActiveLegacyCeremonies uint64 + ActiveSecurityV2Ceremonies uint64 + ActivePermits []PermitSnapshot + // RecentTerminalOutcomes is the gate's own account of what became of the + // permits it has already closed, oldest first and bounded. It is the same + // record the quiescence journal carries, available while the node is still + // running rather than only from a quiescence transition onward. + RecentTerminalOutcomes []TerminalOutcomeRecord + // GateInstance identifies this gate, and so this process, for as long as it + // runs. It is generated once at construction and never persisted. + // + // It exists because the account above lives in memory. A reader that follows + // permits through a node has no way to tell an account that never held a + // record from one that held it and lost it to a restart, and those are + // opposite answers: the first says the node did not do that work, the second + // says nobody can say what it did. Two readings that disagree about this + // value are two different processes, and every record the earlier one held + // is gone. + GateInstance string + // ForgottenTerminalOutcomes counts the closed permits the bounded account + // has dropped to make room, for the same reason. An account at its bound has + // forgotten its oldest records rather than never having held them, and a + // reader joining permits to endings has to know which of those it is looking + // at. + ForgottenTerminalOutcomes uint64 +} + +// Gate issues per-ceremony participation permits with the protocol mode pinned +// from each ceremony's canonical chain anchor. It is the only component that +// derives protocol modes from the chain clock; cryptographic packages never +// query the clock themselves. +type Gate interface { + // Begin issues a permit for a new ceremony. It reads the chain clock + // synchronously, rejects a zero anchor while a cutover schedule is active, + // rejects an anchor ahead of the current height, and derives the mode only + // from the canonical start block. A production gate with quiescence + // persistence also requires exactly one stable PermitIdentity. It returns + // ErrInvalidAnchor, ErrInvalidPermitIdentity, ErrClockUnavailable, or + // ErrQuiescing. + Begin( + ceremony Ceremony, + canonicalStartBlock uint64, + identity ...PermitIdentity, + ) (Permit, error) + // Resume issues a permit for the beacon relay restart path only. The + // caller must have verified on chain that the relay request is still live + // and pass its on-chain start block; the mode pins from that block exactly + // as in Begin. Any other ceremony class returns ErrResumeUnsupported. + Resume( + ceremony Ceremony, + canonicalStartBlock uint64, + identity ...PermitIdentity, + ) (Permit, error) + // State returns a point-in-time observability snapshot. + State() Snapshot + // Quiesce atomically refuses all new permits, keeps existing permits + // alive to natural completion, and refuses penalty commits from the + // transition onward. It is idempotent and always returns the same channel, + // which closes when the active permit count reaches zero or when Close + // force-cancels the remainder. + Quiesce(cause error) <-chan struct{} + // Drained returns a channel that closes once the gate no longer issues + // new permits — quiescence began or Close ran — and every issued permit + // has been released by its owner. Unlike the quiesce channel, which Close + // closes immediately, the drained channel stays open across a forced + // cancellation until the canceled owners finish their cleanup — the + // quarantine and audit writes included — and release their permits. It + // always returns the same channel. + Drained() <-chan struct{} + // QuiescenceSnapshot returns the immutable node-authored inventory + // captured at the first quiescence transition. The boolean is false + // before that transition. Returned slices are defensive copies. + QuiescenceSnapshot() (QuiescenceSnapshot, bool) + // Close is the terminal shutdown: it force-cancels any remaining permits + // with ErrQuiesceDeadline, closes the quiesce channel, and stops the + // clock supervisor. Force-canceled permits remain counted until their + // owners release them; the drained channel reports when that has + // happened. Close is idempotent. + Close() +} + +// GateMetricsRecorder is the minimal metrics sink the gate needs. It is +// satisfied by the client-info performance metrics registry. +type GateMetricsRecorder interface { + IncrementCounter(name string, value float64) + SetGauge(name string, value float64) +} + +type gateOptions struct { + releaseVersion string + releaseRevision string + recorder QuiescenceSnapshotRecorder + now func() time.Time +} + +// GateOption configures node-artifact identity and quiescence persistence +// without changing the required chain-clock and metrics inputs. +type GateOption func(*gateOptions) error + +// WithArtifactIdentity binds the node-authored quiescence snapshot to the +// exact release version and source revision of the running process. +func WithArtifactIdentity(version string, revision string) GateOption { + return func(options *gateOptions) error { + if version == "" { + return fmt.Errorf("release version is required") + } + if revision == "" { + return fmt.Errorf("release revision is required") + } + options.releaseVersion = version + options.releaseRevision = revision + return nil + } +} + +// WithQuiescenceSnapshotRecorder persists the node-authored snapshot into +// storage as part of the first quiescence transition. +func WithQuiescenceSnapshotRecorder( + recorder QuiescenceSnapshotRecorder, +) GateOption { + return func(options *gateOptions) error { + if recorder == nil { + return fmt.Errorf("quiescence snapshot recorder is required") + } + options.recorder = recorder + return nil + } +} + +func withGateTimeSource(now func() time.Time) GateOption { + return func(options *gateOptions) error { + if now == nil { + return fmt.Errorf("gate time source is required") + } + options.now = now + return nil + } +} + +var gateLogger = log.Logger("keep-participation") + +// The gate reports through the client-info performance registry, which adds +// the "performance_" application prefix. Referencing the clientinfo constants +// keeps a single source of truth for the exact exported metric names. +const ( + metricGateState = clientinfo.MetricParticipationGateState + metricCurrentBlock = clientinfo.MetricParticipationCurrentBlock + metricCutoverBlock = clientinfo.MetricParticipationCutoverBlock + metricAllowed = clientinfo.MetricParticipationAllowed + metricActiveCeremonies = clientinfo.MetricParticipationActiveCeremonies + metricActiveLegacyCeremonies = clientinfo.MetricParticipationActiveLegacyCeremonies + metricActiveSecurityV2Ceremonies = clientinfo.MetricParticipationActiveSecurityV2Ceremonies + metricModeLegacyTotal = clientinfo.MetricParticipationModeLegacyTotal + metricModeSecurityV2Total = clientinfo.MetricParticipationModeSecurityV2Total + metricLegacyCompletionsTotal = clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal + metricRefusalsTotal = clientinfo.MetricParticipationRefusalsTotal + metricCommitRefusalsTotal = clientinfo.MetricParticipationCommitRefusalsTotal + metricClockErrorsTotal = clientinfo.MetricParticipationClockErrorsTotal + metricClockAbortsTotal = clientinfo.MetricParticipationClockAbortsTotal + metricQuiesceTotal = clientinfo.MetricParticipationQuiesceTotal + metricQuiesceForcedAbortsTotal = clientinfo.MetricParticipationQuiesceForcedAbortsTotal + metricHeartbeatPenaltySuppressed = clientinfo.MetricHeartbeatPenaltySuppressedTotal +) + +// gateSupervisorPollInterval is how often the clock supervisor synchronously +// polls the current chain height between authoritative per-operation reads. +const gateSupervisorPollInterval = 15 * time.Second + +type permit struct { + gate *chainGate + ceremony Ceremony + canonicalStartBlock uint64 + mode ProtocolMode + workID string + permitID string + identityBound bool + // operatedMembers is the issuance-time copy of the memberships this + // permit's holder operates. It is written once, before the permit is + // reachable by any other goroutine, and only ever read afterwards — every + // reader receives a copy, so no holder of a snapshot can reach back into + // the permit's own record. + operatedMembers MemberIndexes + + ctx context.Context + cancel context.CancelCauseFunc + + closeOnce sync.Once + terminalOutcome *TerminalOutcomeRecord +} + +func (p *permit) Context() context.Context { return p.ctx } +func (p *permit) Ceremony() Ceremony { return p.ceremony } +func (p *permit) CanonicalStartBlock() uint64 { return p.canonicalStartBlock } +func (p *permit) Mode() ProtocolMode { return p.mode } +func (p *permit) WorkID() string { return p.workID } +func (p *permit) PermitID() string { return p.permitID } + +// chainGate is the production Gate implementation, clocked exclusively by the +// shared Ethereum block counter. +type chainGate struct { + schedule Schedule + blockCounter chain.BlockCounter + metrics GateMetricsRecorder + + ctx context.Context + cancel context.CancelFunc + loopDone chan struct{} + + // modeLogLimiter covers mode-selection and legacy-completion logs; + // refusalLogLimiter covers refusal logs. Metrics retain every event. + modeLogLimiter *rate.Limiter + refusalLogLimiter *rate.Limiter + releaseVersion string + releaseRevision string + recorder QuiescenceSnapshotRecorder + now func() time.Time + + closeOnce sync.Once + + // clockSeq issues the ordering tickets for synchronous clock reads. A + // ticket is taken immediately before a read starts, so concurrently + // completing reads apply in initiation order regardless of the order their + // responses arrive in. + clockSeq atomic.Uint64 + + mu sync.Mutex + lastClockTicket uint64 + currentBlock uint64 + clockAvailable bool + quiescing bool + closed bool + quiesceDone chan struct{} + quiesceDoneClosed bool + drained chan struct{} + drainedClosed bool + permits map[*permit]struct{} + activeLegacy uint64 + activeSecurityV2 uint64 + lastState State + nextPermitID uint64 + quiescenceSnapshot *QuiescenceSnapshot + // terminalOutcomes retains what became of the permits this gate has + // already closed, oldest first, bounded to the most recent + // retainedTerminalOutcomes. + // + // The journal only exists from a quiescence transition onward, so before + // one there is nothing node-authored saying what a permit came to — an + // observer watching work cross the cutover has only its own word for it, + // and a report about a ceremony is not evidence about a ceremony. This is + // the gate's own account of the same records, kept while the node is still + // running, so a permit that appeared in a live reading can be followed to + // the outcome its owner recorded rather than to one claimed for it. + terminalOutcomes []TerminalOutcomeRecord + // forgottenTerminalOutcomes counts the records the bound above has dropped. + // A reader cannot otherwise tell an account that never held a permit from one + // that held it and forgot it. + forgottenTerminalOutcomes uint64 + // instance identifies this gate for as long as the process runs. Written once + // at construction and read without the lock is deliberate: it never changes, + // and a reader comparing two readings of it is asking whether they came from + // the same process. + instance string +} + +// retainedTerminalOutcomes bounds the gate's in-memory account of closed +// permits. A node runs indefinitely, so the account has to forget: this holds +// well past the permit population of any one cutover while staying a fixed +// cost. +const retainedTerminalOutcomes = 512 + +// newGateInstance returns an identity for one gate, and so for one process. +// +// It is random rather than derived from the host, the release, or the start +// time: nothing about it needs to be meaningful, and anything meaningful would +// be another nonsecret field to keep out of the diagnostics. A reader only ever +// compares two readings of it for equality. Randomness failing is not a reason +// to refuse to construct the gate, but it must not be a reason to hand two +// processes the same identity either, so the fallback says the identity is +// unknown and every comparison of it reads as a process that cannot be +// followed. +func newGateInstance() string { + raw := make([]byte, 16) + if _, err := rand.Read(raw); err != nil { + return "" + } + return hex.EncodeToString(raw) +} + +// NewGate constructs the production gate from a resolved schedule and the +// shared chain block counter. It synchronously reads the current chain height +// — a clock error at startup is a construction error — arms the cutover-block +// waiter for eager transition telemetry, initializes all fixed metrics, and +// starts the clock supervisor. The supervisor loop is bound to the given +// context and to Close. +func NewGate( + ctx context.Context, + schedule Schedule, + blockCounter chain.BlockCounter, + metrics GateMetricsRecorder, + options ...GateOption, +) (Gate, error) { + return newGate( + ctx, + schedule, + blockCounter, + metrics, + gateSupervisorPollInterval, + options..., + ) +} + +// newGate is the poll-interval-injecting constructor used by tests. +func newGate( + ctx context.Context, + schedule Schedule, + blockCounter chain.BlockCounter, + metrics GateMetricsRecorder, + pollInterval time.Duration, + optionFunctions ...GateOption, +) (*chainGate, error) { + options := &gateOptions{now: time.Now} + for _, option := range optionFunctions { + if option == nil { + return nil, fmt.Errorf("nil gate option") + } + if err := option(options); err != nil { + return nil, fmt.Errorf("invalid gate option: [%w]", err) + } + } + if options.recorder != nil && + (options.releaseVersion == "" || options.releaseRevision == "") { + return nil, fmt.Errorf( + "artifact identity is required when quiescence persistence is enabled", + ) + } + + if blockCounter == nil { + return nil, fmt.Errorf("block counter is required") + } + if metrics == nil { + return nil, fmt.Errorf("metrics recorder is required") + } + if pollInterval <= 0 { + return nil, fmt.Errorf("poll interval must be positive") + } + if err := validateMetricProjectable(schedule.CutoverBlock); err != nil { + return nil, err + } + + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + return nil, fmt.Errorf( + "could not read the chain clock at gate construction: [%w]", + err, + ) + } + // Every height the gate exports must project exactly onto the float64 + // metric surface; a chain reporting an unprojectable height is as unusable + // as one reporting an error. + if err := validateMetricProjectable(currentBlock); err != nil { + return nil, fmt.Errorf( + "could not accept the chain height at gate construction: [%w]", + err, + ) + } + + // The waiter exists only to make transition telemetry eager; every mode + // selection and commit fence uses a synchronous read. The disabled + // schedule has no transition to observe. + var cutoverWaiter <-chan uint64 + if !schedule.Disabled() { + cutoverWaiter, err = blockCounter.BlockHeightWaiter( + schedule.CutoverBlock, + ) + if err != nil { + return nil, fmt.Errorf( + "could not arm the cutover block waiter: [%w]", + err, + ) + } + } + + loopCtx, cancel := context.WithCancel(ctx) + + gate := &chainGate{ + schedule: schedule, + blockCounter: blockCounter, + metrics: metrics, + ctx: loopCtx, + cancel: cancel, + loopDone: make(chan struct{}), + modeLogLimiter: rate.NewLimiter(rate.Every(30*time.Second), 5), + refusalLogLimiter: rate.NewLimiter(rate.Every(30*time.Second), 5), + releaseVersion: options.releaseVersion, + releaseRevision: options.releaseRevision, + recorder: options.recorder, + now: options.now, + quiesceDone: make(chan struct{}), + drained: make(chan struct{}), + permits: make(map[*permit]struct{}), + currentBlock: currentBlock, + clockAvailable: true, + instance: newGateInstance(), + } + + gate.initMetrics() + + gate.mu.Lock() + gate.lastState = gate.stateLocked() + gate.refreshMetricsLocked() + gate.mu.Unlock() + + gateLogger.Infof( + "protocol participation gate constructed [state=%s] "+ + "[currentBlock=%d] [cutoverBlock=%d] [epoch=%s]", + gate.lastState, + currentBlock, + schedule.CutoverBlock, + CompiledEpoch, + ) + + go gate.run(pollInterval, cutoverWaiter) + + return gate, nil +} + +// initMetrics registers every fixed metric at its zero value so scrapers see a +// complete metric set from the start. +func (g *chainGate) initMetrics() { + g.metrics.SetGauge(metricGateState, 0) + g.metrics.SetGauge(metricCurrentBlock, 0) + g.metrics.SetGauge(metricCutoverBlock, 0) + g.metrics.SetGauge(metricAllowed, 0) + g.metrics.SetGauge(metricActiveCeremonies, 0) + g.metrics.SetGauge(metricActiveLegacyCeremonies, 0) + g.metrics.SetGauge(metricActiveSecurityV2Ceremonies, 0) + g.metrics.IncrementCounter(metricModeLegacyTotal, 0) + g.metrics.IncrementCounter(metricModeSecurityV2Total, 0) + g.metrics.IncrementCounter(metricLegacyCompletionsTotal, 0) + g.metrics.IncrementCounter(metricRefusalsTotal, 0) + g.metrics.IncrementCounter(metricCommitRefusalsTotal, 0) + g.metrics.IncrementCounter(metricClockErrorsTotal, 0) + g.metrics.IncrementCounter(metricClockAbortsTotal, 0) + g.metrics.IncrementCounter(metricQuiesceTotal, 0) + g.metrics.IncrementCounter(metricQuiesceForcedAbortsTotal, 0) + g.metrics.IncrementCounter(metricHeartbeatPenaltySuppressed, 0) + for _, ceremony := range AllCeremonies() { + g.metrics.IncrementCounter( + clientinfo.ParticipationRefusalMetricName(string(ceremony)), + 0, + ) + } +} + +// run is the clock supervisor: it polls the current height, watches the +// cutover-block waiter for an eager transition, and converts any clock error +// into the atomic clock-unavailable transition. +func (g *chainGate) run( + pollInterval time.Duration, + cutoverWaiter <-chan uint64, +) { + defer close(g.loopDone) + + ticker := time.NewTicker(pollInterval) + defer ticker.Stop() + + for { + select { + case <-g.ctx.Done(): + return + case _, ok := <-cutoverWaiter: + // A nil channel (disabled schedule, or already handled) blocks + // forever, which is the intended disarm. + cutoverWaiter = nil + if !ok { + // The waiter closed before its target: a clock failure. + g.mu.Lock() + g.signalClockFailureLocked( + "cutover_waiter", + fmt.Errorf("cutover block waiter closed before target"), + ) + g.mu.Unlock() + continue + } + // The waiter exists only to make transition telemetry eager: it + // requests an authoritative synchronous poll and never recovers or + // advances gate state itself. A failing poll here is a clock + // failure even though the waiter reported the target height. + g.poll("cutover_waiter_poll") + case <-ticker.C: + g.poll("supervisor_poll") + } + } +} + +// poll performs one ordered synchronous read of the chain clock. A newest +// failure cancels all permits; a newest success recomputes the current state, +// but previously canceled permits do not revive. A stale outcome is discarded. +func (g *chainGate) poll(operation string) { + ticket, height, err := g.readClock() + + g.mu.Lock() + defer g.mu.Unlock() + + g.applyClockSampleLocked(ticket, height, operation, err) +} + +// readClock performs one ordered synchronous read of the chain clock. The +// ordering ticket is taken immediately before the read starts, so concurrently +// completing reads apply in initiation order regardless of the order their +// responses arrive in. A height the float64 metrics projection cannot +// represent exactly is folded into the read's own error here, at read time: +// the owning operation then fails closed on it exactly like on an RPC error, +// even when a newer concurrent sample supersedes this one. +func (g *chainGate) readClock() (ticket uint64, height uint64, err error) { + ticket = g.clockSeq.Add(1) + height, err = g.blockCounter.CurrentBlock() + if err == nil { + err = validateMetricProjectable(height) + } + return ticket, height, err +} + +// applyClockSampleLocked applies the outcome of one ordered synchronous clock +// read taken through readClock. A sample older than the newest applied one is +// discarded entirely: a stale success arriving after a newer failure can never +// reopen the gate, and a stale failure — an RPC error or an unprojectable +// height — arriving after a newer success cannot spuriously cancel permits. +// The caller must hold g.mu. +func (g *chainGate) applyClockSampleLocked( + ticket uint64, + height uint64, + operation string, + readErr error, +) { + if ticket <= g.lastClockTicket { + return + } + g.lastClockTicket = ticket + + if readErr != nil { + g.clockFailureLocked(operation, readErr) + return + } + + g.clockAvailable = true + g.currentBlock = height + g.refreshMetricsLocked() +} + +// signalClockFailureLocked records a clock failure that arrived as a lifecycle +// signal — the cutover waiter closing before its target — rather than as an +// ordered read outcome. It takes a fresh ticket at application time, so any +// read still in flight was initiated earlier, is stale on arrival, and cannot +// mask this failure; recovery requires a read initiated afterwards. The caller +// must hold g.mu. +func (g *chainGate) signalClockFailureLocked(operation string, err error) { + g.lastClockTicket = g.clockSeq.Add(1) + g.clockFailureLocked(operation, err) +} + +// clockFailureLocked is the atomic clock-unavailable transition: it marks the +// clock unavailable and cancels every not-yet-canceled permit with +// ErrClockUnavailable. Canceled permits remain counted until their owners +// close them. The caller must hold g.mu. +func (g *chainGate) clockFailureLocked(operation string, err error) { + g.clockAvailable = false + g.metrics.IncrementCounter(metricClockErrorsTotal, 1) + + gateLogger.Warnf( + "protocol participation chain clock unavailable [operation=%s] "+ + "[lastCurrentBlock=%d] [error=%s]", + operation, + g.currentBlock, + err, + ) + + aborted := 0 + for p := range g.permits { + if context.Cause(p.ctx) == nil { + p.cancel(ErrClockUnavailable) + aborted++ + } + } + if aborted > 0 { + g.metrics.IncrementCounter(metricClockAbortsTotal, float64(aborted)) + } + + g.refreshMetricsLocked() +} + +// stateLocked computes the externally visible process state. Quiescence is the +// dominant lifecycle condition, then clock failure, then the height-derived +// open state. The caller must hold g.mu. +func (g *chainGate) stateLocked() State { + switch { + case g.closed || g.quiescing: + return StateQuiescing + case !g.clockAvailable: + return StateClockUnavailable + default: + return g.schedule.StateFor(g.currentBlock) + } +} + +// allowedLocked reports whether a new permit can be issued. The caller must +// hold g.mu. +func (g *chainGate) allowedLocked() bool { + return !g.closed && !g.quiescing && g.clockAvailable +} + +// refreshMetricsLocked recomputes all gauges and logs a state transition once +// per process transition. The caller must hold g.mu. +func (g *chainGate) refreshMetricsLocked() { + state := g.stateLocked() + if state != g.lastState { + gateLogger.Infof( + "protocol participation gate transitioned [from=%s] [to=%s] "+ + "[currentBlock=%d] [cutoverBlock=%d] [activeLegacy=%d] "+ + "[activeSecurityV2=%d]", + g.lastState, + state, + g.currentBlock, + g.schedule.CutoverBlock, + g.activeLegacy, + g.activeSecurityV2, + ) + g.lastState = state + } + + g.metrics.SetGauge(metricGateState, float64(state)) + g.metrics.SetGauge(metricCurrentBlock, float64(g.currentBlock)) + g.metrics.SetGauge(metricCutoverBlock, float64(g.schedule.CutoverBlock)) + allowed := float64(0) + if g.allowedLocked() { + allowed = 1 + } + g.metrics.SetGauge(metricAllowed, allowed) + g.metrics.SetGauge( + metricActiveCeremonies, + float64(g.activeLegacy+g.activeSecurityV2), + ) + g.metrics.SetGauge(metricActiveLegacyCeremonies, float64(g.activeLegacy)) + g.metrics.SetGauge( + metricActiveSecurityV2Ceremonies, + float64(g.activeSecurityV2), + ) +} + +// refuseLocked records a Begin/Resume refusal in metrics and the rate-limited +// refusal log and returns the sentinel wrapped with context. The caller must +// hold g.mu. +func (g *chainGate) refuseLocked( + ceremony Ceremony, + canonicalStartBlock uint64, + reason string, + sentinel error, +) error { + g.metrics.IncrementCounter(metricRefusalsTotal, 1) + g.metrics.IncrementCounter( + clientinfo.ParticipationRefusalMetricName(string(ceremony)), + 1, + ) + + if g.refusalLogLimiter.Allow() { + gateLogger.Infof( + "protocol participation refused by release gate [ceremony=%s] "+ + "[reason=%s] [canonicalStartBlock=%d] [currentBlock=%d] "+ + "[cutoverBlock=%d]", + ceremony, + reason, + canonicalStartBlock, + g.currentBlock, + g.schedule.CutoverBlock, + ) + } + + return fmt.Errorf( + "ceremony [%s] with canonical start block [%d] refused (%s): %w", + ceremony, + canonicalStartBlock, + reason, + sentinel, + ) +} + +var knownCeremonies = func() map[Ceremony]struct{} { + known := make(map[Ceremony]struct{}) + for _, ceremony := range AllCeremonies() { + known[ceremony] = struct{}{} + } + return known +}() + +// Begin implements Gate. +func (g *chainGate) Begin( + ceremony Ceremony, + canonicalStartBlock uint64, + identity ...PermitIdentity, +) (Permit, error) { + return g.issue(ceremony, canonicalStartBlock, false, identity...) +} + +// Resume implements Gate. +func (g *chainGate) Resume( + ceremony Ceremony, + canonicalStartBlock uint64, + identity ...PermitIdentity, +) (Permit, error) { + return g.issue(ceremony, canonicalStartBlock, true, identity...) +} + +func (g *chainGate) issue( + ceremony Ceremony, + canonicalStartBlock uint64, + resume bool, + identities ...PermitIdentity, +) (Permit, error) { + if _, known := knownCeremonies[ceremony]; !known { + return nil, fmt.Errorf("unknown ceremony [%s]", ceremony) + } + if len(identities) > 1 { + return nil, fmt.Errorf( + "ceremony [%s] supplied [%d] permit identities: %w", + ceremony, + len(identities), + ErrInvalidPermitIdentity, + ) + } + if g.recorder != nil && len(identities) == 0 { + return nil, fmt.Errorf( + "ceremony [%s] did not supply the permit identity required by "+ + "the production quiescence recorder: %w", + ceremony, + ErrInvalidPermitIdentity, + ) + } + if len(identities) == 1 { + if err := validatePermitIdentityForCeremony( + ceremony, + identities[0], + ); err != nil { + return nil, fmt.Errorf( + "ceremony [%s] permit identity rejected: [%v]: %w", + ceremony, + err, + ErrInvalidPermitIdentity, + ) + } + } + + // The synchronous, authoritative chain read happens outside the lock so a + // slow chain call never blocks fences, closes, or the supervisor. The + // ticket taken before the read orders this sample against concurrent ones. + ticket, height, clockErr := g.readClock() + + g.mu.Lock() + defer g.mu.Unlock() + + g.applyClockSampleLocked(ticket, height, "issue_permit", clockErr) + + if g.closed || g.quiescing { + return nil, g.refuseLocked( + ceremony, + canonicalStartBlock, + "quiescing", + ErrQuiescing, + ) + } + + // This operation fails closed on its own read error — an RPC failure or an + // unprojectable height — even when a newer concurrent sample kept the gate + // available, and equally when its own read succeeded but lost the race to + // a newer applied failure. + if clockErr != nil || !g.clockAvailable { + return nil, g.refuseLocked( + ceremony, + canonicalStartBlock, + "clock_unavailable", + ErrClockUnavailable, + ) + } + + // From here on, decisions use the newest applied height, which is this + // read's own height unless a newer concurrent sample applied first. + height = g.currentBlock + + if resume && ceremony != BeaconRelaySigning { + return nil, g.refuseLocked( + ceremony, + canonicalStartBlock, + "resume_unsupported", + ErrResumeUnsupported, + ) + } + + // A zero anchor is rejected whenever a cutover schedule is active: every + // canonical anchor is an already validated chain event or window block. + // The developer-only disabled schedule accepts zero (genesis) anchors. + if !g.schedule.Disabled() && canonicalStartBlock == 0 { + return nil, g.refuseLocked( + ceremony, + canonicalStartBlock, + "zero_anchor", + ErrInvalidAnchor, + ) + } + if canonicalStartBlock > height { + return nil, g.refuseLocked( + ceremony, + canonicalStartBlock, + "future_anchor", + ErrInvalidAnchor, + ) + } + + mode := g.schedule.ModeFor(canonicalStartBlock) + g.nextPermitID++ + identity := PermitIdentity{ + WorkID: fmt.Sprintf( + "unbound-%s-%d", + ceremony, + canonicalStartBlock, + ), + PermitID: fmt.Sprintf("unbound-%d", g.nextPermitID), + } + identityBound := false + if len(identities) == 1 { + identity = identities[0] + identityBound = true + } + for existing := range g.permits { + if existing.ceremony == ceremony && + existing.canonicalStartBlock == canonicalStartBlock && + existing.workID == identity.WorkID && + existing.permitID == identity.PermitID { + return nil, g.refuseLocked( + ceremony, + canonicalStartBlock, + "duplicate_permit_identity", + ErrInvalidPermitIdentity, + ) + } + } + + ctx, cancel := context.WithCancelCause(g.ctx) + p := &permit{ + gate: g, + ceremony: ceremony, + canonicalStartBlock: canonicalStartBlock, + mode: mode, + workID: identity.WorkID, + permitID: identity.PermitID, + identityBound: identityBound, + // Copied rather than aliased: the caller still holds the slice it + // passed, and a permit whose operated seats can be rewritten after + // issuance is not the immutable record every reading of it claims. + operatedMembers: slices.Clone(identity.OperatedMembers), + ctx: ctx, + cancel: cancel, + } + + g.permits[p] = struct{}{} + switch mode { + case ModeLegacy: + g.activeLegacy++ + g.metrics.IncrementCounter(metricModeLegacyTotal, 1) + case ModeSecurityV2: + g.activeSecurityV2++ + g.metrics.IncrementCounter(metricModeSecurityV2Total, 1) + } + + if g.modeLogLimiter.Allow() { + gateLogger.Infof( + "protocol participation mode selected [ceremony=%s] [mode=%s] "+ + "[canonicalStartBlock=%d] [currentBlock=%d] [cutoverBlock=%d]", + ceremony, + mode, + canonicalStartBlock, + height, + g.schedule.CutoverBlock, + ) + } + + g.refreshMetricsLocked() + + return p, nil +} + +// CheckCommit implements the Permit commit fence. +func (p *permit) CheckCommit(operation string, class CommitClass) error { + g := p.gate + + // The fence always uses its own fresh synchronous height, read outside + // the lock and ordered against concurrent reads by its ticket. + ticket, height, clockErr := g.readClock() + + g.mu.Lock() + defer g.mu.Unlock() + + g.applyClockSampleLocked(ticket, height, "commit_fence", clockErr) + + // The fence fails closed on its own read error — an RPC failure or an + // unprojectable height — even when a newer concurrent sample kept the gate + // available, and equally when its own read succeeded but lost the race to + // a newer applied failure. + if clockErr != nil || !g.clockAvailable { + return g.refuseCommitLocked( + p, + operation, + class, + g.currentBlock, + ErrClockUnavailable, + ) + } + + // Fence decisions use the newest applied height, which is this read's own + // height unless a newer concurrent sample applied first. + height = g.currentBlock + + if cause := context.Cause(p.ctx); cause != nil { + return g.refuseCommitLocked(p, operation, class, height, cause) + } + + if g.closed { + return g.refuseCommitLocked( + p, + operation, + class, + height, + ErrQuiesceDeadline, + ) + } + + if class != CompletionCommit && class != PenaltyCommit { + return g.refuseCommitLocked( + p, + operation, + class, + height, + ErrInvalidCommitClass, + ) + } + + if class == PenaltyCommit { + // Penalty suppression protects the technical grace from turning into + // punishment: it applies to legacy work at or after the cutover block + // and to every permit once quiescence begins. + afterCutoverLegacy := p.mode == ModeLegacy && + !g.schedule.Disabled() && + height >= g.schedule.CutoverBlock + if afterCutoverLegacy || g.quiescing { + return g.suppressPenaltyLocked(p, operation, height) + } + } + + if p.mode == ModeSecurityV2 && + (p.canonicalStartBlock < g.schedule.CutoverBlock || + height < g.schedule.CutoverBlock) { + return g.refuseCommitLocked( + p, + operation, + class, + height, + ErrCommitBeforeCutover, + ) + } + + if p.mode == ModeLegacy && + class == CompletionCommit && + !g.schedule.Disabled() && + height >= g.schedule.CutoverBlock { + g.metrics.IncrementCounter(metricLegacyCompletionsTotal, 1) + if g.modeLogLimiter.Allow() { + gateLogger.Infof( + "protocol participation legacy completion after cutover "+ + "[ceremony=%s] [operation=%s] [canonicalStartBlock=%d] "+ + "[currentBlock=%d]", + p.ceremony, + operation, + p.canonicalStartBlock, + height, + ) + } + } + + return nil +} + +// RecordTerminalOutcome implements Permit. Only the permit owner has this +// method, so an external evidence generator cannot author or replace the +// terminal disposition. The record is retained in memory before quiescence +// and persisted if the permit is captured by the quiescence transition. +func (p *permit) RecordTerminalOutcome( + outcome TerminalOutcome, + evidence TerminalEvidence, +) error { + if err := ValidateTerminalOutcome( + p.ceremony, + p.workID, + outcome, + evidence, + ); err != nil { + return fmt.Errorf( + "terminal outcome for ceremony [%s] rejected: [%v]: %w", + p.ceremony, + err, + ErrInvalidTerminalOutcome, + ) + } + // The seats the holder announced it was operating are read here, not in the + // shared validator, because only the permit knows them. p.operatedMembers is + // written once at issuance and never again, so it needs no lock. + if err := ValidatePermitOperatedOwnership( + p.ceremony, + p.operatedMembers, + outcome, + evidence, + ); err != nil { + return fmt.Errorf( + "terminal outcome for ceremony [%s] rejected: [%v]: %w", + p.ceremony, + err, + ErrInvalidTerminalOutcome, + ) + } + + g := p.gate + g.mu.Lock() + defer g.mu.Unlock() + + if _, active := g.permits[p]; !active { + return fmt.Errorf( + "terminal outcome for ceremony [%s] refused: %w", + p.ceremony, + ErrPermitClosed, + ) + } + if p.terminalOutcome != nil { + if p.terminalOutcome.Outcome == outcome && + p.terminalOutcome.Evidence.Equal(evidence) { + if g.quiescenceSnapshot != nil && g.recorder != nil { + if err := g.recorder.RecordTerminalOutcome( + *p.terminalOutcome, + ); err != nil { + return fmt.Errorf( + "cannot persist terminal outcome for ceremony [%s]: "+ + "[%v]: %w", + p.ceremony, + err, + ErrTerminalOutcomePersistence, + ) + } + } + + return nil + } + + return fmt.Errorf( + "terminal outcome for ceremony [%s] is already [%s]: %w", + p.ceremony, + p.terminalOutcome.Outcome, + ErrTerminalOutcomeAlreadyRecorded, + ) + } + + record := &TerminalOutcomeRecord{ + RecordedAt: g.now().UTC(), + Permit: p.snapshot(), + Outcome: outcome, + Evidence: evidence, + } + // Retain the ceremony owner's immutable disposition before attempting + // persistence. A transient journal failure can then be retried by an + // identical call or by Close without replacing the real outcome with the + // fail-closed unresolved marker. + p.terminalOutcome = record + + if g.quiescenceSnapshot != nil && g.recorder != nil { + if err := g.recorder.RecordTerminalOutcome(*record); err != nil { + return fmt.Errorf( + "cannot persist terminal outcome for ceremony [%s]: [%v]: %w", + p.ceremony, + err, + ErrTerminalOutcomePersistence, + ) + } + } + + return nil +} + +func (p *permit) snapshot() PermitSnapshot { + return PermitSnapshot{ + Ceremony: p.ceremony, + Mode: p.mode.String(), + CanonicalStartBlock: p.canonicalStartBlock, + WorkID: p.workID, + PermitID: p.permitID, + IdentityBound: p.identityBound, + // A snapshot outlives the call that took it — into a diagnostics + // scrape, a quiescence inventory, a journal record — so it carries its + // own copy rather than a window onto the permit's slice. + OperatedMembers: slices.Clone(p.operatedMembers), + } +} + +// refuseCommitLocked records a failed commit fence and returns the sentinel +// wrapped with context. The caller must hold g.mu. +func (g *chainGate) refuseCommitLocked( + p *permit, + operation string, + class CommitClass, + height uint64, + sentinel error, +) error { + g.metrics.IncrementCounter(metricCommitRefusalsTotal, 1) + + gateLogger.Warnf( + "protocol participation commit refused [ceremony=%s] [operation=%s] "+ + "[class=%s] [mode=%s] [state=%s] [currentBlock=%d]", + p.ceremony, + operation, + class, + p.mode, + g.stateLocked(), + height, + ) + + return fmt.Errorf( + "%s commit [%s] for ceremony [%s] refused: %w", + class, + operation, + p.ceremony, + sentinel, + ) +} + +// suppressPenaltyLocked records a suppressed penalty commit. The caller must +// hold g.mu. +func (g *chainGate) suppressPenaltyLocked( + p *permit, + operation string, + height uint64, +) error { + g.metrics.IncrementCounter(metricCommitRefusalsTotal, 1) + if p.ceremony == TBTCHeartbeat || p.ceremony == TBTCInactivityClaim { + g.metrics.IncrementCounter(metricHeartbeatPenaltySuppressed, 1) + } + + gateLogger.Warnf( + "protocol participation penalty suppressed [ceremony=%s] "+ + "[operation=%s] [mode=%s] [currentBlock=%d] [cutoverBlock=%d]", + p.ceremony, + operation, + p.mode, + height, + g.schedule.CutoverBlock, + ) + + return fmt.Errorf( + "penalty commit [%s] for ceremony [%s] suppressed: %w", + operation, + p.ceremony, + ErrPenaltySuppressed, + ) +} + +// Close implements the Permit release. It is idempotent; the permit stops +// being counted as active exactly once. +func (p *permit) Close() { + p.closeOnce.Do(func() { + p.cancel(ErrPermitClosed) + + g := p.gate + g.mu.Lock() + defer g.mu.Unlock() + + if _, active := g.permits[p]; !active { + return + } + + // A permit whose owner recorded nothing left no account of what its + // ceremony came to. That is a disposition in itself and it is the one + // the offline barrier refuses on, so it is written here rather than + // left as a silent absence — under quiescence for the journal, and + // always for the gate's own retained account. + if p.terminalOutcome == nil { + p.terminalOutcome = &TerminalOutcomeRecord{ + RecordedAt: g.now().UTC(), + Permit: p.snapshot(), + Outcome: terminalOutcomeUnresolved, + } + } + g.retainTerminalOutcome(*p.terminalOutcome) + + if g.quiescenceSnapshot != nil && g.recorder != nil { + if err := g.recorder.RecordTerminalOutcome( + *p.terminalOutcome, + ); err != nil { + gateLogger.Warnf( + "protocol participation terminal outcome could not "+ + "be persisted [ceremony=%s] [outcome=%s] [error=%s]", + p.ceremony, + p.terminalOutcome.Outcome, + err, + ) + } + } + + delete(g.permits, p) + switch p.mode { + case ModeLegacy: + g.activeLegacy-- + case ModeSecurityV2: + g.activeSecurityV2-- + } + + if g.quiescing && + g.activeLegacy+g.activeSecurityV2 == 0 && + !g.quiesceDoneClosed { + g.quiesceDoneClosed = true + close(g.quiesceDone) + } + g.maybeCloseDrainedLocked() + + g.refreshMetricsLocked() + }) +} + +// maybeCloseDrainedLocked closes the drained channel once the gate no longer +// issues new permits — quiescence began or the gate closed — and the last +// counted permit has been released. A clock failure alone never drains the +// gate: it cancels permits but issuance resumes on clock recovery. The caller +// must hold g.mu. +func (g *chainGate) maybeCloseDrainedLocked() { + if !g.quiescing && !g.closed { + return + } + if g.activeLegacy+g.activeSecurityV2 > 0 { + return + } + if !g.drainedClosed { + g.drainedClosed = true + close(g.drained) + } +} + +// State implements Gate. +func (g *chainGate) State() Snapshot { + g.mu.Lock() + defer g.mu.Unlock() + + return Snapshot{ + State: g.stateLocked(), + CutoverBlock: g.schedule.CutoverBlock, + CurrentBlock: g.currentBlock, + ClockAvailable: g.clockAvailable, + Quiescing: g.quiescing || g.closed, + Allowed: g.allowedLocked(), + ActiveCeremonies: g.activeLegacy + g.activeSecurityV2, + ActiveLegacyCeremonies: g.activeLegacy, + ActiveSecurityV2Ceremonies: g.activeSecurityV2, + ActivePermits: g.permitSnapshotsLocked(), + RecentTerminalOutcomes: g.terminalOutcomesLocked(), + GateInstance: g.instance, + ForgottenTerminalOutcomes: g.forgottenTerminalOutcomes, + } +} + +// retainTerminalOutcome adds a closed permit's disposition to the gate's own +// account of them, dropping the oldest once the account is full. The caller +// must hold g.mu. +func (g *chainGate) retainTerminalOutcome(record TerminalOutcomeRecord) { + if len(g.terminalOutcomes) == retainedTerminalOutcomes { + copy(g.terminalOutcomes, g.terminalOutcomes[1:]) + g.terminalOutcomes[len(g.terminalOutcomes)-1] = record + g.forgottenTerminalOutcomes++ + return + } + + g.terminalOutcomes = append(g.terminalOutcomes, record) +} + +// terminalOutcomesLocked returns a copy of the gate's account of closed +// permits, oldest first. The caller must hold g.mu. +// +// The order is the order the permits closed in and is not re-sorted. What a +// reader joins on is the permit identity each record carries; the sequence is +// the one piece of information a re-sort would destroy, and it is what says +// which of two dispositions of neighbouring work came first. +func (g *chainGate) terminalOutcomesLocked() []TerminalOutcomeRecord { + if len(g.terminalOutcomes) == 0 { + return nil + } + + records := make([]TerminalOutcomeRecord, 0, len(g.terminalOutcomes)) + for _, record := range g.terminalOutcomes { + // The settlement hangs off the evidence by pointer, so a plain copy + // would hand a reader the gate's own record to hold. + if record.Evidence.ChainSettlement != nil { + settlement := *record.Evidence.ChainSettlement + record.Evidence.ChainSettlement = &settlement + } + // And the memberships travel by slice, which a plain copy shares for + // the same reason. + record.Permit.OperatedMembers = slices.Clone( + record.Permit.OperatedMembers, + ) + if record.Evidence.Contribution != nil { + contribution := *record.Evidence.Contribution + contribution.IncorporatedMembers = slices.Clone( + contribution.IncorporatedMembers, + ) + contribution.LocalMembers = slices.Clone( + contribution.LocalMembers, + ) + contribution.PermitSpaceMembers = slices.Clone( + contribution.PermitSpaceMembers, + ) + record.Evidence.Contribution = &contribution + } + records = append(records, record) + } + + return records +} + +// permitSnapshotsLocked returns a deterministic copy of the real live-permit +// registry. The caller must hold g.mu. +func (g *chainGate) permitSnapshotsLocked() []PermitSnapshot { + snapshots := make([]PermitSnapshot, 0, len(g.permits)) + for permit := range g.permits { + snapshots = append(snapshots, permit.snapshot()) + } + + sort.Slice(snapshots, func(i, j int) bool { + left := snapshots[i] + right := snapshots[j] + if left.Ceremony != right.Ceremony { + return left.Ceremony < right.Ceremony + } + if left.CanonicalStartBlock != right.CanonicalStartBlock { + return left.CanonicalStartBlock < right.CanonicalStartBlock + } + if left.WorkID != right.WorkID { + return left.WorkID < right.WorkID + } + return left.PermitID < right.PermitID + }) + + return snapshots +} + +func cloneQuiescenceSnapshot( + snapshot QuiescenceSnapshot, +) QuiescenceSnapshot { + snapshot.ActivePermits = append( + []PermitSnapshot(nil), + snapshot.ActivePermits..., + ) + // Copying the list is not copying the permits: each one carries the + // memberships its holder operates by slice, and the inventory this returns + // is the immutable record of what the node was holding at the transition. + for i := range snapshot.ActivePermits { + snapshot.ActivePermits[i].OperatedMembers = slices.Clone( + snapshot.ActivePermits[i].OperatedMembers, + ) + } + + return snapshot +} + +// QuiescenceSnapshot implements Gate. +func (g *chainGate) QuiescenceSnapshot() (QuiescenceSnapshot, bool) { + g.mu.Lock() + defer g.mu.Unlock() + + if g.quiescenceSnapshot == nil { + return QuiescenceSnapshot{}, false + } + + return cloneQuiescenceSnapshot(*g.quiescenceSnapshot), true +} + +// Quiesce implements Gate. +func (g *chainGate) Quiesce(cause error) <-chan struct{} { + g.mu.Lock() + defer g.mu.Unlock() + + if !g.quiescing && !g.closed { + g.quiescing = true + g.metrics.IncrementCounter(metricQuiesceTotal, 1) + + captured := QuiescenceSnapshot{ + SchemaVersion: QuiescenceSnapshotSchemaVersion, + CapturedAt: g.now().UTC(), + ReleaseVersion: g.releaseVersion, + ReleaseRevision: g.releaseRevision, + ReleaseEpoch: CompiledEpoch.String(), + CutoverBlock: g.schedule.CutoverBlock, + CurrentBlock: g.currentBlock, + ClockAvailable: g.clockAvailable, + State: StateQuiescing.String(), + QuiesceCause: fmt.Sprint(cause), + ActiveCeremonies: g.activeLegacy + g.activeSecurityV2, + ActiveLegacyCeremonies: g.activeLegacy, + ActiveSecurityV2Ceremonies: g.activeSecurityV2, + ActivePermits: g.permitSnapshotsLocked(), + } + g.quiescenceSnapshot = &captured + + if g.recorder != nil { + if err := g.recorder.Record( + cloneQuiescenceSnapshot(captured), + ); err != nil { + // The transition remains one-way and fail-closed. The stopped + // node's offline audit will refuse rollback when the required + // node-authored record is absent or malformed. + gateLogger.Warnf( + "protocol participation quiescence snapshot could not "+ + "be persisted [error=%s]", + err, + ) + } + for permit := range g.permits { + if permit.terminalOutcome == nil { + continue + } + if err := g.recorder.RecordTerminalOutcome( + *permit.terminalOutcome, + ); err != nil { + gateLogger.Warnf( + "protocol participation terminal outcome could not "+ + "be persisted [ceremony=%s] [outcome=%s] [error=%s]", + permit.ceremony, + permit.terminalOutcome.Outcome, + err, + ) + } + } + } + + gateLogger.Warnf( + "protocol participation quiescing [reason=%s] [currentBlock=%d] "+ + "[activeLegacy=%d] [activeSecurityV2=%d]", + cause, + g.currentBlock, + g.activeLegacy, + g.activeSecurityV2, + ) + + if g.activeLegacy+g.activeSecurityV2 == 0 && !g.quiesceDoneClosed { + g.quiesceDoneClosed = true + close(g.quiesceDone) + } + g.maybeCloseDrainedLocked() + + g.refreshMetricsLocked() + } + + return g.quiesceDone +} + +// Drained implements Gate. The channel is created once at construction and +// never replaced, so no lock is needed to hand it out. +func (g *chainGate) Drained() <-chan struct{} { + return g.drained +} + +// Close implements Gate. +func (g *chainGate) Close() { + g.closeOnce.Do(func() { + g.mu.Lock() + + g.closed = true + + for p := range g.permits { + if context.Cause(p.ctx) == nil { + p.cancel(ErrQuiesceDeadline) + g.metrics.IncrementCounter(metricQuiesceForcedAbortsTotal, 1) + + gateLogger.Warnf( + "protocol participation forced abort at quiesce deadline "+ + "[ceremony=%s] [mode=%s] [canonicalStartBlock=%d] "+ + "[currentBlock=%d]", + p.ceremony, + p.mode, + p.canonicalStartBlock, + g.currentBlock, + ) + } + } + + if !g.quiesceDoneClosed { + g.quiesceDoneClosed = true + close(g.quiesceDone) + } + g.maybeCloseDrainedLocked() + + g.refreshMetricsLocked() + g.mu.Unlock() + + g.cancel() + <-g.loopDone + }) +} diff --git a/pkg/protocol/participation/gate_test.go b/pkg/protocol/participation/gate_test.go new file mode 100644 index 0000000000..19572c2f20 --- /dev/null +++ b/pkg/protocol/participation/gate_test.go @@ -0,0 +1,2774 @@ +package participation + +import ( + "context" + "errors" + "fmt" + "math/big" + "strings" + "sync" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/clientinfo" +) + +// gateBlockCounter is a controllable chain.BlockCounter with real height +// waiter semantics: a waiter channel emits the reached height and closes, or +// can be force-closed without a value to simulate a waiter failure. A one-shot +// read hold lets tests model a slow RPC response, computed from an older chain +// view, arriving after newer reads have completed. +type gateBlockCounter struct { + mu sync.Mutex + block uint64 + err error + waiterErr error + waiters map[uint64][]chan uint64 + reads uint64 + holdStarted chan struct{} + holdRelease chan struct{} +} + +func TestGate_CeremonySpecificPermitIdentityValidation(t *testing.T) { + const cutover = uint64(1_000) + recorder := &recordingQuiescenceSnapshotRecorder{} + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + WithArtifactIdentity("v2.1.0", "revision-test"), + WithQuiescenceSnapshotRecorder(recorder), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + tests := map[string]struct { + ceremony Ceremony + identity PermitIdentity + }{ + "uppercase DKG seed hash": { + ceremony: TBTCDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("A", 64), + PermitID: "1", + OperatedMembers: MemberIndexes{1}, + }, + }, + "truncated DKG seed hash": { + ceremony: BeaconDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("a", 63), + PermitID: "1", + OperatedMembers: MemberIndexes{1}, + }, + }, + "prefixed member index": { + ceremony: TBTCDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "01", + OperatedMembers: MemberIndexes{1}, + }, + }, + "zero member index": { + ceremony: BeaconDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "0", + OperatedMembers: MemberIndexes{1}, + }, + }, + // The seats a per-seat ceremony announces cannot say anything other + // than what its permit ID already says. + "DKG permit operating no seat": { + ceremony: TBTCDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "7", + }, + }, + "DKG permit operating a different seat": { + ceremony: BeaconDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "7", + OperatedMembers: MemberIndexes{8}, + }, + }, + "DKG permit operating more than its own seat": { + ceremony: TBTCDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "7", + OperatedMembers: MemberIndexes{7, 8}, + }, + }, + "relay signing permit operating a different seat": { + ceremony: BeaconRelaySigning, + identity: PermitIdentity{ + WorkID: BeaconRelayWorkID(1_000), + PermitID: "3", + OperatedMembers: MemberIndexes{4}, + }, + }, + // And the ceremonies that operate no seat may not claim one: a + // forwarder relays other members' shares and a timeout report is a + // penalty filing, so a seat here would enter the fleet's ownership map + // with no membership behind it. + "forwarder claiming a seat": { + ceremony: BeaconRelayForwarding, + identity: PermitIdentity{ + WorkID: BeaconRelayWorkID(1_000), + PermitID: "forwarder", + OperatedMembers: MemberIndexes{1}, + }, + }, + "timeout monitor claiming a seat": { + ceremony: BeaconTimeoutReport, + identity: PermitIdentity{ + WorkID: BeaconRelayWorkID(1_000), + PermitID: "timeout-monitor", + OperatedMembers: MemberIndexes{1}, + }, + }, + // The generic set rules hold for every ceremony: one set has exactly + // one encoding, so a reader cannot be shown the same seat twice. + "unordered operated seats": { + ceremony: TBTCSigning, + identity: PermitIdentity{ + WorkID: "wallet-action-unordered", + PermitID: "wallet", + OperatedMembers: MemberIndexes{4, 2}, + }, + }, + "repeated operated seat": { + ceremony: TBTCSigning, + identity: PermitIdentity{ + WorkID: "wallet-action-repeated", + PermitID: "wallet", + OperatedMembers: MemberIndexes{2, 2}, + }, + }, + "invalid operated seat": { + ceremony: TBTCSigning, + identity: PermitIdentity{ + WorkID: "wallet-action-zero-seat", + PermitID: "wallet", + OperatedMembers: MemberIndexes{0}, + }, + }, + "nonmember relay identity": { + ceremony: BeaconRelaySigning, + identity: PermitIdentity{ + WorkID: "relay-request-1000", + PermitID: "member-1", + }, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + if _, err := gate.Begin( + test.ceremony, + cutover, + test.identity, + ); !errors.Is(err, ErrInvalidPermitIdentity) { + t.Fatalf( + "expected ceremony-specific identity rejection, got [%v]", + err, + ) + } + }) + } + + permit, err := gate.Begin( + TBTCDKG, + cutover, + PermitIdentity{ + WorkID: strings.Repeat("a", 64), + PermitID: "255", + OperatedMembers: MemberIndexes{255}, + }, + ) + if err != nil { + t.Fatalf("canonical DKG identity rejected: [%v]", err) + } + permit.Close() +} + +func TestGate_NodeAuthoredTerminalOutcomes(t *testing.T) { + const cutover = uint64(1_000) + capturedAt := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + recorder := &recordingQuiescenceSnapshotRecorder{} + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + WithArtifactIdentity("v2.1.0", "revision-test"), + WithQuiescenceSnapshotRecorder(recorder), + withGateTimeSource(func() time.Time { return capturedAt }), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + completed, err := gate.Begin( + TBTCSigning, + cutover, + PermitIdentity{ + WorkID: "wallet-action-completed", + PermitID: "wallet-completed", + OperatedMembers: MemberIndexes{1}, + }, + ) + if err != nil { + t.Fatal(err) + } + unresolved, err := gate.Begin( + BeaconRelayForwarding, + cutover, + PermitIdentity{ + WorkID: "relay-request-unresolved", + PermitID: "forwarder", + }, + ) + if err != nil { + t.Fatal(err) + } + + gate.Quiesce(fmt.Errorf("rollback drill")) + if err := completed.RecordTerminalOutcome( + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidenceBitcoinTransaction, + Reference: "signed-transaction-hash", + Contribution: testTranscriptContribution(TBTCSigning, 1), + }, + ); err != nil { + t.Fatal(err) + } + if err := completed.RecordTerminalOutcome( + TerminalOutcomeExhausted, + TerminalEvidence{Kind: TerminalEvidenceNoThreshold}, + ); !errors.Is(err, ErrTerminalOutcomeAlreadyRecorded) { + t.Fatalf("expected immutable terminal outcome, got [%v]", err) + } + + completed.Close() + unresolved.Close() + + outcomes := recorder.recordedOutcomes() + if len(outcomes) != 2 { + t.Fatalf("expected two terminal outcomes, got [%d]", len(outcomes)) + } + recorded := make(map[string]TerminalOutcome) + for _, outcome := range outcomes { + recorded[outcome.Permit.WorkID] = outcome.Outcome + } + if recorded["wallet-action-completed"] != TerminalOutcomeCompleted { + t.Errorf("completed permit outcome not persisted: %+v", outcomes) + } + if recorded["relay-request-unresolved"] != terminalOutcomeUnresolved { + t.Errorf("missing owner outcome did not fail closed: %+v", outcomes) + } +} + +// TestGate_RetainsTerminalOutcomesOfClosedPermits asserts the gate keeps its +// own account of what became of the permits it closed, and that the account is +// available without a quiescence transition. +// +// The journal exists only from the first quiescence onward. Before one, a +// permit that finishes simply disappears from the live state: an observer +// watching work cross the cutover sees it held and then sees nothing, and has +// to take some other party's report for how it ended. A report about a +// ceremony is not evidence about a ceremony, and the difference is the whole +// point of the record — so what the ceremony's own owner recorded has to +// outlive the permit while the node is still running. +func TestGate_RetainsTerminalOutcomesOfClosedPermits(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + // No quiescence transition anywhere in this test: the account has to be + // there for a node that is simply running. + completed, err := gate.Begin( + TBTCSigning, + cutover, + PermitIdentity{ + WorkID: "wallet-action", + PermitID: "wallet", + OperatedMembers: MemberIndexes{1}, + }, + ) + if err != nil { + t.Fatal(err) + } + silent, err := gate.Begin( + BeaconRelayForwarding, + cutover, + PermitIdentity{WorkID: "relay-request", PermitID: "forwarder"}, + ) + if err != nil { + t.Fatal(err) + } + + if held := gate.State().RecentTerminalOutcomes; len(held) != 0 { + t.Fatalf("a live permit was accounted for as closed: %+v", held) + } + + if err := completed.RecordTerminalOutcome( + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidenceBitcoinTransaction, + Reference: "signed-transaction-hash", + Contribution: testTranscriptContribution(TBTCSigning, 1), + }, + ); err != nil { + t.Fatal(err) + } + completed.Close() + silent.Close() + + outcomes := gate.State().RecentTerminalOutcomes + if len(outcomes) != 2 { + t.Fatalf("expected two closed permits accounted for: %+v", outcomes) + } + + // The order is the order they closed in, which is what says which of two + // dispositions came first. + if outcomes[0].Permit.WorkID != "wallet-action" || + outcomes[1].Permit.WorkID != "relay-request" { + t.Errorf("closed permits are not in the order they closed: %+v", outcomes) + } + if outcomes[0].Outcome != TerminalOutcomeCompleted { + t.Errorf( + "the owner's recorded outcome was not retained: %+v", + outcomes[0], + ) + } + if outcomes[0].Evidence.Reference != "signed-transaction-hash" { + t.Errorf( + "the owner's recorded evidence was not retained: %+v", + outcomes[0].Evidence, + ) + } + // The permit identity is what a reader joins the record to a live reading + // on; without it the account names dispositions belonging to nobody. + if outcomes[0].Permit.PermitID != "wallet" || + outcomes[0].Permit.Ceremony != TBTCSigning || + outcomes[0].Permit.Mode != ModeSecurityV2.String() { + t.Errorf("the closed permit's identity was not retained: %+v", outcomes[0]) + } + // A permit whose owner recorded nothing is not silently absent. Its + // ceremony came to something the node cannot vouch for, and that is a + // disposition a reader has to be able to see. + if outcomes[1].Outcome != terminalOutcomeUnresolved { + t.Errorf( + "a permit closed without an owner outcome was not accounted "+ + "for as unresolved: %+v", + outcomes[1], + ) + } + + // Closing again must not double-count: an idempotent release that added a + // second record would make one ceremony look like two. + completed.Close() + if again := gate.State().RecentTerminalOutcomes; len(again) != 2 { + t.Errorf("a repeated release was accounted for twice: %+v", again) + } +} + +// TestGate_TerminalOutcomeAccountIsBounded asserts the gate's account of closed +// permits forgets rather than growing without limit, and that it forgets the +// oldest. +// +// A node runs for as long as the release does, so an account that kept every +// permit would be a slow leak in the one component every ceremony passes +// through. +func TestGate_TerminalOutcomeAccountIsBounded(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + const surplus = 3 + for i := 0; i < retainedTerminalOutcomes+surplus; i++ { + permit, err := gate.Begin( + TBTCSigning, + cutover, + PermitIdentity{ + WorkID: fmt.Sprintf("wallet-action-%d", i), + PermitID: fmt.Sprintf("wallet-%d", i), + OperatedMembers: MemberIndexes{1}, + }, + ) + if err != nil { + t.Fatal(err) + } + permit.Close() + } + + outcomes := gate.State().RecentTerminalOutcomes + if len(outcomes) != retainedTerminalOutcomes { + t.Fatalf( + "unexpected account size\nexpected: [%d]\nactual: [%d]", + retainedTerminalOutcomes, + len(outcomes), + ) + } + if first := outcomes[0].Permit.WorkID; first != + fmt.Sprintf("wallet-action-%d", surplus) { + t.Errorf("the account did not forget the oldest permits: [%s]", first) + } + if last := outcomes[len(outcomes)-1].Permit.WorkID; last != + fmt.Sprintf("wallet-action-%d", retainedTerminalOutcomes+surplus-1) { + t.Errorf("the account did not keep the newest permit: [%s]", last) + } + // And it says how many it forgot. Without that a reader joining a permit it + // saw held to the ending its holder recorded cannot tell an account that + // never held the record from one that dropped it, and those are opposite + // answers about whether this node did the work. + if forgotten := gate.State().ForgottenTerminalOutcomes; forgotten != + surplus { + t.Errorf( + "unexpected forgotten count\nexpected: [%d]\nactual: [%d]", + surplus, + forgotten, + ) + } +} + +// TestGate_PublishesAProcessInstanceIdentity asserts each gate names itself, and +// that two gates do not share a name. +// +// The gate's account of closed permits lives in memory. A reader following work +// through a node has no way to tell an account that never held a record from one +// that held it and lost it to a restart, and reading the second as the first +// attributes that node's work to whoever else was on the network. The instance is +// what separates the two: two readings that disagree about it came from different +// processes, and every record the earlier one held is gone. +func TestGate_PublishesAProcessInstanceIdentity(t *testing.T) { + const cutover = uint64(1_000) + + instances := make(map[string]struct{}) + for run := 0; run < 2; run++ { + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + instance := gate.State().GateInstance + if instance == "" { + t.Fatal("a gate published no instance identity") + } + // It has to be the same answer every time it is asked, or a single + // process would read as a restarting one and every reading of its + // account would be unusable. + if again := gate.State().GateInstance; again != instance { + t.Errorf( + "one gate named itself twice over: [%s] then [%s]", + instance, + again, + ) + } + instances[instance] = struct{}{} + } + + if len(instances) != 2 { + t.Error("two gates published one instance identity between them") + } +} + +// TestGate_TerminalOutcomeAccountIsNotAliased asserts a reader cannot reach +// into the gate's own account through the snapshot it is handed. +// +// The evidence carries the chain settlement by pointer, so a shallow copy +// would hand every scrape a live reference to the record the gate keeps. +func TestGate_TerminalOutcomeAccountIsNotAliased(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + // The heartbeat is the one ceremony that dispatches a chain settlement of + // its own, so it is the only permit whose evidence carries the pointer + // this test is about. + permit, err := gate.Begin( + TBTCHeartbeat, + cutover, + PermitIdentity{ + WorkID: "heartbeat-work", + PermitID: "heartbeat", + OperatedMembers: MemberIndexes{1}, + }, + ) + if err != nil { + t.Fatal(err) + } + if err := permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: "heartbeat-result-identity", + ChainSettlement: &ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + }, + Contribution: testTranscriptContribution(TBTCHeartbeat, 1), + }, + ); err != nil { + t.Fatal(err) + } + permit.Close() + + handed := gate.State().RecentTerminalOutcomes + if len(handed) != 1 || handed[0].Evidence.ChainSettlement == nil { + t.Fatalf("the settled outcome was not retained: %+v", handed) + } + handed[0].Evidence.ChainSettlement.Reference = "some-other-claim" + + reread := gate.State().RecentTerminalOutcomes + if reread[0].Evidence.ChainSettlement.Reference != "" { + t.Errorf( + "a reader rewrote the gate's own record: [%s]", + reread[0].Evidence.ChainSettlement.Reference, + ) + } +} + +func TestGate_NodeAuthoredTerminalOutcomeRetriesPersistence(t *testing.T) { + const cutover = uint64(1_000) + capturedAt := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + recorder := &recordingQuiescenceSnapshotRecorder{ + terminalFailures: 1, + terminalErr: errors.New("transient journal failure"), + } + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + WithArtifactIdentity("v2.1.0", "revision-test"), + WithQuiescenceSnapshotRecorder(recorder), + withGateTimeSource(func() time.Time { return capturedAt }), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin( + TBTCSigning, + cutover, + PermitIdentity{ + WorkID: "wallet-action-retry", + PermitID: "wallet-retry", + OperatedMembers: MemberIndexes{1}, + }, + ) + if err != nil { + t.Fatal(err) + } + gate.Quiesce(fmt.Errorf("rollback drill")) + + outcome := TerminalOutcomeCompleted + evidence := TerminalEvidence{ + Kind: TerminalEvidenceBitcoinTransaction, + Reference: "signed-transaction-hash", + Contribution: testTranscriptContribution(TBTCSigning, 1), + } + if err := permit.RecordTerminalOutcome( + outcome, + evidence, + ); !errors.Is(err, ErrTerminalOutcomePersistence) { + t.Fatalf("expected first persistence attempt to fail, got [%v]", err) + } + if err := permit.RecordTerminalOutcome(outcome, evidence); err != nil { + t.Fatalf("expected identical outcome retry to succeed, got [%v]", err) + } + permit.Close() + + outcomes := recorder.recordedOutcomes() + if len(outcomes) != 1 { + t.Fatalf("expected one terminal outcome, got [%d]", len(outcomes)) + } + if outcomes[0].Outcome != outcome || + !outcomes[0].Evidence.Equal(evidence) { + t.Errorf("unexpected terminal outcome after retry: %+v", outcomes[0]) + } +} + +type recordingQuiescenceSnapshotRecorder struct { + mu sync.Mutex + snapshots []QuiescenceSnapshot + outcomes []TerminalOutcomeRecord + err error + terminalFailures int + terminalErr error +} + +func (r *recordingQuiescenceSnapshotRecorder) Record( + snapshot QuiescenceSnapshot, +) error { + r.mu.Lock() + defer r.mu.Unlock() + + r.snapshots = append(r.snapshots, cloneQuiescenceSnapshot(snapshot)) + return r.err +} + +func (r *recordingQuiescenceSnapshotRecorder) RecordTerminalOutcome( + outcome TerminalOutcomeRecord, +) error { + r.mu.Lock() + defer r.mu.Unlock() + + if r.terminalFailures > 0 { + r.terminalFailures-- + return r.terminalErr + } + + for _, existing := range r.outcomes { + if existing.Permit.Equal(outcome.Permit) { + if existing.Equal(outcome) { + return r.err + } + return fmt.Errorf("contradictory terminal outcome") + } + } + r.outcomes = append(r.outcomes, outcome) + return r.err +} + +func (r *recordingQuiescenceSnapshotRecorder) recorded() []QuiescenceSnapshot { + r.mu.Lock() + defer r.mu.Unlock() + + result := make([]QuiescenceSnapshot, len(r.snapshots)) + for i, snapshot := range r.snapshots { + result[i] = cloneQuiescenceSnapshot(snapshot) + } + return result +} + +func (r *recordingQuiescenceSnapshotRecorder) recordedOutcomes() []TerminalOutcomeRecord { + r.mu.Lock() + defer r.mu.Unlock() + + return append([]TerminalOutcomeRecord(nil), r.outcomes...) +} + +func newGateBlockCounter(block uint64) *gateBlockCounter { + return &gateBlockCounter{ + block: block, + waiters: make(map[uint64][]chan uint64), + } +} + +func (f *gateBlockCounter) set(block uint64, err error) { + f.mu.Lock() + defer f.mu.Unlock() + + f.block = block + f.err = err + + if err != nil { + return + } + for height, channels := range f.waiters { + if block >= height { + for _, ch := range channels { + ch <- block + close(ch) + } + delete(f.waiters, height) + } + } +} + +// failWaiters closes all armed waiters without emitting a value, which the +// gate must treat as a clock failure. +func (f *gateBlockCounter) failWaiters() { + f.mu.Lock() + defer f.mu.Unlock() + + for height, channels := range f.waiters { + for _, ch := range channels { + close(ch) + } + delete(f.waiters, height) + } +} + +// deliverWaiters fires every waiter armed at or below the given height without +// changing the current block or error, so a test can make the cutover waiter +// report its target while the synchronous read path stays independently +// controlled. +func (f *gateBlockCounter) deliverWaiters(height uint64) { + f.mu.Lock() + defer f.mu.Unlock() + + for target, channels := range f.waiters { + if height >= target { + for _, ch := range channels { + ch <- height + close(ch) + } + delete(f.waiters, target) + } + } +} + +// holdNextRead arms a one-shot hold: the next CurrentBlock call snapshots its +// result immediately but does not return until release is called. The started +// channel closes once the held read has taken its snapshot; release is +// idempotent. +func (f *gateBlockCounter) holdNextRead() (<-chan struct{}, func()) { + f.mu.Lock() + defer f.mu.Unlock() + + started := make(chan struct{}) + releaseCh := make(chan struct{}) + f.holdStarted = started + f.holdRelease = releaseCh + + var once sync.Once + return started, func() { once.Do(func() { close(releaseCh) }) } +} + +// readCount returns how many CurrentBlock reads have been served, letting a +// test wait deterministically for a background read to have happened. +func (f *gateBlockCounter) readCount() uint64 { + f.mu.Lock() + defer f.mu.Unlock() + return f.reads +} + +func (f *gateBlockCounter) CurrentBlock() (uint64, error) { + f.mu.Lock() + f.reads++ + block, err := f.block, f.err + started, releaseCh := f.holdStarted, f.holdRelease + f.holdStarted, f.holdRelease = nil, nil + f.mu.Unlock() + + if started != nil { + close(started) + <-releaseCh + } + return block, err +} + +func (f *gateBlockCounter) WaitForBlockHeight(uint64) error { return nil } + +func (f *gateBlockCounter) BlockHeightWaiter( + height uint64, +) (<-chan uint64, error) { + f.mu.Lock() + defer f.mu.Unlock() + + if f.waiterErr != nil { + return nil, f.waiterErr + } + + ch := make(chan uint64, 1) + if f.block >= height { + ch <- f.block + close(ch) + return ch, nil + } + f.waiters[height] = append(f.waiters[height], ch) + return ch, nil +} + +func (f *gateBlockCounter) WatchBlocks(ctx context.Context) <-chan uint64 { + ch := make(chan uint64) + go func() { + <-ctx.Done() + close(ch) + }() + return ch +} + +// inertPollInterval keeps the supervisor loop out of a test's way; state is +// then driven exclusively by the waiter and the per-operation reads. +const inertPollInterval = time.Hour + +func newTestGate( + t *testing.T, + schedule Schedule, + initialBlock uint64, + pollInterval time.Duration, +) (*chainGate, *gateBlockCounter, *fakeMetrics) { + t.Helper() + + blockCounter := newGateBlockCounter(initialBlock) + metrics := newFakeMetrics() + + gate, err := newGate( + context.Background(), + schedule, + blockCounter, + metrics, + pollInterval, + ) + if err != nil { + t.Fatalf("failed to construct gate: [%v]", err) + } + t.Cleanup(gate.Close) + + return gate, blockCounter, metrics +} + +// eventually polls the condition until it holds or the timeout elapses. +func eventually(t *testing.T, condition func() bool) { + t.Helper() + + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + if condition() { + return + } + time.Sleep(2 * time.Millisecond) + } + t.Fatal("condition not reached before timeout") +} + +func TestNewGate_Validation(t *testing.T) { + metrics := newFakeMetrics() + blockCounter := newGateBlockCounter(100) + + if _, err := newGate( + context.Background(), Schedule{}, nil, metrics, time.Second, + ); err == nil { + t.Error("expected a nil block counter rejection") + } + + if _, err := newGate( + context.Background(), Schedule{}, blockCounter, nil, time.Second, + ); err == nil { + t.Error("expected a nil metrics recorder rejection") + } + + if _, err := newGate( + context.Background(), Schedule{}, blockCounter, metrics, 0, + ); err == nil { + t.Error("expected a non-positive poll interval rejection") + } + + if _, err := newGate( + context.Background(), + Schedule{CutoverBlock: maxSafeMetricInteger + 1}, + blockCounter, + metrics, + time.Second, + ); err == nil { + t.Error("expected an unprojectable cutover block rejection") + } + + failing := newGateBlockCounter(100) + failing.set(100, fmt.Errorf("clock down")) + if _, err := newGate( + context.Background(), Schedule{}, failing, metrics, time.Second, + ); err == nil { + t.Error("expected a chain-clock error at startup to be rejected") + } + + unprojectable := newGateBlockCounter(maxSafeMetricInteger + 1) + if _, err := newGate( + context.Background(), Schedule{}, unprojectable, metrics, time.Second, + ); err == nil { + t.Error("expected an unprojectable chain height rejection") + } + + noWaiter := newGateBlockCounter(100) + noWaiter.waiterErr = fmt.Errorf("waiter down") + if _, err := newGate( + context.Background(), + Schedule{CutoverBlock: 1000}, + noWaiter, + metrics, + time.Second, + ); err == nil { + t.Error("expected a waiter arming error at startup to be rejected") + } +} + +func TestNewGate_RegistersFixedMetrics(t *testing.T) { + _, _, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 500, inertPollInterval, + ) + + gauges := []string{ + clientinfo.MetricParticipationGateState, + clientinfo.MetricParticipationCurrentBlock, + clientinfo.MetricParticipationCutoverBlock, + clientinfo.MetricParticipationAllowed, + clientinfo.MetricParticipationActiveCeremonies, + clientinfo.MetricParticipationActiveLegacyCeremonies, + clientinfo.MetricParticipationActiveSecurityV2Ceremonies, + } + for _, name := range gauges { + if !metrics.hasGauge(name) { + t.Errorf("gauge [%s] not registered", name) + } + } + + counters := []string{ + clientinfo.MetricParticipationModeLegacyTotal, + clientinfo.MetricParticipationModeSecurityV2Total, + clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal, + clientinfo.MetricParticipationRefusalsTotal, + clientinfo.MetricParticipationCommitRefusalsTotal, + clientinfo.MetricParticipationClockErrorsTotal, + clientinfo.MetricParticipationClockAbortsTotal, + clientinfo.MetricParticipationQuiesceTotal, + clientinfo.MetricParticipationQuiesceForcedAbortsTotal, + clientinfo.MetricHeartbeatPenaltySuppressedTotal, + } + for _, ceremony := range AllCeremonies() { + counters = append( + counters, + clientinfo.ParticipationRefusalMetricName(string(ceremony)), + ) + } + for _, name := range counters { + if !metrics.hasCounter(name) { + t.Errorf("counter [%s] not registered", name) + } + } + + if got := metrics.gauge( + clientinfo.MetricParticipationCutoverBlock, + ); got != 1000 { + t.Errorf("expected cutover block gauge [1000], got [%f]", got) + } + if got := metrics.gauge( + clientinfo.MetricParticipationCurrentBlock, + ); got != 500 { + t.Errorf("expected current block gauge [500], got [%f]", got) + } + if got := metrics.gauge(clientinfo.MetricParticipationAllowed); got != 1 { + t.Errorf("expected allowed gauge [1], got [%f]", got) + } + if got := metrics.gauge( + clientinfo.MetricParticipationGateState, + ); got != float64(StateOpenLegacy) { + t.Errorf("expected state gauge [%d], got [%f]", StateOpenLegacy, got) + } +} + +func TestGate_CeremonyListMatchesClientInfo(t *testing.T) { + fromClientInfo := clientinfo.GetAllParticipationCeremonies() + fromGate := AllCeremonies() + + if len(fromClientInfo) != len(fromGate) { + t.Fatalf( + "ceremony list length drift: clientinfo [%d], participation [%d]", + len(fromClientInfo), + len(fromGate), + ) + } + for i, ceremony := range fromGate { + if fromClientInfo[i] != string(ceremony) { + t.Errorf( + "ceremony list drift at [%d]: clientinfo [%s], "+ + "participation [%s]", + i, + fromClientInfo[i], + ceremony, + ) + } + } +} + +func TestGate_StateTransitionsAtCutoverViaWaiter(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + if state := gate.State().State; state != StateOpenLegacy { + t.Fatalf("expected initial state open_legacy, got [%s]", state) + } + + // The armed cutover waiter must flip the state eagerly, without waiting + // for the (inert) supervisor poll. + blockCounter.set(1000, nil) + eventually(t, func() bool { + return gate.State().State == StateOpenSecurityV2 + }) +} + +func TestGate_BeginModeFromCanonicalAnchor(t *testing.T) { + gate, _, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 1500, inertPollInterval, + ) + + // A pre-cutover chain event confirmed after the cutover block classifies + // by the event's canonical block, not the callback's local arrival height. + legacy, err := gate.Begin(TBTCDKG, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if legacy.Mode() != ModeLegacy { + t.Errorf("expected legacy mode, got [%s]", legacy.Mode()) + } + if legacy.CanonicalStartBlock() != 999 { + t.Errorf( + "expected canonical start block [999], got [%d]", + legacy.CanonicalStartBlock(), + ) + } + if legacy.Ceremony() != TBTCDKG { + t.Errorf("expected ceremony [tbtc_dkg], got [%s]", legacy.Ceremony()) + } + + atCutover, err := gate.Begin(TBTCSigning, 1000) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if atCutover.Mode() != ModeSecurityV2 { + t.Errorf("expected security_v2 mode, got [%s]", atCutover.Mode()) + } + + after, err := gate.Begin(BeaconDKG, 1500) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if after.Mode() != ModeSecurityV2 { + t.Errorf("expected security_v2 mode, got [%s]", after.Mode()) + } + + snapshot := gate.State() + if snapshot.ActiveCeremonies != 3 || + snapshot.ActiveLegacyCeremonies != 1 || + snapshot.ActiveSecurityV2Ceremonies != 2 { + t.Errorf( + "expected active counts 3/1/2, got %d/%d/%d", + snapshot.ActiveCeremonies, + snapshot.ActiveLegacyCeremonies, + snapshot.ActiveSecurityV2Ceremonies, + ) + } + + if got := metrics.counter( + clientinfo.MetricParticipationModeLegacyTotal, + ); got != 1 { + t.Errorf("expected legacy mode counter [1], got [%f]", got) + } + if got := metrics.counter( + clientinfo.MetricParticipationModeSecurityV2Total, + ); got != 2 { + t.Errorf("expected security_v2 mode counter [2], got [%f]", got) + } + + legacy.Close() + atCutover.Close() + after.Close() + + if active := gate.State().ActiveCeremonies; active != 0 { + t.Errorf("expected zero active ceremonies, got [%d]", active) + } + + // Close is idempotent: a second close must not unbalance the counts. + legacy.Close() + if active := gate.State().ActiveCeremonies; active != 0 { + t.Errorf( + "expected zero active ceremonies after double close, got [%d]", + active, + ) + } +} + +func TestGate_BeginRejectsInvalidAnchors(t *testing.T) { + gate, _, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 500, inertPollInterval, + ) + + if _, err := gate.Begin(TBTCDKG, 501); !errors.Is(err, ErrInvalidAnchor) { + t.Errorf("expected a future anchor rejection, got: [%v]", err) + } + + if _, err := gate.Begin(TBTCDKG, 0); !errors.Is(err, ErrInvalidAnchor) { + t.Errorf("expected a zero anchor rejection, got: [%v]", err) + } + + // An anchor equal to the current height is valid: the event is in the + // current block. + permit, err := gate.Begin(TBTCDKG, 500) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + permit.Close() + + if got := metrics.counter( + clientinfo.MetricParticipationRefusalsTotal, + ); got != 2 { + t.Errorf("expected refusals counter [2], got [%f]", got) + } + if got := metrics.counter( + clientinfo.ParticipationRefusalMetricName(string(TBTCDKG)), + ); got != 2 { + t.Errorf("expected tbtc_dkg refusals counter [2], got [%f]", got) + } +} + +func TestGate_UnknownCeremonyRejected(t *testing.T) { + gate, _, _ := newTestGate(t, Schedule{CutoverBlock: 1000}, 500, inertPollInterval) + + if _, err := gate.Begin(Ceremony("bogus"), 100); err == nil { + t.Error("expected an unknown ceremony rejection") + } +} + +func TestGate_DisabledScheduleAlwaysLegacy(t *testing.T) { + gate, _, _ := newTestGate(t, Schedule{}, 50, inertPollInterval) + + if state := gate.State().State; state != StateDisabled { + t.Fatalf("expected disabled state, got [%s]", state) + } + + // The developer-only disabled schedule accepts a genesis anchor and + // always selects legacy. + for _, anchor := range []uint64{0, 50} { + permit, err := gate.Begin(TBTCSigning, anchor) + if err != nil { + t.Fatalf("unexpected error for anchor [%d]: [%v]", anchor, err) + } + if permit.Mode() != ModeLegacy { + t.Errorf( + "anchor [%d]: expected legacy mode, got [%s]", + anchor, + permit.Mode(), + ) + } + + // The disabled schedule never suppresses penalties by height. + if err := permit.CheckCommit( + "test_penalty", PenaltyCommit, + ); err != nil { + t.Errorf("unexpected penalty fence error: [%v]", err) + } + + permit.Close() + } + + if _, err := gate.Begin(TBTCSigning, 51); !errors.Is(err, ErrInvalidAnchor) { + t.Errorf("expected a future anchor rejection, got: [%v]", err) + } +} + +func TestGate_PermitSurvivesCrossingCutover(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if permit.Mode() != ModeLegacy { + t.Fatalf("expected legacy mode, got [%s]", permit.Mode()) + } + + // Crossing the cutover block must not cancel the permit or mutate its + // mode. + blockCounter.set(1005, nil) + + select { + case <-permit.Context().Done(): + t.Fatal("crossing the cutover block must not cancel a permit") + default: + } + + if permit.Mode() != ModeLegacy { + t.Errorf("permit mode mutated to [%s]", permit.Mode()) + } + + // A legacy completion commit after the cutover block is allowed and + // counted. + if err := permit.CheckCommit( + "result_submission", CompletionCommit, + ); err != nil { + t.Errorf("unexpected completion fence error: [%v]", err) + } + if got := metrics.counter( + clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal, + ); got != 1 { + t.Errorf("expected legacy completions counter [1], got [%f]", got) + } + + if state := gate.State().State; state != StateOpenSecurityV2 { + t.Errorf("expected open_security_v2 state, got [%s]", state) + } + if active := gate.State().ActiveLegacyCeremonies; active != 1 { + t.Errorf("expected one active legacy ceremony, got [%d]", active) + } + + permit.Close() +} + +func TestGate_LegacyCompletionBeforeCutoverNotCounted(t *testing.T) { + gate, _, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 500, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 500) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + if err := permit.CheckCommit("broadcast", CompletionCommit); err != nil { + t.Errorf("unexpected completion fence error: [%v]", err) + } + if got := metrics.counter( + clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal, + ); got != 0 { + t.Errorf("expected legacy completions counter [0], got [%f]", got) + } +} + +func TestGate_LegacyPenaltyFence(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + heartbeat, err := gate.Begin(TBTCHeartbeat, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer heartbeat.Close() + + timeoutReport, err := gate.Begin(BeaconTimeoutReport, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer timeoutReport.Close() + + // Below the cutover block, a legacy penalty commit is normal work. + if err := heartbeat.CheckCommit( + "inactivity_claim", PenaltyCommit, + ); err != nil { + t.Fatalf("unexpected penalty fence error below cutover: [%v]", err) + } + + // At and after the cutover block, a legacy penalty commit is suppressed. + blockCounter.set(1000, nil) + err = heartbeat.CheckCommit("inactivity_claim", PenaltyCommit) + if !errors.Is(err, ErrPenaltySuppressed) { + t.Errorf("expected a suppressed penalty, got: [%v]", err) + } + if got := metrics.counter( + clientinfo.MetricHeartbeatPenaltySuppressedTotal, + ); got != 1 { + t.Errorf("expected heartbeat suppression counter [1], got [%f]", got) + } + + // A non-heartbeat penalty suppression counts as a commit refusal but not + // as a heartbeat suppression. + err = timeoutReport.CheckCommit("timeout_report", PenaltyCommit) + if !errors.Is(err, ErrPenaltySuppressed) { + t.Errorf("expected a suppressed penalty, got: [%v]", err) + } + if got := metrics.counter( + clientinfo.MetricHeartbeatPenaltySuppressedTotal, + ); got != 1 { + t.Errorf( + "expected heartbeat suppression counter to stay [1], got [%f]", + got, + ) + } + if got := metrics.counter( + clientinfo.MetricParticipationCommitRefusalsTotal, + ); got != 2 { + t.Errorf("expected commit refusals counter [2], got [%f]", got) + } + + // A completion commit for the same legacy permit remains allowed. + if err := heartbeat.CheckCommit( + "heartbeat_signature", CompletionCommit, + ); err != nil { + t.Errorf("unexpected completion fence error: [%v]", err) + } +} + +func TestGate_SecurityV2CommitFences(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 1200, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 1100) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + if permit.Mode() != ModeSecurityV2 { + t.Fatalf("expected security_v2 mode, got [%s]", permit.Mode()) + } + + if err := permit.CheckCommit("broadcast", CompletionCommit); err != nil { + t.Fatalf("unexpected completion fence error: [%v]", err) + } + if err := permit.CheckCommit("penalty", PenaltyCommit); err != nil { + t.Fatalf("unexpected penalty fence error: [%v]", err) + } + + // After a deep reorg below the cutover block, a security-v2 commit must + // be refused; the permit itself remains alive. + blockCounter.set(999, nil) + err = permit.CheckCommit("broadcast", CompletionCommit) + if !errors.Is(err, ErrCommitBeforeCutover) { + t.Errorf("expected a below-cutover refusal, got: [%v]", err) + } + select { + case <-permit.Context().Done(): + t.Fatal("a refused commit must not cancel the permit") + default: + } + + // Once the chain recovers, the same permit commits normally again. + blockCounter.set(1200, nil) + if err := permit.CheckCommit("broadcast", CompletionCommit); err != nil { + t.Errorf("unexpected completion fence error after recovery: [%v]", err) + } +} + +func TestGate_UnknownCommitClassFailsClosed(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCHeartbeat, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + // An unknown class must not become an implicit completion or penalty. + // Exercise it after C, where treating it as neither would bypass the + // legacy penalty fence. + blockCounter.set(1000, nil) + err = permit.CheckCommit("unknown", CommitClass(0)) + if !errors.Is(err, ErrInvalidCommitClass) { + t.Errorf("expected an invalid commit class refusal, got: [%v]", err) + } + if !IsGateRefusal(err) { + t.Errorf("expected the invalid commit class to be a gate refusal: [%v]", err) + } + if got := metrics.counter( + clientinfo.MetricParticipationCommitRefusalsTotal, + ); got != 1 { + t.Errorf("expected commit refusals counter [1], got [%f]", got) + } + if got := metrics.counter( + clientinfo.MetricParticipationLegacyCompletionsAfterCutoverTotal, + ); got != 0 { + t.Errorf("expected legacy completions counter [0], got [%f]", got) + } + if got := metrics.counter( + clientinfo.MetricHeartbeatPenaltySuppressedTotal, + ); got != 0 { + t.Errorf("expected heartbeat suppressions counter [0], got [%f]", got) + } +} + +func TestGate_ResumeOnlyForBeaconRelaySigning(t *testing.T) { + gate, _, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 1500, inertPollInterval, + ) + + for _, ceremony := range []Ceremony{ + TBTCDKG, + TBTCSigning, + TBTCHeartbeat, + BeaconDKG, + BeaconRelayForwarding, + BeaconTimeoutReport, + } { + if _, err := gate.Resume( + ceremony, 900, + ); !errors.Is(err, ErrResumeUnsupported) { + t.Errorf( + "expected resume rejection for [%s], got: [%v]", + ceremony, + err, + ) + } + } + + // The beacon relay restart path resumes with the mode pinned from the + // on-chain request start block. + legacy, err := gate.Resume(BeaconRelaySigning, 900) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if legacy.Mode() != ModeLegacy { + t.Errorf("expected legacy mode, got [%s]", legacy.Mode()) + } + legacy.Close() + + hardened, err := gate.Resume(BeaconRelaySigning, 1100) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if hardened.Mode() != ModeSecurityV2 { + t.Errorf("expected security_v2 mode, got [%s]", hardened.Mode()) + } + hardened.Close() + + if _, err := gate.Resume( + BeaconRelaySigning, 2000, + ); !errors.Is(err, ErrInvalidAnchor) { + t.Errorf("expected a future anchor rejection, got: [%v]", err) + } +} + +func TestGate_ClockFailureCancelsPermits(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + legacy, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + blockCounter.set(1100, nil) + hardened, err := gate.Begin(TBTCDKG, 1100) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + // A failed synchronous read anywhere atomically fails the whole gate. + blockCounter.set(1100, fmt.Errorf("rpc down")) + if _, err := gate.Begin( + TBTCSigning, 1100, + ); !errors.Is(err, ErrClockUnavailable) { + t.Fatalf("expected a clock-unavailable refusal, got: [%v]", err) + } + + if state := gate.State().State; state != StateClockUnavailable { + t.Errorf("expected clock_unavailable state, got [%s]", state) + } + if gate.State().Allowed { + t.Error("expected the gate to refuse new permits") + } + + for _, p := range []Permit{legacy, hardened} { + select { + case <-p.Context().Done(): + default: + t.Fatal("expected the permit to be canceled by clock failure") + } + if cause := context.Cause( + p.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Errorf( + "expected cancellation cause clock-unavailable, got: [%v]", + cause, + ) + } + if err := p.CheckCommit( + "anything", CompletionCommit, + ); !errors.Is(err, ErrClockUnavailable) { + t.Errorf( + "expected a canceled-permit commit refusal, got: [%v]", + err, + ) + } + } + + if got := metrics.counter( + clientinfo.MetricParticipationClockAbortsTotal, + ); got != 2 { + t.Errorf("expected clock aborts counter [2], got [%f]", got) + } + if got := metrics.counter( + clientinfo.MetricParticipationClockErrorsTotal, + ); got == 0 { + t.Error("expected a nonzero clock errors counter") + } + + // The next successful read recomputes the state, but canceled permits do + // not revive. + blockCounter.set(1100, nil) + fresh, err := gate.Begin(TBTCSigning, 1100) + if err != nil { + t.Fatalf("unexpected error after clock recovery: [%v]", err) + } + if state := gate.State().State; state != StateOpenSecurityV2 { + t.Errorf("expected open_security_v2 state, got [%s]", state) + } + if cause := context.Cause( + legacy.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Error("expected the canceled permit to stay canceled") + } + + fresh.Close() + legacy.Close() + hardened.Close() +} + +func TestGate_ClockFailureViaSupervisorPoll(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, 5*time.Millisecond, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + blockCounter.set(999, fmt.Errorf("rpc down")) + eventually(t, func() bool { + return gate.State().State == StateClockUnavailable + }) + select { + case <-permit.Context().Done(): + default: + t.Fatal("expected the supervisor to cancel the permit") + } + + // Recovery restores the open state without reviving the permit. + blockCounter.set(999, nil) + eventually(t, func() bool { + return gate.State().State == StateOpenLegacy + }) + if cause := context.Cause( + permit.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Error("expected the canceled permit to stay canceled") + } +} + +func TestGate_WaiterCloseWithoutValueIsClockFailure(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + blockCounter.failWaiters() + eventually(t, func() bool { + return gate.State().State == StateClockUnavailable + }) + select { + case <-permit.Context().Done(): + default: + t.Fatal("expected a waiter failure to cancel the permit") + } +} + +func TestGate_QuiesceLifecycle(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + legacy, err := gate.Begin(TBTCHeartbeat, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + blockCounter.set(1100, nil) + hardened, err := gate.Begin(TBTCSigning, 1100) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + done := gate.Quiesce(fmt.Errorf("shutdown signal")) + + if state := gate.State().State; state != StateQuiescing { + t.Errorf("expected quiescing state, got [%s]", state) + } + if _, err := gate.Begin( + TBTCSigning, 1100, + ); !errors.Is(err, ErrQuiescing) { + t.Errorf("expected a quiescing refusal, got: [%v]", err) + } + + // Quiescence keeps existing permits alive to natural completion. + select { + case <-legacy.Context().Done(): + t.Fatal("quiescence must not cancel existing permits") + default: + } + + // Penalty commits are refused for every permit from the transition + // onward; completion commits remain allowed. + if err := legacy.CheckCommit( + "inactivity_claim", PenaltyCommit, + ); !errors.Is(err, ErrPenaltySuppressed) { + t.Errorf("expected a suppressed legacy penalty, got: [%v]", err) + } + if err := hardened.CheckCommit( + "timeout_report", PenaltyCommit, + ); !errors.Is(err, ErrPenaltySuppressed) { + t.Errorf("expected a suppressed security-v2 penalty, got: [%v]", err) + } + if err := legacy.CheckCommit( + "heartbeat_signature", CompletionCommit, + ); err != nil { + t.Errorf("unexpected completion fence error: [%v]", err) + } + if err := hardened.CheckCommit("broadcast", CompletionCommit); err != nil { + t.Errorf("unexpected completion fence error: [%v]", err) + } + + // The quiesce channel closes exactly when the active count reaches zero. + select { + case <-done: + t.Fatal("quiesce channel closed with active permits") + default: + } + + // Quiesce is idempotent and returns the same channel. + if again := gate.Quiesce(fmt.Errorf("second signal")); again != done { + t.Error("expected the same quiesce channel") + } + if got := metrics.counter( + clientinfo.MetricParticipationQuiesceTotal, + ); got != 1 { + t.Errorf("expected quiesce counter [1], got [%f]", got) + } + + legacy.Close() + select { + case <-done: + t.Fatal("quiesce channel closed with one active permit") + default: + } + + hardened.Close() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("quiesce channel did not close at zero active permits") + } +} + +func TestGate_QuiescenceCapturesRealPermitInventoryAtomically(t *testing.T) { + const cutover = uint64(1000) + capturedAt := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + blockCounter := newGateBlockCounter(cutover - 1) + recorder := &recordingQuiescenceSnapshotRecorder{} + + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + blockCounter, + newFakeMetrics(), + inertPollInterval, + WithArtifactIdentity("v2.1.0", "revision-test"), + WithQuiescenceSnapshotRecorder(recorder), + withGateTimeSource(func() time.Time { return capturedAt }), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + if _, err := gate.Begin( + TBTCSigning, + cutover-1, + ); !errors.Is(err, ErrInvalidPermitIdentity) { + t.Fatalf( + "expected production issuance without identity to fail, got [%v]", + err, + ) + } + + legacy, err := gate.Begin( + TBTCSigning, + cutover-1, + PermitIdentity{ + WorkID: "wallet-action-legacy", + PermitID: "wallet-legacy", + }, + ) + if err != nil { + t.Fatal(err) + } + if _, err := gate.Begin( + TBTCSigning, + cutover-1, + PermitIdentity{ + WorkID: "wallet-action-legacy", + PermitID: "wallet-legacy", + }, + ); !errors.Is(err, ErrInvalidPermitIdentity) { + t.Fatalf("expected duplicate permit identity rejection, got [%v]", err) + } + blockCounter.set(cutover, nil) + securityV2, err := gate.Begin( + BeaconRelaySigning, + cutover, + PermitIdentity{ + WorkID: "relay-request-security-v2", + PermitID: "2", + OperatedMembers: MemberIndexes{2}, + }, + ) + if err != nil { + t.Fatal(err) + } + + before := gate.State() + if len(before.ActivePermits) != 2 { + t.Fatalf( + "expected the live gate snapshot to inventory two permits, got [%d]", + len(before.ActivePermits), + ) + } + + gate.Quiesce(fmt.Errorf("rollback drill")) + legacy.Close() + securityV2.Close() + + snapshot, ok := gate.QuiescenceSnapshot() + if !ok { + t.Fatal("expected a quiescence snapshot") + } + if snapshot.CapturedAt != capturedAt { + t.Errorf( + "expected capture time [%s], got [%s]", + capturedAt, + snapshot.CapturedAt, + ) + } + if snapshot.ReleaseVersion != "v2.1.0" || + snapshot.ReleaseRevision != "revision-test" || + snapshot.ReleaseEpoch != CompiledEpoch.String() { + t.Errorf("unexpected artifact identity: %+v", snapshot) + } + if snapshot.CutoverBlock != cutover || + snapshot.State != StateQuiescing.String() || + snapshot.QuiesceCause != "rollback drill" { + t.Errorf("unexpected quiescence binding: %+v", snapshot) + } + if snapshot.ActiveCeremonies != 2 || + snapshot.ActiveLegacyCeremonies != 1 || + snapshot.ActiveSecurityV2Ceremonies != 1 || + len(snapshot.ActivePermits) != 2 { + t.Errorf("unexpected captured inventory: %+v", snapshot) + } + for _, permit := range snapshot.ActivePermits { + if !permit.IdentityBound { + t.Errorf("expected a bound permit identity: %+v", permit) + } + } + + recorded := recorder.recorded() + if len(recorded) != 1 { + t.Fatalf("expected exactly one persisted snapshot, got [%d]", len(recorded)) + } + if fmt.Sprint(recorded[0]) != fmt.Sprint(snapshot) { + t.Errorf( + "persisted snapshot differs from the gate capture\npersisted: %+v\n"+ + "captured: %+v", + recorded[0], + snapshot, + ) + } + + // Closing permits after the transition cannot rewrite history. + if active := gate.State().ActiveCeremonies; active != 0 { + t.Fatalf("expected the live gate to drain, got [%d] permits", active) + } + again, ok := gate.QuiescenceSnapshot() + if !ok || again.ActiveCeremonies != 2 || len(again.ActivePermits) != 2 { + t.Errorf("quiescence inventory mutated after drain: %+v", again) + } + + // Returned slices are defensive copies. + again.ActivePermits[0].WorkID = "mutated" + final, _ := gate.QuiescenceSnapshot() + if final.ActivePermits[0].WorkID == "mutated" { + t.Error("the caller mutated the gate's retained quiescence inventory") + } +} + +func TestGate_QuiesceOnIdleGateClosesImmediately(t *testing.T) { + gate, _, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + done := gate.Quiesce(fmt.Errorf("shutdown signal")) + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("quiesce channel did not close on an idle gate") + } +} + +func TestGate_CloseForcesQuiesceDeadline(t *testing.T) { + gate, _, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + first, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + second, err := gate.Begin(TBTCHeartbeat, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + + done := gate.Quiesce(fmt.Errorf("shutdown signal")) + gate.Close() + + for _, p := range []Permit{first, second} { + select { + case <-p.Context().Done(): + default: + t.Fatal("expected the permit to be force-canceled at close") + } + if cause := context.Cause( + p.Context(), + ); !errors.Is(cause, ErrQuiesceDeadline) { + t.Errorf( + "expected cancellation cause quiesce-deadline, got: [%v]", + cause, + ) + } + } + + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("quiesce channel did not close at gate close") + } + + if got := metrics.counter( + clientinfo.MetricParticipationQuiesceForcedAbortsTotal, + ); got != 2 { + t.Errorf("expected forced aborts counter [2], got [%f]", got) + } + + if _, err := gate.Begin(TBTCSigning, 999); !errors.Is(err, ErrQuiescing) { + t.Errorf("expected a refusal after close, got: [%v]", err) + } + if err := first.CheckCommit( + "anything", CompletionCommit, + ); !errors.Is(err, ErrQuiesceDeadline) { + t.Errorf("expected a forced-abort commit refusal, got: [%v]", err) + } + + // Close is idempotent: no double counting. + gate.Close() + if got := metrics.counter( + clientinfo.MetricParticipationQuiesceForcedAbortsTotal, + ); got != 2 { + t.Errorf( + "expected forced aborts counter to stay [2], got [%f]", + got, + ) + } + + first.Close() + second.Close() +} + +// TestGate_DrainedJoinsForcedPermitRelease pins the two-phase forced +// cancellation: Close closes the quiesce channel immediately, but the drained +// channel stays open until the owner of every force-canceled permit releases +// it — the window in which the cancellation cleanup, quarantine writes +// included, is still running. +func TestGate_DrainedJoinsForcedPermitRelease(t *testing.T) { + gate, _, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + drained := gate.Drained() + + done := gate.Quiesce(fmt.Errorf("shutdown signal")) + gate.Close() + + select { + case <-done: + default: + t.Fatal("quiesce channel must close at gate close") + } + select { + case <-drained: + t.Fatal("drained channel closed while a force-canceled permit was held") + default: + } + + permit.Close() + select { + case <-drained: + case <-time.After(time.Second): + t.Fatal("drained channel did not close at the last permit release") + } +} + +// TestGate_DrainedClosesWithNaturalQuiesceCompletion proves the drained +// channel needs no Close on the natural path: once quiescence begins, the +// release of the last permit both completes the quiesce drain and drains the +// gate. +func TestGate_DrainedClosesWithNaturalQuiesceCompletion(t *testing.T) { + gate, _, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + drained := gate.Drained() + + gate.Quiesce(fmt.Errorf("shutdown signal")) + select { + case <-drained: + t.Fatal("drained channel closed with an active permit") + default: + } + + permit.Close() + select { + case <-drained: + case <-time.After(time.Second): + t.Fatal("drained channel did not close at natural completion") + } +} + +// TestGate_DrainedStaysOpenWhileGateIssuesPermits pins the boundary of the +// drained condition: neither an active count reaching zero during normal +// operation nor a clock-failure cancellation drains the gate, because both +// leave it able to issue permits again. Only quiescence or close does. +func TestGate_DrainedStaysOpenWhileGateIssuesPermits(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + drained := gate.Drained() + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + permit.Close() + select { + case <-drained: + t.Fatal("drained channel closed while the gate still issues permits") + default: + } + + held, err := gate.Begin(TBTCDKG, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + blockCounter.set(999, fmt.Errorf("rpc down")) + if _, err := gate.Begin( + TBTCHeartbeat, 999, + ); !errors.Is(err, ErrClockUnavailable) { + t.Fatalf("expected a clock-unavailable refusal, got: [%v]", err) + } + if cause := context.Cause( + held.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Fatalf("expected a clock-failure cancellation, got: [%v]", cause) + } + held.Close() + select { + case <-drained: + t.Fatal("drained channel closed on a clock-failure cancellation") + default: + } + + gate.Close() + select { + case <-drained: + case <-time.After(time.Second): + t.Fatal("drained channel did not close at the close of an idle gate") + } +} + +func TestGate_ClosedPermitCommitRefused(t *testing.T) { + gate, _, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + permit.Close() + + if err := permit.CheckCommit( + "anything", CompletionCommit, + ); !errors.Is(err, ErrPermitClosed) { + t.Errorf("expected a closed-permit commit refusal, got: [%v]", err) + } +} + +// TestGate_StaleClockSuccessCannotReopenGate pins the clock-sample ordering +// for permit issuance: a successful read initiated before a newer failing read +// but applied after it must be discarded. The issuing operation fails closed +// and the gate stays clock-unavailable instead of silently reopening. +func TestGate_StaleClockSuccessCannotReopenGate(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + existing, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer existing.Close() + + started, release := blockCounter.holdNextRead() + + // This Begin's read snapshots a healthy chain view, then stalls in flight. + staleResult := make(chan error, 1) + go func() { + _, err := gate.Begin(TBTCDKG, 999) + staleResult <- err + }() + <-started + + // A read initiated after the held one fails and must win permanently. + blockCounter.set(999, fmt.Errorf("rpc down")) + if _, err := gate.Begin( + TBTCHeartbeat, 999, + ); !errors.Is(err, ErrClockUnavailable) { + t.Fatalf("expected a clock-unavailable refusal, got: [%v]", err) + } + if state := gate.State().State; state != StateClockUnavailable { + t.Fatalf("expected clock_unavailable state, got [%s]", state) + } + + // The stale success lands last: it must not reopen the gate, must not + // issue a permit, and must not revive the canceled permit. + release() + if err := <-staleResult; !errors.Is(err, ErrClockUnavailable) { + t.Errorf("expected the stale Begin to fail closed, got: [%v]", err) + } + snapshot := gate.State() + if snapshot.State != StateClockUnavailable { + t.Errorf( + "expected the gate to stay clock_unavailable, got [%s]", + snapshot.State, + ) + } + if snapshot.ClockAvailable { + t.Error("expected the clock to stay unavailable") + } + if snapshot.Allowed { + t.Error("expected the gate to keep refusing new permits") + } + if cause := context.Cause( + existing.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Errorf("expected the canceled permit to stay canceled, got: [%v]", cause) + } +} + +// TestGate_StaleClockSuccessCannotReopenGateViaFence pins the same ordering +// for the commit fence path: a stalled successful fence read applied after a +// newer failure is discarded, the fence refuses, and the gate stays failed. +func TestGate_StaleClockSuccessCannotReopenGateViaFence(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + other, err := gate.Begin(TBTCHeartbeat, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer other.Close() + + started, release := blockCounter.holdNextRead() + + staleResult := make(chan error, 1) + go func() { + staleResult <- permit.CheckCommit( + "result_submission", CompletionCommit, + ) + }() + <-started + + blockCounter.set(999, fmt.Errorf("rpc down")) + if err := other.CheckCommit( + "heartbeat_signature", CompletionCommit, + ); !errors.Is(err, ErrClockUnavailable) { + t.Fatalf("expected a clock-unavailable fence refusal, got: [%v]", err) + } + if state := gate.State().State; state != StateClockUnavailable { + t.Fatalf("expected clock_unavailable state, got [%s]", state) + } + + release() + if err := <-staleResult; !errors.Is(err, ErrClockUnavailable) { + t.Errorf("expected the stale fence to fail closed, got: [%v]", err) + } + snapshot := gate.State() + if snapshot.State != StateClockUnavailable { + t.Errorf( + "expected the gate to stay clock_unavailable, got [%s]", + snapshot.State, + ) + } + if snapshot.ClockAvailable { + t.Error("expected the clock to stay unavailable") + } +} + +// TestGate_StaleClockFailureCannotCancelAfterNewerSuccess pins the symmetric +// ordering guarantee: a failing read initiated before a newer successful read +// but applied after it refuses only its own operation. It must not transition +// the gate to clock-unavailable or cancel permits on stale information. +func TestGate_StaleClockFailureCannotCancelAfterNewerSuccess(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + // The held read snapshots a transient failure, then stalls in flight. + blockCounter.set(999, fmt.Errorf("transient rpc error")) + started, release := blockCounter.holdNextRead() + + staleResult := make(chan error, 1) + go func() { + _, err := gate.Begin(TBTCDKG, 999) + staleResult <- err + }() + <-started + + // The chain recovers and a newer read succeeds before the stale failure + // lands. + blockCounter.set(1000, nil) + fresh, err := gate.Begin(BeaconDKG, 1000) + if err != nil { + t.Fatalf("unexpected error after recovery: [%v]", err) + } + defer fresh.Close() + + release() + // The operation whose own read failed still fails closed. + if err := <-staleResult; !errors.Is(err, ErrClockUnavailable) { + t.Errorf("expected the stale Begin to fail closed, got: [%v]", err) + } + // But the stale failure must not have transitioned the gate or canceled + // anything. + snapshot := gate.State() + if snapshot.State != StateOpenSecurityV2 { + t.Errorf("expected open_security_v2 state, got [%s]", snapshot.State) + } + if !snapshot.ClockAvailable { + t.Error("expected the clock to stay available") + } + select { + case <-permit.Context().Done(): + t.Error("a stale clock failure must not cancel permits") + default: + } + if got := metrics.counter( + clientinfo.MetricParticipationClockAbortsTotal, + ); got != 0 { + t.Errorf("expected zero clock aborts, got [%f]", got) + } +} + +// TestGate_WaiterFireWithFailingReadIsClockFailure pins that the cutover +// waiter is telemetry-only: recovery and state advancement require a +// successful synchronous read. A waiter that reports its target while the +// synchronous clock fails must produce clock-unavailable, not a transition, +// and must not project the waiter's height. +func TestGate_WaiterFireWithFailingReadIsClockFailure(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + // Reads fail from here on; the armed cutover waiter stays deliverable. + blockCounter.set(999, fmt.Errorf("rpc down")) + blockCounter.deliverWaiters(1000) + + eventually(t, func() bool { + return gate.State().State == StateClockUnavailable + }) + snapshot := gate.State() + if snapshot.CurrentBlock != 999 { + t.Errorf( + "expected the waiter height to be discarded and the current "+ + "block to stay [999], got [%d]", + snapshot.CurrentBlock, + ) + } + if cause := context.Cause( + permit.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Errorf("expected a clock-unavailable cancellation, got: [%v]", cause) + } +} + +// TestGate_WaiterFireWithLaggingReadStaysAuthoritative pins that a waiter +// firing at the cutover target cannot advance the state past what the +// authoritative synchronous read reports: a healthy read still below the +// cutover block keeps the gate open in legacy state. +func TestGate_WaiterFireWithLaggingReadStaysAuthoritative(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + reads := blockCounter.readCount() + blockCounter.deliverWaiters(1000) + + // Wait for the waiter-triggered authoritative poll to have read the + // (still lagging) clock. + eventually(t, func() bool { + return blockCounter.readCount() > reads + }) + + snapshot := gate.State() + if snapshot.State != StateOpenLegacy { + t.Errorf( + "expected the lagging read to keep open_legacy, got [%s]", + snapshot.State, + ) + } + if snapshot.CurrentBlock != 999 { + t.Errorf( + "expected current block [999] from the authoritative read, "+ + "got [%d]", + snapshot.CurrentBlock, + ) + } + + // Once the synchronous clock itself reports the cutover height, new work + // selects security-v2: the gate stayed live throughout. + blockCounter.set(1000, nil) + permit, err := gate.Begin(TBTCDKG, 1000) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if permit.Mode() != ModeSecurityV2 { + t.Errorf("expected security_v2 mode, got [%s]", permit.Mode()) + } + permit.Close() +} + +// TestGate_UnprojectableHeightIsClockFailure pins that a chain height the +// float64 metrics projection cannot represent exactly is handled as a clock +// failure at runtime instead of being exported imprecisely. +func TestGate_UnprojectableHeightIsClockFailure(t *testing.T) { + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: 1000}, 999, inertPollInterval, + ) + + permit, err := gate.Begin(TBTCSigning, 999) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + + blockCounter.set(maxSafeMetricInteger+1, nil) + if _, err := gate.Begin( + TBTCDKG, 999, + ); !errors.Is(err, ErrClockUnavailable) { + t.Fatalf("expected a clock-unavailable refusal, got: [%v]", err) + } + if state := gate.State().State; state != StateClockUnavailable { + t.Errorf("expected clock_unavailable state, got [%s]", state) + } + if cause := context.Cause( + permit.Context(), + ); !errors.Is(cause, ErrClockUnavailable) { + t.Errorf("expected a clock-unavailable cancellation, got: [%v]", cause) + } + if current := gate.State().CurrentBlock; current != 999 { + t.Errorf( + "expected the unprojectable height to be discarded and the "+ + "current block to stay [999], got [%d]", + current, + ) + } +} + +// TestGate_StaleUnprojectableHeightStillFailsItsOperation pins that an +// unprojectable height belongs to its own read's result: a Begin whose read +// snapshots a height above the metric projection limit fails closed even when +// a newer valid sample applies first, discards the stale sample, and keeps the +// gate available. The discarded sample must not fail the gate or cancel +// permits, exactly like a superseded RPC error. +func TestGate_StaleUnprojectableHeightStillFailsItsOperation(t *testing.T) { + // Constructing at the cutover height fires the construction-armed cutover + // waiter immediately, so its one-shot telemetry poll is the only background + // read; wait it out so nothing races the held-read choreography below. + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 1000, inertPollInterval, + ) + eventually(t, func() bool { return blockCounter.readCount() >= 2 }) + + existing, err := gate.Begin(TBTCSigning, 1000) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer existing.Close() + + // The held read snapshots an unprojectable height, then stalls in flight. + blockCounter.set(maxSafeMetricInteger+1, nil) + started, release := blockCounter.holdNextRead() + + staleResult := make(chan error, 1) + go func() { + p, err := gate.Begin(TBTCDKG, 1000) + if err == nil { + p.Close() + } + staleResult <- err + }() + <-started + + // The chain recovers and a newer read succeeds before the stale + // unprojectable sample lands. + blockCounter.set(1005, nil) + fresh, err := gate.Begin(BeaconDKG, 1005) + if err != nil { + t.Fatalf("unexpected error after recovery: [%v]", err) + } + defer fresh.Close() + + release() + // The operation whose own read was unprojectable still fails closed. + if err := <-staleResult; !errors.Is(err, ErrClockUnavailable) { + t.Errorf("expected the stale Begin to fail closed, got: [%v]", err) + } + // But the discarded stale sample must not have transitioned the gate or + // canceled anything. + snapshot := gate.State() + if snapshot.State != StateOpenSecurityV2 { + t.Errorf("expected open_security_v2 state, got [%s]", snapshot.State) + } + if !snapshot.ClockAvailable { + t.Error("expected the clock to stay available") + } + if snapshot.CurrentBlock != 1005 { + t.Errorf( + "expected the newer valid height [1005] to remain, got [%d]", + snapshot.CurrentBlock, + ) + } + select { + case <-existing.Context().Done(): + t.Error("a stale unprojectable sample must not cancel permits") + default: + } + if got := metrics.counter( + clientinfo.MetricParticipationClockAbortsTotal, + ); got != 0 { + t.Errorf("expected zero clock aborts, got [%f]", got) + } +} + +// TestGate_StaleUnprojectableHeightStillFailsItsFence pins the same guarantee +// for the commit fence path: a fence whose own read snapshots an unprojectable +// height refuses even when a newer valid sample applied first and kept the +// gate available for everyone else. +func TestGate_StaleUnprojectableHeightStillFailsItsFence(t *testing.T) { + gate, blockCounter, metrics := newTestGate( + t, Schedule{CutoverBlock: 1000}, 1000, inertPollInterval, + ) + eventually(t, func() bool { return blockCounter.readCount() >= 2 }) + + permit, err := gate.Begin(TBTCSigning, 1000) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer permit.Close() + other, err := gate.Begin(TBTCHeartbeat, 1000) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + defer other.Close() + + blockCounter.set(maxSafeMetricInteger+1, nil) + started, release := blockCounter.holdNextRead() + + staleResult := make(chan error, 1) + go func() { + staleResult <- permit.CheckCommit( + "result_submission", CompletionCommit, + ) + }() + <-started + + blockCounter.set(1005, nil) + if err := other.CheckCommit( + "heartbeat_signature", CompletionCommit, + ); err != nil { + t.Fatalf("unexpected fence error after recovery: [%v]", err) + } + + release() + // The fence whose own read was unprojectable still fails closed. + if err := <-staleResult; !errors.Is(err, ErrClockUnavailable) { + t.Errorf("expected the stale fence to fail closed, got: [%v]", err) + } + // The discarded stale sample left the gate available on the newer height. + snapshot := gate.State() + if snapshot.State != StateOpenSecurityV2 { + t.Errorf("expected open_security_v2 state, got [%s]", snapshot.State) + } + if !snapshot.ClockAvailable { + t.Error("expected the clock to stay available") + } + for _, p := range []Permit{permit, other} { + select { + case <-p.Context().Done(): + t.Error("a stale unprojectable sample must not cancel permits") + default: + } + } + if got := metrics.counter( + clientinfo.MetricParticipationClockAbortsTotal, + ); got != 0 { + t.Errorf("expected zero clock aborts, got [%f]", got) + } +} + +// TestGate_ConcurrentBeginAcrossCutover races permit issuance, commit fences, +// state reads, and permit closes against the chain crossing the cutover block, +// a mid-flight Quiesce, and the terminal gate Close, all genuinely +// overlapping. The invariants: a permit is pinned legacy for an anchor below C +// and security-v2 at/above C regardless of which goroutine observed C first; +// every fence outcome is one of the exactly allowed sentinels for its commit +// class (the clock never fails here, so a clock sentinel is a bug); and the +// active-permit accounting balances to zero once every owner closed. +func TestGate_ConcurrentBeginAcrossCutover(t *testing.T) { + const cutover = uint64(1000) + + gate, blockCounter, _ := newTestGate( + t, Schedule{CutoverBlock: cutover}, cutover-10, 3*time.Millisecond, + ) + + var wg sync.WaitGroup + lifecycleDone := make(chan struct{}) + + // Advance the chain across the cutover block while workers race, and keep + // stepping until the lifecycle goroutine has closed the gate so crossing, + // quiescence, and close all overlap live traffic. + wg.Add(1) + go func() { + defer wg.Done() + height := cutover - 10 + for { + blockCounter.set(height, nil) + height++ + select { + case <-lifecycleDone: + return + case <-time.After(500 * time.Microsecond): + } + } + }() + + // Quiesce once the gate has observably crossed the cutover block, then + // close shortly after, while workers still hammer the gate. + var quiesceDone <-chan struct{} + wg.Add(1) + go func() { + defer wg.Done() + defer close(lifecycleDone) + for gate.State().CurrentBlock < cutover+2 { + time.Sleep(200 * time.Microsecond) + } + quiesceDone = gate.Quiesce(fmt.Errorf("test quiesce")) + time.Sleep(2 * time.Millisecond) + gate.Close() + }() + + // The exact allowed fence outcomes. Completion commits stay allowed + // through crossing and quiescence and refuse only after the forced + // close; penalty commits are additionally suppressed for legacy permits + // at/after C and for every permit once quiescence begins. + completionAllowed := func(err error) bool { + return err == nil || errors.Is(err, ErrQuiesceDeadline) + } + penaltyAllowed := func(err error) bool { + return err == nil || + errors.Is(err, ErrPenaltySuppressed) || + errors.Is(err, ErrQuiesceDeadline) + } + + for worker := 0; worker < 8; worker++ { + wg.Add(1) + go func() { + defer wg.Done() + + var held []Permit + defer func() { + for _, p := range held { + p.Close() + } + }() + + // Run until the lifecycle finished, then a few bonus iterations + // so the closed-gate paths are exercised too. + bonus := 0 + for bonus <= 10 { + select { + case <-lifecycleDone: + bonus++ + default: + } + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + continue + } + + permit, err := gate.Begin(TBTCHeartbeat, anchor) + if err != nil { + // The gate may have quiesced or closed, and an anchor can + // race one step ahead of a concurrent reader's applied + // height. Nothing else is acceptable. + if !errors.Is(err, ErrQuiescing) && + !errors.Is(err, ErrInvalidAnchor) { + t.Errorf("unexpected Begin error: [%v]", err) + } + } else { + expected := ModeLegacy + if anchor >= cutover { + expected = ModeSecurityV2 + } + if permit.Mode() != expected { + t.Errorf( + "anchor [%d]: expected mode [%s], got [%s]", + anchor, + expected, + permit.Mode(), + ) + } + held = append(held, permit) + } + + // Fence every held permit and assert every outcome; permits + // held across the crossing, the quiescence, and the close are + // exactly the ones the fences must keep classifying. + for _, p := range held { + if err := p.CheckCommit( + "race_completion", CompletionCommit, + ); !completionAllowed(err) { + t.Errorf( + "unexpected completion fence outcome for "+ + "mode [%s]: [%v]", + p.Mode(), + err, + ) + } + if err := p.CheckCommit( + "race_penalty", PenaltyCommit, + ); !penaltyAllowed(err) { + t.Errorf( + "unexpected penalty fence outcome for "+ + "mode [%s]: [%v]", + p.Mode(), + err, + ) + } + } + _ = gate.State() + + // Close the oldest permit so ownership churns while newer + // permits keep spanning the lifecycle transitions. + if len(held) > 3 { + held[0].Close() + held = held[1:] + } + } + }() + } + + wg.Wait() + + if quiesceDone == nil { + t.Fatal("the lifecycle goroutine did not quiesce the gate") + } + select { + case <-quiesceDone: + default: + t.Error("expected the quiesce channel to be closed after gate close") + } + + // Every worker closed all its permits, including force-canceled ones, so + // the accounting must balance exactly. + if active := gate.State().ActiveCeremonies; active != 0 { + t.Errorf("expected zero active ceremonies, got [%d]", active) + } + if _, err := gate.Begin( + TBTCSigning, cutover, + ); !errors.Is(err, ErrQuiescing) { + t.Errorf("expected a quiescing refusal after close, got: [%v]", err) + } +} + +// TestGate_TerminalOutcomeRetryComparesSettlementByValue pins the retry path +// for an outcome carrying a chain settlement. The retry is expected to be an +// identical call, but a ceremony owner rebuilds its evidence each time, so the +// two settlements are equal values at different addresses. Comparing them by +// address would reject the retry as a contradictory second outcome and lose the +// disposition the ceremony actually reached. +func TestGate_TerminalOutcomeRetryComparesSettlementByValue(t *testing.T) { + const cutover = uint64(1_000) + capturedAt := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + recorder := &recordingQuiescenceSnapshotRecorder{ + terminalFailures: 1, + terminalErr: errors.New("transient journal failure"), + } + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + WithArtifactIdentity("v2.1.0", "revision-test"), + WithQuiescenceSnapshotRecorder(recorder), + withGateTimeSource(func() time.Time { return capturedAt }), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin( + TBTCHeartbeat, + cutover, + PermitIdentity{ + WorkID: "heartbeat-retry", + PermitID: "heartbeat-retry", + OperatedMembers: MemberIndexes{1}, + }, + ) + if err != nil { + t.Fatal(err) + } + gate.Quiesce(fmt.Errorf("rollback drill")) + + walletID := make([]byte, inactivityClaimWalletIDLength) + walletID[0] = 0x5c + reference, err := InactivityClaimSettlementReference( + walletID, + big.NewInt(4), + ) + if err != nil { + t.Fatal(err) + } + + // Each attempt builds its own settlement record, exactly as the ceremony + // owner does. + newEvidence := func() TerminalEvidence { + return TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: "heartbeat-result-identity", + ChainSettlement: &ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + Reference: reference, + }, + Contribution: testTranscriptContribution(TBTCHeartbeat, 1), + } + } + + if err := permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + newEvidence(), + ); !errors.Is(err, ErrTerminalOutcomePersistence) { + t.Fatalf("expected first persistence attempt to fail, got [%v]", err) + } + if err := permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + newEvidence(), + ); err != nil { + t.Fatalf("expected identical outcome retry to succeed, got [%v]", err) + } + permit.Close() + + outcomes := recorder.recordedOutcomes() + if len(outcomes) != 1 { + t.Fatalf("expected one terminal outcome, got [%d]", len(outcomes)) + } + if !outcomes[0].Evidence.Equal(newEvidence()) { + t.Errorf( + "unexpected terminal outcome after retry: %+v", + outcomes[0].Evidence, + ) + } + + // A retry that names a different settlement is a contradiction, not a + // retry, and must still be refused. + contradicting := newEvidence() + contradicting.ChainSettlement = &ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + } + if !TerminalEvidence.Equal(newEvidence(), newEvidence()) { + t.Error("two equal evidence values compared unequal") + } + if newEvidence().Equal(contradicting) { + t.Error("an unobserved settlement compared equal to an observed one") + } +} diff --git a/pkg/protocol/participation/identity_ownership_test.go b/pkg/protocol/participation/identity_ownership_test.go new file mode 100644 index 0000000000..04ae1889db --- /dev/null +++ b/pkg/protocol/participation/identity_ownership_test.go @@ -0,0 +1,83 @@ +package participation + +import ( + "fmt" + "go/ast" + "go/parser" + "go/token" + "io/fs" + "path/filepath" + "strings" + "testing" +) + +// TestProductionPermitIssuanceBindsIdentity prevents a new production +// ceremony hook from issuing a permit without the chain-work and local-permit +// identity required by the node-authored quiescence inventory. Tests may use +// the source-compatible unbound path; rollback evidence may not. +func TestProductionPermitIssuanceBindsIdentity(t *testing.T) { + repoRoot, err := filepath.Abs(filepath.Join("..", "..", "..")) + if err != nil { + t.Fatal(err) + } + + var violations []string + for _, scanRoot := range []string{ + filepath.Join(repoRoot, "pkg", "beacon"), + filepath.Join(repoRoot, "pkg", "tbtc"), + } { + err := filepath.WalkDir( + scanRoot, + func(path string, entry fs.DirEntry, err error) error { + if err != nil { + return err + } + if entry.IsDir() { + if entry.Name() == "gen" || entry.Name() == "testdata" { + return filepath.SkipDir + } + return nil + } + if !strings.HasSuffix(path, ".go") || + strings.HasSuffix(path, "_test.go") { + return nil + } + + fileSet := token.NewFileSet() + file, err := parser.ParseFile(fileSet, path, nil, 0) + if err != nil { + return fmt.Errorf("cannot parse [%s]: [%w]", path, err) + } + ast.Inspect(file, func(node ast.Node) bool { + call, ok := node.(*ast.CallExpr) + if !ok { + return true + } + selector, ok := call.Fun.(*ast.SelectorExpr) + if !ok || + (selector.Sel.Name != "Begin" && + selector.Sel.Name != "Resume") { + return true + } + if len(call.Args) != 3 { + violations = append(violations, fmt.Sprintf( + "%s: production participation permits must "+ + "supply exactly one PermitIdentity", + fileSet.Position(call.Pos()), + )) + } + return true + }) + + return nil + }, + ) + if err != nil { + t.Fatal(err) + } + } + + for _, violation := range violations { + t.Error(violation) + } +} diff --git a/pkg/protocol/participation/mode.go b/pkg/protocol/participation/mode.go new file mode 100644 index 0000000000..63c7b1331b --- /dev/null +++ b/pkg/protocol/participation/mode.go @@ -0,0 +1,44 @@ +// Package participation implements the chain-clocked protocol cutover from +// the legacy cryptographic behavior to the hardened security-v2 behavior: the +// compiled release epoch, the one-value cutover schedule and its per-network +// resolver, the participation gate that issues per-ceremony permits with the +// protocol mode pinned from each ceremony's canonical chain anchor, and the +// node-local roster of post-cutover legacy peer sightings. +// +// The gate is the only component that derives protocol modes from the chain +// clock. There is no process-wide mutable mode: a pre-cutover legacy ceremony +// may still be completing while a post-cutover security-v2 ceremony begins, +// and each carries its own immutable permit. +package participation + +// ProtocolMode identifies which cryptographic compatibility mode a ceremony +// participates in. +// +// A mode is selected by the gate from a ceremony's canonical chain anchor at +// permit issuance — legacy below the cutover block, security-v2 at or above +// it — and is pinned in that ceremony's permit for its entire lifetime. +// Components that receive a mode directly (test fixtures, strategy bundles) +// must treat it as immutable for the ceremony it was issued for. +type ProtocolMode uint8 + +const ( + // ModeLegacy is the production-compatible legacy cryptographic mode: the + // session-ID, key-derivation, and hash-to-point behavior of the pre-hardening + // releases. + ModeLegacy ProtocolMode = iota + 1 + // ModeSecurityV2 is the hardened PR #4109 cryptographic mode. + ModeSecurityV2 +) + +// String returns the canonical string form of the protocol mode: exactly +// "legacy" or "security_v2". Any other value renders as "unknown". +func (m ProtocolMode) String() string { + switch m { + case ModeLegacy: + return "legacy" + case ModeSecurityV2: + return "security_v2" + default: + return "unknown" + } +} diff --git a/pkg/protocol/participation/permit_ownership_test.go b/pkg/protocol/participation/permit_ownership_test.go new file mode 100644 index 0000000000..fcf95d3a55 --- /dev/null +++ b/pkg/protocol/participation/permit_ownership_test.go @@ -0,0 +1,615 @@ +package participation + +import ( + "context" + "errors" + "fmt" + "strings" + "sync" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// A permit's operated seats are the fleet's only complete account of who held +// which seat on one piece of work: they are recorded before the ceremony runs, +// so they survive an ending that produced no result at all. These tests hold +// that account to the two things a reader of it depends on — that it is there +// whatever became of the permit, and that the record written afterwards cannot +// contradict it. + +// TestPermit_OperatedSeatsSurviveEveryEnding is the case an ownership map built +// from completions gets wrong. A node that operated a seat and then ended +// without a threshold result operated it just as much as one that finished, and +// a reader that cannot see the seat attributes it to whoever else was on the +// network. +func TestPermit_OperatedSeatsSurviveEveryEnding(t *testing.T) { + const cutover = uint64(1_000) + + tests := map[string]struct { + ceremony Ceremony + identity PermitIdentity + outcome TerminalOutcome + evidence TerminalEvidence + }{ + "a signing that reached no threshold": { + ceremony: TBTCSigning, + identity: PermitIdentity{ + WorkID: "wallet-action-exhausted", + PermitID: "wallet", + OperatedMembers: MemberIndexes{3, 9}, + }, + outcome: TerminalOutcomeExhausted, + evidence: TerminalEvidence{Kind: TerminalEvidenceNoThreshold}, + }, + "a DKG whose key material was quarantined": { + ceremony: BeaconDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("b", 64), + PermitID: "4", + OperatedMembers: MemberIndexes{4}, + }, + outcome: TerminalOutcomeQuarantined, + evidence: TerminalEvidence{ + Kind: TerminalEvidenceQuarantinedBeaconSigner, + }, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin(test.ceremony, cutover, test.identity) + if err != nil { + t.Fatal(err) + } + + // While the permit is live. + held := gate.State().ActivePermits + if len(held) != 1 { + t.Fatalf("expected one held permit, got [%d]", len(held)) + } + assertOperatedMembers( + t, + "the held permit", + held[0].OperatedMembers, + test.identity.OperatedMembers, + ) + + if err := permit.RecordTerminalOutcome( + test.outcome, + test.evidence, + ); err != nil { + t.Fatal(err) + } + permit.Close() + + // And after it ended with nothing to show for itself. + closed := gate.State().RecentTerminalOutcomes + if len(closed) != 1 { + t.Fatalf("expected one closed permit, got [%d]", len(closed)) + } + if closed[0].Outcome != test.outcome { + t.Fatalf("unexpected outcome [%s]", closed[0].Outcome) + } + assertOperatedMembers( + t, + "the closed permit", + closed[0].Permit.OperatedMembers, + test.identity.OperatedMembers, + ) + }) + } +} + +// TestPermit_UnresolvedEndingStillNamesItsSeats covers the ending no ceremony +// owner writes: a permit closed without any disposition at all. It is the +// ending a crashed or abandoned ceremony leaves behind, and it is exactly the +// case where a reader most needs to know which seats the holder was operating. +func TestPermit_UnresolvedEndingStillNamesItsSeats(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin( + BeaconRelaySigning, + cutover, + PermitIdentity{ + WorkID: BeaconRelayWorkID(cutover), + PermitID: "6", + OperatedMembers: MemberIndexes{6}, + }, + ) + if err != nil { + t.Fatal(err) + } + permit.Close() + + closed := gate.State().RecentTerminalOutcomes + if len(closed) != 1 { + t.Fatalf("expected one closed permit, got [%d]", len(closed)) + } + if closed[0].Outcome != terminalOutcomeUnresolved { + t.Fatalf( + "expected the fail-closed unresolved marker, got [%s]", + closed[0].Outcome, + ) + } + assertOperatedMembers( + t, + "the unresolved permit", + closed[0].Permit.OperatedMembers, + MemberIndexes{6}, + ) +} + +// TestPermit_TranscriptCannotClaimAnUnoperatedSeat holds the later account to +// the earlier one. Without this the operated set would be decorative: a holder +// could announce one seat and then record a result produced with another, and a +// reader would have two node-authored answers and no rule for choosing. +func TestPermit_TranscriptCannotClaimAnUnoperatedSeat(t *testing.T) { + const cutover = uint64(1_000) + + tests := map[string]struct { + ceremony Ceremony + identity PermitIdentity + evidence TerminalEvidence + }{ + "a signing transcript naming a seat the permit never operated": { + ceremony: TBTCSigning, + identity: PermitIdentity{ + WorkID: "wallet-action-overclaimed", + PermitID: "wallet", + OperatedMembers: MemberIndexes{3}, + }, + evidence: TerminalEvidence{ + Kind: TerminalEvidenceBitcoinTransaction, + Reference: "signed-transaction-hash", + Contribution: &TranscriptContribution{ + IncorporatedMembers: MemberIndexes{3, 5}, + LocalMembers: MemberIndexes{3, 5}, + }, + }, + }, + "a heartbeat transcript naming only an unoperated seat": { + ceremony: TBTCHeartbeat, + identity: PermitIdentity{ + WorkID: "heartbeat-overclaimed", + PermitID: "wallet", + OperatedMembers: MemberIndexes{3}, + }, + evidence: TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: "heartbeat-result-identity", + Contribution: &TranscriptContribution{ + IncorporatedMembers: MemberIndexes{5, 7}, + LocalMembers: MemberIndexes{5}, + }, + }, + }, + "a beacon DKG persisting a seat the permit never operated": { + ceremony: BeaconDKG, + identity: PermitIdentity{ + WorkID: strings.Repeat("c", 64), + PermitID: "4", + OperatedMembers: MemberIndexes{4}, + }, + evidence: TerminalEvidence{ + Kind: TerminalEvidencePersistedBeaconSigner, + Reference: "persisted-beacon-signer", + MembershipIndex: 5, + Contribution: &TranscriptContribution{ + IncorporatedMembers: MemberIndexes{4, 5}, + LocalMembers: MemberIndexes{5}, + }, + }, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin(test.ceremony, cutover, test.identity) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + err = permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + test.evidence, + ) + if !errors.Is(err, ErrInvalidTerminalOutcome) { + t.Fatalf( + "expected the record to be refused as invalid, got [%v]", + err, + ) + } + }) + } +} + +// TestPermit_TBTCDKGTranscriptIsHeldToItsPermitSeatThroughTheMapping asserts the +// one ceremony whose record is in a different index space than its permit is +// still bound to it, through the mapping its own transcript carries. +// +// A tBTC DKG permit is issued for a DKG member index while its transcript and +// persisted membership are in the final signing group's index space, rebuilt +// after the members this node did not see operating are removed — so a node +// legitimately runs seat 9 of the ceremony and lands in seat 8 of the group. +// Exempting the ceremony from the binding left the one case that remaps +// unchecked; comparing the raw numbers would refuse the correct record. The +// mapping is what makes the comparison mean something, so both the record that +// agrees with it and the record that does not are asserted here. +func TestPermit_TBTCDKGTranscriptIsHeldToItsPermitSeatThroughTheMapping( + t *testing.T, +) { + // Ten members were selected and member 2 was not seen operating, so the + // nine survivors are renumbered 1 through 9 and every seat above the removed + // one shifts down by one. + surviving := MemberIndexes{1, 3, 4, 5, 6, 7, 8, 9, 10} + finalSeats := MemberIndexes{1, 2, 3, 4, 5, 6, 7, 8, 9} + + // DKG seat 9 is the eighth survivor, so it holds final seat 8. + tests := map[string]struct { + operatedDKGSeat group.MemberIndex + accepted bool + }{ + "the seat the mapping traces the record back to": { + operatedDKGSeat: 9, + accepted: true, + }, + // The number the raw comparison would have matched. Reading the two + // spaces as one accepts this and attributes final seat 8 to whoever + // holds DKG seat 8 — which after the removal is a different node. + "the same number in the other index space": { + operatedDKGSeat: 8, + accepted: false, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin( + TBTCDKG, + cutover, + PermitIdentity{ + WorkID: strings.Repeat("d", 64), + PermitID: fmt.Sprint(test.operatedDKGSeat), + OperatedMembers: MemberIndexes{ + test.operatedDKGSeat, + }, + }, + ) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + err = permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: 8, + Contribution: &TranscriptContribution{ + IncorporatedMembers: finalSeats, + LocalMembers: MemberIndexes{8}, + PermitSpaceMembers: surviving, + }, + }, + ) + if test.accepted && err != nil { + t.Fatalf( + "a final seat the transcript traces back to this permit's "+ + "own DKG seat was refused: [%v]", + err, + ) + } + if !test.accepted { + if err == nil { + t.Fatal( + "a final seat produced by another node's DKG seat was " + + "accepted against this permit", + ) + } + if !errors.Is(err, ErrInvalidTerminalOutcome) { + t.Fatalf("unexpected error: [%v]", err) + } + } + }) + } +} + +// TestPermit_TBTCDKGTranscriptRequiresItsPermitSpaceMapping asserts the mapping +// is not optional for the ceremony that needs it. Without it there is nothing to +// hold the transcript to the permit, and a reader is back to comparing seat +// numbers from two different index spaces. +func TestPermit_TBTCDKGTranscriptRequiresItsPermitSpaceMapping(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + permit, err := gate.Begin( + TBTCDKG, + cutover, + PermitIdentity{ + WorkID: strings.Repeat("e", 64), + PermitID: "3", + OperatedMembers: MemberIndexes{3}, + }, + ) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + err = permit.RecordTerminalOutcome( + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: 3, + Contribution: &TranscriptContribution{ + IncorporatedMembers: MemberIndexes{1, 2, 3}, + LocalMembers: MemberIndexes{3}, + }, + }, + ) + if err == nil { + t.Fatal("a tBTC DKG transcript was recorded without its mapping") + } + if !errors.Is(err, ErrInvalidTerminalOutcome) { + t.Fatalf("unexpected error: [%v]", err) + } +} + +// TestValidatePermitOperatedOwnership_RejectsMalformedOperatedSets checks the +// generic set shape at the offline reader's entry point too. The journal is read +// outside the node that wrote it, so a record edited after the fact has to fail +// here rather than reach an ownership map as a seat counted twice. +func TestValidatePermitOperatedOwnership_RejectsMalformedOperatedSets( + t *testing.T, +) { + tests := map[string]MemberIndexes{ + "unordered": {4, 2}, + "repeated": {2, 2}, + "zero index": {0}, + } + + for name, operated := range tests { + t.Run(name, func(t *testing.T) { + if err := ValidatePermitOperatedOwnership( + TBTCSigning, + operated, + TerminalOutcomeExhausted, + TerminalEvidence{Kind: TerminalEvidenceNoThreshold}, + ); err == nil { + t.Fatal("a malformed operated set was accepted") + } + }) + } +} + +// TestPermit_OperatedSeatsAreNotAliased covers the two directions a slice can +// leak. The caller still holds the slice it passed to Begin, and every reader +// receives one from a snapshot, so neither may reach the permit's own record. +func TestPermit_OperatedSeatsAreNotAliased(t *testing.T) { + const cutover = uint64(1_000) + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + supplied := MemberIndexes{3, 9} + permit, err := gate.Begin( + TBTCSigning, + cutover, + PermitIdentity{ + WorkID: "wallet-action-aliasing", + PermitID: "wallet", + OperatedMembers: supplied, + }, + ) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + // The caller's own slice, rewritten after issuance. + supplied[0] = 200 + + held := gate.State().ActivePermits + if len(held) != 1 { + t.Fatalf("expected one held permit, got [%d]", len(held)) + } + assertOperatedMembers( + t, + "the held permit after the caller rewrote its slice", + held[0].OperatedMembers, + MemberIndexes{3, 9}, + ) + + // And a reader's copy, rewritten after the snapshot was taken. + held[0].OperatedMembers[1] = 201 + again := gate.State().ActivePermits + assertOperatedMembers( + t, + "the held permit after a reader rewrote its snapshot", + again[0].OperatedMembers, + MemberIndexes{3, 9}, + ) +} + +// TestPermit_ConcurrentReadersSeeTheirOwnOperatedSeats runs the snapshot path +// against a live gate under -race. A permit's operated set is read by every +// diagnostics scrape while other ceremonies are being issued and closed, and a +// snapshot handing out a window onto shared state would be a data race rather +// than a wrong answer. +func TestPermit_ConcurrentReadersSeeTheirOwnOperatedSeats(t *testing.T) { + const cutover = uint64(1_000) + const ceremonies = 16 + const readers = 8 + + gate, err := newGate( + context.Background(), + Schedule{CutoverBlock: cutover}, + newGateBlockCounter(cutover), + newFakeMetrics(), + inertPollInterval, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + start := make(chan struct{}) + stop := make(chan struct{}) + var writers, scrapers sync.WaitGroup + + for i := 0; i < ceremonies; i++ { + writers.Add(1) + go func(i int) { + defer writers.Done() + <-start + permit, err := gate.Begin( + TBTCSigning, + cutover, + PermitIdentity{ + WorkID: fmt.Sprintf("wallet-action-%d", i), + PermitID: fmt.Sprintf("wallet-%d", i), + OperatedMembers: MemberIndexes{group.MemberIndex(i + 1)}, + }, + ) + if err != nil { + t.Errorf("[%d] refused: [%v]", i, err) + return + } + permit.Close() + }(i) + } + + for i := 0; i < readers; i++ { + scrapers.Add(1) + go func() { + defer scrapers.Done() + <-start + for { + select { + case <-stop: + return + default: + } + snapshot := gate.State() + // Every reading is mutated in place, so an implementation + // handing out shared backing arrays races here rather than + // merely returning a stale value. + for _, held := range snapshot.ActivePermits { + for seat := range held.OperatedMembers { + held.OperatedMembers[seat] = 255 + } + } + for _, closed := range snapshot.RecentTerminalOutcomes { + for seat := range closed.Permit.OperatedMembers { + closed.Permit.OperatedMembers[seat] = 255 + } + } + } + }() + } + + close(start) + writers.Wait() + close(stop) + scrapers.Wait() + + // Nothing a reader did may have reached the gate's own account. + for _, closed := range gate.State().RecentTerminalOutcomes { + if len(closed.Permit.OperatedMembers) != 1 || + closed.Permit.OperatedMembers[0] == 255 { + t.Fatalf( + "a reader's rewrite reached the gate's account of [%s]: %v", + closed.Permit.PermitID, + closed.Permit.OperatedMembers, + ) + } + } +} + +func assertOperatedMembers( + t *testing.T, + subject string, + actual MemberIndexes, + expected MemberIndexes, +) { + t.Helper() + + if len(actual) != len(expected) { + t.Fatalf("%s names %v, expected %v", subject, actual, expected) + } + for i, seat := range expected { + if actual[i] != seat { + t.Fatalf("%s names %v, expected %v", subject, actual, expected) + } + } +} diff --git a/pkg/protocol/participation/quiescence.go b/pkg/protocol/participation/quiescence.go new file mode 100644 index 0000000000..fd46d55e69 --- /dev/null +++ b/pkg/protocol/participation/quiescence.go @@ -0,0 +1,1881 @@ +package participation + +import ( + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io/fs" + "math/big" + "slices" + "sort" + "strconv" + "strings" + "sync" + "time" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +const ( + // QuiescenceSnapshotSchemaVersion is the schema of the node-authored gate + // snapshot persisted at the quiescence transition. + // + // Version 2 added the memberships each live permit's holder operates. The + // snapshot is the issuance-time side of that statement and the journal is + // the terminal side, so an audit that reconciles the two needs the field in + // both: a version 1 snapshot carries no operated seats at all, leaving the + // journal's own account of them the only account, which is exactly the + // unreconciled single-sided claim the field was added to remove. + QuiescenceSnapshotSchemaVersion = uint32(2) + + // QuiescenceSnapshotStorageDirectory and + // QuiescenceSnapshotStorageFile identify the record inside the encrypted + // work/participation persistence namespace. The rollback audit reads this + // exact record from the stopped node's storage snapshot. + QuiescenceSnapshotStorageDirectory = "quiescence" + QuiescenceSnapshotStorageFile = "gate-snapshot.json" + + // TerminalOutcomeJournalSchemaVersion is the schema of the node-authored + // terminal-outcome journal. The journal is bound to the gate snapshot's + // capture time and covers that snapshot's permit inventory one-to-one. + // + // Version 3 added the chain-settlement record. An earlier journal cannot + // carry one, so it cannot distinguish a heartbeat that filed an inactivity + // claim from one that did not, and the audit must reject it outright + // rather than reconcile it under a rule it had no way to satisfy. + // + // Version 4 added the transcript contribution behind a completed result. + // The same reasoning applies with more force: a version 3 journal records + // that a threshold ceremony completed and cannot say which memberships + // produced it, so every reading of it has to fall back on treating a + // completion as a contribution — which is exactly the inference this field + // exists to remove. + // + // Version 5 added the memberships each permit's holder operates, recorded + // at issuance and carried by every permit whatever became of it. A version 4 + // journal names operated seats only inside the transcripts of permits that + // completed, so a reader asking which seats this node held on one piece of + // work has to answer from the subset of them that reached a result — and a + // seat whose permit ended exhausted, quarantined, or unresolved then reads + // as a seat the node never operated. The distinction matters most where the + // inference is least visible: a fleet-wide ownership map missing those seats + // attributes them to whoever else was on the network. + // + // Version 6 added the mapping from a transcript's seats back to the index + // space the permits for the same work were issued in. A version 5 journal + // records a tBTC DKG transcript in the final signing group's space beside + // permits in the ceremony's, with nothing to line the two up, so a reader + // joining them either compares raw numbers from different spaces or gives + // up on the join — and the first of those silently attributes a remapped + // seat to whichever party happens to hold that number in the other space. + TerminalOutcomeJournalSchemaVersion = uint32(6) + + // TerminalOutcomeJournalStorageFile identifies the terminal-outcome + // journal beside the immutable gate snapshot. Both records are encrypted + // by the participation work persistence handle. + TerminalOutcomeJournalStorageFile = "terminal-outcomes.json" +) + +// PermitIdentity binds one local permit to the chain-native work it performs +// and to the stable local membership or action identity that owns it. Neither +// component may contain raw seeds, messages, session IDs, keys, hostnames, or +// network addresses. +type PermitIdentity struct { + WorkID string `json:"work_id"` + PermitID string `json:"permit_id"` + // OperatedMembers are the ceremony memberships this node itself operates + // under this permit, ascending and distinct. It is empty for the permits + // that operate no seat at all — a forwarder relaying other members' shares, + // a timeout monitor filing a penalty. + // + // It is supplied at issuance, before any outcome exists, and that is the + // whole reason it lives here rather than only in a terminal record's + // transcript. A reader asking who operated a seat on one piece of work is + // building an ownership map, and an ownership map assembled from completions + // is incomplete by construction: a node that contributed and then crashed, + // timed out, or ended without a threshold result operated its seats just as + // much as one that finished, and records nothing about them. Reading the + // seats off the permit instead covers every node that was allowed to take + // part, whatever became of it — which is what makes "no node in this fleet + // operated that seat" a statement about the fleet rather than about the + // subset of it that happened to complete. + // + // A node can only ever name its own seats here. The gate refuses a + // transcript whose local memberships are not among the permit's operated + // ones, so the two node-authored accounts of the same permit cannot + // disagree, and no report from another party can add a seat to either. + OperatedMembers MemberIndexes `json:"operated_members,omitempty"` +} + +const ( + maxPermitIdentityComponentLength = 256 + // maxDecimalUint64Length is the width of the widest decimal uint64. + maxDecimalUint64Length = len("18446744073709551615") + // maxTerminalEvidenceReferenceLength bounds a terminal evidence reference. + // It is wider than a permit identity component because one evidence class + // is verifiable rather than merely nameable: a beacon relay entry carries + // the group public key, the previous entry and the entry as public curve + // points precisely so the offline audit can check the signature instead of + // taking the node's word for the result, alongside the decimal start block + // of the request it answers so that proof is bound to one request. Those + // four components with their separators is the widest reference any + // ceremony produces. + maxTerminalEvidenceReferenceLength = maxDecimalUint64Length + + 3*2*beaconRelayEntryComponentLength + 3 +) + +// validateNonsecretToken applies the shared shape of every identity and +// reference the journal carries: nonempty, bounded, and drawn from an alphabet +// that cannot smuggle raw seeds, keys, hostnames, or network addresses past a +// reader. +func validateNonsecretToken(name string, value string, maxLength int) error { + if value == "" { + return fmt.Errorf("%s is empty", name) + } + if len(value) > maxLength { + return fmt.Errorf("%s exceeds [%d] bytes", name, maxLength) + } + for i, character := range []byte(value) { + if (character >= 'a' && character <= 'z') || + (character >= 'A' && character <= 'Z') || + (character >= '0' && character <= '9') || + (i > 0 && (character == '_' || + character == '.' || + character == ':' || + character == '-')) { + continue + } + return fmt.Errorf( + "%s contains an unsupported character at byte [%d]", + name, + i, + ) + } + + return nil +} + +func validatePermitIdentity(identity PermitIdentity) error { + for _, component := range []struct { + name string + value string + }{ + {name: "work ID", value: identity.WorkID}, + {name: "permit ID", value: identity.PermitID}, + } { + if err := validateNonsecretToken( + component.name, + component.value, + maxPermitIdentityComponentLength, + ); err != nil { + return err + } + } + + return nil +} + +// validatePermitIdentityForCeremony applies the identity shape that is +// intrinsic to a ceremony in addition to the generic nonsecret token rules. +// DKG work is always keyed by the lowercase SHA-256 hash of its seed, and +// member-owned DKG/relay permits always use the canonical decimal member +// index. Enforcing these shapes at issuance prevents an unauditable identity +// from entering the authoritative quiescence inventory. +func validatePermitIdentityForCeremony( + ceremony Ceremony, + identity PermitIdentity, +) error { + if err := validatePermitIdentity(identity); err != nil { + return err + } + + switch ceremony { + case TBTCDKG, BeaconDKG: + if len(identity.WorkID) != 64 || + identity.WorkID != strings.ToLower(identity.WorkID) { + return fmt.Errorf( + "work ID must be a lowercase SHA-256 hash of 64 hexadecimal characters", + ) + } + if _, err := hex.DecodeString(identity.WorkID); err != nil { + return fmt.Errorf( + "work ID must be a lowercase SHA-256 hash of 64 hexadecimal characters", + ) + } + } + + switch ceremony { + case TBTCDKG, BeaconDKG, BeaconRelaySigning: + memberIndex, err := strconv.ParseUint(identity.PermitID, 10, 8) + if err != nil || + memberIndex == 0 || + memberIndex > uint64(group.MaxMemberIndex) || + strconv.FormatUint(memberIndex, 10) != identity.PermitID { + return fmt.Errorf( + "permit ID must be a canonical member index from 1 through %d", + group.MaxMemberIndex, + ) + } + } + + return ValidatePermitOperatedShape( + ceremony, + identity.PermitID, + identity.OperatedMembers, + ) +} + +// ValidatePermitOperatedShape checks that a permit's operated seats are the +// shape its ceremony can have, given the permit they belong to. +// +// It is exported because the operated set is what a fleet-wide seat ownership +// map is built from, and the offline audit reads that set out of a stopped +// node's own records rather than watching it being issued. Nothing else in a +// snapshot constrains it: a record whose operated seats were widened after the +// fact still names a real ceremony, a real permit, and a real outcome, and would +// enter an ownership map claiming seats no permit was ever issued to run. +func ValidatePermitOperatedShape( + ceremony Ceremony, + permitID string, + operatedMembers MemberIndexes, +) error { + if err := validateMemberIndexSet( + "operated memberships", + operatedMembers, + ); err != nil { + return err + } + + switch ceremony { + case TBTCDKG, BeaconDKG, BeaconRelaySigning: + // The permit already names the one seat it was issued for, so the + // operated set is not free to say anything else. Two node-authored + // statements about the same permit that disagree would leave a reader + // choosing between them, and either choice is an inference. + if len(operatedMembers) != 1 || + fmt.Sprint(operatedMembers[0]) != permitID { + return fmt.Errorf( + "ceremony [%s] runs one seat per permit, so its operated "+ + "memberships must be exactly its permit ID [%s]", + ceremony, + permitID, + ) + } + case BeaconRelayForwarding, BeaconTimeoutReport: + // Neither operates a seat: a forwarder relays other members' shares + // and computes nothing, and a timeout report is one node's penalty + // filing. A seat claimed here would enter the fleet's ownership map as + // operated by this node without any membership behind it, which is the + // self-attestation the map exists to keep out. + if len(operatedMembers) != 0 { + return fmt.Errorf( + "ceremony [%s] operates no membership, so it must claim none", + ceremony, + ) + } + } + + return nil +} + +// PermitSnapshot is the immutable, nonsecret identity and cutover +// classification of one live permit. +type PermitSnapshot struct { + Ceremony Ceremony `json:"ceremony"` + Mode string `json:"mode"` + CanonicalStartBlock uint64 `json:"canonical_start_block"` + WorkID string `json:"work_id"` + PermitID string `json:"permit_id"` + // IdentityBound is true only when the ceremony owner supplied the + // chain-work and local-permit identity at issuance. The source-compatible + // unbound path exists only for test gates without a persistence recorder; + // production issuance and the rollback audit both refuse it. + IdentityBound bool `json:"identity_bound"` + // OperatedMembers are the memberships this node operates under the permit, + // as its owner named them at issuance. It travels with every reading of the + // permit — live, quiesced, and terminal, whatever the outcome — because a + // reader building a per-work seat ownership map needs the seats of the + // permits that came to nothing just as much as the ones that completed. + OperatedMembers MemberIndexes `json:"operated_members,omitempty"` +} + +// Equal reports whether two permit snapshots describe the same permit. It +// exists because the snapshot carries a slice, so == neither compiles nor +// compares a snapshot with the same snapshot reloaded from the journal. +func (p PermitSnapshot) Equal(other PermitSnapshot) bool { + return p.Ceremony == other.Ceremony && + p.Mode == other.Mode && + p.CanonicalStartBlock == other.CanonicalStartBlock && + p.WorkID == other.WorkID && + p.PermitID == other.PermitID && + p.IdentityBound == other.IdentityBound && + slices.Equal(p.OperatedMembers, other.OperatedMembers) +} + +// QuiescenceSnapshot is the node-authored, immutable record captured under +// the gate lock at the instant the first Quiesce call makes the transition. +// It binds the real active-permit registry to the exact artifact and one-value +// cutover schedule that were running. The node-authored terminal-outcome +// journal must cover this inventory one-to-one; external reconciliation can +// only corroborate it. +type QuiescenceSnapshot struct { + SchemaVersion uint32 `json:"schema_version"` + CapturedAt time.Time `json:"captured_at"` + + ReleaseVersion string `json:"release_version"` + ReleaseRevision string `json:"release_revision"` + ReleaseEpoch string `json:"release_epoch"` + + CutoverBlock uint64 `json:"cutover_block"` + CurrentBlock uint64 `json:"current_block"` + ClockAvailable bool `json:"clock_available"` + State string `json:"state"` + QuiesceCause string `json:"quiesce_cause"` + + ActiveCeremonies uint64 `json:"active_ceremonies"` + ActiveLegacyCeremonies uint64 `json:"active_legacy_ceremonies"` + ActiveSecurityV2Ceremonies uint64 `json:"active_security_v2_ceremonies"` + ActivePermits []PermitSnapshot `json:"active_permits"` +} + +// TerminalOutcome is the final disposition of a permit that was active at the +// quiescence transition. +type TerminalOutcome string + +const ( + // TerminalOutcomeCompleted means the ceremony owner reached a durable + // successful result and recorded evidence identifying that result. + TerminalOutcomeCompleted TerminalOutcome = "completed" + // TerminalOutcomeQuarantined means generated key material was durably + // preserved in the protected quarantine namespace. + TerminalOutcomeQuarantined TerminalOutcome = "quarantined" + // TerminalOutcomeExhausted means the ceremony ended without producing a + // threshold result or durable state transition. + TerminalOutcomeExhausted TerminalOutcome = "exhausted" + // terminalOutcomeUnresolved is written by permit Close when its ceremony + // owner did not record any terminal disposition. It is intentionally not + // accepted by RecordTerminalOutcome and always blocks the offline barrier. + terminalOutcomeUnresolved TerminalOutcome = "unresolved" +) + +// TerminalEvidenceKind identifies the durable state or explicit terminal +// condition behind an outcome. References contain only stable public +// identities or digests; private material and raw protocol inputs are +// forbidden. +type TerminalEvidenceKind string + +const ( + TerminalEvidencePersistedTBTCSinger TerminalEvidenceKind = "persisted_tbtc_signer" + TerminalEvidencePersistedBeaconSigner TerminalEvidenceKind = "persisted_beacon_signer" + TerminalEvidenceQuarantinedTBTCSinger TerminalEvidenceKind = "quarantined_tbtc_signer" + TerminalEvidenceQuarantinedBeaconSigner TerminalEvidenceKind = "quarantined_beacon_signer" + TerminalEvidenceBitcoinTransaction TerminalEvidenceKind = "bitcoin_transaction" + TerminalEvidenceEthereumTransaction TerminalEvidenceKind = "ethereum_transaction" + TerminalEvidenceProtocolResult TerminalEvidenceKind = "protocol_result" + TerminalEvidenceNoThreshold TerminalEvidenceKind = "no_threshold" + TerminalEvidenceForwarderClosed TerminalEvidenceKind = "forwarder_closed" +) + +// TerminalEvidence binds a terminal outcome to the state it left behind. +// Reference is required for persisted signers, transactions, and protocol +// result digests. Explicit no-threshold and forwarder-close outcomes carry no +// reference. +type TerminalEvidence struct { + Kind TerminalEvidenceKind `json:"kind"` + Reference string `json:"reference,omitempty"` + // MembershipIndex identifies the exact persisted membership produced by + // a completed DKG permit. For tBTC this is the final wallet signing-group + // index, which may differ from the original DKG permit index after + // inactive or disqualified members are removed. For beacon it is the + // persisted threshold signer's member index. + MembershipIndex group.MemberIndex `json:"membership_index,omitempty"` + // ChainSettlement reports a chain side effect the ceremony dispatched + // beyond its own protocol result. It is nil when the ceremony dispatched + // none. + ChainSettlement *ChainSettlementRecord `json:"chain_settlement,omitempty"` + // Contribution is the node's own account of which memberships combined + // into the result. It is nil for the endings that produced no threshold + // result and for ceremonies whose owners do not author one. + Contribution *TranscriptContribution `json:"contribution,omitempty"` +} + +// TranscriptContribution is the ceremony owner's own account of which group +// memberships combined into the result behind a completed outcome. +// +// It exists because a completion is not evidence of who produced it. Every +// member of a threshold ceremony records "completed" and names the same result +// identity whatever population actually reached it, so a reader holding only +// completions cannot distinguish a run whose shares came from several parties +// from one party that recovered or persisted the common result on its own. That +// distinction is the whole claim of a mixed-release ceremony, and nothing else +// in a terminal record carries it. +// +// The memberships named here are the ones whose protocol contributions this +// node authenticated against the ceremony's on-chain membership and combined +// into the result it recorded. It is the owner's local view of the transcript +// rather than a report about the transcript: no other party can write it, and a +// node cannot author a membership whose authenticated messages it never +// accepted. +type TranscriptContribution struct { + // IncorporatedMembers are the ceremony member indexes whose authenticated + // contributions were combined into the recorded result, ascending and + // distinct. The node's own memberships are among them: a member + // contributes to its own result. + IncorporatedMembers MemberIndexes `json:"incorporated_members"` + // LocalMembers are the incorporated memberships this node itself operated, + // ascending and distinct. Subtracting the fleet's local memberships from + // the incorporated set leaves the memberships some other node had to + // supply, which is the only part of a transcript a node's own record can + // attribute elsewhere — and so the only way a reader can tell a shared + // ceremony from a solitary one without taking some third party's word for + // the population. + // + // It is empty when none of this node's memberships were in the transcript. + // A wallet action owns its permit and records the signature it observed + // even when the attempt that produced it selected none of the memberships + // this node operates; naming none is the honest answer there, and refusing + // to record the result would leave the permit unresolved over work that + // demonstrably concluded. + LocalMembers MemberIndexes `json:"local_members"` + // PermitSpaceMembers is the ceremony membership behind each entry of + // IncorporatedMembers, in the index space the permits for this work were + // issued in, positionally aligned with IncorporatedMembers and therefore + // exactly as long. + // + // It is present only for the ceremonies whose result is recorded in a + // different index space than their permits — see + // permitSpaceMappingCeremonies — and absent everywhere else, where the two + // spaces are the same and a mapping would be a second answer to a question + // that already has one. + // + // It exists because a transcript is useless to a reader that cannot say + // which node sat in the seats it names. tBTC DKG rebuilds its group after + // removing the members it did not see operating, so a node runs seat 9 of + // the ceremony and lands in seat 7 of the group, and the permits — issued + // before the result exists, hence in the ceremony's space — cannot be held + // against the transcript without this. Reading the two spaces as one is + // worse than having neither: with a middle member removed, every final seat + // shifts down and a reader comparing the raw numbers attributes seats to + // parties that never held them. + // + // The mapping is the accepted result's own: the ascending members this node + // authenticated through every round, which is what the final group was + // built from. It is checked against the author's own permit at record time, + // so a node cannot use it to move its own seat. + PermitSpaceMembers MemberIndexes `json:"permit_space_members,omitempty"` +} + +// MemberIndexes is a set of ceremony member indexes in a record that leaves the +// node. +// +// It exists for its JSON encoding. A member index is a byte, so the default +// encoding of a slice of them is base64 — an offline audit or a diagnostics +// scrape would receive a transcript's membership as an opaque string, and the +// one field that says who produced a result would be the one field a reader +// cannot join to anything. The encoding here is a list of numbers, which is +// also what the reader would write down by hand. +type MemberIndexes []group.MemberIndex + +// MarshalJSON renders the indexes as a JSON array of numbers. +func (m MemberIndexes) MarshalJSON() ([]byte, error) { + // A wider element type is what takes the encoding off the byte-slice path; + // the range is checked back on the way in. + numbers := make([]uint16, len(m)) + for i, index := range m { + numbers[i] = uint16(index) + } + + return json.Marshal(numbers) +} + +// UnmarshalJSON reads a JSON array of numbers back, refusing any value that is +// not a valid member index. The journal is read outside the node that wrote it, +// so a corrupted or forged record must fail here rather than become a truncated +// index further along. +func (m *MemberIndexes) UnmarshalJSON(data []byte) error { + var numbers []uint16 + if err := json.Unmarshal(data, &numbers); err != nil { + return err + } + + indexes := make(MemberIndexes, len(numbers)) + for i, number := range numbers { + if number == 0 || number > uint16(group.MaxMemberIndex) { + return fmt.Errorf( + "[%d] is not a member index from 1 through %d", + number, + group.MaxMemberIndex, + ) + } + indexes[i] = group.MemberIndex(number) + } + *m = indexes + + return nil +} + +// Equal reports whether two contributions describe the same transcript. Like +// TerminalEvidence.Equal it exists because the record travels by pointer and +// carries slices, so == neither compiles for the fields nor compares an +// observation with the same observation reloaded from the journal. +func (c *TranscriptContribution) Equal(other *TranscriptContribution) bool { + if c == nil || other == nil { + return c == other + } + + return slices.Equal(c.IncorporatedMembers, other.IncorporatedMembers) && + slices.Equal(c.LocalMembers, other.LocalMembers) && + slices.Equal(c.PermitSpaceMembers, other.PermitSpaceMembers) +} + +// Equal reports whether two evidence records describe the same durable result. +// It exists because TerminalEvidence carries a pointer: Go's == would compare +// the settlement by address, so two records built separately from the same +// observation — a retried write, a record reloaded from the journal — would +// read as different results. Callers must use this instead of ==. +func (e TerminalEvidence) Equal(other TerminalEvidence) bool { + if e.Kind != other.Kind || + e.Reference != other.Reference || + e.MembershipIndex != other.MembershipIndex { + return false + } + if !e.Contribution.Equal(other.Contribution) { + return false + } + if e.ChainSettlement == nil || other.ChainSettlement == nil { + return e.ChainSettlement == other.ChainSettlement + } + + return *e.ChainSettlement == *other.ChainSettlement +} + +// ChainSettlementKind identifies a chain side effect a ceremony dispatches +// outside its own protocol result. +type ChainSettlementKind string + +const ( + // ChainSettlementInactivityClaim is the operator inactivity claim a + // low-activity tBTC heartbeat files against the WalletRegistry. + ChainSettlementInactivityClaim ChainSettlementKind = "tbtc_inactivity_claim" +) + +// ChainSettlementRecord reports a chain side effect the ceremony handed to a +// chain and the canonical chain state it was resolved to settle into. +// +// Submitting is not settling. The submitting call returns as soon as the +// transaction reaches the provider and the transaction is mined afterwards, it +// can lose the race to another member's submission, and it can be canceled +// mid-flight; in none of those cases does the node know from the call alone +// whether the side effect reached the chain. A submission whose settlement the +// ceremony could not resolve is therefore recorded with an empty Reference: the +// attempt is on the record, its outcome is unknown, and the offline barrier +// must treat it as unreconciled instead of letting a node-authored digest close +// the journal over a penalty that may or may not exist on chain. +// +// The record is absent entirely when the ceremony never reached the submitting +// call. A suppressed, refused, or aborted dispatch left no transaction +// anywhere, and reporting it as an unresolved submission would block a rollback +// over chain state that provably cannot exist. +type ChainSettlementRecord struct { + Kind ChainSettlementKind `json:"kind"` + // Reference is the canonical chain identity of the resolved settlement. + // It is empty when the submission's settlement could not be resolved. + Reference string `json:"reference,omitempty"` +} + +// inactivityClaimWalletIDLength is the byte length of the WalletRegistry +// wallet identifier an inactivity claim names. +const inactivityClaimWalletIDLength = 32 + +// InactivityClaimSettlementReference renders the canonical identity of an +// on-chain tBTC inactivity claim: the wallet it was filed against and the +// claim nonce it settled at. The pair identifies exactly one claim for all +// time, because the WalletRegistry accepts a claim only at the wallet's +// current nonce and increments that nonce in the same call. The offline audit +// can therefore join the reference to exactly one authenticated +// InactivityClaimed log, which is what keeps the node from naming a settlement +// that never happened. +func InactivityClaimSettlementReference( + walletID []byte, + nonce *big.Int, +) (string, error) { + if len(walletID) != inactivityClaimWalletIDLength { + return "", fmt.Errorf( + "inactivity claim wallet identifier must be %d bytes, got [%d]", + inactivityClaimWalletIDLength, + len(walletID), + ) + } + if nonce == nil { + return "", fmt.Errorf("inactivity claim nonce is missing") + } + if nonce.Sign() < 0 { + return "", fmt.Errorf( + "inactivity claim nonce [%s] is negative", + nonce, + ) + } + + return hex.EncodeToString(walletID) + ":" + nonce.String(), nil +} + +// ParseInactivityClaimSettlementReference recovers the wallet identifier and +// claim nonce from a reference produced by +// InactivityClaimSettlementReference. Only the exact rendering that function +// produces is accepted: an uppercase, prefixed, or zero-padded alias would +// name the same claim while failing every string comparison the audit makes +// against it, which is indistinguishable from naming no claim at all. +func ParseInactivityClaimSettlementReference( + reference string, +) ([]byte, *big.Int, error) { + walletIDText, nonceText, separated := strings.Cut(reference, ":") + if !separated { + return nil, nil, fmt.Errorf( + "inactivity claim reference [%s] is not a wallet identifier and "+ + "nonce pair", + reference, + ) + } + + walletID, err := hex.DecodeString(walletIDText) + if err != nil { + return nil, nil, fmt.Errorf( + "inactivity claim reference wallet identifier [%s] is not hex: "+ + "[%v]", + walletIDText, + err, + ) + } + if len(walletID) != inactivityClaimWalletIDLength { + return nil, nil, fmt.Errorf( + "inactivity claim reference wallet identifier [%s] is not %d "+ + "bytes", + walletIDText, + inactivityClaimWalletIDLength, + ) + } + if hex.EncodeToString(walletID) != walletIDText { + return nil, nil, fmt.Errorf( + "inactivity claim reference wallet identifier [%s] is not "+ + "canonically encoded", + walletIDText, + ) + } + + nonce, valid := new(big.Int).SetString(nonceText, 10) + if !valid || nonce.Sign() < 0 { + return nil, nil, fmt.Errorf( + "inactivity claim reference nonce [%s] is not a non-negative "+ + "decimal integer", + nonceText, + ) + } + if nonce.String() != nonceText { + return nil, nil, fmt.Errorf( + "inactivity claim reference nonce [%s] is not canonically encoded", + nonceText, + ) + } + + return walletID, nonce, nil +} + +// beaconRelayEntryComponentLength is the byte length of each component of a +// relay entry identity. A compressed bn256 group public key, a marshaled +// previous entry, and a marshaled recovered entry are all 64 bytes. +const beaconRelayEntryComponentLength = 64 + +// beaconRelayWorkIDPrefix labels the work every beacon relay permit for one +// on-chain relay request shares. The request's start block completes it, so +// every membership, the forwarder and the timeout monitor of a single request +// name the same work while distinct requests never can. +const beaconRelayWorkIDPrefix = "relay-request-" + +// BeaconRelayWorkID renders the work identity of every permit issued for the +// relay request that started at the given block. +func BeaconRelayWorkID(relayRequestStartBlock uint64) string { + return beaconRelayWorkIDPrefix + + strconv.FormatUint(relayRequestStartBlock, 10) +} + +// ParseBeaconRelayWorkID recovers the relay request start block from a work +// identity produced by BeaconRelayWorkID. Only that exact rendering is +// accepted, so a zero-padded or signed alias cannot name a request the audit +// would then compare against a differently rendered one and read as a +// different request. +func ParseBeaconRelayWorkID(workID string) (uint64, error) { + blockText, found := strings.CutPrefix(workID, beaconRelayWorkIDPrefix) + if !found { + return 0, fmt.Errorf( + "beacon relay work identity [%s] does not name a relay request", + workID, + ) + } + + startBlock, err := strconv.ParseUint(blockText, 10, 64) + if err != nil { + return 0, fmt.Errorf( + "beacon relay work identity [%s] does not name a relay request "+ + "start block: [%v]", + workID, + err, + ) + } + if strconv.FormatUint(startBlock, 10) != blockText { + return 0, fmt.Errorf( + "beacon relay work identity [%s] start block is not canonically "+ + "encoded", + workID, + ) + } + + return startBlock, nil +} + +// BeaconRelayTimeoutSettlementReference renders the canonical identity of the +// RandomBeacon's own record that a relay request was terminated by an accepted +// timeout report: the request start block the permit was issued for, the +// beacon's request identifier, and the group the beacon terminated. +// +// A filed report is not a penalty. The submitting call returns once the +// transaction reaches a provider, and a transaction that reverts, is dropped, +// or loses the race to another reporter leaves the beacon exactly as it was. So +// the identity names the beacon's record rather than the node's submission: the +// request identifier and terminated group are the two fields of a +// RelayEntryTimedOut log, which the beacon emits at most once per request, so +// the pair joins the reference to exactly one authenticated log. A node that +// never earned the penalty cannot render this reference at all, because no such +// log exists to read it from. +// +// The request start block leads the identity for the same reason it leads a +// relay entry's: a request identifier alone does not say which permit the +// settlement belongs to, and without that component a genuine settlement from +// another request could stand in as this permit's result. +func BeaconRelayTimeoutSettlementReference( + relayRequestStartBlock uint64, + requestID *big.Int, + terminatedGroupID uint64, +) (string, error) { + if requestID == nil { + return "", fmt.Errorf("relay entry timeout request identifier is missing") + } + if requestID.Sign() < 0 { + return "", fmt.Errorf( + "relay entry timeout request identifier [%s] is negative", + requestID, + ) + } + + return strconv.FormatUint(relayRequestStartBlock, 10) + ":" + + requestID.String() + ":" + + strconv.FormatUint(terminatedGroupID, 10), nil +} + +// ParseBeaconRelayTimeoutSettlementReference recovers the relay request start +// block, the beacon's request identifier and the terminated group from a +// reference produced by BeaconRelayTimeoutSettlementReference. Only that exact +// rendering is accepted: a prefixed or zero-padded alias would name the same +// settlement while failing every comparison the audit makes against the request +// its permit names and against the authenticated log it joins to, which is +// indistinguishable from naming no settlement at all. +func ParseBeaconRelayTimeoutSettlementReference(reference string) ( + relayRequestStartBlock uint64, + requestID *big.Int, + terminatedGroupID uint64, + err error, +) { + parts := strings.Split(reference, ":") + if len(parts) != 3 { + return 0, nil, 0, fmt.Errorf( + "relay entry timeout settlement reference [%s] is not a request "+ + "start block, request identifier and terminated group triple", + reference, + ) + } + + startBlock, err := parseCanonicalUint64( + "relay entry timeout settlement reference request start block", + parts[0], + ) + if err != nil { + return 0, nil, 0, err + } + + id, valid := new(big.Int).SetString(parts[1], 10) + if !valid || id.Sign() < 0 { + return 0, nil, 0, fmt.Errorf( + "relay entry timeout settlement reference request identifier [%s] "+ + "is not a non-negative decimal integer", + parts[1], + ) + } + if id.String() != parts[1] { + return 0, nil, 0, fmt.Errorf( + "relay entry timeout settlement reference request identifier [%s] "+ + "is not canonically encoded", + parts[1], + ) + } + + groupID, err := parseCanonicalUint64( + "relay entry timeout settlement reference terminated group", + parts[2], + ) + if err != nil { + return 0, nil, 0, err + } + + return startBlock, id, groupID, nil +} + +// parseCanonicalUint64 decodes an unsigned decimal component of a terminal +// evidence reference, rejecting any rendering the reference builders do not +// produce. A padded or signed alias names the same number while comparing +// unequal to every reference the audit rebuilds, so it is refused rather than +// normalized. +func parseCanonicalUint64(name string, text string) (uint64, error) { + value, err := strconv.ParseUint(text, 10, 64) + if err != nil { + return 0, fmt.Errorf( + "%s [%s] is not an unsigned decimal integer: [%v]", + name, + text, + err, + ) + } + if strconv.FormatUint(value, 10) != text { + return 0, fmt.Errorf("%s [%s] is not canonically encoded", name, text) + } + + return value, nil +} + +// BeaconRelayEntryReference renders the canonical identity of a recovered +// relay entry: the relay request it answers, the group that signed it, the +// previous entry it signed over, and the entry itself. +// +// Unlike every other protocol result, this identity is not a digest. A relay +// entry is a threshold BLS signature by the group over the previous entry, and +// every component is public beacon state that the chain publishes anyway. +// Carrying them in the clear is what lets the offline audit verify the pairing +// itself: an entry that verifies under a group public key the snapshot holds +// cannot have been authored by anything but that group's threshold key, so the +// node's word is not what makes the record true. A digest would prove only that +// the node was consistent with itself. +// +// The request start block leads the identity because the signature alone says +// which group produced an entry, not which request that entry answers. Naming +// the request inside the reference is what lets the audit hold the record to +// the permit that authorized it and to one request only: a genuine historical +// entry lifted onto an unrelated permit still verifies as a signature, and +// without this component nothing about the record contradicts it. +func BeaconRelayEntryReference( + relayRequestStartBlock uint64, + groupPublicKey []byte, + previousEntry []byte, + entry []byte, +) (string, error) { + for _, component := range []struct { + name string + value []byte + }{ + {name: "group public key", value: groupPublicKey}, + {name: "previous entry", value: previousEntry}, + {name: "entry", value: entry}, + } { + if len(component.value) != beaconRelayEntryComponentLength { + return "", fmt.Errorf( + "relay entry %s must be %d bytes, got [%d]", + component.name, + beaconRelayEntryComponentLength, + len(component.value), + ) + } + } + + return strconv.FormatUint(relayRequestStartBlock, 10) + ":" + + hex.EncodeToString(groupPublicKey) + ":" + + hex.EncodeToString(previousEntry) + ":" + + hex.EncodeToString(entry), nil +} + +// ParseBeaconRelayEntryReference recovers the relay request start block, group +// public key, previous entry and recovered entry from a reference produced by +// BeaconRelayEntryReference. Only that exact rendering is accepted: an +// uppercase, prefixed or zero-padded alias would name the same entry while +// failing every comparison the audit makes against the group identities it +// decoded and the request its permit names, which is indistinguishable from +// naming no group and no request at all. +func ParseBeaconRelayEntryReference(reference string) ( + relayRequestStartBlock uint64, + groupPublicKey []byte, + previousEntry []byte, + entry []byte, + err error, +) { + parts := strings.Split(reference, ":") + if len(parts) != 4 { + return 0, nil, nil, nil, fmt.Errorf( + "relay entry reference [%s] is not a request start block, group, "+ + "previous entry and entry quadruple", + reference, + ) + } + + startBlock, err := strconv.ParseUint(parts[0], 10, 64) + if err != nil { + return 0, nil, nil, nil, fmt.Errorf( + "relay entry reference request start block [%s] is not an "+ + "unsigned decimal integer: [%v]", + parts[0], + err, + ) + } + if strconv.FormatUint(startBlock, 10) != parts[0] { + return 0, nil, nil, nil, fmt.Errorf( + "relay entry reference request start block [%s] is not "+ + "canonically encoded", + parts[0], + ) + } + + decoded := make([][]byte, 0, 3) + for i, name := range []string{ + "group public key", + "previous entry", + "entry", + } { + part := parts[i+1] + + value, err := hex.DecodeString(part) + if err != nil { + return 0, nil, nil, nil, fmt.Errorf( + "relay entry reference %s [%s] is not hex: [%v]", + name, + part, + err, + ) + } + if len(value) != beaconRelayEntryComponentLength { + return 0, nil, nil, nil, fmt.Errorf( + "relay entry reference %s [%s] is not %d bytes", + name, + part, + beaconRelayEntryComponentLength, + ) + } + if hex.EncodeToString(value) != part { + return 0, nil, nil, nil, fmt.Errorf( + "relay entry reference %s [%s] is not canonically encoded", + name, + part, + ) + } + decoded = append(decoded, value) + } + + return startBlock, decoded[0], decoded[1], decoded[2], nil +} + +// TerminalResultReference derives the nonsecret, stable identity of a protocol +// result for a terminal evidence record. Components are length-prefixed under a +// domain label, so no two ceremonies can derive the same digest from different +// inputs and no component boundary can be shifted to forge a collision. Only +// the digest — never the underlying material — reaches the journal, which keeps +// raw protocol inputs out of a record the rollback audit reads outside the +// node's trust boundary. +func TerminalResultReference(domain string, components ...[]byte) string { + digest := sha256.New() + + writeComponent := func(component []byte) { + var length [8]byte + binary.BigEndian.PutUint64(length[:], uint64(len(component))) + digest.Write(length[:]) + digest.Write(component) + } + + writeComponent([]byte(domain)) + for _, component := range components { + writeComponent(component) + } + + return hex.EncodeToString(digest.Sum(nil)) +} + +// TerminalOutcomeRecord is written by the permit owner after real completion +// or quarantine handling. The embedded permit identity is copied from the gate +// and cannot be supplied or changed by an external report generator. +type TerminalOutcomeRecord struct { + RecordedAt time.Time `json:"recorded_at"` + Permit PermitSnapshot `json:"permit"` + Outcome TerminalOutcome `json:"outcome"` + Evidence TerminalEvidence `json:"evidence"` +} + +// Equal reports whether two records report the same disposition of the same +// permit. Like TerminalEvidence.Equal it exists because the embedded evidence +// carries a pointer, so == would separate a record from an identical one +// rebuilt or reloaded elsewhere. Callers must use this instead of ==. +func (r TerminalOutcomeRecord) Equal(other TerminalOutcomeRecord) bool { + return r.RecordedAt.Equal(other.RecordedAt) && + r.Permit.Equal(other.Permit) && + r.Outcome == other.Outcome && + r.Evidence.Equal(other.Evidence) +} + +// TerminalOutcomeJournal is the node-authored terminal record for the exact +// permit population captured at one quiescence transition. +type TerminalOutcomeJournal struct { + SchemaVersion uint32 `json:"schema_version"` + SnapshotCapturedAt time.Time `json:"snapshot_captured_at"` + Outcomes []TerminalOutcomeRecord `json:"outcomes"` +} + +// ValidateTerminalOutcome checks that an outcome and its evidence have a +// supported shape for the owning ceremony, and that evidence naming the work +// it settles names the work the permit was issued for. The live gate and the +// offline state audit use this same validator so corrupted journal data cannot +// exploit schema drift between issuance and reconciliation. +func ValidateTerminalOutcome( + ceremony Ceremony, + workID string, + outcome TerminalOutcome, + evidence TerminalEvidence, +) error { + if outcome != TerminalOutcomeCompleted && + outcome != TerminalOutcomeQuarantined && + outcome != TerminalOutcomeExhausted { + return fmt.Errorf("unsupported terminal outcome [%s]", outcome) + } + + referenceRequired := false + switch evidence.Kind { + case TerminalEvidencePersistedTBTCSinger, + TerminalEvidencePersistedBeaconSigner, + TerminalEvidenceBitcoinTransaction, + TerminalEvidenceEthereumTransaction, + TerminalEvidenceProtocolResult: + referenceRequired = true + case TerminalEvidenceQuarantinedTBTCSinger, + TerminalEvidenceQuarantinedBeaconSigner, + TerminalEvidenceNoThreshold, + TerminalEvidenceForwarderClosed: + default: + return fmt.Errorf( + "unsupported terminal evidence kind [%s]", + evidence.Kind, + ) + } + + if referenceRequired { + if err := validateNonsecretToken( + "terminal evidence reference", + evidence.Reference, + maxTerminalEvidenceReferenceLength, + ); err != nil { + return fmt.Errorf("invalid terminal evidence reference: [%w]", err) + } + } else if evidence.Reference != "" { + return fmt.Errorf( + "terminal evidence kind [%s] must not carry a reference", + evidence.Kind, + ) + } + + dkgSignerEvidence := + evidence.Kind == TerminalEvidencePersistedTBTCSinger || + evidence.Kind == TerminalEvidencePersistedBeaconSigner + if dkgSignerEvidence { + if evidence.MembershipIndex == 0 || + evidence.MembershipIndex > group.MaxMemberIndex { + return fmt.Errorf( + "persisted DKG signer evidence requires a membership index "+ + "from 1 through %d", + group.MaxMemberIndex, + ) + } + } else if evidence.MembershipIndex != 0 { + return fmt.Errorf( + "terminal evidence kind [%s] must not carry a membership index", + evidence.Kind, + ) + } + + switch outcome { + case TerminalOutcomeCompleted: + if evidence.Kind == TerminalEvidenceQuarantinedTBTCSinger || + evidence.Kind == TerminalEvidenceQuarantinedBeaconSigner || + evidence.Kind == TerminalEvidenceNoThreshold { + return fmt.Errorf( + "completed outcome cannot use evidence kind [%s]", + evidence.Kind, + ) + } + case TerminalOutcomeQuarantined: + expected := TerminalEvidenceQuarantinedTBTCSinger + if ceremony == BeaconDKG { + expected = TerminalEvidenceQuarantinedBeaconSigner + } + if (ceremony != TBTCDKG && ceremony != BeaconDKG) || + evidence.Kind != expected { + return fmt.Errorf( + "quarantined outcome for ceremony [%s] requires evidence kind [%s]", + ceremony, + expected, + ) + } + case TerminalOutcomeExhausted: + if evidence.Kind != TerminalEvidenceNoThreshold { + return fmt.Errorf( + "exhausted outcome requires evidence kind [%s]", + TerminalEvidenceNoThreshold, + ) + } + } + + switch ceremony { + case TBTCDKG: + if outcome == TerminalOutcomeExhausted { + return fmt.Errorf( + "exhausted tbtc DKG has no chain-derived proof that another " + + "member did not publish an accepted result", + ) + } + case BeaconDKG: + if outcome == TerminalOutcomeExhausted { + return fmt.Errorf( + "exhausted beacon DKG has no chain-derived proof that another " + + "member did not publish an accepted result", + ) + } + } + + if outcome == TerminalOutcomeCompleted { + expected, known := completedEvidenceKinds[ceremony] + if !known { + return fmt.Errorf( + "ceremony [%s] has no declared completed evidence kind", + ceremony, + ) + } + if evidence.Kind != expected { + return fmt.Errorf( + "completed ceremony [%s] requires evidence kind [%s], got [%s]", + ceremony, + expected, + evidence.Kind, + ) + } + // A relay entry is the one protocol result whose reference is + // verifiable rather than merely well formed, so a malformed one is + // rejected here instead of reaching an audit that could not check it. + if ceremony == BeaconRelaySigning { + referenceStartBlock, _, _, _, err := ParseBeaconRelayEntryReference( + evidence.Reference, + ) + if err != nil { + return fmt.Errorf("invalid relay entry reference: [%w]", err) + } + + // The permit authorizes work on exactly one relay request, so an + // entry answering a different one settles nothing this permit was + // issued for. Refusing the pair here means a node cannot write the + // mismatch at all: a genuine entry from another request cannot be + // lifted onto this permit and still be recorded. + permitStartBlock, err := ParseBeaconRelayWorkID(workID) + if err != nil { + return fmt.Errorf( + "relay entry evidence cannot be bound to its permit: [%w]", + err, + ) + } + if referenceStartBlock != permitStartBlock { + return fmt.Errorf( + "relay entry reference answers request start block [%d], "+ + "but the permit was issued for request start block "+ + "[%d]", + referenceStartBlock, + permitStartBlock, + ) + } + } + + // A timeout report completes on the beacon's own settlement record, so + // the reference must be readable as one and must answer the request the + // permit was issued for. A settlement lifted from another request is a + // real penalty that this permit did not earn. + if ceremony == BeaconTimeoutReport { + referenceStartBlock, _, _, err := + ParseBeaconRelayTimeoutSettlementReference(evidence.Reference) + if err != nil { + return fmt.Errorf( + "invalid relay timeout settlement reference: [%w]", + err, + ) + } + + permitStartBlock, err := ParseBeaconRelayWorkID(workID) + if err != nil { + return fmt.Errorf( + "relay timeout settlement evidence cannot be bound to its "+ + "permit: [%w]", + err, + ) + } + if referenceStartBlock != permitStartBlock { + return fmt.Errorf( + "relay timeout settlement reference terminates request "+ + "start block [%d], but the permit was issued for "+ + "request start block [%d]", + referenceStartBlock, + permitStartBlock, + ) + } + } + } + + if err := validateTranscriptContribution( + ceremony, + outcome, + evidence, + ); err != nil { + return err + } + + return validateChainSettlement(ceremony, outcome, evidence.ChainSettlement) +} + +// transcriptContributionCeremonies names the ceremonies whose owners author the +// transcript behind a completed result, and whose completed record is therefore +// refused without one. +// +// It is a closed list rather than a blanket requirement because only a ceremony +// that authenticates its peers' contributions can author this honestly. A +// ceremony absent from the list produces no threshold transcript: a coordination +// proposal comes from one leader, a forwarder relays other members' shares and +// computes nothing, a timeout report is one node's penalty filing, and an +// inactivity claim carries the population it accuses rather than one that +// produced a result. Inventing a contribution for any of them would be the +// self-attestation this record exists to remove. +var transcriptContributionCeremonies = map[Ceremony]struct{}{ + // The final signing group, which is exactly the DKG members whose shares + // this node validated through every round. + TBTCDKG: {}, + // The signing members whose authenticated done checks carried the + // signature the action recorded. + TBTCSigning: {}, + // The same, for the heartbeat's own signing round. + TBTCHeartbeat: {}, + // The operating members of the accepted result, which are exactly the + // members whose round messages this node accepted through every round. + BeaconDKG: {}, + // The memberships whose signature shares, each authenticated against the + // group public key share published for it, were combined into the entry. + BeaconRelaySigning: {}, +} + +// AuthorsTranscriptContribution reports whether a ceremony's owner authenticates +// the population behind its result and therefore authors the transcript. A +// completed record for such a ceremony is refused without one; a completed +// record for any other ceremony is refused with one. +func AuthorsTranscriptContribution(ceremony Ceremony) bool { + _, authored := transcriptContributionCeremonies[ceremony] + return authored +} + +// validateTranscriptContribution checks the ceremony owner's account of which +// memberships combined into its result: that the ceremony is one that authors +// the account at all, that the memberships are a well-formed set, and that the +// node placed itself inside the population it is describing. +// +// The last part is what stops the record from becoming a report about other +// parties. A node may only ever say "these memberships, including mine, +// produced this result"; it cannot author a transcript it was not part of, and +// it cannot name a persisted membership that is absent from the memberships it +// says produced the result. +func validateTranscriptContribution( + ceremony Ceremony, + outcome TerminalOutcome, + evidence TerminalEvidence, +) error { + contribution := evidence.Contribution + + if outcome != TerminalOutcomeCompleted { + if contribution != nil { + return fmt.Errorf( + "terminal outcome [%s] produced no result and must not carry "+ + "a transcript contribution", + outcome, + ) + } + + return nil + } + + _, authored := transcriptContributionCeremonies[ceremony] + if contribution == nil { + if authored { + return fmt.Errorf( + "completed ceremony [%s] requires the transcript contribution "+ + "behind its result", + ceremony, + ) + } + + return nil + } + if !authored { + return fmt.Errorf( + "ceremony [%s] does not author a transcript contribution", + ceremony, + ) + } + + if len(contribution.IncorporatedMembers) == 0 { + return fmt.Errorf( + "a completed result names no memberships that produced it", + ) + } + if err := validateMemberIndexSet( + "incorporated memberships", + contribution.IncorporatedMembers, + ); err != nil { + return err + } + if err := validateMemberIndexSet( + "local memberships", + contribution.LocalMembers, + ); err != nil { + return err + } + + for _, local := range contribution.LocalMembers { + if !slices.Contains(contribution.IncorporatedMembers, local) { + return fmt.Errorf( + "local membership [%d] is absent from the memberships that "+ + "produced the result", + local, + ) + } + } + + // The persisted membership and the transcript are two statements about one + // ceremony, so a record naming a membership it does not claim to have + // operated is describing two different ceremonies at once. + if evidence.MembershipIndex != 0 && + !slices.Contains(contribution.LocalMembers, evidence.MembershipIndex) { + return fmt.Errorf( + "membership index [%d] is not among the memberships this node "+ + "operated in the transcript", + evidence.MembershipIndex, + ) + } + + _, mapped := permitSpaceMappingCeremonies[ceremony] + if !mapped { + if len(contribution.PermitSpaceMembers) != 0 { + return fmt.Errorf( + "ceremony [%s] records its result in the index space its "+ + "permits were issued in, so its transcript must not map "+ + "between spaces", + ceremony, + ) + } + + return nil + } + if len(contribution.PermitSpaceMembers) == 0 { + return fmt.Errorf( + "ceremony [%s] records its result in a different index space than "+ + "its permits, so its transcript requires the membership behind "+ + "each incorporated seat", + ceremony, + ) + } + if err := validateMemberIndexSet( + "permit-space memberships", + contribution.PermitSpaceMembers, + ); err != nil { + return err + } + // Positional alignment is the whole encoding of the mapping, so a length + // disagreement leaves every seat past the shorter list unmapped and the + // ones before it unverifiable. + if len(contribution.PermitSpaceMembers) != + len(contribution.IncorporatedMembers) { + return fmt.Errorf( + "the transcript names %d incorporated memberships but maps %d of "+ + "them back to the permits' index space", + len(contribution.IncorporatedMembers), + len(contribution.PermitSpaceMembers), + ) + } + + return nil +} + +// PermitSpaceMember translates one membership of a mapped transcript into the +// index space this work's permits were issued in. It reports false when the +// transcript does not name that membership, and it is the identity translation +// for the ceremonies whose result already speaks in the permits' space. +// +// It exists so that every reader joining a transcript to a permit — the gate at +// record time, the offline audit, the fleet ownership map a rehearsal builds — +// performs the translation the same way rather than each re-deriving it from the +// positional convention. +func PermitSpaceMember( + ceremony Ceremony, + contribution *TranscriptContribution, + member group.MemberIndex, +) (group.MemberIndex, bool) { + if _, mapped := permitSpaceMappingCeremonies[ceremony]; !mapped { + return member, true + } + if contribution == nil { + return 0, false + } + + for position, incorporated := range contribution.IncorporatedMembers { + if incorporated != member { + continue + } + if position >= len(contribution.PermitSpaceMembers) { + return 0, false + } + return contribution.PermitSpaceMembers[position], true + } + + return 0, false +} + +// permitSpaceMappingCeremonies names the ceremonies whose terminal record +// speaks in a different membership index space than the permits issued for the +// same work, and whose transcript therefore has to carry the mapping between +// the two. +// +// tBTC DKG is the only one. Its permit is issued for a DKG member index while +// its transcript and persisted membership are in the final signing group's +// index space, which is rebuilt after inactive and disqualified members are +// removed — so the same node legitimately operates seat 9 of the ceremony and +// persists seat 7 of the group. Every other ceremony records its result in the +// space its permits name: the beacon removes no membership between DKG and the +// persisted signer, beacon relay shares are combined in the existing group's +// space, and a wallet action's permit covers this node's seats in the very +// signing group its done checks are counted in. +var permitSpaceMappingCeremonies = map[Ceremony]struct{}{ + TBTCDKG: {}, +} + +// ValidatePermitOperatedOwnership checks that a terminal record claims no seat +// its own permit was not issued to operate. +// +// A permit's operated set and a completed record's local memberships are two +// node-authored statements about one permit, made at different times: the first +// before the ceremony ran, the second after it produced something. Holding the +// second against the first is what keeps the earlier statement load-bearing — +// without it a reader building a per-work ownership map from permits would have +// no assurance that the seats a node later claims to have produced a result with +// are the seats it announced it was operating, and the two accounts could be +// used against each other. +// +// Records whose index space differs from their permits' are compared through the +// transcript's own mapping rather than exempted from comparison. Exempting them +// left the load-bearing statement unchecked for the one ceremony that remaps, +// and a reader translating with the mapping needs to know the author could not +// have used it to move its own seat. +func ValidatePermitOperatedOwnership( + ceremony Ceremony, + operatedMembers MemberIndexes, + outcome TerminalOutcome, + evidence TerminalEvidence, +) error { + if err := validateMemberIndexSet( + "operated memberships", + operatedMembers, + ); err != nil { + return err + } + + _, mapped := permitSpaceMappingCeremonies[ceremony] + if evidence.MembershipIndex != 0 { + // The persisted membership is in the record's own space, so for a + // remapping ceremony it is the transcript that says which ceremony seat + // produced it. + persisted, translated := PermitSpaceMember( + ceremony, + evidence.Contribution, + evidence.MembershipIndex, + ) + if !translated { + return fmt.Errorf( + "membership index [%d] cannot be traced back to a membership "+ + "this permit could have operated", + evidence.MembershipIndex, + ) + } + if !slices.Contains(operatedMembers, persisted) { + if mapped { + return fmt.Errorf( + "membership index [%d] was produced by membership [%d], "+ + "which is not among the memberships [%v] this permit "+ + "was issued to operate", + evidence.MembershipIndex, + persisted, + operatedMembers, + ) + } + return fmt.Errorf( + "membership index [%d] is not among the memberships [%v] this "+ + "permit was issued to operate", + evidence.MembershipIndex, + operatedMembers, + ) + } + } + if evidence.Contribution == nil { + return nil + } + for _, local := range evidence.Contribution.LocalMembers { + operated, translated := PermitSpaceMember( + ceremony, + evidence.Contribution, + local, + ) + if !translated { + return fmt.Errorf( + "outcome [%s] claims membership [%d] in its transcript without "+ + "saying which membership produced it", + outcome, + local, + ) + } + if !slices.Contains(operatedMembers, operated) { + if mapped { + return fmt.Errorf( + "outcome [%s] claims membership [%d] in its transcript, "+ + "produced by membership [%d], which is not among the "+ + "memberships [%v] this permit was issued to operate", + outcome, + local, + operated, + operatedMembers, + ) + } + return fmt.Errorf( + "outcome [%s] claims membership [%d] in its transcript, which "+ + "is not among the memberships [%v] this permit was issued "+ + "to operate", + outcome, + local, + operatedMembers, + ) + } + } + + return nil +} + +// validateMemberIndexSet checks that a member-index list is an ascending, +// duplicate-free set of valid indexes. The ordering requirement is not +// cosmetic: it makes one set have exactly one encoding, so two records of the +// same transcript compare equal and a reader cannot be shown the same +// membership twice to inflate a population. Emptiness is a question for each +// caller, so it is left to them. +func validateMemberIndexSet(name string, indexes MemberIndexes) error { + previous := group.MemberIndex(0) + for _, index := range indexes { + if index == 0 { + return fmt.Errorf("%s contain the invalid membership index 0", name) + } + if index <= previous { + return fmt.Errorf( + "%s must be ascending and distinct; [%d] follows [%d]", + name, + index, + previous, + ) + } + previous = index + } + + return nil +} + +// chainSettlementKinds names the single chain side effect each ceremony may +// dispatch outside its own protocol result. A ceremony absent from the map +// dispatches none: it has no code path that submits to a chain, so a +// settlement recorded against it is a fabricated one, and accepting it would +// let any ceremony attach a penalty submission it could not have made. +var chainSettlementKinds = map[Ceremony]ChainSettlementKind{ + // A low-activity heartbeat files the inactivity claim under its own permit, + // so the heartbeat's terminal record is the only place that submission is + // ever reported. + TBTCHeartbeat: ChainSettlementInactivityClaim, +} + +func validateChainSettlement( + ceremony Ceremony, + outcome TerminalOutcome, + settlement *ChainSettlementRecord, +) error { + if settlement == nil { + return nil + } + + expected, dispatches := chainSettlementKinds[ceremony] + if !dispatches { + return fmt.Errorf( + "ceremony [%s] dispatches no chain settlement", + ceremony, + ) + } + if settlement.Kind != expected { + return fmt.Errorf( + "ceremony [%s] dispatches chain settlement kind [%s], got [%s]", + ceremony, + expected, + settlement.Kind, + ) + } + // Every dispatch path runs downstream of the ceremony's own threshold + // result, so a ceremony that reports no result cannot have reached one. + if outcome != TerminalOutcomeCompleted { + return fmt.Errorf( + "[%s] outcome cannot carry a chain settlement", + outcome, + ) + } + + // An empty reference is the deliberate unresolved-submission record; only a + // reference that claims a settlement has to name one canonically. + if settlement.Reference == "" { + return nil + } + + switch settlement.Kind { + case ChainSettlementInactivityClaim: + if _, _, err := ParseInactivityClaimSettlementReference( + settlement.Reference, + ); err != nil { + return fmt.Errorf("invalid chain settlement reference: [%w]", err) + } + } + + return nil +} + +// completedEvidenceKinds names the single evidence kind each ceremony may use +// to claim a durable result. The mapping is deliberately exhaustive and +// one-to-one: without it, a ceremony whose real result is an external +// transaction — a signed Bitcoin spend, an on-chain penalty submission — could +// settle its permit with TerminalEvidenceProtocolResult, a digest the node +// authors entirely by itself. That would let an ambiguous submission clear the +// rollback journal on the node's own say-so, with nothing for the offline audit +// to reconcile against canonical state. Each ceremony is therefore pinned to +// the evidence class its result actually lives in. +// +// A ceremony added to AllCeremonies without an entry here fails closed: its +// completed outcome is rejected, the permit closes unresolved, and the offline +// barrier blocks until the omission is fixed. +var completedEvidenceKinds = map[Ceremony]TerminalEvidenceKind{ + // The wallet's persisted signing-group membership. + TBTCDKG: TerminalEvidencePersistedTBTCSinger, + // The agreed proposal; it dispatches an action that settles separately. + TBTCWalletCoordination: TerminalEvidenceProtocolResult, + // The signed Bitcoin transaction the action may have broadcast. + TBTCSigning: TerminalEvidenceBitcoinTransaction, + // The threshold signature over the proposed heartbeat message. + TBTCHeartbeat: TerminalEvidenceProtocolResult, + // The claim submission, which is Ethereum state and never node-authored. + TBTCInactivityClaim: TerminalEvidenceEthereumTransaction, + // The persisted threshold signer. + BeaconDKG: TerminalEvidencePersistedBeaconSigner, + // The recovered relay entry, deterministic for a given previous entry. + BeaconRelaySigning: TerminalEvidenceProtocolResult, + // The forwarder relays other members' shares and produces no result of its + // own; reaching its close is the whole of its durable disposition. + BeaconRelayForwarding: TerminalEvidenceForwarderClosed, + // The beacon's own record of the terminated request, which is Ethereum + // state and never node-authored. + BeaconTimeoutReport: TerminalEvidenceEthereumTransaction, +} + +func terminalOutcomeRecordLess( + left TerminalOutcomeRecord, + right TerminalOutcomeRecord, +) bool { + if left.Permit.Ceremony != right.Permit.Ceremony { + return left.Permit.Ceremony < right.Permit.Ceremony + } + if left.Permit.CanonicalStartBlock != right.Permit.CanonicalStartBlock { + return left.Permit.CanonicalStartBlock < + right.Permit.CanonicalStartBlock + } + if left.Permit.WorkID != right.Permit.WorkID { + return left.Permit.WorkID < right.Permit.WorkID + } + return left.Permit.PermitID < right.Permit.PermitID +} + +// QuiescenceSnapshotRecorder persists the node-authored snapshot and the +// ceremony-owner-authored terminal outcomes. Implementations must serialize +// concurrent terminal writes and make contradictory duplicate outcomes fail. +type QuiescenceSnapshotRecorder interface { + Record(snapshot QuiescenceSnapshot) error + RecordTerminalOutcome(outcome TerminalOutcomeRecord) error +} + +// quiescencePersistence is the narrow persistence handle used by the +// recorder. persistence.BasicHandle satisfies it without coupling the gate to +// the rest of the storage API. +type quiescencePersistence interface { + Save(data []byte, directory string, name string) error + Delete(directory string, name string) error +} + +type persistenceQuiescenceSnapshotRecorder struct { + persistence quiescencePersistence + + mutex sync.Mutex + journal TerminalOutcomeJournal +} + +// NewPersistenceQuiescenceSnapshotRecorder constructs the production +// recorder for the encrypted work/participation namespace. +func NewPersistenceQuiescenceSnapshotRecorder( + persistence quiescencePersistence, +) (QuiescenceSnapshotRecorder, error) { + if persistence == nil { + return nil, fmt.Errorf("quiescence snapshot persistence is required") + } + + // A snapshot authorizes only the process run that produced it. Clear any + // prior run's record before the gate becomes available; if the node exits + // or a later write fails, the offline audit then sees a missing or malformed + // record instead of accepting stale inventory. + if err := persistence.Delete( + QuiescenceSnapshotStorageDirectory, + QuiescenceSnapshotStorageFile, + ); err != nil && !errors.Is(err, fs.ErrNotExist) { + return nil, fmt.Errorf( + "cannot invalidate the prior quiescence snapshot: [%w]", + err, + ) + } + if err := persistence.Delete( + QuiescenceSnapshotStorageDirectory, + TerminalOutcomeJournalStorageFile, + ); err != nil && !errors.Is(err, fs.ErrNotExist) { + return nil, fmt.Errorf( + "cannot invalidate the prior terminal-outcome journal: [%w]", + err, + ) + } + + return &persistenceQuiescenceSnapshotRecorder{ + persistence: persistence, + }, nil +} + +func (r *persistenceQuiescenceSnapshotRecorder) Record( + snapshot QuiescenceSnapshot, +) error { + r.mutex.Lock() + defer r.mutex.Unlock() + + content, err := json.MarshalIndent(snapshot, "", " ") + if err != nil { + return fmt.Errorf("cannot encode the quiescence snapshot: [%w]", err) + } + + if err := r.persistence.Save( + content, + QuiescenceSnapshotStorageDirectory, + QuiescenceSnapshotStorageFile, + ); err != nil { + return fmt.Errorf("cannot persist the quiescence snapshot: [%w]", err) + } + + r.journal = TerminalOutcomeJournal{ + SchemaVersion: TerminalOutcomeJournalSchemaVersion, + SnapshotCapturedAt: snapshot.CapturedAt, + } + if err := r.persistJournalLocked(); err != nil { + return err + } + + return nil +} + +func (r *persistenceQuiescenceSnapshotRecorder) RecordTerminalOutcome( + outcome TerminalOutcomeRecord, +) error { + r.mutex.Lock() + defer r.mutex.Unlock() + + if r.journal.SnapshotCapturedAt.IsZero() { + return fmt.Errorf( + "cannot persist a terminal outcome before the quiescence snapshot", + ) + } + + for _, existing := range r.journal.Outcomes { + if existing.Permit.Equal(outcome.Permit) { + if existing.Equal(outcome) { + return nil + } + return fmt.Errorf( + "terminal outcome already recorded for ceremony [%s] "+ + "[workID=%s] [permitID=%s]", + outcome.Permit.Ceremony, + outcome.Permit.WorkID, + outcome.Permit.PermitID, + ) + } + } + + previous := append( + []TerminalOutcomeRecord(nil), + r.journal.Outcomes..., + ) + r.journal.Outcomes = append(r.journal.Outcomes, outcome) + sort.Slice(r.journal.Outcomes, func(i, j int) bool { + return terminalOutcomeRecordLess( + r.journal.Outcomes[i], + r.journal.Outcomes[j], + ) + }) + + if err := r.persistJournalLocked(); err != nil { + r.journal.Outcomes = previous + return err + } + + return nil +} + +func (r *persistenceQuiescenceSnapshotRecorder) persistJournalLocked() error { + content, err := json.MarshalIndent(r.journal, "", " ") + if err != nil { + return fmt.Errorf( + "cannot encode the terminal-outcome journal: [%w]", + err, + ) + } + + if err := r.persistence.Save( + content, + QuiescenceSnapshotStorageDirectory, + TerminalOutcomeJournalStorageFile, + ); err != nil { + return fmt.Errorf( + "cannot persist the terminal-outcome journal: [%w]", + err, + ) + } + + return nil +} diff --git a/pkg/protocol/participation/quiescence_test.go b/pkg/protocol/participation/quiescence_test.go new file mode 100644 index 0000000000..670139dca3 --- /dev/null +++ b/pkg/protocol/participation/quiescence_test.go @@ -0,0 +1,1477 @@ +package participation + +import ( + "bytes" + "encoding/hex" + "encoding/json" + "errors" + "io/fs" + "math" + "math/big" + "slices" + "strconv" + "strings" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +type quiescencePersistenceRecorder struct { + deleted []string + deleteErr error + saved map[string][]byte +} + +func (p *quiescencePersistenceRecorder) Save( + data []byte, + directory string, + name string, +) error { + if p.saved == nil { + p.saved = make(map[string][]byte) + } + p.saved[directory+"/"+name] = append([]byte(nil), data...) + return nil +} + +func (p *quiescencePersistenceRecorder) Delete( + directory string, + name string, +) error { + p.deleted = append(p.deleted, directory+"/"+name) + return p.deleteErr +} + +func TestNewPersistenceQuiescenceSnapshotRecorder_InvalidatesPriorRun( + t *testing.T, +) { + persistence := &quiescencePersistenceRecorder{} + + if _, err := NewPersistenceQuiescenceSnapshotRecorder( + persistence, + ); err != nil { + t.Fatal(err) + } + + expected := map[string]bool{ + QuiescenceSnapshotStorageDirectory + "/" + + QuiescenceSnapshotStorageFile: false, + QuiescenceSnapshotStorageDirectory + "/" + + TerminalOutcomeJournalStorageFile: false, + } + for _, deleted := range persistence.deleted { + if _, ok := expected[deleted]; ok { + expected[deleted] = true + } + } + for record, found := range expected { + if found { + continue + } + t.Errorf( + "expected prior record [%s] to be invalidated; deletes: %v", + record, + persistence.deleted, + ) + } +} + +func TestNewPersistenceQuiescenceSnapshotRecorder_MissingPriorRunIsAllowed( + t *testing.T, +) { + persistence := &quiescencePersistenceRecorder{ + deleteErr: fs.ErrNotExist, + } + + if _, err := NewPersistenceQuiescenceSnapshotRecorder( + persistence, + ); err != nil { + t.Fatal(err) + } +} + +func TestNewPersistenceQuiescenceSnapshotRecorder_DeleteFailureIsFatal( + t *testing.T, +) { + deleteErr := errors.New("delete failed") + persistence := &quiescencePersistenceRecorder{ + deleteErr: deleteErr, + } + + if _, err := NewPersistenceQuiescenceSnapshotRecorder( + persistence, + ); !errors.Is(err, deleteErr) { + t.Fatalf("expected delete failure, got [%v]", err) + } +} + +func TestPersistenceQuiescenceSnapshotRecorder_PersistsTerminalJournal( + t *testing.T, +) { + persistence := &quiescencePersistenceRecorder{} + recorder, err := NewPersistenceQuiescenceSnapshotRecorder(persistence) + if err != nil { + t.Fatal(err) + } + + capturedAt := time.Date(2026, time.July, 28, 12, 0, 0, 0, time.UTC) + if err := recorder.Record(QuiescenceSnapshot{ + SchemaVersion: QuiescenceSnapshotSchemaVersion, + CapturedAt: capturedAt, + }); err != nil { + t.Fatal(err) + } + + outcome := TerminalOutcomeRecord{ + RecordedAt: capturedAt.Add(time.Second), + Permit: PermitSnapshot{ + Ceremony: TBTCSigning, + Mode: ModeSecurityV2.String(), + CanonicalStartBlock: 1_000, + WorkID: "wallet-action", + PermitID: "wallet", + IdentityBound: true, + }, + Outcome: TerminalOutcomeCompleted, + Evidence: TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: "result-digest", + Contribution: testTranscriptContribution(TBTCSigning, 1), + }, + } + if err := recorder.RecordTerminalOutcome(outcome); err != nil { + t.Fatal(err) + } + + content := persistence.saved[QuiescenceSnapshotStorageDirectory+ + "/"+TerminalOutcomeJournalStorageFile] + journal := &TerminalOutcomeJournal{} + if err := json.Unmarshal(content, journal); err != nil { + t.Fatal(err) + } + if journal.SchemaVersion != TerminalOutcomeJournalSchemaVersion { + t.Errorf( + "unexpected journal schema [%d]", + journal.SchemaVersion, + ) + } + if !journal.SnapshotCapturedAt.Equal(capturedAt) { + t.Errorf( + "unexpected snapshot binding [%s]", + journal.SnapshotCapturedAt, + ) + } + if len(journal.Outcomes) != 1 || + !journal.Outcomes[0].Equal(outcome) { + t.Errorf("unexpected terminal outcomes: %+v", journal.Outcomes) + } +} + +func TestValidateTerminalOutcome_RejectsUnsupportedEvidence(t *testing.T) { + err := ValidateTerminalOutcome( + TBTCSigning, + testWorkID(TBTCSigning), + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidenceKind("fabricated_result"), + Reference: "fabricated-result", + }, + ) + if err == nil { + t.Fatal("expected unsupported terminal evidence to be rejected") + } +} + +func TestValidateTerminalOutcome_DKGCompletionRequiresExactMembership( + t *testing.T, +) { + tests := map[string]struct { + ceremony Ceremony + kind TerminalEvidenceKind + }{ + "tbtc": { + ceremony: TBTCDKG, + kind: TerminalEvidencePersistedTBTCSinger, + }, + "beacon": { + ceremony: BeaconDKG, + kind: TerminalEvidencePersistedBeaconSigner, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + evidence := TerminalEvidence{ + Kind: test.kind, + Reference: "persisted-signer", + Contribution: testTranscriptContribution(test.ceremony, 7), + } + if err := ValidateTerminalOutcome( + test.ceremony, + testWorkID(test.ceremony), + TerminalOutcomeCompleted, + evidence, + ); err == nil { + t.Fatal( + "expected DKG completion without a membership index " + + "to be rejected", + ) + } + + evidence.MembershipIndex = group.MemberIndex(7) + if err := ValidateTerminalOutcome( + test.ceremony, + testWorkID(test.ceremony), + TerminalOutcomeCompleted, + evidence, + ); err != nil { + t.Fatalf( + "expected exact persisted membership evidence to pass: [%v]", + err, + ) + } + }) + } +} + +func TestValidateTerminalOutcome_RejectsUnauthenticatedDKGExhaustion( + t *testing.T, +) { + for _, ceremony := range []Ceremony{TBTCDKG, BeaconDKG} { + err := ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeExhausted, + TerminalEvidence{Kind: TerminalEvidenceNoThreshold}, + ) + if err == nil { + t.Errorf( + "expected local no-threshold marker for [%s] to be rejected", + ceremony, + ) + } + } +} + +// TestTerminalResultReference_ComponentBoundariesAreBinding asserts the +// reference builder cannot be made to collide by shifting the boundary between +// two components. Concatenating adjacent components must not reproduce another +// call's digest, otherwise two different ceremony results could claim the same +// journal identity. +func TestTerminalResultReference_ComponentBoundariesAreBinding(t *testing.T) { + shifted := map[string][][]byte{ + "split one way": {[]byte("ab"), []byte("c")}, + "split another way": {[]byte("a"), []byte("bc")}, + "joined": {[]byte("abc")}, + } + + seen := make(map[string]string) + for name, components := range shifted { + reference := TerminalResultReference("domain", components...) + + if previous, taken := seen[reference]; taken { + t.Errorf( + "[%s] collides with [%s] on reference [%s]", + name, + previous, + reference, + ) + } + seen[reference] = name + } +} + +// TestTerminalResultReference_DomainSeparates asserts two ceremonies cannot +// derive the same reference from identical components. +func TestTerminalResultReference_DomainSeparates(t *testing.T) { + components := [][]byte{{0x01, 0x02}} + + if TerminalResultReference("first_domain", components...) == + TerminalResultReference("second_domain", components...) { + t.Error("distinct domains produced the same reference") + } +} + +// testRelayRequestStartBlock is the relay request every relay-flavored fixture +// in this file answers. Relay evidence is only valid against the permit issued +// for its own request, so the reference and the work identity have to agree on +// one block. +const testRelayRequestStartBlock = uint64(4_100_900) + +// testWorkID renders a work identity the given ceremony's permits would carry. +// Only the beacon relay ceremonies constrain it: their evidence names the +// request it answers, and the validator holds that to the request the permit +// was issued for. +func testWorkID(ceremony Ceremony) string { + if isBeaconRelayCeremony(ceremony) { + return BeaconRelayWorkID(testRelayRequestStartBlock) + } + return "work-identity" +} + +// testTranscriptContribution renders the transcript a completed outcome of the +// given ceremony must carry, and nil for the ceremonies that author none. The +// local membership is passed in rather than fixed so a fixture that also names +// a persisted membership index agrees with its own transcript, leaving the +// disagreement to the tests that mean to provoke it. +func testTranscriptContribution( + ceremony Ceremony, + local group.MemberIndex, +) *TranscriptContribution { + if _, authored := transcriptContributionCeremonies[ceremony]; !authored { + return nil + } + + incorporated := []group.MemberIndex{local} + for _, peer := range []group.MemberIndex{1, 2, 3} { + if peer != local { + incorporated = append(incorporated, peer) + } + } + slices.Sort(incorporated) + + return &TranscriptContribution{ + IncorporatedMembers: incorporated, + LocalMembers: []group.MemberIndex{local}, + PermitSpaceMembers: testPermitSpaceMembers(ceremony, incorporated), + } +} + +// testPermitSpaceMembers renders the mapping from a transcript's seats back to +// the index space this work's permits were issued in, for the ceremonies that +// require one, and nil for the ceremonies whose result already speaks in that +// space. +// +// The mapping is the identity so that a fixture naming a persisted membership or +// a permit's operated seat needs no translating to agree with itself. The tests +// that mean to exercise a real remapping build their own; see +// TestPermit_TBTCDKGTranscriptIsHeldToItsPermitSeatThroughTheMapping. +func testPermitSpaceMembers( + ceremony Ceremony, + incorporated []group.MemberIndex, +) MemberIndexes { + if _, mapped := permitSpaceMappingCeremonies[ceremony]; !mapped { + return nil + } + + return slices.Clone(incorporated) +} + +// isBeaconRelayCeremony reports whether a ceremony's permits are issued for an +// on-chain relay request. +func isBeaconRelayCeremony(ceremony Ceremony) bool { + return ceremony == BeaconRelaySigning || + ceremony == BeaconTimeoutReport || + ceremony == BeaconRelayForwarding +} + +// testCompletedResultReference renders a completed-outcome reference the given +// ceremony accepts. Most ceremonies name a digest of their own result; the +// relay ceremonies name the request they answer, so no placeholder can stand +// in for one. A relay entry additionally names the group, the previous entry +// and the entry itself so the offline audit can verify the signature, and a +// timeout report names the beacon's own request identifier and terminated +// group so the audit can join it to an authenticated log. +func testCompletedResultReference( + t *testing.T, + ceremony Ceremony, + digest string, +) string { + t.Helper() + + if ceremony == BeaconTimeoutReport { + reference, err := BeaconRelayTimeoutSettlementReference( + testRelayRequestStartBlock, + big.NewInt(7), + 3, + ) + if err != nil { + t.Fatal(err) + } + return reference + } + if ceremony != BeaconRelaySigning { + return digest + } + + reference, err := BeaconRelayEntryReference( + testRelayRequestStartBlock, + bytes.Repeat([]byte{0x01}, beaconRelayEntryComponentLength), + bytes.Repeat([]byte{0x02}, beaconRelayEntryComponentLength), + bytes.Repeat([]byte{0x03}, beaconRelayEntryComponentLength), + ) + if err != nil { + t.Fatal(err) + } + + return reference +} + +// TestTerminalResultReference_IsAcceptedAsEvidence asserts a derived reference +// passes the journal's own identity rules, so a ceremony that authors one can +// actually record it. +// +// The list is exactly the ceremonies whose durable result is a protocol result. +// A timeout report is deliberately absent: its result is the beacon's own +// settlement record, so it has no derived reference to accept. +func TestTerminalResultReference_IsAcceptedAsEvidence(t *testing.T) { + for _, ceremony := range []Ceremony{ + TBTCHeartbeat, + TBTCWalletCoordination, + BeaconRelaySigning, + } { + if err := ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeCompleted, + TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: testCompletedResultReference( + t, + ceremony, + TerminalResultReference("domain", []byte("result")), + ), + Contribution: testTranscriptContribution(ceremony, 1), + }, + ); err != nil { + t.Errorf( + "derived reference rejected for ceremony [%s]: [%v]", + ceremony, + err, + ) + } + } +} + +// TestBeaconRelayTimeoutSettlementReference_RoundTrips asserts the beacon +// settlement identity survives rendering and parsing unchanged, and that only +// its exact canonical rendering is accepted. An alias that names the same +// settlement while failing every comparison the audit makes against it is +// indistinguishable from naming no settlement at all. +func TestBeaconRelayTimeoutSettlementReference_RoundTrips(t *testing.T) { + requestID := big.NewInt(4_294_967_297) + const terminatedGroupID = uint64(9) + + reference, err := BeaconRelayTimeoutSettlementReference( + testRelayRequestStartBlock, + requestID, + terminatedGroupID, + ) + if err != nil { + t.Fatal(err) + } + + parsedStartBlock, parsedRequestID, parsedGroupID, err := + ParseBeaconRelayTimeoutSettlementReference(reference) + if err != nil { + t.Fatal(err) + } + if parsedStartBlock != testRelayRequestStartBlock { + t.Errorf( + "request start block did not round-trip\n"+ + "expected: [%d]\nactual: [%d]", + testRelayRequestStartBlock, + parsedStartBlock, + ) + } + if parsedRequestID.Cmp(requestID) != 0 { + t.Errorf( + "request identifier did not round-trip\n"+ + "expected: [%s]\nactual: [%s]", + requestID, + parsedRequestID, + ) + } + if parsedGroupID != terminatedGroupID { + t.Errorf( + "terminated group did not round-trip\n"+ + "expected: [%d]\nactual: [%d]", + terminatedGroupID, + parsedGroupID, + ) + } + + // A settlement with no request identifier names no log, so it cannot be + // rendered at all rather than being rendered as an absent component. + for _, test := range []struct { + name string + requestID *big.Int + }{ + {name: "missing request identifier", requestID: nil}, + {name: "negative request identifier", requestID: big.NewInt(-1)}, + } { + if _, err := BeaconRelayTimeoutSettlementReference( + testRelayRequestStartBlock, + test.requestID, + terminatedGroupID, + ); err == nil { + t.Errorf("expected [%s] to be rejected", test.name) + } + } + + for _, test := range []struct { + name string + reference string + }{ + {name: "no components", reference: ""}, + {name: "two components", reference: "1000:11"}, + {name: "four components", reference: reference + ":1"}, + {name: "prefixed alias", reference: "0x" + reference}, + // Any component rendered other than canonically names the same + // settlement while comparing unequal to the reference the audit + // rebuilds, which reads as naming none. + {name: "zero-padded start block", reference: "0" + reference}, + {name: "signed start block", reference: "+" + reference}, + {name: "hexadecimal start block", reference: "0x" + + strconv.FormatUint(testRelayRequestStartBlock, 16) + ":" + + requestID.String() + ":" + + strconv.FormatUint(terminatedGroupID, 10)}, + {name: "zero-padded request identifier", reference: strconv.FormatUint( + testRelayRequestStartBlock, 10) + ":0" + requestID.String() + ":" + + strconv.FormatUint(terminatedGroupID, 10)}, + {name: "signed request identifier", reference: strconv.FormatUint( + testRelayRequestStartBlock, 10) + ":+" + requestID.String() + ":" + + strconv.FormatUint(terminatedGroupID, 10)}, + {name: "negative request identifier", reference: strconv.FormatUint( + testRelayRequestStartBlock, 10) + ":-11:" + + strconv.FormatUint(terminatedGroupID, 10)}, + {name: "zero-padded terminated group", reference: strconv.FormatUint( + testRelayRequestStartBlock, 10) + ":" + requestID.String() + ":09"}, + {name: "non-numeric terminated group", reference: strconv.FormatUint( + testRelayRequestStartBlock, 10) + ":" + requestID.String() + + ":group"}, + } { + if _, _, _, err := ParseBeaconRelayTimeoutSettlementReference( + test.reference, + ); err == nil { + t.Errorf("expected [%s] to be rejected", test.name) + } + } +} + +// TestBeaconRelayEntryReference_RoundTrips asserts the relay entry identity +// survives rendering and parsing unchanged, and that only its exact canonical +// rendering is accepted. An alias that names the same entry while failing every +// comparison the audit makes against it is indistinguishable from naming no +// entry at all. +func TestBeaconRelayEntryReference_RoundTrips(t *testing.T) { + groupPublicKey := bytes.Repeat([]byte{0xa1}, beaconRelayEntryComponentLength) + previousEntry := bytes.Repeat([]byte{0xb2}, beaconRelayEntryComponentLength) + entry := bytes.Repeat([]byte{0xc3}, beaconRelayEntryComponentLength) + + reference, err := BeaconRelayEntryReference( + testRelayRequestStartBlock, + groupPublicKey, + previousEntry, + entry, + ) + if err != nil { + t.Fatal(err) + } + + parsedStartBlock, parsedGroup, parsedPrevious, parsedEntry, err := + ParseBeaconRelayEntryReference(reference) + if err != nil { + t.Fatal(err) + } + if parsedStartBlock != testRelayRequestStartBlock { + t.Errorf( + "request start block did not round-trip\n"+ + "expected: [%d]\nactual: [%d]", + testRelayRequestStartBlock, + parsedStartBlock, + ) + } + for _, component := range []struct { + name string + expected []byte + actual []byte + }{ + {name: "group public key", expected: groupPublicKey, actual: parsedGroup}, + {name: "previous entry", expected: previousEntry, actual: parsedPrevious}, + {name: "entry", expected: entry, actual: parsedEntry}, + } { + if !bytes.Equal(component.expected, component.actual) { + t.Errorf( + "%s did not round-trip\nexpected: [%x]\nactual: [%x]", + component.name, + component.expected, + component.actual, + ) + } + } + + short := bytes.Repeat([]byte{0x01}, beaconRelayEntryComponentLength-1) + for _, test := range []struct { + name string + groupPublicKey []byte + previousEntry []byte + entry []byte + }{ + {name: "short group public key", groupPublicKey: short, previousEntry: previousEntry, entry: entry}, + {name: "short previous entry", groupPublicKey: groupPublicKey, previousEntry: short, entry: entry}, + {name: "short entry", groupPublicKey: groupPublicKey, previousEntry: previousEntry, entry: short}, + } { + if _, err := BeaconRelayEntryReference( + testRelayRequestStartBlock, + test.groupPublicKey, + test.previousEntry, + test.entry, + ); err == nil { + t.Errorf("expected [%s] to be rejected", test.name) + } + } + + components := hex.EncodeToString(groupPublicKey) + ":" + + hex.EncodeToString(previousEntry) + ":" + hex.EncodeToString(entry) + for _, test := range []struct { + name string + reference string + }{ + {name: "no components", reference: ""}, + {name: "three components", reference: components}, + {name: "five components", reference: reference + ":" + + hex.EncodeToString(entry)}, + {name: "uppercase alias", reference: strings.ToUpper(reference)}, + {name: "prefixed alias", reference: "0x" + reference}, + // A start block rendered any way but the canonical one names the same + // request while comparing unequal to the permit's own rendering, which + // is what the binding check reads. + {name: "zero-padded start block", reference: "0" + reference}, + {name: "signed start block", reference: "+" + reference}, + {name: "hexadecimal start block", reference: "0x" + + strconv.FormatUint(testRelayRequestStartBlock, 16) + ":" + + components}, + {name: "non-numeric start block", reference: "block:" + components}, + } { + if _, _, _, _, err := ParseBeaconRelayEntryReference( + test.reference, + ); err == nil { + t.Errorf("expected [%s] to be rejected", test.name) + } + } +} + +// TestBeaconRelayWorkID_RoundTrips asserts a relay permit's work identity +// names exactly one request start block and that only the exact rendering the +// node writes is read back. The audit compares the request a relay entry +// answers against the request its permit was issued for, so a work identity +// that parses loosely would let two renderings of one block — or a block that +// was never rendered at all — pass as agreement. +func TestBeaconRelayWorkID_RoundTrips(t *testing.T) { + for _, startBlock := range []uint64{ + 0, + 1, + testRelayRequestStartBlock, + math.MaxUint64, + } { + workID := BeaconRelayWorkID(startBlock) + + parsed, err := ParseBeaconRelayWorkID(workID) + if err != nil { + t.Fatalf("work identity [%s] was rejected: [%v]", workID, err) + } + if parsed != startBlock { + t.Errorf( + "work identity [%s] did not round-trip\n"+ + "expected: [%d]\nactual: [%d]", + workID, + startBlock, + parsed, + ) + } + } + + for _, test := range []struct { + name string + workID string + }{ + {name: "empty", workID: ""}, + {name: "no request", workID: "relay-entry-17"}, + {name: "no start block", workID: "relay-request-"}, + {name: "zero-padded", workID: "relay-request-017"}, + {name: "signed", workID: "relay-request-+17"}, + {name: "negative", workID: "relay-request--17"}, + {name: "hexadecimal", workID: "relay-request-0x11"}, + {name: "beyond uint64", workID: "relay-request-18446744073709551616"}, + {name: "trailing text", workID: "relay-request-17-timeout-monitor"}, + } { + if _, err := ParseBeaconRelayWorkID(test.workID); err == nil { + t.Errorf("expected [%s] to be rejected", test.name) + } + } +} + +// TestValidateTerminalOutcome_RelayEntryIsBoundToItsRequest asserts a relay +// entry can only settle the permit issued for the request it answers. The +// consequential direction is a genuine historical entry, which verifies as a +// threshold signature forever, standing in as the result of an unrelated +// request whose work this node may never have completed. +func TestValidateTerminalOutcome_RelayEntryIsBoundToItsRequest(t *testing.T) { + relayEvidence := func(startBlock uint64) TerminalEvidence { + reference, err := BeaconRelayEntryReference( + startBlock, + bytes.Repeat([]byte{0xa1}, beaconRelayEntryComponentLength), + bytes.Repeat([]byte{0xb2}, beaconRelayEntryComponentLength), + bytes.Repeat([]byte{0xc3}, beaconRelayEntryComponentLength), + ) + if err != nil { + t.Fatal(err) + } + return TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: reference, + // The relay ceremony authors the population behind its entry, so a + // completed record carrying none is refused before the reference is + // ever read. This case is about the reference, so every record here + // carries a well-formed transcript. + Contribution: &TranscriptContribution{ + IncorporatedMembers: MemberIndexes{1, 2}, + LocalMembers: MemberIndexes{2}, + }, + } + } + + if err := ValidateTerminalOutcome( + BeaconRelaySigning, + BeaconRelayWorkID(testRelayRequestStartBlock), + TerminalOutcomeCompleted, + relayEvidence(testRelayRequestStartBlock), + ); err != nil { + t.Fatalf("an entry answering its own request was rejected: [%v]", err) + } + + // The widest reference a real relay round can produce still has to fit the + // evidence bound. A bound that no longer admits it would reject genuine + // results at the top of the block range as unnameable, which the recorder + // reads as no result at all. + if err := ValidateTerminalOutcome( + BeaconRelaySigning, + BeaconRelayWorkID(math.MaxUint64), + TerminalOutcomeCompleted, + relayEvidence(math.MaxUint64), + ); err != nil { + t.Errorf("the widest relay entry reference was rejected: [%v]", err) + } + + for _, test := range []struct { + name string + workID string + startBlock uint64 + }{ + { + name: "an entry from an earlier request", + workID: BeaconRelayWorkID(testRelayRequestStartBlock), + startBlock: testRelayRequestStartBlock - 1, + }, + { + name: "an entry from a later request", + workID: BeaconRelayWorkID(testRelayRequestStartBlock), + startBlock: testRelayRequestStartBlock + 1, + }, + { + name: "a permit naming no relay request", + workID: "wallet-action", + startBlock: testRelayRequestStartBlock, + }, + { + name: "a permit whose request is not canonically rendered", + workID: "relay-request-0" + strconv.FormatUint(testRelayRequestStartBlock, 10), + startBlock: testRelayRequestStartBlock, + }, + } { + if err := ValidateTerminalOutcome( + BeaconRelaySigning, + test.workID, + TerminalOutcomeCompleted, + relayEvidence(test.startBlock), + ); err == nil { + t.Errorf("expected [%s] to be rejected", test.name) + } + } +} + +// TestValidateTerminalOutcome_CompletedEvidenceKindIsPinnedPerCeremony asserts +// no ceremony can settle a completed permit with an evidence class other than +// the one its result actually lives in. The consequential direction is a +// ceremony whose result is external state — a Bitcoin spend, an Ethereum +// penalty submission — settling on a node-authored protocol digest instead, +// which would leave the offline audit nothing canonical to reconcile against. +func TestValidateTerminalOutcome_CompletedEvidenceKindIsPinnedPerCeremony( + t *testing.T, +) { + everyKind := []TerminalEvidenceKind{ + TerminalEvidencePersistedTBTCSinger, + TerminalEvidencePersistedBeaconSigner, + TerminalEvidenceBitcoinTransaction, + TerminalEvidenceEthereumTransaction, + TerminalEvidenceProtocolResult, + TerminalEvidenceForwarderClosed, + } + + for _, ceremony := range AllCeremonies() { + expected, declared := completedEvidenceKinds[ceremony] + if !declared { + t.Errorf( + "ceremony [%s] has no declared completed evidence kind; a "+ + "completed permit for it can never be recorded", + ceremony, + ) + continue + } + + for _, kind := range everyKind { + evidence := TerminalEvidence{ + Kind: kind, + Contribution: testTranscriptContribution(ceremony, 1), + } + switch kind { + case TerminalEvidencePersistedTBTCSinger, + TerminalEvidencePersistedBeaconSigner: + evidence.MembershipIndex = 1 + evidence.Reference = "persisted-signer-identity" + case TerminalEvidenceForwarderClosed: + default: + evidence.Reference = testCompletedResultReference( + t, + ceremony, + "durable-result-identity", + ) + } + + err := ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeCompleted, + evidence, + ) + if kind == expected && err != nil { + t.Errorf( + "ceremony [%s] rejected its own evidence kind [%s]: [%v]", + ceremony, + kind, + err, + ) + } + if kind != expected && err == nil { + t.Errorf( + "ceremony [%s] accepted foreign evidence kind [%s]; only "+ + "[%s] identifies its durable result", + ceremony, + kind, + expected, + ) + } + } + } +} + +// TestInactivityClaimSettlementReference_RoundTrips asserts the canonical claim +// identity survives a round trip and that only the exact rendering is accepted. +// The audit joins this string to a chain log by comparison, so an alias that +// names the same claim in a different shape is indistinguishable from naming no +// claim at all. +func TestInactivityClaimSettlementReference_RoundTrips(t *testing.T) { + walletID := make([]byte, inactivityClaimWalletIDLength) + for i := range walletID { + walletID[i] = byte(i + 1) + } + + for _, nonce := range []*big.Int{ + big.NewInt(0), + big.NewInt(1), + new(big.Int).Lsh(big.NewInt(1), 200), + } { + reference, err := InactivityClaimSettlementReference(walletID, nonce) + if err != nil { + t.Fatalf("nonce [%s]: [%v]", nonce, err) + } + + decodedWalletID, decodedNonce, err := ParseInactivityClaimSettlementReference( + reference, + ) + if err != nil { + t.Fatalf("reference [%s]: [%v]", reference, err) + } + if !bytes.Equal(decodedWalletID, walletID) { + t.Errorf( + "reference [%s] round-tripped wallet [%x], expected [%x]", + reference, + decodedWalletID, + walletID, + ) + } + if decodedNonce.Cmp(nonce) != 0 { + t.Errorf( + "reference [%s] round-tripped nonce [%s], expected [%s]", + reference, + decodedNonce, + nonce, + ) + } + } +} + +func TestInactivityClaimSettlementReference_RejectsNonCanonicalIdentities( + t *testing.T, +) { + walletID := make([]byte, inactivityClaimWalletIDLength) + walletID[inactivityClaimWalletIDLength-1] = 0xab + + canonical, err := InactivityClaimSettlementReference( + walletID, + big.NewInt(12), + ) + if err != nil { + t.Fatal(err) + } + if _, _, err := ParseInactivityClaimSettlementReference( + canonical, + ); err != nil { + t.Fatalf("the canonical rendering [%s] is rejected: [%v]", canonical, err) + } + + shortWalletID := walletID[:inactivityClaimWalletIDLength-1] + if _, err := InactivityClaimSettlementReference( + shortWalletID, + big.NewInt(12), + ); err == nil { + t.Error("a short wallet identifier produced a claim identity") + } + if _, err := InactivityClaimSettlementReference(walletID, nil); err == nil { + t.Error("a missing nonce produced a claim identity") + } + if _, err := InactivityClaimSettlementReference( + walletID, + big.NewInt(-1), + ); err == nil { + t.Error("a negative nonce produced a claim identity") + } + + for name, reference := range map[string]string{ + "no separator": hex.EncodeToString(walletID) + "12", + "uppercase wallet": strings.ToUpper(hex.EncodeToString(walletID)) + ":12", + "prefixed wallet": "0x" + hex.EncodeToString(walletID) + ":12", + "truncated wallet": hex.EncodeToString(shortWalletID) + ":12", + "zero-padded nonce": hex.EncodeToString(walletID) + ":012", + "hexadecimal nonce": hex.EncodeToString(walletID) + ":0xc", + "signed nonce": hex.EncodeToString(walletID) + ":+12", + "negative nonce": hex.EncodeToString(walletID) + ":-12", + "empty nonce": hex.EncodeToString(walletID) + ":", + "trailing separator": canonical + ":", + "non-numeric nonce": hex.EncodeToString(walletID) + ":twelve", + "non-hexadecimal key": strings.Repeat("z", 64) + ":12", + } { + if _, _, err := ParseInactivityClaimSettlementReference( + reference, + ); err == nil { + t.Errorf( + "%s: alias [%s] was accepted as a canonical claim identity", + name, + reference, + ) + } + } +} + +// TestValidateTerminalOutcome_ChainSettlementIsPinnedPerCeremony asserts only a +// ceremony with a code path that submits to a chain may report a settlement, +// and only of the kind it actually dispatches. Without the restriction any +// ceremony could attach a penalty submission it could not have made. +func TestValidateTerminalOutcome_ChainSettlementIsPinnedPerCeremony( + t *testing.T, +) { + walletID := make([]byte, inactivityClaimWalletIDLength) + walletID[0] = 0x7f + reference, err := InactivityClaimSettlementReference( + walletID, + big.NewInt(3), + ) + if err != nil { + t.Fatal(err) + } + + for _, ceremony := range AllCeremonies() { + expectedKind, known := completedEvidenceKinds[ceremony] + if !known { + continue + } + + evidence := TerminalEvidence{ + Kind: expectedKind, + Contribution: testTranscriptContribution(ceremony, 1), + } + switch expectedKind { + case TerminalEvidencePersistedTBTCSinger, + TerminalEvidencePersistedBeaconSigner: + evidence.MembershipIndex = 1 + evidence.Reference = "persisted-signer-identity" + case TerminalEvidenceForwarderClosed: + default: + evidence.Reference = "durable-result-identity" + } + + dispatches, permitted := chainSettlementKinds[ceremony] + + settled := evidence + settled.ChainSettlement = &ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + Reference: reference, + } + err := ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeCompleted, + settled, + ) + if permitted && dispatches == ChainSettlementInactivityClaim { + if err != nil { + t.Errorf( + "ceremony [%s] rejected the settlement it dispatches: [%v]", + ceremony, + err, + ) + } + } else if err == nil { + t.Errorf( + "ceremony [%s] accepted an inactivity claim settlement it has "+ + "no code path to file", + ceremony, + ) + } + + unknownKind := evidence + unknownKind.ChainSettlement = &ChainSettlementRecord{ + Kind: "some_other_submission", + } + if err := ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeCompleted, + unknownKind, + ); err == nil { + t.Errorf( + "ceremony [%s] accepted an undeclared chain settlement kind", + ceremony, + ) + } + } +} + +func TestValidateTerminalOutcome_ChainSettlementShapes(t *testing.T) { + walletID := make([]byte, inactivityClaimWalletIDLength) + walletID[0] = 0x11 + reference, err := InactivityClaimSettlementReference( + walletID, + big.NewInt(9), + ) + if err != nil { + t.Fatal(err) + } + + heartbeatEvidence := func( + settlement *ChainSettlementRecord, + ) TerminalEvidence { + return TerminalEvidence{ + Kind: TerminalEvidenceProtocolResult, + Reference: "heartbeat-result-identity", + ChainSettlement: settlement, + Contribution: testTranscriptContribution(TBTCHeartbeat, 1), + } + } + + // A dispatch with no observed settlement is the deliberate unreconciled + // record; rejecting it would force the node to either fabricate a claim + // identity or hide the dispatch entirely. + if err := ValidateTerminalOutcome( + TBTCHeartbeat, + testWorkID(TBTCHeartbeat), + TerminalOutcomeCompleted, + heartbeatEvidence(&ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + }), + ); err != nil { + t.Errorf("an unobserved dispatch was rejected: [%v]", err) + } + + if err := ValidateTerminalOutcome( + TBTCHeartbeat, + testWorkID(TBTCHeartbeat), + TerminalOutcomeCompleted, + heartbeatEvidence(&ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + Reference: "not-a-claim-identity", + }), + ); err == nil { + t.Error("a settlement naming no canonical claim was accepted") + } + + // The dispatch runs downstream of the heartbeat's own threshold signature, + // so an outcome reporting no result cannot have reached one. + if err := ValidateTerminalOutcome( + TBTCHeartbeat, + testWorkID(TBTCHeartbeat), + TerminalOutcomeExhausted, + TerminalEvidence{ + Kind: TerminalEvidenceNoThreshold, + ChainSettlement: &ChainSettlementRecord{ + Kind: ChainSettlementInactivityClaim, + Reference: reference, + }, + }, + ); err == nil { + t.Error("an exhausted heartbeat reported a filed penalty") + } +} + +// TestValidateTerminalOutcome_TranscriptContributionIsRequired asserts a +// ceremony whose owner authenticates its peers cannot record a completed +// threshold result without saying which memberships produced it. +// +// Without this the reader of a terminal record is left where every earlier +// reading of a mixed-release ceremony was: every member of one ceremony records +// the same completion and the same result identity, so the record cannot +// distinguish shares that combined from several parties from a single party that +// recovered the common result — and the population has to come from whichever +// party wrote the report. +func TestValidateTerminalOutcome_TranscriptContributionIsRequired(t *testing.T) { + for ceremony, expectedKind := range completedEvidenceKinds { + _, authored := transcriptContributionCeremonies[ceremony] + + evidence := TerminalEvidence{Kind: expectedKind} + switch expectedKind { + case TerminalEvidencePersistedTBTCSinger, + TerminalEvidencePersistedBeaconSigner: + evidence.MembershipIndex = 1 + evidence.Reference = "persisted-signer-identity" + case TerminalEvidenceForwarderClosed: + default: + evidence.Reference = testCompletedResultReference( + t, + ceremony, + "durable-result-identity", + ) + } + + err := ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeCompleted, + evidence, + ) + if authored && err == nil { + t.Errorf( + "ceremony [%s] completed a threshold result without naming "+ + "the memberships that produced it", + ceremony, + ) + } + if !authored && err != nil { + t.Errorf( + "ceremony [%s] rejected its own completed evidence: [%v]", + ceremony, + err, + ) + } + + // The mirror: a ceremony that authenticates no peer population must not + // be able to attach one. A forwarder relays other members' shares and + // computes nothing, and a coordination proposal comes from one leader; + // letting either name a transcript would put a claim about other + // parties into a record that has no local view to support it. + evidence.Contribution = &TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 2}, + LocalMembers: []group.MemberIndex{1}, + PermitSpaceMembers: testPermitSpaceMembers( + ceremony, + []group.MemberIndex{1, 2}, + ), + } + err = ValidateTerminalOutcome( + ceremony, + testWorkID(ceremony), + TerminalOutcomeCompleted, + evidence, + ) + if !authored && err == nil { + t.Errorf( + "ceremony [%s] authored a transcript it cannot observe", + ceremony, + ) + } + if authored && err != nil { + t.Errorf( + "ceremony [%s] rejected its own transcript: [%v]", + ceremony, + err, + ) + } + } +} + +// TestValidateTerminalOutcome_TranscriptContributionShapes asserts the +// transcript is a well-formed set that places the recording node inside the +// population it describes. +func TestValidateTerminalOutcome_TranscriptContributionShapes(t *testing.T) { + signingEvidence := func( + contribution *TranscriptContribution, + ) TerminalEvidence { + return TerminalEvidence{ + Kind: TerminalEvidenceBitcoinTransaction, + Reference: "signed-transaction-hash", + Contribution: contribution, + } + } + + refused := map[string]*TranscriptContribution{ + "no incorporated membership": { + LocalMembers: []group.MemberIndex{1}, + }, + // An index of zero is no membership at all, and a reader that accepted + // it would count a party that cannot exist toward a population. + "a zero index": { + IncorporatedMembers: []group.MemberIndex{0, 2}, + LocalMembers: []group.MemberIndex{2}, + }, + // One encoding per set. Otherwise the same membership listed twice + // inflates a population, and two records of one transcript compare + // unequal for no reason a reader can see. + "a repeated membership": { + IncorporatedMembers: []group.MemberIndex{2, 2, 3}, + LocalMembers: []group.MemberIndex{2}, + }, + "an unordered set": { + IncorporatedMembers: []group.MemberIndex{3, 2}, + LocalMembers: []group.MemberIndex{2}, + }, + "unordered local memberships": { + IncorporatedMembers: []group.MemberIndex{1, 2, 3}, + LocalMembers: []group.MemberIndex{3, 1}, + }, + // The node has to be inside the transcript it authors. A record whose + // local membership is absent from the produced population is a report + // about other parties, which is the one thing this field must not be + // able to become. + "a local membership outside the population": { + IncorporatedMembers: []group.MemberIndex{1, 2}, + LocalMembers: []group.MemberIndex{4}, + }, + } + + for name, contribution := range refused { + if err := ValidateTerminalOutcome( + TBTCSigning, + testWorkID(TBTCSigning), + TerminalOutcomeCompleted, + signingEvidence(contribution), + ); err == nil { + t.Errorf("a transcript with %s was accepted", name) + } + } + + // A node operating several memberships of one group records them all: the + // memberships some other node supplied are what is left after removing + // them, so an omitted local membership would be attributed elsewhere. + if err := ValidateTerminalOutcome( + TBTCSigning, + testWorkID(TBTCSigning), + TerminalOutcomeCompleted, + signingEvidence(&TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 2, 3, 7}, + LocalMembers: []group.MemberIndex{2, 7}, + }), + ); err != nil { + t.Errorf("a node operating several memberships was refused: [%v]", err) + } + + // A wallet action owns its permit whether or not the attempt that produced + // the signature selected any of the memberships this node operates. The + // transcript it observed is still authenticated, and naming no local + // membership is the honest reading of it; refusing the record would leave a + // permit unresolved over work that demonstrably concluded. + if err := ValidateTerminalOutcome( + TBTCSigning, + testWorkID(TBTCSigning), + TerminalOutcomeCompleted, + signingEvidence(&TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 2, 3}, + }), + ); err != nil { + t.Errorf( + "a node that observed a result it did not sign was refused: [%v]", + err, + ) + } +} + +// TestValidateTerminalOutcome_TranscriptContributionBindsPersistedMembership +// asserts a persisted DKG membership and the transcript behind it describe one +// ceremony. A record naming a membership it does not claim to have operated +// joins a result produced in one ceremony to a signer persisted from another. +func TestValidateTerminalOutcome_TranscriptContributionBindsPersistedMembership( + t *testing.T, +) { + evidence := func(local group.MemberIndex) TerminalEvidence { + return TerminalEvidence{ + Kind: TerminalEvidencePersistedTBTCSinger, + Reference: "wallet-storage-key", + MembershipIndex: 4, + Contribution: &TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 4, 9}, + LocalMembers: []group.MemberIndex{local}, + PermitSpaceMembers: testPermitSpaceMembers( + TBTCDKG, + []group.MemberIndex{1, 4, 9}, + ), + }, + } + } + + if err := ValidateTerminalOutcome( + TBTCDKG, + testWorkID(TBTCDKG), + TerminalOutcomeCompleted, + evidence(4), + ); err != nil { + t.Errorf("a persisted membership inside its own transcript was refused: [%v]", err) + } + + if err := ValidateTerminalOutcome( + TBTCDKG, + testWorkID(TBTCDKG), + TerminalOutcomeCompleted, + evidence(9), + ); err == nil { + t.Error("a persisted membership the node never claimed to operate was accepted") + } +} + +// TestValidateTerminalOutcome_TranscriptContributionNeedsAResult asserts an +// ending that produced nothing cannot carry a transcript. Quarantine, exhaustion +// and the unresolved marker left no threshold result behind, so a population +// attached to one would describe a ceremony that did not conclude. +func TestValidateTerminalOutcome_TranscriptContributionNeedsAResult( + t *testing.T, +) { + contribution := &TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 2}, + LocalMembers: []group.MemberIndex{1}, + } + + if err := ValidateTerminalOutcome( + TBTCDKG, + testWorkID(TBTCDKG), + TerminalOutcomeQuarantined, + TerminalEvidence{ + Kind: TerminalEvidenceQuarantinedTBTCSinger, + Contribution: contribution, + }, + ); err == nil { + t.Error("a quarantined signer claimed a produced transcript") + } + + if err := ValidateTerminalOutcome( + TBTCSigning, + testWorkID(TBTCSigning), + TerminalOutcomeExhausted, + TerminalEvidence{ + Kind: TerminalEvidenceNoThreshold, + Contribution: contribution, + }, + ); err == nil { + t.Error("an exhausted ceremony claimed a produced transcript") + } +} + +// TestMemberIndexes_JSONRoundTrip asserts a transcript survives the journal as a +// membership list rather than as an opaque string. A member index is a byte, so +// the default encoding of a set of them is base64: the audit and the diagnostics +// scrape would both receive the one field that says who produced a result in a +// form they cannot join to a membership. +func TestMemberIndexes_JSONRoundTrip(t *testing.T) { + contribution := &TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 4, 255}, + LocalMembers: []group.MemberIndex{4}, + } + + encoded, err := json.Marshal(contribution) + if err != nil { + t.Fatal(err) + } + + expected := `{"incorporated_members":[1,4,255],"local_members":[4]}` + if string(encoded) != expected { + t.Errorf( + "unexpected transcript encoding\nactual: %s\nexpected: %s", + encoded, + expected, + ) + } + + decoded := &TranscriptContribution{} + if err := json.Unmarshal(encoded, decoded); err != nil { + t.Fatal(err) + } + if !decoded.Equal(contribution) { + t.Errorf("the transcript did not survive the journal: %+v", decoded) + } + + // A record read outside the node that wrote it can be corrupt or forged, so + // a value that is not a member index has to fail on the way in rather than + // become a truncated index a reader would treat as a real membership. + for _, invalid := range []string{ + `{"incorporated_members":[256],"local_members":[1]}`, + `{"incorporated_members":[0],"local_members":[1]}`, + `{"incorporated_members":[-1],"local_members":[1]}`, + `{"incorporated_members":"AQQJ","local_members":[1]}`, + } { + if err := json.Unmarshal( + []byte(invalid), + &TranscriptContribution{}, + ); err == nil { + t.Errorf("an invalid transcript was decoded: %s", invalid) + } + } +} + +// TestTranscriptContribution_EqualComparesTheTranscript asserts the record is +// compared by the transcript it names. It travels by pointer and carries +// slices, so the default comparison would separate one observation from the +// same observation reloaded from the journal — which is what the terminal +// recorder's idempotent retry rests on. +func TestTranscriptContribution_EqualComparesTheTranscript(t *testing.T) { + contribution := func() *TranscriptContribution { + return &TranscriptContribution{ + IncorporatedMembers: []group.MemberIndex{1, 2, 3}, + LocalMembers: []group.MemberIndex{2}, + } + } + + if !contribution().Equal(contribution()) { + t.Error("two records of one transcript compared unequal") + } + + var absent *TranscriptContribution + if absent.Equal(contribution()) || contribution().Equal(absent) { + t.Error("a named transcript compared equal to none at all") + } + if !absent.Equal(nil) { + t.Error("two records naming no transcript compared unequal") + } + + differing := contribution() + differing.IncorporatedMembers = []group.MemberIndex{1, 2, 4} + if contribution().Equal(differing) { + t.Error("two different populations compared equal") + } + + // The same population with a different local half is a different node's + // record of the same ceremony, and the recorder must not treat one as a + // retry of the other. + relabeled := contribution() + relabeled.LocalMembers = []group.MemberIndex{3} + if contribution().Equal(relabeled) { + t.Error("two nodes' records of one ceremony compared equal") + } +} diff --git a/pkg/protocol/participation/release.go b/pkg/protocol/participation/release.go new file mode 100644 index 0000000000..62ff87b7e1 --- /dev/null +++ b/pkg/protocol/participation/release.go @@ -0,0 +1,49 @@ +package participation + +// ReleaseEpoch identifies a compiled release artifact of the client with +// respect to the coordinated protocol cutover. It names the artifact, not the +// cryptographic mode of any particular ceremony: a single cutover-release +// process participates in legacy ceremonies before the cutover block and in +// security-v2 ceremonies canonically anchored at or after it. +type ReleaseEpoch uint8 + +const ( + // EpochSecurityV2Cutover identifies the single cutover release: one + // compiled binary carrying both the production-compatible legacy protocol + // behavior and the hardened security-v2 behavior, selecting between them + // per ceremony from the ceremony's canonical chain anchor. + EpochSecurityV2Cutover ReleaseEpoch = iota + 1 +) + +// CompiledEpoch is the release epoch of this artifact. It is changed only by a +// reviewed release commit and MUST NOT be selectable by an environment +// variable, configuration file, CLI flag, mutable image tag, or remote +// service. It is exported through client-info, diagnostics, and the startup +// log so fleet inventory can verify the exact artifact an instance runs. +const CompiledEpoch = EpochSecurityV2Cutover + +// String returns the canonical string form of the release epoch: exactly +// "security_v2_cutover" for the cutover release. Any other value renders as +// "unknown". +func (e ReleaseEpoch) String() string { + switch e { + case EpochSecurityV2Cutover: + return "security_v2_cutover" + default: + return "unknown" + } +} + +// MainnetCutoverBlock is the immutable mainnet cutover block C. A ceremony +// whose canonical chain anchor is below this Ethereum block participates with +// legacy cryptography for its entire lifetime; a ceremony anchored at or after +// it participates with security-v2 cryptography. +// +// The zero placeholder is a deliberate release blocker: mainnet schedule +// resolution fails until a reviewed release commit replaces it with the block +// height published in the operator notice. Like +// tbtc.DepositSweepEveryWindowActivationBlock, this is a release-baked +// constant that every operator must be running before the block is reached; on +// mainnet it cannot be overridden at runtime, and supplying an override at all +// is a startup error. +const MainnetCutoverBlock = uint64(0) // R1 release commit MUST replace diff --git a/pkg/protocol/participation/schedule.go b/pkg/protocol/participation/schedule.go new file mode 100644 index 0000000000..d276c7c8e6 --- /dev/null +++ b/pkg/protocol/participation/schedule.go @@ -0,0 +1,204 @@ +package participation + +import ( + "fmt" + + commonEthereum "github.com/keep-network/keep-common/pkg/chain/ethereum" +) + +// Schedule is the resolved one-value cutover schedule. The cutover block C is +// the only activation height: there is no drain block, stop block, or +// separately selected start block. Crossing C never cancels a permit, mutates +// its mode, or reinterprets persisted messages; it only changes the mode +// selected for ceremonies canonically anchored at or after it. +type Schedule struct { + // CutoverBlock is the cutover block C. Zero means the developer-only + // disabled schedule, in which every ceremony participates in legacy mode; + // zero is rejected during resolution for every production network. + CutoverBlock uint64 +} + +// State is the externally visible protocol participation state of the process. +// It is observability, not the mode selector for already-started work: the +// per-ceremony mode is pinned from the ceremony's canonical chain anchor at +// permit issuance and never changes afterwards. +type State uint8 + +// The State values double as the numeric gate-state metric mapping and must +// keep their exact order: 0=disabled, 1=open_legacy, 2=open_security_v2, +// 3=quiescing, 4=clock_unavailable. +const ( + // StateDisabled is the developer-only all-zero schedule: the gate issues + // legacy permits unconditionally. It is never accepted as production + // cutover evidence. + StateDisabled State = iota + // StateOpenLegacy means the current chain height is below the cutover + // block: new ceremonies begin in legacy mode. + StateOpenLegacy + // StateOpenSecurityV2 means the current chain height is at or above the + // cutover block: new ceremonies begin in security-v2 mode. + StateOpenSecurityV2 + // StateQuiescing means process quiescence began: no new permits are + // issued, existing permits run to natural completion, and penalty commits + // are refused. + StateQuiescing + // StateClockUnavailable means a synchronous chain-clock read failed: the + // gate refuses new work and has canceled all outstanding permits. + StateClockUnavailable +) + +// String returns the canonical string form of the participation state. +func (s State) String() string { + switch s { + case StateDisabled: + return "disabled" + case StateOpenLegacy: + return "open_legacy" + case StateOpenSecurityV2: + return "open_security_v2" + case StateQuiescing: + return "quiescing" + case StateClockUnavailable: + return "clock_unavailable" + default: + return "unknown" + } +} + +// Disabled returns true for the developer-only all-zero schedule. +func (s Schedule) Disabled() bool { + return s.CutoverBlock == 0 +} + +// ModeFor returns the protocol mode for a ceremony with the given canonical +// chain anchor: legacy below the cutover block, security-v2 at or above it. +// The disabled schedule always selects legacy. The result depends only on the +// anchor and the compiled schedule, never on the local current height, so a +// pre-cutover chain event confirmed or delivered after the cutover block still +// classifies as legacy. +func (s Schedule) ModeFor(canonicalStartBlock uint64) ProtocolMode { + if s.Disabled() || canonicalStartBlock < s.CutoverBlock { + return ModeLegacy + } + return ModeSecurityV2 +} + +// StateFor returns the open participation state derived from the given current +// chain height. Quiescence and clock failure are process conditions layered on +// top by the gate; they are not derivable from a height. +func (s Schedule) StateFor(currentBlock uint64) State { + if s.Disabled() { + return StateDisabled + } + if currentBlock < s.CutoverBlock { + return StateOpenLegacy + } + return StateOpenSecurityV2 +} + +// Config is the protocol participation configuration surface. On mainnet the +// cutover block is exclusively the compiled MainnetCutoverBlock and supplying +// any override — including an explicit zero — is a startup error. Testnet +// release rehearsals must supply a nonzero cutover block. Developer mode may +// use zero for the disabled schedule. +type Config struct { + // CutoverBlock is the non-mainnet cutover block override, supplied via the + // [protocolParticipation] configuration section or the + // --protocolParticipation.cutoverBlock flag. + CutoverBlock uint64 + + // CutoverBlockSet records whether the cutover block was explicitly + // supplied at all, via flag or configuration file. Mainnet rejection is + // keyed on this presence, not on the decoded numeric value, so an explicit + // zero is rejected too. It is populated by command wiring from flag/key + // presence and is deliberately not decodable from the configuration file + // itself. + CutoverBlockSet bool `mapstructure:"-"` +} + +// ResolveAndValidate resolves the cutover schedule for the given Ethereum +// network from the compiled mainnet constant and the supplied configuration, +// enforcing the per-network validation rules. It performs configuration-only +// checks and is intended to run at the beginning of client start, before the +// Ethereum connection is established. +func ResolveAndValidate( + network commonEthereum.Network, + config Config, +) (Schedule, error) { + return resolveAndValidate(network, config, MainnetCutoverBlock) +} + +// resolveAndValidate is the compiled-constant-injecting resolver, split out so +// tests can exercise both the zero placeholder rejection and the reviewed +// nonzero release behavior without editing the constant. +func resolveAndValidate( + network commonEthereum.Network, + config Config, + compiledMainnetCutoverBlock uint64, +) (Schedule, error) { + switch network { + case commonEthereum.Mainnet: + if config.CutoverBlockSet { + return Schedule{}, fmt.Errorf( + "the [protocolParticipation.cutoverBlock] setting is not "+ + "allowed on mainnet: the cutover block is a compiled "+ + "release constant; remove the setting (supplied value: "+ + "[%d])", + config.CutoverBlock, + ) + } + if compiledMainnetCutoverBlock == 0 { + return Schedule{}, fmt.Errorf( + "the compiled mainnet cutover block is the zero placeholder; " + + "this artifact is not a reviewed cutover release and " + + "must not participate on mainnet", + ) + } + if err := validateMetricProjectable( + compiledMainnetCutoverBlock, + ); err != nil { + return Schedule{}, err + } + return Schedule{CutoverBlock: compiledMainnetCutoverBlock}, nil + case commonEthereum.Sepolia: + if config.CutoverBlock == 0 { + return Schedule{}, fmt.Errorf( + "testnet requires a nonzero " + + "[protocolParticipation.cutoverBlock]: release " + + "rehearsals must supply the rehearsed cutover block", + ) + } + if err := validateMetricProjectable(config.CutoverBlock); err != nil { + return Schedule{}, err + } + return Schedule{CutoverBlock: config.CutoverBlock}, nil + case commonEthereum.Developer: + if err := validateMetricProjectable(config.CutoverBlock); err != nil { + return Schedule{}, err + } + return Schedule{CutoverBlock: config.CutoverBlock}, nil + default: + return Schedule{}, fmt.Errorf( + "cannot resolve the protocol participation schedule for the "+ + "unrecognized Ethereum network [%v]", + network, + ) + } +} + +// validateMetricProjectable rejects an Ethereum block height that cannot be +// represented exactly by the float64 metrics projection. Decisions always use +// uint64; this only guards the observability contract, under which every +// exported height gauge — the cutover block and the current block — must equal +// the decision value exactly. +func validateMetricProjectable(blockHeight uint64) error { + if blockHeight > maxSafeMetricInteger { + return fmt.Errorf( + "block height [%d] exceeds the maximum precisely projectable "+ + "metric value [%d]", + blockHeight, + maxSafeMetricInteger, + ) + } + return nil +} diff --git a/pkg/protocol/participation/schedule_test.go b/pkg/protocol/participation/schedule_test.go new file mode 100644 index 0000000000..0459e78ef3 --- /dev/null +++ b/pkg/protocol/participation/schedule_test.go @@ -0,0 +1,265 @@ +package participation + +import ( + "strings" + "testing" + + commonEthereum "github.com/keep-network/keep-common/pkg/chain/ethereum" +) + +func TestResolveAndValidate_MainnetRejectsOverride(t *testing.T) { + for _, value := range []uint64{0, 1, 124000} { + _, err := resolveAndValidate( + commonEthereum.Mainnet, + Config{CutoverBlock: value, CutoverBlockSet: true}, + 999, + ) + if err == nil { + t.Fatalf( + "expected mainnet to reject an explicit override of [%d]", + value, + ) + } + if !strings.Contains(err.Error(), "protocolParticipation.cutoverBlock") { + t.Errorf( + "override rejection must name the offending key, got: [%v]", + err, + ) + } + } +} + +func TestResolveAndValidate_MainnetRejectsZeroCompiled(t *testing.T) { + _, err := resolveAndValidate(commonEthereum.Mainnet, Config{}, 0) + if err == nil { + t.Fatal("expected the zero compiled placeholder to be rejected") + } +} + +func TestResolveAndValidate_MainnetPlaceholderIsStillZero(t *testing.T) { + // The public resolver uses the compiled MainnetCutoverBlock. While the + // placeholder is zero, mainnet resolution must fail; once the release + // commit bakes a nonzero C, it must succeed with exactly that value. + schedule, err := ResolveAndValidate(commonEthereum.Mainnet, Config{}) + if MainnetCutoverBlock == 0 { + if err == nil { + t.Fatal( + "mainnet resolution must fail while the compiled cutover " + + "block is the zero placeholder", + ) + } + } else { + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if schedule.CutoverBlock != MainnetCutoverBlock { + t.Errorf( + "expected schedule cutover block [%d], got [%d]", + MainnetCutoverBlock, + schedule.CutoverBlock, + ) + } + } +} + +func TestResolveAndValidate_MainnetUsesCompiledConstant(t *testing.T) { + schedule, err := resolveAndValidate(commonEthereum.Mainnet, Config{}, 12345) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if schedule.CutoverBlock != 12345 { + t.Errorf( + "expected cutover block [12345], got [%d]", + schedule.CutoverBlock, + ) + } +} + +func TestResolveAndValidate_MainnetRejectsUnprojectableCompiled(t *testing.T) { + _, err := resolveAndValidate( + commonEthereum.Mainnet, + Config{}, + maxSafeMetricInteger+1, + ) + if err == nil { + t.Fatal("expected an unprojectable compiled cutover block rejection") + } +} + +func TestResolveAndValidate_TestnetRejectsZero(t *testing.T) { + for _, config := range []Config{ + {}, + {CutoverBlock: 0, CutoverBlockSet: true}, + } { + _, err := resolveAndValidate(commonEthereum.Sepolia, config, 0) + if err == nil { + t.Fatal("expected testnet to reject a zero cutover block") + } + } +} + +func TestResolveAndValidate_TestnetAcceptsNonzero(t *testing.T) { + schedule, err := resolveAndValidate( + commonEthereum.Sepolia, + Config{CutoverBlock: 124000, CutoverBlockSet: true}, + 0, + ) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if schedule.CutoverBlock != 124000 { + t.Errorf( + "expected cutover block [124000], got [%d]", + schedule.CutoverBlock, + ) + } +} + +func TestResolveAndValidate_TestnetRejectsUnprojectable(t *testing.T) { + _, err := resolveAndValidate( + commonEthereum.Sepolia, + Config{CutoverBlock: maxSafeMetricInteger + 1, CutoverBlockSet: true}, + 0, + ) + if err == nil { + t.Fatal("expected an unprojectable cutover block rejection") + } +} + +func TestResolveAndValidate_DeveloperAcceptsZeroAndNonzero(t *testing.T) { + schedule, err := resolveAndValidate(commonEthereum.Developer, Config{}, 0) + if err != nil { + t.Fatalf("unexpected error for developer zero: [%v]", err) + } + if !schedule.Disabled() { + t.Error("expected the developer zero schedule to be disabled") + } + + schedule, err = resolveAndValidate( + commonEthereum.Developer, + Config{CutoverBlock: 42, CutoverBlockSet: true}, + 0, + ) + if err != nil { + t.Fatalf("unexpected error for developer nonzero: [%v]", err) + } + if schedule.CutoverBlock != 42 { + t.Errorf( + "expected cutover block [42], got [%d]", + schedule.CutoverBlock, + ) + } +} + +func TestResolveAndValidate_RejectsUnknownNetwork(t *testing.T) { + _, err := resolveAndValidate(commonEthereum.Unknown, Config{}, 999) + if err == nil { + t.Fatal("expected an unknown network rejection") + } +} + +func TestSchedule_ModeFor(t *testing.T) { + schedule := Schedule{CutoverBlock: 1000} + + for anchor, expected := range map[uint64]ProtocolMode{ + 0: ModeLegacy, + 1: ModeLegacy, + 999: ModeLegacy, + 1000: ModeSecurityV2, + 1001: ModeSecurityV2, + } { + if mode := schedule.ModeFor(anchor); mode != expected { + t.Errorf( + "anchor [%d]: expected mode [%s], got [%s]", + anchor, + expected, + mode, + ) + } + } + + disabled := Schedule{} + for _, anchor := range []uint64{0, 1, 1000000} { + if mode := disabled.ModeFor(anchor); mode != ModeLegacy { + t.Errorf( + "disabled schedule anchor [%d]: expected legacy, got [%s]", + anchor, + mode, + ) + } + } +} + +func TestSchedule_StateFor(t *testing.T) { + schedule := Schedule{CutoverBlock: 1000} + + for currentBlock, expected := range map[uint64]State{ + 0: StateOpenLegacy, + 999: StateOpenLegacy, + 1000: StateOpenSecurityV2, + 1001: StateOpenSecurityV2, + } { + if state := schedule.StateFor(currentBlock); state != expected { + t.Errorf( + "current block [%d]: expected state [%s], got [%s]", + currentBlock, + expected, + state, + ) + } + } + + if state := (Schedule{}).StateFor(1000000); state != StateDisabled { + t.Errorf("disabled schedule: expected disabled state, got [%s]", state) + } +} + +func TestState_StringAndMetricMapping(t *testing.T) { + // The numeric values are the gate-state metric contract: + // 0=disabled, 1=open_legacy, 2=open_security_v2, 3=quiescing, + // 4=clock_unavailable. + expected := map[State]struct { + text string + value uint8 + }{ + StateDisabled: {"disabled", 0}, + StateOpenLegacy: {"open_legacy", 1}, + StateOpenSecurityV2: {"open_security_v2", 2}, + StateQuiescing: {"quiescing", 3}, + StateClockUnavailable: {"clock_unavailable", 4}, + } + + for state, expectation := range expected { + if state.String() != expectation.text { + t.Errorf( + "expected state string [%s], got [%s]", + expectation.text, + state.String(), + ) + } + if uint8(state) != expectation.value { + t.Errorf( + "state [%s]: expected metric value [%d], got [%d]", + expectation.text, + expectation.value, + uint8(state), + ) + } + } + + if State(250).String() != "unknown" { + t.Error("expected an out-of-range state to render as unknown") + } +} + +func TestReleaseEpoch_String(t *testing.T) { + if CompiledEpoch.String() != "security_v2_cutover" { + t.Errorf( + "expected compiled epoch [security_v2_cutover], got [%s]", + CompiledEpoch.String(), + ) + } + if ReleaseEpoch(0).String() != "unknown" { + t.Error("expected an unrecognized epoch to render as unknown") + } +} diff --git a/pkg/protocol/state/sync_machine.go b/pkg/protocol/state/sync_machine.go index a6f2ec20ff..dee16eca5f 100644 --- a/pkg/protocol/state/sync_machine.go +++ b/pkg/protocol/state/sync_machine.go @@ -32,20 +32,30 @@ const syncReceiveBuffer = 128 // if some members expected to participate in the execution are inactive. type SyncMachine struct { logger log.StandardLogger + ctx context.Context channel net.BroadcastChannel blockCounter chain.BlockCounter initialState SyncState // first state from which execution starts } // NewSyncMachine returns a new protocol state machine. +// +// The context passed to NewSyncMachine must be active for the entire lifetime +// of the execution. Canceling it aborts the machine even while it is parked on +// a block wait — the start-block wait and the between-state delay waits are +// interruptible, so a stalled chain cannot hold a canceled execution hostage. +// Per-state work receives a context derived from it, and the machine returns +// the cancellation cause as its error. func NewSyncMachine( logger log.StandardLogger, + ctx context.Context, channel net.BroadcastChannel, blockCounter chain.BlockCounter, initialState SyncState, ) *SyncMachine { return &SyncMachine{ logger: logger, + ctx: ctx, channel: channel, blockCounter: blockCounter, initialState: initialState, @@ -61,7 +71,7 @@ func (sm *SyncMachine) Execute(startBlockHeight uint64) (SyncState, uint64, erro } currentState := sm.initialState - ctx, cancelCtx := context.WithCancel(context.Background()) + ctx, cancelCtx := context.WithCancel(sm.ctx) sm.channel.Recv(ctx, handler) sm.logger.Infof( @@ -69,10 +79,13 @@ func (sm *SyncMachine) Execute(startBlockHeight uint64) (SyncState, uint64, erro currentState.MemberIndex(), startBlockHeight, ) - err := sm.blockCounter.WaitForBlockHeight(startBlockHeight) + err := waitForBlockHeight(ctx, sm.blockCounter, startBlockHeight) if err != nil { cancelCtx() - return nil, 0, fmt.Errorf("failed to wait for the execution start block") + return nil, 0, fmt.Errorf( + "failed to wait for the execution start block: [%w]", + err, + ) } lastStateEndBlockHeight := startBlockHeight @@ -125,7 +138,7 @@ func (sm *SyncMachine) Execute(startBlockHeight uint64) (SyncState, uint64, erro } currentState = nextState - ctx, cancelCtx = context.WithCancel(context.Background()) + ctx, cancelCtx = context.WithCancel(sm.ctx) sm.channel.Recv(ctx, handler) blockWaiter, err = stateTransition( @@ -139,6 +152,15 @@ func (sm *SyncMachine) Execute(startBlockHeight uint64) (SyncState, uint64, erro cancelCtx() return nil, 0, err } + + case <-sm.ctx.Done(): + cancelCtx() + drainAbandonedWaiter(blockWaiter) + return nil, 0, fmt.Errorf( + "execution of state [%T] canceled: [%w]", + currentState, + context.Cause(sm.ctx), + ) } } } @@ -163,10 +185,10 @@ func stateTransition( // In that case, if the message is sent too early, it is lost given that the // syncReceiveBuffer has the retransmissions filtered out. initiateDelay := lastStateEndBlockHeight + currentState.DelayBlocks() - err := blockCounter.WaitForBlockHeight(initiateDelay) + err := waitForBlockHeight(ctx, blockCounter, initiateDelay) if err != nil { return nil, fmt.Errorf( - "failed to wait [%v] blocks entering state [%T]: [%v]", + "failed to wait [%v] blocks entering state [%T]: [%w]", currentState.DelayBlocks(), currentState, err, @@ -197,3 +219,37 @@ func stateTransition( return blockWaiter, nil } + +// waitForBlockHeight blocks until the given height is reached or the context +// ends, whichever happens first. A synchronous WaitForBlockHeight call would +// hold the machine hostage to a stalled chain even after its ceremony was +// canceled; interrupting the wait lets the caller observe the cancellation +// cause and run its recovery path instead. +func waitForBlockHeight( + ctx context.Context, + blockCounter chain.BlockCounter, + blockHeight uint64, +) error { + waiter, err := blockCounter.BlockHeightWaiter(blockHeight) + if err != nil { + return err + } + + select { + case <-waiter: + return nil + case <-ctx.Done(): + drainAbandonedWaiter(waiter) + return context.Cause(ctx) + } +} + +// drainAbandonedWaiter takes ownership of a block-height waiter whose consumer +// is walking away before the notification arrived. The block counter delivers +// exactly one notification per waiter with a blocking send on an unbuffered +// channel once the height is reached; simply abandoning the channel would park +// that sender goroutine forever. The drain goroutine performs the single +// receive so the eventual sender can terminate, and itself exits then. +func drainAbandonedWaiter(waiter <-chan uint64) { + go func() { <-waiter }() +} diff --git a/pkg/protocol/state/sync_machine_test.go b/pkg/protocol/state/sync_machine_test.go index 5a738d534f..bd74dd2786 100644 --- a/pkg/protocol/state/sync_machine_test.go +++ b/pkg/protocol/state/sync_machine_test.go @@ -2,9 +2,12 @@ package state import ( "context" + "errors" "fmt" "reflect" + "sync" "testing" + "time" "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/chain/local_v1" @@ -55,7 +58,13 @@ func TestSyncExecute(t *testing.T) { channel: channel, } - stateMachine := NewSyncMachine(&testutils.MockLogger{}, channel, blockCounter, initialState) + stateMachine := NewSyncMachine( + &testutils.MockLogger{}, + context.Background(), + channel, + blockCounter, + initialState, + ) finalState, endBlockHeight, err := stateMachine.Execute(1) if err != nil { @@ -94,6 +103,391 @@ func TestSyncExecute(t *testing.T) { } } +// TestSyncExecute_ContextCancellation proves canceling the machine's parent +// context aborts the execution between states and surfaces the cancellation +// cause instead of running the protocol to its final state. +func TestSyncExecute_ContextCancellation(t *testing.T) { + testLog = make(map[uint64][]string) + + localChain := local_v1.Connect(10, 5) + blockCounter, _ = localChain.BlockCounter() + provider := netLocal.Connect() + channel, err := provider.BroadcastChannelFor("cancellation_test") + if err != nil { + t.Fatal(err) + } + + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { + return &TestMessage{} + }) + + initialState := testSyncState1{ + memberIndex: group.MemberIndex(1), + channel: channel, + } + + cause := fmt.Errorf("cancellation cause") + ctx, cancel := context.WithCancelCause(context.Background()) + + stateMachine := NewSyncMachine( + &testutils.MockLogger{}, + ctx, + channel, + blockCounter, + initialState, + ) + + go func() { + blockCounter.WaitForBlockHeight(2) + cancel(cause) + }() + + finalState, _, err := stateMachine.Execute(1) + if finalState != nil { + t.Errorf("expected no final state, got [%v]", finalState) + } + if !errors.Is(err, cause) { + t.Errorf( + "expected the cancellation cause in the error chain, got [%v]", + err, + ) + } +} + +// TestSyncExecute_CancellationDuringStartWait proves canceling the machine +// while it is parked on the execution start-block wait — a chain that stalls +// before the ceremony begins — aborts the wait promptly with the cancellation +// cause instead of holding the machine until the start block arrives. +func TestSyncExecute_CancellationDuringStartWait(t *testing.T) { + localChain := local_v1.Connect(10, 5) + heldBlockCounter, _ := localChain.BlockCounter() + provider := netLocal.Connect() + channel, err := provider.BroadcastChannelFor("held_start_wait_test") + if err != nil { + t.Fatal(err) + } + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { + return &TestMessage{} + }) + + initialState := &testHeldWaitSyncState{ + memberIndex: group.MemberIndex(1), + onInitiate: func() { + t.Error("the initial state must not initiate before the start block") + }, + } + + cause := fmt.Errorf("held start wait cancellation cause") + ctx, cancel := context.WithCancelCause(context.Background()) + + stateMachine := NewSyncMachine( + &testutils.MockLogger{}, + ctx, + channel, + heldBlockCounter, + initialState, + ) + + go func() { + heldBlockCounter.WaitForBlockHeight(2) + cancel(cause) + }() + + // A start block the local counter cannot reach within the test keeps the + // machine parked on the initial wait when the cancellation arrives. + finalState, _, err := stateMachine.Execute(100000) + if finalState != nil { + t.Errorf("expected no final state, got [%v]", finalState) + } + if !errors.Is(err, cause) { + t.Errorf( + "expected the cancellation cause in the error chain, got [%v]", + err, + ) + } +} + +// TestSyncExecute_CancellationDuringTransitionDelayWait proves canceling the +// machine while it is parked on a between-state delay wait — after an earlier +// state already completed its work — aborts the held wait promptly with the +// cancellation cause and never initiates the stalled state. +func TestSyncExecute_CancellationDuringTransitionDelayWait(t *testing.T) { + localChain := local_v1.Connect(10, 5) + heldBlockCounter, _ := localChain.BlockCounter() + provider := netLocal.Connect() + channel, err := provider.BroadcastChannelFor("held_delay_wait_test") + if err != nil { + t.Fatal(err) + } + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { + return &TestMessage{} + }) + + // The second state's delay stalls the machine between states, after the + // first state finished; the cancellation must interrupt that held wait. + stalledState := &testHeldWaitSyncState{ + memberIndex: group.MemberIndex(1), + delayBlocks: 100000, + onInitiate: func() { + t.Error("the stalled state must not initiate during its delay wait") + }, + } + initialState := &testHeldWaitSyncState{ + memberIndex: group.MemberIndex(1), + activeBlocks: 1, + next: stalledState, + } + + cause := fmt.Errorf("held delay wait cancellation cause") + ctx, cancel := context.WithCancelCause(context.Background()) + + stateMachine := NewSyncMachine( + &testutils.MockLogger{}, + ctx, + channel, + heldBlockCounter, + initialState, + ) + + go func() { + heldBlockCounter.WaitForBlockHeight(4) + cancel(cause) + }() + + finalState, _, err := stateMachine.Execute(1) + if finalState != nil { + t.Errorf("expected no final state, got [%v]", finalState) + } + if !errors.Is(err, cause) { + t.Errorf( + "expected the cancellation cause in the error chain, got [%v]", + err, + ) + } +} + +// TestWaitForBlockHeight_CancellationReleasesEventualSender proves the +// canceled wait does not strand the block counter's delivery goroutine: after +// the wait is canceled and the requested height is later reached, the +// counter's blocking send completes instead of parking forever on the +// abandoned channel. +func TestWaitForBlockHeight_CancellationReleasesEventualSender(t *testing.T) { + counter := newManualBlockCounter(0) + + cause := fmt.Errorf("wait cancellation cause") + ctx, cancel := context.WithCancelCause(context.Background()) + + waitResult := make(chan error, 1) + go func() { + waitResult <- waitForBlockHeight(ctx, counter, 5) + }() + + // The waiter registers before the wait parks; canceling only afterwards + // deterministically hits a parked wait. + <-counter.waiterRegistered + cancel(cause) + + if err := <-waitResult; !errors.Is(err, cause) { + t.Fatalf( + "expected the cancellation cause in the error chain, got [%v]", + err, + ) + } + + // Reaching the height after the cancellation launches the counter's + // blocking send; it completes only if the abandoned waiter was drained. + counter.advanceTo(5) + + select { + case <-counter.sendCompleted: + case <-time.After(10 * time.Second): + t.Fatal( + "the block counter's sender remained blocked after the " + + "cancellation; the abandoned waiter was not drained", + ) + } +} + +// TestSyncExecute_CancellationReleasesStateEndWaiterSender proves aborting the +// machine while it is parked on a state's end-block waiter does not strand the +// block counter's delivery goroutine: once the end block is later reached, +// every launched sender completes. +func TestSyncExecute_CancellationReleasesStateEndWaiterSender(t *testing.T) { + counter := newManualBlockCounter(1) + + provider := netLocal.Connect() + channel, err := provider.BroadcastChannelFor("drained_end_waiter_test") + if err != nil { + t.Fatal(err) + } + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { + return &TestMessage{} + }) + + initialState := &testHeldWaitSyncState{ + memberIndex: group.MemberIndex(1), + activeBlocks: 4, + } + + cause := fmt.Errorf("end waiter cancellation cause") + ctx, cancel := context.WithCancelCause(context.Background()) + + stateMachine := NewSyncMachine( + &testutils.MockLogger{}, + ctx, + channel, + counter, + initialState, + ) + + execResult := make(chan error, 1) + go func() { + _, _, err := stateMachine.Execute(1) + execResult <- err + }() + + // Execution registers three waiters in order: the start wait, the + // zero-delay transition wait, and the state end-block waiter at block 5. + // The first two are satisfied immediately at height 1; only the third + // keeps the machine parked, so the cancellation abandons exactly it. + for i := 0; i < 3; i++ { + <-counter.waiterRegistered + } + cancel(cause) + + if err := <-execResult; !errors.Is(err, cause) { + t.Fatalf( + "expected the cancellation cause in the error chain, got [%v]", + err, + ) + } + + // The first two senders completed into the consumed waits; reaching block + // 5 launches the third. All three complete only if the machine drained + // the end-block waiter it abandoned on the cancellation. + counter.advanceTo(5) + + for i := 0; i < 3; i++ { + select { + case <-counter.sendCompleted: + case <-time.After(10 * time.Second): + t.Fatalf( + "sender [%d] remained blocked after the cancellation; the "+ + "abandoned end-block waiter was not drained", + i+1, + ) + } + } +} + +// manualBlockCounter is a deterministic chain.BlockCounter test double: its +// height moves only when the test advances it, and it reproduces the +// production waiter contract — exactly one blocking send on an unbuffered +// channel per registered waiter once the height is reached. Registration and +// send completion are observable so tests can order their steps and prove the +// eventual sender terminated, instead of relying on timing. +type manualBlockCounter struct { + mu sync.Mutex + height uint64 + waiters map[uint64][]chan uint64 + + waiterRegistered chan struct{} + sendCompleted chan struct{} +} + +func newManualBlockCounter(height uint64) *manualBlockCounter { + return &manualBlockCounter{ + height: height, + waiters: make(map[uint64][]chan uint64), + waiterRegistered: make(chan struct{}, 128), + sendCompleted: make(chan struct{}, 128), + } +} + +func (mbc *manualBlockCounter) WaitForBlockHeight(blockNumber uint64) error { + waiter, err := mbc.BlockHeightWaiter(blockNumber) + if err != nil { + return err + } + <-waiter + return nil +} + +func (mbc *manualBlockCounter) BlockHeightWaiter( + blockNumber uint64, +) (<-chan uint64, error) { + newWaiter := make(chan uint64) + + mbc.mu.Lock() + if blockNumber <= mbc.height { + go mbc.deliver(newWaiter, blockNumber) + } else { + mbc.waiters[blockNumber] = append(mbc.waiters[blockNumber], newWaiter) + } + mbc.mu.Unlock() + + mbc.waiterRegistered <- struct{}{} + + return newWaiter, nil +} + +func (mbc *manualBlockCounter) CurrentBlock() (uint64, error) { + mbc.mu.Lock() + defer mbc.mu.Unlock() + return mbc.height, nil +} + +func (mbc *manualBlockCounter) WatchBlocks(ctx context.Context) <-chan uint64 { + return make(chan uint64) +} + +// deliver performs the production-style blocking send and then records that +// the sender terminated. +func (mbc *manualBlockCounter) deliver(waiter chan uint64, height uint64) { + waiter <- height + mbc.sendCompleted <- struct{}{} +} + +// advanceTo moves the height forward and launches the production-style +// delivery goroutine for every waiter whose height was reached. +func (mbc *manualBlockCounter) advanceTo(height uint64) { + mbc.mu.Lock() + defer mbc.mu.Unlock() + + for h := mbc.height + 1; h <= height; h++ { + mbc.height = h + for _, waiter := range mbc.waiters[h] { + go mbc.deliver(waiter, h) + } + delete(mbc.waiters, h) + } +} + +// testHeldWaitSyncState is a minimal state for the held-wait cancellation +// tests: its block bounds are configurable, initiation is observable, and it +// hands over to a preset next state. +type testHeldWaitSyncState struct { + memberIndex group.MemberIndex + delayBlocks uint64 + activeBlocks uint64 + next SyncState + onInitiate func() +} + +func (ts *testHeldWaitSyncState) DelayBlocks() uint64 { return ts.delayBlocks } +func (ts *testHeldWaitSyncState) ActiveBlocks() uint64 { return ts.activeBlocks } +func (ts *testHeldWaitSyncState) Initiate(ctx context.Context) error { + if ts.onInitiate != nil { + ts.onInitiate() + } + return nil +} +func (ts *testHeldWaitSyncState) Receive(msg net.Message) error { return nil } +func (ts *testHeldWaitSyncState) Next() (SyncState, error) { return ts.next, nil } +func (ts *testHeldWaitSyncState) MemberIndex() group.MemberIndex { + return ts.memberIndex +} + func addToTestLog(testState SyncState, functionName string) { currentBlock, _ := blockCounter.CurrentBlock() testLog[currentBlock] = append( diff --git a/pkg/sortition/internal/local/chain.go b/pkg/sortition/internal/local/chain.go index 1c85c4e519..12784ca965 100644 --- a/pkg/sortition/internal/local/chain.go +++ b/pkg/sortition/internal/local/chain.go @@ -222,6 +222,12 @@ func (c *Chain) GetOperatorID( } func (c *Chain) SetCurrentTimestamp(currentTimestamp *big.Int) { + // currentTimestamp is read by canRestoreRewardEligibility under + // ineligibleForRewardsUntilMutex, so guard the write with the same mutex to + // avoid racing the monitoring goroutine. + c.ineligibleForRewardsUntilMutex.Lock() + defer c.ineligibleForRewardsUntilMutex.Unlock() + c.currentTimestamp = currentTimestamp } diff --git a/pkg/tbtc/audit.go b/pkg/tbtc/audit.go new file mode 100644 index 0000000000..2d0082fc8c --- /dev/null +++ b/pkg/tbtc/audit.go @@ -0,0 +1,71 @@ +package tbtc + +import ( + "encoding/hex" + "fmt" + + "github.com/ethereum/go-ethereum/crypto" + + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/internal/byteutils" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// SignerAuditRecord is the non-secret identity of one persisted wallet signer +// record, decoded for the offline participation state audit. +type SignerAuditRecord struct { + // WalletStorageKey identifies the wallet the signer belongs to. It is the + // directory name the wallet registry stores the record under, derived + // from the wallet public key. + WalletStorageKey string + // WalletID is the hex-encoded 32-byte ECDSA wallet ID derived from the + // record's wallet public key, computed the same way the on-chain wallet + // registry derives it. It lets the offline audit match a decoded record + // against chain reconciliation evidence without any chain access. + WalletID string + // WalletPublicKeyHash is the hex-encoded 20-byte Bitcoin public key hash + // of the record's wallet public key. + WalletPublicKeyHash string + // MemberIndex is the signer's index within the wallet signing group. + MemberIndex group.MemberIndex + // SigningGroupSize is the size of the wallet signing group the record + // carries. + SigningGroupSize int +} + +// DecodeSignerAuditRecord decodes a persisted wallet signer record exactly +// the way the registry's own loader does — the decode any release's active +// scan must survive — and returns only its non-secret identity fields. The +// private key share is decoded to prove the record parses in full but never +// leaves this function. +func DecodeSignerAuditRecord(recordBytes []byte) (*SignerAuditRecord, error) { + signer := &signer{} + if err := signer.Unmarshal(recordBytes); err != nil { + return nil, err + } + + walletPublicKey := signer.wallet.publicKey + + // The offline audit runs without a chain connection, so the wallet ID is + // derived locally: the keccak256 of the 64-byte chain-format public key, + // exactly the derivation the chain's CalculateWalletID performs. + x, err := byteutils.LeftPadTo32Bytes(walletPublicKey.X.Bytes()) + if err != nil { + return nil, fmt.Errorf("cannot derive the wallet ID: [%v]", err) + } + y, err := byteutils.LeftPadTo32Bytes(walletPublicKey.Y.Bytes()) + if err != nil { + return nil, fmt.Errorf("cannot derive the wallet ID: [%v]", err) + } + walletID := crypto.Keccak256Hash(append(x, y...)) + + walletPublicKeyHash := bitcoin.PublicKeyHash(walletPublicKey) + + return &SignerAuditRecord{ + WalletStorageKey: getWalletStorageKey(walletPublicKey), + WalletID: hex.EncodeToString(walletID[:]), + WalletPublicKeyHash: hex.EncodeToString(walletPublicKeyHash[:]), + MemberIndex: signer.signingGroupMemberIndex, + SigningGroupSize: len(signer.wallet.signingGroupOperators), + }, nil +} diff --git a/pkg/tbtc/audit_test.go b/pkg/tbtc/audit_test.go new file mode 100644 index 0000000000..e90867778e --- /dev/null +++ b/pkg/tbtc/audit_test.go @@ -0,0 +1,82 @@ +package tbtc + +import ( + "encoding/hex" + "testing" + + "github.com/keep-network/keep-core/pkg/bitcoin" +) + +// TestDecodeSignerAuditRecord proves the audit decode accepts exactly what +// the registry loader accepts and reports the identity the loader would use +// for its wallet cache, including a wallet ID byte-identical to the chain's +// own derivation. +func TestDecodeSignerAuditRecord(t *testing.T) { + signer := createMockSigner(t) + + signerBytes, err := signer.Marshal() + if err != nil { + t.Fatal(err) + } + + record, err := DecodeSignerAuditRecord(signerBytes) + if err != nil { + t.Fatalf("unexpected decode error: [%v]", err) + } + + expectedKey := getWalletStorageKey(signer.wallet.publicKey) + if record.WalletStorageKey != expectedKey { + t.Errorf( + "expected wallet storage key [%s], got [%s]", + expectedKey, + record.WalletStorageKey, + ) + } + + chainWalletID, err := Connect().CalculateWalletID(signer.wallet.publicKey) + if err != nil { + t.Fatal(err) + } + if expected := hex.EncodeToString( + chainWalletID[:], + ); record.WalletID != expected { + t.Errorf( + "expected wallet ID [%s], got [%s]", + expected, + record.WalletID, + ) + } + + walletPublicKeyHash := bitcoin.PublicKeyHash(signer.wallet.publicKey) + if expected := hex.EncodeToString( + walletPublicKeyHash[:], + ); record.WalletPublicKeyHash != expected { + t.Errorf( + "expected wallet public key hash [%s], got [%s]", + expected, + record.WalletPublicKeyHash, + ) + } + if record.MemberIndex != signer.signingGroupMemberIndex { + t.Errorf( + "expected member index [%d], got [%d]", + signer.signingGroupMemberIndex, + record.MemberIndex, + ) + } + if record.SigningGroupSize != len(signer.wallet.signingGroupOperators) { + t.Errorf( + "expected signing group size [%d], got [%d]", + len(signer.wallet.signingGroupOperators), + record.SigningGroupSize, + ) + } +} + +// TestDecodeSignerAuditRecord_RejectsUndecodableRecord proves a record the +// registry loader would reject is reported as an error, not misclassified. +func TestDecodeSignerAuditRecord_RejectsUndecodableRecord(t *testing.T) { + if _, err := DecodeSignerAuditRecord([]byte("not a signer record")); err == nil { + t.Error("expected a decode error for an undecodable record") + } +} diff --git a/pkg/tbtc/chain_test.go b/pkg/tbtc/chain_test.go index 365950caa9..ad45053807 100644 --- a/pkg/tbtc/chain_test.go +++ b/pkg/tbtc/chain_test.go @@ -72,6 +72,15 @@ type localChain struct { inactivityNonceMutex sync.Mutex inactivityNonces map[[32]byte]uint64 + // inactivityClaimMiner, when set, defers the on-chain effect of a + // submitted inactivity claim the way a real provider does: the submitting + // call returns once the transaction is accepted and the nonce advance and + // claim event happen only when the test mines it. Without it the chain + // settles inline, which hides every lifecycle race between a submission + // and the settlement it eventually produces. + inactivityClaimMinerMutex sync.Mutex + inactivityClaimMiner func(mine func()) + blocksByTimestampMutex sync.Mutex blocksByTimestamp map[uint64]uint64 @@ -632,19 +641,44 @@ func (lc *localChain) AssembleInactivityClaim( }, nil } +// inactivityClaimedHandlerCount reports how many inactivity claim +// subscriptions are currently installed. It lets a test assert that a +// submitted claim is still being listened for at a given moment, which is the +// difference between resolving a late settlement and never seeing it. +func (lc *localChain) inactivityClaimedHandlerCount() int { + lc.inactivityClaimedHandlersMutex.Lock() + defer lc.inactivityClaimedHandlersMutex.Unlock() + + return len(lc.inactivityClaimedHandlers) +} + +// setInactivityClaimMiner installs a hook that decides when a submitted +// inactivity claim takes effect. The hook receives the closure that advances +// the nonce and emits the claim event, so a test can hold a submission +// unmined, mine it later, or drop it entirely. +func (lc *localChain) setInactivityClaimMiner(miner func(mine func())) { + lc.inactivityClaimMinerMutex.Lock() + defer lc.inactivityClaimMinerMutex.Unlock() + + lc.inactivityClaimMiner = miner +} + func (lc *localChain) SubmitInactivityClaim( claim *InactivityClaim, nonce *big.Int, groupMembers []uint32, ) error { - lc.inactivityClaimedHandlersMutex.Lock() - defer lc.inactivityClaimedHandlersMutex.Unlock() + if err := func() error { + lc.inactivityNonceMutex.Lock() + defer lc.inactivityNonceMutex.Unlock() - lc.inactivityNonceMutex.Lock() - defer lc.inactivityNonceMutex.Unlock() + if nonce.Uint64() != lc.inactivityNonces[claim.WalletID] { + return fmt.Errorf("wrong inactivity claim nonce") + } - if nonce.Uint64() != lc.inactivityNonces[claim.WalletID] { - return fmt.Errorf("wrong inactivity claim nonce") + return nil + }(); err != nil { + return err } blockNumber, err := lc.blockCounter.CurrentBlock() @@ -652,16 +686,36 @@ func (lc *localChain) SubmitInactivityClaim( return fmt.Errorf("failed to get the current block") } - for _, handler := range lc.inactivityClaimedHandlers { - handler(&InactivityClaimedEvent{ - WalletID: claim.WalletID, - Nonce: nonce, - Notifier: "", - BlockNumber: blockNumber, - }) + // settle is everything the registry does once the transaction is included: + // consume the nonce and announce the claim. + settle := func() { + lc.inactivityNonceMutex.Lock() + lc.inactivityNonces[claim.WalletID]++ + lc.inactivityNonceMutex.Unlock() + + lc.inactivityClaimedHandlersMutex.Lock() + defer lc.inactivityClaimedHandlersMutex.Unlock() + + for _, handler := range lc.inactivityClaimedHandlers { + handler(&InactivityClaimedEvent{ + WalletID: claim.WalletID, + Nonce: nonce, + Notifier: "", + BlockNumber: blockNumber, + }) + } + } + + lc.inactivityClaimMinerMutex.Lock() + miner := lc.inactivityClaimMiner + lc.inactivityClaimMinerMutex.Unlock() + + if miner == nil { + settle() + return nil } - lc.inactivityNonces[claim.WalletID]++ + miner(settle) return nil } diff --git a/pkg/tbtc/coordination.go b/pkg/tbtc/coordination.go index 2dd75e9614..c4b1f58ee1 100644 --- a/pkg/tbtc/coordination.go +++ b/pkg/tbtc/coordination.go @@ -20,6 +20,7 @@ import ( "github.com/keep-network/keep-core/pkg/generator" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "golang.org/x/sync/semaphore" ) @@ -346,8 +347,12 @@ func (ce *coordinationExecutor) walletPublicKeyHash() [20]byte { } // coordinate executes the coordination procedure for the given coordination -// window. +// window. The given context bounds the procedure: it is the owning +// coordination permit's context, so a release-gate cancellation — clock +// failure or forced quiescence — ends the procedure like the active phase end +// does. func (ce *coordinationExecutor) coordinate( + ctx context.Context, window *coordinationWindow, ) (*coordinationResult, error) { if lockAcquired := ce.lock.TryAcquire(1); !lockAcquired { @@ -410,7 +415,7 @@ func (ce *coordinationExecutor) coordinate( // The coordination follower cancels the context as soon as it receives // the coordination message. ctx, cancelCtx := withCancelOnBlock( - context.Background(), + ctx, window.activePhaseEndBlock(), ce.waitForBlockFn, ) @@ -432,7 +437,10 @@ func (ce *coordinationExecutor) coordinate( // occur anyway. cancelCtx() coordinationFailed = true - if ce.metricsRecorder != nil { + // A gate-canceled permit ended the routine; that is not an + // ordinary coordination failure of this node. + if ce.metricsRecorder != nil && + !participation.IsGateRefusal(context.Cause(ctx)) { ce.metricsRecorder.IncrementCounter(clientinfo.MetricCoordinationFailedTotal, 1) } return nil, fmt.Errorf( @@ -457,8 +465,11 @@ func (ce *coordinationExecutor) coordinate( if err != nil { coordinationFailed = true // Record as leader timeout observation, not as a failure of this node. - // The actual failure is on the leader's side. - if ce.metricsRecorder != nil { + // The actual failure is on the leader's side. A gate-canceled + // permit ended the routine locally, so it is no observation about + // the leader either. + if ce.metricsRecorder != nil && + !participation.IsGateRefusal(context.Cause(ctx)) { ce.metricsRecorder.IncrementCounter(clientinfo.MetricCoordinationLeaderTimeoutTotal, 1) } // Return a partial result with leader and faults information diff --git a/pkg/tbtc/coordination_byzantine_test.go b/pkg/tbtc/coordination_byzantine_test.go new file mode 100644 index 0000000000..416893f2a2 --- /dev/null +++ b/pkg/tbtc/coordination_byzantine_test.go @@ -0,0 +1,349 @@ +package tbtc + +// Tier-2 Byzantine coordination harness. tBTC's interceptable Byzantine surface +// is the wallet coordination procedure: a per-window leader broadcasts an action +// proposal and followers receive + validate it. Unlike DKG/signing (one shared +// channel, sender-attributed strategies), coordination uses PER-OPERATOR +// channels, so Byzantine behavior is expressed by wrapping a specific operator's +// outbound channel with an interception.Strategy. +// +// This harness lives in package tbtc (not a pkg/internal/* package like dkgtest +// or signingtest) because the coordination machinery - newCoordinationExecutor, +// coordinate, wallet, coordinationWindow, the local chain - is all unexported. +// Exporting it purely for tests would widen the production API for no runtime +// benefit; a test-only helper here is reusable by any test in the package, which +// is the maximal reuse Go visibility allows. It generalizes the setup of +// TestCoordinationExecutor_Coordinate, adding per-operator strategy injection. + +import ( + "context" + "encoding/hex" + "math/big" + "slices" + "testing" + "time" + + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/generator" + "github.com/keep-network/keep-core/pkg/internal/interception" + "github.com/keep-network/keep-core/pkg/net" + netlocal "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// byzantineCoordinationReport is one operator's outcome from a coordination run. +type byzantineCoordinationReport struct { + operatorIndex int + address chain.Address + result *coordinationResult + err error +} + +// dropAll is a channel-level Byzantine strategy: it drops every message the +// operator tries to send. Applied to a leader's channel, it models a leader that +// generates a proposal but never broadcasts it (silent/withholding leader). +func dropAll(interception.Outbound) []net.TaggedMarshaler { return nil } + +// runByzantineCoordination sets up the canonical 3-operator / 10-seat redemption +// coordination scenario (deterministic operator keys => stable leader selection, +// matching TestCoordinationExecutor_Coordinate) and runs coordinate() for each +// operator concurrently. The outbound channel of operator i (1-based) is wrapped +// with strategies[i]; operators absent from the map get interception.PassThrough. +// channelName isolates this run in the process-global local broadcast registry. +func runByzantineCoordination( + t *testing.T, + channelName string, + blockTime time.Duration, + strategies map[int]interception.Strategy, +) []*byzantineCoordinationReport { + publicKeyHex, err := hex.DecodeString( + "0471e30bca60f6548d7b42582a478ea37ada63b402af7b3ddd57f0c95bb6843175" + + "aa0d2053a91a050a6797d85c38f2909cb7027f2344a01986aa2f9f8ca7a0c289", + ) + if err != nil { + t.Fatal(err) + } + + var publicKeyHash [20]byte + buffer, err := hex.DecodeString("aa768412ceed10bd423c025542ca90071f9fb62d") + if err != nil { + t.Fatal(err) + } + copy(publicKeyHash[:], buffer) + + parseScript := func(script string) bitcoin.Script { + parsed, err := hex.DecodeString(script) + if err != nil { + t.Fatal(err) + } + return parsed + } + + coordinationBlock := uint64(900) + + type operatorFixture struct { + chain Chain + address chain.Address + channel net.BroadcastChannel + waitForBlockHeight func(ctx context.Context, blockHeight uint64) error + } + + generateOperator := func(index int, privateKey int64) *operatorFixture { + // Deterministic addresses so leader selection is stable across runs. + privateKeyBigInt := big.NewInt(privateKey) + x, y := local_v1.DefaultCurve.ScalarBaseMult(privateKeyBigInt.Bytes()) + + localChain := ConnectWithKey( + &operator.PrivateKey{ + PublicKey: operator.PublicKey{ + Curve: operator.Secp256k1, + X: x, + Y: y, + }, + D: privateKeyBigInt, + }, + blockTime, + ) + + localChain.setBlockHashByNumber( + coordinationBlock-32, + "1422996cbcbc38fc924a46f4df5f9064279d3ab43396e58386dac9b87440d64f", + ) + + operatorAddress, err := localChain.operatorAddress() + if err != nil { + t.Fatal(err) + } + + _, operatorPublicKey, err := localChain.OperatorKeyPair() + if err != nil { + t.Fatal(err) + } + + strategy := interception.PassThrough + if s, ok := strategies[index]; ok { + strategy = s + } + + // Wrap this operator's outbound channel with its Byzantine strategy. + broadcastChannel, err := interception.NewNetworkWithStrategy( + netlocal.ConnectWithKey(operatorPublicKey), + strategy, + ).BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + + broadcastChannel.SetUnmarshaler(func() net.TaggedUnmarshaler { + return &coordinationMessage{} + }) + + waitForBlockHeight := func(ctx context.Context, blockHeight uint64) error { + blockCounter, err := localChain.BlockCounter() + if err != nil { + return err + } + wait, err := blockCounter.BlockHeightWaiter(blockHeight) + if err != nil { + return err + } + select { + case <-wait: + case <-ctx.Done(): + } + return nil + } + + return &operatorFixture{ + chain: localChain, + address: operatorAddress, + channel: broadcastChannel, + waitForBlockHeight: waitForBlockHeight, + } + } + + operator1 := generateOperator(1, 1) + operator2 := generateOperator(2, 2) + operator3 := generateOperator(3, 3) + + coordinatedWallet := wallet{ + publicKey: mustUnmarshalPublicKey(t, publicKeyHex), + signingGroupOperators: []chain.Address{ + operator2.address, + operator3.address, + operator1.address, + operator1.address, + operator3.address, + operator2.address, + operator2.address, + operator3.address, + operator1.address, + operator1.address, + }, + } + + proposalGenerator := newMockCoordinationProposalGenerator( + func( + walletPublicKeyHash [20]byte, + actionsChecklist []WalletActionType, + _ uint, + ) (CoordinationProposal, error) { + for _, action := range actionsChecklist { + if walletPublicKeyHash == publicKeyHash && action == ActionRedemption { + return &RedemptionProposal{ + RedeemersOutputScripts: []bitcoin.Script{ + parseScript("00148db50eb52063ea9d98b3eac91489a90f738986f6"), + parseScript("76a9148db50eb52063ea9d98b3eac91489a90f738986f688ac"), + }, + RedemptionTxFee: big.NewInt(10000), + }, nil + } + } + return &NoopProposal{}, nil + }, + ) + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + coordinatedWallet.signingGroupOperators, + Connect().Signing(), + ) + + protocolLatch := generator.NewProtocolLatch() + + generateExecutor := func(op *operatorFixture) *coordinationExecutor { + return newCoordinationExecutor( + op.chain, + coordinatedWallet, + coordinatedWallet.membersByOperator(op.address), + op.address, + proposalGenerator, + op.channel, + membershipValidator, + protocolLatch, + op.waitForBlockHeight, + ) + } + + window := newCoordinationWindow(coordinationBlock) + + reportChan := make(chan *byzantineCoordinationReport, 3) + + for i, op := range []*operatorFixture{operator1, operator2, operator3} { + go func(operatorIndex int, op *operatorFixture) { + result, err := generateExecutor(op).coordinate(context.Background(), window) + reportChan <- &byzantineCoordinationReport{ + operatorIndex: operatorIndex, + address: op.address, + result: result, + err: err, + } + }(i+1, op) + } + + reports := make([]*byzantineCoordinationReport, 0, 3) + for len(reports) < 3 { + reports = append(reports, <-reportChan) + } + slices.SortFunc(reports, func(a, b *byzantineCoordinationReport) int { + return a.operatorIndex - b.operatorIndex + }) + + return reports +} + +// TestByzantineCoordination_HonestBaseline runs the harness with no Byzantine +// strategy and confirms it reproduces the known-good coordination outcome: +// operator 2 is leader, all three operators agree on the redemption proposal, +// none errors. This is the equivalence proof that the interception seam (with +// PassThrough) does not perturb the protocol. +func TestByzantineCoordination_HonestBaseline(t *testing.T) { + reports := runByzantineCoordination(t, t.Name(), 100*time.Millisecond, nil) + + testutils.AssertIntsEqual(t, "reports count", 3, len(reports)) + + leader := reports[1].address // operator 2 is the expected leader + + for _, r := range reports { + if r.err != nil { + t.Errorf("operator %d errored: %v", r.operatorIndex, r.err) + continue + } + if r.result == nil { + t.Errorf("operator %d produced a nil result", r.operatorIndex) + continue + } + if r.result.leader != leader { + t.Errorf( + "operator %d saw leader %s; want %s", + r.operatorIndex, r.result.leader, leader, + ) + } + if r.result.proposal.ActionType() != ActionRedemption { + t.Errorf( + "operator %d coordinated action %v; want %v", + r.operatorIndex, r.result.proposal.ActionType(), ActionRedemption, + ) + } + if len(r.result.faults) != 0 { + t.Errorf("operator %d observed faults: %v", r.operatorIndex, r.result.faults) + } + } +} + +// TestByzantineCoordination_WithholdingLeader applies a drop-all strategy to the +// leader's (operator 2's) outbound channel: the leader generates a proposal but +// never broadcasts it. The safety invariant under test is that a silent leader +// causes a denial of service (followers coordinate NO action) but can never make +// followers act on a proposal they did not receive, and cannot split them onto +// divergent outcomes. Fast blocks bound the follower timeout (active phase ends +// at coordinationBlock+80). +func TestByzantineCoordination_WithholdingLeader(t *testing.T) { + reports := runByzantineCoordination( + t, + t.Name(), + 5*time.Millisecond, + map[int]interception.Strategy{2: dropAll}, // operator 2 is the leader + ) + + testutils.AssertIntsEqual(t, "reports count", 3, len(reports)) + + leaderReport := reports[1] // operator 2 + follower1 := reports[0] // operator 1 + follower3 := reports[2] // operator 3 + + // The leader generated its proposal locally and believes it broadcast it. + if leaderReport.err != nil { + t.Errorf("leader (operator 2) errored: %v", leaderReport.err) + } + + // Each follower fails to receive the withheld proposal and coordinates NO + // action - a denial of service, not an unauthorized action. + for _, f := range []*byzantineCoordinationReport{follower1, follower3} { + if f.err == nil { + t.Errorf("follower %d unexpectedly succeeded with a withholding leader", f.operatorIndex) + } + if f.result == nil { + t.Errorf("follower %d produced no result", f.operatorIndex) + continue + } + if f.result.proposal != nil { + t.Errorf( + "SAFETY VIOLATION: follower %d acted on a proposal (%v) it never received", + f.operatorIndex, f.result.proposal.ActionType(), + ) + } + } + + // No split-brain: both followers reached the same (no-proposal) outcome. + if (follower1.result.proposal == nil) != (follower3.result.proposal == nil) { + t.Errorf("followers diverged: f1.proposal=%v f3.proposal=%v", + follower1.result.proposal, follower3.result.proposal) + } + + t.Logf("withholding leader: followers coordinated no action (DoS); leader err=%v", leaderReport.err) +} diff --git a/pkg/tbtc/coordination_test.go b/pkg/tbtc/coordination_test.go index c597048fb0..04696d75bf 100644 --- a/pkg/tbtc/coordination_test.go +++ b/pkg/tbtc/coordination_test.go @@ -359,7 +359,7 @@ func TestCoordinationExecutor_Coordinate(t *testing.T) { operator3, } { go func(operatorIndex int, operator *operatorFixture) { - result, err := generateExecutor(operator).coordinate(window) + result, err := generateExecutor(operator).coordinate(context.Background(), window) reportChan <- &report{ operatorIndex: operatorIndex, diff --git a/pkg/tbtc/coordination_window_metrics_test.go b/pkg/tbtc/coordination_window_metrics_test.go index 274613f765..f1cbc06f4f 100644 --- a/pkg/tbtc/coordination_window_metrics_test.go +++ b/pkg/tbtc/coordination_window_metrics_test.go @@ -1,6 +1,7 @@ package tbtc import ( + "encoding/hex" "fmt" "sync" "testing" @@ -9,6 +10,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) // noopMetrics satisfies clientinfo.PerformanceMetricsRecorder with no side effects. @@ -346,3 +348,100 @@ func TestCoordinationWindowMetrics_Concurrent(t *testing.T) { _ = cwm.GetSummary() _ = cwm.GetRecentWindows(5) } + +// TestRecordCoordinationOutcome_GateAbortLeavesWindowMetricsUntouched proves +// a gate-aborted coordination procedure changes no coordination-window +// accounting — no coordinated, failed, or per-wallet entry — while an +// ordinary coordination failure and an ordinary success of the same wallet +// still reach the window's failure and success views. +func TestRecordCoordinationOutcome_GateAbortLeavesWindowMetricsUntouched( + t *testing.T, +) { + walletPublicKeyBytes, err := hex.DecodeString( + "0471e30bca60f6548d7b42582a478ea37ada63b402af7b3ddd57f0c95bb6843175" + + "aa0d2053a91a050a6797d85c38f2909cb7027f2344a01986aa2f9f8ca7a0c289", + ) + if err != nil { + t.Fatal(err) + } + walletPublicKey := mustUnmarshalPublicKey(t, walletPublicKeyBytes) + + tracker := newCoordinationWindowMetrics(nil, 10) + n := &node{windowMetricsTracker: tracker} + + window := newCoordinationWindow(900) + + recordCoordinationOutcome( + n, + window, + walletPublicKey, + nil, + time.Second, + fmt.Errorf("coordination canceled: %w", participation.ErrQuiescing), + true, + ) + + if _, exists := tracker.GetWindowMetrics(window.index()); exists { + t.Fatal("expected no window accounting after a gate abort") + } + + recordCoordinationOutcome( + n, + window, + walletPublicKey, + nil, + time.Second, + fmt.Errorf("ordinary coordination failure"), + false, + ) + + windowMetrics, exists := tracker.GetWindowMetrics(window.index()) + if !exists { + t.Fatal("expected window accounting after an ordinary failure") + } + testutils.AssertUintsEqual( + t, + "coordinated wallets after the ordinary failure", + 1, + windowMetrics.WalletsCoordinated, + ) + testutils.AssertUintsEqual( + t, + "failed wallets after the ordinary failure", + 1, + windowMetrics.WalletsFailed, + ) + + recordCoordinationOutcome( + n, + window, + walletPublicKey, + &coordinationResult{leader: chain.Address("0xAA")}, + time.Second, + nil, + false, + ) + + windowMetrics, exists = tracker.GetWindowMetrics(window.index()) + if !exists { + t.Fatal("expected window accounting after a success") + } + testutils.AssertUintsEqual( + t, + "coordinated wallets after the success", + 2, + windowMetrics.WalletsCoordinated, + ) + testutils.AssertUintsEqual( + t, + "failed wallets after the success", + 1, + windowMetrics.WalletsFailed, + ) + testutils.AssertUintsEqual( + t, + "successful wallets after the success", + 1, + windowMetrics.WalletsSuccessful, + ) +} diff --git a/pkg/tbtc/cutover_observer.go b/pkg/tbtc/cutover_observer.go new file mode 100644 index 0000000000..02de860aeb --- /dev/null +++ b/pkg/tbtc/cutover_observer.go @@ -0,0 +1,103 @@ +package tbtc + +import ( + "golang.org/x/time/rate" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// announcerMismatchMetrics is the minimal metrics sink used when handling an +// announcer session-ID mismatch. It is satisfied by the client-info performance +// metrics recorder. +type announcerMismatchMetrics interface { + IncrementCounter(name string, value float64) +} + +// announcerMismatchLogger is the minimal logging sink used when handling an +// announcer session-ID mismatch. It is satisfied by *zap.SugaredLogger. +type announcerMismatchLogger interface { + Infof(format string, args ...interface{}) +} + +type participationGateStateReader interface { + State() participation.Snapshot +} + +func currentParticipationGateState(gate participationGateStateReader) string { + if gate == nil { + return "unknown" + } + + return gate.State().State.String() +} + +// handleAnnouncerSessionMismatch centralizes the node-local response to a +// membership-valid, protocol-matched announcement whose session ID differs from +// the local one. It is invoked once per mismatching sender per Announce call +// (the announcer deduplicates) and: +// +// - increments the session-ID mismatch counter for every unequal ID, plus the +// cross-format counter when the difference is legacy<->hardened; +// - attributes the sighting to an operator address (mapping the 1-based sender +// member index through operatorAddresses with an explicit bounds check) and +// records it in the node-local cutover roster, which itself keeps only +// genuine post-cutover legacy stragglers; and +// - emits a rate-limited INFO log that carries only the classified formats, +// never a raw session ID. +// +// Any of metrics, roster, logger, or logLimiter may be nil; each is guarded +// independently so the handler is safe on a client-info-disabled node. +func handleAnnouncerSessionMismatch( + logger announcerMismatchLogger, + logLimiter *rate.Limiter, + metrics announcerMismatchMetrics, + roster *participation.CutoverPeerRoster, + currentMode participation.ProtocolMode, + gateState string, + operatorAddresses chain.Addresses, + protocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, +) { + // Every unequal, membership-valid announcement is a mismatch; only a + // legacy<->hardened difference is a cross-format peer. + if metrics != nil { + metrics.IncrementCounter(clientinfo.MetricAnnouncerSessionIDMismatchTotal, 1) + if announcer.IsCrossFormatMismatch(expectedFormat, observedFormat) { + metrics.IncrementCounter(clientinfo.MetricAnnouncerCrossFormatPeerTotal, 1) + } + } + + // Attribute the sighting to an operator address and record it. The roster + // itself filters to genuine post-cutover legacy stragglers (a security-v2 + // permit observing a legacy peer). + if roster != nil && sender >= 1 && int(sender) <= len(operatorAddresses) { + roster.ObserveLegacy( + protocolID, + sender, + operatorAddresses[sender-1], + currentMode, + expectedFormat, + observedFormat, + ) + } + + if logger != nil && (logLimiter == nil || logLimiter.Allow()) { + logger.Infof( + "protocol announcement rejected: session ID mismatch "+ + "[protocol=%s] [member=%d] [expectedFormat=%s] "+ + "[observedFormat=%s] [permitMode=%s] [gateState=%s]", + protocolID, + sender, + expectedFormat, + observedFormat, + currentMode, + gateState, + ) + } +} diff --git a/pkg/tbtc/cutover_observer_test.go b/pkg/tbtc/cutover_observer_test.go new file mode 100644 index 0000000000..fba9656d6f --- /dev/null +++ b/pkg/tbtc/cutover_observer_test.go @@ -0,0 +1,322 @@ +package tbtc + +import ( + "context" + "fmt" + "strings" + "sync" + "testing" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// cutoverFakeBlockCounter is a minimal chain.BlockCounter returning a fixed +// height. +type cutoverFakeBlockCounter struct { + block uint64 +} + +func (f *cutoverFakeBlockCounter) CurrentBlock() (uint64, error) { return f.block, nil } +func (f *cutoverFakeBlockCounter) WaitForBlockHeight(uint64) error { return nil } +func (f *cutoverFakeBlockCounter) BlockHeightWaiter(uint64) (<-chan uint64, error) { + c := make(chan uint64, 1) + close(c) + return c, nil +} +func (f *cutoverFakeBlockCounter) WatchBlocks(ctx context.Context) <-chan uint64 { + c := make(chan uint64) + go func() { + <-ctx.Done() + close(c) + }() + return c +} + +// cutoverFakeMetrics records counter/gauge writes. It satisfies both the +// announcer mismatch metrics sink and the roster metrics recorder. +type cutoverFakeMetrics struct { + mu sync.Mutex + counters map[string]float64 + gauges map[string]float64 +} + +func newCutoverFakeMetrics() *cutoverFakeMetrics { + return &cutoverFakeMetrics{ + counters: make(map[string]float64), + gauges: make(map[string]float64), + } +} + +func (m *cutoverFakeMetrics) IncrementCounter(name string, value float64) { + m.mu.Lock() + defer m.mu.Unlock() + m.counters[name] += value +} + +func (m *cutoverFakeMetrics) SetGauge(name string, value float64) { + m.mu.Lock() + defer m.mu.Unlock() + m.gauges[name] = value +} + +func (m *cutoverFakeMetrics) counter(name string) float64 { + m.mu.Lock() + defer m.mu.Unlock() + return m.counters[name] +} + +// captureLogger records formatted log lines. +type captureLogger struct { + mu sync.Mutex + lines []string +} + +func (l *captureLogger) Infof(format string, args ...interface{}) { + l.mu.Lock() + defer l.mu.Unlock() + l.lines = append(l.lines, fmt.Sprintf(format, args...)) +} + +func (l *captureLogger) all() []string { + l.mu.Lock() + defer l.mu.Unlock() + out := make([]string, len(l.lines)) + copy(out, l.lines) + return out +} + +type fixedParticipationGateState struct { + state participation.State +} + +func (f fixedParticipationGateState) State() participation.Snapshot { + return participation.Snapshot{State: f.state} +} + +func TestCurrentParticipationGateState(t *testing.T) { + tests := map[string]struct { + gate participationGateStateReader + want string + }{ + "missing gate": { + want: "unknown", + }, + "open security v2": { + gate: fixedParticipationGateState{ + state: participation.StateOpenSecurityV2, + }, + want: "open_security_v2", + }, + "quiescing": { + gate: fixedParticipationGateState{ + state: participation.StateQuiescing, + }, + want: "quiescing", + }, + "clock unavailable": { + gate: fixedParticipationGateState{ + state: participation.StateClockUnavailable, + }, + want: "clock_unavailable", + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + if got := currentParticipationGateState(test.gate); got != test.want { + t.Errorf("expected gate state [%s], got [%s]", test.want, got) + } + }) + } +} + +// operatorAddrs is a set of three valid, distinct operator addresses. +var operatorAddrs = chain.Addresses{ + chain.Address("0x1111111111111111111111111111111111111111"), + chain.Address("0x2222222222222222222222222222222222222222"), + chain.Address("0x3333333333333333333333333333333333333333"), +} + +func newTestRoster( + t *testing.T, + metrics participation.CutoverRosterMetricsRecorder, + block uint64, +) *participation.CutoverPeerRoster { + t.Helper() + roster, err := participation.NewCutoverPeerRoster( + context.Background(), + &cutoverFakeBlockCounter{block: block}, + 1500, + metrics, + ) + if err != nil { + t.Fatalf("cannot build roster: %v", err) + } + t.Cleanup(roster.Close) + return roster +} + +// TestHandleAnnouncerSessionMismatch_LegacyStragglerRecorded proves that a +// legacy peer observed by a security-v2 permit increments both the mismatch and +// cross-format counters, is attributed to the correct operator address in the +// node-local roster, and is logged without any raw session identifier. +func TestHandleAnnouncerSessionMismatch_LegacyStragglerRecorded(t *testing.T) { + metrics := newCutoverFakeMetrics() + roster := newTestRoster(t, metrics, 5000) + logger := &captureLogger{} + + // sender 2 -> operatorAddrs[1] (0x2222...). A nil limiter always logs. + handleAnnouncerSessionMismatch( + logger, + nil, + metrics, + roster, + participation.ModeSecurityV2, + participation.StateOpenSecurityV2.String(), + operatorAddrs, + "tbtc-dkg", + group.MemberIndex(2), + announcer.SessionIDFormatHardenedDKG, + announcer.SessionIDFormatLegacy, + ) + + if got := metrics.counter(clientinfo.MetricAnnouncerSessionIDMismatchTotal); got != 1 { + t.Errorf("mismatch counter = %v, want 1", got) + } + if got := metrics.counter(clientinfo.MetricAnnouncerCrossFormatPeerTotal); got != 1 { + t.Errorf("cross-format counter = %v, want 1", got) + } + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected 1 peer in roster, got %d", len(snapshot.Peers)) + } + if snapshot.Peers[0].OperatorAddress != "0x2222222222222222222222222222222222222222" { + t.Errorf( + "expected operator 0x2222..., got %s", + snapshot.Peers[0].OperatorAddress, + ) + } + + lines := logger.all() + if len(lines) != 1 { + t.Fatalf("expected 1 log line, got %d: %v", len(lines), lines) + } + line := lines[0] + expectedLog := "protocol announcement rejected: session ID mismatch " + + "[protocol=tbtc-dkg] [member=2] [expectedFormat=hardened_dkg] " + + "[observedFormat=legacy] [permitMode=security_v2] " + + "[gateState=open_security_v2]" + if line != expectedLog { + t.Errorf("unexpected mismatch log:\n got: %q\nwant: %q", line, expectedLog) + } + // The observer only ever receives classified formats, never raw IDs; the + // operator address is the only identifier and no session-ID hex appears. + for _, forbidden := range []string{"dkg-", "signing-", "seed", "0xdeadbeef"} { + if strings.Contains(line, forbidden) { + t.Errorf("log line leaked raw material %q: %q", forbidden, line) + } + } +} + +// TestHandleAnnouncerSessionMismatch_HardenedVsHardened proves that a mismatch +// between two hardened formats is counted as a mismatch but not as a +// cross-format peer, and is not recorded in the roster (the observed peer is not +// legacy). +func TestHandleAnnouncerSessionMismatch_HardenedVsHardened(t *testing.T) { + metrics := newCutoverFakeMetrics() + roster := newTestRoster(t, metrics, 5000) + + handleAnnouncerSessionMismatch( + &captureLogger{}, + nil, + metrics, + roster, + participation.ModeSecurityV2, + participation.StateOpenSecurityV2.String(), + operatorAddrs, + "tbtc-signing", + group.MemberIndex(1), + announcer.SessionIDFormatHardenedSigning, + announcer.SessionIDFormatHardenedDKG, + ) + + if got := metrics.counter(clientinfo.MetricAnnouncerSessionIDMismatchTotal); got != 1 { + t.Errorf("mismatch counter = %v, want 1", got) + } + if got := metrics.counter(clientinfo.MetricAnnouncerCrossFormatPeerTotal); got != 0 { + t.Errorf("cross-format counter = %v, want 0", got) + } + if got := len(roster.Snapshot().Peers); got != 0 { + t.Errorf("expected empty roster, got %d peers", got) + } +} + +// TestHandleAnnouncerSessionMismatch_OutOfRangeSender proves the explicit bounds +// check: an out-of-range member index does not panic and is not attributed to +// any operator, though the mismatch is still counted. +func TestHandleAnnouncerSessionMismatch_OutOfRangeSender(t *testing.T) { + metrics := newCutoverFakeMetrics() + roster := newTestRoster(t, metrics, 5000) + + // Only three operators exist; index 99 is out of range. + handleAnnouncerSessionMismatch( + &captureLogger{}, + nil, + metrics, + roster, + participation.ModeSecurityV2, + participation.StateOpenSecurityV2.String(), + operatorAddrs, + "tbtc-dkg", + group.MemberIndex(99), + announcer.SessionIDFormatHardenedDKG, + announcer.SessionIDFormatLegacy, + ) + + if got := metrics.counter(clientinfo.MetricAnnouncerSessionIDMismatchTotal); got != 1 { + t.Errorf("mismatch counter = %v, want 1", got) + } + if got := len(roster.Snapshot().Peers); got != 0 { + t.Errorf("out-of-range sender must not be rostered, got %d peers", got) + } +} + +// TestHandleAnnouncerSessionMismatch_PortZeroNilMetrics proves the handler is +// safe when client-info is disabled (nil metrics sink) and that the roster — +// constructed with a no-op recorder in that mode — still records the sighting. +func TestHandleAnnouncerSessionMismatch_PortZeroNilMetrics(t *testing.T) { + // Port-zero uses the no-op recorder for the roster and passes a nil metrics + // sink to the handler. + roster := newTestRoster(t, &clientinfo.NoOpPerformanceMetrics{}, 5000) + logger := &captureLogger{} + + handleAnnouncerSessionMismatch( + logger, + nil, + nil, // no metrics sink (client-info disabled) + roster, + participation.ModeSecurityV2, + participation.StateOpenSecurityV2.String(), + operatorAddrs, + "tbtc-dkg", + group.MemberIndex(3), + announcer.SessionIDFormatHardenedDKG, + announcer.SessionIDFormatLegacy, + ) + + snapshot := roster.Snapshot() + if len(snapshot.Peers) != 1 { + t.Fatalf("expected 1 peer despite disabled metrics, got %d", len(snapshot.Peers)) + } + if snapshot.Peers[0].OperatorAddress != "0x3333333333333333333333333333333333333333" { + t.Errorf("expected operator 0x3333..., got %s", snapshot.Peers[0].OperatorAddress) + } + if len(logger.all()) != 1 { + t.Errorf("expected the mismatch to still be logged when metrics are disabled") + } +} diff --git a/pkg/tbtc/deduplicator.go b/pkg/tbtc/deduplicator.go index 37d0b1704f..38a1f75753 100644 --- a/pkg/tbtc/deduplicator.go +++ b/pkg/tbtc/deduplicator.go @@ -56,19 +56,10 @@ func (d *deduplicator) notifyDKGStarted( newDKGSeed *big.Int, ) bool { d.dkgSeedCache.Sweep() - // The cache key is the hexadecimal representation of the seed. cacheKey := newDKGSeed.Text(16) - // If the key is not in the cache, that means the seed was not handled - // yet and the client should proceed with the execution. - if !d.dkgSeedCache.Has(cacheKey) { - d.dkgSeedCache.Add(cacheKey) - return true - } - - // Otherwise, the DKG seed is a duplicate and the client should not proceed - // with the execution. - return false + // Add is mutex-serialized: returns true only if the key was not already present. + return d.dkgSeedCache.Add(cacheKey) } // notifyDKGResultSubmitted notifies the client wants to start some actions @@ -85,16 +76,7 @@ func (d *deduplicator) notifyDKGResultSubmitted( hex.EncodeToString(newDKGResultHash[:]) + strconv.Itoa(int(newDKGResultBlock)) - // If the key is not in the cache, that means the result was not handled - // yet and the client should proceed with the execution. - if !d.dkgResultHashCache.Has(cacheKey) { - d.dkgResultHashCache.Add(cacheKey) - return true - } - - // Otherwise, the DKG result is a duplicate and the client should not - // proceed with the execution. - return false + return d.dkgResultHashCache.Add(cacheKey) } func (d *deduplicator) notifyWalletClosed( @@ -104,15 +86,5 @@ func (d *deduplicator) notifyWalletClosed( // Use wallet ID converted to string as the cache key. cacheKey := hex.EncodeToString(WalletID[:]) - - // If the key is not in the cache, that means the wallet closure was not - // handled yet and the client should proceed with the execution. - if !d.walletClosedCache.Has(cacheKey) { - d.walletClosedCache.Add(cacheKey) - return true - } - - // Otherwise, the wallet closure is a duplicate and the client should not - // proceed with the execution. - return false + return d.walletClosedCache.Add(cacheKey) } diff --git a/pkg/tbtc/deduplicator_test.go b/pkg/tbtc/deduplicator_test.go index b75432a8c0..ec4a90eda8 100644 --- a/pkg/tbtc/deduplicator_test.go +++ b/pkg/tbtc/deduplicator_test.go @@ -3,6 +3,8 @@ package tbtc import ( "encoding/hex" "math/big" + "sync" + "sync/atomic" "testing" "time" @@ -116,6 +118,125 @@ func TestNotifyDKGResultSubmitted(t *testing.T) { } } +// TestNotifyDKGStartedConcurrent is the F-13 TOCTOU regression test. +// Before F-13, notify*() did a Has()+Add() pair and two goroutines racing on +// the same key could both see Has() return false and both Add() the key, +// returning true from both calls. The fix relies on cache.TimeCache.Add() +// being mutex-serialized and returning true only for the first inserter. +// This test launches many goroutines that all race on the same key behind a +// barrier and asserts exactly one wins. +func TestNotifyDKGStartedConcurrent(t *testing.T) { + const callers = 100 + + dedup := deduplicator{ + dkgSeedCache: cache.NewTimeCache(testDKGSeedCachePeriod), + } + seed := big.NewInt(42) + + var wins int32 + var ready, start sync.WaitGroup + ready.Add(callers) + start.Add(1) + + results := make(chan bool, callers) + for i := 0; i < callers; i++ { + go func() { + ready.Done() + start.Wait() + results <- dedup.notifyDKGStarted(seed) + }() + } + ready.Wait() + start.Done() + + for i := 0; i < callers; i++ { + if <-results { + atomic.AddInt32(&wins, 1) + } + } + if got := atomic.LoadInt32(&wins); got != 1 { + t.Fatalf("F-13 regression: %d/%d concurrent notifyDKGStarted "+ + "calls returned true; want exactly 1", got, callers) + } +} + +func TestNotifyDKGResultSubmittedConcurrent(t *testing.T) { + const callers = 100 + + dedup := deduplicator{ + dkgResultHashCache: cache.NewTimeCache(testDKGResultHashCachePeriod), + } + hashBytes, err := hex.DecodeString( + "92327ddff69a2b8c7ae787c5d590a2f14586089e6339e942d56e82aa42052cd9", + ) + if err != nil { + t.Fatal(err) + } + var hash [32]byte + copy(hash[:], hashBytes) + + var wins int32 + var ready, start sync.WaitGroup + ready.Add(callers) + start.Add(1) + + results := make(chan bool, callers) + for i := 0; i < callers; i++ { + go func() { + ready.Done() + start.Wait() + results <- dedup.notifyDKGResultSubmitted(big.NewInt(100), hash, 500) + }() + } + ready.Wait() + start.Done() + + for i := 0; i < callers; i++ { + if <-results { + atomic.AddInt32(&wins, 1) + } + } + if got := atomic.LoadInt32(&wins); got != 1 { + t.Fatalf("F-13 regression: %d/%d concurrent notifyDKGResultSubmitted "+ + "calls returned true; want exactly 1", got, callers) + } +} + +func TestNotifyWalletClosedConcurrent(t *testing.T) { + const callers = 100 + + dedup := deduplicator{ + walletClosedCache: cache.NewTimeCache(testWalletClosedCachePeriod), + } + wallet := [32]byte{0x77} + + var wins int32 + var ready, start sync.WaitGroup + ready.Add(callers) + start.Add(1) + + results := make(chan bool, callers) + for i := 0; i < callers; i++ { + go func() { + ready.Done() + start.Wait() + results <- dedup.notifyWalletClosed(wallet) + }() + } + ready.Wait() + start.Done() + + for i := 0; i < callers; i++ { + if <-results { + atomic.AddInt32(&wins, 1) + } + } + if got := atomic.LoadInt32(&wins); got != 1 { + t.Fatalf("F-13 regression: %d/%d concurrent notifyWalletClosed "+ + "calls returned true; want exactly 1", got, callers) + } +} + func TestNotifyWalletClosed(t *testing.T) { deduplicator := deduplicator{ walletClosedCache: cache.NewTimeCache(testWalletClosedCachePeriod), diff --git a/pkg/tbtc/deposit_sweep.go b/pkg/tbtc/deposit_sweep.go index 824ce29d28..093fb77515 100644 --- a/pkg/tbtc/deposit_sweep.go +++ b/pkg/tbtc/deposit_sweep.go @@ -10,6 +10,7 @@ import ( "go.uber.org/zap" "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) const ( @@ -88,6 +89,11 @@ type depositSweepAction struct { broadcastTimeout time.Duration broadcastCheckDelay time.Duration + // permit is the wallet action's participation permit: it pins the + // protocol mode for every signing of this action and is released when the + // action's execution ends. + permit participation.Permit + // metricsRecorder is optional and used for recording performance metrics metricsRecorder interface { IncrementCounter(name string, value float64) @@ -105,12 +111,15 @@ func newDepositSweepAction( proposalProcessingStartBlock uint64, proposalExpiryBlock uint64, waitForBlockFn waitForBlockFn, + permit participation.Permit, ) *depositSweepAction { transactionExecutor := newWalletTransactionExecutor( btcChain, sweepingWallet, signingExecutor, waitForBlockFn, + permit, + "tbtc_deposit_sweep_bitcoin_broadcast", ) return &depositSweepAction{ @@ -126,10 +135,18 @@ func newDepositSweepAction( signingTimeoutSafetyMarginBlocks: depositSweepSigningTimeoutSafetyMarginBlocks, broadcastTimeout: depositSweepBroadcastTimeout, broadcastCheckDelay: depositSweepBroadcastCheckDelay, + permit: permit, } } func (dsa *depositSweepAction) execute() error { + // The action owns its permit from dispatch on; releasing it here ends the + // ceremony's active accounting in the participation gate. The terminal + // outcome is registered afterwards so it runs first and reaches the permit + // while it is still open. + defer dsa.permit.Close() + defer dsa.transactionExecutor.recordTerminalOutcome(dsa.logger) + executionStartTime := time.Now() // Record deposit sweep execution attempt @@ -232,11 +249,14 @@ func (dsa *depositSweepAction) execute() error { dsa.proposalExpiryBlock-dsa.signingTimeoutSafetyMarginBlocks, ) if err != nil { - if dsa.metricsRecorder != nil { + // A gate-caused abort is not an ordinary failure of this action and + // stays out of its failure metrics; the wrapped cause lets the + // dispatcher classify it the same way. + if dsa.metricsRecorder != nil && !participation.IsGateRefusal(err) { dsa.metricsRecorder.IncrementCounter("deposit_sweep_executions_failed_total", 1) dsa.metricsRecorder.RecordDuration("deposit_sweep_execution_duration_seconds", time.Since(executionStartTime)) } - return fmt.Errorf("sign transaction step failed: [%v]", err) + return fmt.Errorf("sign transaction step failed: [%w]", err) } // Record deposit sweep transaction signing duration @@ -256,11 +276,14 @@ func (dsa *depositSweepAction) execute() error { dsa.broadcastCheckDelay, ) if err != nil { - if dsa.metricsRecorder != nil { + // A gate-caused abort is not an ordinary failure of this action and + // stays out of its failure metrics; the wrapped cause lets the + // dispatcher classify it the same way. + if dsa.metricsRecorder != nil && !participation.IsGateRefusal(err) { dsa.metricsRecorder.IncrementCounter("deposit_sweep_executions_failed_total", 1) dsa.metricsRecorder.RecordDuration("deposit_sweep_execution_duration_seconds", time.Since(executionStartTime)) } - return fmt.Errorf("broadcast transaction step failed: [%v]", err) + return fmt.Errorf("broadcast transaction step failed: [%w]", err) } // Record successful deposit sweep execution diff --git a/pkg/tbtc/deposit_sweep_test.go b/pkg/tbtc/deposit_sweep_test.go index c98f75a3c0..607d85c7f4 100644 --- a/pkg/tbtc/deposit_sweep_test.go +++ b/pkg/tbtc/deposit_sweep_test.go @@ -11,6 +11,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tbtc/internal/test" ) @@ -193,6 +194,8 @@ func TestDepositSweepAction_Execute(t *testing.T) { rawSignatures, ) + permit := newTestPermit(participation.TBTCSigning) + action := newDepositSweepAction( logger.With(), hostChain, @@ -205,6 +208,7 @@ func TestDepositSweepAction_Execute(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + permit, ) // Modify the default parameters of the action to make @@ -233,6 +237,28 @@ func TestDepositSweepAction_Execute(t *testing.T) { scenario.ExpectedSweepTransaction.Serialize(), broadcastedSweepTransaction.Serialize(), ) + + // The action must leave the rollback journal pointing at the exact + // Bitcoin transaction it put on the network, recorded while its + // permit was still open. + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceBitcoinTransaction, + ) + + expectedReference := scenario.ExpectedSweepTransactionHash.Hex( + bitcoin.ReversedByteOrder, + ) + if evidence.Reference != expectedReference { + t.Errorf( + "unexpected evidence reference\n"+ + "expected: [%s]\nactual: [%s]", + expectedReference, + evidence.Reference, + ) + } }) } } diff --git a/pkg/tbtc/dkg.go b/pkg/tbtc/dkg.go index 177e225a18..1f7a885ebf 100644 --- a/pkg/tbtc/dkg.go +++ b/pkg/tbtc/dkg.go @@ -1,14 +1,21 @@ package tbtc import ( + "bytes" "context" + "crypto/sha256" + "encoding/hex" "errors" "fmt" "math/big" + "slices" "sort" + "sync" + "sync/atomic" "time" "golang.org/x/exp/maps" + "golang.org/x/time/rate" "go.uber.org/zap" @@ -19,7 +26,9 @@ import ( "github.com/keep-network/keep-core/pkg/generator" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa/dkg" ) @@ -74,6 +83,80 @@ type dkgExecutor struct { SetGauge(name string, value float64) RecordDuration(name string, duration time.Duration) } + + // cutoverPeerRoster is optional and, when set, records post-cutover legacy + // peer sightings observed by the DKG announcer. + cutoverPeerRoster *participation.CutoverPeerRoster + + // participationGate issues the per-member DKG participation permits that + // pin the ceremony's protocol mode from its canonical chain anchor. It is + // wired once during initialization, before any DKG event subscription + // exists; joining DKG without it is refused fail-closed. + participationGate participation.Gate + + // signerQuarantine preserves signer outputs whose activation the gate + // refused before the wallet's on-chain registration was proven. Joining + // DKG without it is refused fail-closed: a gate interruption after key + // generation would otherwise have to drop the generated share. + signerQuarantine *signerQuarantine + + // quarantineReportMutex serializes recounting the quarantine namespace and + // publishing the count, so concurrently preserving members cannot leave an + // older, lower count as the published one. It also guards the four fields + // below. + quarantineReportMutex sync.Mutex + + // preservedOutputFloor names the key material this process wrote to the + // quarantine namespace itself and no successful scan has ruled on yet. The + // namespace is the authority on how much preserved material a rollback has + // to account for, but a scan that fails cannot take away what this process + // knows it persisted, and the published count must never fall below that. + // A scan that does succeed settles every entry — it enumerated the namespace + // after each was written — so it empties this, and what the namespace still + // held is carried by lastScannedOutputs instead. + preservedOutputFloor map[quarantinedSigner]struct{} + + // lastScannedOutputs names what the last successful enumeration found. A + // quarantined output outlives the process that wrote it, so the material + // this process inherited is not its own to remember any other way, and + // dropping it the moment a scan fails would leave everything an earlier + // process preserved out of the count exactly when the namespace can no + // longer be asked about it. + lastScannedOutputs map[quarantinedSigner]struct{} + + // lastPublishedQuarantineCount is the count that currently stands, so a + // failed recount can tell whether what it does know is already covered by + // the published number. + lastPublishedQuarantineCount int + + // incompleteQuarantineOutputs names preservation attempts that exhausted + // their write-grace rounds and are still holding an output whose key + // material and audit record are not both durable. It is guarded by + // quarantineReportMutex and drives the live incomplete-output gauge. The + // map key deduplicates repeated notifications for the same wallet seat. + incompleteQuarantineOutputs map[quarantinedSigner]struct{} + + // announcerMismatchLogLimiter bounds the volume of session-ID mismatch INFO + // logs to a burst of 5 with one line every 30 seconds, matching the + // observability contract. Metrics retain every event. + announcerMismatchLogLimiter *rate.Limiter +} + +func tbtcDKGPermitIdentity( + seed *big.Int, + memberIndex group.MemberIndex, +) participation.PermitIdentity { + seedHash := sha256.Sum256(seed.Bytes()) + return participation.PermitIdentity{ + WorkID: hex.EncodeToString(seedHash[:]), + PermitID: fmt.Sprint(memberIndex), + // The one DKG seat this permit runs. It is the ceremony's own index + // space, not the final signing group's: the final group is not known + // until the result is built, and the seats a reader needs in order to + // tell which node was operating this ceremony are the ones it was + // operating while it ran. + OperatedMembers: participation.MemberIndexes{memberIndex}, + } } // newDkgExecutor creates a new instance of dkgExecutor struct. There should @@ -103,15 +186,16 @@ func newDkgExecutor( ) return &dkgExecutor{ - groupParameters: groupParameters, - operatorIDFn: operatorIDFn, - operatorAddress: operatorAddress, - chain: chain, - netProvider: netProvider, - walletRegistry: walletRegistry, - protocolLatch: protocolLatch, - tecdsaExecutor: tecdsaExecutor, - waitForBlockFn: waitForBlockFn, + groupParameters: groupParameters, + operatorIDFn: operatorIDFn, + operatorAddress: operatorAddress, + chain: chain, + netProvider: netProvider, + walletRegistry: walletRegistry, + protocolLatch: protocolLatch, + tecdsaExecutor: tecdsaExecutor, + waitForBlockFn: waitForBlockFn, + announcerMismatchLogLimiter: rate.NewLimiter(rate.Every(30*time.Second), 5), } } @@ -124,6 +208,12 @@ func (de *dkgExecutor) setMetricsRecorder(recorder interface { de.metricsRecorder = recorder } +// setCutoverPeerRoster sets the node-local cutover peer roster for the DKG +// executor. +func (de *dkgExecutor) setCutoverPeerRoster(roster *participation.CutoverPeerRoster) { + de.cutoverPeerRoster = roster +} + // preParamsCount returns the current count of the ECDSA DKG pre-parameters. func (de *dkgExecutor) preParamsCount() int { return de.tecdsaExecutor.PreParamsCount() @@ -281,6 +371,19 @@ func (de *dkgExecutor) generateSigningGroup( startBlock uint64, delayBlocks uint64, ) { + if de.participationGate == nil { + // Without the gate no permit can pin the ceremony's protocol mode. + // Fail closed. + dkgLogger.Errorf("no participation gate; refusing to join DKG") + return + } + if de.signerQuarantine == nil { + // Without a quarantine store a gate interruption after key generation + // would have to drop the generated share. Fail closed. + dkgLogger.Errorf("no signer quarantine store; refusing to join DKG") + return + } + membershipValidator := group.NewMembershipValidator( dkgLogger, groupSelectionResult.OperatorsAddresses, @@ -305,23 +408,61 @@ func (de *dkgExecutor) generateSigningGroup( // Capture the member index for the goroutine. memberIndex := index + // One participation permit per locally controlled member, issued + // immediately before the member goroutine. The permit pins the + // protocol mode from the ceremony's canonical chain anchor — the DKG + // started event block — for the ceremony's entire lifetime, including + // every retry attempt. A refusal is a gate decision, not an ordinary + // DKG failure. + permit, err := de.participationGate.Begin( + participation.TBTCDKG, + startBlock, + tbtcDKGPermitIdentity(seed, memberIndex), + ) + if err != nil { + dkgLogger.Warnf( + "[member:%v] refused by the participation gate: [%v]", + memberIndex, + err, + ) + continue + } + go func() { + defer permit.Close() + dkgStartTime := time.Now() de.protocolLatch.Lock() defer de.protocolLatch.Unlock() ctx, cancelCtx := withCancelOnBlock( - context.Background(), + permit.Context(), dkgTimeoutBlock, de.waitForBlockFn, ) defer cancelCtx() + // resultSubmitted holds the DKG result this ceremony was seen to + // settle on chain — submitted by this member or any other — before + // the subscription canceled the publication context. Activating the + // generated signer is conditioned on it: a publication context that + // ends without a submitted result must not leave an active signer + // behind. + // + // The event itself is kept rather than the fact that one arrived. + // The key material about to be activated is this member's own, and + // a result that settled for some other ceremony, or for some other + // group, says nothing about it — so what settled has to be readable + // where the generated result is, which is only after key generation + // returns. + var resultSubmitted atomic.Pointer[DKGResultSubmittedEvent] + // TODO: This subscription has to be updated once we implement // re-submitting DKG result to the chain after a challenge. // See https://github.com/keep-network/keep-core/issues/3450 subscription := de.chain.OnDKGResultSubmitted( func(event *DKGResultSubmittedEvent) { + resultSubmitted.Store(event) defer cancelCtx() dkgLogger.Infof( @@ -337,15 +478,59 @@ func (de *dkgExecutor) generateSigningGroup( }) defer subscription.Unsubscribe() + // currentMode is the local node's protocol mode for this ceremony, + // pinned in the participation permit. It classifies our own + // announcement so the mismatch observer can tell legacy peers + // apart from hardened ones during a coordinated cutover. + currentMode := permit.Mode() + // The compatibility strategy bundle carries the permit's mode + // into every tECDSA party this ceremony constructs; each retry + // attempt reuses it unchanged. + strategies, err := compatibility.StrategiesFor(currentMode) + if err != nil { + dkgLogger.Errorf( + "[member:%v] cannot select compatibility strategies: [%v]", + memberIndex, + err, + ) + return + } + // operatorAddresses maps a sender's group member index (1-based) to + // its operator address so a mismatch can be attributed to an + // operator in the node-local cutover roster. + operatorAddresses := groupSelectionResult.OperatorsAddresses + sessionMismatchObserver := func( + protocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + dkgLogger, + de.announcerMismatchLogLimiter, + de.metricsRecorder, + de.cutoverPeerRoster, + currentMode, + currentParticipationGateState(de.participationGate), + operatorAddresses, + protocolID, + sender, + expectedFormat, + observedFormat, + ) + } + announcer := announcer.New( fmt.Sprintf("%v-%v", ProtocolName, "dkg"), broadcastChannel, membershipValidator, + announcer.WithSessionMismatchObserver(sessionMismatchObserver), ) retryLoop := newDkgRetryLoop( dkgLogger, seed, + permit.Mode(), startBlock+delayBlocks, memberIndex, groupSelectionResult.OperatorsAddresses, @@ -379,24 +564,18 @@ func (de *dkgExecutor) generateSigningGroup( de.waitForBlockFn, ) - // sessionID must be different for each attempt. - sessionID := fmt.Sprintf( - "%v-%v", - seed.Text(16), - attempt.number, - ) - result, err := de.tecdsaExecutor.Execute( attemptCtx, dkgAttemptLogger, seed, - sessionID, + attempt.sessionID, memberIndex, de.groupParameters.GroupSize, de.groupParameters.DishonestThreshold(), attempt.excludedMembersIndexes, broadcastChannel, membershipValidator, + strategies, ) if err != nil { dkgAttemptLogger.Errorf( @@ -412,6 +591,19 @@ func (de *dkgExecutor) generateSigningGroup( }, ) if err != nil { + // A gate decision — clock failure, forced quiescence, or a + // closed permit — is not an ordinary DKG failure and must not + // increment the ordinary failure metrics. + if cause := context.Cause(ctx); participation.IsGateRefusal(cause) { + dkgLogger.Warnf( + "[member:%v] DKG canceled by the participation "+ + "gate: [%v]", + memberIndex, + cause, + ) + return + } + if de.metricsRecorder != nil { de.metricsRecorder.IncrementCounter(clientinfo.MetricDKGFailedTotal, 1) de.metricsRecorder.RecordDuration(clientinfo.MetricDKGDurationSeconds, time.Since(dkgStartTime)) @@ -433,63 +625,355 @@ func (de *dkgExecutor) generateSigningGroup( return } - signer, err := de.registerSigner( + activated := de.completeDkgCeremony( + ctx, + dkgLogger, + permit, + seed, result, memberIndex, - groupSelectionResult.OperatorsAddresses, + groupSelectionResult, + func() bool { + return dkgResultSettledLocalCeremony( + dkgLogger, + memberIndex, + seed, + result, + resultSubmitted.Load(), + ) + }, + func(publishCtx context.Context) error { + return de.publishDkgResult( + publishCtx, + dkgLogger, + seed, + memberIndex, + broadcastChannel, + membershipValidator, + result, + groupSelectionResult, + startBlock, + permit, + ) + }, ) - if err != nil { - dkgLogger.Errorf( - "[member:%v] failed to register signing group member: [%v]", - memberIndex, - err, - ) + if activated && de.metricsRecorder != nil { + // The ceremony completed end to end: result published, + // activation fenced, signer active. + de.metricsRecorder.RecordDuration(clientinfo.MetricDKGDurationSeconds, time.Since(dkgStartTime)) } + }() + } +} + +// dkgResultSettledLocalCeremony reports whether the DKG result observed to +// settle on chain is the one this member generated. +// +// Activation persists key material and enters it into the wallet cache under +// the final signing group the local result describes. Reading the subscription +// as nothing but "something settled" makes that a claim about a chain state +// nobody checked: an event for a different ceremony, or for a group rebuilt +// from a different membership, satisfies it equally, and the node then holds an +// active signer whose seat and whose wallet the chain does not agree with. That +// disagreement is exactly what the offline audit exists to find, and finding it +// afterwards is worse than not activating in the first place — so a mismatch +// falls through to the interrupted-signer path, which preserves the share +// without activating it and leaves the audit a record to reconcile. +// +// The three fields compared are the ones the wallet identity and the final +// group are derived from: the ceremony this result answers, the key it produced, +// and the members removed from the group that produced it. +func dkgResultSettledLocalCeremony( + dkgLogger log.StandardLogger, + memberIndex group.MemberIndex, + seed *big.Int, + result *dkg.Result, + submitted *DKGResultSubmittedEvent, +) bool { + if submitted == nil || submitted.Result == nil { + return false + } - dkgLogger.Infof("registered %s", signer) + if seed == nil || submitted.Seed == nil || + seed.Cmp(submitted.Seed) != 0 { + dkgLogger.Warnf( + "[member:%v] observed a DKG result for seed [0x%x] while running "+ + "the ceremony for seed [0x%x]; not activating the generated "+ + "signer against another ceremony's result", + memberIndex, + submitted.Seed, + seed, + ) + return false + } - // Record successful DKG completion - if de.metricsRecorder != nil { - de.metricsRecorder.RecordDuration(clientinfo.MetricDKGDurationSeconds, time.Since(dkgStartTime)) - } + localGroupPublicKey, err := result.GroupPublicKeyBytes() + if err != nil { + dkgLogger.Errorf( + "[member:%v] cannot read the generated group public key to "+ + "compare it with the submitted DKG result: [%v]", + memberIndex, + err, + ) + return false + } + if !sameChainGroupPublicKey( + localGroupPublicKey, + submitted.Result.GroupPublicKey, + ) { + dkgLogger.Warnf( + "[member:%v] the DKG result submitted for this ceremony carries "+ + "group public key [0x%x] while this member generated [0x%x]; "+ + "not activating a signer for a wallet the chain does not have", + memberIndex, + submitted.Result.GroupPublicKey, + localGroupPublicKey, + ) + return false + } - err = de.publishDkgResult( - ctx, + localMisbehaved := result.MisbehavedMembersIndexes() + if !slices.Equal(localMisbehaved, submitted.Result.MisbehavedMembersIndexes) { + dkgLogger.Warnf( + "[member:%v] the DKG result submitted for this ceremony removes "+ + "members %v while this member removed %v; the two describe "+ + "different final signing groups, so the generated signer is "+ + "not activated", + memberIndex, + submitted.Result.MisbehavedMembersIndexes, + localMisbehaved, + ) + return false + } + + return true +} + +// sameChainGroupPublicKey reports whether a locally marshaled group public key +// and one carried by a submitted DKG result are the same key. +// +// The Chain interface does not pin the encoding of the submitted key, and its +// implementations differ: the on-chain binding carries the 64-byte X||Y pair the +// registry stores, while the in-process chain carries the 65-byte uncompressed +// marshaling the local key produces. Both name one point, so the comparison is +// made on the coordinates the two share rather than on whichever prefix each +// happens to include. +func sameChainGroupPublicKey(local []byte, submitted []byte) bool { + uncompressed := func(key []byte) []byte { + if len(key) == 65 && key[0] == 4 { + return key[1:] + } + return key + } + + return bytes.Equal(uncompressed(local), uncompressed(submitted)) +} + +// completeDkgCeremony finalizes one member's DKG after key generation. +// Publication precedes activation: the generated share stays out of the +// active namespace and the wallet cache until the DKG result demonstrably +// reached the chain and the activation fence passed. A clock failure, forced +// quiescence, a publication window that closes without a submitted result, or +// a submitted result that is not the one this member generated therefore never +// leaves an active signer; every such outcome preserves the share through the +// interrupted-signer path instead of dropping or activating it. +// publishResultFn performs the result publication bound to the given context; +// resultSubmittedFn reports whether the result observed to settle on chain for +// this ceremony is this member's own. It returns true only when the signer was +// activated. +func (de *dkgExecutor) completeDkgCeremony( + ctx context.Context, + dkgLogger log.StandardLogger, + permit participation.Permit, + seed *big.Int, + result *dkg.Result, + memberIndex group.MemberIndex, + groupSelectionResult *GroupSelectionResult, + resultSubmittedFn func() bool, + publishResultFn func(context.Context) error, +) bool { + err := publishResultFn(ctx) + if err != nil { + // The submission fence returns its sentinel as an ordinary error; a + // permit cancellation surfaces as a plain context cancellation whose + // gate cause is only in the context. + refusal := err + if !participation.IsGateRefusal(refusal) { + refusal = context.Cause(ctx) + } + switch { + case participation.IsGateRefusal(refusal): + dkgLogger.Warnf( + "[member:%v] DKG result publication refused by the release "+ + "gate; preserving the generated signer without "+ + "activation: [%v]", + memberIndex, + err, + ) + de.preserveInterruptedSigner( dkgLogger, + permit, seed, + result, memberIndex, - broadcastChannel, - membershipValidator, + groupSelectionResult, + "tbtc_dkg_result_publication", + refusal, + ) + return false + case errors.Is(err, context.Canceled) && resultSubmittedFn(): + // The submission subscription observed the result on chain and + // ended the publication; the ceremony completed and the signer + // proceeds to activation. + dkgLogger.Infof( + "[member:%v] DKG result submitted by another member; "+ + "proceeding to signer activation", + memberIndex, + ) + default: + // The publication window closed without an observed submitted + // result, or publication failed outright. The wallet may never + // appear on chain, so the share is preserved without activation + // for the offline state audit to reconcile. + dkgLogger.Errorf( + "[member:%v] DKG result publication ended without a "+ + "submitted result; preserving the generated signer "+ + "without activation: [%v]", + memberIndex, + err, + ) + de.preserveInterruptedSigner( + dkgLogger, + permit, + seed, result, + memberIndex, groupSelectionResult, - startBlock, + "tbtc_dkg_result_publication", + err, ) - if err != nil { - if errors.Is(err, context.Canceled) { - dkgLogger.Infof( - "[member:%v] DKG is no longer awaiting the result; "+ - "aborting DKG result publication", - memberIndex, - ) - return - } + return false + } + } - dkgLogger.Errorf( - "[member:%v] DKG result publication failed [%v]", - memberIndex, - err, - ) - return - } - }() + // The last-moment fence before activating the newly generated key + // material, consulted only after the result publication concluded. A + // refusal — clock failure or process quiescence — preserves the share + // without activating it: in the protected quarantine namespace normally, + // or as a durable non-activated save when the wallet is already + // registered on chain. + if fenceErr := permit.CheckCommit( + "tbtc_dkg_signer_activation", + participation.CompletionCommit, + ); fenceErr != nil { + de.preserveInterruptedSigner( + dkgLogger, + permit, + seed, + result, + memberIndex, + groupSelectionResult, + "tbtc_dkg_signer_activation", + fenceErr, + ) + return false + } + + signer, err := de.registerSigner( + result, + memberIndex, + groupSelectionResult.OperatorsAddresses, + ) + if err != nil { + dkgLogger.Errorf( + "[member:%v] failed to register signing group member; "+ + "preserving the generated signer without activation: [%v]", + memberIndex, + err, + ) + de.preserveInterruptedSigner( + dkgLogger, + permit, + seed, + result, + memberIndex, + groupSelectionResult, + "tbtc_dkg_signer_registration", + err, + ) + return false } + + dkgLogger.Infof("registered %s", signer) + de.recordPermitTerminalOutcome( + dkgLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: getWalletStorageKey( + signer.wallet.publicKey, + ), + MembershipIndex: signer.signingGroupMemberIndex, + Contribution: dkgTranscriptContribution(signer, result), + }, + ) + + return true } -// registerSigner determines the final signing group shape and persists the -// generated signer with a unique key share. Note that the final group members -// may differ from the ones returned by the sortition pool if there was any -// misbehavior or inactivities during the key generation. -func (de *dkgExecutor) registerSigner( +// dkgTranscriptContribution renders the memberships that produced a DKG result, +// in the final signing group's own index space. +// +// The final signing group is built from exactly the DKG members this node saw +// operating: a member whose round messages did not arrive, or arrived without a +// valid membership behind them, was marked inactive and is absent from the final +// group. Every final membership therefore stands for a member whose +// contributions this node authenticated and combined into the key material it +// persisted — which is the local view of the transcript, and the only thing that +// distinguishes a key generated together with other parties from a persisted +// share whose provenance is one party's word. +// +// The index space is the final group's rather than the DKG's because the +// persisted membership this evidence names is a final index, and a transcript +// that mixed the two would join a result produced in one ceremony to a signer +// persisted from another. +// +// Because the permits for this ceremony are in the DKG's space, the transcript +// also carries the seat each final membership was rebuilt from. That mapping is +// the accepted result's own operating members, ascending, which is precisely the +// list finalSigningGroup positions the final group by — so entry i of it is the +// DKG seat that became final seat i+1. +func dkgTranscriptContribution( + persistedSigner *signer, + result *dkg.Result, +) *participation.TranscriptContribution { + operatingMembers := result.Group.OperatingMemberIndexes() + sort.Slice(operatingMembers, func(i, j int) bool { + return operatingMembers[i] < operatingMembers[j] + }) + + finalGroupSize := len(persistedSigner.wallet.signingGroupOperators) + + incorporated := make(participation.MemberIndexes, 0, finalGroupSize) + for seat := 1; seat <= finalGroupSize; seat++ { + incorporated = append(incorporated, group.MemberIndex(seat)) + } + + return &participation.TranscriptContribution{ + IncorporatedMembers: incorporated, + LocalMembers: participation.MemberIndexes{ + persistedSigner.signingGroupMemberIndex, + }, + PermitSpaceMembers: participation.MemberIndexes(operatingMembers), + } +} + +// buildFinalSigner determines the final signing group shape and constructs +// the signer holding the generated key share. Note that the final group +// members may differ from the ones returned by the sortition pool if there +// was any misbehavior or inactivities during the key generation. +func (de *dkgExecutor) buildFinalSigner( result *dkg.Result, memberIndex group.MemberIndex, selectedSigningGroupOperators chain.Addresses, @@ -521,12 +1005,30 @@ func (de *dkgExecutor) registerSigner( ) } - signer := newSigner( + return newSigner( result.PrivateKeyShare.PublicKey(), finalSigningGroupOperators, finalSigningGroupMemberIndex, result.PrivateKeyShare, + ), nil +} + +// registerSigner determines the final signing group shape and persists the +// generated signer with a unique key share, activating it in the wallet +// cache. +func (de *dkgExecutor) registerSigner( + result *dkg.Result, + memberIndex group.MemberIndex, + selectedSigningGroupOperators chain.Addresses, +) (*signer, error) { + signer, err := de.buildFinalSigner( + result, + memberIndex, + selectedSigningGroupOperators, ) + if err != nil { + return nil, err + } err = de.walletRegistry.registerSigner(signer) if err != nil { @@ -540,7 +1042,584 @@ func (de *dkgExecutor) registerSigner( return signer, nil } -// publishDkgResult performs the DKG result publication process. +// preserveInterruptedSigner durably preserves generated key material the +// release gate or a failed ceremony step kept from activating — a clock +// failure, process quiescence, or a publication that ended without a +// submitted result raced with the completing DKG. The share is never dropped +// and never activated by this process: when the wallet is already registered +// on chain the signer is saved to the active namespace without cache +// activation, so a restart's reconciliation can pick it up; otherwise it goes +// to the protected quarantine namespace that no release's active-wallet scan +// reads, for the offline state audit to reconcile. The operation names the +// ceremony step that was refused in the quarantine metadata. +func (de *dkgExecutor) preserveInterruptedSigner( + dkgLogger log.StandardLogger, + permit participation.Permit, + seed *big.Int, + result *dkg.Result, + memberIndex group.MemberIndex, + groupSelectionResult *GroupSelectionResult, + operation string, + fenceErr error, +) { + signer, err := de.buildFinalSigner( + result, + memberIndex, + groupSelectionResult.OperatorsAddresses, + ) + if err != nil { + dkgLogger.Errorf( + "[member:%v] cannot build the interrupted signer; the generated "+ + "share is only in memory: [%v]", + memberIndex, + err, + ) + return + } + + walletRegistered := false + walletID, err := de.chain.CalculateWalletID(signer.wallet.publicKey) + if err == nil { + walletRegistered, err = de.chain.IsWalletRegistered(walletID) + if err != nil { + // An unverifiable registration state is treated as unregistered: + // quarantine preserves the share without exposing it to any + // release's active scan. + walletRegistered = false + } + } + + if walletRegistered { + dkgLogger.Warnf( + "[member:%v] signer activation withheld at [%s] but the wallet "+ + "is registered on chain; saving the signer without "+ + "activation: [%v]", + memberIndex, + operation, + fenceErr, + ) + saveErr := de.walletRegistry.saveSigner(signer) + if saveErr == nil { + de.recordPermitTerminalOutcome( + dkgLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: getWalletStorageKey( + signer.wallet.publicKey, + ), + MembershipIndex: signer.signingGroupMemberIndex, + Contribution: dkgTranscriptContribution(signer, result), + }, + ) + return + } + + // The active namespace refused the share, so the quarantine namespace is + // tried rather than the share being dropped: it is a separate namespace + // with its own failure modes, and a preserved share the audit reports as + // unexpected is recoverable where a lost one is not. The recorded + // operation says the active save was refused, so the audit reads the + // record as what it is — a registered wallet's share that did not reach + // the namespace a restart would load it from — rather than as an + // ordinary pre-registration quarantine. + dkgLogger.Errorf( + "[member:%v] failed to save the interrupted signer for a "+ + "registered wallet; preserving it in the quarantine namespace "+ + "instead: [%v]", + memberIndex, + saveErr, + ) + operation += "_after_refused_active_save" + } + + seedHash := sha256.Sum256(seed.Bytes()) + snapshot := de.participationGate.State() + + walletIDHex := "" + if err == nil { + walletIDHex = hex.EncodeToString(walletID[:]) + } + quarantineOutput := quarantinedSigner{ + walletStorageKey: getWalletStorageKey(signer.wallet.publicKey), + memberIndex: signer.signingGroupMemberIndex, + } + + dkgLogger.Warnf( + "[member:%v] signer activation withheld at [%s]; quarantining the "+ + "generated signer: [%v]", + memberIndex, + operation, + fenceErr, + ) + + quarantineState, quarantineErr := de.signerQuarantine.preserve( + signer, + QuarantinedSignerMetadata{ + ReleaseEpoch: participation.CompiledEpoch.String(), + ProtocolMode: permit.Mode().String(), + CutoverBlock: snapshot.CutoverBlock, + CanonicalStartBlock: permit.CanonicalStartBlock(), + Ceremony: string(permit.Ceremony()), + SeedHash: hex.EncodeToString(seedHash[:]), + WalletID: walletIDHex, + FailedOperation: operation, + LastObservedBlock: snapshot.CurrentBlock, + }, + quarantineObserver{ + // The published count follows the key material alone: a share the + // namespace holds is material a rollback has to account for even + // when the record explaining it did not land, and a share that + // never reached the namespace is not quarantined however much was + // written about it. + // + // It is taken here, at the moment the namespace accepts the share, + // rather than from what preserve returns. A preservation whose + // other half keeps being refused runs until the process ends, so + // the return is not a moment this count can wait for. + // + // What runs here is only what this handoff can afford. The audit + // record is written next, in this same round, and enumerating the + // namespace between the two writes would put a namespace-wide read + // on the path of the one write that turns preserved material into + // explained material. + keyMaterialPreserved: func() { + de.accountForPreservedKeyMaterial(signer) + }, + // Preservation keeps running behind this. It fires once the + // namespace has refused a half for longer than a passing fault + // would last, so the node stops taking new work while it is still + // holding an output the namespace does not fully have. + stillIncomplete: func(state quarantineState, cause error) { + de.markIncompleteQuarantine(quarantineOutput) + de.blockOnIncompleteQuarantine( + dkgLogger, + memberIndex, + state, + cause, + ) + }, + }, + ) + + // The observer above normally reports an incomplete output while Preserve + // is still retrying. A failure before the retry loop begins — for example, + // serialization failure — has no grace callback, and a process lifetime + // that ends before grace can return without one too, so account for both + // here. A completed retry removes the output from the live gauge; the + // cumulative failure counter remains as history. + if quarantineState.complete() { + de.resolveIncompleteQuarantine(quarantineOutput) + } else { + de.markIncompleteQuarantine(quarantineOutput) + } + + // The preservation is over, so the namespace can be read without holding a + // write up behind it. What the write-time accounting published is a floor — + // what this process can vouch for — and this is where it is reconciled + // against what the namespace actually holds, which is the only reading that + // can bring the count back down once a seat has been activated or an + // operator has cleared a record. + if quarantineState.keyMaterialPersisted() { + de.reportQuarantinedSigners(dkgLogger) + } + + // The terminal outcome, unlike the count, needs the whole output. The + // audit record is what names the mode, canonical anchor, ceremony, seat, and + // refused operation of the preserved share; without it the offline audit + // cannot reconcile the material against the chain, so calling the permit + // resolved would hand the rollback decision a quarantine nothing explains. + // Either form the namespace took it whole in settles this — the record pair + // or the single handoff carrying both. Anything less leaves the permit + // unresolved, and the offline barrier keeps blocking on it until an operator + // repairs the namespace. + if !quarantineState.complete() { + de.blockOnIncompleteQuarantine( + dkgLogger, + memberIndex, + quarantineState, + quarantineErr, + ) + return + } + + de.recordPermitTerminalOutcome( + dkgLogger, + permit, + participation.TerminalOutcomeQuarantined, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceQuarantinedTBTCSinger, + }, + ) +} + +// accountForPreservedKeyMaterial adds key material this process durably wrote +// to the count a rollback reads. It is called once the quarantine namespace is +// known to hold the share, which is the only condition it may be called under. +// +// The audit metadata may still be missing. The count is of preserved shares, +// and a share the namespace holds is one whether or not the record explaining +// it landed — leaving it out until the pair completes would under-report exactly +// the material a rollback most needs to find. +// +// The namespace is not enumerated here. This runs between the two writes of one +// preservation round, and a scan that hangs there would hold up the audit record +// the share still needs — the half whose absence leaves preserved material +// unexplained. What it publishes instead is the floor this process can vouch +// for without asking anyone: everything the last successful scan found plus +// everything written since. That floor is never above the truth, so the reading +// it leaves standing until the post-preservation recount cannot be an +// all-clear. +func (de *dkgExecutor) accountForPreservedKeyMaterial(signer *signer) { + de.quarantineReportMutex.Lock() + defer de.quarantineReportMutex.Unlock() + + if de.preservedOutputFloor == nil { + de.preservedOutputFloor = make(map[quarantinedSigner]struct{}) + } + // Keyed by wallet and seat, so preserving the same output twice — a retry, + // a second interruption of the same seat — names it once rather than + // counting the same share as two. + de.preservedOutputFloor[quarantinedSigner{ + walletStorageKey: getWalletStorageKey(signer.wallet.publicKey), + memberIndex: signer.signingGroupMemberIndex, + }] = struct{}{} + + de.publishKnownFloorLocked() +} + +// markIncompleteQuarantine publishes a newly observed incomplete preservation. +// The normal call is the live grace-exhaustion observer; the return-time call +// covers failures that reached no observer. The counter records the first +// grace-exhaustion notification for an output while that output remains +// unresolved; repeated notifications for the same wallet seat coalesce in the +// live gauge. Resolution removes the seat, so a later incomplete episode for +// it is counted again. The gauge remains nonzero for as long as the output lacks +// either key material or its audit record. +func (de *dkgExecutor) markIncompleteQuarantine( + output quarantinedSigner, +) { + de.quarantineReportMutex.Lock() + defer de.quarantineReportMutex.Unlock() + + if de.incompleteQuarantineOutputs == nil { + de.incompleteQuarantineOutputs = + make(map[quarantinedSigner]struct{}) + } + if _, exists := de.incompleteQuarantineOutputs[output]; exists { + return + } + + de.incompleteQuarantineOutputs[output] = struct{}{} + if de.metricsRecorder == nil { + return + } + + de.metricsRecorder.IncrementCounter( + clientinfo.MetricParticipationTBTCQuarantinePreservationFailuresTotal, + 1, + ) + de.metricsRecorder.SetGauge( + clientinfo.MetricParticipationTBTCQuarantineIncompleteOutputs, + float64(len(de.incompleteQuarantineOutputs)), + ) +} + +// resolveIncompleteQuarantine clears the live incomplete-output signal only +// after preservation has made the whole output durable. An output still +// incomplete when the process lifetime ends deliberately remains nonzero in +// the last readable sample; the cumulative counter is never decremented. +func (de *dkgExecutor) resolveIncompleteQuarantine( + output quarantinedSigner, +) { + de.quarantineReportMutex.Lock() + defer de.quarantineReportMutex.Unlock() + + if _, exists := de.incompleteQuarantineOutputs[output]; !exists { + return + } + + delete(de.incompleteQuarantineOutputs, output) + if de.metricsRecorder != nil { + de.metricsRecorder.SetGauge( + clientinfo.MetricParticipationTBTCQuarantineIncompleteOutputs, + float64(len(de.incompleteQuarantineOutputs)), + ) + } +} + +// blockOnIncompleteQuarantine stops this node from beginning new ceremonies +// while a preserved output is missing a half the namespace was supposed to hold. +// +// Either half missing leaves an inventory a rollback cannot reconcile. A share +// that reached no namespace exists only in the goroutine that generated it and +// nothing an operator or the offline audit can read accounts for it. A share +// preserved without its audit metadata is on disk but unexplained: the mode, +// canonical anchor, ceremony, seat, and refused operation that would let the +// audit match it against the chain are exactly what did not land. Taking on more +// work in either state builds further state on a host whose inventory is already +// known to be incomplete. +// +// Quiescence is the blocking state rather than a new one of its own: it refuses +// every new permit, it is already what the gate-state gauge and the quiesce +// counter report, and it lets the permits still running finish normally. It is +// one-way by design — an operator restarts the node once the namespace is +// repaired — which is also why the preservation behind it is given a grace +// budget first, so a namespace that clears on its own does not cost the fleet a +// node. No terminal outcome is recorded, so this permit closes unresolved and +// blocks the offline barrier on its own. +// +// The returned channel is deliberately ignored. This caller holds a permit of +// its own, so waiting for the active permit count to reach zero here would be +// waiting for itself. +func (de *dkgExecutor) blockOnIncompleteQuarantine( + dkgLogger log.StandardLogger, + memberIndex group.MemberIndex, + state quarantineState, + cause error, +) { + if state.keyMaterialPersisted() { + dkgLogger.Errorf( + "[member:%v] the quarantined signer has no audit record "+ + "explaining it; the share is preserved but a rollback cannot "+ + "reconcile it without the record; refusing new ceremonies on "+ + "this node until an operator repairs the quarantine "+ + "namespace: [%v]", + memberIndex, + cause, + ) + } else { + dkgLogger.Errorf( + "[member:%v] generated key material reached no namespace; the "+ + "share is only in memory [auditMetadataPreserved=%v]; "+ + "refusing new ceremonies on this node until an operator "+ + "resolves the quarantine namespace: [%v]", + memberIndex, + state.metadataPersisted, + cause, + ) + } + + if de.participationGate == nil { + return + } + + de.participationGate.Quiesce(fmt.Errorf( + "tbtc key material could not be preserved with its audit record: [%w]", + cause, + )) +} + +// reportQuarantinedSigners publishes how many preserved signer outputs this +// process is holding without having activated them. +// +// The value is recounted from the namespace on every call rather than tracked as +// this process's own tally of preservations. A quarantined output outlives the +// process that wrote it: the count a rollback decision needs is of everything +// preserved on this host, and a tally that starts at zero every restart reports +// none of what an earlier one left behind. Recounting also keeps the comparison +// honest in the other direction — an output whose seat this process did activate +// from the active namespace stops being counted, which a tally could not +// express. +// +// Recount and publication are serialized. Concurrent members of the same +// ceremony quarantine independently, and two interleaved scans could otherwise +// publish out of order, leaving the older, lower count as the last word — the +// direction that reads as an all-clear. +func (de *dkgExecutor) reportQuarantinedSigners(dkgLogger log.StandardLogger) { + if err := de.publishQuarantinedSignerCount(); err != nil { + // The last published count stands. Publishing a zero here would say the + // namespace is empty, which is precisely what could not be established. + dkgLogger.Errorf( + "cannot count the quarantined signer outputs; the reported count "+ + "stays as last published: [%v]", + err, + ) + } +} + +// reportInitialQuarantinedSigners publishes the count this process starts with +// and refuses to start when the namespace cannot be enumerated. +// +// Keeping the last published count is the right answer at runtime because there +// is one: a scan that fails after an earlier scan succeeded leaves a number +// somebody published. At startup there is none. The gauge is registered at zero +// with the rest of the fixed family, so a startup scan that gives up quietly +// leaves that zero as this process's first and only word on the subject, and a +// rollback decision reads it as nothing left to account for — the one answer +// the count must never invent. +// +// So the failure is raised to the caller rather than logged. A node that will +// not start is a visible fault an operator resolves against a namespace whose +// contents are still on disk; a node that starts and reports an empty +// quarantine is an invisible one. +func (de *dkgExecutor) reportInitialQuarantinedSigners() error { + if err := de.publishQuarantinedSignerCount(); err != nil { + return fmt.Errorf( + "cannot count the quarantined signer outputs this process "+ + "starts with: [%w]", + err, + ) + } + + return nil +} + +// publishQuarantinedSignerCount recounts the namespace and publishes how many +// preserved outputs this process holds without having activated them. How the +// caller treats a namespace it cannot enumerate is what tells a startup apart +// from a later recount, so that failure is returned rather than decided here. +// +// A failed scan still publishes when this process knows more than the standing +// count does. The namespace is the authority on the total, but what a scan +// already found and what this process wrote itself are not in doubt: a share +// held and not activated stays held whatever the namespace can be read to say. +// Without that floor, a startup that published zero followed by a first +// post-write scan that failed would leave the zero standing over a namespace +// holding key material — the one answer the count must never invent. +// +// The namespace is enumerated whether or not a recorder is configured. An +// unreadable quarantine is a fault in its own right, and a node that only +// notices it when the client-info endpoint happens to be enabled would start +// over preserved material nobody can account for. +func (de *dkgExecutor) publishQuarantinedSignerCount() error { + if de.signerQuarantine == nil { + return nil + } + + de.quarantineReportMutex.Lock() + defer de.quarantineReportMutex.Unlock() + + outputs, err := de.signerQuarantine.preservedOutputs() + if err != nil { + de.publishKnownFloorLocked() + + return err + } + + de.lastScannedOutputs = make(map[quarantinedSigner]struct{}, len(outputs)) + for _, output := range outputs { + de.lastScannedOutputs[output] = struct{}{} + } + + // The floor exists to name what this process wrote and no scan has ruled on + // yet. This scan ruled on all of it: every entry was added after the + // namespace had taken the record, and this enumeration ran later still, + // under the same lock, so the namespace was asked about every one of them. + // Whatever it did not answer for — a seat an operator activated, a record + // they cleared — is gone, and keeping the identity would let the next failed + // scan union it back in and raise the count over an output that no longer + // exists. + de.preservedOutputFloor = nil + + de.publishQuarantineCountLocked(de.withheldCount(outputs)) + + return nil +} + +// publishKnownFloorLocked publishes what this process can still account for +// without reading the namespace, when the namespace is what it could not read. +// The caller must hold quarantineReportMutex. +// +// It only ever raises the count. A floor is a lower bound — the namespace may +// hold outputs neither the last scan nor this process saw — so letting it lower +// a higher standing count would turn what could not be established into a +// smaller number somebody reads as progress. +func (de *dkgExecutor) publishKnownFloorLocked() { + if floor := de.withheldCount( + de.knownOutputsLocked(), + ); floor > de.lastPublishedQuarantineCount { + de.publishQuarantineCountLocked(floor) + } +} + +// withheldCount counts the preserved outputs whose seat this process has not +// activated from the active namespace. An activated seat stopped being withheld +// material the moment it became a working signer. +func (de *dkgExecutor) withheldCount(outputs []quarantinedSigner) int { + withheld := 0 + for _, output := range outputs { + if de.walletRegistry.isSignerActive( + output.walletStorageKey, + output.memberIndex, + ) { + continue + } + withheld++ + } + + return withheld +} + +// knownOutputsLocked lists the preserved outputs this process can name without +// reading the namespace: everything the last successful scan found together +// with everything this process has persisted since. The caller must hold +// quarantineReportMutex. +// +// Both are needed and neither is enough. The scan is the only account of what +// earlier processes on this host left behind, and it says nothing about a share +// written after it; this process's own writes say nothing about what it +// inherited. Reporting either alone leaves out material that is on disk. +// +// The two overlap freely — a scan taken after a write finds that write — so +// they are merged as identities rather than added as counts, and an output +// named by both is one output. +// +// Only writes a successful scan has not already ruled on survive in the floor, +// so this can name an output the namespace has since let go of only for as long +// as no scan has succeeded to say otherwise. +func (de *dkgExecutor) knownOutputsLocked() []quarantinedSigner { + known := make( + map[quarantinedSigner]struct{}, + len(de.lastScannedOutputs)+len(de.preservedOutputFloor), + ) + for output := range de.lastScannedOutputs { + known[output] = struct{}{} + } + for output := range de.preservedOutputFloor { + known[output] = struct{}{} + } + + outputs := make([]quarantinedSigner, 0, len(known)) + for output := range known { + outputs = append(outputs, output) + } + + return outputs +} + +// publishQuarantineCountLocked publishes the count and remembers it as the one +// that stands. The caller must hold quarantineReportMutex. +// +// Nothing is remembered when there is no recorder to publish to. The remembered +// value is what a failed recount compares its floor against, and a number that +// never reached a gauge would hold that comparison up against a reading nobody +// can see. +func (de *dkgExecutor) publishQuarantineCountLocked(count int) { + if de.metricsRecorder == nil { + return + } + + de.lastPublishedQuarantineCount = count + + de.metricsRecorder.SetGauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + float64(count), + ) +} + +func (de *dkgExecutor) recordPermitTerminalOutcome( + dkgLogger log.StandardLogger, + permit participation.Permit, + outcome participation.TerminalOutcome, + evidence participation.TerminalEvidence, +) { + recordPermitTerminalOutcome(dkgLogger, permit, outcome, evidence) +} + +// publishDkgResult performs the DKG result publication process. The commit +// guard fences the terminal on-chain submission. func (de *dkgExecutor) publishDkgResult( ctx context.Context, dkgLogger log.StandardLogger, @@ -551,6 +1630,7 @@ func (de *dkgExecutor) publishDkgResult( dkgResult *dkg.Result, groupSelectionResult *GroupSelectionResult, startBlock uint64, + commitGuard participation.CommitGuard, ) error { return dkg.Publish( ctx, @@ -566,6 +1646,7 @@ func (de *dkgExecutor) publishDkgResult( de.groupParameters, groupSelectionResult, de.waitForBlockFn, + commitGuard, ), dkgResult, ) diff --git a/pkg/tbtc/dkg_cutover_integration_test.go b/pkg/tbtc/dkg_cutover_integration_test.go new file mode 100644 index 0000000000..c8a7b1b027 --- /dev/null +++ b/pkg/tbtc/dkg_cutover_integration_test.go @@ -0,0 +1,2119 @@ +package tbtc + +// This file carries the in-repository part of the tBTC DKG cutover acceptance +// evidence: the production DKG retry loop and announcer over real local +// network providers, real participation gates clocked by a local chain, and +// complete tECDSA key-generation transcripts in both modes with +// fixture pre-parameters, including generated key material, misbehavior +// evidence, and signer persistence across a registry restart. +// +// The on-chain 90-active/10-misbehaved consequence with reward ineligibility +// belongs to the Solidity suite, and the exact-image mixed-release rehearsals +// — including transcript realness at the full hundred-member scale — to the +// release scripts. What is proven here is the anchor-derived mode selection, +// its immutability across the cutover block, homogeneous legacy and +// security-v2 transcripts, the quorum discipline of the retry loop, the +// real-transcript conversion of post-cutover legacy and silent peers into +// misbehaved-members evidence, mismatch metrics, and roster attribution, and +// the exact production-scale first-attempt exclusion of ten legacy seats at +// the ninety-member quorum. + +import ( + "bytes" + "context" + "crypto/ecdsa" + "fmt" + "math/big" + "sync/atomic" + "testing" + "time" + + "github.com/bnb-chain/tss-lib/ecdsa/keygen" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/timestamppb" + + "github.com/keep-network/keep-common/pkg/persistence" + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/generator" + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/tecdsa/dkg" + "github.com/keep-network/keep-core/pkg/tecdsa/dkg/gen/pb" +) + +// dkgCutoverGroup is a local test group whose seats have distinct wire +// identities: one local provider and one operator per seat. The wire +// operator addresses come from the local chain's signing — raw public keys — +// while the roster operator addresses are normalized Ethereum addresses for +// the same seats, matching the two shapes a production group carries. +type dkgCutoverGroup struct { + localChain *localChain + blockCounter chain.BlockCounter + providers []net.Provider + operators chain.Addresses + rosterOperators chain.Addresses + validator *group.MembershipValidator +} + +// provider returns the network provider of the given 1-based member. +func (g *dkgCutoverGroup) provider(memberIndex group.MemberIndex) net.Provider { + return g.providers[memberIndex-1] +} + +// setupDKGCutoverGroup builds a distinct-identity local group of the given +// size over a chain with the given block time. +func setupDKGCutoverGroup( + t *testing.T, + groupSize int, + blockTime time.Duration, +) *dkgCutoverGroup { + t.Helper() + + g := &dkgCutoverGroup{} + + for i := 0; i < groupSize; i++ { + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + + if i == 0 { + g.localChain = ConnectWithKey(operatorPrivateKey, blockTime) + } + + g.providers = append(g.providers, local.ConnectWithKey(operatorPublicKey)) + + operatorAddress, err := g.localChain.Signing().PublicKeyToAddress( + operatorPublicKey, + ) + if err != nil { + t.Fatal(err) + } + g.operators = append(g.operators, operatorAddress) + + g.rosterOperators = append( + g.rosterOperators, + chain.Address(fmt.Sprintf("0x%040x", i+1)), + ) + } + + g.validator = group.NewMembershipValidator( + &testutils.MockLogger{}, + g.operators, + g.localChain.Signing(), + ) + + blockCounter, err := g.localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + g.blockCounter = blockCounter + + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + return g +} + +// TestDKGCutover_SecurityV2AnchorUsesHardenedSessionIDs proves the +// smoke-gate-2 mode-selection rule for a post-cutover DKG: a canonical DKG +// anchor at the cutover block pins the security-v2 mode even when the local +// callback height is already past the cutover block, and the production retry +// loop derives the exact hardened session ID and proceeds with the full ready +// cohort. +func TestDKGCutover_SecurityV2AnchorUsesHardenedSessionIDs(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 3, + GroupQuorum: 2, + HonestThreshold: 2, + } + + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + + localChain := ConnectWithKey(operatorPrivateKey, 20*time.Millisecond) + localProvider := local.ConnectWithKey(operatorPublicKey) + + operatorAddress, err := localChain.Signing().PublicKeyToAddress( + operatorPublicKey, + ) + if err != nil { + t.Fatal(err) + } + + var operators chain.Addresses + for i := 0; i < groupParameters.GroupSize; i++ { + operators = append(operators, operatorAddress) + } + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + // Cross the cutover block before the ceremony starts: the canonical + // anchor is the cutover block itself while the current height is already + // past it. + cutoverBlock := uint64(2) + if err := blockCounter.WaitForBlockHeight(cutoverBlock + 1); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, blockCounter, cutoverBlock) + + permit, err := gate.Begin(participation.TBTCDKG, cutoverBlock) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + testutils.AssertStringsEqual( + t, + "permit mode for the anchor at the cutover block", + participation.ModeSecurityV2.String(), + permit.Mode().String(), + ) + + seed := big.NewInt(0x77997799) + protocolID := fmt.Sprintf("%v-%v", ProtocolName, "dkg") + channelName := "dkg-cutover-hardened-test" + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + operators, + localChain.Signing(), + ) + + channel, err := localProvider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + hardenedSessionIDs := []string{ + compatibility.SecurityV2().DKGSessionID(seed, 1), + compatibility.SecurityV2().DKGSessionID(seed, 2), + } + for _, memberIndex := range []group.MemberIndex{2, 3} { + startPeerAnnouncer( + peersCtx, + t, + localProvider, + channelName, + membershipValidator, + protocolID, + memberIndex, + hardenedSessionIDs, + ) + } + + loopAnnouncer := announcer.New(protocolID, channel, membershipValidator) + + anchor := permit.CanonicalStartBlock() + retryLoop := newDkgRetryLoop( + logger, + seed, + permit.Mode(), + anchor, + group.MemberIndex(1), + operators, + groupParameters, + loopAnnouncer, + 3, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 30*time.Second, + ) + defer cancelLoopCtx() + + expectedResult := &dkg.Result{} + var attemptSessionIDs []string + var attemptExclusions [][]group.MemberIndex + + result, err := retryLoop.start( + loopCtx, + newChainWaitForBlockFn(blockCounter), + func(attempt *dkgAttemptParams) (*dkg.Result, error) { + attemptSessionIDs = append(attemptSessionIDs, attempt.sessionID) + attemptExclusions = append( + attemptExclusions, + attempt.excludedMembersIndexes, + ) + return expectedResult, nil + }, + ) + cancelPeers() + if err != nil { + t.Fatal(err) + } + if result != expectedResult { + t.Error("expected the attempt's result") + } + + testutils.AssertIntsEqual(t, "attempts", 1, len(attemptSessionIDs)) + testutils.AssertStringsEqual( + t, + "attempt session ID", + fmt.Sprintf("dkg-%v-%016x", seed.Text(16), 1), + attemptSessionIDs[0], + ) + testutils.AssertStringsEqual( + t, + "attempt session ID format", + announcer.SessionIDFormatHardenedDKG.String(), + announcer.ClassifySessionIDFormat(attemptSessionIDs[0]).String(), + ) + testutils.AssertIntsEqual( + t, + "excluded members", + 0, + len(attemptExclusions[0]), + ) +} + +// marshaledTestPreParams converts the given tss-lib local pre-parameters into +// the exact bytes the tECDSA pre-parameters storage persists, so a test +// executor can restore them through the pool's ordinary restart path instead +// of running the CPU-intensive generation. The layout mirrors the production +// PreParams marshaling. +func marshaledTestPreParams( + t *testing.T, + localPreParams keygen.LocalPreParams, +) []byte { + t.Helper() + + pbPreParams := &pb.PreParams{ + Data: &pb.PreParams_LocalPreParams{ + PaillierSK: &pb.PreParams_PrivateKey{ + PublicKey: &pb.PreParams_PublicKey{ + N: localPreParams.PaillierSK.N.Bytes(), + }, + LambdaN: localPreParams.PaillierSK.LambdaN.Bytes(), + PhiN: localPreParams.PaillierSK.PhiN.Bytes(), + }, + NTilde: localPreParams.NTildei.Bytes(), + H1I: localPreParams.H1i.Bytes(), + H2I: localPreParams.H2i.Bytes(), + Alpha: localPreParams.Alpha.Bytes(), + Beta: localPreParams.Beta.Bytes(), + P: localPreParams.P.Bytes(), + Q: localPreParams.Q.Bytes(), + }, + CreationTimestamp: timestamppb.Now(), + } + + preParamsBytes, err := proto.Marshal(pbPreParams) + if err != nil { + t.Fatal(err) + } + + return preParamsBytes +} + +// newSeededTecdsaExecutor builds a real tECDSA DKG executor whose +// pre-parameters pool restores the given member's fixture pre-parameters from +// persistence — the executor's ordinary restart path — so an attempt can run +// the complete key-generation transcript without the CPU-intensive +// pre-parameters generation. The scheduler's permanently locked latch stops +// the pool's background generation within one scheduler tick. +func newSeededTecdsaExecutor( + t *testing.T, + localPreParams keygen.LocalPreParams, +) *dkg.Executor { + t.Helper() + + workPersistence := &mockPersistenceHandle{ + saved: []persistence.DataDescriptor{ + &mockDescriptor{ + name: "pp_seeded", + directory: "preparams", + content: marshaledTestPreParams(t, localPreParams), + }, + }, + } + + return dkg.NewExecutor( + &testutils.MockLogger{}, + newTestScheduler(t), + workPersistence, + 1, // pool size: exactly the seeded entry + 2*time.Minute, // pre-params generation timeout + time.Hour, // pre-params generation delay + 1, // pre-params generation concurrency + 10, // key-generation concurrency, as in the protocol tests + ) +} + +// dkgCutoverMemberOutcome carries one member's DKG retry-loop outcome across +// the per-member goroutine boundary. +type dkgCutoverMemberOutcome struct { + memberIndex group.MemberIndex + result *dkg.Result + sessionIDs []string + err error +} + +// runRealDKGCutoverMember mirrors the production per-member DKG pipeline over +// the given cutover group: one participation permit issued from the canonical +// anchor, the production broadcast-channel setup, announcer, and retry loop, +// and a real tECDSA key-generation execution per attempt. Announcer options +// let a member wire the production session-mismatch observer. The outcome is +// always delivered to the outcomes channel, exactly once. +func runRealDKGCutoverMember( + ctx context.Context, + cutoverGroup *dkgCutoverGroup, + gate participation.Gate, + groupParameters *GroupParameters, + seed *big.Int, + anchor uint64, + memberIndex group.MemberIndex, + tecdsaExecutor *dkg.Executor, + outcomes chan<- *dkgCutoverMemberOutcome, + announcerOptions ...announcer.Option, +) { + outcome := &dkgCutoverMemberOutcome{memberIndex: memberIndex} + defer func() { outcomes <- outcome }() + + permit, err := gate.Begin(participation.TBTCDKG, anchor) + if err != nil { + outcome.err = fmt.Errorf("gate refused the permit: [%w]", err) + return + } + defer permit.Close() + + channelName := fmt.Sprintf("%s-%s", ProtocolName, seed.Text(16)) + channel, err := cutoverGroup.provider(memberIndex).BroadcastChannelFor( + channelName, + ) + if err != nil { + outcome.err = err + return + } + + dkg.RegisterUnmarshallers(channel) + announcer.RegisterUnmarshaller(channel) + if err := channel.SetFilter(cutoverGroup.validator.IsInGroup); err != nil { + outcome.err = err + return + } + + memberAnnouncer := announcer.New( + fmt.Sprintf("%v-%v", ProtocolName, "dkg"), + channel, + cutoverGroup.validator, + announcerOptions..., + ) + + strategies, err := compatibility.StrategiesFor(permit.Mode()) + if err != nil { + outcome.err = err + return + } + + retryLoop := newDkgRetryLoop( + logger, + seed, + permit.Mode(), + anchor, + memberIndex, + cutoverGroup.operators, + groupParameters, + memberAnnouncer, + 3, + ) + + waitFn := newChainWaitForBlockFn(cutoverGroup.blockCounter) + + outcome.result, outcome.err = retryLoop.start( + ctx, + waitFn, + func(attempt *dkgAttemptParams) (*dkg.Result, error) { + outcome.sessionIDs = append(outcome.sessionIDs, attempt.sessionID) + + attemptCtx, cancelAttemptCtx := withCancelOnBlock( + ctx, + attempt.timeoutBlock, + waitFn, + ) + defer cancelAttemptCtx() + + return tecdsaExecutor.Execute( + attemptCtx, + &testutils.MockLogger{}, + seed, + attempt.sessionID, + memberIndex, + groupParameters.GroupSize, + groupParameters.DishonestThreshold(), + attempt.excludedMembersIndexes, + channel, + cutoverGroup.validator, + strategies, + ) + }, + ) +} + +// TestDKGCutover_HomogeneousSecurityV2RealKeyGeneration proves the smoke-gate-2 +// homogeneous security-v2 control with a complete key-generation transcript: a +// full cohort whose ceremony is canonically anchored at the cutover block runs +// the production retry loop and announcer over real local network providers, +// executes the real tECDSA key-generation protocol under the hardened session +// ID, and every member derives the same wallet public key with no misbehavior +// evidence. The generated signers then pass through the production +// result-to-signer transformation and registry persistence, and a registry +// restart — a fresh registry over the same persistence — restores the wallet +// and all memberships. Result-publication fencing is covered separately by the +// completion-fence tests. +func TestDKGCutover_HomogeneousSecurityV2RealKeyGeneration(t *testing.T) { + testDKGCutoverHomogeneousRealKeyGeneration( + t, + participation.ModeSecurityV2, + ) +} + +// TestDKGCutover_HomogeneousLegacyRealKeyGeneration proves that a DKG event +// anchored before the cutover, but executed after the chain has crossed it, +// completes with the historical transcript and session-ID format. The +// dependency's transcript regression tests pin the emitted legacy challenges +// to the prior production formulas; exact-image prior/R1 process +// interoperability remains a release rehearsal gate. +func TestDKGCutover_HomogeneousLegacyRealKeyGeneration(t *testing.T) { + testDKGCutoverHomogeneousRealKeyGeneration(t, participation.ModeLegacy) +} + +func testDKGCutoverHomogeneousRealKeyGeneration( + t *testing.T, + mode participation.ProtocolMode, +) { + groupParameters := &GroupParameters{ + GroupSize: 3, + GroupQuorum: 2, + HonestThreshold: 2, + } + + // A block time roomy enough for the real key-generation transcript to + // complete within one attempt's protocol window under CI package + // parallelism, race detector included. + cutoverGroup := setupDKGCutoverGroup( + t, + groupParameters.GroupSize, + 200*time.Millisecond, + ) + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures( + groupParameters.GroupSize, + ) + if err != nil { + t.Fatal(err) + } + + // Cross the cutover block before the ceremony starts. The canonical anchor + // selects the requested transcript and remains immutable even though the + // current height is already past the cutover. + cutoverBlock := uint64(2) + if err := cutoverGroup.blockCounter.WaitForBlockHeight( + cutoverBlock + 1, + ); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, cutoverGroup.blockCounter, cutoverBlock) + anchor := cutoverBlock + if mode == participation.ModeLegacy { + anchor = cutoverBlock - 1 + } + + expectedStrategies, err := compatibility.StrategiesFor(mode) + if err != nil { + t.Fatal(err) + } + + seed := big.NewInt(0x2C0DE) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 120*time.Second, + ) + defer cancelLoopCtx() + + outcomes := make( + chan *dkgCutoverMemberOutcome, + groupParameters.GroupSize, + ) + for i := 1; i <= groupParameters.GroupSize; i++ { + memberIndex := group.MemberIndex(i) + tecdsaExecutor := newSeededTecdsaExecutor( + t, + testData[i-1].LocalPreParams, + ) + + go runRealDKGCutoverMember( + loopCtx, + cutoverGroup, + gate, + groupParameters, + seed, + anchor, + memberIndex, + tecdsaExecutor, + outcomes, + ) + } + + results := make(map[group.MemberIndex]*dkg.Result) + for i := 0; i < groupParameters.GroupSize; i++ { + outcome := <-outcomes + if outcome.err != nil { + t.Fatalf( + "member [%v] failed: [%v]", + outcome.memberIndex, + outcome.err, + ) + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("attempts of member [%v]", outcome.memberIndex), + 1, + len(outcome.sessionIDs), + ) + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt session ID of member [%v]", outcome.memberIndex), + expectedStrategies.DKGSessionID(seed, 1), + outcome.sessionIDs[0], + ) + + results[outcome.memberIndex] = outcome.result + } + + referencePublicKeyBytes, err := results[1].GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + for memberIndex, result := range results { + testutils.AssertIntsEqual( + t, + fmt.Sprintf("misbehaved members of member [%v]", memberIndex), + 0, + len(result.MisbehavedMembersIndexes()), + ) + + publicKeyBytes, err := result.GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(referencePublicKeyBytes, publicKeyBytes) { + t.Errorf( + "member [%v] derived group public key [0x%x] "+ + "instead of [0x%x]", + memberIndex, + publicKeyBytes, + referencePublicKeyBytes, + ) + } + } + + // The production result-to-signer transformation and persistence: all + // three memberships register against one registry, as a single node + // controlling three seats would. + walletPersistence := &mockPersistenceHandle{} + walletRegistry, err := newWalletRegistry( + walletPersistence, + cutoverGroup.localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + registrar := &dkgExecutor{ + groupParameters: groupParameters, + walletRegistry: walletRegistry, + } + + var walletPublicKey *ecdsa.PublicKey + for memberIndex, result := range results { + registeredSigner, err := registrar.registerSigner( + result, + memberIndex, + cutoverGroup.operators, + ) + if err != nil { + t.Fatalf( + "failed to register the signer of member [%v]: [%v]", + memberIndex, + err, + ) + } + walletPublicKey = registeredSigner.wallet.publicKey + } + + testutils.AssertIntsEqual( + t, + "active signers after registration", + groupParameters.GroupSize, + len(walletRegistry.getSigners(walletPublicKey)), + ) + + // A registry restart: a fresh registry over the same persistence must + // restore the wallet and all generated memberships. + restartedRegistry, err := newWalletRegistry( + walletPersistence, + cutoverGroup.localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + testutils.AssertIntsEqual( + t, + "active signers after the registry restart", + groupParameters.GroupSize, + len(restartedRegistry.getSigners(walletPublicKey)), + ) +} + +// TestDKGCutover_RealKeyGenerationExcludesSilentPeer proves that the +// production retry loop and the real tECDSA key-generation protocol convert a +// silent post-cutover peer into misbehavior evidence: the two live members +// exclude the never-announcing seat at quorum, complete the real transcript +// without it, report it in the result's misbehaved members, and the +// production result-to-signer transformation resolves the reduced final +// signing group with remapped member indexes. This is the off-chain half of +// the 90-active/10-misbehaved consequence; the on-chain acceptance and reward +// ineligibility belong to the Solidity suite. +func TestDKGCutover_RealKeyGenerationExcludesSilentPeer(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 3, + GroupQuorum: 2, + HonestThreshold: 2, + } + + // A block time roomy enough for the real key-generation transcript to + // complete within one attempt's protocol window under CI package + // parallelism, race detector included. + cutoverGroup := setupDKGCutoverGroup( + t, + groupParameters.GroupSize, + 200*time.Millisecond, + ) + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures( + groupParameters.GroupSize, + ) + if err != nil { + t.Fatal(err) + } + + cutoverBlock := uint64(2) + if err := cutoverGroup.blockCounter.WaitForBlockHeight( + cutoverBlock + 1, + ); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, cutoverGroup.blockCounter, cutoverBlock) + + seed := big.NewInt(0x51137) + silentMemberIndex := group.MemberIndex(2) + liveMembersIndexes := []group.MemberIndex{1, 3} + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 120*time.Second, + ) + defer cancelLoopCtx() + + outcomes := make( + chan *dkgCutoverMemberOutcome, + len(liveMembersIndexes), + ) + for _, memberIndex := range liveMembersIndexes { + tecdsaExecutor := newSeededTecdsaExecutor( + t, + testData[memberIndex-1].LocalPreParams, + ) + + go runRealDKGCutoverMember( + loopCtx, + cutoverGroup, + gate, + groupParameters, + seed, + cutoverBlock, + memberIndex, + tecdsaExecutor, + outcomes, + ) + } + + results := make(map[group.MemberIndex]*dkg.Result) + for i := 0; i < len(liveMembersIndexes); i++ { + outcome := <-outcomes + if outcome.err != nil { + t.Fatalf( + "member [%v] failed: [%v]", + outcome.memberIndex, + outcome.err, + ) + } + results[outcome.memberIndex] = outcome.result + } + + referencePublicKeyBytes, err := results[1].GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + for _, memberIndex := range liveMembersIndexes { + result := results[memberIndex] + + misbehaved := result.MisbehavedMembersIndexes() + testutils.AssertIntsEqual( + t, + fmt.Sprintf("misbehaved members of member [%v]", memberIndex), + 1, + len(misbehaved), + ) + testutils.AssertIntsEqual( + t, + fmt.Sprintf("misbehaved member index of member [%v]", memberIndex), + int(silentMemberIndex), + int(misbehaved[0]), + ) + + publicKeyBytes, err := result.GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(referencePublicKeyBytes, publicKeyBytes) { + t.Errorf( + "member [%v] derived group public key [0x%x] "+ + "instead of [0x%x]", + memberIndex, + publicKeyBytes, + referencePublicKeyBytes, + ) + } + } + + // The reduced final signing group: the silent seat is dropped and the + // remaining member indexes are remapped to consecutive positions. + walletRegistry, err := newWalletRegistry( + &mockPersistenceHandle{}, + cutoverGroup.localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + registrar := &dkgExecutor{ + groupParameters: groupParameters, + walletRegistry: walletRegistry, + } + + expectedFinalIndexes := map[group.MemberIndex]group.MemberIndex{ + 1: 1, + 3: 2, + } + for _, memberIndex := range liveMembersIndexes { + registeredSigner, err := registrar.registerSigner( + results[memberIndex], + memberIndex, + cutoverGroup.operators, + ) + if err != nil { + t.Fatalf( + "failed to register the signer of member [%v]: [%v]", + memberIndex, + err, + ) + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("final signing group size of member [%v]", memberIndex), + len(liveMembersIndexes), + len(registeredSigner.wallet.signingGroupOperators), + ) + testutils.AssertIntsEqual( + t, + fmt.Sprintf("final member index of member [%v]", memberIndex), + int(expectedFinalIndexes[memberIndex]), + int(registeredSigner.signingGroupMemberIndex), + ) + } +} + +// TestDKGCutover_LegacyAnchorPinnedThroughRetriesAcrossCutover proves the +// mode-pinning half of the smoke-gate-2 legacy case: a DKG canonically +// anchored below the cutover block keeps the legacy mode through every retry +// attempt — the production retry loop derives the exact prior-release session +// ID for a retry starting at or after the cutover block, and the permit's +// mode never mutates while the process state is already open_security_v2. +// Completed cryptographic legacy interoperability is covered by the +// homogeneous legacy test above. +func TestDKGCutover_LegacyAnchorPinnedThroughRetriesAcrossCutover(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 3, + GroupQuorum: 2, + HonestThreshold: 2, + } + + // Distinct wire identities per seat: the DKG retry algorithm derives the + // attempt-2+ qualified set by excluding operators, which requires more + // than one distinct operator address. + cutoverGroup := setupDKGCutoverGroup(t, groupParameters.GroupSize, 20*time.Millisecond) + blockCounter := cutoverGroup.blockCounter + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + cutoverBlock := anchor + 2 + + gate := newTestGateWithCutover(t, blockCounter, cutoverBlock) + + permit, err := gate.Begin(participation.TBTCDKG, anchor) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + testutils.AssertStringsEqual( + t, + "permit mode for the pre-cutover anchor", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + + seed := big.NewInt(0x881188) + protocolID := fmt.Sprintf("%v-%v", ProtocolName, "dkg") + channelName := "dkg-cutover-pin-test" + + channel, err := cutoverGroup.provider(1).BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + // The legacy peers announce the exact prior-release session IDs of the + // first three attempts, exactly as a prior binary would for this seed. + legacySessionIDs := []string{ + compatibility.Legacy().DKGSessionID(seed, 1), + compatibility.Legacy().DKGSessionID(seed, 2), + compatibility.Legacy().DKGSessionID(seed, 3), + } + for _, memberIndex := range []group.MemberIndex{2, 3} { + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(memberIndex), + channelName, + cutoverGroup.validator, + protocolID, + memberIndex, + legacySessionIDs, + ) + } + + loopAnnouncer := announcer.New(protocolID, channel, cutoverGroup.validator) + + retryLoop := newDkgRetryLoop( + logger, + seed, + permit.Mode(), + anchor, + group.MemberIndex(1), + cutoverGroup.operators, + groupParameters, + loopAnnouncer, + 3, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 60*time.Second, + ) + defer cancelLoopCtx() + + expectedResult := &dkg.Result{} + + type invokedAttempt struct { + number uint + sessionID string + startBlock uint64 + } + var invokedAttempts []invokedAttempt + + // The first invoked attempt fails so the loop retries at a start block + // that is unambiguously at or after the cutover block. The retry + // algorithm's seeded operator exclusion may skip the local member on one + // retry, so the succeeding attempt is the second one actually invoked, + // not necessarily attempt number two. + result, err := retryLoop.start( + loopCtx, + newChainWaitForBlockFn(blockCounter), + func(attempt *dkgAttemptParams) (*dkg.Result, error) { + invokedAttempts = append(invokedAttempts, invokedAttempt{ + number: attempt.number, + sessionID: attempt.sessionID, + startBlock: attempt.startBlock, + }) + + if len(invokedAttempts) == 1 { + return nil, fmt.Errorf("simulated first-attempt failure") + } + return expectedResult, nil + }, + ) + cancelPeers() + if err != nil { + t.Fatal(err) + } + if result != expectedResult { + t.Error("expected the retry attempt's result") + } + + if len(invokedAttempts) < 2 { + t.Fatalf( + "expected at least two invoked attempts, got [%d]", + len(invokedAttempts), + ) + } + for _, attempt := range invokedAttempts { + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt %d session ID", attempt.number), + fmt.Sprintf("%v-%v", seed.Text(16), attempt.number), + attempt.sessionID, + ) + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt %d session ID format", attempt.number), + announcer.SessionIDFormatLegacy.String(), + announcer.ClassifySessionIDFormat(attempt.sessionID).String(), + ) + } + + lastAttempt := invokedAttempts[len(invokedAttempts)-1] + if lastAttempt.startBlock < cutoverBlock { + t.Errorf( + "expected the retry attempt to start at or after the cutover "+ + "block [%d], got [%d]", + cutoverBlock, + lastAttempt.startBlock, + ) + } + + testutils.AssertStringsEqual( + t, + "permit mode after crossing the cutover block", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + snapshot := gate.State() + testutils.AssertStringsEqual( + t, + "gate state after crossing the cutover block", + participation.StateOpenSecurityV2.String(), + snapshot.State.String(), + ) +} + +// TestDKGCutover_PostCutoverSplitExcludesLegacyPeersAtProductionScale proves +// the exact smoke-gate-2 90/10 exclusion arithmetic at the production group +// parameters: a post-cutover DKG selection over a hundred-member group whose +// ten prior-release seats keep announcing legacy session IDs proceeds in the +// first attempt — the security-v2 cohort alone is exactly the group quorum +// of ninety — and excludes exactly the ten legacy seats. This test pins the +// retry-loop exclusion vector only; the real result carrying key material +// and all ten excluded seats as misbehaved-members indexes is produced by +// TestDKGCutover_RealKeyGenerationExcludesTenLegacyPeers at the largest +// group this repository can drive with distinct real pre-parameters. Every +// legacy straggler is attributed to its operator in the node-local roster. +// The on-chain acceptance of the ninety-active boundary and the +// reward-ineligibility consequence live in the Solidity suite; transcript +// realness at this scale stays with the exact-image rehearsals. +func TestDKGCutover_PostCutoverSplitExcludesLegacyPeersAtProductionScale(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 100, + GroupQuorum: 90, + HonestThreshold: 51, + } + + cutoverGroup := setupDKGCutoverGroup(t, groupParameters.GroupSize, 100*time.Millisecond) + blockCounter := cutoverGroup.blockCounter + + gate := newTestGate(t, blockCounter) + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + permit, err := gate.Begin(participation.TBTCDKG, anchor) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + seed := big.NewInt(0x9010) + protocolID := fmt.Sprintf("%v-%v", ProtocolName, "dkg") + channelName := "dkg-cutover-split-production-scale-test" + + channel, err := cutoverGroup.provider(1).BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + recorder := newDispatcherMetricsRecorder() + roster, err := participation.NewCutoverPeerRoster( + context.Background(), + blockCounter, + 1500, + newCutoverFakeMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(roster.Close) + + // The observer is wired exactly like the production DKG executor wires it. + currentMode := permit.Mode() + loopAnnouncer := announcer.New( + protocolID, + channel, + cutoverGroup.validator, + announcer.WithSessionMismatchObserver(func( + observedProtocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + logger, + nil, + recorder, + roster, + currentMode, + participation.StateOpenSecurityV2.String(), + cutoverGroup.rosterOperators, + observedProtocolID, + sender, + expectedFormat, + observedFormat, + ) + }), + ) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + // Members 2-90 are current security-v2 peers — together with the local + // member that is exactly the group quorum of ninety. Members 91-100 are + // prior-release binaries that keep announcing the legacy session ID + // after the cutover. + firstLegacySeat := groupParameters.GroupQuorum + 1 + hardenedSessionIDs := []string{ + compatibility.SecurityV2().DKGSessionID(seed, 1), + compatibility.SecurityV2().DKGSessionID(seed, 2), + } + for seat := 2; seat < firstLegacySeat; seat++ { + memberIndex := group.MemberIndex(seat) + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(memberIndex), + channelName, + cutoverGroup.validator, + protocolID, + memberIndex, + hardenedSessionIDs, + ) + } + legacySessionIDs := []string{ + compatibility.Legacy().DKGSessionID(seed, 1), + compatibility.Legacy().DKGSessionID(seed, 2), + } + for seat := firstLegacySeat; seat <= groupParameters.GroupSize; seat++ { + memberIndex := group.MemberIndex(seat) + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(memberIndex), + channelName, + cutoverGroup.validator, + protocolID, + memberIndex, + legacySessionIDs, + ) + } + + retryLoop := newDkgRetryLoop( + logger, + seed, + permit.Mode(), + anchor, + group.MemberIndex(1), + cutoverGroup.rosterOperators, + groupParameters, + loopAnnouncer, + 3, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 30*time.Second, + ) + defer cancelLoopCtx() + + expectedResult := &dkg.Result{} + var attemptExclusions [][]group.MemberIndex + + result, err := retryLoop.start( + loopCtx, + newChainWaitForBlockFn(blockCounter), + func(attempt *dkgAttemptParams) (*dkg.Result, error) { + attemptExclusions = append( + attemptExclusions, + attempt.excludedMembersIndexes, + ) + return expectedResult, nil + }, + ) + cancelPeers() + if err != nil { + t.Fatal(err) + } + if result != expectedResult { + t.Error("expected the attempt's result") + } + + // The security-v2 cohort proceeded at exactly the ninety-member quorum + // in the first attempt and excluded exactly the ten legacy seats. + testutils.AssertIntsEqual(t, "attempts", 1, len(attemptExclusions)) + excludedMembersIndexes := attemptExclusions[0] + legacySeatCount := groupParameters.GroupSize - groupParameters.GroupQuorum + testutils.AssertIntsEqual( + t, + "excluded members", + legacySeatCount, + len(excludedMembersIndexes), + ) + for i, excludedMemberIndex := range excludedMembersIndexes { + testutils.AssertIntsEqual( + t, + fmt.Sprintf("excluded member at position [%v]", i), + firstLegacySeat+i, + int(excludedMemberIndex), + ) + } + + // Every straggler became mismatch and cross-format evidence attributed + // to its operator in the node-local roster. + if mismatches := recorder.counter( + clientinfo.MetricAnnouncerSessionIDMismatchTotal, + ); mismatches < float64(legacySeatCount) { + t.Errorf( + "expected at least [%v] mismatches, got [%v]", + legacySeatCount, + mismatches, + ) + } + if crossFormat := recorder.counter( + clientinfo.MetricAnnouncerCrossFormatPeerTotal, + ); crossFormat < float64(legacySeatCount) { + t.Errorf( + "expected at least [%v] cross-format peers, got [%v]", + legacySeatCount, + crossFormat, + ) + } + + rosterSnapshot := roster.Snapshot() + testutils.AssertIntsEqual( + t, + "cutover roster operators", + legacySeatCount, + len(rosterSnapshot.Peers), + ) + rosterOperatorAddresses := make(map[string]bool) + for _, peer := range rosterSnapshot.Peers { + rosterOperatorAddresses[peer.OperatorAddress] = true + } + for seat := firstLegacySeat; seat <= groupParameters.GroupSize; seat++ { + operatorAddress := string(cutoverGroup.rosterOperators[seat-1]) + if !rosterOperatorAddresses[operatorAddress] { + t.Errorf( + "legacy seat [%v] operator [%s] missing from the roster", + seat, + operatorAddress, + ) + } + } +} + +// TestDKGCutover_RealKeyGenerationExcludesLegacyPeerAtQuorum proves the +// off-chain half of the smoke-gate-2 90/10 consequence with a real +// transcript: a post-cutover DKG selection contains a live prior-release +// peer that keeps announcing the legacy session ID, the security-v2 cohort +// proceeds once it alone reaches the group quorum, completes the real tECDSA +// key-generation protocol without the legacy seat, and reports that seat in +// the result's misbehaved members — the exact output the submitted result +// carries into the Solidity suite's boundary acceptance and reward-ban +// proof. The straggler also becomes mismatch metrics and roster evidence +// attributed to its operator, and the production result-to-signer +// transformation resolves the reduced final signing group with remapped +// member indexes. +func TestDKGCutover_RealKeyGenerationExcludesLegacyPeerAtQuorum(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 3, + GroupQuorum: 2, + HonestThreshold: 2, + } + + // A block time roomy enough for the real key-generation transcript to + // complete within one attempt's protocol window under CI package + // parallelism, race detector included. + cutoverGroup := setupDKGCutoverGroup( + t, + groupParameters.GroupSize, + 200*time.Millisecond, + ) + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures( + groupParameters.GroupSize, + ) + if err != nil { + t.Fatal(err) + } + + cutoverBlock := uint64(2) + if err := cutoverGroup.blockCounter.WaitForBlockHeight( + cutoverBlock + 1, + ); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, cutoverGroup.blockCounter, cutoverBlock) + + seed := big.NewInt(0x1E6AC1) + legacyMemberIndex := group.MemberIndex(3) + liveMembersIndexes := []group.MemberIndex{1, 2} + + recorder := newDispatcherMetricsRecorder() + roster, err := participation.NewCutoverPeerRoster( + context.Background(), + cutoverGroup.blockCounter, + 1500, + newCutoverFakeMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(roster.Close) + + // The first live member observes announcement mismatches exactly like + // the production DKG executor wires them: stragglers become metrics and + // roster evidence. The permit mode is pinned to security-v2 by the + // member pipeline itself. + mismatchObserver := announcer.WithSessionMismatchObserver(func( + observedProtocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + logger, + nil, + recorder, + roster, + participation.ModeSecurityV2, + participation.StateOpenSecurityV2.String(), + cutoverGroup.rosterOperators, + observedProtocolID, + sender, + expectedFormat, + observedFormat, + ) + }) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + // Seat 3 is a prior-release binary that keeps announcing the legacy + // session IDs after the cutover, on the same channel the live members + // use for the ceremony. + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(legacyMemberIndex), + fmt.Sprintf("%s-%s", ProtocolName, seed.Text(16)), + cutoverGroup.validator, + fmt.Sprintf("%v-%v", ProtocolName, "dkg"), + legacyMemberIndex, + []string{ + compatibility.Legacy().DKGSessionID(seed, 1), + compatibility.Legacy().DKGSessionID(seed, 2), + }, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 120*time.Second, + ) + defer cancelLoopCtx() + + outcomes := make( + chan *dkgCutoverMemberOutcome, + len(liveMembersIndexes), + ) + for _, memberIndex := range liveMembersIndexes { + tecdsaExecutor := newSeededTecdsaExecutor( + t, + testData[memberIndex-1].LocalPreParams, + ) + + var announcerOptions []announcer.Option + if memberIndex == liveMembersIndexes[0] { + announcerOptions = append(announcerOptions, mismatchObserver) + } + + go runRealDKGCutoverMember( + loopCtx, + cutoverGroup, + gate, + groupParameters, + seed, + cutoverBlock, + memberIndex, + tecdsaExecutor, + outcomes, + announcerOptions..., + ) + } + + results := make(map[group.MemberIndex]*dkg.Result) + for i := 0; i < len(liveMembersIndexes); i++ { + outcome := <-outcomes + if outcome.err != nil { + t.Fatalf( + "member [%v] failed: [%v]", + outcome.memberIndex, + outcome.err, + ) + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("attempts of member [%v]", outcome.memberIndex), + 1, + len(outcome.sessionIDs), + ) + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt session ID of member [%v]", outcome.memberIndex), + compatibility.SecurityV2().DKGSessionID(seed, 1), + outcome.sessionIDs[0], + ) + + results[outcome.memberIndex] = outcome.result + } + cancelPeers() + + referencePublicKeyBytes, err := results[1].GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + for _, memberIndex := range liveMembersIndexes { + result := results[memberIndex] + + misbehaved := result.MisbehavedMembersIndexes() + testutils.AssertIntsEqual( + t, + fmt.Sprintf("misbehaved members of member [%v]", memberIndex), + 1, + len(misbehaved), + ) + testutils.AssertIntsEqual( + t, + fmt.Sprintf("misbehaved member index of member [%v]", memberIndex), + int(legacyMemberIndex), + int(misbehaved[0]), + ) + + publicKeyBytes, err := result.GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(referencePublicKeyBytes, publicKeyBytes) { + t.Errorf( + "member [%v] derived group public key [0x%x] "+ + "instead of [0x%x]", + memberIndex, + publicKeyBytes, + referencePublicKeyBytes, + ) + } + } + + // The reduced final signing group: the legacy seat is dropped and the + // remaining member indexes are remapped to consecutive positions. + walletRegistry, err := newWalletRegistry( + &mockPersistenceHandle{}, + cutoverGroup.localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + registrar := &dkgExecutor{ + groupParameters: groupParameters, + walletRegistry: walletRegistry, + } + + expectedFinalIndexes := map[group.MemberIndex]group.MemberIndex{ + 1: 1, + 2: 2, + } + for _, memberIndex := range liveMembersIndexes { + registeredSigner, err := registrar.registerSigner( + results[memberIndex], + memberIndex, + cutoverGroup.operators, + ) + if err != nil { + t.Fatalf( + "failed to register the signer of member [%v]: [%v]", + memberIndex, + err, + ) + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("final signing group size of member [%v]", memberIndex), + len(liveMembersIndexes), + len(registeredSigner.wallet.signingGroupOperators), + ) + testutils.AssertIntsEqual( + t, + fmt.Sprintf("final member index of member [%v]", memberIndex), + int(expectedFinalIndexes[memberIndex]), + int(registeredSigner.signingGroupMemberIndex), + ) + } + + // The straggler became mismatch and cross-format evidence attributed to + // its operator in the node-local roster. + if mismatches := recorder.counter( + clientinfo.MetricAnnouncerSessionIDMismatchTotal, + ); mismatches < 1 { + t.Errorf("expected at least one mismatch, got [%v]", mismatches) + } + if crossFormat := recorder.counter( + clientinfo.MetricAnnouncerCrossFormatPeerTotal, + ); crossFormat < 1 { + t.Errorf("expected at least one cross-format peer, got [%v]", crossFormat) + } + + rosterSnapshot := roster.Snapshot() + testutils.AssertIntsEqual( + t, + "cutover roster operators", + 1, + len(rosterSnapshot.Peers), + ) + testutils.AssertStringsEqual( + t, + "roster operator address", + string(cutoverGroup.rosterOperators[legacyMemberIndex-1]), + rosterSnapshot.Peers[0].OperatorAddress, + ) +} + +// TestDKGCutover_RealKeyGenerationExcludesTenLegacyPeers proves the full +// ten-misbehaved-seat consequence of the post-cutover split with a real +// transcript: ten prior-release seats keep announcing legacy session IDs +// after the cutover, the security-v2 cohort — exactly the group quorum — +// runs the real tECDSA key-generation protocol without them, and every +// cohort member's result carries real key material together with all ten +// excluded seats as misbehaved-members indexes, the exact result shape whose +// hundred-member equivalent the Solidity suite accepts at the ninety-active +// boundary and punishes with the reward ban. The group size is the largest +// this repository can drive with distinct real pre-parameters per live +// member; TestDKGCutover_PostCutoverSplitExcludesLegacyPeersAtProductionScale +// pins the same exclusion arithmetic at the production hundred-member +// parameters with a stubbed protocol execution, and a real transcript at +// that scale remains outstanding with the not-yet-executed exact-image +// rehearsals. All ten stragglers become mismatch metrics +// and deduplicated roster evidence, and the production result-to-signer +// transformation remaps the four survivors to consecutive final indexes. +func TestDKGCutover_RealKeyGenerationExcludesTenLegacyPeers(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 14, + GroupQuorum: 4, + HonestThreshold: 4, + } + + // A block time roomy enough for the four-party key-generation transcript + // to complete within one attempt's protocol window, race detector + // included. + cutoverGroup := setupDKGCutoverGroup( + t, + groupParameters.GroupSize, + 200*time.Millisecond, + ) + + liveMembersIndexes := []group.MemberIndex{1, 12, 13, 14} + legacyMembersIndexes := []group.MemberIndex{2, 3, 4, 5, 6, 7, 8, 9, 10, 11} + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures( + len(liveMembersIndexes), + ) + if err != nil { + t.Fatal(err) + } + + cutoverBlock := uint64(2) + if err := cutoverGroup.blockCounter.WaitForBlockHeight( + cutoverBlock + 1, + ); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, cutoverGroup.blockCounter, cutoverBlock) + + seed := big.NewInt(0x10E14) + + recorder := newDispatcherMetricsRecorder() + roster, err := participation.NewCutoverPeerRoster( + context.Background(), + cutoverGroup.blockCounter, + 1500, + newCutoverFakeMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(roster.Close) + + // The first live member observes announcement mismatches exactly like + // the production DKG executor wires them: stragglers become metrics and + // roster evidence. + mismatchObserver := announcer.WithSessionMismatchObserver(func( + observedProtocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + logger, + nil, + recorder, + roster, + participation.ModeSecurityV2, + participation.StateOpenSecurityV2.String(), + cutoverGroup.rosterOperators, + observedProtocolID, + sender, + expectedFormat, + observedFormat, + ) + }) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + // Seats 2-11 are prior-release binaries that keep announcing the legacy + // session IDs after the cutover, on the same channel the live members + // use for the ceremony. + for _, legacyMemberIndex := range legacyMembersIndexes { + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(legacyMemberIndex), + fmt.Sprintf("%s-%s", ProtocolName, seed.Text(16)), + cutoverGroup.validator, + fmt.Sprintf("%v-%v", ProtocolName, "dkg"), + legacyMemberIndex, + []string{ + compatibility.Legacy().DKGSessionID(seed, 1), + compatibility.Legacy().DKGSessionID(seed, 2), + }, + ) + } + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 300*time.Second, + ) + defer cancelLoopCtx() + + outcomes := make( + chan *dkgCutoverMemberOutcome, + len(liveMembersIndexes), + ) + for i, memberIndex := range liveMembersIndexes { + tecdsaExecutor := newSeededTecdsaExecutor( + t, + testData[i].LocalPreParams, + ) + + var announcerOptions []announcer.Option + if memberIndex == liveMembersIndexes[0] { + announcerOptions = append(announcerOptions, mismatchObserver) + } + + go runRealDKGCutoverMember( + loopCtx, + cutoverGroup, + gate, + groupParameters, + seed, + cutoverBlock, + memberIndex, + tecdsaExecutor, + outcomes, + announcerOptions..., + ) + } + + results := make(map[group.MemberIndex]*dkg.Result) + for i := 0; i < len(liveMembersIndexes); i++ { + outcome := <-outcomes + if outcome.err != nil { + t.Fatalf( + "member [%v] failed: [%v]", + outcome.memberIndex, + outcome.err, + ) + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("attempts of member [%v]", outcome.memberIndex), + 1, + len(outcome.sessionIDs), + ) + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt session ID of member [%v]", outcome.memberIndex), + compatibility.SecurityV2().DKGSessionID(seed, 1), + outcome.sessionIDs[0], + ) + + results[outcome.memberIndex] = outcome.result + } + cancelPeers() + + referencePublicKeyBytes, err := results[1].GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + if len(referencePublicKeyBytes) == 0 { + t.Fatal("expected non-empty group public key bytes") + } + for _, memberIndex := range liveMembersIndexes { + result := results[memberIndex] + + // The real result must report every one of the ten excluded seats — + // and nothing else — as misbehaved. + misbehaved := result.MisbehavedMembersIndexes() + testutils.AssertIntsEqual( + t, + fmt.Sprintf("misbehaved members of member [%v]", memberIndex), + len(legacyMembersIndexes), + len(misbehaved), + ) + for i, legacyMemberIndex := range legacyMembersIndexes { + testutils.AssertIntsEqual( + t, + fmt.Sprintf( + "misbehaved member at position [%v] of member [%v]", + i, + memberIndex, + ), + int(legacyMemberIndex), + int(misbehaved[i]), + ) + } + + publicKeyBytes, err := result.GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(referencePublicKeyBytes, publicKeyBytes) { + t.Errorf( + "member [%v] derived group public key [0x%x] "+ + "instead of [0x%x]", + memberIndex, + publicKeyBytes, + referencePublicKeyBytes, + ) + } + } + + // The reduced final signing group: the ten legacy seats are dropped and + // the four remaining member indexes are remapped to consecutive + // positions. + walletRegistry, err := newWalletRegistry( + &mockPersistenceHandle{}, + cutoverGroup.localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + registrar := &dkgExecutor{ + groupParameters: groupParameters, + walletRegistry: walletRegistry, + } + + expectedFinalIndexes := map[group.MemberIndex]group.MemberIndex{ + 1: 1, + 12: 2, + 13: 3, + 14: 4, + } + for _, memberIndex := range liveMembersIndexes { + registeredSigner, err := registrar.registerSigner( + results[memberIndex], + memberIndex, + cutoverGroup.operators, + ) + if err != nil { + t.Fatalf( + "failed to register the signer of member [%v]: [%v]", + memberIndex, + err, + ) + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("final signing group size of member [%v]", memberIndex), + len(liveMembersIndexes), + len(registeredSigner.wallet.signingGroupOperators), + ) + testutils.AssertIntsEqual( + t, + fmt.Sprintf("final member index of member [%v]", memberIndex), + int(expectedFinalIndexes[memberIndex]), + int(registeredSigner.signingGroupMemberIndex), + ) + } + + // Every straggler became mismatch and cross-format evidence attributed + // to its operator in the node-local roster, deduplicated to the ten + // distinct operators. + if mismatches := recorder.counter( + clientinfo.MetricAnnouncerSessionIDMismatchTotal, + ); mismatches < float64(len(legacyMembersIndexes)) { + t.Errorf( + "expected at least [%v] mismatches, got [%v]", + len(legacyMembersIndexes), + mismatches, + ) + } + if crossFormat := recorder.counter( + clientinfo.MetricAnnouncerCrossFormatPeerTotal, + ); crossFormat < float64(len(legacyMembersIndexes)) { + t.Errorf( + "expected at least [%v] cross-format peers, got [%v]", + len(legacyMembersIndexes), + crossFormat, + ) + } + + rosterSnapshot := roster.Snapshot() + testutils.AssertIntsEqual( + t, + "cutover roster operators", + len(legacyMembersIndexes), + len(rosterSnapshot.Peers), + ) + rosterOperatorAddresses := make(map[string]bool) + for _, peer := range rosterSnapshot.Peers { + rosterOperatorAddresses[peer.OperatorAddress] = true + } + for _, legacyMemberIndex := range legacyMembersIndexes { + operatorAddress := string( + cutoverGroup.rosterOperators[legacyMemberIndex-1], + ) + if !rosterOperatorAddresses[operatorAddress] { + t.Errorf( + "legacy seat [%v] operator [%s] missing from the roster", + legacyMemberIndex, + operatorAddress, + ) + } + } +} + +// TestDKGCutover_SplitBelowQuorumNeverStartsProtocol proves the quorum +// discipline of the post-cutover split: when the security-v2 cohort is below +// the group quorum because prior-release peers keep announcing legacy session +// IDs, the production retry loop never starts the DKG protocol at all, and +// every straggler is reported into mismatch metrics. +func TestDKGCutover_SplitBelowQuorumNeverStartsProtocol(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + } + + cutoverGroup := setupDKGCutoverGroup(t, groupParameters.GroupSize, 20*time.Millisecond) + blockCounter := cutoverGroup.blockCounter + + gate := newTestGate(t, blockCounter) + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + permit, err := gate.Begin(participation.TBTCDKG, anchor) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + seed := big.NewInt(0x6655) + protocolID := fmt.Sprintf("%v-%v", ProtocolName, "dkg") + channelName := "dkg-cutover-split-noquorum-test" + + channel, err := cutoverGroup.provider(1).BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + recorder := newDispatcherMetricsRecorder() + + currentMode := permit.Mode() + loopAnnouncer := announcer.New( + protocolID, + channel, + cutoverGroup.validator, + announcer.WithSessionMismatchObserver(func( + observedProtocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + logger, + nil, + recorder, + nil, + currentMode, + participation.StateOpenSecurityV2.String(), + cutoverGroup.rosterOperators, + observedProtocolID, + sender, + expectedFormat, + observedFormat, + ) + }), + ) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + // Only member 2 is a current security-v2 peer — together with the local + // member that is 2 ready members, below the quorum of 4. Members 3-5 are + // prior-release binaries announcing legacy session IDs. + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(2), + channelName, + cutoverGroup.validator, + protocolID, + group.MemberIndex(2), + []string{compatibility.SecurityV2().DKGSessionID(seed, 1)}, + ) + for _, memberIndex := range []group.MemberIndex{3, 4, 5} { + startPeerAnnouncer( + peersCtx, + t, + cutoverGroup.provider(memberIndex), + channelName, + cutoverGroup.validator, + protocolID, + memberIndex, + []string{compatibility.Legacy().DKGSessionID(seed, 1)}, + ) + } + + retryLoop := newDkgRetryLoop( + logger, + seed, + permit.Mode(), + anchor, + group.MemberIndex(1), + cutoverGroup.rosterOperators, + groupParameters, + loopAnnouncer, + 1, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 30*time.Second, + ) + defer cancelLoopCtx() + + var attemptCalls atomic.Uint64 + + _, err = retryLoop.start( + loopCtx, + newChainWaitForBlockFn(blockCounter), + func(attempt *dkgAttemptParams) (*dkg.Result, error) { + attemptCalls.Add(1) + return nil, fmt.Errorf("must never be reached") + }, + ) + cancelPeers() + + if err == nil { + t.Fatal("expected the loop to end without a result") + } + testutils.AssertUintsEqual( + t, + "DKG protocol invocations below quorum", + 0, + attemptCalls.Load(), + ) + + if mismatches := recorder.counter( + clientinfo.MetricAnnouncerSessionIDMismatchTotal, + ); mismatches < 3 { + t.Errorf("expected at least three mismatches, got [%v]", mismatches) + } + if crossFormat := recorder.counter( + clientinfo.MetricAnnouncerCrossFormatPeerTotal, + ); crossFormat < 3 { + t.Errorf("expected at least three cross-format peers, got [%v]", crossFormat) + } +} + +// TestDKGCutover_GateQuiesceAbortSkipsOrdinaryDKGFailureMetrics proves the +// smoke-gate-2 metric-neutrality rule through the production DKG executor: a +// member goroutine ended by the gate's forced quiesce deadline does not +// increment the ordinary DKG failure counter. +func TestDKGCutover_GateQuiesceAbortSkipsOrdinaryDKGFailureMetrics(t *testing.T) { + localChain := Connect(20 * time.Millisecond) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + + _, operatorPublicKey, err := operator.GenerateKeyPair(local_v1.DefaultCurve) + if err != nil { + t.Fatal(err) + } + + recorder := newDispatcherMetricsRecorder() + + de := &dkgExecutor{ + groupParameters: &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + }, + chain: localChain, + netProvider: local.ConnectWithKey(operatorPublicKey), + protocolLatch: generator.NewProtocolLatch(), + waitForBlockFn: newChainWaitForBlockFn(blockCounter), + participationGate: gate, + metricsRecorder: recorder, + signerQuarantine: newSignerQuarantine( + context.Background(), + logger, + &mockPersistenceHandle{}, + ), + } + + gsr := &GroupSelectionResult{ + OperatorsIDs: chain.OperatorIDs{1, 2, 3, 4, 5}, + OperatorsAddresses: chain.Addresses{ + "0xAA", "0xBB", "0xCC", "0xDD", "0xEE", + }, + } + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + // The single controlled member joins the DKG and blocks in the + // announcement phase: the other four members never announce. + de.generateSigningGroup( + logger.With(), + big.NewInt(0x11), + []uint8{1}, + gsr, + anchor, + 0, + ) + + // Wait for the member permit to be active, then force the quiesce + // deadline while the member goroutine is still in flight. + waitForActiveCeremonies := func(expected uint64) { + t.Helper() + deadline := time.Now().Add(15 * time.Second) + for gate.State().ActiveCeremonies != expected { + if time.Now().After(deadline) { + t.Fatalf( + "gate never reached [%d] active ceremonies", + expected, + ) + } + time.Sleep(5 * time.Millisecond) + } + } + + waitForActiveCeremonies(1) + gate.Quiesce(fmt.Errorf("rollback drill")) + gate.Close() + waitForActiveCeremonies(0) + + testutils.AssertIntsEqual( + t, + "ordinary DKG failures after the gate abort", + 0, + int(recorder.counter(clientinfo.MetricDKGFailedTotal)), + ) +} diff --git a/pkg/tbtc/dkg_loop.go b/pkg/tbtc/dkg_loop.go index 4b7955abc9..125f8af4b2 100644 --- a/pkg/tbtc/dkg_loop.go +++ b/pkg/tbtc/dkg_loop.go @@ -5,12 +5,14 @@ import ( "crypto/sha256" "encoding/binary" "fmt" - "github.com/keep-network/keep-core/pkg/protocol/announcer" "math/big" "github.com/ipfs/go-log/v2" "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa/dkg" "github.com/keep-network/keep-core/pkg/tecdsa/retry" "golang.org/x/exp/slices" @@ -60,6 +62,12 @@ type dkgRetryLoop struct { // Used for the announcement. It never changes. seed *big.Int + // protocolMode is the ceremony's pinned protocol compatibility mode. A + // retry is a phase of its outer ceremony, so every attempt of this loop — + // including attempts starting at or after the cutover block — derives its + // session ID from this one immutable mode. + protocolMode participation.ProtocolMode + memberIndex group.MemberIndex selectedOperators chain.Addresses @@ -80,6 +88,7 @@ type dkgRetryLoop struct { func newDkgRetryLoop( logger log.StandardLogger, seed *big.Int, + protocolMode participation.ProtocolMode, initialStartBlock uint64, memberIndex group.MemberIndex, selectedOperators chain.Addresses, @@ -97,6 +106,7 @@ func newDkgRetryLoop( return &dkgRetryLoop{ logger: logger, seed: seed, + protocolMode: protocolMode, memberIndex: memberIndex, selectedOperators: selectedOperators, groupParameters: groupParameters, @@ -115,6 +125,33 @@ type dkgAttemptParams struct { startBlock uint64 timeoutBlock uint64 excludedMembersIndexes []group.MemberIndex + // sessionID is the GG20 session identifier shared by the announcer and the + // DKG protocol for this attempt. Computed once per attempt by the retry + // loop so both sides cannot drift. + sessionID string +} + +// dkgAttemptSessionID derives the announcer/protocol session ID of a single +// DKG attempt for the given protocol compatibility mode. The exact per-mode +// formats are owned by the compatibility strategy bundle: the legacy form is +// byte-for-byte the pre-hardening production form so a legacy-mode ceremony +// interoperates with prior-release peers; the security-v2 form carries the +// protocol name and a fixed-width attempt so it cannot collide or be replayed +// across protocols. The mode always comes from the ceremony's pinned permit +// mode; there is no implicit default. +func dkgAttemptSessionID( + mode participation.ProtocolMode, + seed *big.Int, + attemptNumber uint, +) string { + strategies, err := compatibility.StrategiesFor(mode) + if err != nil { + panic(fmt.Sprintf( + "dkgAttemptSessionID: protocol mode not set explicitly: [%v]", + err, + )) + } + return strategies.DKGSessionID(seed, attemptNumber) } // dkgAttemptFn represents a function performing a DKG attempt. @@ -194,10 +231,18 @@ func (drl *dkgRetryLoop) start( drl.attemptCounter, ) + // Derive the session ID once per attempt so the announcer and the DKG + // protocol cannot drift apart. + sessionID := dkgAttemptSessionID( + drl.protocolMode, + drl.seed, + drl.attemptCounter, + ) + readyMembersIndexes, err := drl.announcer.Announce( announceCtx, drl.memberIndex, - fmt.Sprintf("%v-%v", drl.seed, drl.attemptCounter), + sessionID, ) if err != nil { drl.logger.Warnf( @@ -271,6 +316,7 @@ func (drl *dkgRetryLoop) start( startBlock: announcementEndBlock, timeoutBlock: timeoutBlock, excludedMembersIndexes: excludedMembersIndexes, + sessionID: sessionID, }) } else { drl.logger.Infof( diff --git a/pkg/tbtc/dkg_loop_test.go b/pkg/tbtc/dkg_loop_test.go index 779b3ac184..a2a99533c1 100644 --- a/pkg/tbtc/dkg_loop_test.go +++ b/pkg/tbtc/dkg_loop_test.go @@ -12,6 +12,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/internal/tecdsatest" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/dkg" ) @@ -84,6 +85,7 @@ func TestDkgRetryLoop(t *testing.T) { startBlock: 211, timeoutBlock: 411, // start block + 200 excludedMembersIndexes: []group.MemberIndex{}, + sessionID: dkgAttemptSessionID(participation.ModeSecurityV2, seed, 1), }, }, "success on initial attempt with missing announcements and quorum": { @@ -109,6 +111,7 @@ func TestDkgRetryLoop(t *testing.T) { startBlock: 211, timeoutBlock: 411, // start block + 200 excludedMembersIndexes: []group.MemberIndex{9, 10}, + sessionID: dkgAttemptSessionID(participation.ModeSecurityV2, seed, 1), }, }, "missing announcements without quorum on initial attempt": { @@ -117,7 +120,7 @@ func TestDkgRetryLoop(t *testing.T) { return context.WithTimeout(context.Background(), 10*time.Second) }, incomingAnnouncementsFn: func(sessionID string) ([]group.MemberIndex, error) { - if sessionID == fmt.Sprintf("%v-%v", seed, 1) { + if sessionID == dkgAttemptSessionID(participation.ModeSecurityV2, seed, 1) { // Non-quorum of members announced their readiness. return []group.MemberIndex{1, 2, 3, 4, 5, 6, 7}, nil } @@ -137,6 +140,7 @@ func TestDkgRetryLoop(t *testing.T) { startBlock: 427, // 211 + 1 * (11 + 200 + 5) timeoutBlock: 627, // start block + 200 excludedMembersIndexes: []group.MemberIndex{2, 5}, + sessionID: dkgAttemptSessionID(participation.ModeSecurityV2, seed, 2), }, }, "announcement error on initial attempt": { @@ -145,7 +149,7 @@ func TestDkgRetryLoop(t *testing.T) { return context.WithTimeout(context.Background(), 10*time.Second) }, incomingAnnouncementsFn: func(sessionID string) ([]group.MemberIndex, error) { - if sessionID == fmt.Sprintf("%v-%v", seed, 1) { + if sessionID == dkgAttemptSessionID(participation.ModeSecurityV2, seed, 1) { return nil, fmt.Errorf("unexpected error") } @@ -163,6 +167,7 @@ func TestDkgRetryLoop(t *testing.T) { startBlock: 427, // 211 + 1 * (11 + 200 + 5) timeoutBlock: 627, // start block + 200 excludedMembersIndexes: []group.MemberIndex{2, 5}, + sessionID: dkgAttemptSessionID(participation.ModeSecurityV2, seed, 2), }, }, "DKG error on initial attempt": { @@ -192,6 +197,7 @@ func TestDkgRetryLoop(t *testing.T) { startBlock: 427, // 211 + 1 * (11 + 200 + 5) timeoutBlock: 627, // start block + 200 excludedMembersIndexes: []group.MemberIndex{2, 5}, + sessionID: dkgAttemptSessionID(participation.ModeSecurityV2, seed, 2), }, }, "executing member excluded": { @@ -221,6 +227,7 @@ func TestDkgRetryLoop(t *testing.T) { startBlock: 643, // 211 + 2 * (11 + 200 + 5) timeoutBlock: 843, // start block + 200 excludedMembersIndexes: []group.MemberIndex{9}, + sessionID: dkgAttemptSessionID(participation.ModeSecurityV2, seed, 3), }, }, "loop context done": { @@ -249,7 +256,7 @@ func TestDkgRetryLoop(t *testing.T) { }, incomingAnnouncementsFn: func(sessionID string) ([]group.MemberIndex, error) { // Force the first attempt's announcement failure. - if sessionID == fmt.Sprintf("%v-%v", seed, 1) { + if sessionID == dkgAttemptSessionID(participation.ModeSecurityV2, seed, 1) { return nil, fmt.Errorf("unexpected error") } @@ -275,6 +282,7 @@ func TestDkgRetryLoop(t *testing.T) { retryLoop := newDkgRetryLoop( &testutils.MockLogger{}, seed, + participation.ModeSecurityV2, 200, test.memberIndex, selectedOperators, @@ -354,6 +362,34 @@ func TestDkgRetryLoop(t *testing.T) { } } +func TestDkgAttemptSessionIDHasMinimumEntropyWidth(t *testing.T) { + seed := big.NewInt(100) + + sessionID := dkgAttemptSessionID(participation.ModeSecurityV2, seed, 1) + + testutils.AssertStringsEqual( + t, + "session ID format", + "dkg-64-0000000000000001", + sessionID, + ) + if len(sessionID) < 16 { + t.Fatal("DKG session ID must satisfy tss-lib SetSessionNonceBytes minimum length") + } + + // The smallest possible inputs must still clear the tss-lib floor; this + // guards against a future format change silently regressing below 16 bytes. + minSessionID := dkgAttemptSessionID(participation.ModeSecurityV2, big.NewInt(0), 0) + if len(minSessionID) < 16 { + t.Fatalf( + "DKG session ID for minimum inputs must satisfy tss-lib "+ + "SetSessionNonceBytes minimum length, got [%v] (%d bytes)", + minSessionID, + len(minSessionID), + ) + } +} + type mockDkgAnnouncer struct { // outgoingAnnouncements holds all announcements that are sent by the // announcer. diff --git a/pkg/tbtc/dkg_submit.go b/pkg/tbtc/dkg_submit.go index eb244d17aa..aeaac6b5e1 100644 --- a/pkg/tbtc/dkg_submit.go +++ b/pkg/tbtc/dkg_submit.go @@ -6,6 +6,7 @@ import ( "github.com/ipfs/go-log/v2" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa/dkg" ) @@ -83,6 +84,10 @@ type dkgResultSubmitter struct { groupSelectionResult *GroupSelectionResult waitForBlockFn waitForBlockFn + + // commitGuard fences the terminal on-chain submission: a refusal is a + // release-gate decision, not an ordinary submission failure. + commitGuard participation.CommitGuard } func newDkgResultSubmitter( @@ -91,6 +96,7 @@ func newDkgResultSubmitter( groupParameters *GroupParameters, groupSelectionResult *GroupSelectionResult, waitForBlockFn waitForBlockFn, + commitGuard participation.CommitGuard, ) *dkgResultSubmitter { return &dkgResultSubmitter{ dkgLogger: dkgLogger, @@ -98,6 +104,7 @@ func newDkgResultSubmitter( groupSelectionResult: groupSelectionResult, groupParameters: groupParameters, waitForBlockFn: waitForBlockFn, + commitGuard: commitGuard, } } @@ -228,5 +235,14 @@ func (drs *dkgResultSubmitter) SubmitResult( len(signatures), ) + // The last-moment fence immediately before the irreversible on-chain + // submission. + if err := drs.commitGuard.CheckCommit( + "tbtc_dkg_result_submission", + participation.CompletionCommit, + ); err != nil { + return err + } + return drs.chain.SubmitDKGResult(dkgResult) } diff --git a/pkg/tbtc/dkg_submit_test.go b/pkg/tbtc/dkg_submit_test.go index 9ced7783fa..53d4740711 100644 --- a/pkg/tbtc/dkg_submit_test.go +++ b/pkg/tbtc/dkg_submit_test.go @@ -14,6 +14,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/internal/tecdsatest" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/dkg" ) @@ -273,6 +274,7 @@ func TestSubmitResult_MemberSubmitsResult(t *testing.T) { groupParameters, groupSelectionResult, testWaitForBlockFn(localChain), + newTestPermit(participation.TBTCDKG), ) testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) @@ -366,6 +368,7 @@ func TestSubmitResult_AnotherMemberSubmitsResult(t *testing.T) { groupParameters, groupSelectionResult, testWaitForBlockFn(localChain), + newTestPermit(participation.TBTCDKG), ) testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) @@ -500,6 +503,7 @@ func TestSubmitResult_InvalidResult(t *testing.T) { groupParameters, groupSelectionResult, testWaitForBlockFn(localChain), + newTestPermit(participation.TBTCDKG), ) testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) @@ -586,6 +590,7 @@ func TestSubmitResult_ContextCancelled(t *testing.T) { groupParameters, groupSelectionResult, testWaitForBlockFn(localChain), + newTestPermit(participation.TBTCDKG), ) testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) @@ -668,6 +673,7 @@ func TestSubmitResult_TooFewSignatures(t *testing.T) { groupParameters, groupSelectionResult, testWaitForBlockFn(localChain), + newTestPermit(participation.TBTCDKG), ) testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) @@ -809,6 +815,7 @@ func TestSubmitResult_StateChangesDuringWait(t *testing.T) { groupParameters, groupSelectionResult, hookedWaitForBlockFn, + newTestPermit(participation.TBTCDKG), ) ctx, cancelCtx := context.WithCancel(context.Background()) diff --git a/pkg/tbtc/dkg_test.go b/pkg/tbtc/dkg_test.go index b177e03d10..6ead778a3c 100644 --- a/pkg/tbtc/dkg_test.go +++ b/pkg/tbtc/dkg_test.go @@ -2,6 +2,8 @@ package tbtc import ( "context" + "crypto/sha256" + "encoding/hex" "fmt" "math/big" "reflect" @@ -20,10 +22,361 @@ import ( "github.com/keep-network/keep-core/pkg/net/local" "github.com/keep-network/keep-core/pkg/operator" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/dkg" ) +func TestTBTCDKGPermitIdentityIsCanonical(t *testing.T) { + seed := big.NewInt(123456789) + seedHash := sha256.Sum256(seed.Bytes()) + + identity := tbtcDKGPermitIdentity(seed, group.MemberIndex(17)) + + if identity.WorkID != hex.EncodeToString(seedHash[:]) { + t.Errorf("unexpected DKG work ID [%s]", identity.WorkID) + } + if identity.PermitID != "17" { + t.Errorf("unexpected DKG permit ID [%s]", identity.PermitID) + } + if !slices.Equal( + identity.OperatedMembers, + participation.MemberIndexes{17}, + ) { + t.Errorf( + "unexpected operated memberships [%v]", + identity.OperatedMembers, + ) + } +} + +// TestDKGTranscriptContributionMapsBackToThePermitSpace asserts the transcript a +// completed tBTC DKG records lines its final signing group seats up with the DKG +// seats its permits were issued for. +// +// This is the one ceremony whose record and permits live in different index +// spaces: the group is rebuilt from the members this node saw operating, so every +// seat above a removed one shifts down. The mapping is what lets a reader join +// the two, and it has to come from the same accepted result the final group was +// built from — a mapping derived any other way would place seats by coincidence +// wherever no member was removed and silently misplace them wherever one was. +func TestDKGTranscriptContributionMapsBackToThePermitSpace(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 3, + HonestThreshold: 3, + } + + selectedOperators := []chain.Address{ + "0xAA", + "0xBB", + "0xCC", + "0xDD", + "0xEE", + } + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + + var tests = map[string]struct { + memberIndex group.MemberIndex + inactiveMemberIndexes []group.MemberIndex + disqualifiedMemberIndexes []group.MemberIndex + + expectedIncorporated participation.MemberIndexes + expectedLocal participation.MemberIndexes + expectedPermitSpace participation.MemberIndexes + }{ + // Nothing was removed, so the two spaces coincide. The mapping still + // has to be there: a reader cannot tell this case from a remapped one + // without it, and its absence is what would send the reader back to + // comparing raw numbers. + "the whole selected group operated": { + memberIndex: 4, + expectedIncorporated: participation.MemberIndexes{1, 2, 3, 4, 5}, + expectedLocal: participation.MemberIndexes{4}, + expectedPermitSpace: participation.MemberIndexes{1, 2, 3, 4, 5}, + }, + // The case a raw comparison gets wrong. DKG seat 4 lands in final seat + // 3, and final seat 3 belongs to DKG seat 4 rather than to whichever + // node ran DKG seat 3. + "a middle member was not seen operating": { + memberIndex: 4, + inactiveMemberIndexes: []group.MemberIndex{2}, + expectedIncorporated: participation.MemberIndexes{1, 2, 3, 4}, + expectedLocal: participation.MemberIndexes{3}, + expectedPermitSpace: participation.MemberIndexes{1, 3, 4, 5}, + }, + "a middle member was disqualified": { + memberIndex: 5, + disqualifiedMemberIndexes: []group.MemberIndex{3}, + expectedIncorporated: participation.MemberIndexes{1, 2, 3, 4}, + expectedLocal: participation.MemberIndexes{4}, + expectedPermitSpace: participation.MemberIndexes{1, 2, 4, 5}, + }, + "members were removed from both ends": { + memberIndex: 3, + inactiveMemberIndexes: []group.MemberIndex{1}, + disqualifiedMemberIndexes: []group.MemberIndex{5}, + expectedIncorporated: participation.MemberIndexes{1, 2, 3}, + expectedLocal: participation.MemberIndexes{2}, + expectedPermitSpace: participation.MemberIndexes{2, 3, 4}, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + persistenceHandle := &mockPersistenceHandle{} + localChain := Connect() + walletRegistry, err := newWalletRegistry( + persistenceHandle, + localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + dkgExecutor := &dkgExecutor{ + groupParameters: groupParameters, + chain: localChain, + walletRegistry: walletRegistry, + } + + dkgGroup := group.NewGroup( + groupParameters.DishonestThreshold(), + groupParameters.GroupSize, + ) + for _, inactive := range test.inactiveMemberIndexes { + dkgGroup.MarkMemberAsInactive(inactive) + } + for _, disqualified := range test.disqualifiedMemberIndexes { + dkgGroup.MarkMemberAsDisqualified(disqualified) + } + + result := &dkg.Result{ + Group: dkgGroup, + PrivateKeyShare: tecdsa.NewPrivateKeyShare(testData[0]), + } + + signer, err := dkgExecutor.buildFinalSigner( + result, + test.memberIndex, + selectedOperators, + ) + if err != nil { + t.Fatal(err) + } + + contribution := dkgTranscriptContribution(signer, result) + + if !slices.Equal( + contribution.IncorporatedMembers, + test.expectedIncorporated, + ) { + t.Errorf( + "unexpected incorporated memberships\n"+ + "expected: %v\n"+ + "actual: %v\n", + test.expectedIncorporated, + contribution.IncorporatedMembers, + ) + } + if !slices.Equal(contribution.LocalMembers, test.expectedLocal) { + t.Errorf( + "unexpected local memberships\n"+ + "expected: %v\n"+ + "actual: %v\n", + test.expectedLocal, + contribution.LocalMembers, + ) + } + if !slices.Equal( + contribution.PermitSpaceMembers, + test.expectedPermitSpace, + ) { + t.Errorf( + "unexpected permit-space memberships\n"+ + "expected: %v\n"+ + "actual: %v\n", + test.expectedPermitSpace, + contribution.PermitSpaceMembers, + ) + } + + // The mapping is only worth carrying if it is the one the gate + // checks the record against, so the record this node would write is + // held to the permit this node was issued exactly as the gate holds + // it. + if err := participation.ValidatePermitOperatedOwnership( + participation.TBTCDKG, + participation.MemberIndexes{test.memberIndex}, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidencePersistedTBTCSinger, + Reference: getWalletStorageKey( + signer.wallet.publicKey, + ), + MembershipIndex: signer.signingGroupMemberIndex, + Contribution: contribution, + }, + ); err != nil { + t.Errorf( + "the transcript this node wrote was refused against the "+ + "DKG seat its own permit was issued for: [%v]", + err, + ) + } + }) + } +} + +// The result a member activates its own key material against has to be the +// result it generated. +// +// Activation persists a share and enters it in the wallet cache under the final +// group the local result describes. A subscription read as nothing but +// "something settled" makes that a claim about chain state nobody looked at: +// another ceremony settling, or this one settling on a group rebuilt from a +// different membership, satisfies it just as well — and the node is then holding +// an active signer for a wallet, or a seat, the chain does not agree with. +func TestDKGResultSettledLocalCeremony(t *testing.T) { + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 3, + HonestThreshold: 3, + } + + seed := big.NewInt(123456789) + memberIndex := group.MemberIndex(3) + + localGroup := group.NewGroup( + groupParameters.DishonestThreshold(), + groupParameters.GroupSize, + ) + localGroup.MarkMemberAsInactive(group.MemberIndex(2)) + result := &dkg.Result{ + Group: localGroup, + PrivateKeyShare: tecdsa.NewPrivateKeyShare(testData[0]), + } + localGroupPublicKey, err := result.GroupPublicKeyBytes() + if err != nil { + t.Fatal(err) + } + + // A different key, built by moving one coordinate byte. The shares this + // package's fixtures carry all belong to one group, so a second fixture + // would produce the very key this case has to differ from. + otherGroupPublicKey := slices.Clone(localGroupPublicKey) + otherGroupPublicKey[len(otherGroupPublicKey)-1] ^= 0xff + + settled := func( + seed *big.Int, + groupPublicKey []byte, + misbehaved []group.MemberIndex, + ) *DKGResultSubmittedEvent { + return &DKGResultSubmittedEvent{ + Seed: seed, + Result: &DKGChainResult{ + SubmitterMemberIndex: group.MemberIndex(1), + GroupPublicKey: groupPublicKey, + MisbehavedMembersIndexes: misbehaved, + }, + BlockNumber: 1_000, + } + } + + var tests = map[string]struct { + submitted *DKGResultSubmittedEvent + expected bool + }{ + "this member's own result settled": { + submitted: settled( + seed, + localGroupPublicKey, + []group.MemberIndex{2}, + ), + expected: true, + }, + // The registry stores the coordinate pair without the uncompressed + // prefix the local marshaling carries, so the same key legitimately + // reaches this comparison in two encodings. + "the same key without its uncompressed prefix": { + submitted: settled( + seed, + localGroupPublicKey[1:], + []group.MemberIndex{2}, + ), + expected: true, + }, + "nothing settled": { + submitted: nil, + expected: false, + }, + "an event carrying no result": { + submitted: &DKGResultSubmittedEvent{Seed: seed}, + expected: false, + }, + "another ceremony settled": { + submitted: settled( + big.NewInt(987654321), + localGroupPublicKey, + []group.MemberIndex{2}, + ), + expected: false, + }, + "a result for another group settled": { + submitted: settled( + seed, + otherGroupPublicKey, + []group.MemberIndex{2}, + ), + expected: false, + }, + // The same key, rebuilt from a different membership. The wallet would + // be the right one and this member's seat in it would not. + "a result removing different members settled": { + submitted: settled( + seed, + localGroupPublicKey, + []group.MemberIndex{4}, + ), + expected: false, + }, + "a result removing nobody settled": { + submitted: settled(seed, localGroupPublicKey, nil), + expected: false, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + actual := dkgResultSettledLocalCeremony( + &testutils.MockLogger{}, + memberIndex, + seed, + result, + test.submitted, + ) + if actual != test.expected { + t.Errorf( + "unexpected settlement verdict\n"+ + "expected: %v\n"+ + "actual: %v\n", + test.expected, + actual, + ) + } + }) + } +} + func TestDkgExecutor_RegisterSigner(t *testing.T) { testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) if err != nil { @@ -803,9 +1156,20 @@ func TestDkgExecutor_GenerateSigningGroup_DKGParametersError(t *testing.T) { c := &dkgParamsErrChain{Connect()} netProvider := local.ConnectWithKey(operatorPublicKey) + blockCounter, err := c.BlockCounter() + if err != nil { + t.Fatal(err) + } + de := &dkgExecutor{ - chain: c, - netProvider: netProvider, + chain: c, + netProvider: netProvider, + participationGate: newTestGate(t, blockCounter), + signerQuarantine: newSignerQuarantine( + context.Background(), + logger, + &mockPersistenceHandle{}, + ), } gsr := &GroupSelectionResult{ @@ -847,9 +1211,22 @@ func (c *dkgParamsErrChain) DKGParameters() (*DKGParameters, error) { // generateSigningGroup returns gracefully when the net.Provider fails to // create a broadcast channel. The function exits before spawning goroutines. func TestDkgExecutor_GenerateSigningGroup_BroadcastChannelError(t *testing.T) { + c := Connect() + + blockCounter, err := c.BlockCounter() + if err != nil { + t.Fatal(err) + } + de := &dkgExecutor{ - chain: Connect(), - netProvider: &errNetProvider{}, + chain: c, + netProvider: &errNetProvider{}, + participationGate: newTestGate(t, blockCounter), + signerQuarantine: newSignerQuarantine( + context.Background(), + logger, + &mockPersistenceHandle{}, + ), } gsr := &GroupSelectionResult{ @@ -881,4 +1258,9 @@ func (p *errNetProvider) ConnectionManager() net.ConnectionManager { return nil func (p *errNetProvider) CreateTransportIdentifier(_ *operator.PublicKey) (net.TransportIdentifier, error) { return nil, nil } -func (p *errNetProvider) BroadcastChannelForwarderFor(_ string) {} +func (p *errNetProvider) BroadcastChannelForwarderFor(_ string) ( + net.Forwarder, + error, +) { + return net.NoopForwarder(), nil +} diff --git a/pkg/tbtc/fuzz_test.go b/pkg/tbtc/fuzz_test.go new file mode 100644 index 0000000000..5d37811e7a --- /dev/null +++ b/pkg/tbtc/fuzz_test.go @@ -0,0 +1,73 @@ +package tbtc + +// Coverage-guided fuzz targets for the NETWORK/coordination protobuf +// unmarshalers in marshaling.go. Each asserts that Unmarshal never panics on +// arbitrary bytes: malformed input must return an error, not crash. The +// signer unmarshaler is intentionally excluded (local key material, not +// untrusted network input). + +import "testing" + +func FuzzSigningDoneMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&signingDoneMessage{}).Unmarshal(data) + }) +} + +func FuzzCoordinationMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&coordinationMessage{}).Unmarshal(data) + }) +} + +func FuzzNoopProposalUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&NoopProposal{}).Unmarshal(data) + }) +} + +func FuzzHeartbeatProposalUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&HeartbeatProposal{}).Unmarshal(data) + }) +} + +func FuzzDepositSweepProposalUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&DepositSweepProposal{}).Unmarshal(data) + }) +} + +func FuzzRedemptionProposalUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&RedemptionProposal{}).Unmarshal(data) + }) +} + +func FuzzMovingFundsProposalUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&MovingFundsProposal{}).Unmarshal(data) + }) +} + +func FuzzMovedFundsSweepProposalUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&MovedFundsSweepProposal{}).Unmarshal(data) + }) +} diff --git a/pkg/tbtc/heartbeat.go b/pkg/tbtc/heartbeat.go index c86afd88db..59f15c1eb7 100644 --- a/pkg/tbtc/heartbeat.go +++ b/pkg/tbtc/heartbeat.go @@ -5,11 +5,13 @@ import ( "encoding/hex" "fmt" "math/big" + "slices" "sync" "github.com/ipfs/go-log/v2" "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -60,19 +62,22 @@ type heartbeatSigningExecutor interface { ctx context.Context, message *big.Int, startBlock uint64, - ) (*tecdsa.Signature, *signingActivityReport, uint64, error) + mode participation.ProtocolMode, + ) (*signingOutcome, error) } // heartbeatInactivityClaimExecutor is an interface meant to decouple the // specific implementation of the inactivity claim executor from the heartbeat -// action. +// action. The commit guard is the heartbeat permit's penalty fence: the claim +// is derived penalty work and inherits the heartbeat ceremony's permit. type heartbeatInactivityClaimExecutor interface { claimInactivity( ctx context.Context, + commitGuard participation.CommitGuard, inactiveMembersIndexes []group.MemberIndex, heartbeatFailed bool, sessionID *big.Int, - ) error + ) (inactivityClaimDisposition, error) } // heartbeatAction is a walletAction implementation handling heartbeat requests @@ -93,6 +98,12 @@ type heartbeatAction struct { expiryBlock uint64 waitForBlockFn waitForBlockFn + + // permit is the heartbeat ceremony's participation permit: it pins the + // protocol mode for the heartbeat signing, fences the penalty path of any + // derived inactivity work, and is released when the action's execution + // ends. + permit participation.Permit } func newHeartbeatAction( @@ -106,6 +117,7 @@ func newHeartbeatAction( startBlock uint64, expiryBlock uint64, waitForBlockFn waitForBlockFn, + permit participation.Permit, ) *heartbeatAction { return &heartbeatAction{ logger: logger, @@ -118,10 +130,170 @@ func newHeartbeatAction( startBlock: startBlock, expiryBlock: expiryBlock, waitForBlockFn: waitForBlockFn, + permit: permit, + } +} + +// heartbeatPenaltyState is the penalty state a heartbeat left behind on top of +// its threshold signature. The inactivity claim derived from a low-activity +// heartbeat runs under the heartbeat's own permit, so the heartbeat's terminal +// record is the only place that disposition is reported. A record naming only +// the signature cannot tell a healthy heartbeat apart from one that went on to +// file an inactivity claim against named members, which is exactly the +// distinction the rollback audit has to reconcile. +type heartbeatPenaltyState struct { + // claimDispatched is true once the action handed a claim to the inactivity + // claim executor. It stays true when publishing then failed or was + // suppressed: the audit must treat a dispatched claim as possibly settled. + claimDispatched bool + // inactiveMembers is the exact member set the dispatched claim names. It is + // empty when no claim was dispatched. + inactiveMembers []group.MemberIndex + // claim is what the dispatched claim did about the chain: whether any + // member reached the submitting call, and which claim slot the settlement + // was resolved to. Dispatch intent is neither of those: the claim can fail + // to publish, be suppressed by the release gate, or be canceled before any + // member submits, and the node cannot tell those apart from a claim that + // landed without asking the chain. + claim inactivityClaimDisposition +} + +// chainSettlement renders the penalty state as the chain-settlement record the +// terminal outcome carries. Three cases have to stay distinct. +// +// A resolved settlement names the claim slot it consumed, which is what lets +// the offline audit join it to an authenticated registry log. A submission the +// executor could not resolve is reported with no reference, which makes the +// barrier treat the penalty as unreconciled rather than absent. A dispatch that +// never reached the submitting call left no transaction anywhere and reports no +// settlement at all: calling it unresolved would block a rollback over a +// penalty that provably cannot exist on chain. +func (hps heartbeatPenaltyState) chainSettlement() *participation.ChainSettlementRecord { + if !hps.claimDispatched { + return nil + } + + settlement := &participation.ChainSettlementRecord{ + Kind: participation.ChainSettlementInactivityClaim, + } + + if hps.claim.settlement == nil { + if !hps.claim.submissionAttempted { + return nil + } + return settlement + } + + reference, err := participation.InactivityClaimSettlementReference( + hps.claim.settlement.walletID[:], + hps.claim.settlement.nonce, + ) + if err != nil { + // A resolved settlement that cannot be rendered canonically is + // indistinguishable to the audit from one that was never resolved, and + // the unresolved record is the conservative of the two. + return settlement + } + + settlement.Reference = reference + + return settlement +} + +// inactiveMemberBytes renders the claimed inactive member set as a canonical, +// order-independent byte string so the derived identity does not depend on the +// order the signing activity report happened to produce. +func (hps heartbeatPenaltyState) inactiveMemberBytes() []byte { + if len(hps.inactiveMembers) == 0 { + return nil } + + members := make([]group.MemberIndex, len(hps.inactiveMembers)) + copy(members, hps.inactiveMembers) + slices.Sort(members) + members = slices.Compact(members) + + bytes := make([]byte, len(members)) + for i, member := range members { + bytes[i] = byte(member) + } + + return bytes +} + +// recordTerminalOutcome reports the heartbeat ceremony's node-owned final +// disposition on its participation permit. The heartbeat's durable result is +// the threshold signature it produced over the proposed message, qualified by +// the penalty state the same permit created; a heartbeat that never reached a +// signature — including one the release gate canceled — left no state behind +// and is recorded as exhausted. +func (ha *heartbeatAction) recordTerminalOutcome( + signature *tecdsa.Signature, + transcript *participation.TranscriptContribution, + penalty heartbeatPenaltyState, +) { + if signature == nil { + recordPermitNoThreshold(ha.logger, ha.permit) + return + } + + // A dispatched claim gets its own domain so a healthy heartbeat's identity + // can never be replayed as the settlement of one that claimed inactivity, + // whatever member set the claim happened to name. + domain := "tbtc_heartbeat_signature" + if penalty.claimDispatched { + domain = "tbtc_heartbeat_signature_with_inactivity_claim" + } + + recordPermitTerminalOutcome( + ha.logger, + ha.permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: participation.TerminalResultReference( + domain, + ha.proposal.Message[:], + signatureComponentBytes(signature.R), + signatureComponentBytes(signature.S), + penalty.inactiveMemberBytes(), + ), + // The signature digest above is node-authored end to end. The + // penalty the same permit may have filed is not the node's to + // assert, so it travels separately as chain state the audit + // reconciles against the WalletRegistry. + ChainSettlement: penalty.chainSettlement(), + // Which memberships produced that signature, as this node + // authenticated them. The digest identifies the result; only this + // says who reached it, and every member records the same digest + // whatever population did. + Contribution: transcript, + }, + ) } func (ha *heartbeatAction) execute() error { + // heartbeatSignature holds the ceremony's durable result once signing + // produces one, heartbeatTranscript the memberships that produced it, and + // heartbeatPenalty the penalty state the same permit went on to create. The + // deferred recorder below reads their final values. + var heartbeatSignature *tecdsa.Signature + var heartbeatTranscript *participation.TranscriptContribution + var heartbeatPenalty heartbeatPenaltyState + + // The action owns its permit from dispatch on; releasing it here ends the + // ceremony's active accounting in the participation gate. The terminal + // outcome is registered afterwards so it runs first and reaches the permit + // while it is still open. + defer ha.permit.Close() + defer func() { + ha.recordTerminalOutcome( + heartbeatSignature, + heartbeatTranscript, + heartbeatPenalty, + ) + }() + // Do not execute the heartbeat action if the operator is unstaking. isUnstaking, err := ha.isOperatorUnstaking() if err != nil { @@ -161,29 +333,43 @@ func (ha *heartbeatAction) execute() error { return fmt.Errorf("invalid proposal expiry block") } + // The signing window is bound to the heartbeat permit: a permit + // cancellation — clock failure, forced quiescence — stops the signing + // exactly like the timeout block does. heartbeatSigningCtx, cancelHeartbeatSigningCtx := withCancelOnBlock( - context.Background(), + ha.permit.Context(), ha.expiryBlock-heartbeatInactivityClaimValidityBlocks, ha.waitForBlockFn, ) defer cancelHeartbeatSigningCtx() - signature, activityReport, _, err := ha.signingExecutor.sign( + outcome, err := ha.signingExecutor.sign( heartbeatSigningCtx, messageToSign, ha.startBlock, + ha.permit.Mode(), ) if err != nil { // Do not count this error as heartbeat inactivity failure. If the // process returned an error here, that likely means the group signing // threshold was not met. In such a case, the inactivity claim does not // have a chance for success anyway (it needs the group threshold to - // be met as well). - return fmt.Errorf("heartbeat signing process errored out: [%v]", err) + // be met as well). The wrapped cause lets the dispatcher tell a + // gate-caused abort apart from an ordinary failure. + return fmt.Errorf("heartbeat signing process errored out: [%w]", err) } + // Signing reached the threshold, so the ceremony has a durable result the + // rollback audit can identify, whatever the activity accounting below + // decides about penalties. The transcript is pinned with it: the result and + // the memberships that produced it are one record. + signature := outcome.signature + heartbeatSignature = signature + heartbeatTranscript = outcome.contribution + // If the number of active members during signing was enough, we can // consider the heartbeat procedure as successful. + activityReport := outcome.activityReport activeMembersCount := len(activityReport.activeMembers) if activeMembersCount >= heartbeatSigningMinimumActiveMembers { ha.logger.Infof( @@ -208,6 +394,23 @@ func (ha *heartbeatAction) execute() error { heartbeatSigningMinimumActiveMembers, ) + // The consecutive-failure counter and any derived claim are new penalty + // state. The permit's penalty fence suppresses both for legacy work at or + // after the cutover block and for every permit once process quiescence + // begins, so a boundary- or shutdown-caused low-activity result cannot + // turn into punishment. + if fenceErr := ha.permit.CheckCommit( + "tbtc_heartbeat_inactivity_accounting", + participation.PenaltyCommit, + ); fenceErr != nil { + ha.logger.Warnf( + "heartbeat inactivity penalty suppressed by the release "+ + "gate: [%v]", + fenceErr, + ) + return nil + } + // Increment the heartbeat inactivity failure counter. ha.failureCounter.increment(walletKey) @@ -232,16 +435,26 @@ func (ha *heartbeatAction) execute() error { } heartbeatInactivityCtx, cancelHeartbeatInactivityCtx := withCancelOnBlock( - context.Background(), + ha.permit.Context(), ha.expiryBlock-heartbeatTimeoutSafetyMarginBlocks, ha.waitForBlockFn, ) defer cancelHeartbeatInactivityCtx() + // Pin the penalty state before dispatching, not after: a claim the executor + // may already have published must show up in the terminal record even if + // the call below then errors out or the permit is canceled mid-flight. + heartbeatPenalty = heartbeatPenaltyState{ + claimDispatched: true, + inactiveMembers: activityReport.inactiveMembers, + } + // The value of consecutive heartbeat inactivity failures exceeds the threshold. - // Proceed with operator inactivity claim. - err = ha.inactivityClaimExecutor.claimInactivity( + // Proceed with operator inactivity claim. The claim is derived penalty + // work: it inherits the heartbeat permit as its commit fence. + claimDisposition, err := ha.inactivityClaimExecutor.claimInactivity( heartbeatInactivityCtx, + ha.permit, // It's safe to consider unstaking members as inactive members in the claim. // Inactive members are set ineligible for on-chain rewards for a certain // period of time. This is a desired outcome for unstaking members as well. @@ -249,6 +462,10 @@ func (ha *heartbeatAction) execute() error { true, messageToSign, ) + // A submission made or a settlement resolved on the way to an error still + // happened; recording the disposition before the error path returns keeps + // the terminal record describing the chain rather than the call. + heartbeatPenalty.claim = claimDisposition if err != nil { return fmt.Errorf( "error while notifying about operator inactivity [%v]]", diff --git a/pkg/tbtc/heartbeat_test.go b/pkg/tbtc/heartbeat_test.go index 4d9339c94d..2f0e7f8740 100644 --- a/pkg/tbtc/heartbeat_test.go +++ b/pkg/tbtc/heartbeat_test.go @@ -6,11 +6,11 @@ import ( "encoding/hex" "fmt" "math/big" - "reflect" "testing" "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -71,6 +71,7 @@ func TestHeartbeatAction_HappyPath(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCHeartbeat), ) err = action.execute() @@ -150,6 +151,7 @@ func TestHeartbeatAction_OperatorUnstaking(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCHeartbeat), ) err = action.execute() @@ -213,14 +215,15 @@ func TestHeartbeatAction_Failure_SigningError(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCHeartbeat), ) // Do not expect the execution to result in an error. Signing error does not // mean the procedure failure. err = action.execute() - expectedError := fmt.Errorf("heartbeat signing process errored out: [oofta]") - if !reflect.DeepEqual(expectedError, err) { + expectedError := "heartbeat signing process errored out: [oofta]" + if err == nil || err.Error() != expectedError { t.Errorf( "unexpected error\n"+ "expected: %v\n"+ @@ -292,6 +295,7 @@ func TestHeartbeatAction_Failure_TooFewActiveOperators(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCHeartbeat), ) // Do not expect the execution to result in an error. Signing error does not @@ -372,6 +376,7 @@ func TestHeartbeatAction_Failure_CounterExceeded(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCHeartbeat), ) // Do not expect the execution to result in an error. Signing error does not @@ -453,6 +458,7 @@ func TestHeartbeatAction_Failure_InactivityExecutionFailure(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCHeartbeat), ) err = action.execute() @@ -600,24 +606,387 @@ func TestHeartbeatFailureCounter_Get(t *testing.T) { ) } +// runHeartbeatAction executes one heartbeat action for the given active-member +// count against the given permit and consecutive-failure counter, returning +// the signing and inactivity-claim executors for assertions. +func runHeartbeatAction( + t *testing.T, + hostChain *localChain, + activeMembers uint32, + failureCounter *heartbeatFailureCounter, + startBlock uint64, + permit participation.Permit, +) (*mockHeartbeatSigningExecutor, *mockInactivityClaimExecutor, error) { + t.Helper() + + walletPublicKeyHex, err := hex.DecodeString(heartbeatTestWalletKey()) + if err != nil { + t.Fatal(err) + } + + proposal := &HeartbeatProposal{ + Message: [16]byte{ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + }, + } + hostChain.setHeartbeatProposalValidationResult(proposal, true) + + mockExecutor := &mockHeartbeatSigningExecutor{} + mockExecutor.activeOperatorsCount = activeMembers + + inactivityClaimExecutor := &mockInactivityClaimExecutor{} + + action := newHeartbeatAction( + logger, + hostChain, + wallet{ + publicKey: mustUnmarshalPublicKey(t, walletPublicKeyHex), + }, + mockExecutor, + proposal, + failureCounter, + inactivityClaimExecutor, + startBlock, + startBlock+heartbeatTotalProposalValidityBlocks, + func(ctx context.Context, blockHeight uint64) error { + return nil + }, + permit, + ) + + return mockExecutor, inactivityClaimExecutor, action.execute() +} + +// heartbeatTestWalletKey returns the uncompressed public key hex of the +// wallet used by runHeartbeatAction, which is also its failure-counter key. +func heartbeatTestWalletKey() string { + return "0471e30bca60f6548d7b42582a478ea37ada63b402af7b3ddd57f0c95bb6843175" + + "aa0d2053a91a050a6797d85c38f2909cb7027f2344a01986aa2f9f8ca7a0c289" +} + +// TestHeartbeatAction_InactivityBandMatrixBeforeCutover exercises the +// inactivity band against a real participation gate whose cutover block is +// far ahead, i.e. the pre-cutover fleet state where every heartbeat permit +// pins the legacy mode and penalty accounting follows the normal current +// rules: 51-69 active members produce a signature but count an inactivity +// failure, the third consecutive failure files exactly one claim, and 70 +// active members reset the counter. +func TestHeartbeatAction_InactivityBandMatrixBeforeCutover(t *testing.T) { + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + + blockCounter, err := hostChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, blockCounter, 1_000_000) + + tests := map[string]struct { + activeMembers uint32 + initialFailures uint + expectedFailures uint64 + expectedClaims int + }{ + "51 members sign but count an inactivity failure": { + activeMembers: 51, + initialFailures: 0, + expectedFailures: 1, + expectedClaims: 0, + }, + "60 members sign but count an inactivity failure": { + activeMembers: 60, + initialFailures: 0, + expectedFailures: 1, + expectedClaims: 0, + }, + "69 members sign but count an inactivity failure": { + activeMembers: 69, + initialFailures: 0, + expectedFailures: 1, + expectedClaims: 0, + }, + "70 members reset the counter": { + activeMembers: heartbeatSigningMinimumActiveMembers, + initialFailures: heartbeatConsecutiveFailureThreshold - 1, + expectedFailures: 0, + expectedClaims: 0, + }, + "third consecutive failure files one claim": { + activeMembers: 51, + initialFailures: heartbeatConsecutiveFailureThreshold - 1, + expectedFailures: heartbeatConsecutiveFailureThreshold, + expectedClaims: 1, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + failureCounter := newHeartbeatFailureCounter() + for i := uint(0); i < test.initialFailures; i++ { + failureCounter.increment(heartbeatTestWalletKey()) + } + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + permit, err := gate.Begin(participation.TBTCHeartbeat, anchor) + if err != nil { + t.Fatal(err) + } + testutils.AssertStringsEqual( + t, + "permit mode before the cutover", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + + mockExecutor, inactivityClaimExecutor, err := runHeartbeatAction( + t, + hostChain, + test.activeMembers, + failureCounter, + anchor, + permit, + ) + if err != nil { + t.Fatal(err) + } + + testutils.AssertStringsEqual( + t, + "signing mode", + participation.ModeLegacy.String(), + mockExecutor.requestedMode.String(), + ) + testutils.AssertUintsEqual( + t, + "consecutive failure counter", + test.expectedFailures, + uint64(failureCounter.get(heartbeatTestWalletKey())), + ) + testutils.AssertIntsEqual( + t, + "inactivity claims", + test.expectedClaims, + inactivityClaimExecutor.calls, + ) + }) + } +} + +// TestHeartbeatAction_LegacyAnchorFinishingAfterCutoverSuppressed proves the +// exact boundary rule of the release gate: a heartbeat anchored below the +// cutover block that finishes at or after it neither increments the +// consecutive-failure counter nor files a claim, even when the counter is one +// failure short of the claim threshold. The permit's mode stays legacy for +// its entire lifetime; only the new penalty state is suppressed. +func TestHeartbeatAction_LegacyAnchorFinishingAfterCutoverSuppressed(t *testing.T) { + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + + blockCounter, err := hostChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + cutoverBlock := anchor + 2 + + gate := newTestGateWithCutover(t, blockCounter, cutoverBlock) + + permit, err := gate.Begin(participation.TBTCHeartbeat, anchor) + if err != nil { + t.Fatal(err) + } + testutils.AssertStringsEqual( + t, + "permit mode for the pre-cutover anchor", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + + // One failure short of the claim threshold: a normal low-activity result + // would increment the counter and file a claim. + failureCounter := newHeartbeatFailureCounter() + for i := uint(0); i < heartbeatConsecutiveFailureThreshold-1; i++ { + failureCounter.increment(heartbeatTestWalletKey()) + } + + // The heartbeat finishes at or after the cutover block. + if err := blockCounter.WaitForBlockHeight(cutoverBlock); err != nil { + t.Fatal(err) + } + + mockExecutor, inactivityClaimExecutor, err := runHeartbeatAction( + t, + hostChain, + 51, + failureCounter, + anchor, + permit, + ) + if err != nil { + t.Fatalf("a suppressed penalty must not be an ordinary failure: [%v]", err) + } + + testutils.AssertStringsEqual( + t, + "signing mode", + participation.ModeLegacy.String(), + mockExecutor.requestedMode.String(), + ) + testutils.AssertUintsEqual( + t, + "consecutive failure counter after suppression", + uint64(heartbeatConsecutiveFailureThreshold-1), + uint64(failureCounter.get(heartbeatTestWalletKey())), + ) + testutils.AssertIntsEqual( + t, + "inactivity claims", + 0, + inactivityClaimExecutor.calls, + ) +} + +// TestHeartbeatAction_SecurityV2AtOrAfterCutoverNormalRules proves a +// heartbeat anchored at or after the cutover block pins the security-v2 mode +// and follows the normal current rules: low-activity results increment the +// counter, the third consecutive failure files exactly one claim, and a +// healthy result resets the counter. +func TestHeartbeatAction_SecurityV2AtOrAfterCutoverNormalRules(t *testing.T) { + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + + blockCounter, err := hostChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + + failureCounter := newHeartbeatFailureCounter() + for i := uint(0); i < heartbeatConsecutiveFailureThreshold-1; i++ { + failureCounter.increment(heartbeatTestWalletKey()) + } + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + permit, err := gate.Begin(participation.TBTCHeartbeat, anchor) + if err != nil { + t.Fatal(err) + } + testutils.AssertStringsEqual( + t, + "permit mode at or after the cutover", + participation.ModeSecurityV2.String(), + permit.Mode().String(), + ) + + // The third consecutive low-activity result files exactly one claim. + mockExecutor, inactivityClaimExecutor, err := runHeartbeatAction( + t, + hostChain, + 69, + failureCounter, + anchor, + permit, + ) + if err != nil { + t.Fatal(err) + } + + testutils.AssertStringsEqual( + t, + "signing mode", + participation.ModeSecurityV2.String(), + mockExecutor.requestedMode.String(), + ) + testutils.AssertUintsEqual( + t, + "consecutive failure counter after the third failure", + uint64(heartbeatConsecutiveFailureThreshold), + uint64(failureCounter.get(heartbeatTestWalletKey())), + ) + testutils.AssertIntsEqual( + t, + "inactivity claims", + 1, + inactivityClaimExecutor.calls, + ) + + // A healthy result resets the counter under the same normal rules. + healthyPermit, err := gate.Begin(participation.TBTCHeartbeat, anchor) + if err != nil { + t.Fatal(err) + } + + _, healthyClaimExecutor, err := runHeartbeatAction( + t, + hostChain, + heartbeatSigningMinimumActiveMembers, + failureCounter, + anchor, + healthyPermit, + ) + if err != nil { + t.Fatal(err) + } + + testutils.AssertUintsEqual( + t, + "consecutive failure counter after the healthy heartbeat", + 0, + uint64(failureCounter.get(heartbeatTestWalletKey())), + ) + testutils.AssertIntsEqual( + t, + "inactivity claims after the healthy heartbeat", + 0, + healthyClaimExecutor.calls, + ) +} + type mockHeartbeatSigningExecutor struct { shouldFail bool activeOperatorsCount uint32 requestedMessage *big.Int requestedStartBlock uint64 + requestedMode participation.ProtocolMode } func (mhse *mockHeartbeatSigningExecutor) sign( ctx context.Context, message *big.Int, startBlock uint64, -) (*tecdsa.Signature, *signingActivityReport, uint64, error) { + mode participation.ProtocolMode, +) (*signingOutcome, error) { mhse.requestedMessage = message mhse.requestedStartBlock = startBlock + mhse.requestedMode = mode if mhse.shouldFail { - return nil, nil, 0, fmt.Errorf("oofta") + return nil, fmt.Errorf("oofta") } activeMembers := make([]group.MemberIndex, 0) @@ -636,26 +1005,38 @@ func (mhse *mockHeartbeatSigningExecutor) sign( inactiveMembers: inactiveMembers, } - return &tecdsa.Signature{}, activityReport, startBlock + 1, nil + return &signingOutcome{ + signature: &tecdsa.Signature{}, + activityReport: activityReport, + contribution: mockSigningTranscript(), + endBlock: startBlock + 1, + }, nil } type mockInactivityClaimExecutor struct { shouldFail bool + // disposition is what the executor reports about the chain, independently + // of shouldFail: a claim can be submitted or even settle and the call still + // error. + disposition inactivityClaimDisposition sessionID *big.Int + calls int } func (mice *mockInactivityClaimExecutor) claimInactivity( ctx context.Context, + commitGuard participation.CommitGuard, inactiveMembersIndexes []group.MemberIndex, heartbeatFailed bool, sessionID *big.Int, -) error { +) (inactivityClaimDisposition, error) { mice.sessionID = sessionID + mice.calls++ if mice.shouldFail { - return fmt.Errorf("mock inactivity claim executor error") + return mice.disposition, fmt.Errorf("mock inactivity claim executor error") } - return nil + return mice.disposition, nil } diff --git a/pkg/tbtc/inactivity.go b/pkg/tbtc/inactivity.go index f65c43d995..63655b2a08 100644 --- a/pkg/tbtc/inactivity.go +++ b/pkg/tbtc/inactivity.go @@ -4,8 +4,10 @@ import ( "context" "errors" "fmt" + "math" "math/big" "sync" + "sync/atomic" "github.com/ipfs/go-log/v2" "go.uber.org/zap" @@ -17,6 +19,7 @@ import ( "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/protocol/inactivity" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) const ( @@ -25,6 +28,17 @@ const ( // by the given member to avoid all members submitting the same inactivity claim // at the same time. inactivityClaimSubmissionDelayStepBlocks = 2 + + // inactivityClaimSettlementResolutionBlocks bounds how long the executor + // keeps looking for the settlement of a claim a member already handed to + // the chain. The submitting call returns once the transaction reaches the + // provider and the transaction is mined afterwards, so without a wait past + // the end of publishing the claim's own event routinely arrives after the + // executor stopped listening and a penalty that did land is reported as + // unresolved. The bound is generous enough for an ordinary inclusion and + // short enough to stay well inside the heartbeat window that owns the + // claim, whose context cuts the wait short in any case. + inactivityClaimSettlementResolutionBlocks = 12 ) // errInactivityClaimExecutorBusy is an error returned when the inactivity claim @@ -66,14 +80,57 @@ func newInactivityClaimExecutor( } } +// inactivityClaimSettlement is the canonical identity of an inactivity claim +// that reached the chain: the wallet it was filed against and the nonce it +// consumed. The registry accepts a claim only at the wallet's current nonce and +// increments that nonce in the same call, so the pair names exactly one settled +// claim for all time. +type inactivityClaimSettlement struct { + walletID [32]byte + nonce *big.Int +} + +// inactivityClaimDisposition reports what one claimInactivity call did about +// the chain. The two facts are kept apart because they answer different +// questions and a rollback reads them differently. +// +// submissionAttempted answers whether a claim transaction can exist at all. It +// is false for every exit that provably precedes the submitting call — a busy +// executor, a failed wallet or nonce lookup, a signing round that never reached +// the threshold, a refused penalty fence, a canceled permit — and calling those +// ambiguous would block a rollback over a penalty that cannot be on chain. +// +// settlement answers whether the claim landed, and is nil only while that +// stays genuinely unknown after the executor has both listened for the +// settlement and asked the chain about it. That single remaining case is the +// one the offline barrier must treat as unreconciled. +type inactivityClaimDisposition struct { + submissionAttempted bool + settlement *inactivityClaimSettlement +} + +// claimInactivity signs and submits an operator inactivity claim. The commit +// guard is the owning ceremony's penalty fence: the terminal on-chain +// submission consults it immediately before submitting, so a claim derived +// from legacy work at or after the cutover block, or raced by process +// quiescence, is suppressed instead of creating new penalty state. +// +// The returned disposition is what the caller's terminal record reports about +// the chain. Dispatching a claim proves nothing by itself: publishing can fail, +// be suppressed, or be canceled mid-flight, a member's submission can lose the +// race to another member's, and a submission that does reach the chain is mined +// after the submitting call has already returned. The disposition separates the +// submission that may exist from the settlement that was resolved, so the +// record never has to guess which happened. func (ice *inactivityClaimExecutor) claimInactivity( ctx context.Context, + commitGuard participation.CommitGuard, inactiveMembersIndexes []group.MemberIndex, heartbeatFailed bool, sessionID *big.Int, -) error { +) (inactivityClaimDisposition, error) { if lockAcquired := ice.lock.TryAcquire(1); !lockAcquired { - return errInactivityClaimExecutorBusy + return inactivityClaimDisposition{}, errInactivityClaimExecutorBusy } defer ice.lock.Release(1) @@ -82,7 +139,10 @@ func (ice *inactivityClaimExecutor) claimInactivity( walletPublicKeyHash := bitcoin.PublicKeyHash(wallet.publicKey) walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { - return fmt.Errorf("cannot marshal wallet public key: [%v]", err) + return inactivityClaimDisposition{}, fmt.Errorf( + "cannot marshal wallet public key: [%v]", + err, + ) } execLogger := logger.With( @@ -92,14 +152,20 @@ func (ice *inactivityClaimExecutor) claimInactivity( walletRegistryData, err := ice.chain.GetWallet(walletPublicKeyHash) if err != nil { - return fmt.Errorf("could not get registry data on wallet: [%v]", err) + return inactivityClaimDisposition{}, fmt.Errorf( + "could not get registry data on wallet: [%v]", + err, + ) } nonce, err := ice.chain.GetInactivityClaimNonce( walletRegistryData.EcdsaWalletID, ) if err != nil { - return fmt.Errorf("could not get nonce for wallet: [%v]", err) + return inactivityClaimDisposition{}, fmt.Errorf( + "could not get nonce for wallet: [%v]", + err, + ) } claim := inactivity.NewClaimPreimage( @@ -111,9 +177,57 @@ func (ice *inactivityClaimExecutor) claimInactivity( groupMembers, err := ice.getWalletOperatorsIDs() if err != nil { - return fmt.Errorf("could not get wallet members info: [%v]", err) + return inactivityClaimDisposition{}, fmt.Errorf( + "could not get wallet members info: [%v]", + err, + ) } + // The wallet identity and nonce this claim is built for are what makes an + // observed event attributable to it. The registry accepts a claim only at + // the wallet's current nonce and increments it in the same call, so exactly + // one on-chain claim can ever carry this pair. + settlement := newInactivityClaimSettlementObserver( + walletRegistryData.EcdsaWalletID, + nonce, + ) + + // Publishing runs under one context and one subscription for the whole + // call. Subscribing per signer tied observation to the goroutine that + // published, but a member returns from the submitting call as soon as the + // transaction reaches the provider, so the claim's own event regularly + // arrives after the last publisher — and its subscription — is gone. + // Observation therefore starts before any member can publish and ends only + // once the settlement below has been resolved or given up on. + publishCtx, cancelPublishCtx := context.WithCancel(ctx) + defer cancelPublishCtx() + + subscription := ice.chain.OnInactivityClaimed( + func(event *InactivityClaimedEvent) { + // The subscription is not filtered on chain, so an unrelated + // wallet's claim arrives here too. Attributing it to this claim + // would both abandon publishing for no reason and put another + // wallet's settlement on this permit's terminal record. + if !settlement.observe(event) { + return + } + + execLogger.Infof( + "Inactivity claim submitted for wallet with ID [0x%x] and "+ + "nonce [%v] by notifier [%v] at block [%v]", + event.WalletID, + event.Nonce, + event.Notifier, + event.BlockNumber, + ) + + // The claim is settled; no member has anything left to publish. + cancelPublishCtx() + }) + defer subscription.Unsubscribe() + + submission := &inactivityClaimSubmissionAttempt{} + wg := sync.WaitGroup{} wg.Add(len(ice.signers)) @@ -129,29 +243,10 @@ func (ice *inactivityClaimExecutor) claimInactivity( signer.signingGroupMemberIndex, ) - signerCtx, cancelSignerCtx := context.WithCancel(ctx) - defer cancelSignerCtx() - - subscription := ice.chain.OnInactivityClaimed( - func(event *InactivityClaimedEvent) { - defer cancelSignerCtx() - - execLogger.Infof( - "[member:%v] Inactivity claim submitted for wallet "+ - "with ID [0x%x] and nonce [%v] by notifier [%v] "+ - "at block [%v]", - signer.signingGroupMemberIndex, - event.WalletID, - event.Nonce, - event.Notifier, - event.BlockNumber, - ) - }) - defer subscription.Unsubscribe() - err := ice.publishInactivityClaim( - signerCtx, + publishCtx, execLogger, + commitGuard, sessionID, signer.signingGroupMemberIndex, wallet.groupSize(), @@ -161,8 +256,22 @@ func (ice *inactivityClaimExecutor) claimInactivity( groupMembers, ice.membershipValidator, claim, + submission, ) if err != nil { + // A refused penalty fence or a gate-canceled permit is a + // deliberate release-gate suppression, not an ordinary + // publishing failure. + if participation.IsGateRefusal(err) || + participation.IsGateRefusal(context.Cause(publishCtx)) { + execLogger.Warnf( + "[member:%v] inactivity claim suppressed by the "+ + "release gate: [%v]", + signer.signingGroupMemberIndex, + err, + ) + return + } if errors.Is(err, context.Canceled) { execLogger.Infof( "[member:%v] inactivity claim is no longer awaiting "+ @@ -185,7 +294,321 @@ func (ice *inactivityClaimExecutor) claimInactivity( // Wait until all controlled signers complete their routine. wg.Wait() - return nil + return ice.resolveInactivityClaim( + ctx, + execLogger, + settlement, + walletRegistryData.EcdsaWalletID, + nonce, + submission.recorded(), + ), nil +} + +// resolveInactivityClaim decides what this call can honestly say about the +// chain once every controlled member has stopped publishing. +// +// An observed event answers directly. Failing that, the nonce answers +// indirectly: the registry accepts a claim only at the wallet's current nonce +// and increments it in the same call, so a nonce past the one this claim was +// built for means the registry emitted the settlement of exactly that claim +// slot, whether or not this node's subscription happened to see it. When +// neither answers and a member did hand a transaction to the chain, the claim +// is still plausibly in flight and is worth a bounded wait; when no member ever +// reached the submitting call there is nothing in flight to wait for and the +// wait would only delay the heartbeat. +func (ice *inactivityClaimExecutor) resolveInactivityClaim( + ctx context.Context, + execLogger log.StandardLogger, + settlement *inactivityClaimSettlementObserver, + walletID [32]byte, + nonce *big.Int, + submissionAttempted bool, +) inactivityClaimDisposition { + disposition := inactivityClaimDisposition{ + submissionAttempted: submissionAttempted, + } + + resolved := func() *inactivityClaimSettlement { + if observed := settlement.settled(); observed != nil { + return &inactivityClaimSettlement{ + walletID: observed.WalletID, + nonce: observed.Nonce, + } + } + if ice.inactivityClaimNonceConsumed(execLogger, walletID, nonce) { + return &inactivityClaimSettlement{ + walletID: walletID, + nonce: nonce, + } + } + return nil + } + + if disposition.settlement = resolved(); disposition.settlement != nil { + return disposition + } + + if !submissionAttempted { + return disposition + } + + ice.waitForInactivityClaimSettlement(ctx, execLogger, settlement) + + if disposition.settlement = resolved(); disposition.settlement != nil { + return disposition + } + + execLogger.Warnf( + "inactivity claim for wallet with ID [0x%x] was submitted but not "+ + "observed to settle at nonce [%v]; the penalty is reported "+ + "unresolved", + walletID, + nonce, + ) + + return disposition +} + +// inactivityClaimNonceConsumed reports whether the chain has moved past the +// nonce this claim was built for, which is the chain's own confirmation that +// the claim slot was settled. An unreadable nonce resolves nothing and leaves +// the claim unresolved, which is the conservative of the two readings. +func (ice *inactivityClaimExecutor) inactivityClaimNonceConsumed( + execLogger log.StandardLogger, + walletID [32]byte, + nonce *big.Int, +) bool { + currentNonce, err := ice.chain.GetInactivityClaimNonce(walletID) + if err != nil { + execLogger.Warnf( + "cannot read the inactivity claim nonce of wallet with ID "+ + "[0x%x] to resolve the claim settlement: [%v]", + walletID, + err, + ) + return false + } + + return currentNonce != nil && nonce != nil && currentNonce.Cmp(nonce) > 0 +} + +// waitForInactivityClaimSettlement gives a submitted claim a bounded number of +// blocks to be mined. The call-wide subscription is still live, so an arriving +// settlement ends the wait immediately instead of letting it run to the +// deadline, and the caller's context — the heartbeat window that owns the claim +// — cuts it short whenever that window closes first. +func (ice *inactivityClaimExecutor) waitForInactivityClaimSettlement( + ctx context.Context, + execLogger log.StandardLogger, + settlement *inactivityClaimSettlementObserver, +) { + blockCounter, err := ice.chain.BlockCounter() + if err != nil { + execLogger.Warnf( + "cannot get the block counter to wait for the inactivity claim "+ + "settlement: [%v]", + err, + ) + return + } + + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + execLogger.Warnf( + "cannot get the current block to wait for the inactivity claim "+ + "settlement: [%v]", + err, + ) + return + } + + waitCtx, cancelWaitCtx := context.WithCancel(ctx) + defer cancelWaitCtx() + + go func() { + select { + case <-settlement.observed(): + cancelWaitCtx() + case <-waitCtx.Done(): + } + }() + + deadline, err := inactivityClaimSettlementDeadline(currentBlock) + if err != nil { + execLogger.Warnf( + "cannot bound the wait for the inactivity claim settlement; the "+ + "claim is left to the resolution that follows: [%v]", + err, + ) + return + } + + execLogger.Infof( + "waiting until block [%v] for the submitted inactivity claim to settle", + deadline, + ) + + if err := ice.waitForBlockFn(waitCtx, deadline); err != nil && + !errors.Is(err, context.Canceled) { + execLogger.Warnf( + "error while waiting for the inactivity claim settlement: [%v]", + err, + ) + } +} + +// errInactivityDeadlineOverflow reports a block deadline the block range cannot +// represent. Both deadlines this file derives are rejected on overflow rather +// than clamped: a wrapped deadline names a block already in the past, so every +// wait keyed to one returns at once and the fence it was standing in for is +// gone without anything saying so. +var errInactivityDeadlineOverflow = errors.New( + "the block deadline is not representable", +) + +// inactivityClaimSubmissionBlock returns the block at which the given member +// may hand its inactivity claim to the chain, or rejects a sum the block range +// cannot represent. +// +// Members stagger their submissions by member index so they do not all pay for +// the same claim. A wrapped submission block names a block already passed, +// which collapses the whole stagger to "now" — but the reason it is rejected +// rather than clamped is the one that matters: this is the last derived +// quantity standing between the caller and an irreversible on-chain +// submission, and arithmetic that overflowed cannot be allowed to authorize it. +func inactivityClaimSubmissionBlock( + currentBlock uint64, + memberIndex group.MemberIndex, +) (uint64, error) { + delayBlocks := uint64(memberIndex-1) * inactivityClaimSubmissionDelayStepBlocks + + if currentBlock > math.MaxUint64-delayBlocks { + return 0, fmt.Errorf( + "%w: current block [%v] plus the [%v] block submission delay of "+ + "member [%v]", + errInactivityDeadlineOverflow, + currentBlock, + delayBlocks, + memberIndex, + ) + } + return currentBlock + delayBlocks, nil +} + +// inactivityClaimSettlementDeadline returns the block the settlement wait runs +// until, or rejects a sum the block range cannot represent. +// +// A wrapped deadline names a block already in the past, so the wait would +// return at once and a claim that settles a block later would be journaled as +// unresolved — a penalty that is on chain recorded as one that is not. Clamping +// to the top of the range instead would leave a wait nothing bounds but the +// heartbeat window. Rejecting reaches the honest disposition, unresolved, and +// says why: the offline barrier holds on an unreconciled claim, which is the +// conservative direction for a penalty nobody can account for. +func inactivityClaimSettlementDeadline(currentBlock uint64) (uint64, error) { + if currentBlock > math.MaxUint64-inactivityClaimSettlementResolutionBlocks { + return 0, fmt.Errorf( + "%w: current block [%v] plus the [%v] block settlement resolution "+ + "bound", + errInactivityDeadlineOverflow, + currentBlock, + inactivityClaimSettlementResolutionBlocks, + ) + } + return currentBlock + inactivityClaimSettlementResolutionBlocks, nil +} + +// inactivityClaimSubmissionAttempt records that a controlled member handed an +// inactivity claim transaction to the chain. Every controlled signer publishes +// on its own goroutine and any of them can be the one that submits, so the +// record is written from arbitrary goroutines and read once publishing ends. +type inactivityClaimSubmissionAttempt struct { + attempted atomic.Bool +} + +// record marks that a claim transaction was handed to the chain. It is called +// immediately before the submitting call rather than after it: a call that +// returns an error may still have broadcast the transaction, and a penalty that +// might be on chain has to reach the terminal record as one. +func (a *inactivityClaimSubmissionAttempt) record() { + a.attempted.Store(true) +} + +// recorded reports whether any controlled member reached the submitting call. +func (a *inactivityClaimSubmissionAttempt) recorded() bool { + return a.attempted.Load() +} + +// inactivityClaimSettlementObserver collects the on-chain settlement of one +// inactivity claim. The chain delivers events off the publishing goroutines and +// keeps delivering them after publishing has ended, so the observation is +// shared state written from arbitrary goroutines and read by the resolution +// that outlives them. +type inactivityClaimSettlementObserver struct { + walletID [32]byte + nonce *big.Int + + mutex sync.Mutex + event *InactivityClaimedEvent + // settlement is closed on the first matching observation so a bounded wait + // can end the moment the claim is seen instead of always running to its + // deadline. + settlement chan struct{} +} + +func newInactivityClaimSettlementObserver( + walletID [32]byte, + nonce *big.Int, +) *inactivityClaimSettlementObserver { + return &inactivityClaimSettlementObserver{ + walletID: walletID, + nonce: nonce, + settlement: make(chan struct{}), + } +} + +// observe records event as this claim's settlement and reports whether it +// belongs to the claim at all. Only the wallet and nonce the claim was built +// for match: the nonce is consumed by the submission that emits it, so a +// matching event is the settlement of this exact claim and not of a later one +// against the same wallet. +func (o *inactivityClaimSettlementObserver) observe( + event *InactivityClaimedEvent, +) bool { + if event == nil || + event.WalletID != o.walletID || + event.Nonce == nil || + o.nonce == nil || + event.Nonce.Cmp(o.nonce) != 0 { + return false + } + + o.mutex.Lock() + defer o.mutex.Unlock() + + // Each signer observes the same settlement; the first observation is the + // one recorded so the reported block cannot drift between members. + if o.event == nil { + o.event = event + close(o.settlement) + } + + return true +} + +// settled returns the observed settlement, or nil when the claim was never +// seen to reach the chain. +func (o *inactivityClaimSettlementObserver) settled() *InactivityClaimedEvent { + o.mutex.Lock() + defer o.mutex.Unlock() + + return o.event +} + +// observed returns a channel closed once this claim's settlement is seen. It +// lets a bounded wait react to the settlement rather than poll for it. +func (o *inactivityClaimSettlementObserver) observed() <-chan struct{} { + return o.settlement } func (ice *inactivityClaimExecutor) getWalletOperatorsIDs() ([]uint32, error) { @@ -217,6 +640,7 @@ func (ice *inactivityClaimExecutor) getWalletOperatorsIDs() ([]uint32, error) { func (ice *inactivityClaimExecutor) publishInactivityClaim( ctx context.Context, inactivityLogger log.StandardLogger, + commitGuard participation.CommitGuard, sessionID *big.Int, memberIndex group.MemberIndex, groupSize int, @@ -224,6 +648,7 @@ func (ice *inactivityClaimExecutor) publishInactivityClaim( groupMembers []uint32, membershipValidator *group.MembershipValidator, inactivityClaim *inactivity.ClaimPreimage, + submission *inactivityClaimSubmissionAttempt, ) error { return inactivity.PublishClaim( ctx, @@ -241,6 +666,8 @@ func (ice *inactivityClaimExecutor) publishInactivityClaim( ice.groupParameters, groupMembers, ice.waitForBlockFn, + commitGuard, + submission, ), inactivityClaim, ) @@ -322,6 +749,16 @@ type inactivityClaimSubmitter struct { groupMembers []uint32 waitForBlockFn waitForBlockFn + + // commitGuard fences the terminal on-chain submission with a penalty + // commit check: a refusal is a release-gate decision, not an ordinary + // submission failure. + commitGuard participation.CommitGuard + + // submission is the executor-wide record of whether any controlled member + // reached the submitting call. Every exit above that call is an exit the + // caller may report as leaving no chain state behind. + submission *inactivityClaimSubmissionAttempt } func newInactivityClaimSubmitter( @@ -330,6 +767,8 @@ func newInactivityClaimSubmitter( groupParameters *GroupParameters, groupMembers []uint32, waitForBlockFn waitForBlockFn, + commitGuard participation.CommitGuard, + submission *inactivityClaimSubmissionAttempt, ) *inactivityClaimSubmitter { return &inactivityClaimSubmitter{ inactivityLogger: inactivityLogger, @@ -337,6 +776,8 @@ func newInactivityClaimSubmitter( groupParameters: groupParameters, groupMembers: groupMembers, waitForBlockFn: waitForBlockFn, + commitGuard: commitGuard, + submission: submission, } } @@ -407,8 +848,19 @@ func (ics *inactivityClaimSubmitter) SubmitClaim( if err != nil { return fmt.Errorf("cannot get current block: [%v]", err) } - delayBlocks := uint64(memberIndex-1) * inactivityClaimSubmissionDelayStepBlocks - submissionBlock := currentBlock + delayBlocks + // The wait below is the last thing between this member and an irreversible + // submission, so the block it waits for is derived before the claim can be + // filed and a derivation that overflowed refuses the claim outright. + submissionBlock, err := inactivityClaimSubmissionBlock( + currentBlock, + memberIndex, + ) + if err != nil { + return fmt.Errorf( + "cannot derive the inactivity claim submission block: [%w]", + err, + ) + } ics.inactivityLogger.Infof( "[member:%v] waiting for block [%v] to submit inactivity claim", @@ -459,6 +911,21 @@ func (ics *inactivityClaimSubmitter) SubmitClaim( len(signatures), ) + // The last-moment penalty fence immediately before the irreversible + // on-chain submission. + if err := ics.commitGuard.CheckCommit( + "tbtc_inactivity_claim_submission", + participation.PenaltyCommit, + ); err != nil { + return err + } + + // Past this point the transaction may reach the provider and the chain, so + // the attempt is recorded before the call and never after it: a submitting + // call that returns an error may still have broadcast, and the executor + // must go on to resolve a penalty that might exist. + ics.submission.record() + err = ics.chain.SubmitInactivityClaim( chainClaim, inactivityNonce, diff --git a/pkg/tbtc/inactivity_test.go b/pkg/tbtc/inactivity_test.go index ce8762a455..31592e2d05 100644 --- a/pkg/tbtc/inactivity_test.go +++ b/pkg/tbtc/inactivity_test.go @@ -2,9 +2,12 @@ package tbtc import ( "context" + "errors" "fmt" + "math" "math/big" "reflect" + "sync" "testing" "time" @@ -20,6 +23,7 @@ import ( "github.com/keep-network/keep-core/pkg/operator" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/protocol/inactivity" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -37,8 +41,9 @@ func TestInactivityClaimExecutor_ClaimInactivity(t *testing.T) { message := big.NewInt(100) inactiveMembersIndexes := []group.MemberIndex{1, 4} - err = executor.claimInactivity( + disposition, err := executor.claimInactivity( ctx, + newTestPermit(participation.TBTCInactivityClaim), inactiveMembersIndexes, true, message, @@ -61,6 +66,588 @@ func TestInactivityClaimExecutor_ClaimInactivity(t *testing.T) { expectedNonceDiff, nonceDiff, ) + + // A member reached the submitting call, so the disposition must say a + // transaction exists rather than leave the caller to infer it. + if !disposition.submissionAttempted { + t.Error("expected the executor to report an attempted submission") + } + + // The claim reached the chain, so the executor must report the settlement + // its caller records; the reported identity has to be the wallet and the + // nonce the claim consumed, not the wallet's current one. + if disposition.settlement == nil { + t.Fatal("expected the submitted inactivity claim to be reported settled") + } + if disposition.settlement.walletID != walletEcdsaID { + t.Errorf( + "unexpected settled wallet\nexpected: [0x%x]\nactual: [0x%x]", + walletEcdsaID, + disposition.settlement.walletID, + ) + } + testutils.AssertBigIntsEqual( + t, + "settled inactivity claim nonce", + initialNonce, + disposition.settlement.nonce, + ) +} + +// TestInactivityClaimExecutor_ClaimInactivity_UnrelatedWalletSettlement checks +// that a claim settled against a different wallet is not reported as this +// claim's settlement. The chain subscription is unfiltered, so the executor is +// the only thing standing between an unrelated penalty and a terminal record +// that names it. +func TestInactivityClaimExecutor_ClaimInactivity_UnrelatedWalletSettlement( + t *testing.T, +) { + walletID := [32]byte{0x01} + nonce := big.NewInt(7) + + observer := newInactivityClaimSettlementObserver(walletID, nonce) + + for _, test := range []struct { + name string + event *InactivityClaimedEvent + }{ + { + name: "no event at all", + event: nil, + }, + { + name: "another wallet at the same nonce", + event: &InactivityClaimedEvent{ + WalletID: [32]byte{0x02}, + Nonce: big.NewInt(7), + }, + }, + { + name: "the same wallet at a later nonce", + event: &InactivityClaimedEvent{ + WalletID: walletID, + Nonce: big.NewInt(8), + }, + }, + { + name: "the same wallet with no nonce", + event: &InactivityClaimedEvent{ + WalletID: walletID, + Nonce: nil, + }, + }, + } { + t.Run(test.name, func(t *testing.T) { + if observer.observe(test.event) { + t.Error("expected the event not to be attributed to the claim") + } + if observer.settled() != nil { + t.Error("expected no settlement to be recorded") + } + }) + } + + matching := &InactivityClaimedEvent{ + WalletID: walletID, + Nonce: big.NewInt(7), + BlockNumber: 1000, + } + if !observer.observe(matching) { + t.Fatal("expected the matching event to be attributed to the claim") + } + + // Every controlled signer observes the same settlement; the record must + // stay on the first observation so members cannot report different blocks + // for one claim. + if !observer.observe(&InactivityClaimedEvent{ + WalletID: walletID, + Nonce: big.NewInt(7), + BlockNumber: 2000, + }) { + t.Fatal("expected a repeated matching event to still match the claim") + } + if settled := observer.settled(); settled != matching { + t.Errorf( + "expected the first observation to be retained, got [%v]", + settled, + ) + } +} + +// TestInactivityClaimExecutor_ResolveInactivityClaim walks every way one claim +// can end once publishing has stopped. +// +// The lifecycle these cases pin is the one a real provider imposes: the +// submitting call returns when the transaction is accepted, and the claim is +// mined — and announced — afterwards. Observation therefore has to outlive +// publishing, and the disposition has to keep "a transaction may exist" apart +// from "the penalty landed", because a rollback reads those differently. +func TestInactivityClaimExecutor_ResolveInactivityClaim(t *testing.T) { + walletID := [32]byte{0x07} + nonce := big.NewInt(0) + + settlementEvent := &InactivityClaimedEvent{ + WalletID: walletID, + Nonce: big.NewInt(0), + BlockNumber: 4242, + } + + // mineCompetingClaim consumes the claim slot on chain without ever + // reaching this node's subscription, which is what a settlement observed + // by nobody here looks like. + mineCompetingClaim := func(t *testing.T, localChain *localChain) { + if err := localChain.SubmitInactivityClaim( + &InactivityClaim{WalletID: walletID}, + new(big.Int).Set(nonce), + nil, + ); err != nil { + t.Fatalf("cannot settle the competing claim: [%v]", err) + } + } + + tests := map[string]struct { + submissionAttempted bool + // before runs against the chain before the resolution starts, standing + // in for whatever already happened while publishing was running. + before func(t *testing.T, localChain *localChain) + // wait stands in for whatever the chain does while the bounded + // settlement wait is in progress. It is nil for the cases that must + // take no wait at all. + wait func( + t *testing.T, + localChain *localChain, + settlement *inactivityClaimSettlementObserver, + ctx context.Context, + ) error + expectedSettlement bool + expectedWait bool + }{ + // The regression this whole lifecycle exists for: the claim's own + // event arrives after every publishing goroutine has returned. The + // call-wide subscription is still listening, so the penalty is + // resolved instead of being reported as an unknown. + "a settlement announced after publishing ended is resolved": { + submissionAttempted: true, + wait: func( + _ *testing.T, + _ *localChain, + settlement *inactivityClaimSettlementObserver, + ctx context.Context, + ) error { + go settlement.observe(settlementEvent) + // Blocking until cancellation makes the wait prove it ends on + // the settlement rather than on its own deadline. + <-ctx.Done() + return ctx.Err() + }, + expectedSettlement: true, + expectedWait: true, + }, + // The subscription can miss the event entirely — a dropped + // notification, a reorganized filter. The consumed nonce is the + // chain's own receipt for the claim slot and resolves it anyway. + "a settlement seen only as a consumed nonce is resolved": { + submissionAttempted: true, + wait: func( + t *testing.T, + localChain *localChain, + _ *inactivityClaimSettlementObserver, + _ context.Context, + ) error { + mineCompetingClaim(t, localChain) + return nil + }, + expectedSettlement: true, + expectedWait: true, + }, + // The one genuinely ambiguous case: a transaction was handed to the + // chain and nothing came back. It must stay unresolved so the offline + // barrier blocks on it. + "a submission that never settles stays unresolved": { + submissionAttempted: true, + wait: func( + _ *testing.T, + _ *localChain, + _ *inactivityClaimSettlementObserver, + _ context.Context, + ) error { + return nil + }, + expectedWait: true, + }, + // No member reached the submitting call, so nothing is in flight. + // Waiting for it would only delay the heartbeat that owns the claim. + "a claim that never reached the chain takes no wait": {}, + // Another node's submission can settle this claim slot while no + // controlled member ever submits; the penalty is still on chain and + // still this permit's. + "a foreign settlement is resolved without any wait": { + before: mineCompetingClaim, + expectedSettlement: true, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + localChain := Connect() + settlement := newInactivityClaimSettlementObserver( + walletID, + new(big.Int).Set(nonce), + ) + + if test.before != nil { + test.before(t, localChain) + } + + var waited bool + executor := &inactivityClaimExecutor{ + chain: localChain, + waitForBlockFn: func(ctx context.Context, _ uint64) error { + waited = true + if test.wait == nil { + return nil + } + return test.wait(t, localChain, settlement, ctx) + }, + } + + disposition := executor.resolveInactivityClaim( + context.Background(), + &testutils.MockLogger{}, + settlement, + walletID, + new(big.Int).Set(nonce), + test.submissionAttempted, + ) + + if disposition.submissionAttempted != test.submissionAttempted { + t.Errorf( + "unexpected submission attempt\nexpected: [%v]\nactual: [%v]", + test.submissionAttempted, + disposition.submissionAttempted, + ) + } + if waited != test.expectedWait { + t.Errorf( + "unexpected settlement wait\nexpected: [%v]\nactual: [%v]", + test.expectedWait, + waited, + ) + } + + if !test.expectedSettlement { + if disposition.settlement != nil { + t.Fatalf( + "expected no resolved settlement, got [%+v]", + disposition.settlement, + ) + } + return + } + + if disposition.settlement == nil { + t.Fatal("expected the claim settlement to be resolved") + } + if disposition.settlement.walletID != walletID { + t.Errorf( + "unexpected settled wallet\nexpected: [0x%x]\nactual: [0x%x]", + walletID, + disposition.settlement.walletID, + ) + } + testutils.AssertBigIntsEqual( + t, + "settled inactivity claim nonce", + nonce, + disposition.settlement.nonce, + ) + }) + } +} + +// TestInactivityClaimExecutor_ResolveInactivityClaim_ContextCancelled asserts a +// claim whose owning window closes mid-wait is reported unresolved rather than +// waited on past the deadline the heartbeat set for it. +func TestInactivityClaimExecutor_ResolveInactivityClaim_ContextCancelled( + t *testing.T, +) { + localChain := Connect() + + walletID := [32]byte{0x09} + nonce := big.NewInt(0) + + settlement := newInactivityClaimSettlementObserver(walletID, nonce) + + ctx, cancelCtx := context.WithCancel(context.Background()) + cancelCtx() + + executor := &inactivityClaimExecutor{ + chain: localChain, + waitForBlockFn: func(ctx context.Context, _ uint64) error { + <-ctx.Done() + return ctx.Err() + }, + } + + disposition := executor.resolveInactivityClaim( + ctx, + &testutils.MockLogger{}, + settlement, + walletID, + nonce, + true, + ) + + if !disposition.submissionAttempted { + t.Error("expected the submission to stay on the record") + } + if disposition.settlement != nil { + t.Errorf( + "expected a canceled resolution to settle nothing, got [%+v]", + disposition.settlement, + ) + } +} + +// TestInactivityClaimExecutor_ClaimInactivity_LateSettlement drives the whole +// executor against a chain that behaves like a real provider: submissions are +// accepted and mined only later. The claim's event therefore arrives after +// every publishing goroutine has returned, which is precisely when a +// per-publisher subscription would already be gone. +func TestInactivityClaimExecutor_ClaimInactivity_LateSettlement(t *testing.T) { + executor, walletEcdsaID, localChain := setupInactivityClaimExecutorScenario(t) + + initialNonce, err := localChain.GetInactivityClaimNonce(walletEcdsaID) + if err != nil { + t.Fatal(err) + } + + var mutex sync.Mutex + var accepted []func() + + localChain.setInactivityClaimMiner(func(mine func()) { + mutex.Lock() + defer mutex.Unlock() + + accepted = append(accepted, mine) + }) + + // Every controlled member waits once for its submission delay and submits + // immediately afterwards, so a wait requested once all of them have been + // accepted can only be the executor's settlement resolution. Mining there + // puts the claim's event strictly after the end of publishing. + signerCount := len(executor.signers) + waitForBlock := executor.waitForBlockFn + listenersAtSettlement := -1 + executor.waitForBlockFn = func(ctx context.Context, block uint64) error { + mutex.Lock() + var mine func() + if len(accepted) == signerCount { + mine, accepted = accepted[0], nil + } + mutex.Unlock() + + if mine != nil { + listenersAtSettlement = localChain.inactivityClaimedHandlerCount() + mine() + } + + return waitForBlock(ctx, block) + } + + ctx, cancelCtx := context.WithCancel(context.Background()) + defer cancelCtx() + + disposition, err := executor.claimInactivity( + ctx, + newTestPermit(participation.TBTCInactivityClaim), + []group.MemberIndex{1, 4}, + true, + big.NewInt(100), + ) + if err != nil { + t.Fatal(err) + } + + if !disposition.submissionAttempted { + t.Error("expected the executor to report an attempted submission") + } + // The point of the lifecycle: the claim was announced with publishing + // already over, and something was still listening for it. + if listenersAtSettlement < 1 { + t.Errorf( + "the claim settled with [%d] subscriptions left listening", + listenersAtSettlement, + ) + } + if disposition.settlement == nil { + t.Fatal( + "a claim mined after publishing ended was not reported settled", + ) + } + if disposition.settlement.walletID != walletEcdsaID { + t.Errorf( + "unexpected settled wallet\nexpected: [0x%x]\nactual: [0x%x]", + walletEcdsaID, + disposition.settlement.walletID, + ) + } + testutils.AssertBigIntsEqual( + t, + "settled inactivity claim nonce", + initialNonce, + disposition.settlement.nonce, + ) +} + +// TestInactivityClaimSettlementDeadline asserts the settlement wait's bound is +// rejected when it overflows rather than clamped to the top of the block range. +// A clamped bound leaves a wait nothing but the heartbeat window ends; the +// rejection reaches the same disposition — the claim stays unresolved and the +// offline barrier holds on it — and says why. +func TestInactivityClaimSettlementDeadline(t *testing.T) { + const resolution = uint64(inactivityClaimSettlementResolutionBlocks) + + tests := map[string]struct { + currentBlock uint64 + expectedDeadline uint64 + expectedError bool + }{ + "an ordinary height": { + currentBlock: 1_000_000, + expectedDeadline: 1_000_000 + resolution, + }, + "the highest height whose deadline is representable": { + currentBlock: math.MaxUint64 - resolution, + expectedDeadline: math.MaxUint64, + }, + "the lowest height whose deadline is not": { + currentBlock: math.MaxUint64 - resolution + 1, + expectedError: true, + }, + "the highest representable height": { + currentBlock: math.MaxUint64, + expectedError: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + deadline, err := inactivityClaimSettlementDeadline( + test.currentBlock, + ) + + if test.expectedError { + if !errors.Is(err, errInactivityDeadlineOverflow) { + t.Errorf( + "expected an overflow rejection\nactual: [%v]", + err, + ) + } + return + } + + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if deadline != test.expectedDeadline { + t.Errorf( + "unexpected settlement deadline\n"+ + "expected: [%d]\nactual: [%d]", + test.expectedDeadline, + deadline, + ) + } + // The property the bound exists for: a deadline the executor waits + // on must never name a block the chain has already passed. + if deadline < test.currentBlock { + t.Errorf( + "settlement deadline [%d] is below the current block [%d]", + deadline, + test.currentBlock, + ) + } + }) + } +} + +// TestInactivityClaimSubmissionBlock asserts the staggered submission block is +// rejected on overflow. It is the last quantity derived before an irreversible +// on-chain claim, so a wrapped value would authorize the submission from +// arithmetic that had already lost its meaning. +func TestInactivityClaimSubmissionBlock(t *testing.T) { + const step = uint64(inactivityClaimSubmissionDelayStepBlocks) + + tests := map[string]struct { + currentBlock uint64 + memberIndex group.MemberIndex + expectedBlock uint64 + expectedError bool + }{ + "the first member never waits": { + currentBlock: 1_000_000, + memberIndex: 1, + expectedBlock: 1_000_000, + }, + "a later member waits its index out": { + currentBlock: 1_000_000, + memberIndex: 5, + expectedBlock: 1_000_000 + 4*step, + }, + "the first member at the top of the block range": { + currentBlock: math.MaxUint64, + memberIndex: 1, + expectedBlock: math.MaxUint64, + }, + "the highest height that still admits the delay": { + currentBlock: math.MaxUint64 - 4*step, + memberIndex: 5, + expectedBlock: math.MaxUint64, + }, + "one height past what the delay admits": { + currentBlock: math.MaxUint64 - 4*step + 1, + memberIndex: 5, + expectedError: true, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + submissionBlock, err := inactivityClaimSubmissionBlock( + test.currentBlock, + test.memberIndex, + ) + + if test.expectedError { + if !errors.Is(err, errInactivityDeadlineOverflow) { + t.Errorf( + "expected an overflow rejection\nactual: [%v]", + err, + ) + } + return + } + + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if submissionBlock != test.expectedBlock { + t.Errorf( + "unexpected submission block\n"+ + "expected: [%d]\nactual: [%d]", + test.expectedBlock, + submissionBlock, + ) + } + if submissionBlock < test.currentBlock { + t.Errorf( + "submission block [%d] is below the current block [%d]", + submissionBlock, + test.currentBlock, + ) + } + }) + } } func TestInactivityClaimExecutor_ClaimInactivity_Busy(t *testing.T) { @@ -74,8 +661,9 @@ func TestInactivityClaimExecutor_ClaimInactivity_Busy(t *testing.T) { errChan := make(chan error, 1) go func() { - err := executor.claimInactivity( + _, err := executor.claimInactivity( ctx, + newTestPermit(participation.TBTCInactivityClaim), inactiveMembersIndexes, true, message, @@ -85,14 +673,28 @@ func TestInactivityClaimExecutor_ClaimInactivity_Busy(t *testing.T) { time.Sleep(100 * time.Millisecond) - err := executor.claimInactivity( + disposition, err := executor.claimInactivity( ctx, + newTestPermit(participation.TBTCInactivityClaim), inactiveMembersIndexes, true, message, ) testutils.AssertErrorsSame(t, errInactivityClaimExecutorBusy, err) + // A refused call never subscribed to anything and never reached a + // submitting call, so it can neither report a settlement observed by the + // call that holds the executor nor claim a transaction of its own. + if disposition.submissionAttempted { + t.Error("expected a refused claim to report no attempted submission") + } + if disposition.settlement != nil { + t.Errorf( + "expected no settlement from a refused claim, got [%+v]", + disposition.settlement, + ) + } + err = <-errChan if err != nil { t.Errorf("unexpected error: [%v]", err) @@ -423,6 +1025,147 @@ func TestVerifySignature_VerifyError(t *testing.T) { } } +// TestSubmitClaim_SubmissionAttemptAccounting pins which exits of the +// submitter count as handing a transaction to the chain. +// +// The distinction decides what a rollback may do. An exit above the submitting +// call — too few signatures, a claim another member already settled, a refused +// penalty fence, a closed window — provably left no transaction anywhere, and +// recording it as a possible penalty would block a homogeneous rollback over +// state that cannot exist. From the submitting call on the opposite holds: a +// call that returns an error may still have broadcast, so the attempt has to +// survive the error. +func TestSubmitClaim_SubmissionAttemptAccounting(t *testing.T) { + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + publicKey := tecdsa.NewPrivateKeyShare(testData[0]).PublicKey() + + ecdsaWalletID := [32]byte{1, 2, 3} + groupMembers := []uint32{1, 2, 2, 3, 5} + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + } + + thresholdSignatures := map[group.MemberIndex][]byte{ + 1: []byte("signature 1"), + 2: []byte("signature 2"), + 3: []byte("signature 3"), + 4: []byte("signature 4"), + } + + tests := map[string]struct { + signatures map[group.MemberIndex][]byte + // claimNonce is the nonce the claim is built for; the wallet starts at + // nonce zero. + claimNonce int64 + // settledFirst mimics another member consuming the claim slot before + // this member wakes from its submission delay. + settledFirst bool + // commitErr mimics the penalty fence refusing the submission. + commitErr error + // cancelled mimics the owning window closing before submission. + cancelled bool + expectedAttempt bool + }{ + "too few signatures to submit anything": { + signatures: map[group.MemberIndex][]byte{ + 1: []byte("signature 1"), + 2: []byte("signature 2"), + }, + }, + "a claim another member already settled": { + signatures: thresholdSignatures, + settledFirst: true, + }, + "a refused penalty fence": { + signatures: thresholdSignatures, + commitErr: participation.ErrPenaltySuppressed, + }, + "a window closed before submission": { + signatures: thresholdSignatures, + cancelled: true, + }, + "a transaction the chain rejected": { + signatures: thresholdSignatures, + // A nonce the registry refuses still reaches it as a transaction. + claimNonce: 12345, + expectedAttempt: true, + }, + "a transaction the chain accepted": { + signatures: thresholdSignatures, + expectedAttempt: true, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + localChain := Connect() + localChain.setWallet( + bitcoin.PublicKeyHash(publicKey), + &WalletChainData{EcdsaWalletID: ecdsaWalletID}, + ) + + if test.settledFirst { + if err := localChain.SubmitInactivityClaim( + &InactivityClaim{WalletID: ecdsaWalletID}, + big.NewInt(0), + groupMembers, + ); err != nil { + t.Fatal(err) + } + } + + permit := newTestPermit(participation.TBTCInactivityClaim) + permit.commitErr = test.commitErr + + submission := &inactivityClaimSubmissionAttempt{} + + submitter := newInactivityClaimSubmitter( + &testutils.MockLogger{}, + localChain, + groupParameters, + groupMembers, + testWaitForBlockFn(localChain), + permit, + submission, + ) + + ctx, cancelCtx := context.WithCancel(context.Background()) + defer cancelCtx() + if test.cancelled { + cancelCtx() + } + + // SubmitClaim reports refusals and rejections through its error, + // which the executor classifies separately; what is under test + // here is only whether a transaction was handed to the chain. + _ = submitter.SubmitClaim( + ctx, + group.MemberIndex(1), + inactivity.NewClaimPreimage( + big.NewInt(test.claimNonce), + publicKey, + []group.MemberIndex{11, 22, 33}, + true, + ), + test.signatures, + ) + + if submission.recorded() != test.expectedAttempt { + t.Errorf( + "unexpected submission attempt\nexpected: [%v]\nactual: [%v]", + test.expectedAttempt, + submission.recorded(), + ) + } + }) + } +} + func TestSubmitClaim_MemberSubmitsClaim(t *testing.T) { testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) if err != nil { @@ -457,6 +1200,8 @@ func TestSubmitClaim_MemberSubmitsClaim(t *testing.T) { groupParameters, groupMembers, testWaitForBlockFn(chain), + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) ctx, cancelCtx := context.WithCancel(context.Background()) @@ -537,6 +1282,8 @@ func TestSubmitClaim_AnotherMemberSubmitsClaim(t *testing.T) { groupParameters, groupMembers, testWaitForBlockFn(chain), + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) ctx, cancelCtx := context.WithCancel(context.Background()) @@ -564,6 +1311,39 @@ func TestSubmitClaim_AnotherMemberSubmitsClaim(t *testing.T) { }, ) + // The second member has to be parked in its submission delay when the + // first member submits, otherwise both submit and the wallet's nonce + // advances twice. Holding its wait open until the first member is done is + // what makes that ordering the test's own rather than the block counter's: + // the delay is two blocks wide, so any wall-clock pause races it. + secondMemberWaiting := make(chan struct{}) + releaseSecondMember := make(chan struct{}) + var waitMutex sync.Mutex + waits := 0 + + waitForBlock := inactivityClaimSubmitter.waitForBlockFn + inactivityClaimSubmitter.waitForBlockFn = func( + ctx context.Context, + block uint64, + ) error { + waitMutex.Lock() + waits++ + held := waits == 1 + waitMutex.Unlock() + + // The second member's goroutine is the only one running until its + // wait is observed, so the first wait to arrive is always its own. + if held { + close(secondMemberWaiting) + select { + case <-releaseSecondMember: + case <-ctx.Done(): + } + } + + return waitForBlock(ctx, block) + } + secondMemberSubmissionChannel := make(chan error) // Attempt to submit claim for the second member on a separate goroutine. go func() { @@ -577,10 +1357,7 @@ func TestSubmitClaim_AnotherMemberSubmitsClaim(t *testing.T) { secondMemberSubmissionChannel <- secondMemberErr }() - // This sleep is needed to give enough time for the second member to - // register their claim submission event handler and act properly on the - // claim submitted by the first member. - time.Sleep(1 * time.Second) + <-secondMemberWaiting // While the second member is waiting for submission eligibility, submit the // claim with the first member. @@ -591,6 +1368,7 @@ func TestSubmitClaim_AnotherMemberSubmitsClaim(t *testing.T) { claim, signatures, ) + close(releaseSecondMember) if firstMemberErr != nil { t.Fatal(firstMemberErr) } @@ -664,6 +1442,8 @@ func TestSubmitClaim_StaleNonceAfterDelayTreatedAsSubmitted(t *testing.T) { groupParameters, groupMembers, func(context.Context, uint64) error { return nil }, + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) var firstMemberSubmitErr error @@ -682,6 +1462,8 @@ func TestSubmitClaim_StaleNonceAfterDelayTreatedAsSubmitted(t *testing.T) { ) return nil }, + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) err = secondMemberSubmitter.SubmitClaim( @@ -745,6 +1527,8 @@ func TestSubmitClaim_InvalidResult(t *testing.T) { groupParameters, groupMembers, testWaitForBlockFn(chain), + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) ctx, cancelCtx := context.WithCancel(context.Background()) @@ -817,6 +1601,8 @@ func TestSubmitClaim_ContextCancelled(t *testing.T) { groupParameters, groupMembers, testWaitForBlockFn(chain), + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) ctx, cancelCtx := context.WithCancel(context.Background()) @@ -900,6 +1686,8 @@ func TestSubmitClaim_TooFewSignatures(t *testing.T) { groupParameters, groupMembers, testWaitForBlockFn(chain), + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) ctx, cancelCtx := context.WithCancel(context.Background()) @@ -1008,6 +1796,8 @@ func TestSubmitClaim_NonceChangesDuringWait(t *testing.T) { groupParameters, groupMembers, hookedWaitForBlockFn, + newTestPermit(participation.TBTCInactivityClaim), + &inactivityClaimSubmissionAttempt{}, ) ctx, cancelCtx := context.WithCancel(context.Background()) diff --git a/pkg/tbtc/moved_funds_sweep.go b/pkg/tbtc/moved_funds_sweep.go index 2569f4557d..f7f7c50cd5 100644 --- a/pkg/tbtc/moved_funds_sweep.go +++ b/pkg/tbtc/moved_funds_sweep.go @@ -7,6 +7,7 @@ import ( "time" "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/participation" "go.uber.org/zap" "github.com/ipfs/go-log/v2" @@ -95,6 +96,11 @@ type movedFundsSweepAction struct { signingTimeoutSafetyMarginBlocks uint64 broadcastTimeout time.Duration broadcastCheckDelay time.Duration + + // permit is the wallet action's participation permit: it pins the + // protocol mode for every signing of this action and is released when the + // action's execution ends. + permit participation.Permit } func newMovedFundsSweepAction( @@ -107,12 +113,15 @@ func newMovedFundsSweepAction( proposalProcessingStartBlock uint64, proposalExpiryBlock uint64, waitForBlockFn waitForBlockFn, + permit participation.Permit, ) *movedFundsSweepAction { transactionExecutor := newWalletTransactionExecutor( btcChain, movedFundsSweepWallet, signingExecutor, waitForBlockFn, + permit, + "tbtc_moved_funds_sweep_bitcoin_broadcast", ) return &movedFundsSweepAction{ @@ -127,10 +136,18 @@ func newMovedFundsSweepAction( signingTimeoutSafetyMarginBlocks: movedFundsSweepSigningTimeoutSafetyMarginBlocks, broadcastTimeout: movedFundsSweepBroadcastTimeout, broadcastCheckDelay: movedFundsSweepBroadcastCheckDelay, + permit: permit, } } func (mfsa *movedFundsSweepAction) execute() error { + // The action owns its permit from dispatch on; releasing it here ends the + // ceremony's active accounting in the participation gate. The terminal + // outcome is registered afterwards so it runs first and reaches the permit + // while it is still open. + defer mfsa.permit.Close() + defer mfsa.transactionExecutor.recordTerminalOutcome(mfsa.logger) + validateProposalLogger := mfsa.logger.With( zap.String("step", "validateProposal"), ) @@ -217,7 +234,7 @@ func (mfsa *movedFundsSweepAction) execute() error { mfsa.proposalExpiryBlock-mfsa.signingTimeoutSafetyMarginBlocks, ) if err != nil { - return fmt.Errorf("sign transaction step failed: [%v]", err) + return fmt.Errorf("sign transaction step failed: [%w]", err) } broadcastTxLogger := mfsa.logger.With( @@ -235,7 +252,7 @@ func (mfsa *movedFundsSweepAction) execute() error { mfsa.broadcastCheckDelay, ) if err != nil { - return fmt.Errorf("broadcast transaction step failed: [%v]", err) + return fmt.Errorf("broadcast transaction step failed: [%w]", err) } return nil @@ -256,7 +273,17 @@ func assembleMovedFundsSweepUtxo( ) } - movingFundsTxValue := movingFundsTx.Outputs[movingFundsTxOutputIdx].Value + // The moving funds transaction is fetched from the Bitcoin node, so its + // output count is untrusted; use the bounds-checked accessor to avoid an + // out-of-range panic on a short or malformed node response. + movingFundsTxOutput, err := movingFundsTx.OutputAt(movingFundsTxOutputIdx) + if err != nil { + return nil, fmt.Errorf( + "could not get moving funds transaction output: [%v]", + err, + ) + } + movingFundsTxValue := movingFundsTxOutput.Value return &bitcoin.UnspentTransactionOutput{ Outpoint: &bitcoin.TransactionOutpoint{ diff --git a/pkg/tbtc/moved_funds_sweep_test.go b/pkg/tbtc/moved_funds_sweep_test.go index 68ae7be032..48e110623e 100644 --- a/pkg/tbtc/moved_funds_sweep_test.go +++ b/pkg/tbtc/moved_funds_sweep_test.go @@ -11,6 +11,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tbtc/internal/test" ) @@ -112,6 +113,7 @@ func TestMovedFundsSweepAction_Execute(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCSigning), ) // Modify the default parameters of the action to make diff --git a/pkg/tbtc/moving_funds.go b/pkg/tbtc/moving_funds.go index 1e9c01b0a3..ec501b31fd 100644 --- a/pkg/tbtc/moving_funds.go +++ b/pkg/tbtc/moving_funds.go @@ -10,6 +10,7 @@ import ( "github.com/keep-network/keep-common/pkg/chain/ethereum" "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/protocol/participation" "go.uber.org/zap" ) @@ -91,6 +92,11 @@ type movingFundsAction struct { signingTimeoutSafetyMarginBlocks uint64 broadcastTimeout time.Duration broadcastCheckDelay time.Duration + + // permit is the wallet action's participation permit: it pins the + // protocol mode for every signing of this action and is released when the + // action's execution ends. + permit participation.Permit } func newMovingFundsAction( @@ -103,12 +109,15 @@ func newMovingFundsAction( proposalProcessingStartBlock uint64, proposalExpiryBlock uint64, waitForBlockFn waitForBlockFn, + permit participation.Permit, ) *movingFundsAction { transactionExecutor := newWalletTransactionExecutor( btcChain, movingFundsWallet, signingExecutor, waitForBlockFn, + permit, + "tbtc_moving_funds_bitcoin_broadcast", ) return &movingFundsAction{ @@ -123,10 +132,18 @@ func newMovingFundsAction( signingTimeoutSafetyMarginBlocks: movingFundsSigningTimeoutSafetyMarginBlocks, broadcastTimeout: movingFundsBroadcastTimeout, broadcastCheckDelay: movingFundsBroadcastCheckDelay, + permit: permit, } } func (mfa *movingFundsAction) execute() error { + // The action owns its permit from dispatch on; releasing it here ends the + // ceremony's active accounting in the participation gate. The terminal + // outcome is registered afterwards so it runs first and reaches the permit + // while it is still open. + defer mfa.permit.Close() + defer mfa.transactionExecutor.recordTerminalOutcome(mfa.logger) + validateProposalLogger := mfa.logger.With( zap.String("step", "validateProposal"), ) @@ -231,7 +248,7 @@ func (mfa *movingFundsAction) execute() error { mfa.proposalExpiryBlock-mfa.signingTimeoutSafetyMarginBlocks, ) if err != nil { - return fmt.Errorf("sign transaction step failed: [%v]", err) + return fmt.Errorf("sign transaction step failed: [%w]", err) } broadcastTxLogger := mfa.logger.With( @@ -249,7 +266,7 @@ func (mfa *movingFundsAction) execute() error { mfa.broadcastCheckDelay, ) if err != nil { - return fmt.Errorf("broadcast transaction step failed: [%v]", err) + return fmt.Errorf("broadcast transaction step failed: [%w]", err) } return nil diff --git a/pkg/tbtc/moving_funds_test.go b/pkg/tbtc/moving_funds_test.go index d1fb2b99d4..313a332422 100644 --- a/pkg/tbtc/moving_funds_test.go +++ b/pkg/tbtc/moving_funds_test.go @@ -11,6 +11,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tbtc/internal/test" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -123,6 +124,7 @@ func TestMovingFundsAction_Execute(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCSigning), ) // Modify the default parameters of the action to make diff --git a/pkg/tbtc/node.go b/pkg/tbtc/node.go index f8f40b9f7c..32525619c6 100644 --- a/pkg/tbtc/node.go +++ b/pkg/tbtc/node.go @@ -3,12 +3,15 @@ package tbtc import ( "context" "crypto/ecdsa" + "encoding/binary" "encoding/hex" "fmt" "math/big" + "slices" "sync" "time" + "github.com/ipfs/go-log/v2" "github.com/keep-network/keep-common/pkg/chain/ethereum" "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/chain" @@ -22,6 +25,7 @@ import ( "github.com/keep-network/keep-core/pkg/protocol/announcer" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/protocol/inactivity" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa/signing" ) @@ -123,6 +127,78 @@ type node struct { // windowMetricsTracker tracks detailed metrics for individual coordination windows windowMetricsTracker *coordinationWindowMetrics + + // cutoverPeerRoster is the node-local, deduplicated record of post-cutover + // legacy peer sightings. It is constructed unconditionally at process + // startup beside the participation gate, including when client-info is + // disabled, and is shared by the DKG and signing executors. It may be nil + // in tests that do not exercise the cutover observability path. + cutoverPeerRoster *participation.CutoverPeerRoster + + // participationGate issues the per-ceremony participation permits that pin + // each ceremony's protocol mode from its canonical chain anchor. It is + // constructed once at process startup beside the cutover peer roster and + // shared with the beacon application. Every tBTC ceremony choke point — + // DKG members, wallet coordination, wallet actions and their signings, + // heartbeat and derived inactivity work — acquires a permit from it and + // fails closed without one. + participationGate participation.Gate +} + +// walletPermitIdentity binds a wallet permit to the wallet and canonical block +// it runs for. operatedMembers is this node's own seats in that wallet's +// signing group, read from the registry at issuance. +func walletPermitIdentity( + workClass string, + walletPublicKey *ecdsa.PublicKey, + canonicalStartBlock uint64, + operatedMembers participation.MemberIndexes, +) participation.PermitIdentity { + walletPublicKeyHash := bitcoin.PublicKeyHash(walletPublicKey) + walletID := hex.EncodeToString(walletPublicKeyHash[:]) + + return participation.PermitIdentity{ + WorkID: fmt.Sprintf( + "%s-%d-%s", + workClass, + canonicalStartBlock, + walletID, + ), + PermitID: walletID, + OperatedMembers: operatedMembers, + } +} + +// walletSigningGroupSeats reports this node's seats in the given wallet's +// signing group, ascending and distinct, as the wallet registry holds them. +// +// It is read at permit issuance rather than derived later from whatever the +// ceremony produced. A wallet action's permit is one permit covering every seat +// this node operates in the wallet, so unlike a per-seat DKG or relay permit it +// cannot carry the seat in its identity — and a reader assembling which seats +// this node held on this wallet action has nowhere else to get them if the +// action ends without a result. +// +// An empty result is a real answer and is recorded as one: a node whose signers +// for the wallet were archived between the coordination window and the action +// operates no seat in it, and saying so is more accurate than declining to +// record the permit at all. +func (n *node) walletSigningGroupSeats( + walletPublicKey *ecdsa.PublicKey, +) participation.MemberIndexes { + signers := n.walletRegistry.getSigners(walletPublicKey) + + seats := make(participation.MemberIndexes, 0, len(signers)) + for _, signer := range signers { + if !slices.Contains(seats, signer.signingGroupMemberIndex) { + seats = append(seats, signer.signingGroupMemberIndex) + } + } + // The gate requires an ascending, duplicate-free set: one set has exactly + // one encoding, so two readings of the same permit compare equal. + slices.Sort(seats) + + return seats } func newNode( @@ -239,6 +315,30 @@ func (n *node) setPerformanceMetrics(metrics interface { n.coordinationExecutorsMutex.Unlock() } +// setCutoverPeerRoster sets the node-local cutover peer roster and propagates it +// into the components that observe announcer session-ID mismatches. It is called +// once during initialization, before the coordination layer starts, so in +// practice no signing executor exists yet; the propagation loop is a defensive +// safeguard for any executor created by an early coordination round. +// +// The field write is guarded by signingExecutorsMutex because getSigningExecutor +// reads n.cutoverPeerRoster under the same lock when wiring a freshly created +// executor; without this the read/write pair would be an unsynchronized race. +func (n *node) setCutoverPeerRoster(roster *participation.CutoverPeerRoster) { + n.signingExecutorsMutex.Lock() + n.cutoverPeerRoster = roster + for _, executor := range n.signingExecutors { + executor.setCutoverPeerRoster(roster) + } + n.signingExecutorsMutex.Unlock() + + // The DKG executor is created once in newNode and never mutated + // concurrently, so it is wired directly. + if n.dkgExecutor != nil { + n.dkgExecutor.setCutoverPeerRoster(roster) + } +} + // GetCoordinationWindowsSummary returns a summary of coordination window metrics. // Returns nil if the window metrics tracker is not initialized. func (n *node) GetCoordinationWindowsSummary() *WindowMetricsSummary { @@ -402,6 +502,7 @@ func (n *node) getSigningExecutor( blockCounter.CurrentBlock, n.waitForBlockHeight, signingAttemptsLimit, + n.participationGate, ) // Wire metrics recorder if available @@ -409,6 +510,12 @@ func (n *node) getSigningExecutor( executor.setMetricsRecorder(n.performanceMetrics) } + // Wire the node-local cutover peer roster so the signing announcer can + // record post-cutover legacy peer sightings. + if n.cutoverPeerRoster != nil { + executor.setCutoverPeerRoster(n.cutoverPeerRoster) + } + n.signingExecutors[executorKey] = executor return executor, true, nil @@ -614,7 +721,18 @@ func (n *node) handleHeartbeatProposal( proposal *HeartbeatProposal, startBlock uint64, expiryBlock uint64, + permit participation.Permit, ) { + // Until the action is dispatched the permit is owned here and every + // early return must release it; after a successful dispatch the action + // owns it for its whole execution. + permitHandedOff := false + defer func() { + if !permitHandedOff { + permit.Close() + } + }() + walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { logger.Errorf("cannot marshal wallet public key: [%v]", err) @@ -683,6 +801,7 @@ func (n *node) handleHeartbeatProposal( startBlock, expiryBlock, n.waitForBlockHeight, + permit, ) err = n.walletDispatcher.dispatch(action) @@ -690,6 +809,7 @@ func (n *node) handleHeartbeatProposal( walletActionLogger.Errorf("cannot dispatch wallet action: [%v]", err) return } + permitHandedOff = true walletActionLogger.Infof("wallet action dispatched successfully") } @@ -701,7 +821,18 @@ func (n *node) handleDepositSweepProposal( proposal *DepositSweepProposal, startBlock uint64, expiryBlock uint64, + permit participation.Permit, ) { + // Until the action is dispatched the permit is owned here and every + // early return must release it; after a successful dispatch the action + // owns it for its whole execution. + permitHandedOff := false + defer func() { + if !permitHandedOff { + permit.Close() + } + }() + walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { logger.Errorf("cannot marshal wallet public key: [%v]", err) @@ -751,6 +882,7 @@ func (n *node) handleDepositSweepProposal( startBlock, expiryBlock, n.waitForBlockHeight, + permit, ) // Wire metrics recorder if available @@ -763,6 +895,7 @@ func (n *node) handleDepositSweepProposal( walletActionLogger.Errorf("cannot dispatch wallet action: [%v]", err) return } + permitHandedOff = true walletActionLogger.Infof("wallet action dispatched successfully") } @@ -774,7 +907,18 @@ func (n *node) handleRedemptionProposal( proposal *RedemptionProposal, startBlock uint64, expiryBlock uint64, + permit participation.Permit, ) { + // Until the action is dispatched the permit is owned here and every + // early return must release it; after a successful dispatch the action + // owns it for its whole execution. + permitHandedOff := false + defer func() { + if !permitHandedOff { + permit.Close() + } + }() + walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { logger.Errorf("cannot marshal wallet public key: [%v]", err) @@ -824,6 +968,7 @@ func (n *node) handleRedemptionProposal( startBlock, expiryBlock, n.waitForBlockHeight, + permit, ) // Wire metrics recorder if available @@ -836,6 +981,7 @@ func (n *node) handleRedemptionProposal( walletActionLogger.Errorf("cannot dispatch wallet action: [%v]", err) return } + permitHandedOff = true walletActionLogger.Infof("wallet action dispatched successfully") } @@ -847,7 +993,18 @@ func (n *node) handleMovingFundsProposal( proposal *MovingFundsProposal, startBlock uint64, expiryBlock uint64, + permit participation.Permit, ) { + // Until the action is dispatched the permit is owned here and every + // early return must release it; after a successful dispatch the action + // owns it for its whole execution. + permitHandedOff := false + defer func() { + if !permitHandedOff { + permit.Close() + } + }() + walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { logger.Errorf("cannot marshal wallet public key: [%v]", err) @@ -897,6 +1054,7 @@ func (n *node) handleMovingFundsProposal( startBlock, expiryBlock, n.waitForBlockHeight, + permit, ) err = n.walletDispatcher.dispatch(action) @@ -904,6 +1062,7 @@ func (n *node) handleMovingFundsProposal( walletActionLogger.Errorf("cannot dispatch wallet action: [%v]", err) return } + permitHandedOff = true walletActionLogger.Infof("wallet action dispatched successfully") } @@ -915,7 +1074,18 @@ func (n *node) handleMovedFundsSweepProposal( proposal *MovedFundsSweepProposal, startBlock uint64, expiryBlock uint64, + permit participation.Permit, ) { + // Until the action is dispatched the permit is owned here and every + // early return must release it; after a successful dispatch the action + // owns it for its whole execution. + permitHandedOff := false + defer func() { + if !permitHandedOff { + permit.Close() + } + }() + walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { logger.Errorf("cannot marshal wallet public key: [%v]", err) @@ -965,6 +1135,7 @@ func (n *node) handleMovedFundsSweepProposal( startBlock, expiryBlock, n.waitForBlockHeight, + permit, ) err = n.walletDispatcher.dispatch(action) @@ -972,6 +1143,7 @@ func (n *node) handleMovedFundsSweepProposal( walletActionLogger.Errorf("cannot dispatch wallet action: [%v]", err) return } + permitHandedOff = true walletActionLogger.Infof("wallet action dispatched successfully") } @@ -987,8 +1159,10 @@ type coordinationLayerSettings struct { ) (*coordinationResult, bool) // processCoordinationResultFn is a function processing the given - // coordination result. + // coordination result. The context bounds the processing lifetime and + // is done when the coordination layer shuts down. processCoordinationResultFn func( + ctx context.Context, node *node, result *coordinationResult, ) @@ -1084,7 +1258,7 @@ func (n *node) runCoordinationLayer( for { select { case result := <-coordinationResultChan: - go cls.processCoordinationResultFn(n, result) + go cls.processCoordinationResultFn(ctx, n, result) case <-ctx.Done(): return } @@ -1139,39 +1313,89 @@ func executeCoordinationProcedure( return nil, false } + if node.participationGate == nil { + // Without the gate no permit can track the procedure for clock + // failure and quiescence. Fail closed. + procedureLogger.Errorf( + "no participation gate; refusing the coordination procedure", + ) + return nil, false + } + + // One coordination permit tracks the procedure for clock failure and + // quiescence, anchored at the window's coordination block. It ends with + // the procedure and does not authorize or select the later wallet + // action's cryptographic mode: the coordination wire format is shared by + // both releases, so the permit's mode is telemetry here and the procedure + // runs in either mode. A refusal is a gate decision, not an ordinary + // coordination failure. + permit, err := node.participationGate.Begin( + participation.TBTCWalletCoordination, + window.coordinationBlock, + walletPermitIdentity( + "wallet-coordination", + walletPublicKey, + window.coordinationBlock, + node.walletSigningGroupSeats(walletPublicKey), + ), + ) + if err != nil { + procedureLogger.Warnf( + "coordination procedure refused by the participation gate: [%v]", + err, + ) + return nil, false + } + // agreedResult holds the coordination procedure's durable result once the + // wallet agrees on one. The deferred recorder below reads its final value. + var agreedResult *coordinationResult + + // The terminal outcome is registered after the release so it runs first and + // reaches the permit while it is still open. + defer permit.Close() + defer func() { + recordCoordinationTerminalOutcome( + procedureLogger, + permit, + walletPublicKeyBytes, + agreedResult, + ) + }() + startTime := time.Now() - result, err := executor.coordinate(window) + result, err := executor.coordinate(permit.Context(), window) duration := time.Since(startTime) if err != nil { - procedureLogger.Errorf("coordination procedure failed: [%v]", err) - // Metrics are already recorded in executor.coordinate() for failures - - // Record window metrics for failed coordination - if node.windowMetricsTracker != nil { - walletPublicKeyHash := bitcoin.PublicKeyHash(walletPublicKey) - // Extract leader and faults from partial result if available - // (e.g., when follower routine fails, we know who the leader was) - leader := chain.Address("") - var faults []*coordinationFault - if result != nil { - leader = result.leader - faults = result.faults - } - node.windowMetricsTracker.recordWalletCoordination( - window, - walletPublicKeyHash, - leader, - "", - false, - duration, - faults, - err, // capture the error message + // A gate-canceled permit — clock failure, forced quiescence — ended + // the procedure; that is a release-gate decision, not an ordinary + // coordination failure. + gateAborted := participation.IsGateRefusal(context.Cause(permit.Context())) + if gateAborted { + procedureLogger.Warnf( + "coordination procedure canceled by the participation "+ + "gate: [%v]", + err, ) + } else { + procedureLogger.Errorf("coordination procedure failed: [%v]", err) } + // Metrics are already recorded in executor.coordinate() for failures + + recordCoordinationOutcome( + node, + window, + walletPublicKey, + result, + duration, + err, + gateAborted, + ) return nil, false } + agreedResult = result + procedureLogger.Infof( "coordination procedure finished successfully with result [%s]", result, @@ -1179,30 +1403,144 @@ func executeCoordinationProcedure( // Metrics are already recorded in executor.coordinate() for successful executions - // Record window metrics for successful coordination - if node.windowMetricsTracker != nil { - walletPublicKeyHash := bitcoin.PublicKeyHash(walletPublicKey) - actionType := "" - if result.proposal != nil { - actionType = result.proposal.ActionType().String() + recordCoordinationOutcome( + node, + window, + walletPublicKey, + result, + duration, + nil, + false, + ) + + return result, true +} + +// recordCoordinationTerminalOutcome reports the coordination ceremony's +// node-owned final disposition on its participation permit. The procedure's +// durable result is the agreed proposal that dispatches the wallet action; the +// action itself runs under its own permit and reports its own outcome. A +// procedure that never agreed on a result — including one the release gate +// canceled — left nothing behind and is recorded as exhausted. +// +// The evidence binds the proposal's full serialized payload, not just its +// action type: a wallet can agree on two different deposit sweeps or two +// different redemptions in the same window across a restart, and an identity +// that only named the type would report both as the same durable result. +func recordCoordinationTerminalOutcome( + procedureLogger log.StandardLogger, + permit participation.Permit, + walletPublicKeyBytes []byte, + result *coordinationResult, +) { + if result == nil { + recordPermitNoThreshold(procedureLogger, permit) + return + } + + // A proposal the node cannot serialize has no faithful identity, and the + // procedure did dispatch a wallet action, so neither a weaker reference nor + // an exhausted record would be honest. Leaving the permit without a + // terminal outcome closes it as unresolved, which blocks the offline + // barrier until an operator reconciles the window by hand. + proposalBytes, err := result.proposal.Marshal() + if err != nil { + procedureLogger.Errorf( + "could not derive the coordination result identity for the "+ + "node-authored terminal outcome: [%v]", + err, + ) + return + } + + coordinationBlock := make([]byte, 8) + binary.BigEndian.PutUint64(coordinationBlock, result.window.coordinationBlock) + + recordPermitTerminalOutcome( + procedureLogger, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceProtocolResult, + Reference: participation.TerminalResultReference( + "tbtc_wallet_coordination_result", + walletPublicKeyBytes, + coordinationBlock, + []byte(result.leader), + []byte(result.proposal.ActionType().String()), + proposalBytes, + ), + }, + ) +} + +// recordCoordinationOutcome records one wallet's coordination outcome in the +// window metrics tracker. A gate-aborted procedure is deliberately not +// recorded at all: the release gate canceling a procedure is not a +// coordination failure of the wallet or the window, so it must not +// contaminate the window's coordinated/failed accounting that operators read +// as ordinary protocol health. +func recordCoordinationOutcome( + node *node, + window *coordinationWindow, + walletPublicKey *ecdsa.PublicKey, + result *coordinationResult, + duration time.Duration, + coordinationErr error, + gateAborted bool, +) { + if node.windowMetricsTracker == nil || gateAborted { + return + } + + walletPublicKeyHash := bitcoin.PublicKeyHash(walletPublicKey) + + if coordinationErr != nil { + // Extract leader and faults from partial result if available + // (e.g., when follower routine fails, we know who the leader was) + leader := chain.Address("") + var faults []*coordinationFault + if result != nil { + leader = result.leader + faults = result.faults } node.windowMetricsTracker.recordWalletCoordination( window, walletPublicKeyHash, - result.leader, - actionType, - true, + leader, + "", + false, duration, - result.faults, - nil, // no error on success + faults, + coordinationErr, // capture the error message ) + return } - return result, true + actionType := "" + if result.proposal != nil { + actionType = result.proposal.ActionType().String() + } + node.windowMetricsTracker.recordWalletCoordination( + window, + walletPublicKeyHash, + result.leader, + actionType, + true, + duration, + result.faults, + nil, // no error on success + ) } -// processCoordinationResult processes the given coordination result. -func processCoordinationResult(node *node, result *coordinationResult) { +// processCoordinationResult processes the given coordination result. The +// context bounds the pre-dispatch wait for the action's start block and is +// done when the coordination layer shuts down. +func processCoordinationResult( + ctx context.Context, + node *node, + result *coordinationResult, +) { logger.Infof("processing coordination result [%s]", result) // TODO: In the future, create coordination faults cache and @@ -1219,6 +1557,43 @@ func processCoordinationResult(node *node, result *coordinationResult) { startBlock := result.window.endBlock() expiryBlock := startBlock + result.proposal.ValidityBlocks() + // Coordination normally concludes during the window's active phase, so + // at this point the chain has not reached the window's end block that + // anchors the action. The gate accepts only anchors at or below the + // current height, hence the permit is acquired once the chain reaches + // the anchor; the anchor itself stays the same for the whole action, + // including all its retries. + if err := node.waitForBlockHeight(ctx, startBlock); err != nil { + logger.Errorf( + "failed to wait for the [%s] wallet action start block [%v]: [%v]", + proposedAction, + startBlock, + err, + ) + return + } + if ctx.Err() != nil { + // waitForBlockHeight returns nil when the context ends before the + // height is reached; the coordination layer is shutting down, so + // the action is dropped without touching the gate. + return + } + + // One action permit, acquired before the handler and the dispatcher are + // set up and anchored at the proposal-processing start block. Every + // signing and terminal commit of the dispatched action derives from it; + // the heartbeat ceremony additionally fences its derived inactivity work + // through it. The handlers hand the permit to the action, which owns it + // until its execution ends. + permit := node.beginWalletActionPermit( + proposedAction, + startBlock, + result.wallet.publicKey, + ) + if permit == nil { + return + } + switch proposedAction { case ActionHeartbeat: if proposal, ok := result.proposal.(*HeartbeatProposal); ok { @@ -1227,7 +1602,9 @@ func processCoordinationResult(node *node, result *coordinationResult) { proposal, startBlock, expiryBlock, + permit, ) + return } case ActionDepositSweep: if proposal, ok := result.proposal.(*DepositSweepProposal); ok { @@ -1236,7 +1613,9 @@ func processCoordinationResult(node *node, result *coordinationResult) { proposal, startBlock, expiryBlock, + permit, ) + return } case ActionRedemption: if proposal, ok := result.proposal.(*RedemptionProposal); ok { @@ -1245,7 +1624,9 @@ func processCoordinationResult(node *node, result *coordinationResult) { proposal, startBlock, expiryBlock, + permit, ) + return } case ActionMovingFunds: if proposal, ok := result.proposal.(*MovingFundsProposal); ok { @@ -1254,7 +1635,9 @@ func processCoordinationResult(node *node, result *coordinationResult) { proposal, startBlock, expiryBlock, + permit, ) + return } case ActionMovedFundsSweep: if proposal, ok := result.proposal.(*MovedFundsSweepProposal); ok { @@ -1263,11 +1646,85 @@ func processCoordinationResult(node *node, result *coordinationResult) { proposal, startBlock, expiryBlock, + permit, ) + return } default: logger.Errorf("no handler for coordination result [%s]", result) } + + // A mismatched proposal type or an unknown action never reached a + // handler, so the permit is released here. + permit.Close() +} + +// beginWalletActionPermit acquires the participation permit for a wallet +// action about to be orchestrated. It fails closed: without a gate or on a +// gate refusal, no permit is returned and the action must not be dispatched. +func (n *node) beginWalletActionPermit( + proposedAction WalletActionType, + startBlock uint64, + walletPublicKeys ...*ecdsa.PublicKey, +) participation.Permit { + if n.participationGate == nil { + logger.Errorf( + "no participation gate; refusing the [%s] wallet action", + proposedAction, + ) + return nil + } + + // The heartbeat is its own ceremony class because its penalty semantics + // differ; every other wallet action is a signing ceremony. + ceremony := participation.TBTCSigning + if proposedAction == ActionHeartbeat { + ceremony = participation.TBTCHeartbeat + } + + var identities []participation.PermitIdentity + if len(walletPublicKeys) > 1 { + logger.Errorf( + "refusing the [%s] wallet action: [%d] wallet identities supplied", + proposedAction, + len(walletPublicKeys), + ) + return nil + } + if len(walletPublicKeys) == 1 { + if walletPublicKeys[0] == nil { + logger.Errorf( + "refusing the [%s] wallet action: nil wallet identity", + proposedAction, + ) + return nil + } + identities = append( + identities, + walletPermitIdentity( + fmt.Sprintf("wallet-action-%d", proposedAction), + walletPublicKeys[0], + startBlock, + n.walletSigningGroupSeats(walletPublicKeys[0]), + ), + ) + } + + permit, err := n.participationGate.Begin( + ceremony, + startBlock, + identities..., + ) + if err != nil { + logger.Warnf( + "[%s] wallet action refused by the participation gate: [%v]", + proposedAction, + err, + ) + return nil + } + + return permit } // archiveClosedWallets archives closed or terminated wallets. @@ -1416,6 +1873,12 @@ func (n *node) waitForBlockHeight(ctx context.Context, blockHeight uint64) error select { case <-wait: case <-ctx.Done(): + // The block counter delivers exactly one notification per waiter + // with a blocking send on an unbuffered channel once the height is + // reached. Simply abandoning the channel would park that sender + // goroutine forever, so a drain goroutine performs the single + // receive and lets the eventual sender terminate. + go func() { <-wait }() } return nil diff --git a/pkg/tbtc/node_operated_seats_test.go b/pkg/tbtc/node_operated_seats_test.go new file mode 100644 index 0000000000..e5b7e9d326 --- /dev/null +++ b/pkg/tbtc/node_operated_seats_test.go @@ -0,0 +1,142 @@ +package tbtc + +import ( + "crypto/ecdsa" + "crypto/rand" + "testing" + + "github.com/btcsuite/btcd/btcec" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/tecdsa" +) + +// A wallet action takes one permit covering every seat this node holds in the +// wallet, so unlike a per-seat DKG or relay permit it cannot carry the seat in +// its identity. These tests cover the reader that supplies them instead, because +// a reader that under-reports leaves a seat outside the fleet's ownership map — +// and a seat outside that map is attributed to whatever other release is on the +// network. + +// TestWalletSigningGroupSeats_NamesEverySeatTheRegistryHolds covers the case a +// per-seat identity cannot express: one node operating several seats of one +// wallet under one permit. +func TestWalletSigningGroupSeats_NamesEverySeatTheRegistryHolds(t *testing.T) { + registry, walletPublicKey := newSeatTestRegistry(t, 9, 2, 5) + node := &node{walletRegistry: registry} + + assertSeats( + t, + "the seats of a wallet this node holds three memberships in", + node.walletSigningGroupSeats(walletPublicKey), + []group.MemberIndex{2, 5, 9}, + ) +} + +// TestWalletSigningGroupSeats_NamesNoSeatOfAnUnheldWallet is the honest empty +// answer, and it has to be empty rather than absent: a permit is still issued +// for the action, and a reader has to be able to tell "this node operated no +// seat here" from "this node was never asked". +func TestWalletSigningGroupSeats_NamesNoSeatOfAnUnheldWallet(t *testing.T) { + registry, _ := newSeatTestRegistry(t, 1) + node := &node{walletRegistry: registry} + + // A genuinely different wallet. The key-share fixtures are all shares of one + // wallet key, so a second fixture would ask about the same wallet under + // another name. + otherKey, err := ecdsa.GenerateKey(btcec.S256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + other := &otherKey.PublicKey + + if seats := node.walletSigningGroupSeats(other); len(seats) != 0 { + t.Fatalf("a wallet this node holds no membership in named %v", seats) + } +} + +// TestWalletPermitIdentity_CarriesTheOperatedSeats holds the wiring itself: the +// seats have to reach the permit identity the gate validates, not merely be +// readable from the registry. +func TestWalletPermitIdentity_CarriesTheOperatedSeats(t *testing.T) { + registry, walletPublicKey := newSeatTestRegistry(t, 4, 1) + node := &node{walletRegistry: registry} + + identity := walletPermitIdentity( + "wallet-action-1", + walletPublicKey, + 1_000, + node.walletSigningGroupSeats(walletPublicKey), + ) + + assertSeats( + t, + "the seats a wallet permit identity carries", + identity.OperatedMembers, + []group.MemberIndex{1, 4}, + ) +} + +// newSeatTestRegistry returns a registry holding one wallet with the given +// signing group seats, registered as a node controlling all of them would. +func newSeatTestRegistry( + t *testing.T, + seats ...group.MemberIndex, +) (*walletRegistry, *ecdsa.PublicKey) { + t.Helper() + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + privateKeyShare := tecdsa.NewPrivateKeyShare(testData[0]) + + registry, err := newWalletRegistry( + &mockPersistenceHandle{}, + func(publicKey *ecdsa.PublicKey) ([32]byte, error) { + return [32]byte{0x5c}, nil + }, + ) + if err != nil { + t.Fatal(err) + } + + for _, seat := range seats { + if err := registry.registerSigner(&signer{ + wallet: wallet{ + publicKey: privateKeyShare.PublicKey(), + signingGroupOperators: []chain.Address{ + "address-1", + "address-2", + "address-3", + }, + }, + signingGroupMemberIndex: seat, + privateKeyShare: privateKeyShare, + }); err != nil { + t.Fatalf("failed to register the signer of seat [%v]: [%v]", seat, err) + } + } + + return registry, privateKeyShare.PublicKey() +} + +func assertSeats( + t *testing.T, + subject string, + actual []group.MemberIndex, + expected []group.MemberIndex, +) { + t.Helper() + + if len(actual) != len(expected) { + t.Fatalf("%s are %v, expected %v", subject, actual, expected) + } + for i, seat := range expected { + if actual[i] != seat { + t.Fatalf("%s are %v, expected %v", subject, actual, expected) + } + } +} diff --git a/pkg/tbtc/node_test.go b/pkg/tbtc/node_test.go index a756c69595..97aadf3dd8 100644 --- a/pkg/tbtc/node_test.go +++ b/pkg/tbtc/node_test.go @@ -7,6 +7,7 @@ import ( "fmt" "math/big" "reflect" + "sync" "testing" "time" @@ -14,10 +15,12 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/generator" "github.com/keep-network/keep-core/pkg/internal/tecdsatest" "github.com/keep-network/keep-core/pkg/net/local" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -379,6 +382,7 @@ func TestNode_RunCoordinationLayer(t *testing.T) { // Simply pass processed results to the channel. processedResultsChan := make(chan *coordinationResult, 5) processCoordinationResultFn := func( + _ context.Context, _ *node, result *coordinationResult, ) { @@ -527,7 +531,7 @@ func TestNode_HandleHeartbeatProposal_WalletNotControlled(t *testing.T) { uncontrolledWallet := uncontrolledWalletFor(signer) proposal := &HeartbeatProposal{Message: [16]byte{0x01}} - n.handleHeartbeatProposal(uncontrolledWallet, proposal, 10, 100) + n.handleHeartbeatProposal(uncontrolledWallet, proposal, 10, 100, newTestPermit(participation.TBTCHeartbeat)) if count := dispatchedActionsCount(n); count != 0 { t.Errorf("expected no dispatched actions for uncontrolled wallet, got %d", count) @@ -547,7 +551,7 @@ func TestNode_HandleHeartbeatProposal_WalletBusy(t *testing.T) { n.walletDispatcher.actions[walletKey] = ActionHeartbeat }() - n.handleHeartbeatProposal(signer.wallet, &HeartbeatProposal{Message: [16]byte{0x02}}, 10, 100) + n.handleHeartbeatProposal(signer.wallet, &HeartbeatProposal{Message: [16]byte{0x02}}, 10, 100, newTestPermit(participation.TBTCHeartbeat)) // The pre-populated entry must still be there -- our call did not modify it. actionType, ok := func() (WalletActionType, bool) { @@ -571,7 +575,7 @@ func TestNode_HandleHeartbeatProposal_WalletBusy(t *testing.T) { func TestNode_HandleHeartbeatProposal_DispatchesAction(t *testing.T) { n, signer := setupNodeForHandlerTests(t) - n.handleHeartbeatProposal(signer.wallet, &HeartbeatProposal{Message: [16]byte{0x03}}, 10, 100) + n.handleHeartbeatProposal(signer.wallet, &HeartbeatProposal{Message: [16]byte{0x03}}, 10, 100, newTestPermit(participation.TBTCHeartbeat)) waitForDispatcherIdle(t, n) @@ -590,7 +594,7 @@ func TestNode_HandleDepositSweepProposal_WalletNotControlled(t *testing.T) { uncontrolledWallet := uncontrolledWalletFor(signer) proposal := &DepositSweepProposal{} - n.handleDepositSweepProposal(uncontrolledWallet, proposal, 10, 100) + n.handleDepositSweepProposal(uncontrolledWallet, proposal, 10, 100, newTestPermit(participation.TBTCSigning)) if count := dispatchedActionsCount(n); count != 0 { t.Errorf("expected no dispatched actions for uncontrolled wallet, got %d", count) @@ -609,7 +613,7 @@ func TestNode_HandleDepositSweepProposal_WalletBusy(t *testing.T) { n.walletDispatcher.actions[walletKey] = ActionDepositSweep }() - n.handleDepositSweepProposal(signer.wallet, &DepositSweepProposal{}, 10, 100) + n.handleDepositSweepProposal(signer.wallet, &DepositSweepProposal{}, 10, 100, newTestPermit(participation.TBTCSigning)) actionType, ok := func() (WalletActionType, bool) { n.walletDispatcher.actionsMutex.Lock() @@ -637,6 +641,7 @@ func TestNode_HandleDepositSweepProposal_DispatchesAction(t *testing.T) { &DepositSweepProposal{SweepTxFee: big.NewInt(0)}, 10, 100, + newTestPermit(participation.TBTCSigning), ) waitForDispatcherIdle(t, n) @@ -656,7 +661,7 @@ func TestNode_HandleRedemptionProposal_WalletNotControlled(t *testing.T) { uncontrolledWallet := uncontrolledWalletFor(signer) proposal := &RedemptionProposal{} - n.handleRedemptionProposal(uncontrolledWallet, proposal, 10, 100) + n.handleRedemptionProposal(uncontrolledWallet, proposal, 10, 100, newTestPermit(participation.TBTCSigning)) if count := dispatchedActionsCount(n); count != 0 { t.Errorf("expected no dispatched actions for uncontrolled wallet, got %d", count) @@ -675,7 +680,7 @@ func TestNode_HandleRedemptionProposal_WalletBusy(t *testing.T) { n.walletDispatcher.actions[walletKey] = ActionRedemption }() - n.handleRedemptionProposal(signer.wallet, &RedemptionProposal{RedemptionTxFee: big.NewInt(0)}, 10, 100) + n.handleRedemptionProposal(signer.wallet, &RedemptionProposal{RedemptionTxFee: big.NewInt(0)}, 10, 100, newTestPermit(participation.TBTCSigning)) actionType, ok := func() (WalletActionType, bool) { n.walletDispatcher.actionsMutex.Lock() @@ -703,6 +708,7 @@ func TestNode_HandleRedemptionProposal_DispatchesAction(t *testing.T) { &RedemptionProposal{RedemptionTxFee: big.NewInt(0)}, 10, 100, + newTestPermit(participation.TBTCSigning), ) waitForDispatcherIdle(t, n) @@ -722,7 +728,7 @@ func TestNode_HandleMovingFundsProposal_WalletNotControlled(t *testing.T) { uncontrolledWallet := uncontrolledWalletFor(signer) proposal := &MovingFundsProposal{} - n.handleMovingFundsProposal(uncontrolledWallet, proposal, 10, 100) + n.handleMovingFundsProposal(uncontrolledWallet, proposal, 10, 100, newTestPermit(participation.TBTCSigning)) if count := dispatchedActionsCount(n); count != 0 { t.Errorf("expected no dispatched actions for uncontrolled wallet, got %d", count) @@ -741,7 +747,7 @@ func TestNode_HandleMovingFundsProposal_WalletBusy(t *testing.T) { n.walletDispatcher.actions[walletKey] = ActionMovingFunds }() - n.handleMovingFundsProposal(signer.wallet, &MovingFundsProposal{}, 10, 100) + n.handleMovingFundsProposal(signer.wallet, &MovingFundsProposal{}, 10, 100, newTestPermit(participation.TBTCSigning)) actionType, ok := func() (WalletActionType, bool) { n.walletDispatcher.actionsMutex.Lock() @@ -764,7 +770,7 @@ func TestNode_HandleMovingFundsProposal_WalletBusy(t *testing.T) { func TestNode_HandleMovingFundsProposal_DispatchesAction(t *testing.T) { n, signer := setupNodeForHandlerTests(t) - n.handleMovingFundsProposal(signer.wallet, &MovingFundsProposal{}, 10, 100) + n.handleMovingFundsProposal(signer.wallet, &MovingFundsProposal{}, 10, 100, newTestPermit(participation.TBTCSigning)) waitForDispatcherIdle(t, n) @@ -783,7 +789,7 @@ func TestNode_HandleMovedFundsSweepProposal_WalletNotControlled(t *testing.T) { uncontrolledWallet := uncontrolledWalletFor(signer) proposal := &MovedFundsSweepProposal{} - n.handleMovedFundsSweepProposal(uncontrolledWallet, proposal, 10, 100) + n.handleMovedFundsSweepProposal(uncontrolledWallet, proposal, 10, 100, newTestPermit(participation.TBTCSigning)) if count := dispatchedActionsCount(n); count != 0 { t.Errorf("expected no dispatched actions for uncontrolled wallet, got %d", count) @@ -802,7 +808,7 @@ func TestNode_HandleMovedFundsSweepProposal_WalletBusy(t *testing.T) { n.walletDispatcher.actions[walletKey] = ActionMovedFundsSweep }() - n.handleMovedFundsSweepProposal(signer.wallet, &MovedFundsSweepProposal{}, 10, 100) + n.handleMovedFundsSweepProposal(signer.wallet, &MovedFundsSweepProposal{}, 10, 100, newTestPermit(participation.TBTCSigning)) actionType, ok := func() (WalletActionType, bool) { n.walletDispatcher.actionsMutex.Lock() @@ -830,6 +836,7 @@ func TestNode_HandleMovedFundsSweepProposal_DispatchesAction(t *testing.T) { &MovedFundsSweepProposal{SweepTxFee: big.NewInt(0)}, 10, 100, + newTestPermit(participation.TBTCSigning), ) waitForDispatcherIdle(t, n) @@ -856,176 +863,180 @@ func TestProcessCoordinationResult_NoopActionReturnsEarly(t *testing.T) { }, } - processCoordinationResult(n, result) + processCoordinationResult(context.Background(), n, result) if count := dispatchedActionsCount(n); count != 0 { t.Errorf("expected no dispatched actions for Noop result, got %d", count) } } -// TestProcessCoordinationResult_HeartbeatRoutesToHandler verifies that -// processCoordinationResult dispatches a heartbeat action when the proposal is -// a HeartbeatProposal and the wallet is controlled by this node. -func TestProcessCoordinationResult_HeartbeatRoutesToHandler(t *testing.T) { - n, signer := setupNodeForHandlerTests(t) - - result := &coordinationResult{ - wallet: signer.wallet, - window: newCoordinationWindow(100), - proposal: &HeartbeatProposal{ - Message: [16]byte{0x04}, - }, - } - - processCoordinationResult(n, result) - - waitForDispatcherIdle(t, n) - - // Dispatcher should be idle; a panicking handler would have made this fail. - if count := dispatchedActionsCount(n); count != 0 { - t.Errorf( - "expected dispatcher to be idle after heartbeat action, got %d active", - count, - ) +// routingTestProposals returns one well-formed proposal per dispatchable +// wallet action, keyed by the action type it must route to. +func routingTestProposals() map[WalletActionType]CoordinationProposal { + return map[WalletActionType]CoordinationProposal{ + ActionHeartbeat: &HeartbeatProposal{Message: [16]byte{0x04}}, + ActionDepositSweep: &DepositSweepProposal{}, + ActionRedemption: &RedemptionProposal{RedemptionTxFee: big.NewInt(0)}, + ActionMovingFunds: &MovingFundsProposal{}, + ActionMovedFundsSweep: &MovedFundsSweepProposal{SweepTxFee: big.NewInt(0)}, } } -// TestProcessCoordinationResult_DepositSweepRoutesToHandler verifies that -// processCoordinationResult attempts to dispatch a deposit sweep action when -// the proposal is a DepositSweepProposal. The wallet is pre-marked busy so -// dispatch returns errWalletBusy immediately, proving the routing path was -// exercised without running the action's execute() method. -func TestProcessCoordinationResult_DepositSweepRoutesToHandler(t *testing.T) { - n, signer := setupNodeForHandlerTests(t) - walletKey := walletKeyFor(t, signer) +// TestProcessCoordinationResult_RoutesToHandler verifies that every +// dispatchable proposal type reaches its handler and the wallet dispatcher +// under a real participation gate. Coordination results arrive before the +// window's end block, so processCoordinationResult must first wait for that +// block and only then acquire the permit anchored at it. The wallet is +// pre-marked busy so dispatch is rejected before the action's execute() method +// runs; the rejected-actions counter increment is positive proof the routed +// handler reached the dispatcher. +func TestProcessCoordinationResult_RoutesToHandler(t *testing.T) { + for action, proposal := range routingTestProposals() { + t.Run(action.String(), func(t *testing.T) { + n, signer, recorder := setupNodeForRoutingTests(t) + walletKey := markWalletBusy(t, n, signer) + + result := &coordinationResult{ + wallet: signer.wallet, + window: newCoordinationWindow(100), + proposal: proposal, + } - // Mark the wallet busy so dispatch is rejected before execute() runs. - func() { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - n.walletDispatcher.actions[walletKey] = ActionNoop - }() + processCoordinationResult(context.Background(), n, result) + + rejected := recorder.counter( + clientinfo.MetricWalletDispatcherRejectedTotal, + ) + if rejected != 1 { + t.Errorf( + "expected exactly one rejected dispatch proving the "+ + "handler was invoked, got %v", + rejected, + ) + } - result := &coordinationResult{ - wallet: signer.wallet, - window: newCoordinationWindow(100), - proposal: &DepositSweepProposal{}, + // The busy sentinel must be untouched: dispatch was attempted but + // returned errWalletBusy without modifying the map entry. + _, ok := func() (WalletActionType, bool) { + n.walletDispatcher.actionsMutex.Lock() + defer n.walletDispatcher.actionsMutex.Unlock() + v, exists := n.walletDispatcher.actions[walletKey] + return v, exists + }() + if !ok { + t.Error( + "expected walletDispatcher to retain the busy sentinel " + + "after routing", + ) + } + }) } +} - processCoordinationResult(n, result) +// TestProcessCoordinationResult_AtCutoverAnchorDispatches verifies the exact +// cutover boundary: a wallet action whose canonical anchor equals the cutover +// block resolves to the security-v2 mode and reaches the dispatcher for every +// dispatchable proposal type. +func TestProcessCoordinationResult_AtCutoverAnchorDispatches(t *testing.T) { + for action, proposal := range routingTestProposals() { + t.Run(action.String(), func(t *testing.T) { + n, signer, lc := setupNodeWithChain(t, 1*time.Millisecond) - // Busy sentinel must still be there: dispatch was attempted (routing worked) - // but returned errWalletBusy without touching the map entry. - _, ok := func() (WalletActionType, bool) { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - v, exists := n.walletDispatcher.actions[walletKey] - return v, exists - }() - if !ok { - t.Error("expected walletDispatcher to retain the busy sentinel after DepositSweep routing") - } -} + blockCounter, err := lc.BlockCounter() + if err != nil { + t.Fatal(err) + } -// TestProcessCoordinationResult_RedemptionRoutesToHandler verifies that -// processCoordinationResult dispatches a redemption action when the proposal is -// a RedemptionProposal and the wallet is controlled by this node. The wallet is -// pre-marked busy so dispatch returns errWalletBusy immediately, proving the -// routing path was exercised without running the action's execute() method. -func TestProcessCoordinationResult_RedemptionRoutesToHandler(t *testing.T) { - n, signer := setupNodeForHandlerTests(t) - walletKey := walletKeyFor(t, signer) + window := newCoordinationWindow(100) + // The permit anchor is the window's end block; make it the exact + // cutover block so the anchor sits right at C. + n.participationGate = newTestGateWithCutover( + t, + blockCounter, + window.endBlock(), + ) - // Mark the wallet busy so dispatch is rejected before execute() runs. - func() { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - n.walletDispatcher.actions[walletKey] = ActionNoop - }() + recorder := newDispatcherMetricsRecorder() + n.walletDispatcher.setMetricsRecorder(recorder) - result := &coordinationResult{ - wallet: signer.wallet, - window: newCoordinationWindow(100), - proposal: &RedemptionProposal{RedemptionTxFee: big.NewInt(0)}, - } + markWalletBusy(t, n, signer) - processCoordinationResult(n, result) + result := &coordinationResult{ + wallet: signer.wallet, + window: window, + proposal: proposal, + } - // Busy sentinel must still be there: dispatch was attempted (routing worked) - // but returned errWalletBusy without touching the map entry. - _, ok := func() (WalletActionType, bool) { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - v, exists := n.walletDispatcher.actions[walletKey] - return v, exists - }() - if !ok { - t.Error("expected walletDispatcher to retain the busy sentinel after Redemption routing") + processCoordinationResult(context.Background(), n, result) + + rejected := recorder.counter( + clientinfo.MetricWalletDispatcherRejectedTotal, + ) + if rejected != 1 { + t.Errorf( + "expected the anchor at the exact cutover block to "+ + "dispatch, got %v rejected-dispatch increments", + rejected, + ) + } + }) } } -// TestProcessCoordinationResult_MovingFundsRoutesToHandler verifies that -// processCoordinationResult dispatches a moving funds action when the proposal -// is a MovingFundsProposal and the wallet is controlled by this node. -func TestProcessCoordinationResult_MovingFundsRoutesToHandler(t *testing.T) { - n, signer := setupNodeForHandlerTests(t) - walletKey := walletKeyFor(t, signer) - - func() { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - n.walletDispatcher.actions[walletKey] = ActionNoop - }() - - result := &coordinationResult{ - wallet: signer.wallet, - window: newCoordinationWindow(100), - proposal: &MovingFundsProposal{}, - } +// TestProcessCoordinationResult_BeforeCutoverAnchorDispatches verifies the +// other side of the cutover boundary: a wallet action whose canonical anchor +// is one block below the cutover block resolves to the legacy mode and reaches +// the dispatcher for every dispatchable proposal type. +func TestProcessCoordinationResult_BeforeCutoverAnchorDispatches(t *testing.T) { + for action, proposal := range routingTestProposals() { + t.Run(action.String(), func(t *testing.T) { + n, signer, lc := setupNodeWithChain(t, 1*time.Millisecond) - processCoordinationResult(n, result) + blockCounter, err := lc.BlockCounter() + if err != nil { + t.Fatal(err) + } - _, ok := func() (WalletActionType, bool) { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - v, exists := n.walletDispatcher.actions[walletKey] - return v, exists - }() - if !ok { - t.Error("expected walletDispatcher to retain the busy sentinel after MovingFunds routing") - } -} + window := newCoordinationWindow(100) + // The permit anchor is the window's end block; put the cutover one + // block above it so the anchor sits at C-1 and pins legacy mode. + n.participationGate = newTestGateWithCutover( + t, + blockCounter, + window.endBlock()+1, + ) -// TestProcessCoordinationResult_MovedFundsSweepRoutesToHandler verifies that -// processCoordinationResult dispatches a moved funds sweep action when the -// proposal is a MovedFundsSweepProposal and the wallet is controlled by this -// node. -func TestProcessCoordinationResult_MovedFundsSweepRoutesToHandler(t *testing.T) { - n, signer := setupNodeForHandlerTests(t) - walletKey := walletKeyFor(t, signer) + recorder := newDispatcherMetricsRecorder() + n.walletDispatcher.setMetricsRecorder(recorder) - func() { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - n.walletDispatcher.actions[walletKey] = ActionNoop - }() + walletKey := markWalletBusy(t, n, signer) - result := &coordinationResult{ - wallet: signer.wallet, - window: newCoordinationWindow(100), - proposal: &MovedFundsSweepProposal{SweepTxFee: big.NewInt(0)}, - } + result := &coordinationResult{ + wallet: signer.wallet, + window: window, + proposal: proposal, + } - processCoordinationResult(n, result) + processCoordinationResult(context.Background(), n, result) + + rejected := recorder.counter( + clientinfo.MetricWalletDispatcherRejectedTotal, + ) + if rejected != 1 { + t.Errorf( + "expected the pre-cutover legacy anchor to dispatch, "+ + "got %v rejected-dispatch increments", + rejected, + ) + } - _, ok := func() (WalletActionType, bool) { - n.walletDispatcher.actionsMutex.Lock() - defer n.walletDispatcher.actionsMutex.Unlock() - v, exists := n.walletDispatcher.actions[walletKey] - return v, exists - }() - if !ok { - t.Error("expected walletDispatcher to retain the busy sentinel after MovedFundsSweep routing") + n.walletDispatcher.actionsMutex.Lock() + _, ok := n.walletDispatcher.actions[walletKey] + n.walletDispatcher.actionsMutex.Unlock() + if !ok { + t.Error("expected the busy sentinel to remain after dispatch") + } + }) } } @@ -1074,6 +1085,110 @@ func setupNodeForClosureTests(t *testing.T) (*node, *signer, *localChain) { return n, signer, lc } +// newTestCutoverRoster builds a node-local cutover peer roster backed by the +// given chain's block counter and a no-op metrics sink, for wiring tests. +func newTestCutoverRoster( + t *testing.T, + ctx context.Context, + lc *localChain, +) *participation.CutoverPeerRoster { + t.Helper() + + blockCounter, err := lc.BlockCounter() + if err != nil { + t.Fatal(err) + } + roster, err := participation.NewCutoverPeerRoster( + ctx, + blockCounter, + 1500, + &clientinfo.NoOpPerformanceMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + return roster +} + +// TestNode_SetCutoverPeerRoster_PropagatesToExistingSigningExecutor verifies +// that installing the cutover peer roster reaches a signing executor that was +// already created before the roster was installed. This guards the +// initialization-ordering window in which a coordination round could create a +// signing executor before the roster is wired: without propagation such an +// executor would silently never record legacy-peer sightings. +func TestNode_SetCutoverPeerRoster_PropagatesToExistingSigningExecutor(t *testing.T) { + n, signer, lc := setupNodeForClosureTests(t) + + // Create the signing executor BEFORE the roster is installed, simulating an + // early coordination round that produced a signing executor. + executor, ok, err := n.getSigningExecutor(signer.wallet.publicKey) + if err != nil { + t.Fatal(err) + } + if !ok { + t.Fatal("node is supposed to control wallet signers") + } + if executor.cutoverPeerRoster != nil { + t.Fatal("signing executor unexpectedly carries a roster before install") + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + roster := newTestCutoverRoster(t, ctx, lc) + defer roster.Close() + + n.setCutoverPeerRoster(roster) + + if executor.cutoverPeerRoster != roster { + t.Error("pre-existing signing executor did not receive the roster") + } + if n.dkgExecutor.cutoverPeerRoster != roster { + t.Error("DKG executor did not receive the roster") + } +} + +// TestNode_SetCutoverPeerRoster_ConcurrentInstall exercises concurrent roster +// installation and signing-executor creation under -race. Both paths take +// signingExecutorsMutex, so the field write and the executor cache read/write +// are serialized; regardless of ordering the resulting executor must carry the +// roster (created after install reads it from the node; created before install +// is reached by the propagation loop). +func TestNode_SetCutoverPeerRoster_ConcurrentInstall(t *testing.T) { + n, signer, lc := setupNodeForClosureTests(t) + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + roster := newTestCutoverRoster(t, ctx, lc) + defer roster.Close() + + var wg sync.WaitGroup + wg.Add(2) + go func() { + defer wg.Done() + n.setCutoverPeerRoster(roster) + }() + go func() { + defer wg.Done() + _, _, _ = n.getSigningExecutor(signer.wallet.publicKey) + }() + wg.Wait() + + executor, ok, err := n.getSigningExecutor(signer.wallet.publicKey) + if err != nil { + t.Fatal(err) + } + if !ok { + t.Fatal("node is supposed to control wallet signers") + } + if executor.cutoverPeerRoster != roster { + t.Error( + "signing executor did not receive the roster after concurrent install", + ) + } +} + // TestArchiveClosedWallets_ArchivesClosedWallet verifies that a wallet whose // on-chain state is StateClosed is removed from the node's registry. func TestArchiveClosedWallets_ArchivesClosedWallet(t *testing.T) { @@ -1211,7 +1326,10 @@ func TestHandleWalletClosure_ReturnsErrorWhenNotConfirmed(t *testing.T) { // setupNodeWithChain creates a fully-initialised node and returns the node, // the signer, and the underlying *localChain so callers can manipulate chain // state (e.g. close/terminate a wallet) after creation. -func setupNodeWithChain(t *testing.T) (*node, *signer, *localChain) { +func setupNodeWithChain( + t *testing.T, + blockTime ...time.Duration, +) (*node, *signer, *localChain) { t.Helper() groupParameters := &GroupParameters{ @@ -1220,7 +1338,7 @@ func setupNodeWithChain(t *testing.T) (*node, *signer, *localChain) { HonestThreshold: 3, } - lc := Connect() + lc := Connect(blockTime...) localProvider := local.Connect() signer := createMockSigner(t) @@ -1260,6 +1378,73 @@ func setupNodeForHandlerTests(t *testing.T) (*node, *signer) { return n, signer } +// dispatcherMetricsRecorder counts walletDispatcher counter increments so +// tests can positively observe that a dispatch was attempted. +type dispatcherMetricsRecorder struct { + mu sync.Mutex + counters map[string]float64 +} + +func newDispatcherMetricsRecorder() *dispatcherMetricsRecorder { + return &dispatcherMetricsRecorder{counters: make(map[string]float64)} +} + +func (r *dispatcherMetricsRecorder) IncrementCounter(name string, value float64) { + r.mu.Lock() + defer r.mu.Unlock() + r.counters[name] += value +} + +func (r *dispatcherMetricsRecorder) SetGauge(string, float64) {} + +func (r *dispatcherMetricsRecorder) RecordDuration(string, time.Duration) {} + +func (r *dispatcherMetricsRecorder) counter(name string) float64 { + r.mu.Lock() + defer r.mu.Unlock() + return r.counters[name] +} + +// setupNodeForRoutingTests builds a node on a fast-block chain with a real +// participation gate whose cutover is already crossed, so that +// processCoordinationResult can wait for the coordination window's end block +// and acquire a security-v2 permit against it. The returned recorder counts +// walletDispatcher metrics and proves dispatch attempts. +func setupNodeForRoutingTests( + t *testing.T, +) (*node, *signer, *dispatcherMetricsRecorder) { + t.Helper() + + n, signer, lc := setupNodeWithChain(t, 1*time.Millisecond) + + blockCounter, err := lc.BlockCounter() + if err != nil { + t.Fatal(err) + } + n.participationGate = newTestGate(t, blockCounter) + + recorder := newDispatcherMetricsRecorder() + n.walletDispatcher.setMetricsRecorder(recorder) + + return n, signer, recorder +} + +// markWalletBusy plants a busy sentinel for the signer's wallet in the +// dispatcher so a routed action is rejected with errWalletBusy before its +// execute() method runs; the rejection is observable through the dispatcher +// rejected-actions counter. +func markWalletBusy(t *testing.T, n *node, s *signer) string { + t.Helper() + + walletKey := walletKeyFor(t, s) + + n.walletDispatcher.actionsMutex.Lock() + defer n.walletDispatcher.actionsMutex.Unlock() + n.walletDispatcher.actions[walletKey] = ActionNoop + + return walletKey +} + // uncontrolledWalletFor returns a wallet whose public key is NOT registered in // the given signer's keystore -- constructed by doubling the signer's key. func uncontrolledWalletFor(s *signer) wallet { diff --git a/pkg/tbtc/participation.go b/pkg/tbtc/participation.go new file mode 100644 index 0000000000..5787ddc8ca --- /dev/null +++ b/pkg/tbtc/participation.go @@ -0,0 +1,65 @@ +package tbtc + +import ( + "fmt" + "math" +) + +// MaximumLegacyCompletionBlocks returns the maximum number of Ethereum blocks +// that any already-started tBTC protocol work may legitimately need to reach +// its natural completion: the largest of the DKG and signing retry-loop +// bounds, the coordination window, and every wallet-action proposal validity. +// +// The bound sizes cutover-rehearsal timing, local straggler-roster retention, +// graceful rollback quiescence, and alerts about unexpectedly long legacy +// overlap after the protocol cutover block. It is deliberately not an +// activation height and must never gate new work; each protocol's existing +// validity context remains the hard end of any in-flight grace behavior. +func MaximumLegacyCompletionBlocks() uint64 { + bounds := []uint64{ + uint64(dkgAttemptsLimit) * uint64(dkgAttemptMaximumBlocks()), + uint64(signingAttemptsLimit) * uint64(signingAttemptMaximumBlocks()), + coordinationDurationBlocks, + heartbeatTotalProposalValidityBlocks, + depositSweepProposalValidityBlocks, + redemptionProposalValidityBlocks, + movingFundsProposalValidityBlocks, + movedFundsSweepProposalValidityBlocks, + } + + maximum := uint64(0) + for _, bound := range bounds { + if bound > maximum { + maximum = bound + } + } + return maximum +} + +// cutoverPeerRosterRetentionMarginBlocks is the reviewed margin added to the +// maximum legacy completion bound when deriving the cutover peer roster +// retention: it covers RPC and processing skew beyond the longest in-flight +// work bound. +const cutoverPeerRosterRetentionMarginBlocks = uint64(300) + +// CutoverPeerRosterRetentionBlocks derives how long a legacy peer sighting is +// retained without a fresh observation before it is evicted as "not recently +// observed": the maximum number of blocks any already-started tBTC work may +// legitimately still be running, plus the reviewed margin. The roster records +// sightings from the tBTC DKG and signing announcers, so the tBTC completion +// bound governs the retention. Deriving from the completion bound keeps +// retention in lockstep with the protocol validity windows; the addition is +// overflow-checked because the retention feeds the roster's gauge projection +// and eviction arithmetic. +func CutoverPeerRosterRetentionBlocks() (uint64, error) { + bound := MaximumLegacyCompletionBlocks() + if bound > math.MaxUint64-cutoverPeerRosterRetentionMarginBlocks { + return 0, fmt.Errorf( + "cutover peer roster retention overflows: completion bound [%d] "+ + "plus margin [%d]", + bound, + cutoverPeerRosterRetentionMarginBlocks, + ) + } + return bound + cutoverPeerRosterRetentionMarginBlocks, nil +} diff --git a/pkg/tbtc/participation_gate_test.go b/pkg/tbtc/participation_gate_test.go new file mode 100644 index 0000000000..7aa7ca5f52 --- /dev/null +++ b/pkg/tbtc/participation_gate_test.go @@ -0,0 +1,3756 @@ +package tbtc + +import ( + "context" + "crypto/ecdsa" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "math" + "math/big" + "reflect" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/keep-network/keep-common/pkg/persistence" + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/tecdsa" + "github.com/keep-network/keep-core/pkg/tecdsa/dkg" +) + +func TestSigningExecutor_Sign_RefusesUnsetMode(t *testing.T) { + executor := &signingExecutor{} + + _, err := executor.sign( + nil, + big.NewInt(100), + 0, + participation.ProtocolMode(0), + ) + if err == nil { + t.Fatal("expected an unset-mode refusal error") + } + if !strings.Contains(err.Error(), "cannot select compatibility strategies") { + t.Errorf("unexpected refusal error: [%v]", err) + } +} + +// TestNode_BeginWalletActionPermit exercises the wallet action permit +// acquisition: the heartbeat maps to its own ceremony class, other actions +// are signing ceremonies, quiescence refuses, and a legacy-mode permit remains +// available while the chain is below the cutover block. +func TestNode_BeginWalletActionPermit(t *testing.T) { + localChain := Connect() + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + t.Run("heartbeat ceremony class", func(t *testing.T) { + n := &node{participationGate: newTestGate(t, blockCounter)} + + permit := n.beginWalletActionPermit(ActionHeartbeat, 1) + if permit == nil { + t.Fatal("expected a permit") + } + defer permit.Close() + + testutils.AssertStringsEqual( + t, + "permit ceremony", + string(participation.TBTCHeartbeat), + string(permit.Ceremony()), + ) + testutils.AssertStringsEqual( + t, + "permit mode", + participation.ModeSecurityV2.String(), + permit.Mode().String(), + ) + }) + + t.Run("signing ceremony class", func(t *testing.T) { + n := &node{participationGate: newTestGate(t, blockCounter)} + + permit := n.beginWalletActionPermit(ActionDepositSweep, 1) + if permit == nil { + t.Fatal("expected a permit") + } + defer permit.Close() + + testutils.AssertStringsEqual( + t, + "permit ceremony", + string(participation.TBTCSigning), + string(permit.Ceremony()), + ) + }) + + t.Run("refused while quiescing", func(t *testing.T) { + gate := newTestGate(t, blockCounter) + gate.Quiesce(fmt.Errorf("shutdown")) + + n := &node{participationGate: gate} + + if permit := n.beginWalletActionPermit(ActionRedemption, 1); permit != nil { + permit.Close() + t.Error("expected no permit while quiescing") + } + }) + + t.Run("refused without a gate", func(t *testing.T) { + n := &node{} + + if permit := n.beginWalletActionPermit(ActionRedemption, 1); permit != nil { + permit.Close() + t.Error("expected no permit without a gate") + } + }) + + t.Run("legacy mode admitted", func(t *testing.T) { + // A cutover block far ahead pins every current anchor to the legacy + // mode. + gate, err := participation.NewGate( + t.Context(), + participation.Schedule{CutoverBlock: 1_000_000}, + blockCounter, + testGateMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + n := &node{participationGate: gate} + + permit := n.beginWalletActionPermit(ActionMovingFunds, 1) + if permit == nil { + t.Fatal("expected a permit for the legacy mode") + } + testutils.AssertStringsEqual( + t, + "legacy permit mode", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + + snapshot := gate.State() + testutils.AssertUintsEqual( + t, + "active ceremonies while the legacy permit is held", + 1, + snapshot.ActiveCeremonies, + ) + permit.Close() + }) +} + +// TestDkgExecutor_PreserveInterruptedSigner_Quarantines proves a refused +// activation of a signer whose wallet is not registered on chain preserves +// the share only in the protected quarantine namespace — never in the active +// wallet storage and never in the in-memory wallet cache. +func TestDkgExecutor_PreserveInterruptedSigner_Quarantines(t *testing.T) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + + de.preserveInterruptedSigner( + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + if len(registryHandle.saved) != 0 { + t.Errorf( + "expected no active-namespace save, got [%d]", + len(registryHandle.saved), + ) + } + if signers := de.walletRegistry.getSigners( + result.PrivateKeyShare.PublicKey(), + ); len(signers) != 0 { + t.Errorf("expected no activated signers, got [%d]", len(signers)) + } + + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + + var metadataContent []byte + expectedDirectory := getWalletStorageKey(result.PrivateKeyShare.PublicKey()) + for _, descriptor := range quarantineHandle.saved { + testutils.AssertStringsEqual( + t, + "quarantine directory", + expectedDirectory, + descriptor.Directory(), + ) + if strings.HasPrefix(descriptor.Name(), "/metadata_") { + metadataContent, _ = descriptor.Content() + } + } + if metadataContent == nil { + t.Fatal("expected a quarantine metadata record") + } + + var metadata QuarantinedSignerMetadata + if err := json.Unmarshal(metadataContent, &metadata); err != nil { + t.Fatal(err) + } + + testutils.AssertUintsEqual( + t, + "metadata schema version", + uint64(QuarantineSchemaVersion), + uint64(metadata.SchemaVersion), + ) + testutils.AssertStringsEqual( + t, + "metadata release epoch", + participation.CompiledEpoch.String(), + metadata.ReleaseEpoch, + ) + testutils.AssertStringsEqual( + t, + "metadata protocol mode", + participation.ModeSecurityV2.String(), + metadata.ProtocolMode, + ) + testutils.AssertStringsEqual( + t, + "metadata ceremony", + string(participation.TBTCDKG), + metadata.Ceremony, + ) + testutils.AssertStringsEqual( + t, + "metadata failed operation", + "tbtc_dkg_signer_activation", + metadata.FailedOperation, + ) + if metadata.SeedHash == "" { + t.Error("expected a seed hash in the quarantine metadata") + } + if strings.Contains(metadata.SeedHash, big.NewInt(1).Text(16)) && + len(metadata.SeedHash) < 64 { + t.Error("the raw seed must not appear in the quarantine metadata") + } + expectedWalletPKH := bitcoin.PublicKeyHash(result.PrivateKeyShare.PublicKey()) + testutils.AssertStringsEqual( + t, + "metadata wallet public key hash", + hex.EncodeToString(expectedWalletPKH[:]), + metadata.WalletPublicKeyHash, + ) +} + +// TestDkgExecutor_PreserveInterruptedSigner_SavesRegisteredWithoutActivation +// proves a refused activation of a signer whose wallet is already registered +// on chain saves the share durably in the active namespace — a prior binary +// may legitimately load it — but never activates it in this process's wallet +// cache. +func TestDkgExecutor_PreserveInterruptedSigner_SavesRegisteredWithoutActivation( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + walletPublicKey := result.PrivateKeyShare.PublicKey() + walletID, err := de.chain.CalculateWalletID(walletPublicKey) + if err != nil { + t.Fatal(err) + } + de.chain.(*localChain).setWallet( + bitcoin.PublicKeyHash(walletPublicKey), + &WalletChainData{EcdsaWalletID: walletID, State: StateLive}, + ) + + permit := newTestPermit(participation.TBTCDKG) + + de.preserveInterruptedSigner( + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 1, + len(registryHandle.saved), + ) + if len(quarantineHandle.saved) != 0 { + t.Errorf( + "expected no quarantine records, got [%d]", + len(quarantineHandle.saved), + ) + } + if signers := de.walletRegistry.getSigners(walletPublicKey); len(signers) != 0 { + t.Errorf("expected no activated signers, got [%d]", len(signers)) + } +} + +// setupPreserveScenario builds a dkgExecutor with observable active and +// quarantine persistence plus a completed DKG result, for exercising the +// interrupted-signer preservation paths. +func setupPreserveScenario(t *testing.T) ( + *dkgExecutor, + *dkg.Result, + *GroupSelectionResult, + *mockPersistenceHandle, + *mockPersistenceHandle, +) { + t.Helper() + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 3, + HonestThreshold: 2, + } + + localChain := Connect() + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + registryHandle := &mockPersistenceHandle{} + walletRegistry, err := newWalletRegistry( + registryHandle, + localChain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + quarantineHandle := &mockPersistenceHandle{} + + de := &dkgExecutor{ + groupParameters: groupParameters, + chain: localChain, + walletRegistry: walletRegistry, + participationGate: newTestGate(t, blockCounter), + signerQuarantine: newSignerQuarantine( + context.Background(), + logger, + quarantineHandle, + ), + } + + result := &dkg.Result{ + Group: group.NewGroup( + groupParameters.DishonestThreshold(), + groupParameters.GroupSize, + ), + PrivateKeyShare: tecdsa.NewPrivateKeyShare(testData[0]), + } + + gsr := &GroupSelectionResult{ + OperatorsIDs: chain.OperatorIDs{1, 2, 3, 4, 5}, + OperatorsAddresses: chain.Addresses{"0xAA", "0xBB", "0xCC", "0xDD", "0xEE"}, + } + + return de, result, gsr, registryHandle, quarantineHandle +} + +// preserveOneSigner quarantines the signer the preserve scenario generates for +// the given seat and returns the wallet directory it was written under. +func preserveOneSigner( + t *testing.T, + de *dkgExecutor, + result *dkg.Result, + gsr *GroupSelectionResult, + memberIndex group.MemberIndex, +) string { + t.Helper() + + de.preserveInterruptedSigner( + logger.With(), + newTestPermit(participation.TBTCDKG), + big.NewInt(1), + result, + memberIndex, + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + return getWalletStorageKey(result.PrivateKeyShare.PublicKey()) +} + +// TestDkgExecutor_ReportQuarantinedSigners_CountsOutputsNotRecords proves the +// reported count is of preserved signer outputs, not of the records preservation +// writes: each output is stored as a membership beside its audit metadata, and +// counting both would report every quarantined share twice. +func TestDkgExecutor_ReportQuarantinedSigners_CountsOutputsNotRecords(t *testing.T) { + de, result, gsr, _, quarantineHandle := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "records written for one preserved output", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "reported quarantined signers", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportQuarantinedSigners_ExcludesActivatedSeat proves an +// output whose seat this process has activated stops being counted. The metric +// is of preserved material a rollback still has to account for, and a share the +// running node holds active is accounted for by the wallet cache itself. +func TestDkgExecutor_ReportQuarantinedSigners_ExcludesActivatedSeat(t *testing.T) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + walletStorageKey := preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers before activation", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) + + // The same wallet and the same seat, now activated from the active + // namespace — the state a restart that adopted the share would be in. + signer, err := de.buildFinalSigner( + result, + group.MemberIndex(1), + gsr.OperatorsAddresses, + ) + if err != nil { + t.Fatal(err) + } + if err := de.walletRegistry.registerSigner(signer); err != nil { + t.Fatal(err) + } + testutils.AssertBoolsEqual( + t, + "the preserved seat reads as active", + true, + de.walletRegistry.isSignerActive( + walletStorageKey, + group.MemberIndex(1), + ), + ) + + de.reportQuarantinedSigners(logger.With()) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after activation", + 0, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportQuarantinedSigners_KeepsCountOnUnreadableNamespace +// proves a namespace that cannot be enumerated leaves the last published count +// standing. Publishing zero there would report an empty quarantine, which is +// precisely what could not be established, and it is the one answer that reads +// as nothing left to account for. +func TestDkgExecutor_ReportQuarantinedSigners_KeepsCountOnUnreadableNamespace( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers before the namespace fails", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) + + de.signerQuarantine = newSignerQuarantine( + context.Background(), + logger, + &unreadableHandle{}, + ) + + de.reportQuarantinedSigners(logger.With()) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after the namespace fails", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportQuarantinedSigners_AddsANewOutputToWhatTheLastScanFound +// proves a recount that fails after a new share was preserved reports the +// inherited outputs and the new one together. +// +// The material this process wrote and the material it inherited are known from +// different places: the first from its own confirmed writes, the second only +// from a scan. A node holding both that falls back to its own writes alone +// reports fewer preserved outputs than the namespace held before it started — +// and the more an earlier process left behind, the further under the truth the +// number lands. +func TestDkgExecutor_ReportQuarantinedSigners_AddsANewOutputToWhatTheLastScanFound( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + // Two outputs an earlier process on this host preserved, in a namespace + // that serves the startup scan and nothing after it. + handle := &scanBudgetHandle{readableScans: 1} + for _, name := range []string{"/membership_1", "/membership_2"} { + if err := handle.Save([]byte("x"), "inherited-wallet", name); err != nil { + t.Fatal(err) + } + } + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + + if err := de.reportInitialQuarantinedSigners(); err != nil { + t.Fatal(err) + } + testutils.AssertIntsEqual( + t, + "quarantined signers this process inherited", + 2, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) + + // A third, distinct output, preserved once the namespace can no longer be + // enumerated: the write is confirmed, every recount around it fails. + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after preserving a third output", + 3, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportQuarantinedSigners_CountsARepreservedSeatOnce proves +// preserving the same output twice does not raise the count twice. +// +// The count is of preserved shares, not of preservations. One seat of one +// wallet is one share however many times it is written — a retry, a second +// interruption of the same ceremony — and a count that adds a share the +// namespace holds one copy of describes a namespace this host does not have. +func TestDkgExecutor_ReportQuarantinedSigners_CountsARepreservedSeatOnce( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + handle := &scanBudgetHandle{readableScans: 1} + for _, name := range []string{"/membership_1", "/membership_2"} { + if err := handle.Save([]byte("x"), "inherited-wallet", name); err != nil { + t.Fatal(err) + } + } + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + + if err := de.reportInitialQuarantinedSigners(); err != nil { + t.Fatal(err) + } + + // The same wallet and the same seat, preserved twice while the namespace + // cannot be read back to settle what it holds. + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after re-preserving one seat", + 3, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportQuarantinedSigners_DoesNotResurrectAClearedOutput +// proves an output a successful scan found gone is not brought back by the next +// scan that fails. +// +// What this process wrote is only a floor until the namespace can be asked +// about it. Once a scan does succeed, it has enumerated the namespace after +// every one of those writes, so a write it does not find is a record an +// operator cleared or a seat that was activated. Carrying that identity forward +// would let the next unreadable namespace raise the count back over an output +// nobody holds — a rollback sent looking for material that is not there, from a +// node that is otherwise reporting correctly. +func TestDkgExecutor_ReportQuarantinedSigners_DoesNotResurrectAClearedOutput( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + // Two enumerations: the one preservation takes when it is done, and the one + // that follows the operator's repair. Everything after that fails. + handle := &scanBudgetHandle{readableScans: 2} + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + + walletStorageKey := preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after preserving one output", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) + + // The operator clears the preserved record, having accounted for the share + // by hand, and the recount that follows sees the namespace without it. + if err := handle.Delete(walletStorageKey, "/membership_1"); err != nil { + t.Fatal(err) + } + + de.reportQuarantinedSigners(logger.With()) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after the record was cleared", + 0, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) + + // The namespace stops being readable. What this process wrote is no longer + // its own to vouch for: a scan already established the namespace does not + // hold it. + de.reportQuarantinedSigners(logger.With()) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers once the namespace stops answering", + 0, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// scanBudgetHandle is a protected namespace that serves a fixed number of +// enumerations and fails every one after that, as a namespace does when its +// directory stops being readable while the process runs. Its writes always +// succeed: what it models is a node that can still preserve material it can no +// longer count. +type scanBudgetHandle struct { + mockPersistenceHandle + + readableScans int + + mu sync.Mutex + scans int +} + +func (h *scanBudgetHandle) Save( + data []byte, + directory string, + name string, +) error { + h.mu.Lock() + defer h.mu.Unlock() + + return h.mockPersistenceHandle.Save(data, directory, name) +} + +func (h *scanBudgetHandle) Delete(directory string, name string) error { + h.mu.Lock() + defer h.mu.Unlock() + + return h.mockPersistenceHandle.Delete(directory, name) +} + +func (h *scanBudgetHandle) ReadAll() ( + <-chan persistence.DataDescriptor, + <-chan error, +) { + h.mu.Lock() + h.scans++ + readable := h.scans <= h.readableScans + saved := append([]persistence.DataDescriptor(nil), h.saved...) + h.mu.Unlock() + + descriptors := make(chan persistence.DataDescriptor, len(saved)) + errs := make(chan error, 1) + + defer close(descriptors) + defer close(errs) + + if !readable { + errs <- fmt.Errorf("the quarantine directory is unreadable") + return descriptors, errs + } + + for _, descriptor := range saved { + descriptors <- descriptor + } + + return descriptors, errs +} + +// TestDkgExecutor_ReportInitialQuarantinedSigners_BlocksOnUnreadableNamespace +// proves a process that cannot enumerate its quarantine namespace at startup +// reports the failure to its caller and publishes no count at all. +// +// Keeping the last published count is the right answer at runtime, where a +// number somebody published exists. On a cold start there is none: the gauge is +// registered at zero with the rest of the fixed family, so a scan that gives up +// quietly leaves that zero as the process's only word on the subject, and a +// fleet reading it concludes there is nothing left to account for. This is the +// one direction the count must never invent, so it is watched from a recorder +// that can tell a published zero from an unpublished one. +func TestDkgExecutor_ReportInitialQuarantinedSigners_BlocksOnUnreadableNamespace( + t *testing.T, +) { + de, _, _, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + de.signerQuarantine = newSignerQuarantine( + context.Background(), + logger, + &unreadableHandle{}, + ) + + err := de.reportInitialQuarantinedSigners() + if err == nil { + t.Fatal( + "an unreadable quarantine namespace must stop a cold start, not " + + "leave the registered zero standing as the count", + ) + } + if !strings.Contains(err.Error(), "unreadable") { + t.Errorf("expected the underlying read error, got [%v]", err) + } + + if value, published := recorder.gaugePublished( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + ); published { + t.Errorf( + "a count that could not be established was published as [%v]", + value, + ) + } +} + +// TestDkgExecutor_ReportInitialQuarantinedSigners_PublishesWhatIsPreserved +// proves a readable namespace is counted and published at startup, so a +// restart inherits the outputs earlier processes on this host preserved rather +// than starting the count over. +func TestDkgExecutor_ReportInitialQuarantinedSigners_PublishesWhatIsPreserved( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + preserveOneSigner(t, de, result, gsr, group.MemberIndex(2)) + + // A restart's first scan: the same namespace, a recorder that has + // published nothing yet. + restarted := newDispatchGaugeRecorder() + de.metricsRecorder = restarted + + if err := de.reportInitialQuarantinedSigners(); err != nil { + t.Fatal(err) + } + + value, published := restarted.gaugePublished( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + ) + if !published { + t.Fatal("expected the inherited outputs to be counted at startup") + } + testutils.AssertIntsEqual( + t, + "quarantined signers a restart inherits", + 2, + int(value), + ) +} + +// TestDkgExecutor_ReportQuarantinedSigners_SerializesRecountAndPublication +// proves no two reporters recount the namespace at the same time, and that +// concurrent reporters agree on what they publish. +// +// Members of one ceremony quarantine independently, so reports can be raised +// concurrently. Interleaved scans do not corrupt anything — every piece of state +// they touch is individually guarded — but they can publish out of order and +// leave an earlier, smaller scan's count as the last word, which is the +// direction that reads as an all-clear. Serializing the scan with its +// publication is what rules that out, so the enumeration itself is what this +// watches: a scan that begins while another is open is the interleaving. +func TestDkgExecutor_ReportQuarantinedSigners_SerializesRecountAndPublication( + t *testing.T, +) { + de, result, gsr, _, quarantineHandle := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + preserveOneSigner(t, de, result, gsr, group.MemberIndex(2)) + + // The same records, behind a namespace that holds every enumeration open + // until released and reports how many were ever open at once. + tracking := &scanTrackingHandle{ + mockPersistenceHandle: *quarantineHandle, + entered: make(chan struct{}, reporterCount), + release: make(chan struct{}), + } + de.signerQuarantine = newSignerQuarantine(context.Background(), logger, tracking) + + var wg sync.WaitGroup + for range reporterCount { + wg.Add(1) + go func() { + defer wg.Done() + de.reportQuarantinedSigners(logger.With()) + }() + } + + // One reporter is inside its enumeration; the wait then gives every other + // reporter time to reach its own before any of them can finish. Reporters + // that recount one at a time cannot join it there however long they are + // given, so this bound decides how much evidence of interleaving the test + // collects, never whether a serialized implementation passes. + <-tracking.entered + time.Sleep(100 * time.Millisecond) + close(tracking.release) + wg.Wait() + + testutils.AssertIntsEqual( + t, + "greatest number of enumerations open at once", + 1, + int(tracking.peak.Load()), + ) + testutils.AssertIntsEqual( + t, + "reported quarantined signers after concurrent reports", + 2, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// reporterCount is how many concurrent reporters the serialization test raises. +const reporterCount = 4 + +// scanTrackingHandle is a protected namespace that holds every enumeration open +// until released and records the greatest number that were open at once. +type scanTrackingHandle struct { + mockPersistenceHandle + + entered chan struct{} + release chan struct{} + inFlight atomic.Int32 + peak atomic.Int32 +} + +func (h *scanTrackingHandle) ReadAll() ( + <-chan persistence.DataDescriptor, + <-chan error, +) { + descriptors := make(chan persistence.DataDescriptor) + errs := make(chan error) + + open := h.inFlight.Add(1) + for { + peak := h.peak.Load() + if open <= peak || h.peak.CompareAndSwap(peak, open) { + break + } + } + + select { + case h.entered <- struct{}{}: + default: + } + + go func() { + defer close(descriptors) + defer close(errs) + defer h.inFlight.Add(-1) + + <-h.release + + for _, descriptor := range h.saved { + descriptors <- descriptor + } + }() + + return descriptors, errs +} + +// TestSignerQuarantine_PreservedOutputs_CountsMembershipsOnly proves only the +// records carrying key material are read as signer outputs — the membership of +// a pair and the combined handoff — and that a seat named by both is one +// output. The namespace also holds the audit metadata, and a later schema or an +// operator may leave other names there; none of them is a preserved share. +func TestSignerQuarantine_PreservedOutputs_CountsMembershipsOnly(t *testing.T) { + handle := &mockPersistenceHandle{} + quarantine := newSignerQuarantine(context.Background(), logger, handle) + + for _, name := range []string{ + "/membership_1", + "/metadata_1", + // The same seat, written again as one record when the namespace would + // not complete the pair. One share, however many times it was offered. + "/handoff_1", + "/membership_17", + "/metadata_17", + // A seat the namespace only ever took whole. + "/handoff_23", + // Not signer outputs: seat zero is no seat, an unparsable seat names + // none, and neither a later schema's record nor a stray file is a share. + "/membership_0", + "/membership_two", + "/handoff_0", + "/attestation_1", + "/notes.txt", + } { + if err := handle.Save([]byte("x"), "wallet", name); err != nil { + t.Fatal(err) + } + } + + outputs, err := quarantine.preservedOutputs() + if err != nil { + t.Fatal(err) + } + + testutils.AssertIntsEqual(t, "preserved outputs", 3, len(outputs)) + + seats := make(map[group.MemberIndex]string) + for _, output := range outputs { + seats[output.memberIndex] = output.walletStorageKey + } + for _, seat := range []group.MemberIndex{1, 17, 23} { + directory, found := seats[seat] + if !found { + t.Errorf("seat [%v] was not read as a preserved output", seat) + continue + } + testutils.AssertStringsEqual( + t, + fmt.Sprintf("wallet directory of seat [%v]", seat), + "wallet", + directory, + ) + } +} + +// TestSignerQuarantine_PreservedOutputs_FailsOnUnreadableNamespace proves an +// enumeration error is returned rather than absorbed into a shorter list: a +// truncated count of preserved material is indistinguishable from an accurate +// low one. +func TestSignerQuarantine_PreservedOutputs_FailsOnUnreadableNamespace(t *testing.T) { + quarantine := newSignerQuarantine( + context.Background(), + logger, + &unreadableHandle{}, + ) + + outputs, err := quarantine.preservedOutputs() + if err == nil { + t.Fatalf("expected an error, got [%d] outputs", len(outputs)) + } + if outputs != nil { + t.Errorf("expected no outputs beside the error, got [%v]", outputs) + } + if !strings.Contains(err.Error(), "unreadable") { + t.Errorf("expected the underlying read error, got [%v]", err) + } +} + +// unreadableHandle is a protected namespace whose enumeration fails, as a disk +// namespace does when its directory cannot be read. +type unreadableHandle struct { + mockPersistenceHandle +} + +func (h *unreadableHandle) ReadAll() ( + <-chan persistence.DataDescriptor, + <-chan error, +) { + descriptors := make(chan persistence.DataDescriptor) + errs := make(chan error, 1) + + errs <- fmt.Errorf("the quarantine directory is unreadable") + close(descriptors) + close(errs) + + return descriptors, errs +} + +// unwritableRecordHandle is a protected namespace that refuses the record names +// it is given while writing their neighbours, as a disk namespace does when +// particular files cannot be written. +// +// The names are a list because a preserved output is offered under more than +// one of them: refusing the record pair and refusing the output are different +// namespaces, and only the second one costs the node a share. +type unwritableRecordHandle struct { + mockPersistenceHandle + + // refusedNamePrefixes name the records this namespace will not accept. + refusedNamePrefixes []string +} + +func (h *unwritableRecordHandle) refuses(name string) bool { + for _, prefix := range h.refusedNamePrefixes { + if strings.HasPrefix(name, prefix) { + return true + } + } + + return false +} + +func (h *unwritableRecordHandle) Save( + data []byte, + directory string, + name string, +) error { + if h.refuses(name) { + return fmt.Errorf("cannot write [%s]", name) + } + + return h.mockPersistenceHandle.Save(data, directory, name) +} + +// quarantineLogCapture records the error lines a preservation path emits, so a +// test can hold the operator's account of what was preserved to what the +// namespace actually holds. The operator log is the only account of a +// quarantine an operator reads at the time, and it is the one that was +// reporting a preserved share as lost. +type quarantineLogCapture struct { + testutils.MockLogger + + mu sync.Mutex + errors []string + warnings []string +} + +func (c *quarantineLogCapture) Errorf(format string, args ...interface{}) { + c.mu.Lock() + defer c.mu.Unlock() + c.errors = append(c.errors, fmt.Sprintf(format, args...)) +} + +func (c *quarantineLogCapture) Warnf(format string, args ...interface{}) { + c.mu.Lock() + defer c.mu.Unlock() + c.warnings = append(c.warnings, fmt.Sprintf(format, args...)) +} + +func (c *quarantineLogCapture) joined() string { + c.mu.Lock() + defer c.mu.Unlock() + return strings.Join(c.errors, "\n") +} + +// joinedWarnings is kept apart from joined because the two say different +// things: an error line is a state an operator has to act on, a warning line is +// the record of what happened to an output. +func (c *quarantineLogCapture) joinedWarnings() string { + c.mu.Lock() + defer c.mu.Unlock() + return strings.Join(c.warnings, "\n") +} + +// savedNames lists the record names a namespace accepted, in write order. +func savedNames(handle *mockPersistenceHandle) []string { + names := make([]string, 0, len(handle.saved)) + for _, descriptor := range handle.saved { + names = append(names, descriptor.Name()) + } + return names +} + +// TestSignerQuarantine_Preserve_AttemptsBothRecordsAndReportsWhatLanded proves +// no record of a quarantined output is skipped because another failed, and that +// the caller is told which of them the namespace actually holds. +// +// The records mean different things — the membership is the key material a +// rollback has to account for, the metadata is the record explaining it, the +// handoff is both at once — and the error alone cannot say which are on disk. A +// caller that guesses is how the operator log, the published count, and the +// offline audit come to describe the same directory differently. +// +// A name the namespace refuses does not decide the outcome: a round that cannot +// complete the pair offers the output whole under a name of its own, so only a +// namespace refusing everything leaves a share with nowhere to go. +func TestSignerQuarantine_Preserve_AttemptsBothRecordsAndReportsWhatLanded( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + signer, err := de.buildFinalSigner( + result, + group.MemberIndex(1), + gsr.OperatorsAddresses, + ) + if err != nil { + t.Fatal(err) + } + + tests := map[string]struct { + refusedNamePrefix string + expectedSaved []string + membershipPersisted bool + metadataPersisted bool + handoffPersisted bool + }{ + "both records land": { + refusedNamePrefix: "/nothing_is_refused", + expectedSaved: []string{"/membership_1", "/metadata_1"}, + membershipPersisted: true, + metadataPersisted: true, + }, + "the metadata is refused": { + refusedNamePrefix: "/metadata_", + expectedSaved: []string{"/membership_1", "/handoff_1"}, + membershipPersisted: true, + metadataPersisted: false, + handoffPersisted: true, + }, + "the membership is refused": { + refusedNamePrefix: "/membership_", + expectedSaved: []string{"/metadata_1", "/handoff_1"}, + membershipPersisted: false, + metadataPersisted: true, + handoffPersisted: true, + }, + "the namespace refuses every record": { + refusedNamePrefix: "/", + expectedSaved: []string{}, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + handle := &unwritableRecordHandle{ + refusedNamePrefixes: []string{test.refusedNamePrefix}, + } + // One round, then the process is taken to have ended: this asks + // what a single pass writes and reports, not how long the retry + // behind it lasts. + quarantine := newTestSignerQuarantine(handle, 1) + + state, err := quarantine.preserve( + signer, + QuarantinedSignerMetadata{ + ReleaseEpoch: participation.CompiledEpoch.String(), + Ceremony: string(participation.TBTCDKG), + }, + quarantineObserver{}, + ) + + expectedComplete := test.handoffPersisted || + (test.membershipPersisted && test.metadataPersisted) + if expectedComplete && err != nil { + t.Fatalf("expected no error, got [%v]", err) + } + if !expectedComplete && err == nil { + t.Fatal("expected a preservation error") + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + test.membershipPersisted, + state.membershipPersisted, + ) + testutils.AssertBoolsEqual( + t, + "metadata persisted", + test.metadataPersisted, + state.metadataPersisted, + ) + testutils.AssertBoolsEqual( + t, + "handoff persisted", + test.handoffPersisted, + state.handoffPersisted, + ) + + got := savedNames(&handle.mockPersistenceHandle) + if !reflect.DeepEqual(got, test.expectedSaved) { + t.Errorf( + "namespace holds %v, expected %v", + got, + test.expectedSaved, + ) + } + }) + } +} + +// TestDkgExecutor_PreserveInterruptedSigner_RefusedMetadataLeavesThePermitUnresolved +// proves a share the namespace accepted without any audit record is counted but +// not called resolved: the published count includes the output, the permit ends +// with no terminal outcome, and the node stops taking new ceremonies. +// +// The key material is on disk, so a rollback has to account for it — that is the +// count. What is missing is everything that would let the offline audit match it +// against the chain: the mode, the canonical anchor, the ceremony, the seat, and +// the operation that was refused. Recording the permit as quarantined on the key +// material alone would hand the rollback decision a preserved share nothing +// explains and call the inventory complete. +// +// Both records that carry the audit fields are refused here, because either one +// of them landing is enough to explain the share. +func TestDkgExecutor_PreserveInterruptedSigner_RefusedMetadataLeavesThePermitUnresolved( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + handle := &unwritableRecordHandle{ + refusedNamePrefixes: []string{"/metadata_", "/handoff_"}, + } + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + permit := newTestPermit(participation.TBTCDKG) + operatorLog := &quarantineLogCapture{} + + de.preserveInterruptedSigner( + operatorLog, + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/membership_1"}, + ) { + t.Errorf("namespace holds %v, expected only the membership", got) + } + + // What the operator is told has to be what the namespace holds. Reporting + // a preserved share as lost is how key material ends up on a host that + // nobody goes looking on. + if logged := operatorLog.joined(); strings.Contains( + logged, + "only in memory", + ) || !strings.Contains(logged, "the share is preserved") { + t.Errorf( + "the operator log must report the share as preserved, got [%s]", + logged, + ) + } + + if outcomes := permit.recordedTerminalOutcomes(); len(outcomes) != 0 { + t.Errorf( + "a preserved share with no audit record explaining it must "+ + "leave the permit unresolved, got %v", + outcomes, + ) + } + + value, published := recorder.gaugePublished( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + ) + if !published { + t.Fatal("expected the preserved share to be counted") + } + testutils.AssertIntsEqual(t, "reported quarantined signers", 1, int(value)) + + testutils.AssertBoolsEqual( + t, + "the gate quiesced on the unexplained share", + true, + de.participationGate.State().Quiescing, + ) +} + +// TestDkgExecutor_PreserveInterruptedSigner_ProlongedRefusalStillEndsQuarantined +// proves the permit does resolve once both halves are durable, however long the +// namespace took to accept them. +// +// The blocking state exists to keep an incomplete output from being called +// resolved, not to make every namespace hiccup permanent. A metadata write that +// lands on a later round leaves an output the offline audit can reconcile, so +// the permit ends quarantined — while the node still stops taking new work, +// because it spent longer than a passing fault holding an output the namespace +// did not have. +func TestDkgExecutor_PreserveInterruptedSigner_ProlongedRefusalStillEndsQuarantined( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + // Refused far past the grace budget, then accepted — the namespace an + // operator is repairing while the node holds the share. The handoff is + // refused for as long, so what the retry is waiting on is the pair itself. + handle := &flakyRecordHandle{ + namePrefixes: []string{"/metadata_", "/handoff_"}, + refusals: quarantineGraceAttempts * 4, + } + de.signerQuarantine = newTestSignerQuarantine(handle, 50) + + permit := newTestPermit(participation.TBTCDKG) + + de.preserveInterruptedSigner( + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/membership_1", "/metadata_1"}, + ) { + t.Errorf("namespace holds %v, expected both halves", got) + } + testutils.AssertIntsEqual( + t, + "metadata write attempts", + quarantineGraceAttempts*4+1, + handle.attemptCount(), + ) + + terminalOutcomes := permit.recordedTerminalOutcomes() + testutils.AssertIntsEqual(t, "terminal outcomes", 1, len(terminalOutcomes)) + if len(terminalOutcomes) == 1 && + terminalOutcomes[0].outcome != + participation.TerminalOutcomeQuarantined { + t.Errorf( + "unexpected terminal outcome [%s]", + terminalOutcomes[0].outcome, + ) + } + + testutils.AssertBoolsEqual( + t, + "the gate quiesced while the namespace was refusing", + true, + de.participationGate.State().Quiescing, + ) +} + +// TestDkgExecutor_PreserveInterruptedSigner_RefusedMembershipIsNotQuarantined +// proves a share no record of the namespace took is not reported as +// quarantined: the permit records no terminal outcome and no count is +// published, while the audit metadata naming the lost share is still written. +// +// The metadata is what tells the offline audit a share was generated and not +// preserved. Skipping it because the records carrying key material failed first +// would leave the loss with no record at all. +// +// Both of those records are refused here. A namespace that turns down only the +// membership still has the handoff to take the share in, and this is about the +// state where nothing did. +func TestDkgExecutor_PreserveInterruptedSigner_RefusedMembershipIsNotQuarantined( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + handle := &unwritableRecordHandle{ + refusedNamePrefixes: []string{"/membership_", "/handoff_"}, + } + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + permit := newTestPermit(participation.TBTCDKG) + operatorLog := &quarantineLogCapture{} + + de.preserveInterruptedSigner( + operatorLog, + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/metadata_1"}, + ) { + t.Errorf("namespace holds %v, expected only the metadata", got) + } + + // The share really is only in memory here, and the operator log has to say + // so — and say that the audit record naming it did survive, since that is + // the only thing left pointing at the loss. + if logged := operatorLog.joined(); !strings.Contains( + logged, + "only in memory", + ) || !strings.Contains(logged, "auditMetadataPreserved=true") { + t.Errorf( + "the operator log must report the share as lost beside the "+ + "record that survived it, got [%s]", + logged, + ) + } + + if outcomes := permit.recordedTerminalOutcomes(); len(outcomes) != 0 { + t.Errorf( + "a share that never reached the namespace must not end the "+ + "permit, got %v", + outcomes, + ) + } + + if _, published := recorder.gaugePublished( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + ); published { + t.Error( + "no count may be published for a share the namespace refused", + ) + } +} + +// flakyRecordHandle refuses the first refusals writes of the named record and +// accepts every write after that, counting the attempts made on it. It stands +// for a namespace that is momentarily unwritable — a mount being restored, a +// disk an operator is draining — which is the condition a single-attempt write +// would report as permanently lost key material. +type flakyRecordHandle struct { + mockPersistenceHandle + + // namePrefixes name the records this namespace refuses while it is + // unwritable. The first of them is the one whose attempts are counted: + // preservation writes it once per round for as long as it has not landed, + // so its attempt number is the round number. + namePrefixes []string + refusals int + + mu sync.Mutex + attempts int +} + +func (h *flakyRecordHandle) Save( + data []byte, + directory string, + name string, +) error { + refused := false + for _, prefix := range h.namePrefixes { + if strings.HasPrefix(name, prefix) { + refused = true + break + } + } + if !refused { + return h.mockPersistenceHandle.Save(data, directory, name) + } + + h.mu.Lock() + if strings.HasPrefix(name, h.namePrefixes[0]) { + h.attempts++ + } + attempt := h.attempts + h.mu.Unlock() + + if attempt <= h.refusals { + return fmt.Errorf("cannot write [%s] yet", name) + } + + return h.mockPersistenceHandle.Save(data, directory, name) +} + +func (h *flakyRecordHandle) attemptCount() int { + h.mu.Lock() + defer h.mu.Unlock() + return h.attempts +} + +// newTestSignerQuarantine builds a quarantine store that retries exactly the way +// the production one does but ends its process lifetime after the given number +// of rounds instead of spending the real waits between them. +// +// Production preservation stops only with the process, because the key material +// it is holding cannot be generated again and no elapsed time makes discarding +// it safe. A test driving a namespace that never accepts the write therefore has +// to supply the ending itself: the round count is where this process is taken to +// have gone away. +func newTestSignerQuarantine( + handle persistence.ProtectedHandle, + roundsBeforeShutdown int, +) *signerQuarantine { + quarantine := newSignerQuarantine(context.Background(), logger, handle) + + // Counted atomically because a store is shared by the members of a ceremony + // that quarantine concurrently. + var rounds atomic.Int64 + quarantine.wait = func(context.Context, time.Duration) bool { + return rounds.Add(1) < int64(roundsBeforeShutdown) + } + + return quarantine +} + +// TestSignerQuarantine_Preserve_KeepsTryingThroughAProlongedRefusal proves key +// material is not declared lost because a namespace stayed unwritable for a +// while: preservation keeps the share in hand across far more refusals than a +// passing fault produces, and still writes it when the namespace comes back. +// +// The material on this path cannot be generated again, and the conditions that +// refuse a write are the ones an operator repairs — a mount being restored, a +// full disk being drained. A fixed attempt budget turns the length of that +// repair into the difference between a preserved share and a lost one, which is +// not a distinction the node is entitled to make. +func TestSignerQuarantine_Preserve_KeepsTryingThroughAProlongedRefusal( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + signer, err := de.buildFinalSigner( + result, + group.MemberIndex(1), + gsr.OperatorsAddresses, + ) + if err != nil { + t.Fatal(err) + } + + const refusals = quarantineGraceAttempts * 8 + + // The handoff is refused for as long as the membership, so what is being + // held across the repair is the key material itself and not a record that + // already put it somewhere. + handle := &flakyRecordHandle{ + namePrefixes: []string{"/membership_", "/handoff_"}, + refusals: refusals, + } + + // The notification the node acts on fires once and does not end the + // attempt, so it is counted rather than allowed to stand in for a result. + notifications := 0 + + quarantine := newTestSignerQuarantine(handle, refusals*2) + + operatorLog := &quarantineLogCapture{} + quarantine.logger = operatorLog + + state, err := quarantine.preserve( + signer, + QuarantinedSignerMetadata{ + ReleaseEpoch: participation.CompiledEpoch.String(), + Ceremony: string(participation.TBTCDKG), + }, + quarantineObserver{ + stillIncomplete: func(quarantineState, error) { notifications++ }, + }, + ) + if err != nil { + t.Fatalf( + "expected the retried write to preserve the share, got [%v]", + err, + ) + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + true, + state.membershipPersisted, + ) + testutils.AssertIntsEqual( + t, + "membership write attempts", + refusals+1, + handle.attemptCount(), + ) + testutils.AssertIntsEqual(t, "block notifications", 1, notifications) + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/metadata_1", "/membership_1"}, + ) { + t.Errorf("namespace holds %v, expected both halves", got) + } + + // The node was told, and the operator record says, that this share reached + // no namespace. Leaving that as the last word would have an operator repair + // a loss the namespace had already stopped being. + if logged := operatorLog.joinedWarnings(); !strings.Contains( + logged, + "took the tbtc key material it had been refusing", + ) || !strings.Contains( + logged, + fmt.Sprintf("[round=%d]", refusals+1), + ) { + t.Errorf( + "the operator record must say which round the namespace took the "+ + "material in, got [%s]", + logged, + ) + } +} + +// TestSignerQuarantine_Preserve_GivesUpOnlyWhenTheProcessEnds proves the retry +// has no deadline of its own: a namespace that never accepts the write is +// attempted every round until the process itself goes away, and the error says +// that is what ended it. +// +// The node is told once, well before that, that it is holding key material the +// namespace does not have — but being told is not the same as being finished, +// and preservation carries on behind the notification. +func TestSignerQuarantine_Preserve_GivesUpOnlyWhenTheProcessEnds( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + signer, err := de.buildFinalSigner( + result, + group.MemberIndex(1), + gsr.OperatorsAddresses, + ) + if err != nil { + t.Fatal(err) + } + + const roundsBeforeShutdown = quarantineGraceAttempts * 9 + + handle := &flakyRecordHandle{ + namePrefixes: []string{"/membership_", "/handoff_"}, + refusals: math.MaxInt32, + } + + notifications := 0 + + state, err := newTestSignerQuarantine( + handle, + roundsBeforeShutdown, + ).preserve( + signer, + QuarantinedSignerMetadata{ + ReleaseEpoch: participation.CompiledEpoch.String(), + Ceremony: string(participation.TBTCDKG), + }, + quarantineObserver{ + stillIncomplete: func(quarantineState, error) { notifications++ }, + }, + ) + if err == nil { + t.Fatal("expected a preservation error") + } + + testutils.AssertBoolsEqual( + t, + "membership persisted", + false, + state.membershipPersisted, + ) + testutils.AssertBoolsEqual( + t, + "metadata persisted", + true, + state.metadataPersisted, + ) + testutils.AssertBoolsEqual( + t, + "handoff persisted", + false, + state.handoffPersisted, + ) + testutils.AssertIntsEqual( + t, + "membership write attempts", + roundsBeforeShutdown, + handle.attemptCount(), + ) + testutils.AssertIntsEqual(t, "block notifications", 1, notifications) + + if want := fmt.Sprintf( + "in %d rounds before the process ended", + roundsBeforeShutdown, + ); !strings.Contains(err.Error(), want) { + t.Errorf( + "the error must say the process ending is what stopped the "+ + "retry, got [%v]", + err, + ) + } +} + +// latchedMembershipHandle refuses both halves of a preserved output until the +// given round, then takes the key material and goes on refusing the audit record +// for good. +// +// It stands for the namespace state a count has to survive: the share is down on +// disk while the record explaining it is still missing, and the preservation +// holding the pair open has not returned and, on this namespace, never will. +type latchedMembershipHandle struct { + mockPersistenceHandle + + // membershipTakenAtRound is the round from which the key material is + // accepted. The membership is attempted once per round for as long as it + // has not landed, so its attempt count is the round number. + membershipTakenAtRound int + + mu sync.Mutex + membershipAttempts int +} + +func (h *latchedMembershipHandle) Save( + data []byte, + directory string, + name string, +) error { + if !strings.HasPrefix(name, "/membership_") { + return fmt.Errorf("cannot write [%s]", name) + } + + h.mu.Lock() + h.membershipAttempts++ + round := h.membershipAttempts + h.mu.Unlock() + + if round < h.membershipTakenAtRound { + return fmt.Errorf("cannot write [%s] yet", name) + } + + return h.mockPersistenceHandle.Save(data, directory, name) +} + +// TestDkgExecutor_PreserveInterruptedSigner_CountsTheShareTheNamespaceTakesMidRetry +// proves the reported count rises at the moment the namespace takes the key +// material, while the preservation that wrote it is still running and the audit +// record beside it is still being refused. +// +// The node is told once, after the grace rounds, that it is holding an output the +// namespace does not fully have, and quiescing on that notification is one-way. +// What follows it is not: a namespace can take the share several rounds later and +// go on refusing the record, and the preservation then keeps running for the rest +// of the process. A count taken from that notification, or from what preserve +// eventually returns, would report an empty quarantine for all of that time over +// a namespace holding key material — the all-clear a rollback decision must never +// be given. +func TestDkgExecutor_PreserveInterruptedSigner_CountsTheShareTheNamespaceTakesMidRetry( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + // The share lands well after the grace rounds are spent, so the node has + // already been told the pair is incomplete by the time the namespace takes + // it, and the preservation outlives the write by several more rounds. + const membershipTakenAtRound = quarantineGraceAttempts + 3 + const roundsBeforeShutdown = membershipTakenAtRound + 3 + + handle := &latchedMembershipHandle{ + membershipTakenAtRound: membershipTakenAtRound, + } + + quarantine := newSignerQuarantine(context.Background(), logger, handle) + + // Sampled between rounds, which is inside the preservation: a count read + // after preserve has returned cannot tell a gauge that rose at the write + // from one that rose at the return, and the two are the whole question here. + countAfterRound := make([]int, 0, roundsBeforeShutdown) + round := 0 + quarantine.wait = func(context.Context, time.Duration) bool { + round++ + countAfterRound = append(countAfterRound, int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + ))) + return round < roundsBeforeShutdown + } + de.signerQuarantine = quarantine + + permit := newTestPermit(participation.TBTCDKG) + + de.preserveInterruptedSigner( + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + testutils.AssertIntsEqual( + t, + "rounds the preservation spent", + roundsBeforeShutdown, + len(countAfterRound), + ) + + for spentRound, count := range countAfterRound { + // countAfterRound[i] is what the count said once round i+1 was over. + expected := 0 + if spentRound+1 >= membershipTakenAtRound { + expected = 1 + } + + testutils.AssertIntsEqual( + t, + fmt.Sprintf("reported quarantined signers after round %d", + spentRound+1), + expected, + count, + ) + } + + // The record explaining the share never landed, so this is the state the + // count had to be right about rather than one preservation resolved. + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/membership_1"}, + ) { + t.Errorf("namespace holds %v, expected only the key material", got) + } + + if outcomes := permit.recordedTerminalOutcomes(); len(outcomes) != 0 { + t.Errorf( + "an output whose audit record never landed must not end the "+ + "permit, got %v", + outcomes, + ) + } +} + +// TestDkgExecutor_PreserveInterruptedSigner_WritesTheAuditRecordAheadOfAnyScan +// proves the audit record is written without waiting on a namespace-wide read, +// and that no such read happens until both halves of the output are down. +// +// The share and the record explaining it are written in one round, the share +// first. Everything that runs between them delays the record, and a +// namespace-wide read is the slowest thing a node does to that namespace: the +// same directory trouble that makes an enumeration hang is what a preservation +// is racing in the first place. A share whose record was held up behind a scan +// is preserved material an offline audit cannot reconcile — the state this node +// quiesces over — reached because of the count rather than because the +// namespace refused the write. +func TestDkgExecutor_PreserveInterruptedSigner_WritesTheAuditRecordAheadOfAnyScan( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + handle := &heldScanHandle{release: make(chan struct{})} + releaseScans := sync.OnceFunc(func() { close(handle.release) }) + defer releaseScans() + + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + + preserved := make(chan struct{}) + go func() { + defer close(preserved) + de.preserveInterruptedSigner( + logger.With(), + newTestPermit(participation.TBTCDKG), + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + }() + + // Both halves must land while every enumeration this namespace is asked for + // is still held open. The bound decides how long the test waits for a write + // that is not blocked, never whether an unblocked implementation passes. + deadline := time.After(10 * time.Second) + for { + written := handle.written() + if reflect.DeepEqual( + written, + []string{"/membership_1", "/metadata_1"}, + ) { + break + } + + select { + case <-deadline: + t.Fatalf( + "the quarantine pair was not written while the namespace scan "+ + "was held open; namespace holds %v", + written, + ) + case <-time.After(5 * time.Millisecond): + } + } + + releaseScans() + <-preserved + + // The recount still happens — it is what brings the count back to what the + // namespace holds — but only once the output it is counting is complete. + if writes := handle.writesBeforeScans(); len(writes) == 0 { + t.Fatal("the namespace was never enumerated after the preservation") + } else { + for scan, written := range writes { + testutils.AssertIntsEqual( + t, + fmt.Sprintf("records written when enumeration %d began", scan+1), + 2, + written, + ) + } + } + + testutils.AssertIntsEqual( + t, + "reported quarantined signers", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// heldScanHandle is a protected namespace whose enumerations are all held open +// until released, while its writes go through immediately — the shape of a +// namespace whose directory listing hangs. It records how many records had been +// written by the time each enumeration began, which is what says whether a +// write was waiting behind a scan or the other way round. +type heldScanHandle struct { + mockPersistenceHandle + + release chan struct{} + + mu sync.Mutex + names []string + writesAtScanTime []int +} + +func (h *heldScanHandle) Save( + data []byte, + directory string, + name string, +) error { + h.mu.Lock() + defer h.mu.Unlock() + + h.names = append(h.names, name) + + return h.mockPersistenceHandle.Save(data, directory, name) +} + +func (h *heldScanHandle) written() []string { + h.mu.Lock() + defer h.mu.Unlock() + + return append([]string(nil), h.names...) +} + +func (h *heldScanHandle) writesBeforeScans() []int { + h.mu.Lock() + defer h.mu.Unlock() + + return append([]int(nil), h.writesAtScanTime...) +} + +func (h *heldScanHandle) ReadAll() ( + <-chan persistence.DataDescriptor, + <-chan error, +) { + h.mu.Lock() + h.writesAtScanTime = append(h.writesAtScanTime, len(h.names)) + h.mu.Unlock() + + descriptors := make(chan persistence.DataDescriptor) + errs := make(chan error) + + go func() { + defer close(descriptors) + defer close(errs) + + <-h.release + + h.mu.Lock() + saved := append([]persistence.DataDescriptor(nil), h.saved...) + h.mu.Unlock() + + for _, descriptor := range saved { + descriptors <- descriptor + } + }() + + return descriptors, errs +} + +// TestSignerQuarantine_Preserve_WritesOnlyTheHalfTheNamespaceLacks proves a +// round does not rewrite a half that already landed. The retry exists for the +// missing record, and rewriting the preserved one would keep touching key +// material the namespace has already accepted. +func TestSignerQuarantine_Preserve_WritesOnlyTheHalfTheNamespaceLacks( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + signer, err := de.buildFinalSigner( + result, + group.MemberIndex(1), + gsr.OperatorsAddresses, + ) + if err != nil { + t.Fatal(err) + } + + handle := &flakyRecordHandle{ + namePrefixes: []string{"/metadata_", "/handoff_"}, + refusals: quarantineGraceAttempts + 2, + } + + if _, err := newTestSignerQuarantine(handle, 50).preserve( + signer, + QuarantinedSignerMetadata{ + ReleaseEpoch: participation.CompiledEpoch.String(), + Ceremony: string(participation.TBTCDKG), + }, + quarantineObserver{}, + ); err != nil { + t.Fatalf("expected the retried write to preserve the share, got [%v]", err) + } + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/membership_1", "/metadata_1"}, + ) { + t.Errorf( + "namespace holds %v, expected each half written exactly once", + got, + ) + } +} + +// TestDkgExecutor_PreserveInterruptedSigner_LostShareBlocksNewCeremonies proves +// a share that reached no namespace stops this node from starting new work. +// +// The share existed only in the goroutine that generated it and the namespace +// refused it for the whole retry budget, so nothing an operator or the offline +// audit can read accounts for it. A node that keeps taking ceremonies after that +// builds further state on a host whose inventory is already known to be +// incomplete, and the rollback audit reconciles namespaces against the chain — +// it cannot reconcile a share nobody wrote down. +func TestDkgExecutor_PreserveInterruptedSigner_LostShareBlocksNewCeremonies( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + de.signerQuarantine = newTestSignerQuarantine( + &unwritableRecordHandle{ + refusedNamePrefixes: []string{"/membership_", "/handoff_"}, + }, + 1, + ) + + before := de.participationGate.State() + testutils.AssertBoolsEqual( + t, + "the gate issues permits before the loss", + true, + before.Allowed, + ) + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + after := de.participationGate.State() + testutils.AssertBoolsEqual( + t, + "the gate quiesced on the lost share", + true, + after.Quiescing, + ) + testutils.AssertBoolsEqual( + t, + "the gate still issues permits", + false, + after.Allowed, + ) + + // A quiescing gate refuses before it looks at the anchor at all, which is + // why the refusal has to be read by sentinel rather than by the fact that + // one happened. + if _, err := de.participationGate.Begin( + participation.TBTCDKG, + before.CurrentBlock, + tbtcDKGPermitIdentity(big.NewInt(2), group.MemberIndex(1)), + ); !errors.Is(err, participation.ErrQuiescing) { + t.Errorf( + "a node holding a lost share must refuse new ceremonies, got [%v]", + err, + ) + } +} + +// TestDkgExecutor_PreserveInterruptedSigner_PublishesIncompleteWhileProcessLives +// proves total quarantine refusal is visible while the node can still answer a +// scrape. The preservation retry is deliberately held inside a live process +// lifetime after its grace notification; neither the counter nor the gauge may +// wait for teardown to report the output the node still holds only in memory. +func TestDkgExecutor_PreserveInterruptedSigner_PublishesIncompleteWhileProcessLives( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + lifetime, cancelLifetime := context.WithCancel(context.Background()) + defer cancelLifetime() + + quarantine := newSignerQuarantine( + lifetime, + logger, + &unwritableRecordHandle{refusedNamePrefixes: []string{"/"}}, + ) + quarantine.graceAttempts = 1 + + waitingWithLiveOutput := make(chan struct{}) + var notifyWaiting sync.Once + quarantine.wait = func(ctx context.Context, _ time.Duration) bool { + notifyWaiting.Do(func() { close(waitingWithLiveOutput) }) + <-ctx.Done() + return false + } + + de.signerQuarantine = quarantine + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preservationDone := make(chan struct{}) + go func() { + defer close(preservationDone) + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + }() + + select { + case <-waitingWithLiveOutput: + case <-time.After(10 * time.Second): + t.Fatal("preservation did not reach its live incomplete retry") + } + + if lifetime.Err() != nil { + t.Fatal("the process lifetime ended before the signal was inspected") + } + if got := recorder.counter( + clientinfo.MetricParticipationTBTCQuarantinePreservationFailuresTotal, + ); got != 1 { + t.Errorf( + "tBTC quarantine-preservation failures = [%v] while live, want 1", + got, + ) + } + if got := recorder.gauge( + clientinfo.MetricParticipationTBTCQuarantineIncompleteOutputs, + ); got != 1 { + t.Errorf( + "tBTC incomplete quarantine outputs = [%v] while live, want 1", + got, + ) + } + + cancelLifetime() + select { + case <-preservationDone: + case <-time.After(10 * time.Second): + t.Fatal("preservation did not return after the process lifetime ended") + } +} + +// TestDkgExecutor_PreserveInterruptedSigner_ClearsIncompleteAfterRecovery +// proves the live gauge describes current incomplete outputs rather than +// latching every historical failure. The counter retains the grace-exhausting +// episode, while a namespace that recovers and takes the full output brings the +// gauge back to zero. +func TestDkgExecutor_PreserveInterruptedSigner_ClearsIncompleteAfterRecovery( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + handle := &flakyRecordHandle{ + namePrefixes: []string{"/membership_", "/handoff_"}, + refusals: quarantineGraceAttempts + 1, + } + de.signerQuarantine = newTestSignerQuarantine(handle, 50) + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + if got := recorder.counter( + clientinfo.MetricParticipationTBTCQuarantinePreservationFailuresTotal, + ); got != 1 { + t.Errorf( + "tBTC quarantine-preservation failures after recovery = [%v], want 1", + got, + ) + } + if got := recorder.gauge( + clientinfo.MetricParticipationTBTCQuarantineIncompleteOutputs, + ); got != 0 { + t.Errorf( + "tBTC incomplete quarantine outputs after recovery = [%v], want 0", + got, + ) + } +} + +// TestDkgExecutor_PreserveInterruptedSigner_RefusedActiveSaveFallsBackToQuarantine +// proves a registered wallet's share the active namespace refused is preserved +// in the quarantine namespace rather than dropped, and that the audit record +// says the active save is what was refused. +// +// The active namespace is where a restart would load the share from, so a write +// refused there leaves a registered wallet short a signer. The quarantine +// namespace is a separate namespace with its own failure modes: a share +// preserved there is recoverable and the offline audit reports it, where a +// dropped one is neither. +func TestDkgExecutor_PreserveInterruptedSigner_RefusedActiveSaveFallsBackToQuarantine( + t *testing.T, +) { + de, result, gsr, _, quarantineHandle := setupPreserveScenario(t) + + walletPublicKey := result.PrivateKeyShare.PublicKey() + walletID, err := de.chain.CalculateWalletID(walletPublicKey) + if err != nil { + t.Fatal(err) + } + de.chain.(*localChain).setWallet( + bitcoin.PublicKeyHash(walletPublicKey), + &WalletChainData{EcdsaWalletID: walletID, State: StateLive}, + ) + + // The active namespace refuses the very record the registered wallet needs. + activeHandle := &unwritableRecordHandle{refusedNamePrefixes: []string{"/membership_"}} + de.walletRegistry, err = newWalletRegistry( + activeHandle, + de.chain.CalculateWalletID, + ) + if err != nil { + t.Fatal(err) + } + + permit := newTestPermit(participation.TBTCDKG) + + de.preserveInterruptedSigner( + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + if got := savedNames(quarantineHandle); !reflect.DeepEqual( + got, + []string{"/membership_1", "/metadata_1"}, + ) { + t.Errorf( + "quarantine namespace holds %v, expected the refused share", + got, + ) + } + + metadata := &QuarantinedSignerMetadata{} + for _, descriptor := range quarantineHandle.saved { + if descriptor.Name() != "/metadata_1" { + continue + } + content, err := descriptor.Content() + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(content, metadata); err != nil { + t.Fatal(err) + } + } + if want := "tbtc_dkg_signer_activation_after_refused_active_save"; metadata. + FailedOperation != want { + t.Errorf( + "audit record says [%s], expected [%s]", + metadata.FailedOperation, + want, + ) + } + + terminalOutcomes := permit.recordedTerminalOutcomes() + testutils.AssertIntsEqual(t, "terminal outcomes", 1, len(terminalOutcomes)) + if len(terminalOutcomes) == 1 && + terminalOutcomes[0].outcome != + participation.TerminalOutcomeQuarantined { + t.Errorf( + "unexpected terminal outcome [%s]", + terminalOutcomes[0].outcome, + ) + } +} + +// TestDkgExecutor_ReportQuarantinedSigners_PersistedShareSurvivesAnUnreadableRecount +// proves a share this process durably persisted is still counted when the +// recount that follows the write cannot read the namespace. +// +// Keeping the last published count is right when that count came from a scan +// that saw the namespace. It is wrong immediately after a write: the standing +// number is a cold start's zero, the namespace now holds key material, and +// leaving the zero up says a rollback has nothing to account for. What the scan +// failure cannot take away is what this process itself wrote. +func TestDkgExecutor_ReportQuarantinedSigners_PersistedShareSurvivesAnUnreadableRecount( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + // The cold start an operator reads first: an empty, readable namespace + // counted and published as zero. + if err := de.reportInitialQuarantinedSigners(); err != nil { + t.Fatal(err) + } + if value, published := recorder.gaugePublished( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + ); !published || value != 0 { + t.Fatalf( + "expected a published zero to start from, got [%v] published=[%v]", + value, + published, + ) + } + + // A namespace that takes the write and then cannot be listed. + de.signerQuarantine = newTestSignerQuarantine(&unreadableHandle{}, 1) + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + testutils.AssertIntsEqual( + t, + "reported quarantined signers after the recount failed", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportInitialQuarantinedSigners_CountsAShareTheNamespaceTookLate +// proves the whole handoff survives a restart: a share the namespace refused for +// far longer than a passing fault, and then accepted, is counted by the next +// process that starts over the same namespace. +// +// The count is what a rollback decision reads, and it is taken by whichever +// process comes next rather than by the one that wrote. A preservation only the +// writing process knew about would leave the material invisible to exactly the +// decision it exists for — which is the same reason the retry is allowed to +// outlast any particular fault in the first place. +func TestDkgExecutor_ReportInitialQuarantinedSigners_CountsAShareTheNamespaceTookLate( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + handle := &flakyRecordHandle{ + namePrefixes: []string{"/membership_", "/handoff_"}, + refusals: quarantineGraceAttempts * 5, + } + de.signerQuarantine = newTestSignerQuarantine(handle, 100) + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/metadata_1", "/membership_1"}, + ) { + t.Fatalf("namespace holds %v, expected both halves", got) + } + + // The next process: a new executor over the namespace the last one left, + // sharing none of its state — no floor, no standing count, no cache of + // what was written. + restarted, _, _, _, _ := setupPreserveScenario(t) + recorder := newDispatchGaugeRecorder() + restarted.metricsRecorder = recorder + restarted.signerQuarantine = newTestSignerQuarantine(handle, 1) + + if err := restarted.reportInitialQuarantinedSigners(); err != nil { + t.Fatal(err) + } + + testutils.AssertIntsEqual( + t, + "preserved outputs the next process counts", + 1, + int(recorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) +} + +// TestDkgExecutor_ReportInitialQuarantinedSigners_RecoversAWholeOutputTheNamespaceWouldNotPair +// proves what a later process recovers when the namespace refuses the key +// material's own record for good: the combined handoff carries the share and +// everything that explains it, and the next process over the same namespace +// finds all of it. +// +// This is the state that used to cost a node a share. The membership record is +// where preservation prefers to put the material, the metadata beside it is only +// the explanation, and a namespace that took the second while refusing the first +// left a note about a share that reached no disk — the one half no ceremony can +// generate a second time. The handoff is one write carrying both, so the output +// survives under a name the namespace will take, and a process that starts over +// that namespace can read back the material, the seat and wallet it belongs to, +// and the mode, canonical anchor, ceremony, and refused operation the offline +// audit reconciles against the chain. +func TestDkgExecutor_ReportInitialQuarantinedSigners_RecoversAWholeOutputTheNamespaceWouldNotPair( + t *testing.T, +) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + // A namespace that will not take the key material's own record at all, and + // takes the combined one only well after the grace rounds are spent — so the + // node has already been told it is holding a share nothing has, and the + // process is on its way out when the namespace finally comes back. + const handoffTakenAtRound = quarantineGraceAttempts * 2 + handle := &latchedHandoffHandle{handoffTakenAtRound: handoffTakenAtRound} + de.signerQuarantine = newTestSignerQuarantine( + handle, + handoffTakenAtRound+1, + ) + + permit := newTestPermit(participation.TBTCDKG) + + de.preserveInterruptedSigner( + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + "tbtc_dkg_signer_activation", + fmt.Errorf("activation refused"), + ) + + if got := savedNames(&handle.mockPersistenceHandle); !reflect.DeepEqual( + got, + []string{"/metadata_1", "/handoff_1"}, + ) { + t.Fatalf( + "namespace holds %v, expected the output preserved whole", + got, + ) + } + + // The namespace holds the material and its explanation, so this permit is + // resolved rather than left for the offline barrier to block on. + terminalOutcomes := permit.recordedTerminalOutcomes() + testutils.AssertIntsEqual(t, "terminal outcomes", 1, len(terminalOutcomes)) + if len(terminalOutcomes) == 1 && + terminalOutcomes[0].outcome != + participation.TerminalOutcomeQuarantined { + t.Errorf( + "unexpected terminal outcome [%s]", + terminalOutcomes[0].outcome, + ) + } + + // The next process: a new executor over the namespace the last one left, + // sharing none of its state — no floor, no standing count, no memory of what + // was written. + restarted, _, _, _, _ := setupPreserveScenario(t) + restartedRecorder := newDispatchGaugeRecorder() + restarted.metricsRecorder = restartedRecorder + restarted.signerQuarantine = newSignerQuarantine( + context.Background(), + logger, + handle, + ) + + if err := restarted.reportInitialQuarantinedSigners(); err != nil { + t.Fatal(err) + } + + testutils.AssertIntsEqual( + t, + "preserved outputs the next process counts", + 1, + int(restartedRecorder.gauge( + clientinfo.MetricParticipationQuarantinedTBTCSigners, + )), + ) + + // Counting the record is not the same as being able to use it. The material + // is recovery evidence, so the next process has to be able to read back both + // what was preserved and what explains it. + preserved := handle.savedRecord(t, "/handoff_1") + content, err := preserved.Content() + if err != nil { + t.Fatal(err) + } + + handoff, err := DecodeQuarantinedSignerHandoff(content) + if err != nil { + t.Fatalf("the preserved output cannot be read back: [%v]", err) + } + + record, err := DecodeSignerAuditRecord(handoff.Signer) + if err != nil { + t.Fatalf("the preserved key material cannot be read back: [%v]", err) + } + + testutils.AssertIntsEqual( + t, + "seat the preserved material was generated for", + 1, + int(record.MemberIndex), + ) + if expected := getWalletStorageKey( + result.PrivateKeyShare.PublicKey(), + ); record.WalletStorageKey != expected { + t.Errorf( + "preserved material belongs to wallet [%s], expected [%s]", + record.WalletStorageKey, + expected, + ) + } + + // The fields the offline audit matches against the chain travel with the + // material, so a share recovered this way is reconcilable rather than just + // countable. + metadata := handoff.Metadata + testutils.AssertStringsEqual( + t, + "protocol mode the preserved output was generated under", + permit.Mode().String(), + metadata.ProtocolMode, + ) + testutils.AssertUintsEqual( + t, + "canonical anchor the preserved output was generated under", + permit.CanonicalStartBlock(), + metadata.CanonicalStartBlock, + ) + testutils.AssertStringsEqual( + t, + "ceremony the preserved output was generated in", + string(participation.TBTCDKG), + metadata.Ceremony, + ) + testutils.AssertStringsEqual( + t, + "operation that was refused", + "tbtc_dkg_signer_activation", + metadata.FailedOperation, + ) + testutils.AssertStringsEqual( + t, + "release epoch that preserved the output", + participation.CompiledEpoch.String(), + metadata.ReleaseEpoch, + ) +} + +// latchedHandoffHandle refuses the record carrying key material for good and +// takes the combined handoff only from the given round, as a namespace does when +// one particular file cannot be written and the rest of the directory is +// part-way through an operator's repair. +type latchedHandoffHandle struct { + mockPersistenceHandle + + // handoffTakenAtRound is the round from which the combined record is + // accepted. The membership is attempted once per round for as long as it has + // not landed, and it never lands here, so its attempt count is the round + // number. + handoffTakenAtRound int + + mu sync.Mutex + membershipAttempts int +} + +func (h *latchedHandoffHandle) Save( + data []byte, + directory string, + name string, +) error { + h.mu.Lock() + defer h.mu.Unlock() + + if strings.HasPrefix(name, "/membership_") { + h.membershipAttempts++ + return fmt.Errorf("cannot write [%s]", name) + } + + if strings.HasPrefix(name, "/handoff_") && + h.membershipAttempts < h.handoffTakenAtRound { + return fmt.Errorf("cannot write [%s] yet", name) + } + + return h.mockPersistenceHandle.Save(data, directory, name) +} + +// savedRecord returns the record the namespace holds under the given name. +func (h *latchedHandoffHandle) savedRecord( + t *testing.T, + name string, +) persistence.DataDescriptor { + t.Helper() + + h.mu.Lock() + defer h.mu.Unlock() + + for _, descriptor := range h.saved { + if descriptor.Name() == name { + return descriptor + } + } + + t.Fatalf("the namespace holds no record named [%s]", name) + return nil +} + +// TestSignerQuarantine_PreservedOutputs_RestartSeesWhatEachWriteFailureLeft +// proves what a later process finds in the namespace after a refused write. +// Whichever record of the pair the namespace would not take, the handoff +// carries the output whole, so the restart still finds one preserved share to +// account for. Only a namespace that refuses every record leaves nothing. +// +// The count is read by whichever process comes next, not by the one that wrote, +// so it is taken here by a store that shares nothing with the one that failed. +func TestSignerQuarantine_PreservedOutputs_RestartSeesWhatEachWriteFailureLeft( + t *testing.T, +) { + tests := map[string]struct { + refusedNamePrefixes []string + expectedOutputs int + expectedRecords []string + expectedFailures int + expectedIncomplete int + }{ + "the metadata write is refused": { + refusedNamePrefixes: []string{"/metadata_"}, + expectedOutputs: 1, + expectedRecords: []string{"/membership_1", "/handoff_1"}, + expectedFailures: 0, + expectedIncomplete: 0, + }, + "the membership write is refused": { + refusedNamePrefixes: []string{"/membership_"}, + expectedOutputs: 1, + expectedRecords: []string{"/metadata_1", "/handoff_1"}, + expectedFailures: 0, + expectedIncomplete: 0, + }, + "the share persists but both audit record forms are refused": { + refusedNamePrefixes: []string{"/metadata_", "/handoff_"}, + expectedOutputs: 1, + expectedRecords: []string{"/membership_1"}, + expectedFailures: 1, + expectedIncomplete: 1, + }, + "every write is refused": { + refusedNamePrefixes: []string{"/"}, + expectedOutputs: 0, + expectedRecords: []string{}, + expectedFailures: 1, + expectedIncomplete: 1, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + de, result, gsr, _, _ := setupPreserveScenario(t) + + handle := &unwritableRecordHandle{ + refusedNamePrefixes: test.refusedNamePrefixes, + } + de.signerQuarantine = newTestSignerQuarantine(handle, 1) + recorder := newDispatchGaugeRecorder() + de.metricsRecorder = recorder + + preserveOneSigner(t, de, result, gsr, group.MemberIndex(1)) + + outputs, err := newTestSignerQuarantine(handle, 1).preservedOutputs() + if err != nil { + t.Fatal(err) + } + testutils.AssertIntsEqual( + t, + "preserved outputs a restart finds", + test.expectedOutputs, + len(outputs), + ) + + if got := savedNames( + &handle.mockPersistenceHandle, + ); !reflect.DeepEqual(got, test.expectedRecords) { + t.Errorf( + "namespace holds %v, expected %v", + got, + test.expectedRecords, + ) + } + + testutils.AssertIntsEqual( + t, + "terminal quarantine-preservation failures", + test.expectedFailures, + int(recorder.counter( + clientinfo. + MetricParticipationTBTCQuarantinePreservationFailuresTotal, + )), + ) + testutils.AssertIntsEqual( + t, + "live incomplete quarantine outputs", + test.expectedIncomplete, + int(recorder.gauge( + clientinfo. + MetricParticipationTBTCQuarantineIncompleteOutputs, + )), + ) + }) + } +} + +// TestHeartbeatAction_PenaltySuppressedByFence proves a refused penalty +// fence suppresses the whole inactivity penalty path of a low-activity +// heartbeat: the consecutive-failure counter is not incremented, no claim is +// requested, and the action completes without an ordinary failure. +func TestHeartbeatAction_PenaltySuppressedByFence(t *testing.T) { + walletPublicKeyHex, err := hex.DecodeString( + "0471e30bca60f6548d7b42582a478ea37ada63b402af7b3ddd57f0c95bb6843175" + + "aa0d2053a91a050a6797d85c38f2909cb7027f2344a01986aa2f9f8ca7a0c289", + ) + if err != nil { + t.Fatal(err) + } + + walletPublicKeyStr := hex.EncodeToString(walletPublicKeyHex) + + proposal := &HeartbeatProposal{ + Message: [16]byte{ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + }, + } + + heartbeatFailureCounter := newHeartbeatFailureCounter() + + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + hostChain.setHeartbeatProposalValidationResult(proposal, true) + + // Enough active members to sign, too few for a healthy heartbeat: the + // normal path would count an inactivity failure. + mockExecutor := &mockHeartbeatSigningExecutor{} + mockExecutor.activeOperatorsCount = heartbeatSigningMinimumActiveMembers - 1 + + inactivityClaimExecutor := &mockInactivityClaimExecutor{} + + permit := newTestPermit(participation.TBTCHeartbeat) + permit.commitErr = participation.ErrPenaltySuppressed + + action := newHeartbeatAction( + logger, + hostChain, + wallet{ + publicKey: mustUnmarshalPublicKey(t, walletPublicKeyHex), + }, + mockExecutor, + proposal, + heartbeatFailureCounter, + inactivityClaimExecutor, + 10, + 10+heartbeatTotalProposalValidityBlocks, + func(ctx context.Context, blockHeight uint64) error { + return nil + }, + permit, + ) + + if err := action.execute(); err != nil { + t.Fatalf("a suppressed penalty must not be an ordinary failure: [%v]", err) + } + + testutils.AssertUintsEqual( + t, + "consecutive failure counter after suppression", + 0, + uint64(heartbeatFailureCounter.get(walletPublicKeyStr)), + ) + if inactivityClaimExecutor.sessionID != nil { + t.Error("expected no inactivity claim after suppression") + } + if !permit.isClosed() { + t.Error("expected the action to release its permit") + } +} + +// TestHeartbeatAction_PenaltySuppressedByQuiescingGate proves the real gate's +// quiescence suppresses a pending heartbeat penalty: a low-activity result +// during process quiescence neither increments the consecutive-failure +// counter nor files a claim, even when the counter is one failure short of +// the claim threshold. +func TestHeartbeatAction_PenaltySuppressedByQuiescingGate(t *testing.T) { + walletPublicKeyHex, err := hex.DecodeString( + "0471e30bca60f6548d7b42582a478ea37ada63b402af7b3ddd57f0c95bb6843175" + + "aa0d2053a91a050a6797d85c38f2909cb7027f2344a01986aa2f9f8ca7a0c289", + ) + if err != nil { + t.Fatal(err) + } + + walletPublicKeyStr := hex.EncodeToString(walletPublicKeyHex) + + proposal := &HeartbeatProposal{ + Message: [16]byte{ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + }, + } + + // One failure short of the claim threshold: a normal low-activity result + // would increment the counter and file a claim. + heartbeatFailureCounter := newHeartbeatFailureCounter() + for i := uint(0); i < heartbeatConsecutiveFailureThreshold-1; i++ { + heartbeatFailureCounter.increment(walletPublicKeyStr) + } + + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + hostChain.setHeartbeatProposalValidationResult(proposal, true) + + blockCounter, err := hostChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + permit, err := gate.Begin(participation.TBTCHeartbeat, 1) + if err != nil { + t.Fatal(err) + } + + // Quiescence begins while the heartbeat is in flight: the permit stays + // alive to natural completion but new penalty state is suppressed. + gate.Quiesce(fmt.Errorf("shutdown")) + + mockExecutor := &mockHeartbeatSigningExecutor{} + mockExecutor.activeOperatorsCount = heartbeatSigningMinimumActiveMembers - 1 + + inactivityClaimExecutor := &mockInactivityClaimExecutor{} + + action := newHeartbeatAction( + logger, + hostChain, + wallet{ + publicKey: mustUnmarshalPublicKey(t, walletPublicKeyHex), + }, + mockExecutor, + proposal, + heartbeatFailureCounter, + inactivityClaimExecutor, + 10, + 10+heartbeatTotalProposalValidityBlocks, + func(ctx context.Context, blockHeight uint64) error { + return nil + }, + permit, + ) + + if err := action.execute(); err != nil { + t.Fatalf("a suppressed penalty must not be an ordinary failure: [%v]", err) + } + + testutils.AssertUintsEqual( + t, + "consecutive failure counter after suppression", + uint64(heartbeatConsecutiveFailureThreshold-1), + uint64(heartbeatFailureCounter.get(walletPublicKeyStr)), + ) + if inactivityClaimExecutor.sessionID != nil { + t.Error("expected no inactivity claim after suppression") + } +} + +// TestWalletTransactionExecutor_BroadcastRefusedByGate proves the commit +// fence runs before every Bitcoin broadcast attempt: a refused fence +// surfaces the gate sentinel and the transaction never reaches the Bitcoin +// chain. +func TestWalletTransactionExecutor_BroadcastRefusedByGate(t *testing.T) { + permit := newTestPermit(participation.TBTCSigning) + permit.commitErr = participation.ErrQuiescing + + btcChain := newLocalBitcoinChain() + + wte := &walletTransactionExecutor{ + btcChain: btcChain, + permit: permit, + broadcastOperation: "tbtc_deposit_sweep_bitcoin_broadcast", + } + + tx := &bitcoin.Transaction{Version: 1} + + err := wte.broadcastTransaction( + logger.With(), + tx, + 10*time.Second, + time.Millisecond, + ) + if !errors.Is(err, participation.ErrQuiescing) { + t.Fatalf("expected the gate sentinel, got [%v]", err) + } + + if _, err := btcChain.GetTransaction(tx.Hash()); err == nil { + t.Error("expected the transaction to never reach the Bitcoin chain") + } + + operations := permit.commitOperations() + testutils.AssertIntsEqual(t, "fence consultations", 1, len(operations)) + testutils.AssertStringsEqual( + t, + "fence operation", + "tbtc_deposit_sweep_bitcoin_broadcast", + operations[0], + ) +} + +// quarantinedFailedOperation decodes the single quarantine metadata record in +// the given handle and returns its failed-operation name. +func quarantinedFailedOperation( + t *testing.T, + quarantineHandle *mockPersistenceHandle, +) string { + t.Helper() + + var metadataContent []byte + for _, descriptor := range quarantineHandle.saved { + if strings.HasPrefix(descriptor.Name(), "/metadata_") { + metadataContent, _ = descriptor.Content() + } + } + if metadataContent == nil { + t.Fatal("expected a quarantine metadata record") + } + + var metadata QuarantinedSignerMetadata + if err := json.Unmarshal(metadataContent, &metadata); err != nil { + t.Fatal(err) + } + + return metadata.FailedOperation +} + +// TestDkgExecutor_CompleteDkgCeremony_ActivatesAfterPublication proves the +// completion order of a DKG ceremony: the result publication concludes first, +// the activation fence is consulted only afterwards, and only then is the +// signer persisted in the active namespace and activated in the wallet cache. +func TestDkgExecutor_CompleteDkgCeremony_ActivatesAfterPublication(t *testing.T) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + + published := false + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(context.Context) error { + // The activation fence must not have been consulted before the + // publication concluded. + testutils.AssertIntsEqual( + t, + "fence consultations during publication", + 0, + len(permit.commitOperations()), + ) + published = true + return nil + }, + ) + + if !published { + t.Fatal("expected the result publication to run") + } + if !activated { + t.Fatal("expected the signer to be activated") + } + + operations := permit.commitOperations() + testutils.AssertIntsEqual(t, "fence consultations", 1, len(operations)) + testutils.AssertStringsEqual( + t, + "fence operation", + "tbtc_dkg_signer_activation", + operations[0], + ) + + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 1, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "activated signers", + 1, + len(de.walletRegistry.getSigners(result.PrivateKeyShare.PublicKey())), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 0, + len(quarantineHandle.saved), + ) + + terminalOutcomes := permit.recordedTerminalOutcomes() + testutils.AssertIntsEqual( + t, + "terminal outcomes", + 1, + len(terminalOutcomes), + ) + if len(terminalOutcomes) == 1 { + if terminalOutcomes[0].outcome != + participation.TerminalOutcomeCompleted { + t.Errorf( + "unexpected terminal outcome [%s]", + terminalOutcomes[0].outcome, + ) + } + if terminalOutcomes[0].evidence.MembershipIndex != + group.MemberIndex(1) { + t.Errorf( + "terminal outcome names membership [%d], expected [1]", + terminalOutcomes[0].evidence.MembershipIndex, + ) + } + } +} + +// TestDkgExecutor_CompleteDkgCeremony_RegistrationFailureQuarantinesOnly +// proves a registration failure between the concluded result publication and +// the wallet-cache activation preserves the generated share only in the +// protected quarantine namespace. The wallet ID calculation is the last +// fallible registration step, and its failure must not leave a partial record +// in the active namespace that a restart's — or any release's — active scan +// would load beside the quarantined copy. +func TestDkgExecutor_CompleteDkgCeremony_RegistrationFailureQuarantinesOnly( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + // The registry's wallet ID calculation fails while the chain's own + // calculation keeps succeeding, so the preservation path can still check + // the wallet's on-chain registration and choose quarantine. + failingRegistry, err := newWalletRegistry( + registryHandle, + func(*ecdsa.PublicKey) ([32]byte, error) { + return [32]byte{}, fmt.Errorf("wallet ID calculation failed") + }, + ) + if err != nil { + t.Fatal(err) + } + de.walletRegistry = failingRegistry + + permit := newTestPermit(participation.TBTCDKG) + + published := false + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(context.Context) error { + published = true + return nil + }, + ) + + if !published { + t.Fatal("expected the result publication to run") + } + if activated { + t.Fatal("expected no signer activation") + } + + operations := permit.commitOperations() + testutils.AssertIntsEqual(t, "fence consultations", 1, len(operations)) + + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "activated signers", + 0, + len(de.walletRegistry.getSigners(result.PrivateKeyShare.PublicKey())), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertStringsEqual( + t, + "quarantined failed operation", + "tbtc_dkg_signer_registration", + quarantinedFailedOperation(t, quarantineHandle), + ) +} + +// TestDkgExecutor_CompleteDkgCeremony_PublicationGateRefusalQuarantines +// proves a submission fence refusal during result publication preserves the +// generated share only in the protected quarantine namespace: the activation +// fence is never consulted and the signer is neither saved to the active +// namespace nor activated in the wallet cache. +func TestDkgExecutor_CompleteDkgCeremony_PublicationGateRefusalQuarantines( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(context.Context) error { + return fmt.Errorf( + "completion commit refused: %w", + participation.ErrClockUnavailable, + ) + }, + ) + + if activated { + t.Fatal("expected no signer activation") + } + testutils.AssertIntsEqual( + t, + "fence consultations", + 0, + len(permit.commitOperations()), + ) + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "activated signers", + 0, + len(de.walletRegistry.getSigners(result.PrivateKeyShare.PublicKey())), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertStringsEqual( + t, + "quarantined failed operation", + "tbtc_dkg_result_publication", + quarantinedFailedOperation(t, quarantineHandle), + ) +} + +// TestDkgExecutor_CompleteDkgCeremony_ClockLossDuringPublicationQuarantines +// proves a clock-failure permit cancellation racing the result publication — +// the publication itself surfaces only a plain context cancellation, the gate +// cause lives in the permit context — preserves the share only in quarantine +// and never activates it. +func TestDkgExecutor_CompleteDkgCeremony_ClockLossDuringPublicationQuarantines( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(publishCtx context.Context) error { + // The gate loses the chain clock while the publication is in + // flight: the permit is canceled with the gate cause and the + // publication ends with a plain context cancellation. + permit.cancel(participation.ErrClockUnavailable) + <-publishCtx.Done() + return publishCtx.Err() + }, + ) + + if activated { + t.Fatal("expected no signer activation") + } + testutils.AssertIntsEqual( + t, + "fence consultations", + 0, + len(permit.commitOperations()), + ) + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "activated signers", + 0, + len(de.walletRegistry.getSigners(result.PrivateKeyShare.PublicKey())), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertStringsEqual( + t, + "quarantined failed operation", + "tbtc_dkg_result_publication", + quarantinedFailedOperation(t, quarantineHandle), + ) +} + +// TestDkgExecutor_CompleteDkgCeremony_ActivatesWhenAnotherMemberSubmitted +// proves a publication ended by the on-chain submission event — another +// member submitted the result first — still activates the signer through the +// activation fence: the ceremony completed and the share is needed. +func TestDkgExecutor_CompleteDkgCeremony_ActivatesWhenAnotherMemberSubmitted( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return true }, + func(context.Context) error { + return context.Canceled + }, + ) + + if !activated { + t.Fatal("expected the signer to be activated") + } + operations := permit.commitOperations() + testutils.AssertIntsEqual(t, "fence consultations", 1, len(operations)) + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 1, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "activated signers", + 1, + len(de.walletRegistry.getSigners(result.PrivateKeyShare.PublicKey())), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 0, + len(quarantineHandle.saved), + ) +} + +// TestDkgExecutor_CompleteDkgCeremony_TimeoutWithoutSubmissionQuarantines +// proves a publication window that closes without any observed submitted +// result preserves the share only in quarantine: the wallet may never appear +// on chain, so activating the signer would leave an active signer for an +// unpublished result. +func TestDkgExecutor_CompleteDkgCeremony_TimeoutWithoutSubmissionQuarantines( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(context.Context) error { + return context.Canceled + }, + ) + + if activated { + t.Fatal("expected no signer activation") + } + testutils.AssertIntsEqual( + t, + "fence consultations", + 0, + len(permit.commitOperations()), + ) + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertStringsEqual( + t, + "quarantined failed operation", + "tbtc_dkg_result_publication", + quarantinedFailedOperation(t, quarantineHandle), + ) +} + +// TestDkgExecutor_CompleteDkgCeremony_ActivationFenceRefusalQuarantines +// proves a refused activation fence after a successful publication preserves +// the share only in quarantine when the wallet is not yet registered on +// chain, and never activates it in this process. +func TestDkgExecutor_CompleteDkgCeremony_ActivationFenceRefusalQuarantines( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + permit := newTestPermit(participation.TBTCDKG) + permit.commitErr = participation.ErrQuiesceDeadline + + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(context.Context) error { + return nil + }, + ) + + if activated { + t.Fatal("expected no signer activation") + } + operations := permit.commitOperations() + testutils.AssertIntsEqual(t, "fence consultations", 1, len(operations)) + testutils.AssertStringsEqual( + t, + "fence operation", + "tbtc_dkg_signer_activation", + operations[0], + ) + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertStringsEqual( + t, + "quarantined failed operation", + "tbtc_dkg_signer_activation", + quarantinedFailedOperation(t, quarantineHandle), + ) +} + +// TestDkgExecutor_CompleteDkgCeremony_QuiesceDeadlineRaceWithRealGate proves +// the deterministic forced-quiescence race against a real gate: the process +// shutdown deadline arrives while the result publication is in flight, the +// permit is force-canceled with the gate cause, and the generated share ends +// up only in quarantine — never active, never dropped. +func TestDkgExecutor_CompleteDkgCeremony_QuiesceDeadlineRaceWithRealGate( + t *testing.T, +) { + de, result, gsr, registryHandle, quarantineHandle := setupPreserveScenario(t) + + blockCounter, err := de.chain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + de.participationGate = gate + + permit, err := gate.Begin(participation.TBTCDKG, 1) + if err != nil { + t.Fatal(err) + } + + activated := de.completeDkgCeremony( + permit.Context(), + logger.With(), + permit, + big.NewInt(1), + result, + group.MemberIndex(1), + gsr, + func() bool { return false }, + func(publishCtx context.Context) error { + // The shutdown deadline arrives mid-publication: Close + // force-cancels the permit with the gate cause and the + // publication ends with a plain context cancellation. + gate.Close() + <-publishCtx.Done() + return publishCtx.Err() + }, + ) + + if activated { + t.Fatal("expected no signer activation") + } + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(registryHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "activated signers", + 0, + len(de.walletRegistry.getSigners(result.PrivateKeyShare.PublicKey())), + ) + testutils.AssertIntsEqual( + t, + "quarantined records", + 2, + len(quarantineHandle.saved), + ) + testutils.AssertStringsEqual( + t, + "quarantined failed operation", + "tbtc_dkg_result_publication", + quarantinedFailedOperation(t, quarantineHandle), + ) +} + +// TestSigningExecutor_Sign_GateCancellationSkipsFailureMetrics proves a +// gate-caused signing cancellation — clock failure carried as the context +// cause — leaves the ordinary signing failure and timeout counters unchanged +// and surfaces the gate sentinel, while an ordinary cancellation still counts +// as a failure and a timeout. +func TestSigningExecutor_Sign_GateCancellationSkipsFailureMetrics(t *testing.T) { + executor := setupSigningExecutor(t) + + recorder := newDispatcherMetricsRecorder() + executor.setMetricsRecorder(recorder) + + gateCtx, cancelGateCtx := context.WithCancelCause(context.Background()) + cancelGateCtx(participation.ErrClockUnavailable) + + _, err := executor.sign( + gateCtx, + big.NewInt(100), + 0, + participation.ModeSecurityV2, + ) + if !errors.Is(err, participation.ErrClockUnavailable) { + t.Fatalf("expected the gate sentinel, got [%v]", err) + } + + if failed := recorder.counter(clientinfo.MetricSigningFailedTotal); failed != 0 { + t.Errorf("expected no ordinary signing failures, got [%v]", failed) + } + if timeouts := recorder.counter(clientinfo.MetricSigningTimeoutsTotal); timeouts != 0 { + t.Errorf("expected no ordinary signing timeouts, got [%v]", timeouts) + } + + // An ordinary cancellation without a gate cause still counts as an + // ordinary failure and timeout. + plainCtx, cancelPlainCtx := context.WithCancel(context.Background()) + cancelPlainCtx() + + _, err = executor.sign( + plainCtx, + big.NewInt(101), + 0, + participation.ModeSecurityV2, + ) + if err == nil { + t.Fatal("expected an error from the canceled signing") + } + if errors.Is(err, participation.ErrClockUnavailable) { + t.Fatalf("expected no gate sentinel, got [%v]", err) + } + + if failed := recorder.counter(clientinfo.MetricSigningFailedTotal); failed != 1 { + t.Errorf("expected one ordinary signing failure, got [%v]", failed) + } + if timeouts := recorder.counter(clientinfo.MetricSigningTimeoutsTotal); timeouts != 1 { + t.Errorf("expected one ordinary signing timeout, got [%v]", timeouts) + } +} + +// dispatchGaugeRecorder extends the counting recorder with gauge capture so +// tests can wait for the dispatcher's active-actions gauge to return to zero +// — the gauge is reset only after an action's goroutine finished all its +// metric accounting. +type dispatchGaugeRecorder struct { + *dispatcherMetricsRecorder + + gaugeMu sync.Mutex + gauges map[string]float64 +} + +func newDispatchGaugeRecorder() *dispatchGaugeRecorder { + return &dispatchGaugeRecorder{ + dispatcherMetricsRecorder: newDispatcherMetricsRecorder(), + gauges: make(map[string]float64), + } +} + +func (r *dispatchGaugeRecorder) SetGauge(name string, value float64) { + r.gaugeMu.Lock() + defer r.gaugeMu.Unlock() + r.gauges[name] = value +} + +func (r *dispatchGaugeRecorder) gauge(name string) float64 { + r.gaugeMu.Lock() + defer r.gaugeMu.Unlock() + return r.gauges[name] +} + +// gaugePublished reports the value alongside whether the gauge was published at +// all. A gauge nobody set reads back as zero, which is the one value a +// quarantine count must never be confused with. +func (r *dispatchGaugeRecorder) gaugePublished(name string) (float64, bool) { + r.gaugeMu.Lock() + defer r.gaugeMu.Unlock() + value, published := r.gauges[name] + return value, published +} + +// TestWalletDispatcher_Dispatch_GateRefusalSkipsFailureMetrics proves a +// wallet action ended by a gate refusal is counted neither as an ordinary +// action failure nor as a success, on both the aggregate and the per-action +// counters, while an ordinary action error still counts as a failure. +func TestWalletDispatcher_Dispatch_GateRefusalSkipsFailureMetrics(t *testing.T) { + walletDispatcher := newWalletDispatcher() + recorder := newDispatchGaugeRecorder() + walletDispatcher.setMetricsRecorder(recorder) + + actionWallet := generateWallet(big.NewInt(100)) + + dispatchAndWait := func(action *mockWalletAction) { + t.Helper() + + if err := walletDispatcher.dispatch(action); err != nil { + t.Fatal(err) + } + // The active-actions gauge returns to zero only in the action + // goroutine's final cleanup, after every counter update. + deadline := time.Now().Add(10 * time.Second) + for recorder.gauge(clientinfo.MetricWalletDispatcherActiveActions) != 0 { + if time.Now().After(deadline) { + t.Fatal("the dispatched action never completed") + } + time.Sleep(time.Millisecond) + } + } + + dispatchAndWait(&mockWalletAction{ + executeFn: func() error { + return fmt.Errorf( + "broadcast refused: %w", + participation.ErrQuiesceDeadline, + ) + }, + actionWallet: actionWallet, + }) + + failedName := clientinfo.WalletActionMetricName("noop", "failed_total") + successName := clientinfo.WalletActionMetricName("noop", "success_total") + + if failed := recorder.counter(clientinfo.MetricWalletActionFailedTotal); failed != 0 { + t.Errorf("expected no aggregate action failures, got [%v]", failed) + } + if failed := recorder.counter(failedName); failed != 0 { + t.Errorf("expected no per-action failures, got [%v]", failed) + } + if success := recorder.counter(successName); success != 0 { + t.Errorf("expected no action successes, got [%v]", success) + } + + dispatchAndWait(&mockWalletAction{ + executeFn: func() error { + return fmt.Errorf("ordinary failure") + }, + actionWallet: actionWallet, + }) + + if failed := recorder.counter(clientinfo.MetricWalletActionFailedTotal); failed != 1 { + t.Errorf("expected one aggregate action failure, got [%v]", failed) + } + if failed := recorder.counter(failedName); failed != 1 { + t.Errorf("expected one per-action failure, got [%v]", failed) + } + if success := recorder.counter(successName); success != 0 { + t.Errorf("expected no action successes, got [%v]", success) + } +} + +// TestWalletTransactionExecutor_BroadcastAbortSurfacesGateCause proves an +// ended broadcast window caused by a gate permit cancellation surfaces the +// gate sentinel instead of the ordinary broadcast timeout. +func TestWalletTransactionExecutor_BroadcastAbortSurfacesGateCause(t *testing.T) { + permit := newTestPermit(participation.TBTCSigning) + permit.cancel(participation.ErrClockUnavailable) + + wte := &walletTransactionExecutor{ + btcChain: newLocalBitcoinChain(), + permit: permit, + } + + err := wte.broadcastTransaction( + logger.With(), + &bitcoin.Transaction{Version: 1}, + 10*time.Second, + time.Millisecond, + ) + if !errors.Is(err, participation.ErrClockUnavailable) { + t.Fatalf("expected the gate sentinel, got [%v]", err) + } +} diff --git a/pkg/tbtc/participation_outcome.go b/pkg/tbtc/participation_outcome.go new file mode 100644 index 0000000000..35f6b1cc7e --- /dev/null +++ b/pkg/tbtc/participation_outcome.go @@ -0,0 +1,67 @@ +package tbtc + +import ( + "math/big" + + "github.com/ipfs/go-log/v2" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// recordPermitTerminalOutcome records the node-owned final disposition of a +// ceremony on its participation permit. Only the ceremony owner can author it, +// so this is the single place every tBTC ceremony reports what it durably left +// behind. A failure to record is logged and never propagated: the permit still +// closes and the gate falls back to the unresolved marker, which fails the +// offline rollback barrier closed rather than inventing an outcome. +func recordPermitTerminalOutcome( + actionLogger log.StandardLogger, + permit participation.Permit, + outcome participation.TerminalOutcome, + evidence participation.TerminalEvidence, +) { + if permit == nil { + return + } + + if err := permit.RecordTerminalOutcome(outcome, evidence); err != nil { + actionLogger.Warnf( + "could not persist the node-authored terminal outcome "+ + "[ceremony=%s] [permit=%s] [outcome=%s]: [%v]", + permit.Ceremony(), + permit.PermitID(), + outcome, + err, + ) + } +} + +// recordPermitNoThreshold records that a ceremony ended without producing a +// threshold result or any durable state transition this node owns. It is the +// honest disposition for a ceremony that never got past its protocol steps, +// including one the release gate canceled: no key material, no signed +// transaction, and no chain submission was left behind. +func recordPermitNoThreshold( + actionLogger log.StandardLogger, + permit participation.Permit, +) { + recordPermitTerminalOutcome( + actionLogger, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceNoThreshold, + }, + ) +} + +// signatureComponentBytes returns the big-endian bytes of a tECDSA signature +// component. The signature type admits nil components — its own equality check +// compares them — so the terminal recorder, which runs deferred on the action's +// exit path, must not dereference one. +func signatureComponentBytes(component *big.Int) []byte { + if component == nil { + return nil + } + + return component.Bytes() +} diff --git a/pkg/tbtc/participation_outcome_test.go b/pkg/tbtc/participation_outcome_test.go new file mode 100644 index 0000000000..a30261069f --- /dev/null +++ b/pkg/tbtc/participation_outcome_test.go @@ -0,0 +1,891 @@ +package tbtc + +import ( + "bytes" + "context" + "crypto/ecdsa" + "crypto/rand" + "encoding/hex" + "errors" + "math/big" + "testing" + "time" + + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/tecdsa" +) + +// assertRecordedTerminalOutcome checks that the ceremony owner authored exactly +// one terminal disposition of the expected shape and that the live gate's own +// validator accepts it for that ceremony. Running the authored record through +// the validator is what keeps the journal usable: an outcome the node writes but +// the gate rejects never reaches the rollback audit at all. +func assertRecordedTerminalOutcome( + t *testing.T, + permit *testPermit, + expectedOutcome participation.TerminalOutcome, + expectedKind participation.TerminalEvidenceKind, +) participation.TerminalEvidence { + t.Helper() + + recorded := permit.recordedTerminalOutcomes() + if len(recorded) != 1 { + t.Fatalf( + "expected exactly one terminal outcome, got [%d]", + len(recorded), + ) + } + + if recorded[0].outcome != expectedOutcome { + t.Errorf( + "unexpected terminal outcome\nexpected: [%s]\nactual: [%s]", + expectedOutcome, + recorded[0].outcome, + ) + } + + if recorded[0].evidence.Kind != expectedKind { + t.Errorf( + "unexpected terminal evidence kind\nexpected: [%s]\nactual: [%s]", + expectedKind, + recorded[0].evidence.Kind, + ) + } + + if err := participation.ValidateTerminalOutcome( + permit.Ceremony(), + permit.WorkID(), + recorded[0].outcome, + recorded[0].evidence, + ); err != nil { + t.Errorf( + "the gate rejects the node-authored outcome for ceremony [%s]: [%v]", + permit.Ceremony(), + err, + ) + } + + return recorded[0].evidence +} + +// TestWalletTransactionExecutor_TerminalOutcome_NoSignedTransaction covers a +// wallet action that never reached a signed Bitcoin transaction. Nothing this +// node owns can land on the Bitcoin chain, so the rollback journal records the +// ceremony as exhausted. +func TestWalletTransactionExecutor_TerminalOutcome_NoSignedTransaction(t *testing.T) { + permit := newTestPermit(participation.TBTCSigning) + + executor := &walletTransactionExecutor{ + permit: permit, + btcChain: newLocalBitcoinChain(), + } + + executor.recordTerminalOutcome(&testutils.MockLogger{}) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidenceNoThreshold, + ) + + if evidence.Reference != "" { + t.Errorf( + "expected no evidence reference, got [%s]", + evidence.Reference, + ) + } +} + +// TestWalletTransactionExecutor_TerminalOutcome_SignedTransaction covers a +// wallet action whose signing reached the threshold. The signed transaction is +// the action's durable result and any wallet member may put it on the Bitcoin +// network, so its hash must reach the journal for offline reconciliation. +func TestWalletTransactionExecutor_TerminalOutcome_SignedTransaction(t *testing.T) { + privateKeyScalar := big.NewInt(100) + executingWallet := generateWallet(privateKeyScalar) + + btcChain, transactionBuilder := buildSignTransactionFixture(t, executingWallet) + + sigHashes, err := transactionBuilder.ComputeSignatureHashes() + if err != nil { + t.Fatal(err) + } + + privateKey := &ecdsa.PrivateKey{ + PublicKey: *executingWallet.publicKey, + D: privateKeyScalar, + } + signatures := make([]*tecdsa.Signature, len(sigHashes)) + for i, sigHash := range sigHashes { + r, s, err := ecdsa.Sign(rand.Reader, privateKey, sigHash.Bytes()) + if err != nil { + t.Fatal(err) + } + signatures[i] = &tecdsa.Signature{R: r, S: s} + } + + const startBlock = uint64(0) + signingExecutor := newMockWalletSigningExecutor() + signingExecutor.setSignatures(sigHashes, startBlock, signatures) + + permit := newTestPermit(participation.TBTCSigning) + + executor := &walletTransactionExecutor{ + permit: permit, + btcChain: btcChain, + executingWallet: executingWallet, + signingExecutor: signingExecutor, + waitForBlockFn: func(ctx context.Context, _ uint64) error { + select { + case <-ctx.Done(): + case <-time.After(5 * time.Second): + } + return nil + }, + } + + signedTransaction, err := executor.signTransaction( + &testutils.MockLogger{}, + transactionBuilder, + startBlock, + 1000, + ) + if err != nil { + t.Fatal(err) + } + + // The action is recorded without any broadcast at all: the ceremony's + // durable result is the signed transaction, and whether it reached the + // Bitcoin network is exactly what the offline audit reconciles from this + // hash. + executor.recordTerminalOutcome(&testutils.MockLogger{}) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceBitcoinTransaction, + ) + + expectedReference := signedTransaction.Hash().Hex(bitcoin.ReversedByteOrder) + if evidence.Reference != expectedReference { + t.Errorf( + "unexpected evidence reference\nexpected: [%s]\nactual: [%s]", + expectedReference, + evidence.Reference, + ) + } +} + +// runHeartbeatActionForOutcome executes one heartbeat action against the given +// permit, using a fresh failure counter so no earlier heartbeat can push this +// one over the consecutive-failure threshold. +func runHeartbeatActionForOutcome( + t *testing.T, + hostChain *localChain, + signingExecutor *mockHeartbeatSigningExecutor, + proposal *HeartbeatProposal, + permit participation.Permit, +) { + t.Helper() + + runHeartbeatActionWithFailureCounter( + t, + hostChain, + signingExecutor, + &mockInactivityClaimExecutor{}, + newHeartbeatFailureCounter(), + proposal, + permit, + ) +} + +// runHeartbeatActionWithFailureCounter executes one heartbeat action against a +// caller-owned failure counter and claim executor, so consecutive low-activity +// heartbeats can be driven up to the threshold that dispatches a claim. +func runHeartbeatActionWithFailureCounter( + t *testing.T, + hostChain *localChain, + signingExecutor *mockHeartbeatSigningExecutor, + claimExecutor *mockInactivityClaimExecutor, + failureCounter *heartbeatFailureCounter, + proposal *HeartbeatProposal, + permit participation.Permit, +) { + t.Helper() + + walletPublicKeyBytes, err := hex.DecodeString(heartbeatTestWalletKey()) + if err != nil { + t.Fatal(err) + } + + hostChain.setHeartbeatProposalValidationResult(proposal, true) + + const startBlock = uint64(10) + + action := newHeartbeatAction( + logger, + hostChain, + wallet{ + publicKey: mustUnmarshalPublicKey(t, walletPublicKeyBytes), + }, + signingExecutor, + proposal, + failureCounter, + claimExecutor, + startBlock, + startBlock+heartbeatTotalProposalValidityBlocks, + func(ctx context.Context, blockHeight uint64) error { return nil }, + permit, + ) + + // The action's own error is not the subject here: every exit, successful or + // not, must leave exactly one terminal disposition behind. + _ = action.execute() +} + +// TestHeartbeatAction_TerminalOutcomeBindsDispatchedInactivityClaim asserts a +// heartbeat that went on to file an inactivity claim is distinguishable in the +// journal from a healthy one. The claim runs under the heartbeat's own permit +// and leaves no separate record, so a reference naming only the signature would +// let the audit clear penalty state it never saw. +func TestHeartbeatAction_TerminalOutcomeBindsDispatchedInactivityClaim( + t *testing.T, +) { + proposal := &HeartbeatProposal{ + Message: [16]byte{ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + }, + } + + // runToClaimThreshold drives consecutive low-activity heartbeats until the + // claim threshold is crossed and returns the last heartbeat's evidence + // reference together with the number of claims that were dispatched. + runToClaimThreshold := func( + t *testing.T, + activeMembers uint32, + runs int, + ) (string, int) { + t.Helper() + + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + + claimExecutor := &mockInactivityClaimExecutor{} + failureCounter := newHeartbeatFailureCounter() + + var reference string + for run := 0; run < runs; run++ { + permit := newTestPermit(participation.TBTCHeartbeat) + + runHeartbeatActionWithFailureCounter( + t, + hostChain, + &mockHeartbeatSigningExecutor{ + activeOperatorsCount: activeMembers, + }, + claimExecutor, + failureCounter, + proposal, + permit, + ) + + reference = assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ).Reference + } + + return reference, claimExecutor.calls + } + + claimed, claimCalls := runToClaimThreshold( + t, + heartbeatSigningMinimumActiveMembers-1, + heartbeatConsecutiveFailureThreshold, + ) + if claimCalls != 1 { + t.Fatalf( + "expected exactly one dispatched inactivity claim, got [%d]", + claimCalls, + ) + } + + healthy, healthyClaimCalls := runToClaimThreshold( + t, + heartbeatSigningMinimumActiveMembers, + heartbeatConsecutiveFailureThreshold, + ) + if healthyClaimCalls != 0 { + t.Fatalf( + "expected no dispatched inactivity claim, got [%d]", + healthyClaimCalls, + ) + } + + if claimed == healthy { + t.Errorf( + "a heartbeat that filed an inactivity claim and one that did not "+ + "produced the same evidence reference [%s]", + claimed, + ) + } +} + +// TestHeartbeatAction_TerminalOutcomeReportsObservedClaimSettlement drives a +// heartbeat past the claim threshold and asserts the terminal record describes +// what the chain did, not what the action tried to do. The claim is filed on +// Ethereum under the heartbeat's own permit, so a record that treats the +// dispatch call as the settlement would let a penalty that never landed — or +// one that landed while the call errored — clear the rollback journal. +func TestHeartbeatAction_TerminalOutcomeReportsObservedClaimSettlement( + t *testing.T, +) { + proposal := &HeartbeatProposal{ + Message: [16]byte{ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, + }, + } + + walletID := [32]byte{ + 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff, 0x00, 0x11, + 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, + 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff, 0x00, 0x11, + 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, + } + settledReference, err := participation.InactivityClaimSettlementReference( + walletID[:], + big.NewInt(41), + ) + if err != nil { + t.Fatal(err) + } + + settled := &inactivityClaimSettlement{ + walletID: walletID, + nonce: big.NewInt(41), + } + + tests := map[string]struct { + activeMembers uint32 + claimFails bool + disposition inactivityClaimDisposition + expectedSettlement bool + expectedReference string + }{ + "a healthy heartbeat dispatches nothing": { + activeMembers: heartbeatSigningMinimumActiveMembers, + }, + "a dispatch resolved to a settlement names the claim": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + disposition: inactivityClaimDisposition{ + submissionAttempted: true, + settlement: settled, + }, + expectedSettlement: true, + expectedReference: settledReference, + }, + // A submission whose settlement stayed unresolved is the one genuinely + // ambiguous case: the penalty may be on chain and the barrier has to + // block on it. + "a submitted claim with no resolved settlement names none": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + disposition: inactivityClaimDisposition{ + submissionAttempted: true, + }, + expectedSettlement: true, + }, + // Nothing reached the chain, so there is no chain state to reconcile + // and the record must not manufacture an ambiguity. + "a dispatch that never reached the chain reports no settlement": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + }, + // Another member's submission settling the claim is still this + // permit's penalty, whether or not a controlled member submitted. + "a settlement resolved without a local submission names the claim": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + disposition: inactivityClaimDisposition{ + settlement: settled, + }, + expectedSettlement: true, + expectedReference: settledReference, + }, + "a failed dispatch that still settled names the claim": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + claimFails: true, + disposition: inactivityClaimDisposition{ + submissionAttempted: true, + settlement: settled, + }, + expectedSettlement: true, + expectedReference: settledReference, + }, + "a failed dispatch that never reached the chain reports no settlement": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + claimFails: true, + }, + } + + for name, test := range tests { + t.Run(name, func(t *testing.T) { + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + + claimExecutor := &mockInactivityClaimExecutor{ + shouldFail: test.claimFails, + disposition: test.disposition, + } + failureCounter := newHeartbeatFailureCounter() + + // The claim is only dispatched once consecutive low-activity + // heartbeats cross the threshold, so the disposition under test is + // the last run's. + var permit *testPermit + for run := 0; run < heartbeatConsecutiveFailureThreshold; run++ { + permit = newTestPermit(participation.TBTCHeartbeat) + + runHeartbeatActionWithFailureCounter( + t, + hostChain, + &mockHeartbeatSigningExecutor{ + activeOperatorsCount: test.activeMembers, + }, + claimExecutor, + failureCounter, + proposal, + permit, + ) + } + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ) + + if !test.expectedSettlement { + if evidence.ChainSettlement != nil { + t.Fatalf( + "a heartbeat that left nothing on chain reported "+ + "settlement [%+v]", + evidence.ChainSettlement, + ) + } + return + } + + if evidence.ChainSettlement == nil { + t.Fatal( + "a heartbeat whose claim may have reached the chain " + + "reported no chain settlement", + ) + } + if evidence.ChainSettlement.Kind != + participation.ChainSettlementInactivityClaim { + t.Errorf( + "unexpected settlement kind\nexpected: [%s]\nactual: [%s]", + participation.ChainSettlementInactivityClaim, + evidence.ChainSettlement.Kind, + ) + } + if evidence.ChainSettlement.Reference != test.expectedReference { + t.Errorf( + "unexpected settlement reference\nexpected: [%s]\nactual: [%s]", + test.expectedReference, + evidence.ChainSettlement.Reference, + ) + } + }) + } +} + +// TestHeartbeatPenaltyState_UnrenderableSettlementStaysUnobserved asserts a +// settlement the node cannot name canonically is reported as unobserved. The +// two records mean different things to the barrier — unreconciled versus +// reconciled against a named claim — and only the unreconciled one is safe +// when the identity is unusable. +func TestHeartbeatPenaltyState_UnrenderableSettlementStaysUnobserved( + t *testing.T, +) { + penalty := heartbeatPenaltyState{ + claimDispatched: true, + inactiveMembers: []group.MemberIndex{1, 2}, + claim: inactivityClaimDisposition{ + submissionAttempted: true, + settlement: &inactivityClaimSettlement{ + walletID: [32]byte{0x01}, + // A claim the chain never assigns a nonce to cannot be joined + // to any InactivityClaimed log. + nonce: nil, + }, + }, + } + + settlement := penalty.chainSettlement() + if settlement == nil { + t.Fatal("expected the submission itself to still be reported") + } + if settlement.Reference != "" { + t.Errorf( + "expected no settlement reference, got [%s]", + settlement.Reference, + ) + } +} + +// TestHeartbeatPenaltyState_InactiveMemberBytesAreCanonical asserts the claimed +// member set contributes a deterministic identity: the signing activity report's +// ordering is incidental, so two records of the same claim must agree. +func TestHeartbeatPenaltyState_InactiveMemberBytesAreCanonical(t *testing.T) { + ordered := heartbeatPenaltyState{ + claimDispatched: true, + inactiveMembers: []group.MemberIndex{3, 9, 14}, + } + shuffled := heartbeatPenaltyState{ + claimDispatched: true, + inactiveMembers: []group.MemberIndex{14, 3, 9, 3}, + } + + if !bytes.Equal(ordered.inactiveMemberBytes(), shuffled.inactiveMemberBytes()) { + t.Errorf( + "one claimed member set produced two identities [%x] and [%x]", + ordered.inactiveMemberBytes(), + shuffled.inactiveMemberBytes(), + ) + } + + disjoint := heartbeatPenaltyState{ + claimDispatched: true, + inactiveMembers: []group.MemberIndex{3, 9, 15}, + } + if bytes.Equal(ordered.inactiveMemberBytes(), disjoint.inactiveMemberBytes()) { + t.Errorf( + "two different claimed member sets produced the same identity [%x]", + ordered.inactiveMemberBytes(), + ) + } + + empty := heartbeatPenaltyState{} + if empty.inactiveMemberBytes() != nil { + t.Errorf( + "a heartbeat that dispatched no claim named members [%x]", + empty.inactiveMemberBytes(), + ) + } +} + +// TestHeartbeatAction_TerminalOutcome walks every exit of the heartbeat action +// and asserts the disposition it leaves in the rollback journal. The heartbeat's +// durable result is the threshold signature; the inactivity accounting that +// follows a low-activity signing does not change that. +func TestHeartbeatAction_TerminalOutcome(t *testing.T) { + tests := map[string]struct { + operatorUnstaking bool + signingFails bool + activeMembers uint32 + expectedOutcome participation.TerminalOutcome + expectedKind participation.TerminalEvidenceKind + }{ + "signing produced a signature": { + activeMembers: heartbeatSigningMinimumActiveMembers, + expectedOutcome: participation.TerminalOutcomeCompleted, + expectedKind: participation.TerminalEvidenceProtocolResult, + }, + "signing produced a signature below the activity threshold": { + activeMembers: heartbeatSigningMinimumActiveMembers - 1, + expectedOutcome: participation.TerminalOutcomeCompleted, + expectedKind: participation.TerminalEvidenceProtocolResult, + }, + "signing failed": { + signingFails: true, + expectedOutcome: participation.TerminalOutcomeExhausted, + expectedKind: participation.TerminalEvidenceNoThreshold, + }, + "operator is unstaking": { + operatorUnstaking: true, + activeMembers: heartbeatSigningMinimumActiveMembers, + expectedOutcome: participation.TerminalOutcomeExhausted, + expectedKind: participation.TerminalEvidenceNoThreshold, + }, + } + + for testName, test := range tests { + t.Run(testName, func(t *testing.T) { + hostChain := Connect() + if test.operatorUnstaking { + hostChain.setOperatorsEligibleStake(big.NewInt(0)) + } else { + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + } + + permit := newTestPermit(participation.TBTCHeartbeat) + signingExecutor := &mockHeartbeatSigningExecutor{ + shouldFail: test.signingFails, + activeOperatorsCount: test.activeMembers, + } + + runHeartbeatActionForOutcome( + t, + hostChain, + signingExecutor, + &HeartbeatProposal{ + Message: [16]byte{ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, + }, + }, + permit, + ) + + if test.operatorUnstaking && + signingExecutor.requestedMessage != nil { + t.Error("an unstaking operator must not sign") + } + + assertRecordedTerminalOutcome( + t, + permit, + test.expectedOutcome, + test.expectedKind, + ) + }) + } +} + +// TestHeartbeatAction_TerminalOutcomeBindsResultToProposal asserts the recorded +// evidence identifies the exact heartbeat that ran. Two heartbeats of the same +// wallet must not be interchangeable in the journal, otherwise the audit cannot +// tell which proposal a recorded result belongs to. +func TestHeartbeatAction_TerminalOutcomeBindsResultToProposal(t *testing.T) { + hostChain := Connect() + hostChain.setOperatorsEligibleStake(big.NewInt(100000)) + + references := make([]string, 0, 2) + for _, message := range [][16]byte{ + {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}, + {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xfe}, + } { + permit := newTestPermit(participation.TBTCHeartbeat) + + runHeartbeatActionForOutcome( + t, + hostChain, + &mockHeartbeatSigningExecutor{ + activeOperatorsCount: heartbeatSigningMinimumActiveMembers, + }, + &HeartbeatProposal{Message: message}, + permit, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ) + references = append(references, evidence.Reference) + } + + if references[0] == references[1] { + t.Errorf( + "two distinct heartbeat proposals produced the same evidence "+ + "reference [%s]", + references[0], + ) + } +} + +// TestCoordinationTerminalOutcome covers the wallet coordination procedure's +// disposition. The procedure's durable result is the proposal the wallet agreed +// on; the wallet action it dispatches runs under its own permit and reports its +// own outcome. +func TestCoordinationTerminalOutcome(t *testing.T) { + walletPublicKeyBytes, err := hex.DecodeString(heartbeatTestWalletKey()) + if err != nil { + t.Fatal(err) + } + + t.Run("no agreed result", func(t *testing.T) { + permit := newTestPermit(participation.TBTCWalletCoordination) + + recordCoordinationTerminalOutcome( + &testutils.MockLogger{}, + permit, + walletPublicKeyBytes, + nil, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeExhausted, + participation.TerminalEvidenceNoThreshold, + ) + + if evidence.Reference != "" { + t.Errorf( + "expected no evidence reference, got [%s]", + evidence.Reference, + ) + } + }) + + t.Run("distinct windows produce distinct references", func(t *testing.T) { + references := make([]string, 0, 2) + for _, coordinationBlock := range []uint64{900, 1800} { + permit := newTestPermit(participation.TBTCWalletCoordination) + + recordCoordinationTerminalOutcome( + &testutils.MockLogger{}, + permit, + walletPublicKeyBytes, + &coordinationResult{ + window: newCoordinationWindow(coordinationBlock), + leader: chain.Address("0xleader"), + proposal: &HeartbeatProposal{}, + }, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ) + references = append(references, evidence.Reference) + } + + if references[0] == references[1] { + t.Errorf( + "two coordination windows produced the same evidence "+ + "reference [%s]", + references[0], + ) + } + }) + + // A wallet can agree on two different redemptions of the same action type + // in the same window across a restart. An identity that named only the + // action type would report both as the same durable result, so the offline + // audit could clear one window's journal entry with the other's settlement. + t.Run("distinct proposals of one action type produce distinct references", func(t *testing.T) { + proposals := []CoordinationProposal{ + &RedemptionProposal{ + RedeemersOutputScripts: []bitcoin.Script{{0x01}}, + RedemptionTxFee: big.NewInt(1000), + }, + &RedemptionProposal{ + RedeemersOutputScripts: []bitcoin.Script{{0x02}}, + RedemptionTxFee: big.NewInt(1000), + }, + } + + references := make([]string, 0, len(proposals)) + for _, proposal := range proposals { + permit := newTestPermit(participation.TBTCWalletCoordination) + + recordCoordinationTerminalOutcome( + &testutils.MockLogger{}, + permit, + walletPublicKeyBytes, + &coordinationResult{ + window: newCoordinationWindow(900), + leader: chain.Address("0xleader"), + proposal: proposal, + }, + ) + + evidence := assertRecordedTerminalOutcome( + t, + permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidenceProtocolResult, + ) + references = append(references, evidence.Reference) + } + + if references[0] == references[1] { + t.Errorf( + "two distinct redemption proposals in one window produced the "+ + "same evidence reference [%s]", + references[0], + ) + } + }) + + // A proposal the node cannot serialize has no faithful identity. Recording + // a weaker reference would hand the audit a result it cannot pin down, and + // recording exhausted would deny a dispatched wallet action, so the permit + // must be left to close unresolved and block the offline barrier. + t.Run("unserializable proposal records nothing", func(t *testing.T) { + permit := newTestPermit(participation.TBTCWalletCoordination) + + recordCoordinationTerminalOutcome( + &testutils.MockLogger{}, + permit, + walletPublicKeyBytes, + &coordinationResult{ + window: newCoordinationWindow(900), + leader: chain.Address("0xleader"), + proposal: &unmarshalableProposal{}, + }, + ) + + if recorded := permit.recordedTerminalOutcomes(); len(recorded) != 0 { + t.Errorf( + "expected no terminal outcome for an unidentifiable result, "+ + "got [%+v]", + recorded, + ) + } + }) +} + +// unmarshalableProposal stands in for a proposal whose serialization fails. +type unmarshalableProposal struct{} + +func (up *unmarshalableProposal) ActionType() WalletActionType { + return ActionRedemption +} + +func (up *unmarshalableProposal) ValidityBlocks() uint64 { + return 0 +} + +func (up *unmarshalableProposal) Marshal() ([]byte, error) { + return nil, errors.New("proposal cannot be serialized") +} + +func (up *unmarshalableProposal) Unmarshal([]byte) error { + return errors.New("proposal cannot be deserialized") +} + +// TestRecordPermitTerminalOutcome_NilPermit asserts the recorder tolerates a nil +// permit. It runs deferred on the action's exit path, so a panic there would +// take down the action's goroutine after the work already finished. +func TestRecordPermitTerminalOutcome_NilPermit(t *testing.T) { + recordPermitNoThreshold(&testutils.MockLogger{}, nil) +} + +// TestSignatureComponentBytes covers the nil signature component the tECDSA +// signature type admits. +func TestSignatureComponentBytes(t *testing.T) { + if bytes := signatureComponentBytes(nil); bytes != nil { + t.Errorf("expected nil bytes for a nil component, got [%v]", bytes) + } + + bytes := signatureComponentBytes(big.NewInt(258)) + if len(bytes) != 2 || bytes[0] != 0x01 || bytes[1] != 0x02 { + t.Errorf("unexpected component bytes [%v]", bytes) + } +} diff --git a/pkg/tbtc/participation_permit_test.go b/pkg/tbtc/participation_permit_test.go new file mode 100644 index 0000000000..6f683ae667 --- /dev/null +++ b/pkg/tbtc/participation_permit_test.go @@ -0,0 +1,165 @@ +package tbtc + +import ( + "context" + "sync" + "testing" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// testGateMetrics is a no-op metrics sink for test participation gates. +type testGateMetrics struct{} + +func (testGateMetrics) IncrementCounter(string, float64) {} +func (testGateMetrics) SetGauge(string, float64) {} + +// newTestGate constructs a real participation gate over the given block +// counter with an already-crossed cutover block, so every permit with a +// nonzero anchor pins the security-v2 mode — the only mode the tECDSA stack +// can run. +func newTestGate( + t *testing.T, + blockCounter chain.BlockCounter, +) participation.Gate { + t.Helper() + + return newTestGateWithCutover(t, blockCounter, 1) +} + +// newTestGateWithCutover constructs a real participation gate over the given +// block counter with the given cutover block, letting boundary tests choose +// the protocol mode a permit anchor resolves to. +func newTestGateWithCutover( + t *testing.T, + blockCounter chain.BlockCounter, + cutoverBlock uint64, +) participation.Gate { + t.Helper() + + gate, err := participation.NewGate( + context.Background(), + participation.Schedule{CutoverBlock: cutoverBlock}, + blockCounter, + testGateMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(gate.Close) + + return gate +} + +// testPermit is a minimal participation.Permit for exercising wallet actions +// and executors without a running gate: it pins the security-v2 mode, keeps +// an always-open (or preset failing) commit fence, and records commit +// operations and closure for assertions. +type testPermit struct { + ctx context.Context + cancel context.CancelCauseFunc + + mode participation.ProtocolMode + ceremony participation.Ceremony + anchor uint64 + + // commitErr, when set, is returned by every CheckCommit call, modeling a + // refused fence. + commitErr error + + mu sync.Mutex + commits []string + terminalOutcomes []testTerminalOutcome + closed bool +} + +type testTerminalOutcome struct { + outcome participation.TerminalOutcome + evidence participation.TerminalEvidence +} + +func newTestPermit(ceremony participation.Ceremony) *testPermit { + ctx, cancel := context.WithCancelCause(context.Background()) + + return &testPermit{ + ctx: ctx, + cancel: cancel, + mode: participation.ModeSecurityV2, + ceremony: ceremony, + anchor: 1, + } +} + +func (tp *testPermit) Context() context.Context { return tp.ctx } + +func (tp *testPermit) Ceremony() participation.Ceremony { return tp.ceremony } + +func (tp *testPermit) CanonicalStartBlock() uint64 { return tp.anchor } + +func (tp *testPermit) Mode() participation.ProtocolMode { return tp.mode } + +func (tp *testPermit) WorkID() string { return "test-work" } + +func (tp *testPermit) PermitID() string { return "test-permit" } + +func (tp *testPermit) RecordTerminalOutcome( + outcome participation.TerminalOutcome, + evidence participation.TerminalEvidence, +) error { + tp.mu.Lock() + defer tp.mu.Unlock() + + tp.terminalOutcomes = append( + tp.terminalOutcomes, + testTerminalOutcome{outcome: outcome, evidence: evidence}, + ) + + return nil +} + +func (tp *testPermit) CheckCommit( + operation string, + class participation.CommitClass, +) error { + tp.mu.Lock() + defer tp.mu.Unlock() + + tp.commits = append(tp.commits, operation) + + return tp.commitErr +} + +func (tp *testPermit) Close() { + tp.mu.Lock() + defer tp.mu.Unlock() + + if !tp.closed { + tp.closed = true + tp.cancel(participation.ErrPermitClosed) + } +} + +func (tp *testPermit) recordedTerminalOutcomes() []testTerminalOutcome { + tp.mu.Lock() + defer tp.mu.Unlock() + + return append([]testTerminalOutcome(nil), tp.terminalOutcomes...) +} + +func (tp *testPermit) isClosed() bool { + tp.mu.Lock() + defer tp.mu.Unlock() + + return tp.closed +} + +func (tp *testPermit) commitOperations() []string { + tp.mu.Lock() + defer tp.mu.Unlock() + + operations := make([]string, len(tp.commits)) + copy(operations, tp.commits) + + return operations +} diff --git a/pkg/tbtc/participation_test.go b/pkg/tbtc/participation_test.go new file mode 100644 index 0000000000..933b7d0a64 --- /dev/null +++ b/pkg/tbtc/participation_test.go @@ -0,0 +1,85 @@ +package tbtc + +import "testing" + +// TestMaximumLegacyCompletionBlocks pins the derived in-flight completion +// bound. The dominant constituent is the deposit sweep proposal validity. +func TestMaximumLegacyCompletionBlocks(t *testing.T) { + if maximum := MaximumLegacyCompletionBlocks(); maximum != 1200 { + t.Errorf( + "expected maximum legacy completion bound [1200], got [%d]", + maximum, + ) + } +} + +// TestCutoverPeerRosterRetentionBlocks pins the derived roster retention: the +// maximum legacy completion bound plus the reviewed margin. A different value +// means the retention review must be redone deliberately, not that this test +// should be updated casually. +func TestCutoverPeerRosterRetentionBlocks(t *testing.T) { + retention, err := CutoverPeerRosterRetentionBlocks() + if err != nil { + t.Fatalf("unexpected retention derivation error: [%v]", err) + } + if retention != 1500 { + t.Errorf("expected roster retention [1500], got [%d]", retention) + } + if cutoverPeerRosterRetentionMarginBlocks != 300 { + t.Errorf( + "reviewed retention margin changed: expected [300], got [%d]; "+ + "re-review the roster retention derivation", + cutoverPeerRosterRetentionMarginBlocks, + ) + } +} + +// TestMaximumLegacyCompletionBlocksConstituents is a drift test: it fails when +// any constituent protocol constant changes without the completion bound — +// and everything derived from it, such as roster retention and rollback +// quiescence deadlines — being deliberately re-reviewed. +func TestMaximumLegacyCompletionBlocksConstituents(t *testing.T) { + constituents := map[string]struct { + actual uint64 + expected uint64 + }{ + "dkg retry loop": { + uint64(dkgAttemptsLimit) * uint64(dkgAttemptMaximumBlocks()), + 216, + }, + "signing retry loop": { + uint64(signingAttemptsLimit) * uint64(signingAttemptMaximumBlocks()), + 205, + }, + "coordination window": {coordinationDurationBlocks, 100}, + "heartbeat proposal validity": { + heartbeatTotalProposalValidityBlocks, + 600, + }, + "deposit sweep proposal validity": { + depositSweepProposalValidityBlocks, + 1200, + }, + "redemption proposal validity": {redemptionProposalValidityBlocks, 600}, + "moving funds proposal validity": { + movingFundsProposalValidityBlocks, + 650, + }, + "moved funds sweep proposal validity": { + movedFundsSweepProposalValidityBlocks, + 600, + }, + } + + for name, constituent := range constituents { + if constituent.actual != constituent.expected { + t.Errorf( + "%s changed: expected [%d] blocks, got [%d]; re-review the "+ + "maximum legacy completion bound and its derived values", + name, + constituent.expected, + constituent.actual, + ) + } + } +} diff --git a/pkg/tbtc/quarantine.go b/pkg/tbtc/quarantine.go new file mode 100644 index 0000000000..a8fdcb96e5 --- /dev/null +++ b/pkg/tbtc/quarantine.go @@ -0,0 +1,700 @@ +package tbtc + +import ( + "context" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "strconv" + "strings" + "sync" + "time" + + "github.com/ipfs/go-log/v2" + + "github.com/keep-network/keep-common/pkg/persistence" + "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// QuarantineSchemaVersion versions the quarantined-signer metadata document +// for the offline state-audit tooling. +const QuarantineSchemaVersion uint32 = 1 + +// QuarantineHandoffSchemaVersion versions the combined handoff document for the +// offline state-audit tooling. +const QuarantineHandoffSchemaVersion uint32 = 1 + +// QuarantinedSignerHandoff carries one quarantined signer output whole: the key +// material and the audit record that explains it, in a single document written +// with a single save. +// +// The membership and metadata records preservation prefers are two independent +// writes, and a namespace that takes one but refuses the other leaves the output +// split. One of those halves cannot be split off harmlessly: a refused +// membership write leaves an audit record describing a share that reached no +// disk, and the share is the half no ceremony can generate again. This document +// is the form that cannot reach a reader in halves — every field an audit needs +// travels with the material it explains, and a document a crash left half +// written fails the encrypted handle's authentication rather than decoding as +// the part that got through — and it is written under a name of its own, so a +// name the namespace refuses does not decide whether the output survives. +type QuarantinedSignerHandoff struct { + SchemaVersion uint32 `json:"schema_version"` + Metadata QuarantinedSignerMetadata `json:"metadata"` + // Signer is the marshaled signer, byte for byte what the membership record + // holds, so a reader decodes either form the same way. + Signer []byte `json:"signer"` +} + +// DecodeQuarantinedSignerHandoff reads a combined handoff record back into the +// halves it carries, for a later process and for the offline state audit. +// +// A document naming a schema this binary does not know is refused rather than +// read past. The handoff is the only account of an output preserved this way, +// so a reader that guessed at unknown fields would be inventing the evidence a +// rollback decision is made on. +func DecodeQuarantinedSignerHandoff( + recordBytes []byte, +) (*QuarantinedSignerHandoff, error) { + handoff := &QuarantinedSignerHandoff{} + if err := json.Unmarshal(recordBytes, handoff); err != nil { + return nil, fmt.Errorf( + "could not decode the quarantine handoff: [%v]", + err, + ) + } + + if handoff.SchemaVersion != QuarantineHandoffSchemaVersion { + return nil, fmt.Errorf( + "quarantine handoff has schema version [%d], expected [%d]", + handoff.SchemaVersion, + QuarantineHandoffSchemaVersion, + ) + } + + if len(handoff.Signer) == 0 { + return nil, fmt.Errorf( + "quarantine handoff carries no key material", + ) + } + + return handoff, nil +} + +// QuarantinedSignerMetadata describes one quarantined tBTC signer output for +// the offline state audit, without any private material: the key share itself +// stays only inside the encrypted membership record it accompanies. The seed +// is recorded as a hash, never raw. +type QuarantinedSignerMetadata struct { + SchemaVersion uint32 `json:"schema_version"` + ReleaseEpoch string `json:"release_epoch"` + ProtocolMode string `json:"protocol_mode"` + CutoverBlock uint64 `json:"cutover_block"` + CanonicalStartBlock uint64 `json:"canonical_start_block"` + Ceremony string `json:"ceremony"` + SeedHash string `json:"seed_hash"` + MemberIndex uint8 `json:"member_index"` + WalletID string `json:"wallet_id"` + WalletPublicKeyHash string `json:"wallet_public_key_hash"` + FailedOperation string `json:"failed_operation"` + LastObservedBlock uint64 `json:"last_observed_block"` + PreservedAt time.Time `json:"preserved_at"` +} + +// signerQuarantine preserves tBTC signer outputs whose activation the +// participation gate refused — clock failure, forced quiescence, or a refused +// commit fence — before the wallet's on-chain registration was proven. The +// handle MUST be rooted in a dedicated protected namespace that no release's +// active-wallet scan reads: quarantined records use the same membership +// encoding as active ones, so placing them beside active membership files +// would make a prior binary load them as active signers, which is not +// rollback-safe. Quarantined material is recovery evidence for the offline +// state audit; it is never activated by the running process. +type signerQuarantine struct { + logger log.StandardLogger + handle persistence.ProtectedHandle + + // lifetime bounds how long a preservation keeps trying to write the output + // it is holding. It is the process lifetime rather than the ceremony's: + // the ceremony context is normally already canceled by the very refusal + // that sent the share here, and until the process itself is going away the + // generated key material is still this node's to write down. It is held on + // the store because the choke points that preserve run several call levels + // below the startup that knows the process context. + lifetime context.Context + + // graceAttempts, retryDelay, and maxRetryDelay shape the retry, and wait + // pauses between rounds unless the lifetime ends first. They are fields so + // a test does not have to spend the real delays. + graceAttempts int + retryDelay time.Duration + maxRetryDelay time.Duration + wait func(context.Context, time.Duration) bool +} + +// quarantineGraceAttempts bounds how many rounds a preservation makes before +// the node is told it is holding key material the namespace does not have, and +// quarantineRetryDelay and quarantineMaxRetryDelay bound the wait between +// rounds. +// +// The grace budget is not a deadline. A refused write is often transient — a +// namespace being remounted, a disk an operator is draining — so the first +// rounds pass without disturbing the fleet; what follows is a node that stops +// taking new work while it keeps trying, not a node that gives the share up. +// The material on this path cannot be generated again, so the retry ends only +// with the process, and the backoff grows to keep a namespace that is down for +// an operator's whole repair from being hammered. +const ( + quarantineGraceAttempts = 3 + quarantineRetryDelay = 100 * time.Millisecond + quarantineMaxRetryDelay = 30 * time.Second +) + +// newSignerQuarantine creates a quarantine store over the given protected +// handle, preserving outputs for as long as the given process lifetime lasts. +func newSignerQuarantine( + lifetime context.Context, + logger log.StandardLogger, + handle persistence.ProtectedHandle, +) *signerQuarantine { + return &signerQuarantine{ + logger: logger, + handle: handle, + lifetime: lifetime, + graceAttempts: quarantineGraceAttempts, + retryDelay: quarantineRetryDelay, + maxRetryDelay: quarantineMaxRetryDelay, + wait: waitWithinLifetime, + } +} + +// waitWithinLifetime pauses between preservation rounds, reporting whether the +// process is still around to make another one. +func waitWithinLifetime(lifetime context.Context, delay time.Duration) bool { + if lifetime == nil { + time.Sleep(delay) + return true + } + + // Checked before the wait so a lifetime that has already ended stops the + // retry rather than racing the timer for it. + if lifetime.Err() != nil { + return false + } + + timer := time.NewTimer(delay) + defer timer.Stop() + + select { + case <-lifetime.Done(): + return false + case <-timer.C: + return true + } +} + +// quarantineState reports which halves of a preserved output reached the +// namespace. A caller cannot infer this from the error alone, and the two +// halves mean different things: the membership is the key material a rollback +// has to account for, while the metadata is the audit record explaining it. +// Reporting the wrong one is how an operator log, a published count, and the +// offline audit come to disagree about the same directory. +type quarantineState struct { + membershipPersisted bool + metadataPersisted bool + // handoffPersisted reports whether the namespace holds the combined record + // carrying both halves at once. It is written only when the pair could not + // be completed, and once it lands the output is whole whatever the pair is + // still missing. + handoffPersisted bool +} + +// keyMaterialPersisted reports whether the namespace holds the generated share +// in either of the forms preservation writes it in. This is what the count of +// preserved material follows: a share on disk is material a rollback has to +// account for however it got written down. +func (s quarantineState) keyMaterialPersisted() bool { + return s.membershipPersisted || s.handoffPersisted +} + +// complete reports whether the namespace holds the whole output — the key +// material and the audit record explaining it. The pair says so when both its +// halves landed, and the combined record says so on its own, since it carries +// both. +func (s quarantineState) complete() bool { + return (s.membershipPersisted && s.metadataPersisted) || s.handoffPersisted +} + +// quarantineObserver receives what a preservation learns while it is still +// holding the output, for the things a caller must not wait for the return +// value to know. Both callbacks run on the preserving goroutine. +type quarantineObserver struct { + // keyMaterialPreserved is called the moment the namespace takes the key + // material, whether or not the audit record beside it has landed. + // + // The count of preserved shares follows the material alone, and the + // preservation still waiting on the other half can run for the rest of the + // process. A caller that learned this from the return value would leave a + // share the namespace already holds unreported for exactly as long as the + // metadata keeps being refused — the stale all-clear the count exists to + // prevent. + // + // It runs between the two writes of the round the material landed in, so + // it must do only what belongs on that path. Anything slow here — a + // namespace-wide read above all — delays the audit record that turns the + // preserved share into an explained one, and delays it for as long as + // whatever it waited on takes. Reconciliation that can wait belongs after + // preserve returns. + keyMaterialPreserved func() + + // stillIncomplete is called once, after graceAttempts rounds have left a + // half unwritten, with what the namespace holds so far. It exists so the + // node can stop taking new work while it is still holding an output no + // namespace fully has — not to end the attempt, which continues behind it + // until the pair is durable or the process ends. + // + // It is one-shot on purpose: quiescence is one-way, so saying it twice + // changes nothing, and key material that lands in a later round is + // reported by keyMaterialPreserved rather than by a second notification + // here. + stillIncomplete func(quarantineState, error) +} + +// preserve durably saves the signer membership and its audit metadata under +// the quarantine namespace, mirroring the active storage layout so the same +// decoding path can interpret both. It keeps ownership of the generated output +// until both halves are durable, retrying for as long as the process lives, and +// returns early only when the process is going away with a half still missing. +// +// Both records are attempted in every round and what actually landed is +// returned beside the error, because the two halves mean different things and a +// caller cannot infer either from the error alone. A membership without +// metadata is unexplained key material; metadata without a membership is a +// share that was lost. What must not happen is the node reporting a state the +// namespace contradicts. +// +// The membership is attempted first so that a process killed between the two +// writes leaves the key material behind rather than only the note describing +// it: an unexplained share is recoverable, a lost one is not. +// +// A round that cannot complete the pair falls back on the combined handoff +// record, which carries both halves in one write under a name of its own. It is +// what keeps a namespace refusing one particular record from costing the node a +// share it can never generate again, and once it lands the output is whole +// however little of the pair the namespace took. +// +// The observer is told what the namespace takes while the preservation is still +// running, because a caller that only reads the returned state learns nothing +// until an attempt that may outlast the process is over. +func (q *signerQuarantine) preserve( + signer *signer, + metadata QuarantinedSignerMetadata, + observer quarantineObserver, +) (quarantineState, error) { + var state quarantineState + + signerBytes, err := signer.Marshal() + if err != nil { + return state, fmt.Errorf( + "could not marshal the quarantined signer: [%v]", + err, + ) + } + + walletPublicKeyHash := bitcoin.PublicKeyHash(signer.wallet.publicKey) + + metadata.SchemaVersion = QuarantineSchemaVersion + metadata.MemberIndex = uint8(signer.signingGroupMemberIndex) + metadata.WalletPublicKeyHash = hex.EncodeToString(walletPublicKeyHash[:]) + metadata.PreservedAt = time.Now().UTC() + + metadataBytes, err := json.Marshal(metadata) + if err != nil { + return state, fmt.Errorf( + "could not marshal the quarantine metadata: [%v]", + err, + ) + } + + handoffBytes, err := json.Marshal(QuarantinedSignerHandoff{ + SchemaVersion: QuarantineHandoffSchemaVersion, + Metadata: metadata, + Signer: signerBytes, + }) + if err != nil { + return state, fmt.Errorf( + "could not marshal the quarantine handoff: [%v]", + err, + ) + } + + directory := getWalletStorageKey(signer.wallet.publicKey) + memberSuffix := fmt.Sprint(signer.signingGroupMemberIndex) + + // One line names the output and what the namespace holds of it, so the + // operator record and the namespace cannot drift apart. An incomplete pair + // is a finding the offline audit will raise, so it reads as an error rather + // than like an ordinary quarantine. + report := func(rounds int, complete bool) { + logQuarantine := q.logger.Warnf + if !complete { + logQuarantine = q.logger.Errorf + } + logQuarantine( + "quarantined a tbtc signer output [walletPKH=0x%s] [member=%v] "+ + "[mode=%s] [canonicalStartBlock=%d] [failedOperation=%s] "+ + "[lastObservedBlock=%d] [keyMaterialPreserved=%v] "+ + "[auditMetadataPreserved=%v] [preservedAsOneRecord=%v] "+ + "[rounds=%d]", + metadata.WalletPublicKeyHash, + signer.signingGroupMemberIndex, + metadata.ProtocolMode, + metadata.CanonicalStartBlock, + metadata.FailedOperation, + metadata.LastObservedBlock, + state.keyMaterialPersisted(), + state.metadataPersisted || state.handoffPersisted, + state.handoffPersisted, + rounds, + ) + } + + rounds, lastErr := q.persistOutput( + &state, + directory, + memberSuffix, + signerBytes, + metadataBytes, + handoffBytes, + observer, + ) + if lastErr == nil { + report(rounds, true) + return state, nil + } + + report(rounds, false) + + return state, fmt.Errorf( + "could not preserve the quarantined tbtc signer output in %d rounds "+ + "before the process ended [keyMaterialPreserved=%v] "+ + "[auditMetadataPreserved=%v]: %w", + rounds, + state.keyMaterialPersisted(), + state.metadataPersisted || state.handoffPersisted, + lastErr, + ) +} + +// persistOutput writes whichever records of a preserved output the namespace +// has not taken yet, round after round, until it holds the whole output or the +// process ends. It reports how many rounds were spent and the last round's +// failure, which is nil exactly when the output is durable. +// +// The preferred form is the pair — a membership record beside its metadata — +// because it is the layout the active namespace uses and the one every reader +// already understands. A round that cannot complete the pair falls back on the +// combined handoff record, which carries both halves under a name of its own, so +// no namespace that refuses one particular record can leave key material with +// nowhere to go. A landed handoff ends the attempt, since there is nothing left +// the namespace does not hold. +// +// A record counts as landed on the namespace's word that it took the write, not +// on a reader's. The disk persistence behind this handle creates the file, +// writes the document, and syncs it, with no temporary record renamed into +// place, so a write a crash interrupts leaves a truncated document behind — and +// confirming each write by enumerating the namespace would put a share this node +// is still holding behind a directory listing that may never return, which is +// the more expensive way to lose it. What a torn write leaves is caught on the +// way out instead: the document fails the encrypted handle's authentication, so +// the offline audit reads it as an unreadable record and blocks on it rather +// than any reader taking it for a preserved output. +// +// The state is updated in place as each record lands so that a caller reading it +// after an interrupted preservation sees what the namespace actually has, and +// so a record that succeeded is never rewritten by a later round. +func (q *signerQuarantine) persistOutput( + state *quarantineState, + directory string, + memberSuffix string, + signerBytes []byte, + metadataBytes []byte, + handoffBytes []byte, + observer quarantineObserver, +) (int, error) { + graceAttempts := q.graceAttempts + if graceAttempts < 1 { + graceAttempts = 1 + } + wait := q.wait + if wait == nil { + wait = waitWithinLifetime + } + delay := q.retryDelay + + notified := false + + // announcedLostMaterial remembers that the operator record says this share + // reached no namespace. It is what makes a later write worth a line of its + // own: until one is written, the standing account of this output is an error + // saying the material is only in memory, over a namespace that now holds it. + announcedLostMaterial := false + + var lastErr error + + // materialAccountedFor keeps the observer's count of preserved shares to one + // notification per output. The material can reach the namespace as the + // membership record or inside the handoff, and it is the same share either + // way. + materialAccountedFor := false + accountForMaterial := func(round int) { + if announcedLostMaterial { + announcedLostMaterial = false + // Named by the directory the record lives under rather than by the + // wallet hash the other quarantine lines carry: an operator reading + // this is going to the namespace to confirm the material is there. + q.logger.Warnf( + "the quarantine namespace took the tbtc key material it had "+ + "been refusing [walletStorageKey=%s] [member=%s] "+ + "[round=%d]; the share this node reported as only in "+ + "memory is on disk", + directory, + memberSuffix, + round, + ) + } + + if materialAccountedFor { + return + } + materialAccountedFor = true + + if observer.keyMaterialPreserved != nil { + observer.keyMaterialPreserved() + } + } + + for round := 1; ; round++ { + var roundErrs []error + + if !state.membershipPersisted { + if err := q.handle.Save( + signerBytes, + directory, + "/membership_"+memberSuffix, + ); err != nil { + roundErrs = append(roundErrs, fmt.Errorf( + "could not persist the quarantined signer: [%v]", + err, + )) + } else { + // Reported from inside the round rather than from the return, + // because the round the metadata lands in may never come: this + // is the only moment the material is known to be held that a + // caller is guaranteed to see. + state.membershipPersisted = true + + accountForMaterial(round) + } + } + + if !state.metadataPersisted { + if err := q.handle.Save( + metadataBytes, + directory, + "/metadata_"+memberSuffix, + ); err != nil { + roundErrs = append(roundErrs, fmt.Errorf( + "could not persist the quarantine metadata: [%v]", + err, + )) + } else { + state.metadataPersisted = true + } + } + + // The pair is what this round could not finish, so the output is + // offered whole under a name of its own. A namespace refusing one + // particular record — a leftover file nothing can overwrite, a name an + // operator's repair left behind — still has somewhere to put a share + // that cannot be generated a second time. + // + // When the half that did land was the membership, the namespace ends up + // holding the material twice. That is the cheaper mistake: both copies + // are the same encrypted bytes under the same handle, readers count the + // seat once, and the alternative is choosing which refusals are worth + // leaving an output incomplete for. + if len(roundErrs) > 0 && !state.handoffPersisted { + if err := q.handle.Save( + handoffBytes, + directory, + "/handoff_"+memberSuffix, + ); err != nil { + roundErrs = append(roundErrs, fmt.Errorf( + "could not persist the quarantine handoff: [%v]", + err, + )) + } else { + state.handoffPersisted = true + + q.logger.Warnf( + "preserved a tbtc signer output as a single handoff record "+ + "[walletStorageKey=%s] [member=%s] [round=%d]; the "+ + "namespace would not take the record pair, and the key "+ + "material and its audit record are held together "+ + "instead", + directory, + memberSuffix, + round, + ) + + accountForMaterial(round) + } + } + + if state.complete() { + return round, nil + } + + lastErr = errors.Join(roundErrs...) + + // The node is told once the grace rounds are spent, so a namespace that + // clears on its own does not take the node out of the fleet, and one + // that does not stops it from building further state it cannot account + // for. Preservation does not end here: the share is still in hand and + // the retry keeps running behind the notification. + if !notified && round >= graceAttempts { + notified = true + announcedLostMaterial = !state.keyMaterialPersisted() + if observer.stillIncomplete != nil { + observer.stillIncomplete(*state, lastErr) + } + } + + if !wait(q.lifetime, delay) { + return round, lastErr + } + + if delay *= 2; delay > q.maxRetryDelay { + delay = q.maxRetryDelay + } + } +} + +// quarantinedSigner names one preserved signer output by the wallet it belongs +// to and the seat it was generated for — the pair an active signer is also +// identified by, so the two namespaces can be compared without decoding either +// side's key material. +type quarantinedSigner struct { + walletStorageKey string + memberIndex group.MemberIndex +} + +// preservedOutputs lists the signer outputs currently held in the quarantine +// namespace. +// +// Only the records carrying key material are counted. A preserved output is +// written either as a membership beside its audit metadata or as a single +// handoff carrying both, and in each form exactly one record holds the share; +// counting the metadata beside it would report the same output twice. Nothing +// here reads a record's content: the pair identifying an output is carried by +// the wallet directory and the record name, and the share inside stays +// encrypted and unread. +// +// The same seat can be named by both forms — a preservation that wrote the +// membership, was refused the metadata, and fell back on the handoff leaves +// both on disk — so outputs are collected as identities rather than counted per +// record. One seat of one wallet is one share whatever it took to write it +// down. +// +// A namespace that cannot be enumerated returns an error rather than a short +// list. The count exists to say how much preserved material a rollback still +// has to account for, and a truncated one reads as an all-clear — the single +// answer this must never invent. +func (q *signerQuarantine) preservedOutputs() ([]quarantinedSigner, error) { + descriptorsChan, errorsChan := q.handle.ReadAll() + + // The descriptor and error channels are unbuffered and written to in an + // order this side cannot predict, so both are drained concurrently. This + // mirrors the active wallet storage scan. + var wg sync.WaitGroup + wg.Add(2) + + found := make(map[quarantinedSigner]struct{}) + go func() { + defer wg.Done() + for descriptor := range descriptorsChan { + memberIndex, ok := quarantinedMemberIndex(descriptor.Name()) + if !ok { + continue + } + found[quarantinedSigner{ + walletStorageKey: descriptor.Directory(), + memberIndex: memberIndex, + }] = struct{}{} + } + }() + + var readErrs []error + go func() { + defer wg.Done() + for err := range errorsChan { + readErrs = append(readErrs, err) + } + }() + + wg.Wait() + + if len(readErrs) > 0 { + return nil, fmt.Errorf( + "could not enumerate the signer quarantine namespace: %w", + errors.Join(readErrs...), + ) + } + + outputs := make([]quarantinedSigner, 0, len(found)) + for output := range found { + outputs = append(outputs, output) + } + + return outputs, nil +} + +// quarantinedMemberIndex reads the seat a preserved record was written for out +// of its name, reporting whether the name belongs to a record holding key +// material at all. +// +// The names are this package's own: preserve writes "membership_" beside +// "metadata_", and falls back on "handoff_" carrying both. The two +// that hold the share count; anything else in the namespace — the metadata +// documents, a name a later schema adds, an operator's stray file — is not a +// signer output and is not counted as one. +// +// A leading separator is tolerated because the name is written with one and not +// every handle hands it back the same way: the disk implementation joins it into +// a path and enumerates the bare file name, while a handle that keeps what it +// was given returns the name as this package wrote it. Both spell the same +// record, so neither is allowed to decide whether it counts. +func quarantinedMemberIndex(name string) (group.MemberIndex, bool) { + bare := strings.TrimPrefix(name, "/") + + var suffix string + for _, prefix := range []string{"membership_", "handoff_"} { + if cut, found := strings.CutPrefix(bare, prefix); found { + suffix = cut + break + } + } + if suffix == "" { + return 0, false + } + + seat, err := strconv.ParseUint(suffix, 10, 8) + if err != nil || seat == 0 { + return 0, false + } + + return group.MemberIndex(seat), true +} diff --git a/pkg/tbtc/redemption.go b/pkg/tbtc/redemption.go index 1dd950c95f..623580601d 100644 --- a/pkg/tbtc/redemption.go +++ b/pkg/tbtc/redemption.go @@ -13,6 +13,7 @@ import ( "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/protocol/participation" ) const ( @@ -120,6 +121,11 @@ type redemptionAction struct { feeDistribution redemptionFeeDistributionFn transactionShape RedemptionTransactionShape + // permit is the wallet action's participation permit: it pins the + // protocol mode for every signing of this action and is released when the + // action's execution ends. + permit participation.Permit + // metricsRecorder is optional and used for recording performance metrics metricsRecorder interface { IncrementCounter(name string, value float64) @@ -137,12 +143,15 @@ func newRedemptionAction( proposalProcessingStartBlock uint64, proposalExpiryBlock uint64, waitForBlockFn waitForBlockFn, + permit participation.Permit, ) *redemptionAction { transactionExecutor := newWalletTransactionExecutor( btcChain, redeemingWallet, signingExecutor, waitForBlockFn, + permit, + "tbtc_redemption_bitcoin_broadcast", ) feeDistribution := withRedemptionTotalFee(proposal.RedemptionTxFee.Int64()) @@ -161,10 +170,18 @@ func newRedemptionAction( broadcastCheckDelay: redemptionBroadcastCheckDelay, feeDistribution: feeDistribution, transactionShape: RedemptionChangeFirst, + permit: permit, } } func (ra *redemptionAction) execute() error { + // The action owns its permit from dispatch on; releasing it here ends the + // ceremony's active accounting in the participation gate. The terminal + // outcome is registered afterwards so it runs first and reaches the permit + // while it is still open. + defer ra.permit.Close() + defer ra.transactionExecutor.recordTerminalOutcome(ra.logger) + startTime := time.Now() // Record redemption execution attempt @@ -269,10 +286,13 @@ func (ra *redemptionAction) execute() error { ra.proposalExpiryBlock-ra.signingTimeoutSafetyMarginBlocks, ) if err != nil { - if ra.metricsRecorder != nil { + // A gate-caused abort is not an ordinary failure of this action and + // stays out of its failure metrics; the wrapped cause lets the + // dispatcher classify it the same way. + if ra.metricsRecorder != nil && !participation.IsGateRefusal(err) { ra.metricsRecorder.IncrementCounter(clientinfo.MetricRedemptionExecutionsFailedTotal, 1) } - return fmt.Errorf("sign transaction step failed: [%v]", err) + return fmt.Errorf("sign transaction step failed: [%w]", err) } broadcastTxLogger := ra.logger.With( @@ -287,10 +307,13 @@ func (ra *redemptionAction) execute() error { ra.broadcastCheckDelay, ) if err != nil { - if ra.metricsRecorder != nil { + // A gate-caused abort is not an ordinary failure of this action and + // stays out of its failure metrics; the wrapped cause lets the + // dispatcher classify it the same way. + if ra.metricsRecorder != nil && !participation.IsGateRefusal(err) { ra.metricsRecorder.IncrementCounter(clientinfo.MetricRedemptionExecutionsFailedTotal, 1) } - return fmt.Errorf("broadcast transaction step failed: [%v]", err) + return fmt.Errorf("broadcast transaction step failed: [%w]", err) } // Record successful redemption execution diff --git a/pkg/tbtc/redemption_test.go b/pkg/tbtc/redemption_test.go index 0a6897dd94..e6b7aed368 100644 --- a/pkg/tbtc/redemption_test.go +++ b/pkg/tbtc/redemption_test.go @@ -12,6 +12,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/bitcoin" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tbtc/internal/test" ) @@ -135,6 +136,7 @@ func TestRedemptionAction_Execute(t *testing.T) { func(ctx context.Context, blockHeight uint64) error { return nil }, + newTestPermit(participation.TBTCSigning), ) // Modify the default parameters of the action to make diff --git a/pkg/tbtc/registry.go b/pkg/tbtc/registry.go index 8e5e33595d..065b8c6a64 100644 --- a/pkg/tbtc/registry.go +++ b/pkg/tbtc/registry.go @@ -8,6 +8,7 @@ import ( "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/crypto/secp256k1" + "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-common/pkg/persistence" ) @@ -117,34 +118,60 @@ func (wr *walletRegistry) getWalletsPublicKeys() []*ecdsa.PublicKey { return keys } -// registerSigner registers the given signer using in the walletRegistry. -func (wr *walletRegistry) registerSigner(signer *signer) error { +// saveSigner durably persists the given signer in the active wallet storage +// namespace without activating it in the in-memory wallet cache. The signer +// becomes visible to this process only after a restart's storage scan. It is +// the durable-save half of registerSigner, used when the release gate refuses +// activation but the wallet is already registered on chain: the share must +// survive, and any release's active scan may legitimately load it. +func (wr *walletRegistry) saveSigner(signer *signer) error { wr.mutex.Lock() defer wr.mutex.Unlock() - err := wr.walletStorage.saveSigner(signer) - if err != nil { + if err := wr.walletStorage.saveSigner(signer); err != nil { return fmt.Errorf("cannot save signer in the storage: [%w]", err) } + return nil +} + +// registerSigner registers the given signer using in the walletRegistry: it +// durably persists the signer and activates it in the in-memory wallet cache. +// Every fallible step runs before the durable save and the cache activation +// follows a successful save unconditionally, so an error return guarantees +// this call left no record in the active storage namespace: an interrupted +// ceremony can then preserve the signer in quarantine without a second, +// active copy surfacing in a restart's — or any release's — active scan. +func (wr *walletRegistry) registerSigner(signer *signer) error { + wr.mutex.Lock() + defer wr.mutex.Unlock() + walletStorageKey := getWalletStorageKey(signer.wallet.publicKey) - // If the wallet cache does not have the given entry yet, initialize - // the value and compute the wallet ID and wallet public key hash. This way, - // the hashes are computed only once. No need to initialize signers slice as - // appending works with nil values. + // If the wallet cache does not have the given entry yet, prepare the value + // with the wallet ID and wallet public key hash. This way, the hashes are + // computed only once. No need to initialize signers slice as appending + // works with nil values. + var newCacheValue *walletCacheValue if _, ok := wr.walletCache[walletStorageKey]; !ok { walletID, err := wr.calculateWalletIdFunc(signer.wallet.publicKey) if err != nil { return fmt.Errorf("cannot calculate wallet ID: [%v]", err) } - wr.walletCache[walletStorageKey] = &walletCacheValue{ + newCacheValue = &walletCacheValue{ walletPublicKeyHash: bitcoin.PublicKeyHash(signer.wallet.publicKey), walletID: walletID, } } + if err := wr.walletStorage.saveSigner(signer); err != nil { + return fmt.Errorf("cannot save signer in the storage: [%w]", err) + } + + if newCacheValue != nil { + wr.walletCache[walletStorageKey] = newCacheValue + } wr.walletCache[walletStorageKey].signers = append( wr.walletCache[walletStorageKey].signers, signer, @@ -167,6 +194,35 @@ func (wr *walletRegistry) getSigners( return nil } +// isSignerActive reports whether the wallet cache holds an active signer for +// the given wallet storage key and signing group seat. +// +// It answers the question the quarantine count is about, and it is asked in the +// cache's own terms: a preserved output is identified by the wallet directory it +// was written under and the seat it was generated for, which is the pair the +// cache is keyed and its signers indexed by. Comparing the two namespaces this +// way needs neither side's key material. +func (wr *walletRegistry) isSignerActive( + walletStorageKey string, + memberIndex group.MemberIndex, +) bool { + wr.mutex.Lock() + defer wr.mutex.Unlock() + + value, ok := wr.walletCache[walletStorageKey] + if !ok { + return false + } + + for _, signer := range value.signers { + if signer.signingGroupMemberIndex == memberIndex { + return true + } + } + + return false +} + // getWalletByPublicKeyHash gets the given wallet by its 20-byte wallet // public key hash. Second boolean return value denotes whether the wallet // was found in the registry or not. diff --git a/pkg/tbtc/registry_test.go b/pkg/tbtc/registry_test.go index f0d4964ce1..bf9e1455b1 100644 --- a/pkg/tbtc/registry_test.go +++ b/pkg/tbtc/registry_test.go @@ -68,6 +68,46 @@ func TestWalletRegistry_RegisterSigner(t *testing.T) { ) } +// TestWalletRegistry_RegisterSigner_WalletIdFailureLeavesNoActiveRecord +// proves a wallet ID calculation failure aborts the registration before the +// durable save: the active storage namespace and the wallet cache both stay +// untouched, so the caller can preserve the signer elsewhere without leaving +// a second, active copy behind. +func TestWalletRegistry_RegisterSigner_WalletIdFailureLeavesNoActiveRecord( + t *testing.T, +) { + persistenceHandle := &mockPersistenceHandle{} + + walletRegistry, err := newWalletRegistry( + persistenceHandle, + func(*ecdsa.PublicKey) ([32]byte, error) { + return [32]byte{}, fmt.Errorf("wallet ID calculation failed") + }, + ) + if err != nil { + t.Fatal(err) + } + + signer := createMockSigner(t) + + if err := walletRegistry.registerSigner(signer); err == nil { + t.Fatal("expected the registration to fail") + } + + testutils.AssertIntsEqual( + t, + "active-namespace saves", + 0, + len(persistenceHandle.saved), + ) + testutils.AssertIntsEqual( + t, + "cached wallets", + 0, + len(walletRegistry.walletCache), + ) +} + func TestWalletRegistry_GetSigners(t *testing.T) { persistenceHandle := &mockPersistenceHandle{} chain := Connect() @@ -548,7 +588,15 @@ func (mph *mockPersistenceHandle) Archive(directory string) error { } func (mph *mockPersistenceHandle) Delete(directory string, name string) error { - panic("not implemented") + for i, descriptor := range mph.saved { + if descriptor.Directory() == directory && descriptor.Name() == name { + mph.saved = append(mph.saved[:i], mph.saved[i+1:]...) + return nil + } + } + + // Deleting an absent entry is a no-op, matching the disk implementation. + return nil } type mockDescriptor struct { diff --git a/pkg/tbtc/session_id_test.go b/pkg/tbtc/session_id_test.go new file mode 100644 index 0000000000..da303defdc --- /dev/null +++ b/pkg/tbtc/session_id_test.go @@ -0,0 +1,120 @@ +package tbtc + +import ( + "math/big" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// TestDkgAttemptSessionID_ExactForms pins both compatibility forms of the DKG +// attempt session ID byte-for-byte: the legacy form is exactly what the +// pre-hardening production releases announce, and the security-v2 form is the +// hardened protocol-named, fixed-width form. The announcer's wire-format +// classifier must agree with the producer on both. +func TestDkgAttemptSessionID_ExactForms(t *testing.T) { + seed := new(big.Int).SetBytes([]byte{0xAB, 0xCD, 0xEF}) + + legacy := dkgAttemptSessionID(participation.ModeLegacy, seed, 7) + if legacy != "abcdef-7" { + t.Errorf("expected legacy session ID [abcdef-7], got [%s]", legacy) + } + if format := announcer.ClassifySessionIDFormat( + legacy, + ); format != announcer.SessionIDFormatLegacy { + t.Errorf( + "expected the legacy session ID to classify as legacy, got [%s]", + format, + ) + } + + hardened := dkgAttemptSessionID(participation.ModeSecurityV2, seed, 7) + if hardened != "dkg-abcdef-0000000000000007" { + t.Errorf( + "expected hardened session ID [dkg-abcdef-0000000000000007], "+ + "got [%s]", + hardened, + ) + } + if format := announcer.ClassifySessionIDFormat( + hardened, + ); format != announcer.SessionIDFormatHardenedDKG { + t.Errorf( + "expected the hardened session ID to classify as hardened DKG, "+ + "got [%s]", + format, + ) + } +} + +// TestSigningAttemptSessionID_ExactForms pins both compatibility forms of the +// signing attempt session ID byte-for-byte. The legacy form carries no attempt +// start block — exactly as the pre-hardening production releases announce — +// while the security-v2 form carries the protocol name and fixed-width start +// block and attempt. +func TestSigningAttemptSessionID_ExactForms(t *testing.T) { + message := new(big.Int).SetBytes([]byte{0x01, 0x23, 0x45}) + + legacy := signingAttemptSessionID(participation.ModeLegacy, message, 206, 12) + if legacy != "12345-12" { + t.Errorf("expected legacy session ID [12345-12], got [%s]", legacy) + } + if format := announcer.ClassifySessionIDFormat( + legacy, + ); format != announcer.SessionIDFormatLegacy { + t.Errorf( + "expected the legacy session ID to classify as legacy, got [%s]", + format, + ) + } + + hardened := signingAttemptSessionID( + participation.ModeSecurityV2, + message, + 206, + 12, + ) + if hardened != "signing-12345-00000000000000ce-000000000000000c" { + t.Errorf( + "expected hardened session ID "+ + "[signing-12345-00000000000000ce-000000000000000c], got [%s]", + hardened, + ) + } + if format := announcer.ClassifySessionIDFormat( + hardened, + ); format != announcer.SessionIDFormatHardenedSigning { + t.Errorf( + "expected the hardened session ID to classify as hardened "+ + "signing, got [%s]", + format, + ) + } +} + +// TestAttemptSessionID_UnsetModePanics proves there is no implicit protocol +// mode: an unset mode is a programming error that must fail loudly rather +// than silently produce either wire format. +func TestAttemptSessionID_UnsetModePanics(t *testing.T) { + assertPanics := func(name string, fn func()) { + defer func() { + if recover() == nil { + t.Errorf("%s: expected a panic for an unset protocol mode", name) + } + }() + fn() + } + + assertPanics("dkg", func() { + dkgAttemptSessionID(participation.ProtocolMode(0), big.NewInt(1), 1) + }) + assertPanics("signing", func() { + signingAttemptSessionID( + participation.ProtocolMode(0), + big.NewInt(1), + 1, + 1, + ) + }) +} diff --git a/pkg/tbtc/signing.go b/pkg/tbtc/signing.go index 346b6b0446..cd3a3f5020 100644 --- a/pkg/tbtc/signing.go +++ b/pkg/tbtc/signing.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "math/big" + "slices" "strings" "sync" "time" @@ -12,11 +13,14 @@ import ( "github.com/keep-network/keep-core/pkg/generator" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/signing" "go.uber.org/zap" "golang.org/x/sync/semaphore" + "golang.org/x/time/rate" ) const ( @@ -67,6 +71,21 @@ type signingExecutor struct { SetGauge(name string, value float64) RecordDuration(name string, duration time.Duration) } + + // cutoverPeerRoster is optional and, when set, records post-cutover legacy + // peer sightings observed by the signing announcer. + cutoverPeerRoster *participation.CutoverPeerRoster + + // participationGate is the process gate that issued the wallet action + // permit. Signing uses the permit's immutable mode for protocol decisions; + // the shared gate is retained only so mismatch logs report the process's + // current gate state. + participationGate participation.Gate + + // announcerMismatchLogLimiter bounds the volume of session-ID mismatch INFO + // logs to a burst of 5 with one line every 30 seconds, matching the + // observability contract. Metrics retain every event. + announcerMismatchLogLimiter *rate.Limiter } func newSigningExecutor( @@ -78,36 +97,51 @@ func newSigningExecutor( getCurrentBlockFn getCurrentBlockFn, waitForBlockFn waitForBlockFn, signingAttemptsLimit uint, + participationGate participation.Gate, ) *signingExecutor { return &signingExecutor{ - lock: semaphore.NewWeighted(1), - signers: signers, - broadcastChannel: broadcastChannel, - membershipValidator: membershipValidator, - groupParameters: groupParameters, - protocolLatch: protocolLatch, - getCurrentBlockFn: getCurrentBlockFn, - waitForBlockFn: waitForBlockFn, - signingAttemptsLimit: signingAttemptsLimit, + lock: semaphore.NewWeighted(1), + signers: signers, + broadcastChannel: broadcastChannel, + membershipValidator: membershipValidator, + groupParameters: groupParameters, + protocolLatch: protocolLatch, + getCurrentBlockFn: getCurrentBlockFn, + waitForBlockFn: waitForBlockFn, + signingAttemptsLimit: signingAttemptsLimit, + participationGate: participationGate, + announcerMismatchLogLimiter: rate.NewLimiter(rate.Every(30*time.Second), 5), } } +// setCutoverPeerRoster sets the node-local cutover peer roster for the signing +// executor. +func (se *signingExecutor) setCutoverPeerRoster(roster *participation.CutoverPeerRoster) { + se.cutoverPeerRoster = roster +} + // signBatch performs the signing process for each message from the given // messages batch, one after another. If at least one message cannot be signed, // this function returns an error. If all messages were signed successfully, // a slice of signatures is returned. Order of the returned signatures matches // the order of the messages in the batch, i.e. the first signature corresponds -// to the first message, and so on. +// to the first message, and so on. The protocol mode comes from the wallet +// action's participation permit and applies to every message and retry of the +// batch. func (se *signingExecutor) signBatch( ctx context.Context, messages []*big.Int, startBlock uint64, -) ([]*tecdsa.Signature, error) { + mode participation.ProtocolMode, +) ([]*tecdsa.Signature, *participation.TranscriptContribution, error) { wallet := se.wallet() walletPublicKeyBytes, err := marshalPublicKey(wallet.publicKey) if err != nil { - return nil, fmt.Errorf("cannot marshal wallet public key: [%v]", err) + return nil, nil, fmt.Errorf( + "cannot marshal wallet public key: [%v]", + err, + ) } messagesDigests := make([]string, len(messages)) @@ -141,6 +175,11 @@ func (se *signingExecutor) signBatch( signingStartBlock := startBlock // start block for the first signing signatures := make([]*tecdsa.Signature, len(messages)) endBlocks := make([]uint64, len(messages)) + transcripts := make( + []*participation.TranscriptContribution, + 0, + len(messages), + ) for i, message := range messages { signingBatchMessageLogger := signingBatchLogger.With( @@ -154,44 +193,166 @@ func (se *signingExecutor) signBatch( signingStartBlock = endBlocks[i-1] + signingBatchInterludeBlocks } - signature, _, endBlock, err := se.sign(ctx, message, signingStartBlock) + outcome, err := se.sign( + ctx, + message, + signingStartBlock, + mode, + ) if err != nil { // Error metrics are recorded in the sign() method for all error paths. - return nil, err + return nil, nil, err } signingBatchMessageLogger.Infof( "generated signature [%v] for message at block [%v]", - signature, - endBlock, + outcome.signature, + outcome.endBlock, ) - signatures[i] = signature - endBlocks[i] = endBlock + signatures[i] = outcome.signature + endBlocks[i] = outcome.endBlock + transcripts = append(transcripts, outcome.contribution) + } + + return signatures, intersectTranscripts(transcripts), nil +} + +// intersectTranscripts reduces the transcripts behind a batch's signatures to +// the memberships present in every one of them. +// +// A wallet action's durable result is one Bitcoin transaction carrying every +// signature in its batch, and each signature's attempt selects its own members, +// so the populations can differ across the batch. The intersection is the only +// reading that cannot overclaim: a membership that carried one signature and not +// another did not contribute to the transaction as a whole, and counting it +// would let one share stand for the several a threshold needs. +// +// It can underclaim, and does so deliberately. Attempts that each reached an +// honest majority through different members intersect to fewer memberships than +// any one of them, and in the limit to none — a transaction every member +// contributed to somewhere can name a population smaller than a threshold. That +// is the safe direction for a reader deciding whether several parties produced a +// result: the record can only understate the population, never invent one. +// +// TODO: record each signature's population separately if a caller ever needs to +// establish participation per component rather than across the whole result — a +// cross-release wallet action whose batch retried with different member sets is +// the case where the intersection is too coarse to show that both releases took +// part. That means carrying a list of transcripts through TerminalEvidence +// instead of one, and teaching the offline audit and the participation validator +// to read a list; the single-population shape here is what every current caller +// needs, so the list is not worth its cost yet. +func intersectTranscripts( + transcripts []*participation.TranscriptContribution, +) *participation.TranscriptContribution { + if len(transcripts) == 0 { + return nil + } + + intersection := &participation.TranscriptContribution{ + IncorporatedMembers: transcripts[0].IncorporatedMembers, + LocalMembers: transcripts[0].LocalMembers, + } + for _, transcript := range transcripts[1:] { + intersection.IncorporatedMembers = intersectMemberIndexes( + intersection.IncorporatedMembers, + transcript.IncorporatedMembers, + ) + intersection.LocalMembers = intersectMemberIndexes( + intersection.LocalMembers, + transcript.LocalMembers, + ) + } + + return intersection +} + +// intersectMemberIndexes returns the memberships both sets name, preserving the +// ascending order the journal requires. +func intersectMemberIndexes( + left participation.MemberIndexes, + right participation.MemberIndexes, +) participation.MemberIndexes { + common := make(participation.MemberIndexes, 0, len(left)) + for _, index := range left { + if slices.Contains(right, index) { + common = append(common, index) + } } - return signatures, nil + return common +} + +// transcriptContribution renders the local view of who produced a signature: the +// memberships whose authenticated done checks carried it, and the ones among +// them this node operates. The local half is what lets a reader of several +// nodes' records subtract the fleet's own memberships and see which memberships +// some other node had to supply. +func (se *signingExecutor) transcriptContribution( + resultSigners participation.MemberIndexes, +) *participation.TranscriptContribution { + local := make(participation.MemberIndexes, 0, len(se.signers)) + for _, signer := range se.signers { + if slices.Contains(resultSigners, signer.signingGroupMemberIndex) { + local = append(local, signer.signingGroupMemberIndex) + } + } + slices.Sort(local) + + return &participation.TranscriptContribution{ + IncorporatedMembers: resultSigners, + LocalMembers: local, + } +} + +// signingOutcome is what one successful signing operation leaves its caller. +type signingOutcome struct { + signature *tecdsa.Signature + // activityReport is the announcement-phase activity accounting the + // heartbeat's penalty decision reads. + activityReport *signingActivityReport + // contribution is the local view of which memberships produced the + // signature, recorded as the transcript behind the ceremony's terminal + // evidence. + contribution *participation.TranscriptContribution + // endBlock is common for all wallet signers so can be used as a + // synchronization point. + endBlock uint64 } // sign performs the signing process for the given message. The process is // triggered according to the given start block. If the message cannot be signed -// within a limited time window, an error is returned. If the message was -// signed successfully, this function returns the signature along with the -// number of active members that participated in signing, the block at which the -// signature was calculated. The end block is common for all wallet signers so -// can be used as a synchronization point. +// within a limited time window, an error is returned. If the message was signed +// successfully, this function returns the signature along with the activity of +// the signing group members, the memberships whose authenticated done checks +// carried the signature, and the block at which the signature was calculated. +// The protocol mode comes from the wallet action's participation permit and +// applies to every retry attempt. func (se *signingExecutor) sign( ctx context.Context, message *big.Int, startBlock uint64, -) (*tecdsa.Signature, *signingActivityReport, uint64, error) { + mode participation.ProtocolMode, +) (*signingOutcome, error) { + // The compatibility strategy bundle carries the wallet action's mode into + // every tECDSA party this operation constructs; each retry attempt reuses + // it unchanged. + strategies, err := compatibility.StrategiesFor(mode) + if err != nil { + return nil, fmt.Errorf( + "cannot select compatibility strategies: [%v]", + err, + ) + } + if lockAcquired := se.lock.TryAcquire(1); !lockAcquired { // Record failure metrics for lock acquisition failure if se.metricsRecorder != nil { se.metricsRecorder.IncrementCounter(clientinfo.MetricSigningOperationsTotal, 1) se.metricsRecorder.IncrementCounter(clientinfo.MetricSigningFailedTotal, 1) } - return nil, nil, 0, errSigningExecutorBusy + return nil, errSigningExecutorBusy } defer se.lock.Release(1) @@ -209,7 +370,7 @@ func (se *signingExecutor) sign( if se.metricsRecorder != nil { se.metricsRecorder.IncrementCounter(clientinfo.MetricSigningFailedTotal, 1) } - return nil, nil, 0, fmt.Errorf("cannot marshal wallet public key: [%v]", err) + return nil, fmt.Errorf("cannot marshal wallet public key: [%v]", err) } loopTimeoutBlock := startBlock + @@ -222,12 +383,6 @@ func (se *signingExecutor) sign( zap.Uint64("signingTimeoutBlock", loopTimeoutBlock), ) - type signingOutcome struct { - signature *tecdsa.Signature - activityReport *signingActivityReport - endBlock uint64 - } - wg := sync.WaitGroup{} wg.Add(len(se.signers)) signingOutcomeChan := make(chan *signingOutcome, len(se.signers)) @@ -239,10 +394,42 @@ func (se *signingExecutor) sign( defer wg.Done() + // currentMode is the local node's protocol mode for this ceremony, + // pinned in the wallet action's participation permit. It + // classifies our own announcement so the mismatch observer can + // tell legacy peers apart from hardened ones during a coordinated + // cutover. + currentMode := mode + // operatorAddresses maps a sender's signing-group member index + // (1-based) to its operator address so a mismatch can be attributed + // to an operator in the node-local cutover roster. + operatorAddresses := wallet.signingGroupOperators + sessionMismatchObserver := func( + protocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + signingLogger, + se.announcerMismatchLogLimiter, + se.metricsRecorder, + se.cutoverPeerRoster, + currentMode, + currentParticipationGateState(se.participationGate), + operatorAddresses, + protocolID, + sender, + expectedFormat, + observedFormat, + ) + } + announcer := announcer.New( fmt.Sprintf("%v-%v", ProtocolName, "signing"), se.broadcastChannel, se.membershipValidator, + announcer.WithSessionMismatchObserver(sessionMismatchObserver), ) doneCheck := newSigningDoneCheck( @@ -254,6 +441,7 @@ func (se *signingExecutor) sign( retryLoop := newSigningRetryLoop( signingLogger, message, + mode, startBlock, signer.signingGroupMemberIndex, wallet.signingGroupOperators, @@ -313,17 +501,11 @@ func (se *signingExecutor) sign( se.waitForBlockFn, ) - sessionID := fmt.Sprintf( - "%v-%v", - message.Text(16), - attempt.number, - ) - result, err := signing.Execute( attemptCtx, signingAttemptLogger, message, - sessionID, + attempt.sessionID, signer.signingGroupMemberIndex, signer.privateKeyShare, wallet.groupSize(), @@ -333,6 +515,7 @@ func (se *signingExecutor) sign( attempt.excludedMembersIndexes, se.broadcastChannel, se.membershipValidator, + strategies, ) if err != nil { return nil, 0, err @@ -394,7 +577,10 @@ func (se *signingExecutor) sign( signingOutcomeChan <- &signingOutcome{ signature: loopResult.result.Signature, activityReport: loopResult.activityReport, - endBlock: loopResult.latestEndBlock, + contribution: se.transcriptContribution( + loopResult.resultSigners, + ), + endBlock: loopResult.latestEndBlock, } }(currentSigner) } @@ -415,8 +601,19 @@ func (se *signingExecutor) sign( se.metricsRecorder.IncrementCounter(clientinfo.MetricSigningSuccessTotal, 1) se.metricsRecorder.RecordDuration(clientinfo.MetricSigningDurationSeconds, time.Since(startTime)) } - return outcome.signature, outcome.activityReport, outcome.endBlock, nil + return outcome, nil default: + // A gate decision — clock failure, forced quiescence, or a closed + // permit — canceled the signing; it is not an ordinary protocol + // failure or timeout and must not increment the ordinary failure + // metrics. The gate records the abort in its own metrics; the wrapped + // cause lets every caller layer classify the outcome the same way. + if cause := context.Cause(ctx); participation.IsGateRefusal(cause) { + return nil, fmt.Errorf( + "signing canceled by the participation gate: %w", + cause, + ) + } if se.metricsRecorder != nil { // All signers failed to produce a signature within the timeout period. // This is counted as both a failure and a timeout. @@ -427,7 +624,7 @@ func (se *signingExecutor) sign( se.metricsRecorder.IncrementCounter(clientinfo.MetricSigningTimeoutsTotal, 1) se.metricsRecorder.RecordDuration(clientinfo.MetricSigningDurationSeconds, time.Since(startTime)) } - return nil, nil, 0, fmt.Errorf("all signers failed") + return nil, fmt.Errorf("all signers failed") } } diff --git a/pkg/tbtc/signing_cutover_integration_test.go b/pkg/tbtc/signing_cutover_integration_test.go new file mode 100644 index 0000000000..245dbc3870 --- /dev/null +++ b/pkg/tbtc/signing_cutover_integration_test.go @@ -0,0 +1,950 @@ +package tbtc + +// This file carries the in-repository part of the tBTC signing cutover +// acceptance evidence: real local network providers, the production announcer +// and retry logic, real participation gates clocked by a local chain, and +// completed homogeneous tECDSA transcripts in both legacy and security-v2 +// modes. Exact-image bidirectional mixed-binary evidence remains part of the +// release rehearsal. + +import ( + "context" + "crypto/ecdsa" + "errors" + "fmt" + "math/big" + "slices" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/keep-network/keep-core/internal/testutils" + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/clientinfo" + "github.com/keep-network/keep-core/pkg/generator" + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/announcer" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" + "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" + "github.com/keep-network/keep-core/pkg/tecdsa" + "github.com/keep-network/keep-core/pkg/tecdsa/signing" +) + +// newChainWaitForBlockFn builds a waitForBlockFn over the given block counter, +// mirroring the production node's implementation. +func newChainWaitForBlockFn(blockCounter chain.BlockCounter) waitForBlockFn { + return func(ctx context.Context, blockHeight uint64) error { + waiter, err := blockCounter.BlockHeightWaiter(blockHeight) + if err != nil { + return err + } + + select { + case <-waiter: + return nil + case <-ctx.Done(): + return ctx.Err() + } + } +} + +// startPeerAnnouncer simulates one remote peer of a ceremony: it keeps +// announcing the given member's participation with each of the given session +// IDs on its own broadcast channel instance until ctx is done. A peer stuck on +// a fixed wire format — a prior-release binary after the cutover block — is +// modeled by announcing that format's session IDs. +func startPeerAnnouncer( + ctx context.Context, + t *testing.T, + provider net.Provider, + channelName string, + membershipValidator *group.MembershipValidator, + protocolID string, + memberIndex group.MemberIndex, + sessionIDs []string, +) { + t.Helper() + + channel, err := provider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + peerAnnouncer := announcer.New(protocolID, channel, membershipValidator) + + for _, sessionID := range sessionIDs { + go func(sessionID string) { + for { + announceCtx, cancelAnnounceCtx := context.WithTimeout( + ctx, + 10*local.RetransmissionTick, + ) + // A canceled announcement is this goroutine's exit signal, + // not an error. + _, _ = peerAnnouncer.Announce(announceCtx, memberIndex, sessionID) + cancelAnnounceCtx() + + select { + case <-ctx.Done(): + return + default: + } + } + }(sessionID) + } +} + +// TestSigningCutover_HomogeneousSecurityV2AfterCutover proves smoke-gate case +// 9.2.2/9.2.4: a homogeneous R1 cohort whose wallet action is canonically +// anchored at the cutover block signs successfully in security-v2 mode with +// the production announcer and retry logic, even though the local callback +// height is already past the cutover block, and the completion fence admits +// the terminal commit. +func TestSigningCutover_HomogeneousSecurityV2AfterCutover(t *testing.T) { + testSigningCutoverHomogeneous(t, participation.ModeSecurityV2) +} + +// TestSigningCutover_HomogeneousLegacyAfterCutover proves that an R1 signing +// cohort selected by a pre-cutover canonical anchor completes with the +// historical transcript and session-ID format after the chain crosses the +// cutover. Dependency-level transcript regression tests pin those legacy +// challenges to the prior production formulas; exact-image bidirectional +// prior/R1 interoperability remains a release rehearsal gate. +func TestSigningCutover_HomogeneousLegacyAfterCutover(t *testing.T) { + testSigningCutoverHomogeneous(t, participation.ModeLegacy) +} + +func testSigningCutoverHomogeneous( + t *testing.T, + mode participation.ProtocolMode, +) { + executor, localChain := setupSigningExecutorWithChain(t) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + // Cross the cutover block before the ceremony starts. The canonical + // anchor is the cutover block itself while the current height is already + // past it — the late-confirmation shape of a post-cutover event. + cutoverBlock := uint64(2) + if err := blockCounter.WaitForBlockHeight(cutoverBlock + 1); err != nil { + t.Fatal(err) + } + + gate := newTestGateWithCutover(t, blockCounter, cutoverBlock) + anchor := cutoverBlock + if mode == participation.ModeLegacy { + anchor = cutoverBlock - 1 + } + + permit, err := gate.Begin(participation.TBTCSigning, anchor) + if err != nil { + t.Fatal(err) + } + testutils.AssertStringsEqual( + t, + "permit mode for the canonical anchor", + mode.String(), + permit.Mode().String(), + ) + + message := big.NewInt(100) + + outcome, err := executor.sign( + permit.Context(), + message, + 0, + permit.Mode(), + ) + if err != nil { + t.Fatal(err) + } + + walletPublicKey := executor.wallet().publicKey + if !ecdsa.Verify( + walletPublicKey, + message.Bytes(), + outcome.signature.R, + outcome.signature.S, + ) { + t.Errorf("invalid signature: [%+v]", outcome.signature) + } + if outcome.endBlock == 0 { + t.Error("expected a nonzero end block") + } + + // The transcript travels with the signature: the memberships whose + // authenticated done checks carried it, and the one this node operated. + // Without it the ceremony's terminal record could say a threshold result + // exists and not which parties reached it. + // This executor operates every membership of the group, so the memberships + // whose done checks carried the signature and the ones it operated are the + // same set — the attempt's members, an honest majority or more. + if outcome.contribution == nil || + len(outcome.contribution.IncorporatedMembers) < + executor.groupParameters.HonestThreshold || + !slices.Equal( + outcome.contribution.LocalMembers, + outcome.contribution.IncorporatedMembers, + ) { + t.Errorf( + "unexpected transcript behind the signature: %+v", + outcome.contribution, + ) + } + + if err := permit.CheckCommit( + "tbtc_signing_test_completion", + participation.CompletionCommit, + ); err != nil { + t.Errorf("expected the completion fence to admit the commit: [%v]", err) + } + + permit.Close() + + snapshot := gate.State() + testutils.AssertUintsEqual( + t, + "active ceremonies after the permit release", + 0, + snapshot.ActiveCeremonies, + ) +} + +// TestSigningCutover_LegacyAnchorPinnedThroughRetriesAcrossCutover proves the +// mode-pinning half of smoke-gate case 9.2.3: a wallet action canonically +// anchored below the cutover block keeps the legacy mode through every retry +// attempt — the production retry loop derives the exact prior-release session +// ID for an attempt starting at or after the cutover block, legacy peers +// announcing those IDs stay ready, the permit's mode never mutates while the +// process state is already open_security_v2, and the legacy completion commit +// remains admitted while a new penalty commit is refused. Completed +// cryptographic legacy interoperability is covered by the homogeneous legacy +// test above. +func TestSigningCutover_LegacyAnchorPinnedThroughRetriesAcrossCutover(t *testing.T) { + // The group size equals the honest threshold so the loop's member-count + // trimming cannot exclude the local member: every ready member is needed + // for every attempt. + groupParameters := &GroupParameters{ + GroupSize: 2, + GroupQuorum: 2, + HonestThreshold: 2, + } + + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + + localChain := ConnectWithKey(operatorPrivateKey, 50*time.Millisecond) + localProvider := local.ConnectWithKey(operatorPublicKey) + + operatorAddress, err := localChain.Signing().PublicKeyToAddress( + operatorPublicKey, + ) + if err != nil { + t.Fatal(err) + } + + var operators []chain.Address + for i := 0; i < groupParameters.GroupSize; i++ { + operators = append(operators, operatorAddress) + } + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + cutoverBlock := anchor + 2 + + gate := newTestGateWithCutover(t, blockCounter, cutoverBlock) + + permit, err := gate.Begin(participation.TBTCSigning, anchor) + if err != nil { + t.Fatal(err) + } + testutils.AssertStringsEqual( + t, + "permit mode for the pre-cutover anchor", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + + message := big.NewInt(2211) + protocolID := fmt.Sprintf("%v-%v", ProtocolName, "signing") + channelName := "signing-cutover-pin-test" + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + operators, + localChain.Signing(), + ) + + channel, err := localProvider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + // The legacy peers announce the exact prior-release session IDs of the + // first two attempts, exactly as a prior binary would for this message. + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + legacySessionIDs := []string{ + compatibility.Legacy().SigningSessionID(message, 0, 1), + compatibility.Legacy().SigningSessionID(message, 0, 2), + } + startPeerAnnouncer( + peersCtx, + t, + localProvider, + channelName, + membershipValidator, + protocolID, + group.MemberIndex(2), + legacySessionIDs, + ) + + loopAnnouncer := announcer.New(protocolID, channel, membershipValidator) + + expectedResult := &signing.Result{ + Signature: &tecdsa.Signature{R: big.NewInt(1), S: big.NewInt(2)}, + } + doneCheck := &mockSigningDoneCheck{ + waitUntilAllDoneOutcomeFn: func( + attemptNumber uint64, + ) (*signing.Result, uint64, error) { + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + return nil, 0, err + } + return expectedResult, currentBlock, nil + }, + } + + retryLoop := newSigningRetryLoop( + logger, + message, + permit.Mode(), + anchor, + group.MemberIndex(1), + operators, + groupParameters, + loopAnnouncer, + doneCheck, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 60*time.Second, + ) + defer cancelLoopCtx() + + var attemptSessionIDs []string + var attemptStartBlocks []uint64 + + result, err := retryLoop.start( + loopCtx, + newChainWaitForBlockFn(blockCounter), + blockCounter.CurrentBlock, + func(attempt *signingAttemptParams) (*signing.Result, uint64, error) { + attemptSessionIDs = append(attemptSessionIDs, attempt.sessionID) + attemptStartBlocks = append(attemptStartBlocks, attempt.startBlock) + + // The first attempt fails so the loop retries at a start block + // that is unambiguously at or after the cutover block. + if len(attemptSessionIDs) == 1 { + return nil, 0, fmt.Errorf("simulated first-attempt failure") + } + + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + return nil, 0, err + } + return expectedResult, currentBlock, nil + }, + ) + cancelPeers() + if err != nil { + t.Fatal(err) + } + if result.result != expectedResult { + t.Error("expected the second attempt's result") + } + + testutils.AssertIntsEqual(t, "attempts", 2, len(attemptSessionIDs)) + for i, sessionID := range attemptSessionIDs { + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt %d session ID", i+1), + fmt.Sprintf("%v-%v", message.Text(16), i+1), + sessionID, + ) + testutils.AssertStringsEqual( + t, + fmt.Sprintf("attempt %d session ID format", i+1), + announcer.SessionIDFormatLegacy.String(), + announcer.ClassifySessionIDFormat(sessionID).String(), + ) + } + + if attemptStartBlocks[1] < cutoverBlock { + t.Errorf( + "expected the retry attempt to start at or after the cutover "+ + "block [%d], got [%d]", + cutoverBlock, + attemptStartBlocks[1], + ) + } + + // The permit's mode never mutated even though the process state has + // crossed to open_security_v2. + testutils.AssertStringsEqual( + t, + "permit mode after crossing the cutover block", + participation.ModeLegacy.String(), + permit.Mode().String(), + ) + snapshot := gate.State() + testutils.AssertStringsEqual( + t, + "gate state after crossing the cutover block", + participation.StateOpenSecurityV2.String(), + snapshot.State.String(), + ) + + // A legacy completion after the cutover block is admitted; a new legacy + // penalty is not. + if err := permit.CheckCommit( + "tbtc_signing_test_completion", + participation.CompletionCommit, + ); err != nil { + t.Errorf("expected the legacy completion to be admitted: [%v]", err) + } + if err := permit.CheckCommit( + "tbtc_signing_test_penalty", + participation.PenaltyCommit, + ); !errors.Is(err, participation.ErrPenaltySuppressed) { + t.Errorf("expected the legacy penalty to be suppressed, got [%v]", err) + } + + permit.Close() +} + +// TestSigningCutover_PostCutoverSplitFailsClosedWithEvidence proves smoke-gate +// case 9.2.5 end to end through the production signing executor: a post-cutover +// wallet action whose signing group is split between security-v2 members and +// prior-release peers that keep announcing legacy session IDs exhausts its +// retries below the signing threshold, returns no signature, and turns the +// stragglers into mismatch metrics and node-local cutover roster evidence +// attributed to their operator. +func TestSigningCutover_PostCutoverSplitFailsClosedWithEvidence(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + } + + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + + localChain := ConnectWithKey(operatorPrivateKey, 50*time.Millisecond) + localProvider := local.ConnectWithKey(operatorPublicKey) + + operatorAddress, err := localChain.Signing().PublicKeyToAddress( + operatorPublicKey, + ) + if err != nil { + t.Fatal(err) + } + + // The membership validator resolves wire senders through the local + // chain's signing, whose addresses are raw public keys rather than + // 20-byte Ethereum addresses. The roster inventory key is a normalized + // Ethereum address, so the signers' operator list — the roster + // attribution source — carries a proper address for the same seats. + var operators []chain.Address + rosterOperatorAddress := chain.Address( + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ) + var rosterOperators []chain.Address + for i := 0; i < groupParameters.GroupSize; i++ { + operators = append(operators, operatorAddress) + rosterOperators = append(rosterOperators, rosterOperatorAddress) + } + + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures( + groupParameters.GroupSize, + ) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + + // The local node controls only two of the five signers — below the + // signing threshold on its own. + signers := make([]*signer, 2) + for i := range signers { + privateKeyShare := tecdsa.NewPrivateKeyShare(testData[i]) + signers[i] = &signer{ + wallet: wallet{ + publicKey: privateKeyShare.PublicKey(), + signingGroupOperators: rosterOperators, + }, + signingGroupMemberIndex: group.MemberIndex(i + 1), + privateKeyShare: privateKeyShare, + } + } + + channelName := "signing-cutover-split-test" + channel, err := localProvider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + signing.RegisterUnmarshallers(channel) + announcer.RegisterUnmarshaller(channel) + channel.SetUnmarshaler(func() net.TaggedUnmarshaler { + return &signingDoneMessage{} + }) + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + operators, + localChain.Signing(), + ) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + + currentBlock, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + permit, err := gate.Begin(participation.TBTCSigning, currentBlock) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + testutils.AssertStringsEqual( + t, + "permit mode after the cutover", + participation.ModeSecurityV2.String(), + permit.Mode().String(), + ) + + executor := newSigningExecutor( + signers, + channel, + membershipValidator, + groupParameters, + generator.NewProtocolLatch(), + blockCounter.CurrentBlock, + newChainWaitForBlockFn(blockCounter), + 2, + gate, + ) + + recorder := newDispatcherMetricsRecorder() + executor.setMetricsRecorder(recorder) + + roster, err := participation.NewCutoverPeerRoster( + context.Background(), + blockCounter, + 1500, + newCutoverFakeMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(roster.Close) + executor.setCutoverPeerRoster(roster) + + // The prior-release peers keep announcing the legacy session IDs of the + // first two attempts after the cutover block. + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + message := big.NewInt(3344) + legacySessionIDs := []string{ + compatibility.Legacy().SigningSessionID(message, 0, 1), + compatibility.Legacy().SigningSessionID(message, 0, 2), + } + for _, memberIndex := range []group.MemberIndex{3, 4, 5} { + startPeerAnnouncer( + peersCtx, + t, + localProvider, + channelName, + membershipValidator, + fmt.Sprintf("%v-%v", ProtocolName, "signing"), + memberIndex, + legacySessionIDs, + ) + } + + outcome, err := executor.sign( + permit.Context(), + message, + currentBlock+2, + permit.Mode(), + ) + cancelPeers() + + if err == nil || !strings.Contains(err.Error(), "all signers failed") { + t.Fatalf("expected the retries to exhaust below threshold, got [%v]", err) + } + if outcome != nil { + t.Errorf("expected no signing outcome, got [%+v]", outcome) + } + + // The failure is an ordinary signing failure of the split cohort. + testutils.AssertIntsEqual( + t, + "ordinary signing failures", + 1, + int(recorder.counter(clientinfo.MetricSigningFailedTotal)), + ) + + // The legacy stragglers became mismatch and cross-format evidence. + if mismatches := recorder.counter( + clientinfo.MetricAnnouncerSessionIDMismatchTotal, + ); mismatches < 3 { + t.Errorf( + "expected at least three session ID mismatches, got [%v]", + mismatches, + ) + } + if crossFormat := recorder.counter( + clientinfo.MetricAnnouncerCrossFormatPeerTotal, + ); crossFormat < 3 { + t.Errorf( + "expected at least three cross-format peers, got [%v]", + crossFormat, + ) + } + + // The roster deduplicates the three seats to their one operator and + // retains the per-seat sightings. + rosterSnapshot := roster.Snapshot() + testutils.AssertIntsEqual( + t, + "cutover roster operators", + 1, + len(rosterSnapshot.Peers), + ) + testutils.AssertStringsEqual( + t, + "roster operator address", + string(rosterOperatorAddress), + rosterSnapshot.Peers[0].OperatorAddress, + ) + if sightings := len(rosterSnapshot.Peers[0].Sightings); sightings < 3 { + t.Errorf("expected at least three sightings, got [%d]", sightings) + } +} + +// TestSigningCutover_LoopNeverInvokesSigningBelowThreshold proves the +// never-without-quorum half of smoke-gate case 9.2.5 at the retry-loop level: +// with the ready cohort below the signing threshold, the production retry loop +// never invokes the signing protocol at all, and every legacy peer is reported +// through the production mismatch handler into metrics and the node-local +// roster. +func TestSigningCutover_LoopNeverInvokesSigningBelowThreshold(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + } + + operatorPrivateKey, operatorPublicKey, err := operator.GenerateKeyPair( + local_v1.DefaultCurve, + ) + if err != nil { + t.Fatal(err) + } + + localChain := ConnectWithKey(operatorPrivateKey, 50*time.Millisecond) + localProvider := local.ConnectWithKey(operatorPublicKey) + + operatorAddress, err := localChain.Signing().PublicKeyToAddress( + operatorPublicKey, + ) + if err != nil { + t.Fatal(err) + } + + // As in the executor-level split test, the wire identities come from the + // local chain's signing while the roster attribution uses a proper + // normalized Ethereum address for the same seats. + var operators []chain.Address + rosterOperatorAddress := chain.Address( + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + ) + var rosterOperators []chain.Address + for i := 0; i < groupParameters.GroupSize; i++ { + operators = append(operators, operatorAddress) + rosterOperators = append(rosterOperators, rosterOperatorAddress) + } + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + + anchor, err := blockCounter.CurrentBlock() + if err != nil { + t.Fatal(err) + } + + permit, err := gate.Begin(participation.TBTCSigning, anchor) + if err != nil { + t.Fatal(err) + } + defer permit.Close() + + message := big.NewInt(4455) + protocolID := fmt.Sprintf("%v-%v", ProtocolName, "signing") + channelName := "signing-cutover-threshold-test" + + membershipValidator := group.NewMembershipValidator( + &testutils.MockLogger{}, + operators, + localChain.Signing(), + ) + + channel, err := localProvider.BroadcastChannelFor(channelName) + if err != nil { + t.Fatal(err) + } + announcer.RegisterUnmarshaller(channel) + + recorder := newDispatcherMetricsRecorder() + roster, err := participation.NewCutoverPeerRoster( + context.Background(), + blockCounter, + 1500, + newCutoverFakeMetrics(), + ) + if err != nil { + t.Fatal(err) + } + t.Cleanup(roster.Close) + + // The observer is wired exactly like the production signing executor + // wires it. + currentMode := permit.Mode() + loopAnnouncer := announcer.New( + protocolID, + channel, + membershipValidator, + announcer.WithSessionMismatchObserver(func( + observedProtocolID string, + sender group.MemberIndex, + expectedFormat announcer.SessionIDFormat, + observedFormat announcer.SessionIDFormat, + ) { + handleAnnouncerSessionMismatch( + logger, + nil, + recorder, + roster, + currentMode, + gate.State().State.String(), + rosterOperators, + observedProtocolID, + sender, + expectedFormat, + observedFormat, + ) + }), + ) + + peersCtx, cancelPeers := context.WithCancel(context.Background()) + defer cancelPeers() + + legacySessionIDs := []string{ + compatibility.Legacy().SigningSessionID(message, 0, 1), + compatibility.Legacy().SigningSessionID(message, 0, 2), + } + for _, memberIndex := range []group.MemberIndex{4, 5} { + startPeerAnnouncer( + peersCtx, + t, + localProvider, + channelName, + membershipValidator, + protocolID, + memberIndex, + legacySessionIDs, + ) + } + + retryLoop := newSigningRetryLoop( + logger, + message, + permit.Mode(), + anchor, + group.MemberIndex(1), + operators, + groupParameters, + loopAnnouncer, + &mockSigningDoneCheck{}, + ) + + loopCtx, cancelLoopCtx := context.WithTimeout( + context.Background(), + 8*time.Second, + ) + defer cancelLoopCtx() + + var attemptCalls atomic.Uint64 + + _, err = retryLoop.start( + loopCtx, + newChainWaitForBlockFn(blockCounter), + blockCounter.CurrentBlock, + func(attempt *signingAttemptParams) (*signing.Result, uint64, error) { + attemptCalls.Add(1) + return nil, 0, fmt.Errorf("must never be reached") + }, + ) + cancelPeers() + + if err == nil { + t.Fatal("expected the loop to end without a result") + } + testutils.AssertUintsEqual( + t, + "signing protocol invocations below threshold", + 0, + attemptCalls.Load(), + ) + + // Both legacy peers were reported into metrics and the roster. + if mismatches := recorder.counter( + clientinfo.MetricAnnouncerSessionIDMismatchTotal, + ); mismatches < 2 { + t.Errorf("expected at least two mismatches, got [%v]", mismatches) + } + if crossFormat := recorder.counter( + clientinfo.MetricAnnouncerCrossFormatPeerTotal, + ); crossFormat < 2 { + t.Errorf("expected at least two cross-format peers, got [%v]", crossFormat) + } + + rosterSnapshot := roster.Snapshot() + testutils.AssertIntsEqual( + t, + "cutover roster operators", + 1, + len(rosterSnapshot.Peers), + ) + + sightedMembers := make(map[group.MemberIndex]bool) + for _, sighting := range rosterSnapshot.Peers[0].Sightings { + sightedMembers[sighting.MemberIndex] = true + } + if !sightedMembers[4] || !sightedMembers[5] { + t.Errorf( + "expected sightings for members 4 and 5, got [%v]", + rosterSnapshot.Peers[0].Sightings, + ) + } +} + +// TestSigningCutover_GateQuiesceAbortSkipsOrdinaryFailureMetrics proves +// smoke-gate case 9.2.6 with a real gate: a signing canceled by the gate's +// forced quiesce deadline surfaces the gate sentinel and increments neither +// the ordinary signing failure nor the timeout counter. +func TestSigningCutover_GateQuiesceAbortSkipsOrdinaryFailureMetrics(t *testing.T) { + executor, localChain := setupSigningExecutorWithChain(t) + + recorder := newDispatcherMetricsRecorder() + executor.setMetricsRecorder(recorder) + + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + if err := blockCounter.WaitForBlockHeight(1); err != nil { + t.Fatal(err) + } + + gate := newTestGate(t, blockCounter) + + permit, err := gate.Begin(participation.TBTCSigning, 1) + if err != nil { + t.Fatal(err) + } + + // Quiescence begins and the shutdown deadline arrives while the permit is + // still active: the gate force-cancels it. + quiesceDone := gate.Quiesce(fmt.Errorf("rollback drill")) + gate.Close() + <-quiesceDone + + _, err = executor.sign( + permit.Context(), + big.NewInt(555), + 0, + permit.Mode(), + ) + if !errors.Is(err, participation.ErrQuiesceDeadline) { + t.Fatalf("expected the gate sentinel, got [%v]", err) + } + + testutils.AssertIntsEqual( + t, + "signing operations", + 1, + int(recorder.counter(clientinfo.MetricSigningOperationsTotal)), + ) + testutils.AssertIntsEqual( + t, + "ordinary signing failures", + 0, + int(recorder.counter(clientinfo.MetricSigningFailedTotal)), + ) + testutils.AssertIntsEqual( + t, + "ordinary signing timeouts", + 0, + int(recorder.counter(clientinfo.MetricSigningTimeoutsTotal)), + ) +} diff --git a/pkg/tbtc/signing_done.go b/pkg/tbtc/signing_done.go index 0f88b7d1bf..9e7f69f949 100644 --- a/pkg/tbtc/signing_done.go +++ b/pkg/tbtc/signing_done.go @@ -4,11 +4,13 @@ import ( "context" "fmt" "math/big" + "slices" "sync" "time" "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/signing" ) @@ -50,11 +52,51 @@ type signingDoneCheck struct { broadcastChannel net.BroadcastChannel membershipValidator *group.MembershipValidator - receiveCtx context.Context - cancelReceiveCtx context.CancelFunc - expectedSignersCount int - doneSigners map[group.MemberIndex]*signingDoneMessage - doneSignersMutex sync.Mutex + // receiveCtx, cancelReceiveCtx and attempt hold the state of the attempt + // this check is currently working on. They are written by listen and read + // by waitUntilAllDone, which the retry loop calls one after the other on + // its own goroutine; no other goroutine touches them. + receiveCtx context.Context + cancelReceiveCtx context.CancelFunc + attempt *signingDoneAttempt +} + +// signingDoneAttempt is the done-check state of a single signing attempt: the +// memberships the attempt selected and the done messages received from them. +// +// It is owned by the listen call that created it, and that call's listener +// goroutine closes over it, because an attempt's listener is stopped by +// cancelling its context and the next attempt does not wait for it to notice. +// A listener still draining buffered messages from the previous attempt would +// otherwise write into state the current attempt installed — filling a fresh +// map with a finished attempt's messages, and validating them against the +// wrong attempt's party set. +type signingDoneAttempt struct { + // members are the memberships the attempt selected to sign, and the only + // senders whose done messages describe its transcript. Immutable once the + // attempt is constructed, so it needs no lock. + members []group.MemberIndex + + doneSigners map[group.MemberIndex]*signingDoneMessage + doneSignersMutex sync.Mutex +} + +// recordDoneSigner stores a validated done message under its sender. +func (sda *signingDoneAttempt) recordDoneSigner(doneMessage *signingDoneMessage) { + sda.doneSignersMutex.Lock() + defer sda.doneSignersMutex.Unlock() + + sda.doneSigners[doneMessage.senderID] = doneMessage +} + +// isDoneSigner reports whether a done message from the given sender was +// already accepted for this attempt. +func (sda *signingDoneAttempt) isDoneSigner(senderID group.MemberIndex) bool { + sda.doneSignersMutex.Lock() + defer sda.doneSignersMutex.Unlock() + + _, done := sda.doneSigners[senderID] + return done } func newSigningDoneCheck( @@ -70,17 +112,19 @@ func newSigningDoneCheck( } // listen runs the signing done check listening routine. This function listens -// for incoming signing done checks from members participating in the given -// signing attempt. Messages are filtered out based on the attempt number. Only -// one message for the given attempt can be sent by the given signing group -// member. This function should be called before the signing attempt starts to -// ensure signing done messages are getting received as early as possible. This -// is especially important when the current member is the slowest one with -// executing the signing. +// for incoming signing done checks from the members the given signing attempt +// selected. Messages are filtered out based on the sending membership, the +// attempt number, and the attempt's own protocol window. Only one message for +// the given attempt can be sent by the given signing group member. This +// function should be called before the signing attempt starts to ensure signing +// done messages are getting received as early as possible. This is especially +// important when the current member is the slowest one with executing the +// signing. func (sdc *signingDoneCheck) listen( ctx context.Context, message *big.Int, attemptNumber uint64, + attemptStartBlock uint64, attemptTimeoutBlock uint64, attemptMembersIndexes []group.MemberIndex, ) { @@ -88,16 +132,22 @@ func (sdc *signingDoneCheck) listen( // consuming goroutine are closed when the `waitUntilAllDone` completes its // work. Leaving a dangling receiver without the message processing loop // causes warnings on the channel level. - sdc.receiveCtx, sdc.cancelReceiveCtx = context.WithCancel(ctx) + receiveCtx, cancelReceiveCtx := context.WithCancel(ctx) + sdc.receiveCtx, sdc.cancelReceiveCtx = receiveCtx, cancelReceiveCtx messagesChan := make(chan net.Message, signingDoneReceiveBuffer) - sdc.broadcastChannel.Recv(sdc.receiveCtx, func(message net.Message) { + sdc.broadcastChannel.Recv(receiveCtx, func(message net.Message) { messagesChan <- message }) - sdc.expectedSignersCount = len(attemptMembersIndexes) - sdc.doneSigners = make(map[group.MemberIndex]*signingDoneMessage) + attempt := &signingDoneAttempt{ + members: slices.Clone(attemptMembersIndexes), + doneSigners: make(map[group.MemberIndex]*signingDoneMessage), + } + sdc.attempt = attempt + // The goroutine works on the attempt and context it was started with rather + // than on the check's fields, which the next attempt's listen replaces. go func() { for { select { @@ -108,20 +158,20 @@ func (sdc *signingDoneCheck) listen( } if !sdc.isValidDoneMessage( + attempt, doneMessage, netMessage.SenderPublicKey(), message, attemptNumber, + attemptStartBlock, attemptTimeoutBlock, ) { continue } - sdc.doneSignersMutex.Lock() - sdc.doneSigners[doneMessage.senderID] = doneMessage - sdc.doneSignersMutex.Unlock() + attempt.recordDoneSigner(doneMessage) - case <-sdc.receiveCtx.Done(): + case <-receiveCtx.Done(): return } } @@ -149,48 +199,81 @@ func (sdc *signingDoneCheck) signalDone( // waitUntilAllDone blocks until it receives all the required done checks from // members or until the passed context is done. In the first case, it returns -// the signature computed by the signing members and the block at which the -// slowest signer completed the signature computation process. If the expected -// done checks are not received on time, the function returns an error. If at -// least one signature is different from others, the function returns an error. +// the signature computed by the signing members, the memberships whose done +// checks carried it, and the block at which the slowest signer completed the +// signature computation process. If the expected done checks are not received +// on time, the function returns an error. If at least one signature is +// different from others, the function returns an error. +// +// The returned memberships are the local view of who produced the signature, +// and the only one this node has. Every done check counted here was +// authenticated against the wallet's on-chain signing group, sent by a +// membership this attempt selected, ended inside this attempt's own protocol +// window, and carried a signature equal to every other — so a membership in that +// list confirmed this exact result under an identity the chain accounts for, and +// a membership absent from it confirmed nothing about it. That distinction is +// what a reader otherwise has to take from whichever party wrote the report: a +// completed ceremony reads identically whether its shares came from several +// parties or one party recovered the common result alone. +// +// It is an attestation by each named membership rather than a proof that it +// computed a share, and the difference is bounded by what the wire carries. A +// done message names the message, the attempt number, the signature, and the +// block its sender finished at; nothing in it is derived from this attempt's +// protocol transcript. The window check is therefore what separates the runs: it +// refuses the earlier run's messages, whose end blocks lie outside this attempt's +// window, whether they were honestly retransmitted or replayed by anybody who +// captured them. A selected membership choosing to assert an in-window end block +// over an output it did not compute is not separated by it, and separating it +// would mean binding the message to a session the prior release does not +// compute — the wire change a compatibility release cannot make. func (sdc *signingDoneCheck) waitUntilAllDone(ctx context.Context) ( *signing.Result, + participation.MemberIndexes, uint64, error, ) { defer sdc.cancelReceiveCtx() + attempt := sdc.attempt + ticker := time.NewTicker(signingDoneCheckInterval) defer ticker.Stop() for { select { case <-ctx.Done(): - return nil, 0, errWaitDoneTimedOut + return nil, nil, 0, errWaitDoneTimedOut case <-ticker.C: - result, endBlock, done, err := func() ( + result, signers, endBlock, done, err := func() ( *signing.Result, + participation.MemberIndexes, uint64, bool, error, ) { - sdc.doneSignersMutex.Lock() - defer sdc.doneSignersMutex.Unlock() + attempt.doneSignersMutex.Lock() + defer attempt.doneSignersMutex.Unlock() - if sdc.expectedSignersCount != len(sdc.doneSigners) { - return nil, 0, false, nil + if len(attempt.members) != len(attempt.doneSigners) { + return nil, nil, 0, false, nil } var signature *tecdsa.Signature var latestEndBlock uint64 + signers := make( + participation.MemberIndexes, + 0, + len(attempt.doneSigners), + ) - for _, doneMessage := range sdc.doneSigners { + for senderID, doneMessage := range attempt.doneSigners { if signature == nil { signature = doneMessage.signature } else { if !signature.Equals(doneMessage.signature) { - return nil, 0, true, fmt.Errorf( + return nil, nil, 0, true, fmt.Errorf( "not matching signatures detected: [%v] and [%v]", signature, doneMessage.signature, @@ -201,13 +284,24 @@ func (sdc *signingDoneCheck) waitUntilAllDone(ctx context.Context) ( if doneMessage.endBlock > latestEndBlock { latestEndBlock = doneMessage.endBlock } + + signers = append(signers, senderID) } - return &signing.Result{Signature: signature}, latestEndBlock, true, nil + // Map iteration order is random, and this list identifies a + // transcript in a record two nodes have to agree on. Sorting + // gives one population exactly one rendering. + slices.Sort(signers) + + return &signing.Result{Signature: signature}, + signers, + latestEndBlock, + true, + nil }() if done { - return result, endBlock, err + return result, signers, endBlock, err } } } @@ -216,14 +310,15 @@ func (sdc *signingDoneCheck) waitUntilAllDone(ctx context.Context) ( // isValidDoneMessage validates the given signingDoneMessage in the context // of the given signing attempt. func (sdc *signingDoneCheck) isValidDoneMessage( + attempt *signingDoneAttempt, doneMessage *signingDoneMessage, senderPublicKey []byte, message *big.Int, attemptNumber uint64, + attemptStartBlock uint64, attemptTimeoutBlock uint64, ) bool { - _, signerDone := sdc.doneSigners[doneMessage.senderID] - if signerDone { + if attempt.isDoneSigner(doneMessage.senderID) { // only one done message allowed return false } @@ -235,6 +330,19 @@ func (sdc *signingDoneCheck) isValidDoneMessage( return false } + // A valid wallet membership is not necessarily one of this attempt's + // signers. The members the attempt excluded compute nothing and signal + // nothing, so a done message from one of them attests to a transcript that + // never existed — and the count check in waitUntilAllDone cannot tell the + // difference, because an excluded member's message standing in for a lost + // message from a selected one reaches the expected total over the wrong + // population. The attempt would then conclude on a signature it never + // gathered from its own signers and name a membership that never signed as + // having produced it. + if !slices.Contains(attempt.members, doneMessage.senderID) { + return false + } + if doneMessage.message.Cmp(message) != 0 { return false } @@ -243,7 +351,16 @@ func (sdc *signingDoneCheck) isValidDoneMessage( return false } - if doneMessage.endBlock > attemptTimeoutBlock { + // The end block says when the sender finished computing, and the attempt's + // protocol window says when finishing this attempt was possible. The + // message and attempt number do not identify an attempt on their own: the + // same wallet can be asked to sign the same message again under a later + // canonical anchor, and its attempt numbering restarts from zero. A done + // message left over from such an earlier run — retransmitted past its own + // window, or replayed — describes that run's transcript, and its end block + // lies below the block this attempt's protocol started at. + if doneMessage.endBlock < attemptStartBlock || + doneMessage.endBlock > attemptTimeoutBlock { return false } diff --git a/pkg/tbtc/signing_done_test.go b/pkg/tbtc/signing_done_test.go index d946052584..854ada01d4 100644 --- a/pkg/tbtc/signing_done_test.go +++ b/pkg/tbtc/signing_done_test.go @@ -18,6 +18,7 @@ import ( "github.com/keep-network/keep-core/pkg/net/local" "github.com/keep-network/keep-core/pkg/operator" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/signing" ) @@ -46,6 +47,11 @@ func TestSigningDoneCheck(t *testing.T) { message := big.NewInt(100) attemptNumber := uint64(2) + // The members below exchange `500 + memberIndex` as their end block, so the + // lowest of them sits exactly on the attempt's start block. The window's + // lower edge is inclusive: a member that finished the moment the protocol + // started did finish inside this attempt. + attemptStartBlock := uint64(501) attemptTimeoutBlock := uint64(1000) attemptMemberIndexes := memberIndexes[:groupParameters.HonestThreshold] result := &signing.Result{ @@ -59,6 +65,7 @@ func TestSigningDoneCheck(t *testing.T) { type outcome struct { memberIndex group.MemberIndex result *signing.Result + signers participation.MemberIndexes endBlock uint64 err error } @@ -77,6 +84,7 @@ func TestSigningDoneCheck(t *testing.T) { ctx, message, attemptNumber, + attemptStartBlock, attemptTimeoutBlock, attemptMemberIndexes, ) @@ -96,11 +104,12 @@ func TestSigningDoneCheck(t *testing.T) { } } - result, endBlock, err := doneCheck.waitUntilAllDone(ctx) + result, signers, endBlock, err := doneCheck.waitUntilAllDone(ctx) outcomesChan <- &outcome{ memberIndex: memberIndex, result: result, + signers: signers, endBlock: endBlock, err: err, } @@ -143,6 +152,26 @@ func TestSigningDoneCheck(t *testing.T) { expectedEndBlock, int(outcome.endBlock), ) + + // Every member — including the ones the attempt excluded, which only + // listen — comes away with the same population: the memberships whose + // authenticated done checks carried this signature, in ascending order. + // This is the fact a terminal record needs and a completion cannot + // supply, so a member that could not name it would leave the ceremony's + // participants to whichever party wrote the report. + if !slices.Equal( + outcome.signers, + participation.MemberIndexes(attemptMemberIndexes), + ) { + t.Errorf( + "unexpected done signers for member [%v]\n"+ + "expected: [%v]\n"+ + "actual: [%v]", + outcome.memberIndex, + attemptMemberIndexes, + outcome.signers, + ) + } } } @@ -168,6 +197,7 @@ func TestSigningDoneCheck_MissingConfirmation(t *testing.T) { message := big.NewInt(100) attemptNumber := uint64(1) + attemptStartBlock := uint64(50) attemptTimeoutBlock := uint64(1000) attemptMemberIndexes := memberIndexes[:groupParameters.HonestThreshold] result := &signing.Result{ @@ -182,6 +212,7 @@ func TestSigningDoneCheck_MissingConfirmation(t *testing.T) { ctx, message, attemptNumber, + attemptStartBlock, attemptTimeoutBlock, attemptMemberIndexes, ) @@ -200,11 +231,14 @@ func TestSigningDoneCheck_MissingConfirmation(t *testing.T) { } } - returnedResult, endBlock, err := doneCheck.waitUntilAllDone(ctx) + returnedResult, signers, endBlock, err := doneCheck.waitUntilAllDone(ctx) if returnedResult != nil { t.Errorf("expected nil result, has [%v]", returnedResult) } + if len(signers) != 0 { + t.Errorf("expected no done signers, has [%v]", signers) + } testutils.AssertIntsEqual(t, "end block", 0, int(endBlock)) testutils.AssertErrorsSame(t, errWaitDoneTimedOut, err) } @@ -231,6 +265,7 @@ func TestSigningDoneCheck_AnotherSignature(t *testing.T) { message := big.NewInt(100) attemptNumber := uint64(1) + attemptStartBlock := uint64(50) attemptTimeoutBlock := uint64(1000) attemptMemberIndexes := memberIndexes[:groupParameters.HonestThreshold] correctResult := &signing.Result{ @@ -252,6 +287,7 @@ func TestSigningDoneCheck_AnotherSignature(t *testing.T) { ctx, message, attemptNumber, + attemptStartBlock, attemptTimeoutBlock, attemptMemberIndexes, ) @@ -287,17 +323,342 @@ func TestSigningDoneCheck_AnotherSignature(t *testing.T) { // Give some time for the message handler goroutine time.Sleep(100 * time.Millisecond) - returnedResult, endBlock, err := doneCheck.waitUntilAllDone(ctx) + returnedResult, signers, endBlock, err := doneCheck.waitUntilAllDone(ctx) if returnedResult != nil { t.Errorf("expected nil result, has [%v]", returnedResult) } + // A population is only ever reported alongside a result the whole attempt + // agreed on; members naming different signatures did not produce one + // transcript, so there is nobody to name. + if len(signers) != 0 { + t.Errorf("expected no done signers, has [%v]", signers) + } testutils.AssertIntsEqual(t, "end block", 0, int(endBlock)) if !strings.Contains(err.Error(), "not matching signatures detected") { t.Errorf("unexpected error: [%v]", err) } } +// TestSigningDoneCheck_SenderOutsideTheAttempt covers a done message from a +// signing group member the attempt did not select. +// +// Such a member computes nothing and signals nothing, so its done message +// attests to a transcript that never existed. Counting it is not merely +// untidy bookkeeping: the wait concludes as soon as the number of accepted +// messages reaches the number of selected members, so one message from an +// excluded member standing in for a lost message from a selected one both ends +// the attempt early and names a membership that never signed as having +// produced the signature. That name then travels into the release evidence as +// the population behind the result. +// +// The case is built to reach exactly that count: two of the three selected +// members signal, and an excluded member's message would complete the total. +func TestSigningDoneCheck_SenderOutsideTheAttempt(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + } + + doneCheck := setupSigningDoneCheck(t, groupParameters) + + // The listener outlives the wait on purpose. The barrier below has to be + // reachable on a loaded machine, while the wait's own deadline is the thing + // this test asserts on, so the two get separate budgets instead of one + // deadline that either makes the barrier flaky or the timeout slow. + listenCtx, cancelListenCtx := context.WithTimeout( + context.Background(), + signingDoneListenBudget, + ) + defer cancelListenCtx() + + message := big.NewInt(100) + attemptNumber := uint64(1) + attemptStartBlock := uint64(50) + attemptTimeoutBlock := uint64(1000) + attemptMemberIndexes := []group.MemberIndex{1, 2, 3} + // Member 4 is a valid wallet member the attempt excluded, and it signals + // first: messages reach the check's receive loop in the order they were sent + // and are ruled on one at a time, so a membership accepted after this one is + // evidence that this one was already seen and refused. + signalingMemberIndexes := []group.MemberIndex{4, 1, 2} + result := &signing.Result{ + Signature: &tecdsa.Signature{ + R: big.NewInt(200), + S: big.NewInt(300), + RecoveryID: 2, + }, + } + + doneCheck.listen( + listenCtx, + message, + attemptNumber, + attemptStartBlock, + attemptTimeoutBlock, + attemptMemberIndexes, + ) + + for _, memberIndex := range signalingMemberIndexes { + err := doneCheck.signalDone( + listenCtx, + memberIndex, + message, + attemptNumber, + result, + 100, + ) + if err != nil { + t.Fatal(err) + } + } + + // The two selected members are accepted and the excluded one that preceded + // them is not, which is only readable once the check has ruled on all three. + awaitDoneSigners(t, doneCheck, participation.MemberIndexes{1, 2}) + + waitCtx, cancelWaitCtx := context.WithTimeout( + context.Background(), + signingDoneWaitTimeout, + ) + defer cancelWaitCtx() + + returnedResult, signers, endBlock, err := doneCheck.waitUntilAllDone(waitCtx) + + if returnedResult != nil { + t.Errorf("expected nil result, has [%v]", returnedResult) + } + if len(signers) != 0 { + t.Errorf("expected no done signers, has [%v]", signers) + } + testutils.AssertIntsEqual(t, "end block", 0, int(endBlock)) + testutils.AssertErrorsSame(t, errWaitDoneTimedOut, err) +} + +// TestSigningDoneCheck_DoneChecksFromAnotherRun covers done messages produced by +// an earlier run of the same message under a different canonical anchor. +// +// The message and the attempt number do not identify an attempt: a wallet can be +// asked to sign the same message again from a later coordination window, and the +// new run's attempt numbering restarts from zero. Done messages retransmitted +// past their own window, or replayed, therefore match on both fields while +// describing the earlier run's transcript — a different set of selected members, +// reaching a signature over a different sequence of protocol messages. Accepting +// them would let the release evidence for this attempt be populated by a +// transcript this attempt had no part in. +// +// What separates the two runs is the block window each protocol ran in. The +// earlier run finished before this one's protocol started, so its end blocks lie +// below this attempt's start block. +func TestSigningDoneCheck_DoneChecksFromAnotherRun(t *testing.T) { + groupParameters := &GroupParameters{ + GroupSize: 5, + GroupQuorum: 4, + HonestThreshold: 3, + } + + doneCheck := setupSigningDoneCheck(t, groupParameters) + + listenCtx, cancelListenCtx := context.WithTimeout( + context.Background(), + signingDoneListenBudget, + ) + defer cancelListenCtx() + + message := big.NewInt(100) + attemptNumber := uint64(1) + attemptStartBlock := uint64(900) + attemptTimeoutBlock := uint64(930) + attemptMemberIndexes := []group.MemberIndex{1, 2, 3, 4} + // The memberships whose earlier-run messages are replayed here, and the one + // whose in-window message follows them. The barrier membership sends nothing + // from the earlier run, so the check accepting it and nothing else says + // exactly one thing: every replayed message ahead of it was ruled on and + // refused. + staleMemberIndexes := []group.MemberIndex{1, 2, 3} + barrierMemberIndex := group.MemberIndex(4) + // The end block the earlier run's members finished at, before this attempt's + // protocol started. + staleEndBlock := uint64(130) + // And a block inside this attempt's own window, which is what the check is + // asking the earlier run's messages for. + currentEndBlock := uint64(910) + result := &signing.Result{ + Signature: &tecdsa.Signature{ + R: big.NewInt(200), + S: big.NewInt(300), + RecoveryID: 2, + }, + } + + doneCheck.listen( + listenCtx, + message, + attemptNumber, + attemptStartBlock, + attemptTimeoutBlock, + attemptMemberIndexes, + ) + + for _, memberIndex := range staleMemberIndexes { + err := doneCheck.signalDone( + listenCtx, + memberIndex, + message, + attemptNumber, + result, + staleEndBlock, + ) + if err != nil { + t.Fatal(err) + } + } + + if err := doneCheck.signalDone( + listenCtx, + barrierMemberIndex, + message, + attemptNumber, + result, + currentEndBlock, + ); err != nil { + t.Fatal(err) + } + + awaitDoneSigners( + t, + doneCheck, + participation.MemberIndexes{barrierMemberIndex}, + ) + + waitCtx, cancelWaitCtx := context.WithTimeout( + context.Background(), + signingDoneWaitTimeout, + ) + defer cancelWaitCtx() + + returnedResult, signers, endBlock, err := doneCheck.waitUntilAllDone(waitCtx) + + if returnedResult != nil { + t.Errorf("expected nil result, has [%v]", returnedResult) + } + if len(signers) != 0 { + t.Errorf("expected no done signers, has [%v]", signers) + } + testutils.AssertIntsEqual(t, "end block", 0, int(endBlock)) + testutils.AssertErrorsSame(t, errWaitDoneTimedOut, err) +} + +// How long a refusal test's listener stays up. It is also the barrier's +// deadline: the two are the same budget because a barrier can only observe the +// check ruling on a message while the check is still listening. +// +// Generous because it is never spent. The barrier returns as soon as its +// evidence is in, so this bounds a broken build's failure rather than a working +// build's runtime. +const signingDoneListenBudget = 10 * time.Second + +// How long a refusal test then waits for a conclusion it must not reach. Several +// done-check intervals wide, so the wait actually evaluates the accepted +// population and declines to conclude on it rather than expiring before its +// first tick. +const signingDoneWaitTimeout = 500 * time.Millisecond + +// How often a barrier re-reads the accepted population. This bounds how long a +// test lingers after its evidence has arrived, not how long it may wait for it. +const signingDoneBarrierPollInterval = time.Millisecond + +// awaitDoneSigners blocks until the check has accepted exactly the given +// memberships, and fails the test if it has not before the listener's budget +// runs out. +// +// A refusal is only observable once the check has seen the message it refused, +// and the accepted population reads identically before a message arrives and +// after it was refused. A fixed sleep followed by an assertion about what is +// absent therefore holds either way, and holds most readily on the machine least +// likely to have processed anything — the one running the whole suite at once — +// so it is a test that reports a refusal it never witnessed. +// +// Ordering is what makes the wait deterministic instead. Everything a test +// offers reaches one receive channel in send order and is ruled on by a single +// goroutine, so a membership present in the accepted population is evidence that +// every message sent before it was already accepted or refused. A test therefore +// offers the message it expects to be admitted last, and this function returns +// only on exactly the population it names. +func awaitDoneSigners( + t *testing.T, + doneCheck *signingDoneCheck, + expected participation.MemberIndexes, +) { + t.Helper() + + deadline := time.NewTimer(signingDoneListenBudget) + defer deadline.Stop() + + ticker := time.NewTicker(signingDoneBarrierPollInterval) + defer ticker.Stop() + + for { + actual := recordedDoneSigners(doneCheck) + if slices.Equal(actual, expected) { + return + } + + // The accepted population only ever grows, so a membership outside the + // expected one is already a message the check admitted and had to + // refuse. Saying so here names the defect, where waiting out the + // deadline would only report that the population never settled. + for _, signer := range actual { + if !slices.Contains(expected, signer) { + t.Fatalf( + "the check accepted a done message from membership [%v]\n"+ + "expected: [%v]\n"+ + "actual: [%v]", + signer, + expected, + actual, + ) + } + } + + select { + case <-deadline.C: + t.Fatalf( + "timed out waiting for the accepted done signers\n"+ + "expected: [%v]\n"+ + "actual: [%v]", + expected, + recordedDoneSigners(doneCheck), + ) + case <-ticker.C: + } + } +} + +// recordedDoneSigners returns the memberships whose done messages the check +// accepted for the attempt it is listening for, ascending. A test that is about +// a refusal reads this so it asserts on a message the check saw and rejected +// rather than on one that had not arrived yet. +func recordedDoneSigners( + doneCheck *signingDoneCheck, +) participation.MemberIndexes { + doneCheck.attempt.doneSignersMutex.Lock() + defer doneCheck.attempt.doneSignersMutex.Unlock() + + signers := make( + participation.MemberIndexes, + 0, + len(doneCheck.attempt.doneSigners), + ) + for senderID := range doneCheck.attempt.doneSigners { + signers = append(signers, senderID) + } + slices.Sort(signers) + + return signers +} + // signingDoneCheckComponents holds the shared state used to construct one or // more signingDoneCheck instances that communicate over the same channel. type signingDoneCheckComponents struct { diff --git a/pkg/tbtc/signing_loop.go b/pkg/tbtc/signing_loop.go index 7e787f1975..c09e53b289 100644 --- a/pkg/tbtc/signing_loop.go +++ b/pkg/tbtc/signing_loop.go @@ -13,7 +13,9 @@ import ( "github.com/ipfs/go-log/v2" "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa/retry" "github.com/keep-network/keep-core/pkg/tecdsa/signing" "golang.org/x/exp/slices" @@ -62,6 +64,7 @@ type signingDoneCheckStrategy interface { ctx context.Context, message *big.Int, attemptNumber uint64, + attemptStartBlock uint64, attemptTimeoutBlock uint64, attemptMembersIndexes []group.MemberIndex, ) @@ -75,7 +78,12 @@ type signingDoneCheckStrategy interface { endBlock uint64, ) error - waitUntilAllDone(ctx context.Context) (*signing.Result, uint64, error) + waitUntilAllDone(ctx context.Context) ( + *signing.Result, + participation.MemberIndexes, + uint64, + error, + ) } // signingRetryLoop is a struct that encapsulates the signing retry logic. @@ -84,6 +92,12 @@ type signingRetryLoop struct { message *big.Int + // protocolMode is the ceremony's pinned protocol compatibility mode. A + // retry is a phase of its outer ceremony, so every attempt of this loop — + // including attempts starting at or after the cutover block — derives its + // session ID from this one immutable mode. + protocolMode participation.ProtocolMode + signingGroupMemberIndex group.MemberIndex signingGroupOperators chain.Addresses @@ -101,6 +115,7 @@ type signingRetryLoop struct { func newSigningRetryLoop( logger log.StandardLogger, message *big.Int, + protocolMode participation.ProtocolMode, initialStartBlock uint64, signingGroupMemberIndex group.MemberIndex, signingGroupOperators chain.Addresses, @@ -118,6 +133,7 @@ func newSigningRetryLoop( return &signingRetryLoop{ logger: logger, message: message, + protocolMode: protocolMode, signingGroupMemberIndex: signingGroupMemberIndex, signingGroupOperators: signingGroupOperators, groupParameters: groupParameters, @@ -135,6 +151,35 @@ type signingAttemptParams struct { startBlock uint64 timeoutBlock uint64 excludedMembersIndexes []group.MemberIndex + // sessionID is the GG20 session identifier shared by the announcer and the + // signing protocol for this attempt. Computed once per attempt by the retry + // loop so both sides cannot drift. + sessionID string +} + +// signingAttemptSessionID derives the announcer/protocol session ID of a +// single signing attempt for the given protocol compatibility mode. The exact +// per-mode formats are owned by the compatibility strategy bundle: the legacy +// form is byte-for-byte the pre-hardening production form — it carries no +// attempt start block — so a legacy-mode ceremony interoperates with +// prior-release peers; the security-v2 form carries the protocol name and +// fixed-width start block and attempt so it cannot collide or be replayed +// across protocols or windows. The mode always comes from the ceremony's +// pinned permit mode; there is no implicit default. +func signingAttemptSessionID( + mode participation.ProtocolMode, + message *big.Int, + attemptStartBlock uint64, + attemptNumber uint, +) string { + strategies, err := compatibility.StrategiesFor(mode) + if err != nil { + panic(fmt.Sprintf( + "signingAttemptSessionID: protocol mode not set explicitly: [%v]", + err, + )) + } + return strategies.SigningSessionID(message, attemptStartBlock, attemptNumber) } // signingAttemptFn represents a function performing a signing attempt. @@ -161,6 +206,11 @@ type signingRetryLoopResult struct { // attemptTimeoutBlock is the block at which the successful attempt times // out. attemptTimeoutBlock uint64 + // resultSigners are the memberships whose authenticated done checks carried + // the signature in result, ascending. They are the local view of who + // produced it, which is what distinguishes shares that combined from + // several parties from one party that arrived at the common result alone. + resultSigners participation.MemberIndexes } // start begins the signing retry loop using the given signing attempt function. @@ -257,10 +307,19 @@ func (srl *signingRetryLoop) start( srl.attemptCounter, ) + // Derive the session ID once per attempt so the announcer and the + // signing protocol cannot drift apart. + sessionID := signingAttemptSessionID( + srl.protocolMode, + srl.message, + announcementEndBlock, + srl.attemptCounter, + ) + readyMembersIndexes, err := srl.announcer.Announce( announceCtx, srl.signingGroupMemberIndex, - fmt.Sprintf("%v-%v", srl.message, srl.attemptCounter), + sessionID, ) if err != nil { srl.logger.Warnf( @@ -339,10 +398,16 @@ func (srl *signingRetryLoop) start( // participants have a chance to receive signingDoneMessage. doneCheckTimeoutCtx, _ := withCancelOnBlock(ctx, timeoutBlock, waitForBlockFn) + // The announcement end block is where this attempt's protocol starts, so + // it is the earliest block a member of this attempt can have finished + // at. It bounds the done messages from below the way the timeout bounds + // them from above, which is what ties them to this attempt rather than + // to any run of the same message that carried the same attempt number. srl.doneCheck.listen( doneCheckTimeoutCtx, srl.message, uint64(srl.attemptCounter), + announcementEndBlock, timeoutBlock, includedMembersIndexes, ) @@ -359,6 +424,7 @@ func (srl *signingRetryLoop) start( startBlock: announcementEndBlock, timeoutBlock: timeoutBlock, excludedMembersIndexes: excludedMembersIndexes, + sessionID: sessionID, }) if err != nil { srl.logger.Warnf( @@ -404,7 +470,8 @@ func (srl *signingRetryLoop) start( ) } - result, latestEndBlock, err := srl.doneCheck.waitUntilAllDone(doneCheckTimeoutCtx) + result, resultSigners, latestEndBlock, err := + srl.doneCheck.waitUntilAllDone(doneCheckTimeoutCtx) if err != nil { srl.logger.Warnf( "[member:%v] cannot wait for signing done "+ @@ -426,6 +493,7 @@ func (srl *signingRetryLoop) start( activityReport: activityReport, latestEndBlock: latestEndBlock, attemptTimeoutBlock: timeoutBlock, + resultSigners: resultSigners, }, nil } } diff --git a/pkg/tbtc/signing_loop_test.go b/pkg/tbtc/signing_loop_test.go index 5d9a5caaf6..840882f8e3 100644 --- a/pkg/tbtc/signing_loop_test.go +++ b/pkg/tbtc/signing_loop_test.go @@ -6,12 +6,14 @@ import ( "math" "math/big" "reflect" + "slices" "testing" "time" "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "github.com/keep-network/keep-core/pkg/tecdsa/signing" ) @@ -116,6 +118,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 206, timeoutBlock: 236, // start block of the first attempt + 30 excludedMembersIndexes: []group.MemberIndex{3, 7, 8, 10}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 206, 1), }, outgoingAnnouncementsCount: 1, }, @@ -170,6 +173,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 206, timeoutBlock: 236, // start block of the first attempt + 30 excludedMembersIndexes: []group.MemberIndex{4, 5, 8, 10}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 206, 1), }, outgoingAnnouncementsCount: 1, }, @@ -184,7 +188,7 @@ func TestSigningRetryLoop(t *testing.T) { incomingAnnouncementsFn: func( sessionID string, ) ([]group.MemberIndex, error) { - if sessionID == fmt.Sprintf("%v-%v", message, 1) { + if sessionID == signingAttemptSessionID(participation.ModeSecurityV2, message, 206, 1) { // Minority of members announced their readiness. return []group.MemberIndex{1, 2, 3, 6, 7}, nil } @@ -231,6 +235,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 247, // 206 + 1 * (6 + 30 + 5) timeoutBlock: 277, // start block of the second attempt + 30 excludedMembersIndexes: []group.MemberIndex{1, 2, 5, 9}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 2), }, outgoingAnnouncementsCount: 2, }, @@ -245,7 +250,7 @@ func TestSigningRetryLoop(t *testing.T) { incomingAnnouncementsFn: func( sessionID string, ) ([]group.MemberIndex, error) { - if sessionID == fmt.Sprintf("%v-%v", message, 1) { + if sessionID == signingAttemptSessionID(participation.ModeSecurityV2, message, 206, 1) { return nil, fmt.Errorf("unexpected error") } @@ -291,6 +296,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 247, // 206 + 1 * (6 + 30 + 5) timeoutBlock: 277, // start block of the second attempt + 30 excludedMembersIndexes: []group.MemberIndex{1, 2, 5, 9}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 2), }, outgoingAnnouncementsCount: 2, }, @@ -351,6 +357,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 247, // 206 + 1 * (6 + 30 + 5) timeoutBlock: 277, // start block of the second attempt + 30 excludedMembersIndexes: []group.MemberIndex{1, 2, 5, 9}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 2), }, outgoingAnnouncementsCount: 2, }, @@ -400,6 +407,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 206, timeoutBlock: 236, // start block of the first attempt + 30 excludedMembersIndexes: []group.MemberIndex{3, 7, 8, 10}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 206, 1), }, // The second announcement is done at the beginning of the // second attempt for which member 2 is eventually excluded. @@ -478,6 +486,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 247, // 206 + 1 * (6 + 30 + 5) timeoutBlock: 277, // start block of the second attempt + 30 excludedMembersIndexes: []group.MemberIndex{1, 2, 5, 9}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 2), }, outgoingAnnouncementsCount: 2, }, @@ -587,6 +596,7 @@ func TestSigningRetryLoop(t *testing.T) { startBlock: 247, // 206 + 1 * (6 + 30 + 5) timeoutBlock: 277, // start block of the second attempt + 30 excludedMembersIndexes: []group.MemberIndex{1, 2, 5, 9}, + sessionID: signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 2), }, // just the second announcement, the first one was skipped outgoingAnnouncementsCount: 1, @@ -607,6 +617,7 @@ func TestSigningRetryLoop(t *testing.T) { retryLoop := newSigningRetryLoop( &testutils.MockLogger{}, message, + participation.ModeSecurityV2, 200, test.signingGroupMemberIndex, signingGroupOperators, @@ -663,6 +674,34 @@ func TestSigningRetryLoop(t *testing.T) { } if test.expectedLastExecutedAttempt != nil { + // The done check decides which incoming done messages belong to + // an attempt from the window it was handed, so that window has + // to be the executed attempt's own. Handing over a window from + // another attempt would leave the check admitting done messages + // produced outside the run it is attesting to. + listen := doneCheck.listenFor( + uint64(test.expectedLastExecutedAttempt.number), + ) + if listen == nil { + t.Errorf( + "done check was never asked to listen for attempt [%v]", + test.expectedLastExecutedAttempt.number, + ) + } else { + testutils.AssertIntsEqual( + t, + "done check listen start block", + int(test.expectedLastExecutedAttempt.startBlock), + int(listen.startBlock), + ) + testutils.AssertIntsEqual( + t, + "done check listen timeout block", + int(test.expectedLastExecutedAttempt.timeoutBlock), + int(listen.timeoutBlock), + ) + } + testutils.AssertIntsEqual( t, "outgoing announcements count", @@ -700,7 +739,16 @@ func TestSigningRetryLoop(t *testing.T) { } } -func TestSigningRetryLoop_GetCurrentBlockErrorCausesRetry(t *testing.T) { +// TestSigningRetryLoop_CarriesTheDoneCheckPopulation proves the loop hands its +// caller the memberships the done check reported as having carried the signature, +// unchanged. +// +// The loop is the only place that sees both the result and the population behind +// it, and the ceremony's terminal record is written from what it returns. A loop +// that dropped the population would leave the record able to say a threshold +// result exists and unable to say which parties reached it — the point at which +// a reader has to fall back on whichever party wrote the report. +func TestSigningRetryLoop_CarriesTheDoneCheckPopulation(t *testing.T) { message := big.NewInt(100) groupParameters := &GroupParameters{ @@ -709,227 +757,130 @@ func TestSigningRetryLoop_GetCurrentBlockErrorCausesRetry(t *testing.T) { } signingGroupOperators := chain.Addresses{ - "address-1", "address-2", "address-8", "address-4", - "address-2", "address-6", "address-7", "address-8", - "address-9", "address-8", + "address-1", + "address-2", + "address-3", + "address-4", + "address-5", + "address-6", + "address-7", + "address-8", + "address-9", + "address-10", } - retryLoop := newSigningRetryLoop( - &testutils.MockLogger{}, - message, - 200, - 1, - signingGroupOperators, - groupParameters, - &mockSigningAnnouncer{ - outgoingAnnouncements: make(map[string]group.MemberIndex), - incomingAnnouncementsFn: func(string) ([]group.MemberIndex, error) { - panic("should not be reached: announcer invoked when getCurrentBlock always errors") - }, - }, - &mockSigningDoneCheck{ - waitUntilAllDoneOutcomeFn: func(uint64) (*signing.Result, uint64, error) { - panic("should not be reached") - }, - }, - ) - - ctx, cancelCtx := context.WithTimeout(context.Background(), 50*time.Millisecond) - defer cancelCtx() + signingGroupMembersIndexes := make([]group.MemberIndex, 0) + for i := range signingGroupOperators { + signingGroupMembersIndexes = append( + signingGroupMembersIndexes, + group.MemberIndex(i+1), + ) + } - _, err := retryLoop.start( - ctx, - func(context.Context, uint64) error { return nil }, - func() (uint64, error) { return 0, fmt.Errorf("rpc unavailable") }, - func(*signingAttemptParams) (*signing.Result, uint64, error) { - panic("should not be reached: signing invoked when getCurrentBlock always errors") + testResult := &signing.Result{ + Signature: &tecdsa.Signature{ + R: big.NewInt(300), + S: big.NewInt(400), + RecoveryID: 2, }, - ) - - if err != context.DeadlineExceeded { - t.Errorf( - "unexpected error\nexpected: [%v]\nactual: [%v]", - context.DeadlineExceeded, - err, - ) } -} -func TestSigningRetryLoop_WaitForBlockErrorCausesRetry(t *testing.T) { - message := big.NewInt(100) + doneSigners := participation.MemberIndexes{1, 2, 3, 4, 5, 6} - groupParameters := &GroupParameters{ - GroupSize: 10, - HonestThreshold: 6, + announcer := &mockSigningAnnouncer{ + outgoingAnnouncements: make(map[string]group.MemberIndex), + incomingAnnouncementsFn: func( + sessionID string, + ) ([]group.MemberIndex, error) { + return signingGroupMembersIndexes, nil + }, } - - signingGroupOperators := chain.Addresses{ - "address-1", "address-2", "address-8", "address-4", - "address-2", "address-6", "address-7", "address-8", - "address-9", "address-8", + doneCheck := &mockSigningDoneCheck{ + waitUntilAllDoneOutcomeFn: func( + attemptNumber uint64, + ) (*signing.Result, uint64, error) { + return testResult, 215, nil + }, + doneSigners: doneSigners, } retryLoop := newSigningRetryLoop( &testutils.MockLogger{}, message, + participation.ModeSecurityV2, 200, 1, signingGroupOperators, groupParameters, - &mockSigningAnnouncer{ - outgoingAnnouncements: make(map[string]group.MemberIndex), - incomingAnnouncementsFn: func(string) ([]group.MemberIndex, error) { - panic("should not be reached: announcer invoked when waitForBlock always errors") - }, - }, - &mockSigningDoneCheck{ - waitUntilAllDoneOutcomeFn: func(uint64) (*signing.Result, uint64, error) { - panic("should not be reached") - }, - }, + announcer, + doneCheck, ) - ctx, cancelCtx := context.WithTimeout(context.Background(), 50*time.Millisecond) + ctx, cancelCtx := context.WithTimeout(context.Background(), 10*time.Second) defer cancelCtx() - _, err := retryLoop.start( + result, err := retryLoop.start( ctx, - func(context.Context, uint64) error { return fmt.Errorf("rpc timeout") }, - func() (uint64, error) { return 200, nil }, // behind announcementEndBlock so attempt is not skipped - func(*signingAttemptParams) (*signing.Result, uint64, error) { - panic("should not be reached: signing invoked when waitForBlock always errors") + func(context.Context, uint64) error { + return nil + }, + func() (uint64, error) { + return 200, nil + }, + func(params *signingAttemptParams) (*signing.Result, uint64, error) { + return testResult, 215, nil }, ) + if err != nil { + t.Fatal(err) + } - if err != context.DeadlineExceeded { + if !slices.Equal(result.resultSigners, doneSigners) { t.Errorf( - "unexpected error\nexpected: [%v]\nactual: [%v]", - context.DeadlineExceeded, - err, + "unexpected population behind the result\n"+ + "expected: [%v]\n"+ + "actual: [%v]", + doneSigners, + result.resultSigners, ) } } -func TestSigningRetryLoop_ContextCancelled(t *testing.T) { - groupParameters := &GroupParameters{ - GroupSize: 10, - HonestThreshold: 6, - } - - signingGroupOperators := chain.Addresses{ - "address-1", "address-2", "address-8", "address-4", - "address-2", "address-6", "address-7", "address-8", - "address-9", "address-8", - } - - retryLoop := newSigningRetryLoop( - &testutils.MockLogger{}, - big.NewInt(100), - 200, - 1, - signingGroupOperators, - groupParameters, - &mockSigningAnnouncer{ - outgoingAnnouncements: make(map[string]group.MemberIndex), - incomingAnnouncementsFn: func(string) ([]group.MemberIndex, error) { - panic("should not be reached: context already cancelled") - }, - }, - &mockSigningDoneCheck{ - waitUntilAllDoneOutcomeFn: func(uint64) (*signing.Result, uint64, error) { - panic("should not be reached") - }, - }, - ) +func TestSigningAttemptSessionIDIncludesAttemptStartBlock(t *testing.T) { + message := big.NewInt(100) - ctx, cancel := context.WithCancel(context.Background()) - cancel() // cancel before start -- the loop should exit at the first ctx.Err() check + firstCeremony := signingAttemptSessionID(participation.ModeSecurityV2, message, 206, 1) + repeatedDigestCeremony := signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 1) + retryAttempt := signingAttemptSessionID(participation.ModeSecurityV2, message, 247, 2) - _, err := retryLoop.start( - ctx, - func(context.Context, uint64) error { return nil }, - func() (uint64, error) { return 200, nil }, - func(*signingAttemptParams) (*signing.Result, uint64, error) { - panic("should not be reached") - }, + testutils.AssertStringsEqual( + t, + "session ID format", + "signing-64-00000000000000ce-0000000000000001", + firstCeremony, ) - - if err != context.Canceled { - t.Errorf("expected context.Canceled, got: %v", err) + if len(firstCeremony) < 16 { + t.Fatal("signing session ID must satisfy tss-lib SetSessionNonceBytes minimum length") } -} - -// TestSigningRetryLoop_SuccessAfterRetry verifies that the retry loop -// recovers when the announcer returns an error on the first attempt and -// succeeds on the second -- the retry path must actually produce a result. -func TestSigningRetryLoop_SuccessAfterRetry(t *testing.T) { - message := big.NewInt(100) - groupParameters := &GroupParameters{ - GroupSize: 10, - HonestThreshold: 6, - } - - signingGroupOperators := chain.Addresses{ - "address-1", "address-2", "address-8", "address-4", - "address-2", "address-6", "address-7", "address-8", - "address-9", "address-8", + // The smallest possible inputs must still clear the tss-lib floor; this + // guards against a future format change silently regressing below 16 bytes. + minSessionID := signingAttemptSessionID(participation.ModeSecurityV2, big.NewInt(0), 0, 0) + if len(minSessionID) < 16 { + t.Fatalf( + "signing session ID for minimum inputs must satisfy tss-lib "+ + "SetSessionNonceBytes minimum length, got [%v] (%d bytes)", + minSessionID, + len(minSessionID), + ) } - testResult := &signing.Result{ - Signature: &tecdsa.Signature{ - R: big.NewInt(300), - S: big.NewInt(400), - RecoveryID: 2, - }, + if firstCeremony == repeatedDigestCeremony { + t.Fatal("same digest and attempt number must not reuse the session ID across ceremonies") } - // Session IDs use fmt.Sprintf("%v-%v", message, attemptCounter). - firstAttemptSession := fmt.Sprintf("%v-%v", message, 1) - - retryLoop := newSigningRetryLoop( - &testutils.MockLogger{}, - message, - 200, - 1, - signingGroupOperators, - groupParameters, - &mockSigningAnnouncer{ - outgoingAnnouncements: make(map[string]group.MemberIndex), - incomingAnnouncementsFn: func(sessionID string) ([]group.MemberIndex, error) { - if sessionID == firstAttemptSession { - return nil, fmt.Errorf("announcer unavailable on first attempt") - } - return []group.MemberIndex{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil - }, - }, - &mockSigningDoneCheck{ - waitUntilAllDoneOutcomeFn: func(uint64) (*signing.Result, uint64, error) { - return testResult, 215, nil - }, - }, - ) - - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cancel() - - result, err := retryLoop.start( - ctx, - func(context.Context, uint64) error { return nil }, - func() (uint64, error) { return 200, nil }, - func(*signingAttemptParams) (*signing.Result, uint64, error) { - return testResult, 215, nil - }, - ) - - if err != nil { - t.Fatalf("expected no error after retry, got: %v", err) - } - if result == nil { - t.Fatal("expected non-nil result") - } - if result.result == nil || !result.result.Signature.Equals(testResult.Signature) { - t.Errorf("unexpected result signature: %v", result) + if repeatedDigestCeremony == retryAttempt { + t.Fatal("attempts within a ceremony must not reuse the session ID") } } @@ -959,16 +910,54 @@ type mockSigningDoneCheck struct { outgoingDoneChecks []*signingDoneMessage currentAttemptNumber uint64 waitUntilAllDoneOutcomeFn func(attemptNumber uint64) (*signing.Result, uint64, error) + // doneSigners stands in for the memberships the real done check reports as + // having carried the result. It is nil unless a test is about the + // transcript, so a case that does not set it expects the loop to carry + // nothing rather than to invent a population. + doneSigners participation.MemberIndexes + // listens records the attempt the loop asked the done check to listen for, + // so a test can hold the loop to handing over that attempt's own protocol + // window rather than some other attempt's. + listens []*mockSigningDoneCheckListen +} + +// mockSigningDoneCheckListen is one recorded listen call. +type mockSigningDoneCheckListen struct { + attemptNumber uint64 + startBlock uint64 + timeoutBlock uint64 + membersIndexes []group.MemberIndex +} + +// listenFor returns the recorded listen call for the given attempt number, or +// nil when the loop never asked to listen for that attempt. +func (msdc *mockSigningDoneCheck) listenFor( + attemptNumber uint64, +) *mockSigningDoneCheckListen { + for _, listen := range msdc.listens { + if listen.attemptNumber == attemptNumber { + return listen + } + } + + return nil } func (msdc *mockSigningDoneCheck) listen( ctx context.Context, message *big.Int, attemptNumber uint64, + attemptStartBlock uint64, attemptTimeoutBlock uint64, attemptMembersIndexes []group.MemberIndex, ) { msdc.currentAttemptNumber = attemptNumber + msdc.listens = append(msdc.listens, &mockSigningDoneCheckListen{ + attemptNumber: attemptNumber, + startBlock: attemptStartBlock, + timeoutBlock: attemptTimeoutBlock, + membersIndexes: attemptMembersIndexes, + }) } func (msdc *mockSigningDoneCheck) signalDone( @@ -990,6 +979,15 @@ func (msdc *mockSigningDoneCheck) signalDone( return nil } -func (msdc *mockSigningDoneCheck) waitUntilAllDone(ctx context.Context) (*signing.Result, uint64, error) { - return msdc.waitUntilAllDoneOutcomeFn(msdc.currentAttemptNumber) +func (msdc *mockSigningDoneCheck) waitUntilAllDone(ctx context.Context) ( + *signing.Result, + participation.MemberIndexes, + uint64, + error, +) { + result, endBlock, err := msdc.waitUntilAllDoneOutcomeFn( + msdc.currentAttemptNumber, + ) + + return result, msdc.doneSigners, endBlock, err } diff --git a/pkg/tbtc/signing_test.go b/pkg/tbtc/signing_test.go index 3e7367fa43..18dab998e0 100644 --- a/pkg/tbtc/signing_test.go +++ b/pkg/tbtc/signing_test.go @@ -5,6 +5,7 @@ import ( "crypto/ecdsa" "crypto/elliptic" "math/big" + "slices" "strings" "testing" "time" @@ -17,6 +18,7 @@ import ( "github.com/keep-network/keep-core/pkg/net/local" "github.com/keep-network/keep-core/pkg/operator" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -29,7 +31,7 @@ func TestSigningExecutor_Sign(t *testing.T) { message := big.NewInt(100) startBlock := uint64(0) - signature, _, endBlock, err := executor.sign(ctx, message, startBlock) + outcome, err := executor.sign(ctx, message, startBlock, participation.ModeSecurityV2) if err != nil { t.Fatal(err) } @@ -39,15 +41,36 @@ func TestSigningExecutor_Sign(t *testing.T) { if !ecdsa.Verify( walletPublicKey, message.Bytes(), - signature.R, - signature.S, + outcome.signature.R, + outcome.signature.S, ) { - t.Errorf("invalid signature: [%+v]", signature) + t.Errorf("invalid signature: [%+v]", outcome.signature) } - if endBlock <= startBlock { + if outcome.endBlock <= startBlock { t.Errorf("wrong end block") } + + // The signature arrives with the local view of who produced it. Every + // membership here confirmed this exact signature with an authenticated done + // check, and the local half names the memberships this node operates, so a + // reader of the terminal record can tell a result several parties reached + // from one this node reached alone. + // This executor operates every membership of the group, so the memberships + // that confirmed the signature and the ones it operated are the same set — + // the attempt's members, which is an honest majority or more. + if outcome.contribution == nil || + len(outcome.contribution.IncorporatedMembers) < + executor.groupParameters.HonestThreshold || + !slices.Equal( + outcome.contribution.LocalMembers, + outcome.contribution.IncorporatedMembers, + ) { + t.Errorf( + "unexpected transcript behind the signature: %+v", + outcome.contribution, + ) + } } func TestSigningExecutor_Sign_Busy(t *testing.T) { @@ -61,13 +84,13 @@ func TestSigningExecutor_Sign_Busy(t *testing.T) { errChan := make(chan error, 1) go func() { - _, _, _, err := executor.sign(ctx, message, startBlock) + _, err := executor.sign(ctx, message, startBlock, participation.ModeSecurityV2) errChan <- err }() time.Sleep(100 * time.Millisecond) - _, _, _, err := executor.sign(ctx, message, startBlock) + _, err := executor.sign(ctx, message, startBlock, participation.ModeSecurityV2) testutils.AssertErrorsSame(t, errSigningExecutorBusy, err) err = <-errChan @@ -89,11 +112,39 @@ func TestSigningExecutor_SignBatch(t *testing.T) { } startBlock := uint64(0) - signatures, err := executor.signBatch(ctx, messages, startBlock) + signatures, transcript, err := executor.signBatch( + ctx, + messages, + startBlock, + participation.ModeSecurityV2, + ) if err != nil { t.Fatal(err) } + // The batch's signatures become one Bitcoin transaction, so the transcript + // it carries names the memberships present in every one of them. Anything + // wider would let a membership that produced one signature stand as a + // contributor to the whole transaction. + if transcript == nil || + len(transcript.IncorporatedMembers) == 0 || + !slices.Equal( + transcript.LocalMembers, + transcript.IncorporatedMembers, + ) { + t.Errorf("unexpected transcript behind the batch: %+v", transcript) + } + // Each message's attempt selects its own members, so the intersection can + // be narrower than any single signature's population — and never wider. + for _, index := range transcript.IncorporatedMembers { + if int(index) < 1 || int(index) > executor.groupParameters.GroupSize { + t.Errorf( + "the batch transcript names membership [%d] outside the group", + index, + ) + } + } + walletPublicKey := executor.wallet().publicKey for i, signature := range signatures { @@ -121,13 +172,13 @@ func TestSigningExecutor_Sign_ContextCancelled(t *testing.T) { // rather than hanging. cancelCtx() - signature, _, _, _ := executor.sign(ctx, message, startBlock) + outcome, _ := executor.sign(ctx, message, startBlock, participation.ModeSecurityV2) // A cancelled context may return nil signature with nil error (early exit) // or an error -- both are acceptable. What must NOT happen is a hang or // a successful signature returned despite cancellation. - if signature != nil { - t.Errorf("expected nil signature on context cancel, got: %+v", signature) + if outcome != nil { + t.Errorf("expected nil outcome on context cancel, got: %+v", outcome) } } @@ -143,12 +194,12 @@ func TestSigningExecutor_Sign_AllSignersFailed(t *testing.T) { message := big.NewInt(100) startBlock := uint64(0) - signature, _, _, err := executor.sign(ctx, message, startBlock) + outcome, err := executor.sign(ctx, message, startBlock, participation.ModeSecurityV2) // With zero attempts, all signers cannot succeed. We expect either // errSigningExecutorBusy (if the lock is still held) or an error/nil // result -- but not a completed valid signature. - if signature != nil && err == nil { + if outcome != nil && err == nil { t.Error("expected failure when signingAttemptsLimit is 0, but got a valid signature") } } @@ -163,7 +214,7 @@ func TestSigningExecutor_Sign_MarshalError(t *testing.T) { ctx, cancelCtx := context.WithCancel(context.Background()) defer cancelCtx() - _, _, _, err := executor.sign(ctx, big.NewInt(100), 0) + _, err := executor.sign(ctx, big.NewInt(100), 0, participation.ModeSecurityV2) if err == nil { t.Fatal("expected error from sign, got nil") @@ -184,7 +235,7 @@ func TestSigningExecutor_SignBatch_PartialFailure(t *testing.T) { messages := []*big.Int{big.NewInt(1), big.NewInt(2), big.NewInt(3)} - _, err := executor.signBatch(ctx, messages, 0) + _, _, err := executor.signBatch(ctx, messages, 0, participation.ModeSecurityV2) if err == nil { t.Error("expected error from signBatch when all signers fail, got nil") @@ -194,6 +245,15 @@ func TestSigningExecutor_SignBatch_PartialFailure(t *testing.T) { // setupSigningExecutor sets up an instance of the signing executor ready // to perform test signing. func setupSigningExecutor(t *testing.T) *signingExecutor { + executor, _ := setupSigningExecutorWithChain(t) + return executor +} + +// setupSigningExecutorWithChain sets up an instance of the signing executor +// ready to perform test signing and returns it together with the local chain +// it is connected to, so tests can drive gates and fences from the same chain +// clock. +func setupSigningExecutorWithChain(t *testing.T) (*signingExecutor, *localChain) { groupParameters := &GroupParameters{ GroupSize: 5, GroupQuorum: 4, @@ -287,5 +347,5 @@ func setupSigningExecutor(t *testing.T) *signingExecutor { // Set more attempts to give more time for computations. executor.signingAttemptsLimit *= 8 - return executor + return executor, localChain } diff --git a/pkg/tbtc/tbtc.go b/pkg/tbtc/tbtc.go index fa009348b9..788651655d 100644 --- a/pkg/tbtc/tbtc.go +++ b/pkg/tbtc/tbtc.go @@ -15,6 +15,7 @@ import ( "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/generator" "github.com/keep-network/keep-core/pkg/net" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/sortition" ) @@ -101,12 +102,21 @@ type Config struct { // Initialize kicks off the TBTC by initializing internal state, ensuring // preconditions like staking are met, and then kicking off the internal TBTC // implementation. Returns an error if this failed. +// +// The participation gate and the cutover peer roster are constructed once at +// process startup, immediately after the Ethereum connection, and shared with +// the beacon application; this function receives those exact instances and +// must not construct its own. A nil gate or roster is forbidden: every tBTC +// ceremony's protocol mode must ultimately derive from a permit issued by the +// shared gate, and legacy peer sightings from the DKG and signing announcers +// feed the shared roster. func Initialize( ctx context.Context, chain Chain, btcChain bitcoin.Chain, netProvider net.Provider, keyStorePersistence persistence.ProtectedHandle, + quarantinePersistence persistence.ProtectedHandle, workPersistence persistence.BasicHandle, scheduler *generator.Scheduler, proposalGenerator CoordinationProposalGenerator, @@ -114,7 +124,19 @@ func Initialize( clientInfo *clientinfo.Registry, perfMetrics *clientinfo.PerformanceMetrics, ethereumNetwork ethereum.Network, + participationGate participation.Gate, + cutoverRoster *participation.CutoverPeerRoster, ) error { + if participationGate == nil { + return fmt.Errorf("the participation gate is required") + } + if quarantinePersistence == nil { + return fmt.Errorf("the signer quarantine persistence is required") + } + if cutoverRoster == nil { + return fmt.Errorf("the cutover peer roster is required") + } + groupParameters := defaultGroupParameters(ethereumNetwork) if ethChain, ok := chain.(interface { @@ -155,6 +177,52 @@ func Initialize( return fmt.Errorf("cannot set up TBTC node: [%v]", err) } + // The gate, quarantine store, and roster are installed BEFORE the + // coordination layer starts and BEFORE any chain event subscription + // exists, so every ceremony choke point already carries them when the + // first ceremony can possibly begin. The gate is stored for the ceremony + // choke points; their lifecycles are owned by the process startup that + // constructed them. + node.participationGate = participationGate + node.dkgExecutor.participationGate = participationGate + node.dkgExecutor.signerQuarantine = newSignerQuarantine( + ctx, + logger, + quarantinePersistence, + ) + node.setCutoverPeerRoster(cutoverRoster) + + // The recorder is wired before the first quarantine scan so that scan has + // somewhere to publish, and both happen before the coordination layer + // starts. + if clientInfo != nil { + if perfMetrics == nil { + perfMetrics = clientinfo.NewPerformanceMetrics(ctx, clientInfo) + } + node.setPerformanceMetrics(perfMetrics) + } + + // Published once here so the count covers what earlier processes on this + // host preserved, not only what this one goes on to preserve. A restart + // inherits the quarantine namespace and the rollback decision the count + // informs is about all of it; reporting nothing until this process happens + // to quarantine its own output would show an empty namespace as long as it + // never does. + // + // A namespace this first scan cannot read stops startup. There is no + // earlier count to fall back on, so the registered zero would stand as the + // answer, and preserved key material would be invisible to the fleet that + // has to account for it. + // + // The scan runs before runCoordinationLayer, which installs the block + // watchers a coordination procedure begins from. A startup that fails + // closed has to fail before any ceremony can be issued a permit, otherwise + // the node does the very work it refused to start for — and the coordination + // layer is the first thing here that can reach a choke point. + if err := node.dkgExecutor.reportInitialQuarantinedSigners(); err != nil { + return fmt.Errorf("cannot set up TBTC node: [%w]", err) + } + err = node.runCoordinationLayer(ctx) if err != nil { return fmt.Errorf("cannot run coordination layer: [%w]", err) @@ -173,11 +241,6 @@ func Initialize( }, ) - if perfMetrics == nil { - perfMetrics = clientinfo.NewPerformanceMetrics(ctx, clientInfo) - } - node.setPerformanceMetrics(perfMetrics) - // Register coordination windows as a diagnostic source clientInfo.RegisterApplicationSource( "coordination_windows", diff --git a/pkg/tbtc/tbtc_startup_test.go b/pkg/tbtc/tbtc_startup_test.go new file mode 100644 index 0000000000..4ef1e36a94 --- /dev/null +++ b/pkg/tbtc/tbtc_startup_test.go @@ -0,0 +1,114 @@ +package tbtc + +import ( + "context" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/keep-network/keep-common/pkg/chain/ethereum" + + "github.com/keep-network/keep-core/pkg/chain" + "github.com/keep-network/keep-core/pkg/net/local" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// coordinationWatchRecordingChain records every reach for the chain clock the +// coordination layer watches windows on. +// +// The block counter is the first thing runCoordinationLayer asks the chain for, +// and nothing earlier in tBTC startup asks for it, so a read of it is the mark +// that the layer started. +type coordinationWatchRecordingChain struct { + Chain + + blockCounterReads atomic.Int32 +} + +func (c *coordinationWatchRecordingChain) BlockCounter() ( + chain.BlockCounter, + error, +) { + c.blockCounterReads.Add(1) + + return c.Chain.BlockCounter() +} + +// TestInitialize_UnreadableQuarantineStopsStartupBeforeCoordination proves a +// quarantine namespace that cannot be enumerated stops tBTC startup before the +// coordination layer installs a single block watcher. +// +// Failing closed after that layer is running is not failing closed. It watches +// coordination windows and launches coordination procedures, so a permit can be +// issued — and a ceremony begun — during a startup that has already established +// it must not proceed, on a host whose preserved key material nobody can count. +// +// The namespace also has to be read whether or not the client-info endpoint is +// configured: an unreadable quarantine is a fault in its own right, and a node +// that only notices it when metrics happen to be enabled starts over material +// nobody can account for. This startup is given no client-info registry for +// exactly that reason. +func TestInitialize_UnreadableQuarantineStopsStartupBeforeCoordination( + t *testing.T, +) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + localChain := Connect() + blockCounter, err := localChain.BlockCounter() + if err != nil { + t.Fatal(err) + } + + retention, err := CutoverPeerRosterRetentionBlocks() + if err != nil { + t.Fatal(err) + } + roster, err := participation.NewCutoverPeerRoster( + ctx, + blockCounter, + retention, + testGateMetrics{}, + ) + if err != nil { + t.Fatal(err) + } + + watchedChain := &coordinationWatchRecordingChain{Chain: localChain} + + err = Initialize( + ctx, + watchedChain, + newLocalBitcoinChain(), + local.Connect(), + &mockPersistenceHandle{}, + &unreadableHandle{}, + &mockPersistenceHandle{}, + newTestScheduler(t), + &mockCoordinationProposalGenerator{}, + Config{PreParamsPoolSize: 1, PreParamsGenerationTimeout: time.Hour}, + nil, + nil, + ethereum.Developer, + newTestGate(t, blockCounter), + roster, + ) + if err == nil { + t.Fatal( + "an unreadable quarantine namespace must stop tBTC startup, not " + + "leave the registered zero standing as the count", + ) + } + if !strings.Contains(err.Error(), "unreadable") { + t.Errorf("expected the underlying read error, got [%v]", err) + } + + if reads := watchedChain.blockCounterReads.Load(); reads != 0 { + t.Errorf( + "a startup that failed closed still started the coordination "+ + "layer: [%d] block-counter reads", + reads, + ) + } +} diff --git a/pkg/tbtc/wallet.go b/pkg/tbtc/wallet.go index ca346dec69..574d6027ae 100644 --- a/pkg/tbtc/wallet.go +++ b/pkg/tbtc/wallet.go @@ -18,6 +18,7 @@ import ( "github.com/keep-network/keep-core/pkg/clientinfo" "github.com/keep-network/keep-core/pkg/crypto/secp256k1" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" "go.uber.org/zap" ) @@ -244,6 +245,18 @@ func (wd *walletDispatcher) dispatch(action walletAction) error { err := action.execute() if err != nil { + // A gate decision — clock failure, forced quiescence, or a refused + // commit fence — ended the action; it is not an ordinary action + // failure and must not increment the ordinary failure metrics. + // The gate records the abort in its own metrics. + if participation.IsGateRefusal(err) { + walletActionLogger.Warnf( + "action execution canceled by the participation "+ + "gate: [%v]", + err, + ) + return + } walletActionLogger.Errorf( "action execution terminated with error: [%v]", err, @@ -281,11 +294,15 @@ type walletSigningExecutor interface { ctx context.Context, messages []*big.Int, startBlock uint64, - ) ([]*tecdsa.Signature, error) + mode participation.ProtocolMode, + ) ([]*tecdsa.Signature, *participation.TranscriptContribution, error) } // walletTransactionExecutor is a component allowing to sign and broadcast -// wallet Bitcoin transactions. +// wallet Bitcoin transactions. Every cryptographic and terminal decision it +// makes is scoped to the owning wallet action's participation permit: signing +// uses the permit's pinned protocol mode and context, and each Bitcoin +// broadcast attempt passes the permit's completion commit fence first. type walletTransactionExecutor struct { btcChain bitcoin.Chain @@ -293,6 +310,28 @@ type walletTransactionExecutor struct { signingExecutor walletSigningExecutor waitForBlockFn waitForBlockFn + + permit participation.Permit + // broadcastOperation names the action-specific Bitcoin broadcast in the + // commit fence, e.g. "tbtc_deposit_sweep_bitcoin_broadcast". + broadcastOperation string + + // signedTransaction is the fully signed Bitcoin transaction this action + // produced, if the signing ceremony reached the threshold. It is the + // wallet action's durable result: once a valid signed transaction exists, + // any wallet member may put it on the Bitcoin network, so the offline + // rollback audit must reconcile this exact transaction against mempool and + // chain to decide whether it was broadcast, mined, or is absent. It is + // written and read only on the action goroutine that owns the executor. + signedTransaction *bitcoin.Transaction + + // signedTransactionTranscript is the local view of which memberships + // produced every signature the transaction above rests on. It travels with + // the transaction because the two are one result: a hash alone says the + // action reached a threshold signature, and every member of the ceremony + // records the same hash whatever population actually produced it. It is + // written and read on the same single goroutine as the transaction. + signedTransactionTranscript *participation.TranscriptContribution } func newWalletTransactionExecutor( @@ -300,12 +339,16 @@ func newWalletTransactionExecutor( executingWallet wallet, signingExecutor walletSigningExecutor, waitForBlockFn waitForBlockFn, + permit participation.Permit, + broadcastOperation string, ) *walletTransactionExecutor { return &walletTransactionExecutor{ - btcChain: btcChain, - executingWallet: executingWallet, - signingExecutor: signingExecutor, - waitForBlockFn: waitForBlockFn, + btcChain: btcChain, + executingWallet: executingWallet, + signingExecutor: signingExecutor, + waitForBlockFn: waitForBlockFn, + permit: permit, + broadcastOperation: broadcastOperation, } } @@ -330,21 +373,25 @@ func (wte *walletTransactionExecutor) signTransaction( signTxLogger.Infof("signing transaction's sig hashes") + // The signing window is bound to the wallet action's permit: a permit + // cancellation — clock failure, forced quiescence — stops the signing + // exactly like the timeout block does. signingCtx, cancelSigningCtx := withCancelOnBlock( - context.Background(), + wte.permit.Context(), signingTimeoutBlock, wte.waitForBlockFn, ) defer cancelSigningCtx() - signatures, err := wte.signingExecutor.signBatch( + signatures, transcript, err := wte.signingExecutor.signBatch( signingCtx, sigHashes, signingStartBlock, + wte.permit.Mode(), ) if err != nil { return nil, fmt.Errorf( - "error while signing transaction's sig hashes: [%v]", + "error while signing transaction's sig hashes: [%w]", err, ) } @@ -370,9 +417,44 @@ func (wte *walletTransactionExecutor) signTransaction( signTxLogger.Infof("transaction created successfully") + // The threshold signature is now bound to a concrete Bitcoin transaction, + // which is what the rollback audit has to reconcile. Pin it before the + // broadcast so a permit canceled mid-broadcast still reports the + // transaction it may have put on the network, together with the transcript + // that produced it — the two are recorded as one result or not at all. + wte.signedTransaction = tx + wte.signedTransactionTranscript = transcript + return tx, nil } +// recordTerminalOutcome reports the wallet action's node-owned final +// disposition on its participation permit. A signed Bitcoin transaction is the +// action's durable result and is recorded by hash for offline reconciliation; +// an action that never reached a signed transaction provably left no state +// behind and is recorded as exhausted. +func (wte *walletTransactionExecutor) recordTerminalOutcome( + actionLogger log.StandardLogger, +) { + if wte.signedTransaction == nil { + recordPermitNoThreshold(actionLogger, wte.permit) + return + } + + recordPermitTerminalOutcome( + actionLogger, + wte.permit, + participation.TerminalOutcomeCompleted, + participation.TerminalEvidence{ + Kind: participation.TerminalEvidenceBitcoinTransaction, + Reference: wte.signedTransaction.Hash().Hex( + bitcoin.ReversedByteOrder, + ), + Contribution: wte.signedTransactionTranscript, + }, + ) +} + // broadcastTransaction broadcasts a signed Bitcoin transaction until // the transaction lands in the Bitcoin mempool or the provided timeout // is hit, whichever comes first. @@ -384,8 +466,11 @@ func (wte *walletTransactionExecutor) broadcastTransaction( ) error { txHash := tx.Hash() + // The broadcast window is bound to the wallet action's permit so a permit + // cancellation ends the retry loop instead of leaving it running on an + // unowned background context. broadcastCtx, cancelBroadcastCtx := context.WithTimeout( - context.Background(), + wte.permit.Context(), timeout, ) defer cancelBroadcastCtx() @@ -395,10 +480,20 @@ func (wte *walletTransactionExecutor) broadcastTransaction( for { select { case <-broadcastCtx.Done(): - return fmt.Errorf("broadcast timeout exceeded") + return broadcastAbortError(broadcastCtx) default: broadcastAttempt++ + // The last-moment fence before every irreversible Bitcoin + // broadcast attempt. A refusal is a release-gate decision, not an + // ordinary broadcast failure. + if err := wte.permit.CheckCommit( + wte.broadcastOperation, + participation.CompletionCommit, + ); err != nil { + return err + } + broadcastTxLogger.Infof( "broadcasting transaction on the Bitcoin chain - attempt [%v]", broadcastAttempt, @@ -424,7 +519,7 @@ func (wte *walletTransactionExecutor) broadcastTransaction( select { case <-time.After(checkDelay): case <-broadcastCtx.Done(): - return fmt.Errorf("broadcast timeout exceeded") + return broadcastAbortError(broadcastCtx) } broadcastTxLogger.Infof( @@ -447,6 +542,17 @@ func (wte *walletTransactionExecutor) broadcastTransaction( } } +// broadcastAbortError classifies an ended broadcast window: a gate-caused +// permit cancellation surfaces its sentinel so upper layers keep it out of +// ordinary failure accounting; everything else is the ordinary broadcast +// timeout. +func broadcastAbortError(ctx context.Context) error { + if cause := context.Cause(ctx); participation.IsGateRefusal(cause) { + return fmt.Errorf("bitcoin broadcast aborted: %w", cause) + } + return fmt.Errorf("broadcast timeout exceeded") +} + // wallet represents a tBTC wallet. A wallet is one of the basic building // blocks of the system that takes BTC under custody during the deposit // process and gives that BTC back during redemptions. diff --git a/pkg/tbtc/wallet_test.go b/pkg/tbtc/wallet_test.go index 9ef4e41576..dccb6df070 100644 --- a/pkg/tbtc/wallet_test.go +++ b/pkg/tbtc/wallet_test.go @@ -20,6 +20,7 @@ import ( "github.com/keep-network/keep-core/pkg/bitcoin" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/protocol/group" + "github.com/keep-network/keep-core/pkg/protocol/participation" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -203,7 +204,8 @@ func TestEnsureWalletSyncedBetweenChains_TransactionWithoutInputs(t *testing.T) if err == nil { t.Fatal("expected transaction-without-inputs error") } - if !strings.Contains(err.Error(), "has no inputs") { + if !strings.Contains(err.Error(), "out of range") && + !strings.Contains(err.Error(), "has no inputs") { t.Fatalf("unexpected error: [%v]", err) } } @@ -508,6 +510,7 @@ func TestWalletTransactionExecutor_SignTransaction_Success(t *testing.T) { mockExec.setSignatures(sigHashes, startBlock, sigs) executor := &walletTransactionExecutor{ + permit: newTestPermit(participation.TBTCSigning), btcChain: btcChain, executingWallet: walletObj, signingExecutor: mockExec, @@ -540,6 +543,7 @@ func TestWalletTransactionExecutor_SignTransaction_Timeout(t *testing.T) { mockExec := newMockWalletSigningExecutor() executor := &walletTransactionExecutor{ + permit: newTestPermit(participation.TBTCSigning), btcChain: newLocalBitcoinChain(), executingWallet: walletObj, signingExecutor: mockExec, @@ -563,6 +567,7 @@ func TestWalletTransactionExecutor_SignTransaction_InsufficientSigners(t *testin mockExec := newMockWalletSigningExecutor() // no signatures set -> always errors executor := &walletTransactionExecutor{ + permit: newTestPermit(participation.TBTCSigning), btcChain: newLocalBitcoinChain(), executingWallet: walletObj, signingExecutor: mockExec, @@ -628,7 +633,8 @@ func (mwse *mockWalletSigningExecutor) signBatch( ctx context.Context, messages []*big.Int, startBlock uint64, -) ([]*tecdsa.Signature, error) { + mode participation.ProtocolMode, +) ([]*tecdsa.Signature, *participation.TranscriptContribution, error) { mwse.signaturesMutex.Lock() defer mwse.signaturesMutex.Unlock() @@ -636,10 +642,20 @@ func (mwse *mockWalletSigningExecutor) signBatch( signatures, ok := mwse.signatures[key] if !ok { - return nil, fmt.Errorf("signing error") + return nil, nil, fmt.Errorf("signing error") } - return signatures, nil + return signatures, mockSigningTranscript(), nil +} + +// mockSigningTranscript stands in for the local view a real signing operation +// carries out of its done check: the memberships whose authenticated done +// checks named the signature, and the one this node operated. +func mockSigningTranscript() *participation.TranscriptContribution { + return &participation.TranscriptContribution{ + IncorporatedMembers: participation.MemberIndexes{1, 2, 3}, + LocalMembers: participation.MemberIndexes{1}, + } } func (mwse *mockWalletSigningExecutor) setSignatures( @@ -683,6 +699,7 @@ func (c *noConfirmBtcChain) GetTransactionConfirmations(bitcoin.Hash) (uint, err func TestWalletTransactionExecutor_BroadcastTransaction_Success(t *testing.T) { executor := &walletTransactionExecutor{ + permit: newTestPermit(participation.TBTCSigning), btcChain: newLocalBitcoinChain(), executingWallet: generateWallet(big.NewInt(1)), } @@ -717,6 +734,7 @@ func TestWalletTransactionExecutor_BroadcastTransaction_Success(t *testing.T) { func TestWalletTransactionExecutor_BroadcastTransaction_Timeout(t *testing.T) { executor := &walletTransactionExecutor{ + permit: newTestPermit(participation.TBTCSigning), btcChain: &noConfirmBtcChain{newLocalBitcoinChain()}, executingWallet: generateWallet(big.NewInt(1)), } diff --git a/pkg/tecdsa/common/compatibility.go b/pkg/tecdsa/common/compatibility.go new file mode 100644 index 0000000000..0ff9378bb3 --- /dev/null +++ b/pkg/tecdsa/common/compatibility.go @@ -0,0 +1,36 @@ +package common + +import ( + "github.com/bnb-chain/tss-lib/tss" + + "github.com/keep-network/keep-core/pkg/crypto/ephemeral" + "github.com/keep-network/keep-core/pkg/protocol/participation" +) + +// CompatibilityStrategies is the narrow view of the per-ceremony protocol +// compatibility bundle the tECDSA protocols require: the ECDH symmetric-key +// derivation and the proof-transcript configuration of the local TSS +// parties, pinned together to one protocol mode for the ceremony's entire +// lifetime. Every DKG and signing member construction takes the bundle +// explicitly — there is no default — so a ceremony can never mix modes or +// fall back to an implicit transcript. The production bundle is provided by +// the pkg/protocol/compatibility package; passing anything else is reserved +// for tests. +type CompatibilityStrategies interface { + // Mode returns the protocol mode this bundle implements. + Mode() participation.ProtocolMode + + // ECDH derives the symmetric key for the given key pair. The info label + // provides the security-v2 protocol/peer domain separation; the legacy + // derivation has no domain separation by design and ignores it. + ECDH( + privateKey *ephemeral.PrivateKey, + publicKey *ephemeral.PublicKey, + info []byte, + ) *ephemeral.SymmetricEcdhKey + + // ConfigureTSSParameters applies the bundle's proof-transcript decision + // to the given TSS parameters before any local party is constructed from + // them. + ConfigureTSSParameters(parameters *tss.Parameters, sessionID string) error +} diff --git a/pkg/tecdsa/dkg/dkg.go b/pkg/tecdsa/dkg/dkg.go index a2acfef876..12e2dbf4a0 100644 --- a/pkg/tecdsa/dkg/dkg.go +++ b/pkg/tecdsa/dkg/dkg.go @@ -13,6 +13,7 @@ import ( "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/protocol/state" + "github.com/keep-network/keep-core/pkg/tecdsa/common" ) // Executor represents an ECDSA distributed key generation process executor. @@ -55,6 +56,11 @@ func NewExecutor( // a member index to use in the group, dishonest threshold, and block height // when DKG protocol should start. // +// The strategies bundle pins the ceremony's compatibility decisions — the +// ECDH derivation and the TSS proof-transcript configuration — and must be +// selected explicitly from the ceremony's participation permit; there is no +// default. +// // This function also supports DKG execution with a subset of the selected // group by passing a non-empty excludedMembers slice holding the members that // should be excluded. @@ -69,6 +75,7 @@ func (e *Executor) Execute( excludedMembersIndexes []group.MemberIndex, channel net.BroadcastChannel, membershipValidator *group.MembershipValidator, + strategies common.CompatibilityStrategies, ) (*Result, error) { logger.Debugf("[member:%v] initializing member", memberIndex) @@ -80,6 +87,7 @@ func (e *Executor) Execute( dishonestThreshold, membershipValidator, sessionID, + strategies, e.tssPreParamsPool.GetNow, e.keyGenerationConcurrency, ) diff --git a/pkg/tecdsa/dkg/fuzz_test.go b/pkg/tecdsa/dkg/fuzz_test.go new file mode 100644 index 0000000000..9065ee5bfc --- /dev/null +++ b/pkg/tecdsa/dkg/fuzz_test.go @@ -0,0 +1,57 @@ +package dkg + +// Native coverage-guided fuzz targets for the network-message protobuf +// unmarshalers in this package. Each target asserts that Unmarshal never +// panics on arbitrary bytes; a non-nil error on malformed input is fine. +// PreParams is intentionally excluded: it is local key material loaded from +// the operator's own disk, not untrusted network input. + +import "testing" + +func FuzzEphemeralPublicKeyMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&ephemeralPublicKeyMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundOneMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundOneMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundTwoMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundTwoMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundThreeMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundThreeMessage{}).Unmarshal(data) + }) +} + +func FuzzTssFinalizationMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssFinalizationMessage{}).Unmarshal(data) + }) +} + +func FuzzResultSignatureMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&resultSignatureMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/tecdsa/dkg/member.go b/pkg/tecdsa/dkg/member.go index 0418d03bb2..8f3a5f2a30 100644 --- a/pkg/tecdsa/dkg/member.go +++ b/pkg/tecdsa/dkg/member.go @@ -26,6 +26,10 @@ type member struct { membershipValidator *group.MembershipValidator // Identifier of the particular DKG session this member is part of. sessionID string + // Per-ceremony compatibility strategy bundle pinning the ECDH derivation + // and the TSS proof-transcript configuration to the ceremony's protocol + // mode for its entire lifetime. + strategies common.CompatibilityStrategies // TSS pre-parameters getter. preParamsFn func() (*PreParams, error) // Concurrency level of TSS key-generation protocol. @@ -43,6 +47,7 @@ func newMember( dishonestThreshold int, membershipValidator *group.MembershipValidator, sessionID string, + strategies common.CompatibilityStrategies, preParamsFn func() (*PreParams, error), keyGenerationConcurrency int, ) *member { @@ -52,6 +57,7 @@ func newMember( group: group.NewGroup(dishonestThreshold, groupSize), membershipValidator: membershipValidator, sessionID: sessionID, + strategies: strategies, preParamsFn: preParamsFn, keyGenerationConcurrency: keyGenerationConcurrency, identityConverter: &identityConverter{seed: seed}, @@ -142,6 +148,18 @@ func (skgm *symmetricKeyGeneratingMember) initializeTssRoundOne() ( len(groupTssPartiesIDs), skgm.group.HonestThreshold()-1, ) + // Apply the ceremony's proof-transcript configuration; for security-v2 + // this binds GG20 proof challenges to the existing protocol session. + err := skgm.strategies.ConfigureTSSParameters( + tssParameters, + skgm.sessionID, + ) + if err != nil { + return nil, fmt.Errorf( + "failed configuring TSS parameters: [%w]", + err, + ) + } tssParameters.SetConcurrency(skgm.keyGenerationConcurrency) tssOutgoingMessagesChan := make(chan tss.Message, len(groupTssPartiesIDs)) diff --git a/pkg/tecdsa/dkg/member_test.go b/pkg/tecdsa/dkg/member_test.go index 3421c1b3ec..befd93fa77 100644 --- a/pkg/tecdsa/dkg/member_test.go +++ b/pkg/tecdsa/dkg/member_test.go @@ -12,7 +12,9 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/chain" "github.com/keep-network/keep-core/pkg/chain/local_v1" + "github.com/keep-network/keep-core/pkg/internal/tecdsatest" "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" ) @@ -91,6 +93,7 @@ func TestShouldAcceptMessage(t *testing.T) { groupSize-honestThreshold, membershipValdator, "1", + compatibility.SecurityV2(), func() (*PreParams, error) { return &PreParams{ data: &keygen.LocalPreParams{}, @@ -184,3 +187,41 @@ func TestIdentityConverter_TssPartyIDToMemberIndex_Corrupted(t *testing.T) { testutils.AssertIntsEqual(t, "member ID", 0, int(memberIndex)) } + +func TestInitializeTssRoundOneConfiguresLegacyTranscript(t *testing.T) { + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + + member := newMember( + &testutils.MockLogger{}, + big.NewInt(200), + group.MemberIndex(1), + 2, + 0, + nil, + "64757a1f-1", + compatibility.Legacy(), + func() (*PreParams, error) { + return &PreParams{ + data: &testData[0].LocalPreParams, + }, nil + }, + 1, + ) + + roundOneMember, err := member. + initializeEphemeralKeysGeneration(). + initializeSymmetricKeyGeneration(). + initializeTssRoundOne() + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if roundOneMember.tssParameters.ProtocolMode() != tss.ProtocolModeLegacy { + t.Errorf( + "expected legacy TSS mode, got [%v]", + roundOneMember.tssParameters.ProtocolMode(), + ) + } +} diff --git a/pkg/tecdsa/dkg/protocol.go b/pkg/tecdsa/dkg/protocol.go index de333b62e7..df9d2249e1 100644 --- a/pkg/tecdsa/dkg/protocol.go +++ b/pkg/tecdsa/dkg/protocol.go @@ -83,9 +83,12 @@ func (skgm *symmetricKeyGeneratingMember) generateSymmetricKeys( ephemeralPubKeyMessage.ephemeralPublicKeys[skgm.id] // Create symmetric key for the current group member and the other - // group member by ECDH'ing the public and private key. - symmetricKey := thisMemberEphemeralPrivateKey.Ecdh( + // group member by ECDH'ing the public and private key, using the + // ceremony's pinned key-derivation strategy. + symmetricKey := skgm.strategies.ECDH( + thisMemberEphemeralPrivateKey, otherMemberEphemeralPublicKey, + dkgEcdhInfo(skgm.id, otherMember), ) skgm.symmetricKeys[otherMember] = symmetricKey } @@ -466,6 +469,18 @@ func (sm *signingMember) verifyDKGResultSignatures( return receivedValidResultSignatures } +// dkgEcdhInfo returns the HKDF info label for ECDH-derived keys in the tECDSA +// DKG protocol. The pair is sorted so both peers compute the same info +// regardless of which side initiates. Each MemberIndex is encoded as a single +// byte; the compile-time assertion in pkg/protocol/group/group.go enforces the +// uint8 invariant this relies on. +func dkgEcdhInfo(id1, id2 group.MemberIndex) []byte { + if id1 > id2 { + id1, id2 = id2, id1 + } + return []byte{'t', 'e', 'c', 'd', 's', 'a', '-', 'd', 'k', 'g', byte(id1), byte(id2)} +} + // submitDKGResult submits the DKG result along with the supporting signatures // to the provided result submitter. func (sm *submittingMember) submitDKGResult( diff --git a/pkg/tecdsa/dkg/protocol_ecdh_info_test.go b/pkg/tecdsa/dkg/protocol_ecdh_info_test.go new file mode 100644 index 0000000000..79659021c7 --- /dev/null +++ b/pkg/tecdsa/dkg/protocol_ecdh_info_test.go @@ -0,0 +1,51 @@ +package dkg + +import ( + "bytes" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// TestDkgEcdhInfoSortSymmetry verifies that the info label is independent of +// argument order. Both peers must derive the same session key regardless of +// who initiates. +func TestDkgEcdhInfoSortSymmetry(t *testing.T) { + for a := group.MemberIndex(1); a < group.MaxMemberIndex; a++ { + for b := group.MemberIndex(1); b < group.MaxMemberIndex; b++ { + if !bytes.Equal(dkgEcdhInfo(a, b), dkgEcdhInfo(b, a)) { + t.Fatalf("info label not symmetric for (%d, %d)", a, b) + } + } + } +} + +// TestDkgEcdhInfoDistinctPerPair verifies that every distinct sorted member +// pair produces a distinct info label. This is the F-03 invariant that would +// silently break if MemberIndex is ever widened past uint8 without updating +// the encoder. +func TestDkgEcdhInfoDistinctPerPair(t *testing.T) { + seen := make(map[string][2]group.MemberIndex) + for a := group.MemberIndex(1); a < group.MaxMemberIndex; a++ { + for b := a; b < group.MaxMemberIndex; b++ { + label := string(dkgEcdhInfo(a, b)) + if prev, ok := seen[label]; ok { + t.Fatalf( + "info label collision: (%d, %d) and (%d, %d) both produce %x", + prev[0], prev[1], a, b, label, + ) + } + seen[label] = [2]group.MemberIndex{a, b} + } + } +} + +// TestDkgEcdhInfoEncoding pins the wire format. Any change here is a +// protocol-breaking change and requires a coordinated network upgrade. +func TestDkgEcdhInfoEncoding(t *testing.T) { + got := dkgEcdhInfo(7, 3) + want := []byte{'t', 'e', 'c', 'd', 's', 'a', '-', 'd', 'k', 'g', 3, 7} + if !bytes.Equal(got, want) { + t.Fatalf("encoding drift: got %v, want %v", got, want) + } +} diff --git a/pkg/tecdsa/dkg/protocol_test.go b/pkg/tecdsa/dkg/protocol_test.go index c997a29b02..0f883e962f 100644 --- a/pkg/tecdsa/dkg/protocol_test.go +++ b/pkg/tecdsa/dkg/protocol_test.go @@ -11,6 +11,7 @@ import ( "testing" "time" + "github.com/bnb-chain/tss-lib/common" "github.com/bnb-chain/tss-lib/crypto/paillier" "github.com/bnb-chain/tss-lib/ecdsa/keygen" "github.com/bnb-chain/tss-lib/tss" @@ -18,6 +19,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/crypto/ephemeral" "github.com/keep-network/keep-core/pkg/crypto/secp256k1" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" ) @@ -30,7 +32,7 @@ import ( const ( groupSize = 3 dishonestThreshold = 0 - sessionID = "session-1" + sessionID = "session-1-with-128-bits" ) func TestGenerateEphemeralKeyPair(t *testing.T) { @@ -186,6 +188,7 @@ func TestGenerateSymmetricKeys(t *testing.T) { expectedKey := ephemeral.SymmetricKey( member.ephemeralKeyPairs[otherMemberID].PrivateKey.Ecdh( otherMemberEphemeralPublicKey, + dkgEcdhInfo(member.id, otherMemberID), ), ) @@ -248,6 +251,39 @@ func TestGenerateSymmetricKeys_InvalidEphemeralPublicKeyMessage(t *testing.T) { } } +func TestInitializeTssRoundOneSetsSessionNonce(t *testing.T) { + members, err := initializeTssRoundOneMembersGroup( + dishonestThreshold, + groupSize, + ) + if err != nil { + t.Fatal(err) + } + + expectedNonce := new(big.Int).SetBytes(common.SHA512_256([]byte(sessionID))) + for _, member := range members { + testutils.AssertBigIntsEqual( + t, + fmt.Sprintf("session nonce for member [%v]", member.id), + expectedNonce, + member.tssParameters.SessionNonce(), + ) + } + + otherSessionSource := members[0].symmetricKeyGeneratingMember + originalSessionID := otherSessionSource.sessionID + otherSessionSource.sessionID = "other-session-with-128-bits" + otherSessionMember, err := otherSessionSource.initializeTssRoundOne() + otherSessionSource.sessionID = originalSessionID + if err != nil { + t.Fatal(err) + } + + if expectedNonce.Cmp(otherSessionMember.tssParameters.SessionNonce()) == 0 { + t.Fatal("initialized TSS members should use different nonces for different session IDs") + } +} + func TestTssRoundOne(t *testing.T) { members, err := initializeTssRoundOneMembersGroup( dishonestThreshold, @@ -1283,7 +1319,7 @@ func TestVerifyDKGResultSignatures(t *testing.T) { resultHash: ResultSignatureHash{11: 11}, signature: []byte("sign 2"), publicKey: []byte("pubKey 2"), - sessionID: "session-1", + sessionID: sessionID, }, &verificationOutcome{ isValid: true, @@ -1296,7 +1332,7 @@ func TestVerifyDKGResultSignatures(t *testing.T) { resultHash: ResultSignatureHash{11: 11}, signature: []byte("sign 3"), publicKey: []byte("pubKey 3"), - sessionID: "session-1", + sessionID: sessionID, }, &verificationOutcome{ isValid: true, @@ -1319,7 +1355,7 @@ func TestVerifyDKGResultSignatures(t *testing.T) { resultHash: ResultSignatureHash{12: 12}, signature: []byte("sign 2"), publicKey: []byte("pubKey 2"), - sessionID: "session-1", + sessionID: sessionID, }, &verificationOutcome{ isValid: true, @@ -1340,7 +1376,7 @@ func TestVerifyDKGResultSignatures(t *testing.T) { resultHash: ResultSignatureHash{11: 11}, signature: []byte("sign 2"), publicKey: []byte("pubKey 2"), - sessionID: "session-1", + sessionID: sessionID, }, &verificationOutcome{ isValid: false, @@ -1360,7 +1396,7 @@ func TestVerifyDKGResultSignatures(t *testing.T) { resultHash: ResultSignatureHash{11: 11}, signature: []byte("bad sign"), publicKey: []byte("pubKey 2"), - sessionID: "session-1", + sessionID: sessionID, }, &verificationOutcome{ isValid: false, @@ -1491,6 +1527,7 @@ func initializeEphemeralKeyPairGeneratingMembersGroup( id: id, group: dkgGroup, sessionID: sessionID, + strategies: compatibility.SecurityV2(), preParamsFn: preParamsFn, keyGenerationConcurrency: 10, identityConverter: &identityConverter{seed: big.NewInt(200)}, diff --git a/pkg/tecdsa/retry/retry_property_test.go b/pkg/tecdsa/retry/retry_property_test.go new file mode 100644 index 0000000000..c975184ddf --- /dev/null +++ b/pkg/tecdsa/retry/retry_property_test.go @@ -0,0 +1,257 @@ +package retry + +import ( + "fmt" + "reflect" + "testing" + + "pgregory.net/rapid" + + "github.com/keep-network/keep-core/pkg/chain" +) + +// Property-based (model-based) coverage for the retry-participant selection, +// targeting the class of security-audit finding F-009: the seat-counting +// eligibility filter must never return a participant subset that is too small +// to retry with. The original F-009 defect mis-counted +// one operator's seats when judging triplet eligibility, which could admit a +// triplet whose exclusion left FEWER than retryParticipantsCount seats. The +// "large enough" invariant below is exactly what such a defect violates, now +// checked across a wide, randomly generated input space rather than a handful +// of hand-picked tables. +// +// Success and failure are asserted explicitly against a capacity model rather +// than skipped on error: a regression that makes the selection always fail +// (or succeed past its exclusion capacity) fails the suite instead of +// silently passing it. + +// drawGroupMembers builds a random operator group: N distinct operators, each +// holding a random number of seats (a seat == one entry in groupMembers). +// Returns the expanded member slice and the total seat count. +func drawGroupMembers(t *rapid.T) ([]chain.Address, int) { + nOps := rapid.IntRange(4, 10).Draw(t, "numOperators") + var groupMembers []chain.Address + for i := 0; i < nOps; i++ { + op := chain.Address(fmt.Sprintf("operator-%d", i)) + seats := rapid.IntRange(1, 5).Draw(t, fmt.Sprintf("seats-%d", i)) + for s := 0; s < seats; s++ { + groupMembers = append(groupMembers, op) + } + } + return groupMembers, len(groupMembers) +} + +// drawSeed draws a full-range int64 seed. Production seeds are message +// hashes reinterpreted as int64, so negative values are reachable and must +// be exercised. +func drawSeed(t *rapid.T) int64 { + return rapid.Int64().Draw(t, "seed") +} + +// keyGenExclusionCapacity mirrors the documented exclusion model of +// EvaluateRetryParticipantsForKeyGeneration: retries walk eligible single +// operators, then eligible pairs, then eligible triplets, and fail once all +// are exhausted. The capacity is therefore the count of exclusion candidates +// whose removal still leaves at least retryParticipantsCount seats; the +// function must succeed for retryCount below it and fail at or above it. +func keyGenExclusionCapacity( + groupMembers []chain.Address, + retryParticipantsCount int, +) int { + total := len(groupMembers) + seatCount := map[chain.Address]int{} + for _, m := range groupMembers { + seatCount[m]++ + } + + var ops []chain.Address + for op, seats := range seatCount { + if total-seats >= retryParticipantsCount { + ops = append(ops, op) + } + } + + capacity := len(ops) + for i := 0; i < len(ops)-1; i++ { + for j := i + 1; j < len(ops); j++ { + if total-seatCount[ops[i]]-seatCount[ops[j]] >= retryParticipantsCount { + capacity++ + } + } + } + for i := 0; i < len(ops)-2; i++ { + for j := i + 1; j < len(ops)-1; j++ { + for k := j + 1; k < len(ops); k++ { + if total-seatCount[ops[i]]-seatCount[ops[j]]-seatCount[ops[k]] >= + retryParticipantsCount { + capacity++ + } + } + } + } + return capacity +} + +// distinctOperators returns the number of distinct operators holding at least +// one seat in members. +func distinctOperators(members []chain.Address) int { + set := map[chain.Address]bool{} + for _, m := range members { + set[m] = true + } + return len(set) +} + +// assertRetryInvariants checks the properties every SUCCESSFUL selection must +// satisfy: it is a sub-multiset of the group, operators are included +// all-or-nothing (an operator never loses only part of its seats), it retains +// at least retryParticipantsCount seats (the F-009 invariant), and it is +// deterministic for a fixed (seed, retryCount). The call itself must succeed; +// callers are responsible for only requesting satisfiable selections and for +// asserting the error path separately. +func assertRetryInvariants( + t *rapid.T, + fn func([]chain.Address, int64, uint, uint) ([]chain.Address, error), + groupMembers []chain.Address, + seed int64, + retryCount uint, + retryParticipantsCount int, +) []chain.Address { + subset, err := fn(groupMembers, seed, retryCount, uint(retryParticipantsCount)) + if err != nil { + t.Fatalf( + "selection failed for a satisfiable request: %v (group=%d, count=%d, retryCount=%d, seed=%d)", + err, len(groupMembers), retryParticipantsCount, retryCount, seed, + ) + } + + // (1) sub-multiset: the subset cannot contain more seats of any operator + // than the group holds. + groupSeats := map[chain.Address]int{} + for _, m := range groupMembers { + groupSeats[m]++ + } + subsetSeats := map[chain.Address]int{} + for _, m := range subset { + subsetSeats[m]++ + if subsetSeats[m] > groupSeats[m] { + t.Fatalf("subset holds more seats of %q than the group does", m) + } + } + + // (2) all-or-nothing: selection operates on whole operators, so an + // included operator must keep every seat it holds in the group. + for op, n := range subsetSeats { + if n != groupSeats[op] { + t.Fatalf( + "operator %q partially included: %d of %d seats", + op, n, groupSeats[op], + ) + } + } + + // (3) F-009: the surviving subset must retain at least + // retryParticipantsCount seats. A mis-counted eligibility filter that + // admits an over-large exclusion breaks exactly this. + if len(subset) < retryParticipantsCount { + t.Fatalf( + "subset too small: got %d seats, need >= %d (group=%d, retryCount=%d, seed=%d)", + len(subset), retryParticipantsCount, len(groupMembers), retryCount, seed, + ) + } + + // (4) determinism: identical inputs must yield an identical subset. + subset2, err2 := fn(groupMembers, seed, retryCount, uint(retryParticipantsCount)) + if err2 != nil || !reflect.DeepEqual(subset, subset2) { + t.Fatalf("non-deterministic selection for fixed inputs") + } + + return subset +} + +func TestRapidEvaluateRetryParticipantsForKeyGeneration(t *testing.T) { + rapid.Check(t, func(t *rapid.T) { + groupMembers, total := drawGroupMembers(t) + retryParticipantsCount := rapid.IntRange(1, total).Draw(t, "retryParticipantsCount") + seed := drawSeed(t) + retryCount := uint(rapid.IntRange(0, 60).Draw(t, "retryCount")) + + capacity := keyGenExclusionCapacity(groupMembers, retryParticipantsCount) + + if int(retryCount) >= capacity { + // Every eligible single/pair/triplet exclusion is exhausted: + // the function must report that, not fabricate a selection. + _, err := EvaluateRetryParticipantsForKeyGeneration( + groupMembers, seed, retryCount, uint(retryParticipantsCount), + ) + if err == nil { + t.Fatalf( + "expected exhaustion error: retryCount=%d >= capacity=%d", + retryCount, capacity, + ) + } + return + } + + subset := assertRetryInvariants( + t, + EvaluateRetryParticipantsForKeyGeneration, + groupMembers, seed, retryCount, retryParticipantsCount, + ) + + // Key-generation retries work by exclusion: every successful + // selection removes exactly one single, pair, or triplet of + // operators. An implementation that excludes nobody (returns the + // group unchanged) satisfies the size invariants but defeats the + // retry mechanism entirely; this assertion catches it. + excluded := distinctOperators(groupMembers) - distinctOperators(subset) + if excluded < 1 || excluded > 3 { + t.Fatalf( + "expected 1-3 operators excluded, got %d (retryCount=%d)", + excluded, retryCount, + ) + } + }) +} + +func TestRapidEvaluateRetryParticipantsForSigning(t *testing.T) { + rapid.Check(t, func(t *rapid.T) { + groupMembers, total := drawGroupMembers(t) + retryParticipantsCount := rapid.IntRange(1, total).Draw(t, "retryParticipantsCount") + seed := drawSeed(t) + retryCount := uint(rapid.IntRange(0, 60).Draw(t, "retryCount")) + + // Signing selection only fails when more seats are requested than + // exist, which the generators never do — so every call here must + // succeed. (Unlike key generation there is no exclusion guarantee: + // requesting all seats legitimately selects the whole group.) + assertRetryInvariants( + t, + EvaluateRetryParticipantsForSigning, + groupMembers, seed, retryCount, retryParticipantsCount, + ) + }) +} + +// TestRapidEvaluateRetryParticipantsRejectsOversizedRequest pins the one +// documented error path shared by both selection functions: requesting more +// seats than the group holds must fail rather than return a too-small subset. +func TestRapidEvaluateRetryParticipantsRejectsOversizedRequest(t *testing.T) { + rapid.Check(t, func(t *rapid.T) { + groupMembers, total := drawGroupMembers(t) + oversized := uint(rapid.IntRange(total+1, 2*total+1).Draw(t, "oversized")) + seed := drawSeed(t) + retryCount := uint(rapid.IntRange(0, 60).Draw(t, "retryCount")) + + if _, err := EvaluateRetryParticipantsForSigning( + groupMembers, seed, retryCount, oversized, + ); err == nil { + t.Fatalf("signing: expected error for %d seats of %d", oversized, total) + } + if _, err := EvaluateRetryParticipantsForKeyGeneration( + groupMembers, seed, retryCount, oversized, + ); err == nil { + t.Fatalf("keygen: expected error for %d seats of %d", oversized, total) + } + }) +} diff --git a/pkg/tecdsa/signing/fuzz_test.go b/pkg/tecdsa/signing/fuzz_test.go new file mode 100644 index 0000000000..13664e6160 --- /dev/null +++ b/pkg/tecdsa/signing/fuzz_test.go @@ -0,0 +1,86 @@ +package signing + +// Coverage-guided fuzz targets for the network-message protobuf unmarshalers. +// Each asserts that Unmarshal never panics on arbitrary input bytes. + +import "testing" + +func FuzzEphemeralPublicKeyMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&ephemeralPublicKeyMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundOneMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundOneMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundTwoMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundTwoMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundThreeMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundThreeMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundFourMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundFourMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundFiveMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundFiveMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundSixMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundSixMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundSevenMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundSevenMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundEightMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundEightMessage{}).Unmarshal(data) + }) +} + +func FuzzTssRoundNineMessageUnmarshal(f *testing.F) { + f.Add([]byte(nil)) + f.Add([]byte{0x08, 0x01}) + f.Fuzz(func(t *testing.T, data []byte) { + _ = (&tssRoundNineMessage{}).Unmarshal(data) + }) +} diff --git a/pkg/tecdsa/signing/integration_test.go b/pkg/tecdsa/signing/integration_test.go new file mode 100644 index 0000000000..c33024665a --- /dev/null +++ b/pkg/tecdsa/signing/integration_test.go @@ -0,0 +1,102 @@ +// Package signing_test contains whole-protocol integration tests for tECDSA +// signing, driving signing.Execute end to end over a local broadcast channel +// via the signingtest harness. This complements the per-phase unit tests in +// protocol_test.go (whose TODO asks for exactly these integration tests) and +// is the entry point for Byzantine signing scenarios (Tier 2). +package signing_test + +import ( + "math/big" + "testing" + "time" + + "github.com/keep-network/keep-core/pkg/internal/byzantine" + "github.com/keep-network/keep-core/pkg/internal/interception" + "github.com/keep-network/keep-core/pkg/internal/signingtest" + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +func TestSigningExecute_HappyPath(t *testing.T) { + groupSize := 5 + dishonestThreshold := 0 + message := big.NewInt(0xDEADBEEF) + + result, err := signingtest.RunTest( + message, + groupSize, + dishonestThreshold, + interception.PassThrough, + ) + if err != nil { + t.Fatal(err) + } + + // Every member completes, all agree on one signature, and it verifies + // against the group public key for the signed message. + signingtest.AssertSignatureGenerated(t, result, groupSize) + signingtest.AssertMemberFailuresCount(t, result, 0) + signingtest.AssertSameSignature(t, result) + + keyShare, err := signingtest.GroupPublicKey() + if err != nil { + t.Fatal(err) + } + signingtest.AssertValidSignature(t, result, keyShare.PublicKey(), message) +} + +// TestSigningExecute_Byzantine_Withhold_member2 demonstrates the harness +// carrying a Byzantine strategy through full signing. tECDSA signing is +// all-or-nothing for the chosen signing set: if a participant withholds, the +// session cannot complete. The safety invariant under test is that this causes +// a denial of service (no completion) but NEVER splits the group onto divergent +// signatures. A short execution bound keeps the (expected) DoS quick to observe. +func TestSigningExecute_Byzantine_Withhold_member2(t *testing.T) { + groupSize := 5 + dishonestThreshold := 0 + message := big.NewInt(0xDEADBEEF) + + strategy := byzantine.Inactive(group.MemberIndex(2)) + + result, err := signingtest.RunTestWithTimeout( + message, + groupSize, + dishonestThreshold, + 15*time.Second, + strategy, + ) + if err != nil { + t.Fatal(err) + } + + // Liveness: tECDSA signing is all-or-nothing for the active signing set. + // With dishonestThreshold=0 the honest threshold is the whole group, so a + // single withholding participant prevents EVERY member from completing - the + // session is a total denial of service. These are the falsifiable contract: + // a regression that let any member complete, or that changed how many members + // fail, trips them. (The previous version asserted only no-divergence, which + // loops over an empty slice when zero members complete and so passed + // unconditionally.) + signingtest.AssertSignatureGenerated(t, result, 0) + signingtest.AssertMemberFailuresCount(t, result, groupSize) + + // Safety: no member may output a signature that disagrees with another, and + // any signature that is produced must verify against the group key. These are + // vacuous while zero members complete (asserted above) - all-or-nothing + // signing over the shared broadcast channel cannot yield a partial, divergent + // result, and the committed fixtures are a threshold-(groupSize-1) key that + // only the full set can sign, so a non-vacuous fork cannot be induced here. + // They are kept as a guard: if a future change ever lets members complete + // under this scenario, a fork or an invalid signature fails loudly instead of + // passing silently. + signingtest.AssertNoDivergentSignatures(t, result) + keyShare, err := signingtest.GroupPublicKey() + if err != nil { + t.Fatal(err) + } + signingtest.AssertValidSignature(t, result, keyShare.PublicKey(), message) + + t.Logf( + "withhold(member2): %d signatures produced, %d member failures (total DoS, as required)", + len(result.GetSignatures()), len(result.GetMemberFailures()), + ) +} diff --git a/pkg/tecdsa/signing/member.go b/pkg/tecdsa/signing/member.go index d506b8aa2d..ed12d6441f 100644 --- a/pkg/tecdsa/signing/member.go +++ b/pkg/tecdsa/signing/member.go @@ -1,6 +1,7 @@ package signing import ( + "context" "fmt" "math/big" @@ -28,6 +29,10 @@ type member struct { membershipValidator *group.MembershipValidator // Identifier of the particular signing session this member is part of. sessionID string + // Per-ceremony compatibility strategy bundle pinning the ECDH derivation + // and the TSS proof-transcript configuration to the ceremony's protocol + // mode for its entire lifetime. + strategies common.CompatibilityStrategies // Message that is the subject of the signing process. message *big.Int // tECDSA private key share of the member. @@ -44,6 +49,7 @@ func newMember( dishonestThreshold int, membershipValidator *group.MembershipValidator, sessionID string, + strategies common.CompatibilityStrategies, message *big.Int, privateKeyShare *tecdsa.PrivateKeyShare, ) *member { @@ -53,6 +59,7 @@ func newMember( group: group.NewGroup(dishonestThreshold, groupSize), membershipValidator: membershipValidator, sessionID: sessionID, + strategies: strategies, message: message, privateKeyShare: privateKeyShare, identityConverter: &identityConverter{keys: privateKeyShare.Data().Ks}, @@ -123,7 +130,10 @@ type symmetricKeyGeneratingMember struct { } // initializeTssRoundOne returns a member to perform next protocol operations. -func (skgm *symmetricKeyGeneratingMember) initializeTssRoundOne() *tssRoundOneMember { +func (skgm *symmetricKeyGeneratingMember) initializeTssRoundOne() ( + *tssRoundOneMember, + error, +) { // Set up the local TSS party using only operating members. This effectively // removes all excluded members who were marked as disqualified at the // beginning of the protocol. @@ -140,9 +150,22 @@ func (skgm *symmetricKeyGeneratingMember) initializeTssRoundOne() *tssRoundOneMe len(groupTssPartiesIDs), skgm.group.HonestThreshold()-1, ) + // Apply the ceremony's proof-transcript configuration; for security-v2 + // this binds GG20 proof challenges to the existing protocol session. + err := skgm.strategies.ConfigureTSSParameters( + tssParameters, + skgm.sessionID, + ) + if err != nil { + return nil, fmt.Errorf( + "failed configuring TSS parameters: [%w]", + err, + ) + } tssOutgoingMessagesChan := make(chan tss.Message, len(groupTssPartiesIDs)) tssResultChan := make(chan tsslibcommon.SignatureData, 1) + fullBytesLen := (tecdsa.Curve.Params().N.BitLen() + 7) / 8 tssParty := signing.NewLocalParty( skgm.message, @@ -150,6 +173,7 @@ func (skgm *symmetricKeyGeneratingMember) initializeTssRoundOne() *tssRoundOneMe skgm.privateKeyShare.Data(), tssOutgoingMessagesChan, tssResultChan, + fullBytesLen, ) return &tssRoundOneMember{ @@ -158,7 +182,7 @@ func (skgm *symmetricKeyGeneratingMember) initializeTssRoundOne() *tssRoundOneMe tssParameters: tssParameters, tssOutgoingMessagesChan: tssOutgoingMessagesChan, tssResultChan: tssResultChan, - } + }, nil } // tssRoundOneMember represents one member in a signing group performing the @@ -298,6 +322,84 @@ func (fm *finalizingMember) Result() *Result { return &Result{Signature: tecdsa.NewSignature(fm.tssResult)} } +// receiveTSSResult waits for the tss-lib signing result to arrive on the result +// channel, or for the context to be cancelled, and returns an +// independently-owned SignatureData that carries the produced signature. +// +// Ownership boundary. tss-lib's signing.NewLocalParty requires a value-typed +// result channel (`end chan<- common.SignatureData`) and delivers the outcome +// with `end <- *round.data` (ecdsa/signing/finalize.go). common.SignatureData +// is a protobuf message whose embedded protoimpl.MessageState carries a +// `[0]sync.Mutex` DoNotCopy marker, so every consumer of that channel must copy +// a lock-bearing struct on receive. go vet's copylock analyzer flags that copy, +// even though it is benign here: the delivered value is a freshly built, +// never-locked data carrier, and copying it once is exactly the contract the +// value-typed channel imposes on all callers. +// +// Rather than evade the analyzer with reflection - which still copies the same +// struct while making the copy invisible - the single unavoidable receive is +// performed by the type-safe generic helper receiveFromChannel, and the fields +// are then re-homed into a brand new SignatureData built with a composite +// literal. The returned message therefore owns a fresh, zero-value +// MessageState; the transient received value is never retained or propagated +// past this boundary, and NewSignature reads only the R, S and recovery byte +// slices from it. +// +// The tss-lib dependency is pinned by commit in go.mod. Its external security +// review is a separate release-gate action (not yet archived), so this comment +// does not assert the dependency is already audited. Its value-typed API is +// deliberately not forked to a pointer channel as part of this release; +// changing the channel element type is the correct upstream fix and is tracked +// separately. Until that upstream change lands, the single mandated copy is +// confined here and its lock-bearing MessageState is never retained. +func (fm *finalizingMember) receiveTSSResult( + ctx context.Context, +) (*tsslibcommon.SignatureData, error) { + received, ok, err := receiveFromChannel(ctx, fm.tssResultChan) + if err != nil { + // The context was cancelled before a result was produced. + return nil, fmt.Errorf("TSS result was not generated on time") + } + + if !ok { + return nil, fmt.Errorf("TSS result channel was closed unexpectedly") + } + + // Re-home the produced fields into a freshly allocated, independently-owned + // SignatureData. The received value (and the lock-bearing MessageState it + // copied from tss-lib) is not kept beyond this point. + return &tsslibcommon.SignatureData{ + Signature: received.GetSignature(), + SignatureRecovery: received.GetSignatureRecovery(), + R: received.GetR(), + S: received.GetS(), + M: received.GetM(), + }, nil +} + +// receiveFromChannel performs a context-aware receive from ch. It reports the +// received value, whether the channel delivered a value (false once the channel +// is closed and drained), and a non-nil error if the context was cancelled or +// its deadline passed before a value arrived. +// +// It is generic over the element type so a single, unit-tested primitive covers +// the value receive that tss-lib's value-typed result channel forces on the +// caller. Keeping the receive here - instead of inline at every call site - +// confines the one lock-bearing protobuf copy tss-lib mandates (see +// receiveTSSResult) to a single, well-documented place. +func receiveFromChannel[T any]( + ctx context.Context, + ch <-chan T, +) (T, bool, error) { + select { + case value, ok := <-ch: + return value, ok, nil + case <-ctx.Done(): + var zero T + return zero, false, ctx.Err() + } +} + // identityConverter implements the common.IdentityConverter for tECDSA signing. // It does the conversion using the predefined keys list obtained from Ks // party ID array available in TSS key share. diff --git a/pkg/tecdsa/signing/member_receive_test.go b/pkg/tecdsa/signing/member_receive_test.go new file mode 100644 index 0000000000..4c4e3beb3d --- /dev/null +++ b/pkg/tecdsa/signing/member_receive_test.go @@ -0,0 +1,162 @@ +package signing + +import ( + "context" + "errors" + "testing" + + tsslibcommon "github.com/bnb-chain/tss-lib/common" +) + +// newFinalizingMemberWithResultChan builds the minimal embedded-struct chain +// required to exercise receiveTSSResult in isolation. receiveTSSResult reads +// only the promoted tssResultChan field (defined on tssRoundOneMember), so the +// rest of the chain is left zero-valued on purpose. +func newFinalizingMemberWithResultChan( + ch <-chan tsslibcommon.SignatureData, +) *finalizingMember { + return &finalizingMember{ + tssRoundNineMember: &tssRoundNineMember{ + tssRoundEightMember: &tssRoundEightMember{ + tssRoundSevenMember: &tssRoundSevenMember{ + tssRoundSixMember: &tssRoundSixMember{ + tssRoundFiveMember: &tssRoundFiveMember{ + tssRoundFourMember: &tssRoundFourMember{ + tssRoundThreeMember: &tssRoundThreeMember{ + tssRoundTwoMember: &tssRoundTwoMember{ + tssRoundOneMember: &tssRoundOneMember{ + tssResultChan: ch, + }, + }, + }, + }, + }, + }, + }, + }, + }, + } +} + +func TestReceiveFromChannel_DeliversValue(t *testing.T) { + ch := make(chan int, 1) + ch <- 42 + + value, ok, err := receiveFromChannel(context.Background(), ch) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if !ok { + t.Fatal("expected ok to be true for a delivered value") + } + if value != 42 { + t.Fatalf("expected value 42, got [%v]", value) + } +} + +func TestReceiveFromChannel_ContextCancelled(t *testing.T) { + ch := make(chan int) // never delivers + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + value, ok, err := receiveFromChannel(ctx, ch) + if !errors.Is(err, context.Canceled) { + t.Fatalf("expected context.Canceled error, got [%v]", err) + } + if ok { + t.Fatal("expected ok to be false when the context is cancelled") + } + if value != 0 { + t.Fatalf("expected zero value, got [%v]", value) + } +} + +func TestReceiveFromChannel_ChannelClosed(t *testing.T) { + ch := make(chan int) + close(ch) + + value, ok, err := receiveFromChannel(context.Background(), ch) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if ok { + t.Fatal("expected ok to be false for a closed channel") + } + if value != 0 { + t.Fatalf("expected zero value from a closed channel, got [%v]", value) + } +} + +func TestReceiveTSSResult_DeliversSignature(t *testing.T) { + ch := make(chan tsslibcommon.SignatureData, 1) + // Seed the channel with a composite literal so no existing lock-bearing + // value is copied by the test itself. + ch <- tsslibcommon.SignatureData{ + Signature: []byte{0xaa, 0xbb}, + SignatureRecovery: []byte{0x01}, + R: []byte{0x11, 0x22}, + S: []byte{0x33, 0x44}, + M: []byte{0x55}, + } + + fm := newFinalizingMemberWithResultChan(ch) + + result, err := fm.receiveTSSResult(context.Background()) + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if result == nil { + t.Fatal("expected a non-nil result") + } + + assertBytesEqual(t, "Signature", result.GetSignature(), []byte{0xaa, 0xbb}) + assertBytesEqual(t, "SignatureRecovery", result.GetSignatureRecovery(), []byte{0x01}) + assertBytesEqual(t, "R", result.GetR(), []byte{0x11, 0x22}) + assertBytesEqual(t, "S", result.GetS(), []byte{0x33, 0x44}) + assertBytesEqual(t, "M", result.GetM(), []byte{0x55}) +} + +func TestReceiveTSSResult_ContextCancelled(t *testing.T) { + ch := make(chan tsslibcommon.SignatureData) // never delivers + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + fm := newFinalizingMemberWithResultChan(ch) + + result, err := fm.receiveTSSResult(ctx) + if result != nil { + t.Fatalf("expected nil result on cancellation, got [%v]", result) + } + if err == nil || err.Error() != "TSS result was not generated on time" { + t.Fatalf("expected 'not generated on time' error, got [%v]", err) + } +} + +func TestReceiveTSSResult_ChannelClosed(t *testing.T) { + ch := make(chan tsslibcommon.SignatureData) + close(ch) + + fm := newFinalizingMemberWithResultChan(ch) + + result, err := fm.receiveTSSResult(context.Background()) + if result != nil { + t.Fatalf("expected nil result on channel closure, got [%v]", result) + } + if err == nil || err.Error() != "TSS result channel was closed unexpectedly" { + t.Fatalf("expected 'channel was closed unexpectedly' error, got [%v]", err) + } +} + +func assertBytesEqual(t *testing.T, field string, actual, expected []byte) { + t.Helper() + if len(actual) != len(expected) { + t.Fatalf("%s: expected % x, got % x", field, expected, actual) + } + for i := range expected { + if actual[i] != expected[i] { + t.Fatalf("%s: expected % x, got % x", field, expected, actual) + } + } +} diff --git a/pkg/tecdsa/signing/member_test.go b/pkg/tecdsa/signing/member_test.go index a4f525a2d2..8ff384f732 100644 --- a/pkg/tecdsa/signing/member_test.go +++ b/pkg/tecdsa/signing/member_test.go @@ -12,6 +12,7 @@ import ( "github.com/keep-network/keep-core/pkg/chain/local_v1" "github.com/keep-network/keep-core/pkg/internal/tecdsatest" "github.com/keep-network/keep-core/pkg/operator" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -95,6 +96,7 @@ func TestShouldAcceptMessage(t *testing.T) { groupSize-honestThreshold, membershipValdator, "1", + compatibility.SecurityV2(), big.NewInt(100), tecdsa.NewPrivateKeyShare(testData[0]), ) @@ -204,3 +206,36 @@ func TestIdentityConverter_TssPartyIDToMemberIndex_Corrupted(t *testing.T) { testutils.AssertIntsEqual(t, "member ID", 0, int(memberIndex)) } + +func TestInitializeTssRoundOneConfiguresLegacyTranscript(t *testing.T) { + testData, err := tecdsatest.LoadPrivateKeyShareTestFixtures(1) + if err != nil { + t.Fatalf("failed to load test data: [%v]", err) + } + + member := newMember( + &testutils.MockLogger{}, + group.MemberIndex(1), + 2, + 0, + nil, + "64757a1f-1", + compatibility.Legacy(), + big.NewInt(100), + tecdsa.NewPrivateKeyShare(testData[0]), + ) + + roundOneMember, err := member. + initializeEphemeralKeysGeneration(). + initializeSymmetricKeyGeneration(). + initializeTssRoundOne() + if err != nil { + t.Fatalf("unexpected error: [%v]", err) + } + if roundOneMember.tssParameters.ProtocolMode() != tss.ProtocolModeLegacy { + t.Errorf( + "expected legacy TSS mode, got [%v]", + roundOneMember.tssParameters.ProtocolMode(), + ) + } +} diff --git a/pkg/tecdsa/signing/protocol.go b/pkg/tecdsa/signing/protocol.go index 9814a0c1a9..8fe1367c3b 100644 --- a/pkg/tecdsa/signing/protocol.go +++ b/pkg/tecdsa/signing/protocol.go @@ -83,9 +83,12 @@ func (skgm *symmetricKeyGeneratingMember) generateSymmetricKeys( ephemeralPubKeyMessage.ephemeralPublicKeys[skgm.id] // Create symmetric key for the current group member and the other - // group member by ECDH'ing the public and private key. - symmetricKey := thisMemberEphemeralPrivateKey.Ecdh( + // group member by ECDH'ing the public and private key, using the + // ceremony's pinned key-derivation strategy. + symmetricKey := skgm.strategies.ECDH( + thisMemberEphemeralPrivateKey, otherMemberEphemeralPublicKey, + signingEcdhInfo(skgm.id, otherMember), ) skgm.symmetricKeys[otherMember] = symmetricKey } @@ -733,13 +736,23 @@ func (fm *finalizingMember) tssFinalize( } } - select { - case tssResult := <-fm.tssResultChan: - fm.tssResult = &tssResult - return nil - case <-ctx.Done(): - return fmt.Errorf( - "TSS result was not generated on time", - ) + tssResult, err := fm.receiveTSSResult(ctx) + if err != nil { + return err + } + fm.tssResult = tssResult + + return nil +} + +// signingEcdhInfo returns the HKDF info label for ECDH-derived keys in the +// tECDSA signing protocol. The pair is sorted so both peers compute the same +// info regardless of which side initiates. Each MemberIndex is encoded as a +// single byte; the compile-time assertion in pkg/protocol/group/group.go +// enforces the uint8 invariant this relies on. +func signingEcdhInfo(id1, id2 group.MemberIndex) []byte { + if id1 > id2 { + id1, id2 = id2, id1 } + return []byte{'t', 'e', 'c', 'd', 's', 'a', '-', 's', 'i', 'g', 'n', byte(id1), byte(id2)} } diff --git a/pkg/tecdsa/signing/protocol_ecdh_info_test.go b/pkg/tecdsa/signing/protocol_ecdh_info_test.go new file mode 100644 index 0000000000..e055131383 --- /dev/null +++ b/pkg/tecdsa/signing/protocol_ecdh_info_test.go @@ -0,0 +1,51 @@ +package signing + +import ( + "bytes" + "testing" + + "github.com/keep-network/keep-core/pkg/protocol/group" +) + +// TestSigningEcdhInfoSortSymmetry verifies that the info label is independent +// of argument order. Both peers must derive the same session key regardless +// of who initiates. +func TestSigningEcdhInfoSortSymmetry(t *testing.T) { + for a := group.MemberIndex(1); a < group.MaxMemberIndex; a++ { + for b := group.MemberIndex(1); b < group.MaxMemberIndex; b++ { + if !bytes.Equal(signingEcdhInfo(a, b), signingEcdhInfo(b, a)) { + t.Fatalf("info label not symmetric for (%d, %d)", a, b) + } + } + } +} + +// TestSigningEcdhInfoDistinctPerPair verifies that every distinct sorted +// member pair produces a distinct info label. This is the F-03 invariant that +// would silently break if MemberIndex is ever widened past uint8 without +// updating the encoder. +func TestSigningEcdhInfoDistinctPerPair(t *testing.T) { + seen := make(map[string][2]group.MemberIndex) + for a := group.MemberIndex(1); a < group.MaxMemberIndex; a++ { + for b := a; b < group.MaxMemberIndex; b++ { + label := string(signingEcdhInfo(a, b)) + if prev, ok := seen[label]; ok { + t.Fatalf( + "info label collision: (%d, %d) and (%d, %d) both produce %x", + prev[0], prev[1], a, b, label, + ) + } + seen[label] = [2]group.MemberIndex{a, b} + } + } +} + +// TestSigningEcdhInfoEncoding pins the wire format. Any change here is a +// protocol-breaking change and requires a coordinated network upgrade. +func TestSigningEcdhInfoEncoding(t *testing.T) { + got := signingEcdhInfo(7, 3) + want := []byte{'t', 'e', 'c', 'd', 's', 'a', '-', 's', 'i', 'g', 'n', 3, 7} + if !bytes.Equal(got, want) { + t.Fatalf("encoding drift: got %v, want %v", got, want) + } +} diff --git a/pkg/tecdsa/signing/protocol_test.go b/pkg/tecdsa/signing/protocol_test.go index d5bc520379..cbab18fb3c 100644 --- a/pkg/tecdsa/signing/protocol_test.go +++ b/pkg/tecdsa/signing/protocol_test.go @@ -17,6 +17,7 @@ import ( "github.com/keep-network/keep-core/internal/testutils" "github.com/keep-network/keep-core/pkg/crypto/ephemeral" "github.com/keep-network/keep-core/pkg/internal/tecdsatest" + "github.com/keep-network/keep-core/pkg/protocol/compatibility" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/tecdsa" ) @@ -28,7 +29,7 @@ import ( const ( groupSize = 3 dishonestThreshold = 0 - sessionID = "session-1" + sessionID = "session-1-with-128-bits" ) func TestGenerateEphemeralKeyPair(t *testing.T) { @@ -199,6 +200,7 @@ func TestGenerateSymmetricKeys(t *testing.T) { expectedKey := ephemeral.SymmetricKey( member.ephemeralKeyPairs[otherMemberID].PrivateKey.Ecdh( otherMemberEphemeralPublicKey, + signingEcdhInfo(member.id, otherMemberID), ), ) @@ -261,6 +263,39 @@ func TestGenerateSymmetricKeys_InvalidEphemeralPublicKeyMessage(t *testing.T) { } } +func TestInitializeTssRoundOneSetsSessionNonce(t *testing.T) { + members, err := initializeTssRoundOneMembersGroup( + dishonestThreshold, + groupSize, + ) + if err != nil { + t.Fatal(err) + } + + expectedNonce := new(big.Int).SetBytes(common.SHA512_256([]byte(sessionID))) + for _, member := range members { + testutils.AssertBigIntsEqual( + t, + fmt.Sprintf("session nonce for member [%v]", member.id), + expectedNonce, + member.tssParameters.SessionNonce(), + ) + } + + otherSessionSource := members[0].symmetricKeyGeneratingMember + originalSessionID := otherSessionSource.sessionID + otherSessionSource.sessionID = "other-session-with-128-bits" + otherSessionMember, err := otherSessionSource.initializeTssRoundOne() + if err != nil { + t.Fatal(err) + } + otherSessionSource.sessionID = originalSessionID + + if expectedNonce.Cmp(otherSessionMember.tssParameters.SessionNonce()) == 0 { + t.Fatal("initialized TSS members should use different nonces for different session IDs") + } +} + func TestTssRoundOne(t *testing.T) { members, err := initializeTssRoundOneMembersGroup( dishonestThreshold, @@ -2436,6 +2471,7 @@ func initializeEphemeralKeyPairGeneratingMembersGroup( id: id, group: signingGroup, sessionID: sessionID, + strategies: compatibility.SecurityV2(), message: big.NewInt(100), privateKeyShare: tecdsa.NewPrivateKeyShare(testData[i-1]), identityConverter: &identityConverter{keys: testData[i-1].Ks}, @@ -2528,10 +2564,16 @@ func initializeTssRoundOneMembersGroup( ) } - tssRoundOneMembers = append( - tssRoundOneMembers, - member.initializeTssRoundOne(), - ) + tssRoundOneMember, err := member.initializeTssRoundOne() + if err != nil { + return nil, fmt.Errorf( + "cannot initialize TSS round one for member [%v]: [%v]", + member.id, + err, + ) + } + + tssRoundOneMembers = append(tssRoundOneMembers, tssRoundOneMember) } return tssRoundOneMembers, nil diff --git a/pkg/tecdsa/signing/signing.go b/pkg/tecdsa/signing/signing.go index f8b981cb84..24344d9eeb 100644 --- a/pkg/tecdsa/signing/signing.go +++ b/pkg/tecdsa/signing/signing.go @@ -11,6 +11,7 @@ import ( "github.com/keep-network/keep-core/pkg/net" "github.com/keep-network/keep-core/pkg/protocol/group" "github.com/keep-network/keep-core/pkg/tecdsa" + "github.com/keep-network/keep-core/pkg/tecdsa/common" ) // Execute runs the tECDSA signing protocol, given a message to sign, @@ -18,6 +19,11 @@ import ( // a member index to use in the group, private key share, dishonest threshold, // and block height when signing protocol should start. // +// The strategies bundle pins the ceremony's compatibility decisions — the +// ECDH derivation and the TSS proof-transcript configuration — and must be +// selected explicitly from the ceremony's participation permit; there is no +// default. +// // This function also supports signing execution with a subset of the signing // group by passing a non-empty excludedMembers slice holding the members that // should be excluded. @@ -33,6 +39,7 @@ func Execute( excludedMembersIndexes []group.MemberIndex, channel net.BroadcastChannel, membershipValidator *group.MembershipValidator, + strategies common.CompatibilityStrategies, ) (*Result, error) { logger.Debugf("[member:%v] initializing member", memberIndex) @@ -43,6 +50,7 @@ func Execute( dishonestThreshold, membershipValidator, sessionID, + strategies, message, privateKeyShare, ) diff --git a/pkg/tecdsa/signing/states.go b/pkg/tecdsa/signing/states.go index 47259dab5c..f633486b3e 100644 --- a/pkg/tecdsa/signing/states.go +++ b/pkg/tecdsa/signing/states.go @@ -99,10 +99,15 @@ func (skgs *symmetricKeyGenerationState) CanTransition() bool { } func (skgs *symmetricKeyGenerationState) Next() (state.AsyncState, error) { + member, err := skgs.member.initializeTssRoundOne() + if err != nil { + return nil, err + } + return &tssRoundOneState{ BaseAsyncState: skgs.BaseAsyncState, channel: skgs.channel, - member: skgs.member.initializeTssRoundOne(), + member: member, }, nil } diff --git a/renovate.json b/renovate.json new file mode 100644 index 0000000000..5db72dd6a9 --- /dev/null +++ b/renovate.json @@ -0,0 +1,6 @@ +{ + "$schema": "https://docs.renovatebot.com/renovate-schema.json", + "extends": [ + "config:recommended" + ] +} diff --git a/scripts/release/pr4109/README.md b/scripts/release/pr4109/README.md new file mode 100644 index 0000000000..0b0df49742 --- /dev/null +++ b/scripts/release/pr4109/README.md @@ -0,0 +1,2327 @@ +# Release rehearsal and smoke harnesses + +This directory holds the release-engineering scaffolding for the coordinated +security release: + +1. the container smoke matrix for the temporary `clientInfo.port` **9601 + compatibility default** — `clientinfo-port-smoke.sh` and `compose.yaml`; +2. the single-release **cutover rehearsal** scaffold — `rehearse.sh`, + `compose.rehearsal.yaml`, and `rehearsal-evidence.schema.json`, driven + manually or through the `cutover-rehearsal` workflow; and +3. the **release manifest** binding the service-manager termination grace to + the client's compiled protocol bounds — `release-manifest.json`, + `release-manifest.schema.json`, and the deployment scaffold under + `deploy/`. + +## Cutover rehearsal scaffold + +The chain-clocked cutover machinery — the participation gate, per-ceremony +permits, commit fences, quiescence and the signal lifecycle controller, and +the signer quarantine namespace — is implemented in this tree and proven by +repository-local Go tests, together with the tBTC cutover ceremony +acceptance suites under the race detector: real security-v2 key-generation +transcripts — including the ten-misbehaved-seat real result — the +production-scale 90/10 split exclusion, heartbeat inactivity bands, and +cutover roster wiring, ending with an explicit report of every skipped case. +Run those proofs, which need no Docker or chain, with: + +``` +./rehearse.sh local-proofs +``` + +Three sibling stages cover the rest of the changed risk surface locally: +`./rehearse.sh static-analysis` runs the CI-enforced Go analyzers with +every tool at an immutable version — gofmt, `go vet ./...` (strictly wider +than CI's root-only vet), staticcheck 2025.1.1, gosec v2.28.0 (CI's own +gosec action floats on `master`; the pin keeps the evidence reproducible), +and golangci-lint v2.12.2 — `./rehearse.sh solidity-proofs` builds and +tests the ECDSA contracts exactly as the contracts workflow's +`contracts-build-and-test` job does: the exact Node release that job pins, +read out of it rather than restated here, plus the Corepack-managed yarn +from `packageManager` and a never-skipped `yarn install --immutable` before +`yarn build` and `yarn test` — and `./rehearse.sh shell-analysis` analyzes this scaffold +itself: `bash -n` and ShellCheck over every script here, actionlint v1.7.12 +over the scaffold's own workflows (scoped to them on purpose; the unrelated +workflows carry pre-existing findings, and a gate that is red for reasons +outside its scope stops being read), and all boundary self-tests. That +last stage is the one CI runs unconditionally — see +`.github/workflows/cutover-scaffold-lint.yml` below — because the checkers +that admit rehearsal evidence must never be proved only by a manual +dispatch. Every stage stamps the exact source commit into its log, +marking any divergence from `HEAD` — untracked files included — as +`-dirty`. Setting `PR4109_EXPECTED_SOURCE_COMMIT` makes the stamp a +fail-closed binding instead: the stage refuses to run at all unless the +tree under test is exactly that commit, so a log carrying a verified stamp +is proof the stamped bytes were the tested bytes. + +For a fast local or automation-loop completion check, use the repository-owned +subset gate from the repository root: + +``` +make verify-cutover +``` + +It runs `go build ./...`, `go vet ./...`, the full unit suite, the protocol +packages under the race detector, and `shell-analysis` as one fail-fast +command. A Go-only completion command is not sufficient for changes that can +reach this scaffold, because it does not exercise the workflow and evidence +validators. This is an automation-completion subset, not release approval: it +does not run the separately evidenced `local-proofs`, `static-analysis`, or +`solidity-proofs` stages and does not exercise either exact-image rehearsal. + +To run every repository-local release stage through one entry point while +preserving each stage's separate evidence log, use: + +``` +make verify-cutover-release-local +``` + +That target adds `local-proofs`, `static-analysis`, and `solidity-proofs` to +the build, vet, full unit suite, protocol race suite, and `shell-analysis`. +It still cannot authorize a release: the native-platform exact-image cutover +and homogeneous rollback rehearsals remain mandatory external gates. Set +`EVIDENCE_DIR` for either target to place logs somewhere other than the ignored +repository-root `rehearsal-evidence/` directory. + +The offline state classification the rollback barrier requires runs with +`go run ./cmd/participation-state-audit --storage-snapshot `: it +records the snapshot identity (aggregate checksum and access mode), flags any +entry outside the expected storage layout, and — when the storage password is +supplied — interprets the beacon active, beacon quarantine, and tBTC active +namespaces with the same decode paths the client's own loaders use, +cross-validating quarantine metadata against its schema, epoch, mode/anchor +arithmetic, storage location, and decrypted membership. Namespace consistency +alone is never rollback-ready: the audit exits nonzero until references to +the chain reconciliation, Bitcoin reconciliation, quiescence outcome, and +prior-reader compatibility evidence are supplied via its +`--*-evidence`/`--quiescence-report` flags **and** the expected operational +identities the evidence must bind to are supplied via its `--expected-*` +flags — Ethereum chain ID, Bitcoin network, the exact prior and current +release versions, revisions, and immutable image digests, the release epoch, +the armed cutover block, and the evidence freshness bound. Chain evidence has +additional independent inputs: the exact WalletRegistry and RandomBeacon +addresses, a finalized Ethereum block number/hash obtained outside the evidence +generator, and the lowercase hexadecimal Ed25519 public key of the trusted chain +collector. Its output never authorizes activating quarantined material by +itself. +For every reconciled tBTC wallet whose DKG settlement is not `none`, schema +v8 requires the complete `DkgStarted`, `DkgResultSubmitted`, +`DkgResultApproved`, and `WalletCreated` event lineage. Each event names its +transaction hash, block hash/number, and log index; the submitted event carries +the full on-chain result tuple, not caller-supplied summary fields. The same +record carries the successful receipt projections and exact address, +topics/data, and log indexes. The audit verifies the collector signature over +the entire record, requires each receipt block in the signed canonical set, +binds that set to the independently supplied finalized block, rejects logs +from any address other than the independently supplied WalletRegistry, and +re-derives every decoded event field from the raw log bytes. A self-consistent +history produced by an untrusted generator, a failed receipt, an unrelated +contract, or a non-canonical block therefore blocks rollback. The audit also +recomputes `keccak256(abi.encode(result))`, the operating-members hash, the +seed hash, and the wallet ID, requires approval and wallet creation to share +one receipt, and derives the original and final group shapes from those bytes. +Its `signing_member_indexes` decode at the contract's full `uint256` width +before the protocol's one-byte group bounds are enforced. For an approved +wallet it then maps each original DKG permit to the persisted final membership; +a forged result hash, unrelated approved wallet, or two persisted memberships +swapped between permits blocks rollback. + +The same record settles the penalties a node recorded as its permits' results. +A heartbeat's inactivity claim must be corroborated by a canonical +`InactivityClaimed` log for the exact wallet and nonce named, emitted by the +supplied WalletRegistry, and the punished wallet must be the one the permit is +about. A relay entry timeout report must be corroborated by a canonical +`RelayEntryTimedOut` log naming the exact request identifier and terminated +group, emitted by the supplied RandomBeacon, whose `RelayEntryRequested` log +sits in the very block the reporting permit was issued for and which no +`RelayEntrySubmitted` log answers. Filing a report is not earning a penalty — a +transaction that reverts, is dropped, or loses the race to another reporter +leaves the beacon exactly as it was and renders exactly the same node-authored +reference — so a settlement the beacon's own logs do not carry blocks rollback. + +A completed relay signing outcome is bound the same way. Its recovered entry is +a threshold BLS signature the audit verifies outright, but authorship is +forever: nothing in the record ties the entry to the ceremony the permit was +issued for except a start block the node wrote beside it. So the supplied +RandomBeacon must carry a `RelayEntryRequested` log in that block signing over +the very previous entry the record names, and where a `RelayEntrySubmitted` log +accepted an entry for that request it must be the entry the node named. A +submission is not required — the group's threshold recovers the entry whoever +publishes it, and a submission that reverted or lost the race says nothing about +the ceremony — but a missing or contradicted request does block. + +That request also names the group the beacon selected, and the entry must be +that group's. Relay entries are deterministic, so an entry another group the +node holds a membership of produced over the same previous entry is a real +signature that verifies and answers this very request, while recording work the +selected group never did. The selection names a registry index, and only the +`GroupRegistered` log binds that index to the hash of the key the group signs +under, so **the selected group's registration receipt is required too** — it is +older than the request, often by months, and is the one receipt that gathering +evidence around the request will not produce by itself. Omitting it blocks +rollback rather than passing, because an omission that waived the check would be +indistinguishable from the omission an attacker needs. The generator must +therefore include the beacon's relay request, submission, and timeout receipts +for every relay permit the drained node recorded a result for, plus the +`GroupRegistered` receipt of the group each of those requests selected. + +The rollback rehearsal runs that audit **twice over the same snapshot**, and +the order is the whole point. Every external record must carry the audited +snapshot's `snapshot_aggregate_sha256`, and the audit rejects any that names +another — so that checksum is a fact about state this rehearsal has only just +produced by draining the fleet and copying it out. Evidence handed in before +the fleet drained could not have known it, and evidence that named it anyway +would be describing a drain that had not happened. So the first pass runs with +no evidence at all and exists to derive the snapshot identity and interpreted +inventory; the supplied `PR4109_ROLLBACK_EVIDENCE_GENERATOR` is then executed +as ` ` and must write +`chain-reconciliation.json`, `bitcoin-reconciliation.json`, +`quiescence-report.json`, and `prior-reader-compatibility.json` for that +snapshot; and the second pass, over those records, is the one that authorizes +anything. A generator that failed, that wrote only some of the four, or that +cannot be run leaves the barrier unestablished rather than the audit refusing. +The node writes an encrypted +`work/participation/quiescence/gate-snapshot.json` artifact while the gate +holds the same lock that changes its state to `quiescing`. That node-authored +artifact binds the exact active-permit registry and total, legacy, and +security-v2 counts to the running version, revision, compiled epoch, C, +current block, cause, and transition instant. It is part of the stopped +storage snapshot and therefore of `snapshot_aggregate_sha256`; the external +evidence generator cannot replace it with a second self-attested inventory. +Each process run invalidates the prior artifact before constructing its gate, +so a restart or failed new capture leaves the rollback audit fail-closed +instead of exposing stale inventory. +The schema-v8 quiescence record supplies only the later +`active_permits_at_quiescence` terminal-outcome list, which must cover the +node-authored inventory one-to-one and reproduce all three counts. An empty or +shortened outcome list over a nonempty gate artifact blocks rollback. Both +records name `work_id` and `permit_id` as well as ceremony, mode, and anchor. +DKG claims use the canonical SHA-256 seed hash (exactly 64 lowercase +hexadecimal characters) and the canonical decimal member index (1 through +255) respectively. Beacon relay-signing permits likewise use the member +index; other work and permit identities use the driver's stable identifier +alphabet. The audit rejects an unbound or repeated full permit identity, a +zero canonical anchor under an armed schedule, and any identity contradicting +the cutover arithmetic (`legacy` below C, `security_v2` at or above C), +including completed outcomes. Quarantined claims additionally match the exact +local permit rather than letting one output cover another event or membership +at the same block. + +The identities the audit binds that evidence to — the release being rolled +back: version, revision, epoch, and armed C — come from what the R1 fleet +itself reported while it was still up, so the rollback is authorized against +what ran rather than against what anyone believed ran. The stage then reads +`rollback_barrier_ready` out of the manifest the authorizing pass wrote. Both +manifests and the generated records are kept under `state-audit/` in the +evidence directory whether the audit authorized the rollback or refused it — +a refusal is the part of a rollback decision most worth reading — and one +level down rather than beside the rehearsal record, because the acceptance +stage validates every top-level JSON against the rehearsal record schema and +an audit manifest is a different document. + +The two **container** rehearsals are mandatory release gates that cannot run +from this repository alone. For every platform the detached release provenance +publishes, they need a native runner, the immutable prior-production and R1 +runtime image digests, an equally immutable probe image digest, and an isolated +rehearsal chain that begins below its own C with deployed beacon/tBTC contracts +and its chain id. Each platform also needs per-node operator keys and configs +under `//`, with the configs naming that chain's +contracts and declaring a nonzero `clientInfo.port`; reusing one already-crossed +chain for another platform is not a rehearsal of that platform's cutover. The +shared inputs are a work driver that originates protocol work on those chains +and (for rollback) a directory to capture each drained node's state into. +`rehearse.sh preflight` validates those inputs and reports `BLOCKED` with the +exact missing one. The rollback gate additionally needs the audit inputs no +storage snapshot can supply — the +rollback evidence generator that produces the chain and Bitcoin +reconciliation, quiescence outcome, and prior-reader compatibility records for +each captured snapshot, the independently provisioned WalletRegistry/finalized +block/collector-key trust inputs, plus the Bitcoin network and the prior +artifact's version and revision — because without them the audit can classify +namespaces and authorize nothing. + +Once preflight passes, `single-release` and `rollback` **run**: each drives +its gate as an explicit sequence of steps, starting the fleet from the +immutable digests, reading every number it records from the nodes' own +client-info ports over the internal rehearsal network, and recording each +step's own outcome. Which services a gate starts is part of what it proves: +the cutover rehearsal needs the prior binary on the network from the start, +because it *is* the straggler the negative control is about, while the +rollback rehearsal starts only the R1 fleet — its whole subject is that no +prior binary participates until the barrier holds, and a fleet that brought +the prior service up with everything else would have put the thing under test +on the network before the first step ran. It does *stage* the prior +container: created from the audited digest, proved not running, and left off +the network, because `compose start` can only start something that exists and +a rollback project that created nothing would leave the release step +recording a rollback it never performed. + +Before either gate touches the fleet it proves the containers are running the +supplied digests — image IDs compared against what the daemon actually created +each container from, because a stale local tag or an edited compose file +otherwise produces a fleet running other bytes under a record naming these +ones — and then captures what that fleet says it is: version, revision, +compiled protocol epoch, and armed cutover block, from *every* R1 node and not +the first. Any disagreement between nodes refuses the run, as does a revision +that is not exactly the commit the run is bound to — an abbreviation names a +commit only as far as it goes, which is why the release workflow stamps the +whole SHA and `shell-analysis` holds it to that. The same gate requires both +release jobs to resolve their version with `release-trigger-tag.sh` from the +exact `github.ref`/`github.ref_name` pair that caused the run. Repository tag +discovery is forbidden because a stable tag and a candidate tag can point at +the same commit without identifying which one triggered the workflow. The +publisher then passes that single identity to `release-docker-tags.sh`: every +tag publishes a versioned image, but only an exact stable +`vMAJOR.MINOR.PATCH` release may move `latest` or `mainnet`. The GitHub Release +prerelease decision consumes the same identity. A candidate or any +unrecognized version shape remains version-only, so publishing an image for +rehearsal cannot change either production alias. +The fleet capture also refuses an armed cutover block that is not the rehearsed +C, or a protocol epoch that is not the one the reviewed manifest was derived +for. The record is +then built from what was captured rather than from what the driver was told, +so its epoch and C are the fleet's own and not a restatement of the +environment. Capturing up front is also what lets the rollback gate emit a +record at all: by the time it concludes it has stopped every R1 node on +purpose, and a reading taken then would be no reading. A step this release cannot execute is recorded `blocked` +with the reason rather than aborting the run, because the steps after it are +independent proofs and losing them tells a reviewer less than a record naming +exactly which step could not run. A step that *did* run and observed the +property violated is recorded `fail` the same way, and an acceptance +assertion is written `true` only where the run watched the property hold, so +an unobserved one reads as refused rather than as satisfied. + +Every run therefore ends with an evidence record on disk — checked for +per-record admissibility by the acceptance stage's own validator — and the +stage's exit is decided from the recorded outcomes. A platform runner does not +ask the archive-wide completeness question while emitting: the other runners' +records do not exist in its workspace yet. A failed step is the strongest +verdict and exits +`FAIL`: the rehearsal reached the property, watched it, and watched it break, +which outranks anything the run could not reach. A step that never executed +exits `BLOCKED`: the gate is unproved rather than disproved. A refused +acceptance assertion with no step behind it exits `FAIL` too. Only a run with +none of the three reports success. A partial rehearsal can never read as a +passed gate, a failed one can never read as either, and a refused gate is +never silent about what it did prove. + +Each step is held to the property it names rather than to a proxy for it. The +crossing step establishes the pre-C side first — every node reporting +`open_legacy` at a block below the C it armed — because a fleet started after +C already reports `open_security_v2` and would satisfy every closing check +without having crossed anything; it names a permit mode in the record only +where security-v2 permits were actually observed. The homogeneous positive +control requires the fleet's security-v2 permit total to *rise* while the +work driver runs, since a zero legacy counter is equally true of a fleet that +ran nothing — and it compares the legacy counter as a delta across the step, +because that counter is cumulative and the pre-C legacy controls this same +gate requires would otherwise fail this step on permits taken before C. It +also requires the driver to have reported the transactions it submitted, so a +counter that moved for some unrelated reason is not credited to ceremonies +nobody can show were originated. + +The in-flight half of the crossing names the permits rather than counting +them. The gate publishes its live permits at `/diagnostics` +(`protocol_participation.active_permits`), so the control reads the fleet's own +list of what it holds before C, again at the instant every gate reports +`open_security_v2`, and requires both to be exactly the permits the driver said +it put there — no named permit the gates never held, and no unnamed permit +crossing beside them. A count that moves in step is satisfied by any two +unrelated ceremonies, and a permit gone by the crossing now fails the step +rather than being read off a completion counter that happened to move +afterwards. Only identity-bound permits count: an unbound permit names no chain +work, so matching one would be reading the count again under another name. + +One permit legitimately arrives between the two readings — the quiescence +control's seed, put on the chain after this work was originated and before the +crossing — and it is excused by identity, not by node. What is excused is the +intersection of the two independent readings of that seeding: the identities +the driver said it originated on the seed node, and the identities that node's +own gate reported holding below C. The gate's reading alone is that node's +whole legacy population, so excusing it wholesale would wave through any permit +that merely turned up there between the samples; the driver's alone is the +driver's word for a below-C anchor, which is what the seeding is checked for. A +gate that could not be read below C agrees with nothing and excuses nothing. + +#### What a permit's ending rests on + +The in-flight half above names permits from the gates' own live readings. What +became of each named permit used to come from somewhere else entirely: the +crossing verdicts read terminal outcomes, contributor identities, and +per-permit results out of the work driver's report and checked them against +fleet-wide counters and the chain. The driver both originates the work and +reports how it ended, so those two readings were not independent — a counter +moving the expected amount beside a driver's claim that it moved for the named +permits is satisfied by a report that simply says so. + +The node's own answer now stands between them. A permit used to disappear from +`protocol_participation` the moment its ceremony finished, so before a +quiescence transition there was no node-authored record of an ending to read at +all. The gate retains what its ceremony owners recorded for the permits it +closed and emits it at +`protocol_participation.recent_terminal_outcomes`, carrying the same +`work_id`/`permit_id` identities the live list carries — so a permit the +crossing control saw held is followed to the disposition its own node recorded. + +A completed record also names who reached the result, not only that one exists. +The ceremonies whose owners authenticate their peers publish the memberships +whose contributions they combined into the result and, separately, the +memberships they operated themselves, and their records are refused without +both; a ceremony that authenticates no population — a forwarder relaying other +members' shares, a coordination proposal from one leader — is refused with one. +A completed DKG record additionally names the membership it persisted, which is +a final signing-group seat rather than the DKG seat its permit was issued under. +Every reading below takes those fields as it finds them: a record that omits one +is unreadable rather than filled in. + +The surviving-work verdict, both quiescence verdicts, both pre-cutover +verdicts, and the homogeneous positive control decide on that reading. Every +permit they named must appear in the closed-permit account exactly once, and +the ending they require is the one the holder recorded, not the one the driver +reported: + +- A gate that cannot be asked leaves the crossing unobserved rather than + shortening the account, since a node that answers nothing and a node whose + permits all ended unrecorded read the same otherwise. +- A named permit with no record at all blocks. So does eviction — the gate's + account is bounded and forgets its oldest first — because to a reader those + are the same thing: no node will vouch for how this permit ended. +- A permit with two records blocks. One permit ends once, so a second record is + either a duplicate or two dispositions for one ceremony, and neither can be + read as the answer. +- A permit whose owner recorded nothing is written `unresolved` by the gate + itself and *fails* the control rather than blocking it. It is a real ending, + and a permit whose holder cannot say where its ceremony went is not one that + was allowed to finish across the crossing. +- An ending outside what the control allows fails the same way, even where the + driver reports a settlement for it. Which endings those are is the control's + own question: the crossing requires work to complete, while a drain is + satisfied by completion or by audited quarantine. +- A chain side effect the holder dispatched and could not name blocks. The gate + lets a settlement be recorded without its canonical identity on purpose — a + node that submitted a transaction and could not learn what became of it has to + say so rather than record a settlement it cannot name or leave the submission + invisible — and a step reading past it decides on work while this fleet may + have left chain state behind that nobody can account for. Resolving it is the + offline audit's job. + +The quiescence controls take their reading *inside* the drain window rather +than after it, for the same reason the refusal counter is sampled there — the +node stops answering when the drain finishes, and its account of what it closed +goes with it. The account only grows as permits close, so the last reading taken +before the node goes away is the one carrying every permit that ended in the +window. + +Within those six verdicts the driver's account is kept beside the decision +rather than making it: it carries the settlement identities and transaction +hashes the chain corroborates, neither of which a gate scrape knows, and it no +longer says how a permit ended. Each of them also refuses an outcome for work +that phase did not originate on the transaction that originated it, and +originated work no outcome covers — a control that reads only the work its +driver chose to report on is satisfied by the subset that went well. + +The rollback reconciliation reads the same account, sampled per node inside its +own drain window for the same reason the quiescence controls sample theirs. +Every permit a node held at the stop must appear in that node's own record of +what it let go of, exactly once, ending in a completion, an exhaustion, or a +quarantine. `unresolved` — what the gate writes for a permit closed by an owner +that recorded nothing — fails the step, because that is precisely what a process +going down while holding a permit leaves behind, and a driver terminal record +says the same thing about it as about a ceremony that finished. + +The contributor set — which parties a settled transcript incorporated — is what +the mixed-release steps distinguish a mixed committee from a homogeneous one by. +Its R1 half is no longer the driver's word. Every R1 node publishes the permits +it closed, the durable result each one produced, and which of the seats behind +that result were its own, so "an R1 node was a party to this transcript" is +answered out of that node's own record, at the whole permit identity and for the +exact piece of chain work, with the driver taking no part in producing the +answer. The driver's list is then reconciled against that derivation in both +directions before any verdict reads it: a claimed R1 party the fleet published no +contribution for is refused, including a real holder named under a permit it +never took — which is how one contribution would otherwise be counted as the +several a threshold needs — and a holder that recorded only watching the result +finish, which is a completion and not a contribution — and a contribution the +fleet did publish that the list omits is refused too, so the set is the +population that ran rather than a subset of it. A holder that is neither the prior binary nor a +node this rehearsal runs is refused outright, because a third name is neither +half of the claim. The two releases are required to meet in one *piece of work* +rather than in one ceremony, matched in the driver's vocabulary and not the +gate's: the gate spells a wallet action and a signing alike, so a prior share on +the wallet action and an R1 completion on the signing beside it stays two +homogeneous transcripts. + +The prior share is no longer the driver's word either, and it is read without +the prior binary having to say anything. Each R1 holder publishes the memberships +whose authenticated contributions it combined into its result and, separately, +the memberships it operated itself: a seat in the first set that no node in the +fleet claims in the second is a seat some node outside the fleet was sitting in, +and the only other release on this network is the prior binary. A run this fleet +performed alone therefore leaves no such seat however the report describes it, +and a driver cannot add one — the seats come from done checks each R1 node +validated against the wallet's on-chain signing group, all carrying the one +signature the attempt agreed on; from the final signing group a DKG built out of +exactly the members whose rounds it accepted; from the operating members of the +beacon DKG result this node's rounds produced; and from the relay shares each +authenticated against the group public key share published for the membership +that sent it. Every gated ceremony that reaches a threshold result publishes one, +and the gate refuses a completed record for any of them that publishes none, so +each required ceremony is held to the same node-authored reading with no fallback +to the report. + +Both halves are read out of **one** published transcript, never out of the +fleet's several transcripts added together. The fleet-wide account is used for +one thing only — which seats some node under test operated, the ownership map — +and the mixed reading then asks a single record's incorporated population to +contain a seat inside that map and a seat outside it. Aggregating the +populations instead would invent transcripts: a threshold output can be recovered +from different subsets of the same ceremony, so two holders naming one result +need not name one population, and an R1-only record beside a prior-only +observer's record of the same work would union into a mixed seat set that neither +record contains and no ceremony ever had. + +The ownership map is built from **permits** rather than from transcripts, and +that distinction is what makes "outside the map" a statement about the fleet. A +transcript exists only where a ceremony reached a result, so a map assembled from +transcripts covers the holders that finished and silently omits the ones that +contributed and then crashed, timed out, exhausted their retries, or closed +without recording anything. Every one of those operated its seats. The gate +therefore records the seats each holder operates when the permit is *issued* — +before any outcome exists — and carries them on every reading of that permit, +live, quiesced and terminal, whatever the ending; the field is +`operated_members` on each permit in the diagnostics scrape and in the terminal +journal, which is what version 5 of that journal and version 2 of the gate +snapshot add. The snapshot side is not redundant: it is the issuance-time copy of +the same set, so the offline audit reconciles the two rather than taking the +journal's own account of what it was issued. A map missing those seats +has one false positive and it is in the direction the control is used: an all-R1 +ceremony one of whose contributors never published an ending presents that +contributor's seat as outside the fleet, and a homogeneous run then satisfies a +control whose whole claim is that two releases combined into one output. + +The map spans the permits a node still holds as well as the ones it closed, and +both halves come out of one gate response. An ending is the only thing that puts +a permit in the closed account, while the driver reports when the chain settles +and a holder closes on its own schedule — so a contributor whose permit outlives +the report appears in neither list, its seat leaves the map with no counter +moving anywhere, and the ordinary outcome of that race reads as a seat the other +release supplied. Splitting the two readings across separate requests has the +same effect on a permit that closes between them, which is why the held permits, +the endings, and the provenance of the account they came from are read together. + +The two node-authored accounts of one permit are held against each other. A +completed record whose transcript claims a seat its own permit was not issued to +operate is refused at the moment it is written and refused again by the offline +audit, and the mixed reading declines to classify the work at all rather than +choose between them. + +The shape rules on `operated_members` are applied by the evidence reader as well +as by the gate that enforces them at issuance. Holding a transcript to its own +permit only binds the permits that published a transcript, and the records the +ownership map most depends on are the ones that did not: a DKG or relay signing +permit ending unresolved carries no transcript to be checked against, so a seat +widened after the fact would enter the map unopposed, take that seat from the +node whose permit really held it, and leave the seat reading as supplied from +outside the fleet. Those ceremonies run one seat per permit and name it in the +permit ID, which is the statement the reader holds the operated set to. + +tBTC DKG is compared through a mapping rather than exempted from the comparison. +Its permit names a DKG member index while its transcript and persisted membership +are in the final signing group's index space, rebuilt after inactive and +disqualified members are removed, so the same node legitimately runs seat 9 of +the ceremony and lands in seat 8 of the group. The transcript therefore carries +the ceremony seat behind each of its own seats, positionally aligned with them +and taken from the accepted result's own operating members — the field is +`permit_space_members` on the transcript contribution, which is what version 6 of +the terminal journal adds. It is required for that ceremony and refused for the +ones whose record already speaks in the space their permits name. + +The mapping is what makes an ownership map readable against such a transcript at +all. Without it the map's seats and the transcript's seats are numbers from two +unrelated spaces, and with a middle ceremony member removed every final seat +shifts down: joining the two by number attributes a final seat to whichever party +holds that number in the other space, which manufactures the mixed reading out of +a homogeneous run. Where a work published no usable mapping — none at all, or two +holders disagreeing about how one final group was rebuilt — the fleet's ownership +of its transcripts is reported as unknown and the step blocks, rather than the +reading treating the seats it could not place as seats the fleet never operated. A +holder the ceremony removed is a separate case and not an unknown: its seat is +absent from the survivor list, which says definitely that it holds no final seat. + +Offline, the whole mapping is derived from the accepted result rather than +checked at the recording node's own seat. A final signing group is the accepted +result's members with its misbehaved seats removed, ascending, so final seat *i* +came from the *i*-th survivor and none of the map is the recording node's to +state. Checking only the author's own entry left every other entry — which is +precisely the part that says who the other seats belonged to — accepted on the +author's word: a record whose remote entries were rewritten still names the right +seed, the right anchor, the right length and the right local seat, and the +fleet-wide map translated through it hands final seats to members that never held +them. + +The runtime side of the same binding is what a node activates against. A DKG +signer is activated only when the result observed to settle on chain is the +result this member generated — same ceremony, same key, same members removed — +rather than when any result settled at all. A mismatch takes the path a +publication window closing empty already takes: the share is preserved without +activation, for the offline audit to reconcile. + +The account those readings come out of lives in memory, so whether it can be +followed is published with it. The gate names the process the account belongs to +at `protocol_participation.gate_instance` and counts the records its bounded +account has dropped at `forgotten_terminal_outcomes`. Both are read either side of +a drive: a node answering from a different process than the one the work ran on +has lost every record the old process held, and a node whose account forgot +records while the work ran has lost some of them and cannot say which. Either +blocks the step, because an account missing a permit reads exactly like a node +that took no part in work it may well have done. A process lost before it closes +anything writes no record at any point, and no reading of the account recovers a +seat that was never written to it — the provenance check is what keeps that +silence from being spent as evidence. + +Provenance a node declined to state blocks on sight rather than being compared. +The gate composes its identity from the system entropy source and says the +identity is unknown when that source fails, and a comparison carrying that +answer through as text finds the two readings equal — which is the followable +verdict, drawn on the case that was meant to be unfollowable. The reader renders +an identity that is not one, and a dropped-permit count that is not a count, as +the one token that can be neither. A count that went backwards is refused the +same way: the account only ever forgets, so a smaller number is a reading no gate +produced rather than a negative quantity of dropped permits. + +Completing a ceremony and contributing to it are held apart. A wallet action owns +its permit and records the signature it saw settle even when the attempt that +produced it selected none of the memberships this node operates, which is the +honest ending of a permit whose ceremony ran without it — and not a contribution. +The R1 half of every mixed claim is therefore a seat the holder says was its own, +so an R1 node that only watched a prior-only result finish supplies neither half +of the claim, and a driver naming it as a party is refused. + +What a seat in a transcript attests is bounded by the wire the two releases +share, and the bound belongs in the evidence rather than in a reader's +assumptions. A compatibility release cannot add a field to a message the prior +binary already parses, so neither of the two signing populations can be bound to +the protocol session it is joined to: + +- A **tBTC done check** carries the message, the attempt number, the signature, + and the block its sender finished at, and nothing derived from the attempt's + own transcript. Holding the end block to the attempt's window is what refuses + the earlier run's messages — the same message and attempt number recur when a + wallet is asked to sign again under a later anchor, with numbering restarting — + whether they were honestly retransmitted or replayed by anyone who captured + them. A membership the attempt selected that asserts an in-window end block + over an output it did not compute is not refused by it. A seat here therefore + says "this membership confirmed this result, under an identity the chain + accounts for, inside this attempt's window", which is an attestation and not a + proof of computation. +- A **beacon relay share** is authenticated against the group public key share + published for the membership that sent it and against the previous entry being + signed, and names no request. A relay entry is deterministic in its previous + entry, so a request following one that timed out without a submission signs the + same previous entry and a share belonging to either is valid for both. A seat + here says "this membership held the private share behind its seat and put it + into this recovered entry" — the entry, not the request an audit joined it to. + +Both limits are load-bearing for how the mixed-release verdicts should be read: +they say the prior release's shares combined with this fleet's into one threshold +output, which is the compatibility claim, and they do not by themselves establish +that a given party was live at a given anchor. That the prior binary was up and +serving across each step is evidence the rehearsal collects separately, from the +container it runs rather than from any transcript. + +The straggler control reads the announcer's own account of the sighting +rather than the gate's refusal counter, which counts a node declining its own +`Begin` for reasons that need no legacy announcement behind them. It requires +the whole chain: a session-ID mismatch arrived, this node recognized it as +cross-format, that recognition became a legacy roster addition, and the +roster names an operator it had not already seen. A mismatch nothing +recognized as cross-format fails the step rather than leaving a gap — the +release's premise is that a straggler is identified — and the roster object +exists from startup with an empty peer list, so its presence proves nothing. + +The clock-failure step reads its contract as two halves and needs evidence +for both. With the endpoint severed the gate must report `clock_unavailable`; +it must have canceled every ceremony it was holding, counted from the +clock-abort counter rather than from the active gauge, because permits stay +counted until their owners close them and a falling gauge is the owners +noticing rather than the gate acting; and it must refuse work *offered to it +while it is blind*, which the step originates and then requires a refusal to +be recorded against. A node nobody asked produces exactly the same unchanged +permit counter as one that refused. A node that was idle when its clock +failed exercises only the refusal half and records the step blocked rather +than passing. + +Quiescence requires a security-v2 ceremony to be in flight when the stop is +issued, stops the node under the reviewed manifest's grace rather than a +restated number, and watches the whole drain. It offers new work once the +node reports `quiescing` and decides issuance from the permit counter rather +than from a peak of the active gauge, which a permit taken and closed between +two samples never raises. It requires the in-flight count to have been *seen* +at zero, because a node that stopped answering while still holding permits is +indistinguishable in its last reading from one that finished them, and it +blocks rather than passes on a counter it could not read. + +The refusal it records has to belong to the work it offered. The offer retains +the ceremonies the driver put on the chain, and one of *those* per-ceremony +refusal counters must be the one that moved. A per-ceremony delta on its own +only says the node refused something: a rehearsal chain carries other traffic, +and any unrelated ceremony refused for its own reasons moves the total and one +per-ceremony counter together, which is precisely the reading this step looks +for. An offer that named no ceremony it originated blocks, because nothing can +then be tied back to it. + +The work driver reports what it originated rather than only whether it +succeeded: its stdout is a JSON object whose optional `transaction_hashes` +array carries the chain transactions it submitted — those enter the step being +recorded, so a reviewer can follow a step back to the transactions that caused +it — and whose optional `ceremony_results` array carries `{ceremony, +canonical_start_block, work_id, outcome, transaction_hash}` objects naming the +terminal result of each ceremony those transactions started. The results are there +because no fleet counter carries them: a permit says a node was allowed to +begin, and the positive control is about a ceremony finishing. An optional +`originated_ceremonies` array carries `{ceremony, canonical_start_block, +work_id, transaction_hash, holders}` objects naming what the driver put on the +chain whatever became of it, for the phases whose subject is work still in +flight — a drain, a forced deadline — which have no terminal outcome to read, +since by the time one exists the work it was about is over. `holders` is an +array of `{service, permit_id}` records, one for every local permit rather than +a set of node names. Every array is validated strictly, and a report that +cannot be read stops the step — a driver whose account is unreadable has left +the step unable to say what it drove, and recording that as "nothing happened" +would enter silence as evidence. + +Every successful result additionally carries a `contributors` array of +`{service, permit_id}` records naming each party whose share the settled +transcript incorporated, no party twice. The mixed prior/R1 controls read it +and nothing else can answer the question they ask: that the prior binary's +container was running says only that it was running. Unselected, partitioned, +and cryptographically excluded all leave it up beside a ceremony that settled +without it, which from outside is the same reading interoperation produces. The +pre-cutover steps therefore require, in *each* required family, one settled +transcript whose contributors include both the prior service and one of the +rehearsed R1 services. Each half of that is load-bearing. Per family, because a +prior binary in a tBTC signing says nothing about the beacon's separate path +into the gate. Both services, because a release that settles a ceremony among +its own kind is homogeneous whichever release it is, and a control that only +looked for the prior share would read a prior-only ceremony as interoperation. +Within one transcript, because a prior-only wallet action beside an R1-only +signing is two homogeneous ceremonies, and what the control claims is that the +two wire formats combined into a single threshold output — which only a +transcript naming both parties ever witnesses. A service that is neither the +prior binary nor a rehearsed R1 node satisfies neither side, so a stray +container cannot stand in for either release. A driver that cannot report who +contributed cannot support a mixed-fleet claim, so the field is mandatory on +success rather than optional. + +Each outcome is bound to the work it belongs to, and controls are decided on +the bound form rather than on the arrays beside it. One chain work item is +`@@`: the canonical start block pins the mode, while +`work_id` is the chain-native request, group, wallet/action, or DKG-seed +identity that distinguishes several events in one block. One local permit is +that work identity plus `~`. The permit ID is the local +membership/member index or stable wallet/action identity, so one node +controlling two members at the same anchor retains two permit records. A piece +of chain work is originated once and ends once, while each local permit is +also unique. A repeated identity, one transaction claimed by two pieces of +work, or one work item changing transactions stops the step rather than being +counted twice or rebound downstream. A later terminal phase must retain the +exact transaction recorded by the originating phase. + +Everything above is the report checked against itself, and a report that is +internally consistent and entirely invented passes all of it. So the chain is +asked. Every transaction a report names must have a receipt on the endpoint +supplied as `ETH_RPC_URL`, that receipt must say the transaction succeeded, and +each piece of work must be anchored at or after the block its own transaction +landed in — an anchor before it is a permit pinning its mode from a block at +which the work did not exist, which is precisely what invented anchors look +like. A reverted transaction is the same shape as a successful one from +outside, and an unmined one is the same shape as work in flight, so both stop +the step rather than being read as the work a control was decided on. The +endpoint is itself checked against the rehearsed chain id, asked of the +endpoint rather than restated from the dispatch input: one answering about +another chain confirms transactions that have nothing to do with this +rehearsal, in exactly the same shape. + +A result must name a `transaction_hash` the same report accounted for +originating; without that, the hashes and the outcomes are two independent +populations, and a stale or unrelated hash sitting beside an unrelated result +satisfies any control that reads them in parallel. A result that succeeded must +carry a `result` identity — the threshold output the ceremony left behind — +because "succeeded" is a word and a positive control that cannot name what was +produced has read a report rather than watched a ceremony settle. A result that +did not succeed must carry a `termination` of `retry_exhausted` or +`no_threshold`, because a bare "failed" is equally what a ceremony still +retrying looks like from outside, and a fails-closed control cannot be read off +work still in progress. + +Every outcome the driver reports is carried forward, not only the successes. +A phase that kept the successes alone cannot tell a clean run from one where a +required ceremony failed beside a passing one, and cannot see a ceremony +succeeding where the property under test is that it must not. + +The pre-cutover steps name the ceremonies they must see settle one by one — +`tbtc_dkg`, `tbtc_signing`, `tbtc_heartbeat`, `beacon_dkg` and +`beacon_signing`, with `tbtc_wallet_action` added to the step whose subject is +the longest practical action. The mandate is that mixed prior/R1 tBTC signing, +DKG and heartbeat and the beacon controls all succeed below C, and neither a +family nor a work-class requirement can state that. "A tBTC threshold ceremony +settled" is satisfied by a signing alone, which leaves the DKG and heartbeat +paths into the gate undriven even though each anchors differently and refuses +separately; a fixture covering both halves of the release and both work classes +can still be three mandated ceremonies short. The heartbeat is the clearest +case: it settles like a signing and carries the inactivity penalty path the +crossing has to keep quiet, so a step that drove none says nothing about the +path most in need of the evidence. Naming the ceremonies means the step reports +on the work the mandate describes rather than on whichever ceremony the driver +happened to pick. + +The homogeneous control is decided against both halves of its own name, over +the whole report, on bound records. "security-v2 controls" needs a ceremony the +driver watched complete on a transaction it originated and that left a +threshold output behind, not only permits the fleet issued — and it needs one +from each half of the release. tBTC and the beacon take their permits from the same gate +through different call paths, so a driver that only ever drove tBTC leaves the +beacon's path unexercised however many tBTC ceremonies settled; a control +covering half the release cannot support a claim made about all of it, and the +step blocks. Anything the driver reported as failed or timed out refutes the +control outright: a report is taken whole, so the half that passed cannot +record the control on its own. "with no legacy sightings" is read where a +sighting would appear — the announcer's cross-format recognition counter and +the legacy roster, summed and unioned across the whole R1 fleet — because the +legacy permit counter is about work this fleet took on, not about what it saw. +The straggler is quarantined before this step runs, so a recognition or a +roster entry during it means the fleet was not homogeneous. + +Both halves of that report are what a step reads to decide work was offered at +all. A driver call that exited nonzero, that was never supplied, or that named +no transaction leaves the fleet in exactly the state a fleet nobody asked is +in — unchanged permit counters, an untouched roster — so a step whose contract +is that the gate *refused* something requires a clean exit and at least one +named transaction before it treats its readings as a refusal. The clock-failure +and quiescence probes distinguish the two: a driver that failed while +attempting the offer is recorded as a broken instrument naming its exit status, +not as a gate nobody challenged. + +A clean offer is still not a refusal, though — an unchanged permit counter is +equally the shape of work that never reached the node at all. So the quiescence +step also requires the node's *own* account: the gate counts every refusal it +makes, and counts it per ceremony, so the quiescing node's refusal total must +move and a per-ceremony counter must move with it. That is what puts the +refusal on the node rather than on the prober's inference, and what names which +ceremony was refused; a total that moved with no ceremony behind it blocks, +because a refusal a release cannot attribute to a ceremony is not evidence +about that ceremony. + +The rollback drain reads the fleet's in-flight security-v2 permits at the +moment the stop is issued: a `compose stop` that returns zero over an idle +fleet evidences that stopping works, not that a node holding protocol work +drains rather than dropping it. It also reads *what kind* of work was in +flight, from the driver's `originated_ceremonies` — a permit total counts +ceremonies without distinguishing a threshold round from a Bitcoin wallet +action, and the two fail differently when a shutdown interrupts them: a +threshold ceremony loses a share and can be re-run, a wallet action can leave a +Bitcoin transaction the fleet has already signed for. A rollback authorized +over one class says nothing about the other, so both must be in flight at once. + +Then every permit is followed to an outcome, per node and per piece of work +rather than in aggregate. A fleet total of zero after the drain is equally +produced by permits that finished and by processes that exited holding them, +and the difference is exactly the state a rollback restores onto. Each node's +permits at the stop must therefore land somewhere a later reader can see: +completed, evidenced by that node being observed without them, or +force-canceled at the quiesce deadline — which the gate counts and which the +offline audit must have written a quarantine record for. A force-cancel with no +quarantine record behind it is in-flight state the rollback would restore onto +with nothing describing it, and an unreadable counter blocks rather than +subtracting like a zero, which is how a permit nobody could account for would +otherwise disappear from the sum. The records must also be *enough* of them: +one record does not describe three abandoned permits, so a count short of the +force-cancels leaves the difference unaccounted for and refutes the step. + +The permits and the work must be the same population before any of that means +anything. Each node's held count is compared against the pieces of work the +driver said it put on that node, and a node holding more permits than there was +work to hold them for — or fewer — blocks: an outcome for one piece of work +reconciles no particular permit when the two accounts are of different sizes, +which is how one reported result came to stand in for however many permits +happened to be outstanding. + +The quarantine records are read the same way. DKG records retain the seed hash +as their chain-work ID and the member index as their local permit ID, and are +matched one-to-one rather than projected down to ceremony and block. Thus two +memberships on one node at one anchor remain two records, and a record for a +permit this drain never put on that node refutes the step instead of padding +the count. Records are filtered by the instant the stop was issued because a +quarantine namespace accumulates: records an earlier interruption wrote are +still there, and a bare count lets state from a run nobody is reconciling stand +in for permits this drain abandoned. The driver's own work classes are +translated into the gate's ceremony vocabulary for that comparison — every +non-heartbeat wallet action is gated as a signing ceremony, and the beacon's +signing class is its relay signing — because otherwise work that has a record +would appear to have none. + +Neither is a permit simply "completed" because the gauge holding it fell. +Being gone is what a ceremony that finished and a process that exited holding +one both look like from outside. So the driver is asked, once the drain is +over and the outcomes exist to be read, what became of the work this gate +originated — the `rollback-terminal` phase — and every piece of that work must +have reached a terminal outcome of its own or appear in a fresh quarantine +record. Permits that were not force-canceled reconcile against the outcome of +the work they were issued for rather than against the gauge; a gate that never +asked, a terminal report from a driver that exited nonzero, and an outcome for +work this drain never originated all block rather than passing. + +The single-release quiescence gate is decided the same way, for the same +reason. It retains the work the node was holding when the stop was issued — +which pieces, that the node's permit count matches how many there were, and +that the node's own gate named those same permits — and asks the driver in a +`quiesce-terminal` phase what became of each once the drain is over. A held +permit whose work never reached an outcome is a permit the process took with +it, and that is indistinguishable from completion in every counter the node +publishes. Work that ended by giving up inside the grace blocks rather than +passing: this gate audits no quarantined state, so a ceremony that exhausted +its retries evidences neither that it was allowed to finish nor that what it +left behind is accounted for. + +Its legacy half needs a permit the gate issued on the legacy side of C and is +still holding long after the crossing, which nothing originated at that point +in the run can be. So the driver is asked for it in a `quiesce-legacy-seed` +phase while the fleet is still below C, and the node's own gate is read there: +the permits it reports holding must include the ones the driver named. A driver +that merely claims a below-C anchor for work it originates after the crossing +is claiming exactly the thing the control exists to observe. The drain is then +required to happen while a permit of the other mode is live on the same node — +a gate draining a single population never has to keep the two modes apart, +which is the fence quiescence is for. + +That second population is held to the same standard as the first, because the +gate's promise covers every permit it was holding and the gate asks that both +live modes finish or enter audited quarantine. So the `quiesce-legacy-inflight` +phase must name the work it puts in flight beside the seeded permit, that work +must be anchored on the other side of C from the drained population — a second +population on the same side exercises no fence — the node's own gate must +report holding exactly those permits for that mode, and the +`quiesce-legacy-terminal` phase must report an outcome for every one of them +alongside the seeded population's. A gate list that is merely non-empty says +the fence was there and nothing about what the far side of it ended up doing. + +The straggler control binds its roster entry to the straggler's own operator. +The prior node publishes the address it signs as at `/diagnostics` +(`client_info.chain_address`), and that address is read off it while it is +still on the network and compared — insensitive to EIP-55 versus lowercase +spelling — against the operators the observing node's roster newly named. A +roster that moved without naming that operator is the release attributing a +legacy sighting to the wrong node, which is worse evidence than none: the name +is what a release decision would act on. + +Being named is not being refused, so that control also reads what the ceremony +it drove produced. The rehearsal fleet is sized so the driven post-C ceremony +needs the straggler to reach threshold: a result that settled means either the +straggler was not refused after all or the ceremony never depended on it, and +a control that did not need the node it is about has not exercised the failure +path it claims to. A ceremony with no terminal outcome at all blocks rather +than passing — retry exhaustion is what makes "produced no threshold output" +a statement about a ceremony that finished, and the roster deltas would +otherwise be read off one still in flight. That is why the driver must say +which of the two terminations it reached rather than only that the ceremony +failed, and why the record names the transaction and the termination it +decided on. + +The rollback gate's own barrier has two halves and neither substitutes for +the other. Every release candidate must be provably down, and the prior binary +must have been absent for the whole of it — so the drain runs while the prior +artifact is sampled repeatedly, from before the drain starts to after it +finishes, rather than probed once at the end. A single closing probe is +satisfied by a prior binary that participated for all of quiescence and stopped +a second before the probe, which is exactly the sequence the barrier forbids. + +"The prior binary" is daemon-wide too, for the same reason "every release +candidate" is, and each sample takes two readings of it. The node probe answers +whether the prior service *this project* staged is serving; the daemon +enumeration answers which containers anywhere were created from the prior +image, including ones this project neither named nor started. Neither reading +subsumes the other: a prior container left by another gate's project — or +started directly under no compose project at all — is invisible to a probe +keyed on this project's service name while watching the same rehearsal chain, +and a process still answering its client-info port after the daemon called its +container stopped is invisible to the enumeration. A prior container is counted +as participating when it is running and can still reach anything, quarantined +when running and reaching nothing, and its unreadable state blocks rather than +passes. Reachability is read from the container's network mode beside its +network map, because the map alone answers it backwards in both directions: a +container run with `container:`/`service:` mode owns no network entry precisely +because it holds another container's stack, and Docker lists `none` in the map +like any other network, so genuine isolation does not present as an empty map. +As on the candidate side, an enumeration that cannot see this project's own +staged prior container blocks: an empty active set read from a blind +instrument looks exactly like a barrier that holds. + +"Every release candidate" is daemon-wide, not this project's two services. A +rollback rehearsal runs after a cutover rehearsal, and the cutover fleet is a +fleet of the same candidate artifact watching the same rehearsal chain: a +distinct compose project is a distinct namespace, not a distinct chain, so a +candidate another gate left running would go on submitting against the same +contracts while the prior binary was released beside it. The barrier therefore +enumerates every container on the daemon that was created from the candidate +image or belongs to any `pr4109-*` project, and requires each one to be stopped +or reaching nothing, read the same way. Attachment comes from the daemon +rather than from the node's own HTTP surface, because a candidate whose +client-info listener died while its protocol stack kept running answers +nothing and is still on the network; conversely a service that does answer is +promoted back to active whatever the daemon believes, since a node serving +requests is participating. An enumeration that cannot see the containers the +asking stage itself created blocks rather than passing — an empty active set +read from a blind instrument looks exactly like a barrier that holds. The +cutover stage closes by stopping its own fleet and recording the same verdict, +and the workflow stops that project again before dispatching the rollback +gate, so a single-release stage that failed halfway cannot leave the next gate +measuring a barrier against a fleet nobody accounted for. + +The second half is the offline state audit reporting `rollback_barrier_ready` +for every snapshot: an all-down fleet says two releases cannot write the same +state at once, not that the state they left is safe to roll back onto. Those +snapshots are captured here, out of the containers the drain stopped, with +the storage path read off each container rather than restated — a supplied +snapshot is only a claim about what the fleet left behind, and an older +capture or another node's audits exactly as cleanly. The audit's result is +taken as a whole: its output path is cleared before it runs, so an earlier +manifest cannot stand in for one this run never produced, and a nonzero exit +refuses regardless of what the manifest claims, because the tool also exits +nonzero on an inconsistent namespace — a refusal its ready flag does not +carry. The prior binary is started only when both halves hold; every R1 node +down with an audit that authorized nothing records a blocked step and starts +nothing. + +`compose.rehearsal.yaml` is the fleet shell: one prior node (no gate — the +deliberate straggler) and two R1 nodes with the non-mainnet +`--protocolParticipation.cutoverBlock` override and persistent volumes. Each +service mounts `KEYSTORE_DIR//` read-only at `/mnt/keystore` and +starts with `--config /mnt/keystore/config.toml`; that per-node config must +carry the rehearsal contract addresses, the key file path under +`/mnt/keystore`, and storage directory `/mnt/storage` (the persistent +volume). The fleet spans two networks: the internal `rehearsal` network +carries inter-node protocol traffic and evidence probes with no host +publication of any port, while `chain-egress` exists only so nodes can reach +the external `ETH_WS_URL` endpoint. + +Every accepted rehearsal run must produce an evidence record conforming to +`rehearsal-evidence.schema.json`: exact source SHA, per-architecture image +digests, unique run identity, exact service/container/operator fleet, chain ID +and C, the sha256 of the reviewed `release-manifest.json` the fleet's +termination grace was taken from and the grace value itself, per-stage +canonical/callback blocks, permit modes, gauge snapshots, transaction hashes, +supporting-artifact identities, and non-secret state checksums. Screenshots +alone are insufficient. The emitter validates its one record with +`validate_evidence_record_set`; `./rehearse.sh validate-evidence` is the +archive entry point and checks every record under `EVIDENCE_DIR` against the +schema (ajv pinned to exact versions) before checking archive completeness. Both +require the recorded manifest hash *and* the recorded termination grace to +equal the checked-in manifest's — the hash alone would accept a record that +names the right manifest while claiming the fleet ran under some other +grace — so an admissible record links the termination-grace record to the +exact artifact and chain identity it carries. + +Admissible is not accepted. Everything above decides whether a record is one +this release may read at all — well formed, from the attested commit, +measured against the reviewed manifest — and says nothing about what it +says. A record is precisely where a rehearsal reports that a mandatory step +failed or an acceptance assertion does not hold, so a schema-valid, +correctly bound record can be exactly the evidence that a gate must be +refused. `validate-evidence` therefore asks the second question separately +against the exact gate contract. The single-release record must carry its 15 +named stages and nine named assertions; rollback must carry its 11 stages +and seven assertions. Every entry occurs exactly once and in execution order, +unknown entries are rejected, and every assertion must cite its designated +passing stage rather than any convenient passing step. Single-release must +also name the reviewed work-driver and tss-lib-review digests; rollback must +name the work-driver and reviewed rollback-evidence-generator digests. Any +recorded failed step or refused assertion, in any record in the directory, +exits `FAIL`; anything missing, duplicated, misbound, blocked, or produced +without its required reviewed instrument exits `BLOCKED`. Only the complete +exact contracts are evidence of satisfied gates. A passing record beside a +failing one accepts nothing. The in-process emitter applies this same contract +to the record it just wrote before a rehearsal is allowed to print success. + +Those comparisons only mean something while the checked-in manifest is +still the compiled bounds' own manifest, so the stage refuses to measure +anything until it holds the receipt proving that. `local-proofs` writes it +under `EVIDENCE_DIR/attestation` as its last step, after every proof has +passed: `release-manifest validate` accepts the reviewed file against the +compiled bounds, `release-manifest derive` records the bounds themselves in +`derived-manifest.json`, `reviewed-manifest.sha256` names the exact bytes +that were validated, `source-commit.txt` names the commit those bounds +were compiled from, and `release-ready.txt` records whether the manifest +names a release at all. `validate-evidence` requires all four files, the hash +to match the manifest as it stands now, and the derived bounds to match the +reviewed ones field by field — hash-matching alone would accept an +attestation and a manifest regenerated together around numbers no compiled +binary produces. The comparison covers the compiled cutover block and chain +id beside the termination grace; the source commit and image digests are +outputs of a build that happens after the manifest is reviewed, so `derive` +cannot speak for them and the manifest hash is what pins them. Only the +free-form notes and the generation stamp may differ, and keys are canonically +ordered so reformatting a reviewed manifest cannot read as drift. + +The readiness verdict is why a placeholder manifest cannot anchor accepted +evidence. `local-proofs` records what `release-manifest validate +--release-ready` says without failing on it — a development run legitimately +has a manifest naming no release, and what that stage proves is the code — +and `validate-evidence` BLOCKS on a receipt reading `no`. Records measured +against a manifest with a zero cutover block, no source commit, and no image +digests describe a rehearsal of the code rather than a release anything could +be accepted as, and the checked-in manifest is in exactly that state today: +until a reviewed release commit sets the mainnet cutover block and the built +commit and image digests are recorded, `validate-evidence` refuses, by +design. The attestation lives in a subdirectory +because the record glob and the workflow's record probe both look at the +top level of `EVIDENCE_DIR` only: writing the receipt never makes a +dispatch that produced no rehearsal record look like it produced one. +Running `validate-evidence` without a matching attestation is BLOCKED, not +accepted — regenerate it by re-running `local-proofs` at the same commit. +A receipt missing any one of its four files is a fragment, never a receipt. + +The two halves of that verdict are proved together. +`test-validate-evidence.sh` hand-authors the verdict file, which holds the +refusal but never runs the producer, so deleting the `--release-ready` +invocation or hard-coding the word it writes would leave that whole suite +green over a receipt nothing derived. +`test-attest-release-manifest.sh` closes the seam: it runs the real producer +and requires the recorded verdict to equal what an independent +`--release-ready` invocation answers, then drives the produced receipt — +unedited, and again with only the verdict flipped — through the consumer that +gates on it. Pinning a fixture to `yes` is not possible while the mainnet +cutover block is the zero placeholder, since `validate` requires the manifest +to carry the compiled block and readiness requires that block to be nonzero; +holding the producer to the binary is what covers the seam either way, and it +keeps holding once the reviewed block lands and the answer flips. It runs from +`local-proofs` rather than `shell-analysis` because asking the binary needs +the Go toolchain. + +The readiness and provenance success branches need a binary that compiles a +real cutover, so the suite always runs one — and asks the compiler which it +is rather than reading the source. `release-manifest derive` reports the +compiled block; while it is the zero placeholder a Go build overlay supplies +one for the duration of those cases, without touching the tree and with +`GOFLAGS` carrying it into the `go run` invocations the producer makes on its +own, and once the reviewed block is set the same cases run against the +shipping binary with no substitution at all. The block the cases actually +compile is then read back out of the binary, so a constant declared some other +way fails loudly instead of quietly turning every ready case into a second +copy of the "no" branch. + +Readiness says the reviewed manifest names a real cutover. It cannot say which +artifact runs it, and no edit to that document could make it: the commit +finally built and the image digests are outputs of a build over the manifest's +own bytes, so a manifest naming them would have to contain a hash of the tree +containing it — writing the value changes the value. A check demanding the two +agree is unsatisfiable by any real checkout rather than merely unsatisfied. + +That half of the identity therefore lives in a **detached release +provenance**: a document generated after the build, recording the +`manifest_sha256` it was taken over, the `source_commit` built, and one +digest-pinned image `reference` per platform. It is never committed to the +tree it describes — the producer refuses a provenance path tracked in this +repository, because committing it would put the commit back inside the commit +it names — and `release-manifest verify-provenance` checks the pair: the +manifest against the compiled bounds and against `--release-ready`, the +provenance against its shape, and the recorded hash against the reviewed +bytes. `local-proofs` takes the verified document into the receipt when +`PR4109_RELEASE_PROVENANCE` points at one, copying it rather than remembering +a path, so acceptance reads the run's own sealed account. + +The dispatched workflow is what has to be able to hand it in. A release +dispatch supplies the base64 of that document in the `release_provenance_b64` +input; the `local-proofs` job decodes it into `RUNNER_TEMP` — outside the +checkout, because the proof stage refuses to produce evidence from a tree that +diverges from the dispatched commit, untracked files included — mounts it into +the proof container at the container root, and passes +`PR4109_RELEASE_PROVENANCE`. That job then requires the archived receipt to +carry the document byte-for-byte, and `container-rehearsal` requires the +receipt it downloads to carry one whenever it records a release-ready +manifest, so a dispatch that cannot produce admissible evidence is refused +before it pulls an image rather than at the emitter of a rehearsal that has +already run every mandatory step. A development dispatch supplies none, runs +everything, and produces a receipt acceptance refuses only once the reviewed +cutover block is set. Every link in that path is read out of the workflow by +`shell-analysis` (`verify_release_provenance_wiring`), because losing one has +no local symptom: every proof still passes and the receipt still looks +complete. + +`validate-evidence` then closes both bindings the manifest could not close +alone. Once the receipt records a release-ready manifest, provenance is +mandatory: a ready receipt without it names a cutover and no artifact. Its +`manifest_sha256` must be the reviewed manifest's hash, so it cannot be +provenance for a release reviewed under other bounds; its `source_commit` must +be the commit the attestation was taken at — already proved a clean id and, on +a bound run, the dispatched one, and already required to equal every record's +`source_sha`. And the images are checked in two places, because they answer two +different questions. + +Per record, the one platform in `artifacts.r1_image_digests` must be one the +provenance publishes, at exactly the reference it publishes: an extra platform +is a fleet running something the release never shipped, a substituted digest is +a rehearsal of an artifact this release does not publish, and a record naming +no image at all would agree with every release by having nothing to disagree +about. A record naming two published platforms is refused too: one runner did +not execute both. The comparison is over references rather than bare digests, +so the same content pulled from another repository is refused as well. + +What a record does *not* name is not asked of it. A multi-architecture digest +names a manifest list whose children are the real runtime images, and a runner +resolves exactly one of them: every container the fleet started came from that +child, so a record honestly names one platform. Recording the whole list would +put architectures nothing executed into a record — an amd64 rehearsal claiming +the arm64 artifact — so the resolver names only the child the daemon actually +ran, refusing rather than guessing when the index publishes none for that +platform, more than one, or nothing readable at all. + +Covering the published set is therefore a property of the archive, checked once +over every record as the exact product of the mandatory gates +`single_release`/`rollback` and the provenance's platforms. A release publishing +two architectures is evidenced by rehearsing both gates on each. A wholly absent +rollback gate no longer disappears from a map inferred from existing records, +and two records for one gate/platform are refused as competing accounts rather +than collapsed into a set. The workflow derives its native-runner matrix from +the sealed provenance, gives every runner a platform suffix and an isolated +chain input, uploads only its uniquely named top-level records for fan-in, and +runs `validate-evidence` once in `aggregate-rehearsal-evidence` after all +platform jobs. Full per-platform artifacts remain separate because their audit +trees legitimately contain colliding filenames. + +The dispatch boundary is executable under fixtures too. +`test-rehearsal-matrix.sh` extracts the Node program from the workflow step +that builds the runner matrix and drives that exact program over accepted +amd64/arm64 mappings plus malformed base64 and UTF-8, duplicate JSON members, +bad endpoint and integer forms, unsupported or repeated provenance platforms, +and missing or surplus platform inputs. The proof and shell-analysis stages run +this test alongside the evidence validator, so changing the workflow changes +the implementation under test rather than leaving a copied validator behind. + +A receipt belongs to one run at one commit, and three rules keep it that +way. `local-proofs` destroys the receipt it inherits — interrupted staging +directories included — *before* it proves anything, so a run failing at any +proof leaves nothing behind; without that, a reused evidence directory kept +whichever earlier run happened to succeed in it and the acceptance stage +read that as this run's receipt. The new receipt is built beside its +destination and published by a single rename, so a reader sees a complete +receipt or none, never a half-written one or parts from two runs. And the +receipt carries the commit the binding check *proved* rather than the raw +stamp — `build-image` mode verifies a tree that legitimately diverges from +`HEAD`, so the raw stamp would call the very tree it just accepted `-dirty` +— which `validate-evidence` then requires to equal both its own +`PR4109_EXPECTED_SOURCE_COMMIT` and every record's `source_sha`. A receipt +taken at one commit can otherwise admit records from another whenever the +manifest bytes did not change between them, since the hash and bounds +comparisons have nothing to see in that case. Anything but a clean 40-hex +commit — a `-dirty` stamp, the `unknown` of a run outside a checkout — is +refused outright, and `validate-evidence` verifies its own source binding +like any other proof stage, because the manifest, schema, and comparison +rules it judges by all come out of the tree it runs from. + +The validator proves itself before validating anything: +`test-validate-evidence.sh` drives the stage over fixture records — correct +binding, wrong hash, wrong grace, wrong source commit, missing binding +fields, malformed timestamp, empty record set — over incomplete and malformed +gate contracts — a one-stage passing subset, a duplicate stage replacing a +required one, an unknown assertion, a true assertion borrowing an unrelated +passing stage, missing required release inputs — the reviewed instruments and +the archived dependency review alike — and an emitted one-stage run — over correctly bound records whose *outcomes* deny the gate — a failed +step, a refused assertion with every step passing, a step that never executed, +a failure alongside an unexecuted step, and a failing record sitting beside a +passing one — over fixture attestations — +absent, incomplete, a leftover staging directory, taken over other manifest +bytes, contradicting the reviewed bounds, taken at another commit than the +run is bound to, taken on a divergent tree, and one differing only in +notes, stamp, and key order — and over a divergent tree the stage must +refuse to judge from, and the stage runs that self-test first on every +invocation. It also drives the fleet-identity capture the container stages +open with, over fleets whose nodes disagree with each other, whose revision +is not the commit the run is bound to — foreign, abbreviated, or absent — +and whose armed cutover block is not the rehearsed C; and it resolves every +helper those stages name in command position, because neither stage runs +anywhere but a real rehearsal and a call site left pointing at a renamed +function otherwise surfaces there. + +The step verdicts those stages reach are proved the same way. The clock, +quiescence, and straggler decisions are functions over their observation +slots that touch no fleet, so the self-test drives them against constructed +readings: an unchallenged permit counter, work that never reached the gate, a +partial cancellation behind a drained and behind an unreadable active count, +a permit issued and closed between two samples, permits never seen at zero, a +mismatch nothing recognized as cross-format, a cross-format sighting that +entered no roster, and unreadable refusal, issuance, and forced-abort +counters. A ladder this layered is exactly the kind that goes on passing on a +proxy for the property until something can exercise it directly. The daemon +is a seam in the same way: the prior-container staging and the storage +capture run against a fixture daemon, over a container that came up running, +a create that produced nothing, a container built from other bytes, a live +node, a missing and a doubled volume, a failed copy, and an inherited capture +— and the audit against a tool that refused while a ready manifest sat at its +output path, and one that wrote nothing at all. + +The chain identity is an observation too. The client publishes +`protocol_participation.ethereum_chain_id` — the chain id its own endpoint +returned when the chain handle was built, checked there against the configured +network — and the identity capture requires every R1 node to report the chain +the record is written against. A cutover block is a count on one chain, so a +fleet pointed at another chain crossed a different schedule and every block, +crossing, and reconciliation in the record would be attributed to a chain the +fleet was never on. The supplied `CHAIN_ID` is now what the observation is +checked against rather than what the record asserts, and a fleet that will not +name its chain cannot be evidenced at all. The receipt lifecycle is proved +through `stage_local_proofs` +itself rather than through the invalidation function alone: a reused +evidence directory is given a valid inherited receipt, the stage's proof +seam is failed the way any proof failure fails it, and the case requires +that the receipt was already gone when the proofs started, that none +survives the failure, and that the acceptance stage is blocked afterwards. +Moving the invalidation anywhere later in the stage, or dropping it, fails +those cases. Its cases run against throwaway git checkouts it creates, not +against the working tree, so every verdict is the same mid-edit on a +workstation and on a bound CI dispatch. The +`cutover-rehearsal` workflow (manually dispatched, in +`.github/workflows/cutover-rehearsal.yml`) runs the local proofs, the +static analyzers, and the contracts build/test on every dispatch — and the +container preflight when the image digests and chain inputs are supplied. +Each stage's log is archived in a per-SHA artifact whether the stage +passes or fails, and the per-SHA name is backed by an in-stage proof, not +just labeling: the workflow hands every proof stage the dispatched SHA via +`PR4109_EXPECTED_SOURCE_COMMIT`, and for the build-image stage it mounts +the checkout's `.git` and `scripts/` read-only into the container +(`.dockerignore` keeps both out of the build context) and sets +`PR4109_SOURCE_BINDING_MODE=build-image`. Under that mode `rehearse.sh` +accepts exactly the image's documented construction and nothing else. +First it restores the commit's own `.gitignore` files — only where absent, +byte-exact from the commit under verification, so restoration can mask +nothing while a tampered ignore file keeps its modified status — because +the image drops every root dotfile and would otherwise report its own +gitignored build outputs (the `keep-client` binary, the `tmp/contracts` +artifact trees) as untracked noise. Then every remaining status line must +be explained: a deletion only for a path `.dockerignore` keeps out of the +context (honoring the `.clusterfuzzlite` negations — those files must be +present; no context-excluded path holds Go code the proof stages compile), +and the families the image regenerates from published artifacts +(`**/gen/**/*.go` and `**/gen/_address/*`, minus the negated `gen/pb/*.go`, +`gen/gen.go`, and `gen/cmd/cmd.go`, which the final `COPY` overwrites with +committed bytes — the committed protobuf code the tests compile can never +differ) are never accepted as found: each one is restored byte-exact from +the dispatched commit — `git show` against the read-only-mounted `.git`, +whose `HEAD` was already proven equal to the dispatched SHA — before any +test compiles it, with the pre-restore image hash recorded for forensics. +Untracked files and any other status are always fatal, a path that cannot +be restored is fatal, and the whole tree is re-checked after restoration: +anything left beyond the context-excluded absences fails the stage. The +resolved contract artifact tarballs under `tmp/contracts` — name, exact +version, sha256 — are still recorded as the image build's input identity +(the workflow pins the `ENVIRONMENT` build-arg from its +`artifact_environment` input instead of riding the Makefile's implicit +default), but they are forensic context only: whatever npm tag or version +the image was built from, the bytes the proof stages compile are the +dispatched commit's bytes by construction. The verifier is itself under +test: `test-source-binding.sh` drives it through checkout- and +image-shaped throwaway repositories — clean image, expected absences +alone, arbitrary bytes in every regenerated family proven replaced on disk +by the committed bytes, an unrestorable path, injected or deleted source, +tampered committed generated code, missing metadata, SHA mismatch — and +runs both as an early workflow step on the runner and inside +`local-proofs`, so its verdicts land in the archived evidence. +`./rehearse.sh verify-source-binding` runs the binding check alone and +records it under `EVIDENCE_DIR`. + +Which absences that verifier may explain away is decided by a +classification of the build context written out in `rehearse.sh`, and a +hand-written mirror drifts silently whenever the thing it mirrors changes. +Both `local-proofs` and `shell-analysis` therefore hold it to the commit's +own `.dockerignore`, compiled the way the build daemon reads it — comments +and blanks dropped, negations split off, patterns path-cleaned, `*` stopping +at a separator, `**` spanning whole segments, last match winning, a path +excluded when it or any ancestor matches — and compared against the script's +verdict for every tracked path. The rules are read from the commit rather +than from disk, because inside the build image the file is one of the paths +its own `.*` rule kept out. A path the script calls context-excluded while +`.dockerignore` keeps it in the context is the dangerous direction and +always fails: build-image mode would otherwise explain that file's absence +as the image's construction and accept a tree missing it. The opposite +direction is safe but still drift, and is tolerated only for the families +the image regenerates by design — the ones the verifier restores byte-exact +rather than explains away. A pattern construct the compilation does not +model, an absent `.dockerignore`, and one carrying no rule at all each fail +closed. `test-source-binding.sh` proves all of it against throwaway trees +carrying the checked-in rules and deliberate drifts of them, and refuses to +build a drift case out of a filter that removes no line, so a case cannot +pass because the rule it targets was renamed. + +Which ignore file the build reads is itself decided elsewhere: the builder +selects `.dockerignore` when the commit carries one and the root +`.dockerignore` only otherwise, and which Dockerfile that is comes out of the +rehearsal workflow's build step, not out of this scaffold. So the identity is +read from that step rather than restated beside the classification — the +single `docker/build-push-action` step's `context` and `file` inputs, taken +from the commit under test. A constant restating them would go stale the +moment the build step moved, silently and in the direction where the mirror +keeps checking itself against rules the build has stopped applying. Every +step shape the resolution cannot read the way the workflow parser does is +refused by name rather than guessed at: a context that is not the repository +root (the classification is written over repository-relative paths), an unset +context (the action's default is the Git context, not this checkout), a +Dockerfile named by a workflow expression or resolving outside the context or +absent from the commit, inputs written as a flow mapping, and more or fewer +than one build step. + +The rehearsal workflow writes its evidence into the workspace root rather +than the script's own default, and every proof stage refuses to run on a +tree that diverges from the dispatched commit — untracked files included — +so `/rehearsal-evidence/` is an ignore rule the repository's root +`.gitignore` carries alongside the script-local one. Without it a stage's +own log would count as divergence and fail the stage that wrote it. + +Everything above runs only when somebody dispatches it, which is the wrong +gate for the checkers that decide what may become release evidence. The +`cutover-scaffold-lint` workflow +(`.github/workflows/cutover-scaffold-lint.yml`) is what runs without being +asked: on every push and pull request touching the scaffold it runs +`./rehearse.sh shell-analysis`, which puts shell syntax, ShellCheck, +actionlint, the build-context mirror check, and all three boundary self-tests +over every change to `rehearse.sh`, to a self-test, and to the workflows +themselves. Whether failing it also *blocks a merge* is a setting outside this +repository, and one whose standing is recorded — not assumed — under "An +enforcing ruleset behind the scaffold gate" in **Hard external dependencies**. +Its path filters cover the build inputs the trust model is derived +from as well as the scaffold's own files — `.dockerignore`, both ignore files +the build could select, the root and nested `.gitignore` rules, `Dockerfile`, +and the root and per-package `Makefile`s — because each of them decides what +the verifier accepts just as directly as its own code does, and a change to +any of them can widen what an image tree is allowed to be missing without +touching a line under `scripts/`. It builds no image and runs no Go suite, so +it is cheap enough to require. + +Those filters decide when this gate runs at all, so neither they nor the list +they are measured against is maintained by hand: a build moved onto another +Dockerfile takes its ignore file with it, and a filter list left behind would +leave every later change to that file ungated while the mirror check went on +passing, on a file nobody was told had changed. `shell-analysis` therefore +enumerates the required inputs out of the commit under test — the three +workflows, the resolved Dockerfile, the ignore file that Dockerfile selects, +the root `.dockerignore`, every file under this directory, and every committed +`.gitignore` and `Makefile` — and requires each `push` and `pull_request` +trigger to run on all of them, or to carry no filter at all, which runs on +everything and covers everything. Adding a `gen/Makefile` or a nested +`.gitignore` therefore extends what the gate must cover without anyone +remembering to say so. + +The list is read the way the workflow parser reads it, in order with the last +matching entry deciding, so an entry listed and then negated further down is +not coverage; a pattern construct this scaffold has no reading for is refused +rather than guessed at. A `paths-ignore` list, an empty filter list, and a +workflow reachable only by dispatch each fail closed. + +Trigger shape is held to the same standard, because a restriction there is +invisible to a check that reads only paths and leaves the same hole. A `push` +trigger fires only after a branch has already moved, so a `pull_request` +trigger is required outright and refused if it carries `branches` or +`branches-ignore` — every restriction of it exempts some merge — or if its +`types` list drops one of `opened`, `synchronize`, `reopened`: without +`synchronize` the gate runs when a pull request opens and never again on what +is pushed into it afterwards. The `push` trigger's own `branches: [main]` is +accepted, since the `pull_request` trigger beside it is what holds the merge. + +All of that says when the gate runs and none of it says that reaching it runs +anything, so the invocation is placed too: a workflow firing on every change +to every input while its job no longer calls `rehearse.sh shell-analysis` is +the same ungated state spelled differently, and it satisfies every rule above. +`shell-analysis` requires exactly one such invocation — two would leave it +unable to say which placement the rest of the reading belongs to — and refuses +an `if:` on either that step or the job around it, along with a +`continue-on-error` on either that is not spelled `false`. A condition is +refused rather than evaluated: nothing here can tell which runs it would hold +for, and a gate whose reachability rests on a condition nothing reads is not +one this scaffold has proved reachable. A condition on some *other* step is +untouched — the evidence upload runs under `if: always()` precisely so a +failing analyzer's log survives. + +Matching text is not a run, so the invocation is read rather than found. It is +looked for only in a step's `run:` body, because that is the only key that +runs anything: the same text in a step name, an `env:` value or an action's +inputs labels a step that can do nothing at all. Inside that body the shell is +read as the shell takes it — lines joined across a trailing backslash, so an +invocation continued over two of them is one command — and the command has to +be the analysis itself and the *last* thing the body does, since a step reports +its last command's exit status. Around it, the shapes that would leave the text +running while the result went nowhere are refused by name: a pipeline, a `&&` +or `||` chain, a `;` list, a background `&`, a redirection, a command +substitution, a subshell, a compound-statement keyword, and the builtins that +decide what the shell does with the lines after them — `set -n` reads a body +without executing a line of it. Anything after the invocation is refused for +the same reason: a trailing `echo` is what a failing analysis would then be +reported as. + +The two ways a body runs under something other than the shell it was written +for are held the same way. A step-level `shell:` is accepted only spelled +`bash`, the runner's own default: `shell: cat {0}` leaves every line exactly +as it was and executes none of them. `defaults:` — on the job or on the +workflow — sets that same thing further out and is refused outright rather +than followed. Workflow expressions inside the body are read too, because the +runner substitutes their values before the shell parses the line: only the +runner's own contexts (`github.workspace`, `runner.temp` and their siblings) +are accepted, and an expression carrying pull-request text is refused, since +its value is what would decide the command. + +Reading the command is still not reading the run. The same invocation, spelled +character for character as the checked-in one, reaches something else entirely +when the environment or the tree around it changes, and none of that touches a +line the paragraphs above read. So four more shapes are refused: + +- **`env:`**, on the step, the job or the workflow. The runner writes those + names into the step's shell before it parses anything, and a `BASH_ENV` + there names a file that shell sources first — where a function can be + defined under the entrypoint's own name. The accepted command word then + resolves to that function and returns whatever it says. An assignment + written onto the invocation itself (`BASH_ENV=… ./rehearse.sh + shell-analysis`) is the same interception with no key to hold it, so the + assignments ahead of the command are read rather than skipped: only + `EVIDENCE_DIR`, the one environment name this entrypoint documents itself as + reading, is accepted there. +- **`working-directory:`**. The invocation is a relative path; resolved from + another directory it names another file, and what was proved to run is an + analysis somewhere else. +- **`container:`** on the job. The image decides both what `bash` is and what + stands at the entrypoint's path, and nothing here reads images. +- **a preceding `run:` step in the same job.** It needs no key on the analysis + step at all: it can write another file over the entrypoint in the checkout, + or append `BASH_ENV` to `$GITHUB_ENV` for every step after it. + +What none of that proves is that a run happened, or that a run reporting +success ran this file. Everything `shell-analysis` reads is the head commit's — +the workflow, the steps around the invocation, the entrypoint itself — and the +check that would notice the invocation being deleted lives *behind* that +invocation: a commit that removes the step also removes the run that would have +objected. + +**A branch-protection rule requiring the `scaffold-lint` check does not close +this**, and this scaffold does not claim it does. That rule requires a +conclusion under a job name, and the job producing it is defined by the same +head commit under test: a commit keeping the name while its job runs something +else reports success and merges. The four refusals above narrow that to shapes +this parser reads; they are a narrowing and not a closure, and shapes outside +them remain. A preceding `uses:` step runs code from another repository and +reaches `$GITHUB_ENV` and the checkout just as directly, and it is accepted +here only because refusing it would refuse the checkout the analysis needs to +read anything at all. An ordinary command ahead of the invocation in the +analysis step's own body is accepted for a narrower reason: the words ahead of +the invocation are read for the shell they open and nothing else, and a `cp` +over the entrypoint opens none — after which the accepted final command runs +whatever the copy left at that path. Both are pinned as accepted cases in +`test-source-binding.sh`, because a boundary a reader has to infer from the +absence of a case is one the next rewording moves. + +The control that does close it has to be defined where the pull request cannot +edit it, which is why it cannot be **this workflow under a rule**: a +`workflows` ruleset entry names one source repository and one path, and an +entry naming this file names something every pull request here can rewrite. +What closes the boundary is a **separate workflow, sourced from a repository no +pull request into `keep-core` can write to, pinned by commit SHA, and carrying +its own copy of the analysis** rather than calling back into the commit under +test for it. This file stays advisory however that is configured. The +requirement is tracked as an outstanding external dependency, with what was and +was not checkable from here, under "An enforcing ruleset behind the scaffold +gate" in **Hard external dependencies**. +`shell-analysis`'s own log says exactly that rather than claiming otherwise: it +reports what the commit under test says, and names this file for the rest. + +The same reasoning covers the other claim this scaffold makes about work it +did not do itself. `solidity-proofs` says its evidence is +`contracts-ecdsa.yml`'s `contracts-build-and-test` job's evidence, and that +holds only while the stage and the dispatch that provisions it run the Node +release that job pins — a release picked precisely because another one broke +hardhat's compile artifacts. So it is read out of that job rather than +restated beside the claim: `shell-analysis` resolves it from the named job +(not from the workflow around it, whose other jobs pin other releases) and +requires the rehearsal workflow's own `solidity-proofs` setup-node to match, +while the stage itself blocks on any other interpreter. A pin loose enough +for the runner to choose, one decided by a workflow expression, a job that +sets up Node twice or not at all, and a renamed or absent job each fail +closed. That is why `contracts-ecdsa.yml` is one of the lint's path filters: +a bump there touches no line under `scripts/`. + +On a hosted runner the per-node keystore comes from the +`REHEARSAL_KEYSTORE_BUNDLE_B64` repository secret: a base64-encoded tar.gz +whose top level holds one platform key (`amd64`, `arm64`, or `arm64-v8`) per +published image and one `/` directory per rehearsal node beneath it, +each with its platform chain's `config.toml` and rehearsal-only key material. +Generate it from a prepared root with `tar -cz -C "$KEYSTORE_ROOT" . | base64`. +The bundle MUST contain throwaway rehearsal keys only — never production +operator keys — and the dispatch reports `BLOCKED` when the secret or the +current platform's subtree is not provisioned. The companion +`REHEARSAL_KEEP_ETHEREUM_PASSWORD` secret carries the key files' password. + +The rest of what a container rehearsal needs is chain-side, which is to say +outside this repository, and arrives the same way. The dispatch inputs name +the artifacts and the chain: the prior, R1, and probe digests (all three +immutable — every evidence reading is a scrape through the probe, so a +mutable probe tag would leave the reading instrument outside the record's +provenance), the Bitcoin network, and the prior version/revision the rollback +state audit binds its verdict to. + +All platform chain coordinates travel in the single +`rehearsal_chain_inputs_b64` dispatch input. Decode it as one JSON object whose +keys match detached provenance one-to-one and whose values contain exactly +`eth_ws_url`, `eth_rpc_url`, `cutover_block`, and `chain_id`; `cutover_block` +is a positive JSON integer and `chain_id` is a positive decimal string. For +example, prepare (with real isolated endpoints): + +```json +{ + "amd64": { + "eth_ws_url": "wss://chain-amd64.invalid/ws", + "eth_rpc_url": "https://chain-amd64.invalid/rpc", + "cutover_block": 124000, + "chain_id": "1337" + } +} +``` + +Then encode the document as one base64 line. The matrix builder refuses +missing, extra, malformed, or duplicate platform entries — and duplicate +fields inside an entry — before it schedules a native runner. This one +structured input removes the workflow dispatch input ceiling: adding a +reviewed platform requires a runner mapping and one document member, not four +new top-level inputs. Every platform still needs its own chain beginning below +its own `C`; the JSON-RPC endpoint is where every transaction a driver reports +is confirmed and preflight requires it to answer with the recorded chain id. + +`arm64` and `arm64/v8` are distinct coverage slots when detached provenance +publishes both. They therefore owe two single-release and two rollback records, +two isolated chains, and separate keystore/trust subtrees (`arm64` and +`arm64-v8`) even though both jobs use `ubuntu-24.04-arm`. The workflow sets +`DOCKER_DEFAULT_PLATFORM` to the exact provenance platform so those jobs +cannot both resolve and rehearse the same child image. A release that intends +one arm64 artifact should publish one spelling, not duplicate it under both. + +The +`REHEARSAL_CHAIN_INPUTS_BUNDLE_B64` secret carries two executables, not data: +a base64-encoded tar.gz holding `work-driver` — called with the phase name, +because the fleet only reacts to chain events and without something +originating deposits, DKG requests, and relay requests there is no ceremony to +observe — and `rollback-evidence-generator`, called once per drained node with +that node's identity audit manifest and an output directory. The evidence is +generated rather than shipped because each record must name the aggregate +checksum of the snapshot it speaks for, and that snapshot does not exist until +this run has drained the fleet; a bundle unpacked before the fleet started +could not know a checksum computed later. Both members are checked as they are +unpacked, so a bundle missing one blocks before the fleet starts rather than +halfway through a rehearsal. + +The authenticated Ethereum half of rollback has a separate trust path. +`REHEARSAL_AUDIT_TRUST_B64` is base64 JSON keyed by the same platform keys as +the keystore. Each value must contain `wallet_registry_address`, +`random_beacon_address`, `finalized_ethereum_block_number`, +`finalized_ethereum_block_hash`, and the lowercase-hex +`chain_evidence_public_key`. The workflow validates their canonical shapes and +exports them only for that platform's job. This document MUST be provisioned +independently of `rollback-evidence-generator`: letting the program supply the +contract addresses, finalized anchor, or public key that authenticate its own +record would turn signature and canonical-chain verification into +self-attestation. + +An executable bit is not provenance, though, and that secret is mutable. Both +programs produce readings that become release evidence — the driver's account +of what it originated and what became of it is the terminal half of every +control that watches work settle — so a stale, replaced, or simply wrong +program manufactures an internally consistent passing account while every +check in this repository stays green. Preflight therefore hashes each supplied +program and compares it against `chain-inputs.sha256`, a reviewed control +checked in beside this file, before any node is started; a mismatch, or a +program the control does not name, stops the rehearsal. The digests are +recorded into the evidence document under `chain_inputs`, and the acceptance +stage refuses a record naming a digest the control does not pin, one carrying +chain transactions while naming no driver at all, or an otherwise complete +gate omitting a release input its contract uses. Single-release requires the +driver and the archived dependency review; rollback requires the driver and +the generator. That control currently pins the all-zero placeholder for all +three, which matches no file: no driver or generator has been written and +reviewed and no dependency review has been archived, so every dispatch that +supplies one blocks until a reviewed digest is recorded in a reviewed commit. +An unpinned control that admitted anything would be worse than an absent one, +because it would read as having been exercised. + +Everything provisioned lands outside the checkout, under the runner's +temporary directory. The container stages verify their own source binding +before they emit or judge a record, and that check counts untracked files as +divergence — so a keystore or an input bundle unpacked into the workspace +would fail the very stage it was provisioned to enable. The evidence +directory is the one exception, and only because the commit's own +`.gitignore` covers it. + +Storage snapshots are not among the supplied inputs. The rollback stage +captures each drained node's state itself, straight out of the container it +just stopped, into `STORAGE_SNAPSHOT_DIR`; a supplied snapshot is only a +claim about what the fleet left behind, and an older capture or another +node's audits exactly as cleanly as the real thing. Those captures hold live +protocol state — key shares included — so they stay on the runner and are +never archived. What a reviewer reads is what each capture produces under +`state-audit/` in the evidence directory: the identity manifest, the records +the generator wrote for that snapshot, and the authorizing manifest over them. + +The container job is bound to the same commit as every other proof stage, +and the receipt that binds it — the local-proofs stage's attestation of the +reviewed manifest against the compiled bounds — is downloaded from that +job's artifact before any rehearsal runs, because a rehearsal that reaches +its emitter without one blocks there, in the one place a dispatch cannot +diagnose from the log it archives. The rollback rehearsal runs on whatever +verdict the cutover rehearsal reached: a refused cutover is exactly when the +rollback gate's evidence matters most. Only a failed preflight stops it, +since that means the inputs never validated and the record would be about +nothing. Both fleets are torn down and both records are archived whatever +happened. + +## Node-local roster evidence window + +The single-release rehearsal opens the node-local roster evidence window on +every authoritative R1 service with `SIGUSR1`. It requires every process to +author its activation line and two clock-healthy empty roster snapshots with +advancing blocks, separated by the production five-minute cadence. The +timestamped relevant lines and a per-service summary are written beneath the +rehearsal evidence directory before `SIGUSR2` is delivered; every process must +then author its close line. An unsignaled service, failed delivery, unreadable +log, one-off snapshot, unavailable/stalled clock, nonempty-only capture, or +missing close fails the mandatory rehearsal step. The capture helper refuses +to overwrite an existing archive. + +Before the window opens, the rehearsal creates unique run and capture +identities and binds the capture to the record's exact source revision, +executed image/platform, protocol epoch, chain ID, C, and authoritative +service/container/operator fleet. The archive directory name is derived from +the capture identity. The final record repeats the run/fleet context and the +step records the capture/archive identities plus the SHA-256 of `result.json`. +Archive-set acceptance rejects one run, capture, archive, or summary claimed by +more than one record, so a capture from another run or native platform cannot +be lent wholesale. + +`validate-evidence` does not trust those strings by shape: it rejects path +traversal and symlinks, recomputes the summary digest, and verifies the hashed +`window-open.json` checkpoint that was written before `SIGUSR2`. It requires +`complete=true` with an empty failure list and the exact authoritative R1 +service set, verifies every archived service log against +`relevant_log_sha256`, and independently rechecks one activation, two +clock-healthy empty snapshots 270–360 seconds apart with advancing blocks, and +one close in that order. Supported UTC timestamps must prove +`opened <= archived-before-close <= closed <= record-generated`; activation +and both accepted snapshots precede the archive checkpoint, while close +follows it. The archive must therefore be preserved beside the top-level +record; deleting it, copying it under another identity, editing summary, +checkpoint, or log bytes, or borrowing both reference and digest from another +capture makes acceptance fail closed. + +For a host process the equivalent manual controls are `kill -USR1 ` and +`kill -USR2 `; for a container use the runtime's named-signal operation. +While the window is open, each node emits the deterministic roster snapshot +INFO line every five minutes even when the roster is empty. Outside the +window, the same cadence emits only while the post-cutover legacy-peer roster +is nonempty. + +These signals control logging only. They cannot select a protocol mode, issue +a permit, classify a peer, or authorize a commit. A termination signal holds +the window open before quiescence begins, and `SIGUSR2` cannot close that held +rollback window while the process drains. + +## Release manifest: service-manager termination grace + +A terminating node drains instead of dying: the first SIGTERM quiesces the +participation gate, already-started ceremonies run to natural completion, and +only the in-process backstop — `(maximum legacy completion bound + reviewed +margin) × upper block interval + RPC/processing allowance`, armed by +`quiesceBackstopDeadline` in `cmd/start.go` — forces the remainder through the +audited forced-cancellation path. All of that is useless if the external +service manager SIGKILLs the process first: the Kubernetes default grace is +30 s and systemd's is typically 90 s, both hours short of the drain a node may +legitimately need. The release manifest exists to close that gap fail-closed +rather than by hand-tuned deployment values. + +`release-manifest.json` records every input of the external grace — the tBTC +and beacon completion bounds with the beacon chain configuration they came +from, the reviewed quiesce margin, the upper block interval, the +RPC/processing allowance, and the resulting in-process backstop — plus the +forced-cancellation allowance between the backstop firing and SIGKILL, +itself a compiled constant: after the forced cancellation the lifecycle +controller keeps the run context alive until every canceled permit owner +finishes its quarantine/audit cleanup and releases its permit, waiting at +most exactly that constant. Validation checks the recorded allowance against +the compiled value like every other number — never re-deriving around the +manifest's own field — so a manifest whose allowance, grace, and scaffold +values were all recomputed coherently around a different allowance is still +rejected, and a `cmd` test additionally pins the runtime wait to the +checked-in manifest's recorded allowance. The service manager counts its grace from SIGTERM delivery, but the +backstop timer arms only after the controller has been scheduled and has +quiesced the gate, the allowance timer only after the gate has closed, and +the logging and teardown run after both — so the manifest adds the compiled +process-exit headroom (`processExitHeadroomSeconds`) on top of the two timed +waits, and the external grace strictly outlasts the complete internal +shutdown sequence rather than merely equaling the sum of its timers. The +authoritative external grace is the checked sum +`in_process_backstop_seconds + forced_cancellation_allowance_seconds + +process_exit_headroom_seconds` (currently `19800 + 300 + 60 = 20160` +seconds); a lifecycle test walks a forced shutdown from the termination +instant to exit readiness and requires the controller's overhead to fit +inside that headroom. The client never reads the manifest at runtime; its +bounds are compiled in, and the manifest exists so the SIGKILL deadline is +derived from those same bounds. + +Beside the grace, the manifest names which chain and block the cutover is +for. The grace binds the document to the compiled bounds of the binary +validating it, which establishes the numbers are right for some build of this +source; it does not establish which chain and block the cutover it describes +is for. Every statement made about the release afterwards — smoke gate +evidence, the inventory's per-instance digest attestation, a rollback decision +taken against a block height — is stated against those identities, so +`release_identity` carries them: the mainnet `chain_id` and the +`cutover_block` C. + +Both are checked by every validate run. `chain_id` is re-derived from the same +network definition the client verifies the remote endpoint against at connect +time, and `cutover_block` must equal `participation.MainnetCutoverBlock` as +compiled into the validating binary — so the reviewed document can never +publish a height no node observes, and the release commit that sets C forces +the manifest to be regenerated and re-reviewed. + +What the release was *built into* is deliberately not in this document. The +commit finally built and the immutable image digests are outputs of a build +over these very bytes, so recording them here would require the file to +contain a hash of the tree containing it. They live in the detached release +provenance instead — generated after the build, never committed to the tree it +describes, bound back to the reviewed manifest by its hash, and checked by +`release-manifest verify-provenance`. Each image reference there must end with +exactly the digest recorded beside it: a tag is a name the registry may +repoint, so a tagged reference would run whatever it resolves to at pull time +rather than the artifact the evidence describes. A manifest still carrying +either relocated key is refused by the loader with that instruction, rather +than as an unknown field. + +Validity and release-readiness are therefore separate verdicts. +`release-manifest validate` asks whether the document contradicts the binary, +which it must not at any point in development. `release-manifest validate +--release-ready` additionally asks whether the reviewed cutover block is set, +and it is the check a release-acceptance decision is taken against — together +with `verify-provenance`, which answers the half about the artifact. The +checked-in manifest passes the first and deliberately fails the second; the +failure names what is outstanding, which today is the zero cutover block — the +blocker that lives in the client rather than in the document, clearable only +by a reviewed release commit and not by any edit to this file. A `cmd` test +holds that standing state in place from both directions: it fails if the +manifest stops validating, and it fails if release-readiness starts passing +while the compiled placeholder stands. + +Neither verdict is advisory. `local-proofs` records the readiness answer in +the attestation receipt and `validate-evidence` refuses every record measured +against a manifest the receipt reports as not release-ready. Past that point +the run is a release-acceptance run, so the same stage requires the receipt to +carry detached provenance and holds every record to it — the commit built and +the exact image set — which is what keeps the acceptance path from issuing a +verdict over a document that identifies no artifact. + +The chain is enforced at three layers, each fail-closed: + +- `keep-client release-manifest derive` prints the manifest derived from the + binary's compiled bounds, and `keep-client release-manifest validate + --manifest ` re-derives every number and rejects the manifest on any + mismatch, reporting every violation at once. Because the subcommand ships in + the client binary, the exact-image rehearsal can validate the manifest with + the very artifact under test. `derive` fills in only the identity the binary + can speak for — a binary cannot honestly name the tree it was built from or + the registry addresses it was packaged into — so its output carries an empty + source commit and image list, exactly what `--release-ready` then refuses. +- `go test ./cmd/ -run TestReleaseManifest` pins the checked-in manifest to + the compiled bounds and the deployment scaffold to the manifest, so a + changed protocol constant, a stale manifest, and a drifted scaffold value + all fail the ordinary test suite; the strict loader additionally rejects + unknown fields, trailing content, and non-integer numbers. The + `local-proofs` stage runs these checks under the race detector. +- `release-manifest.schema.json` describes the document shape for external + tooling; schema validity alone is never authority — only the compiled-bound + validation is. + +`deploy/` carries the two scaffold fragments operators apply, each holding +exactly the manifest's grace: `keep-client-termination-grace.k8s-patch.yaml` +(`spec.template.spec.terminationGracePeriodSeconds`, applied with `kubectl +patch --patch-file`) and `keep-client-termination-grace.systemd-dropin.conf` +(`TimeoutStopSec` plus an explicit `KillSignal=SIGTERM`, installed as a +`.service.d/` drop-in). The rehearsal fleet carries the same contract: +both R1 services in `compose.rehearsal.yaml` set the manifest's grace as +their `stop_grace_period`, because Docker's 10-second default would SIGKILL +a draining node long before its backstop and no rollback rehearsal could +ever evidence natural completion — the prior node deliberately keeps the +default, having no drain semantics to protect. The grace is a ceiling, not a +wait — a node whose drain completes exits immediately. Changing any compiled +bound — the cleanup allowance included — requires regenerating the manifest +with `derive`, re-reviewing it, and updating every scaffold site; the `cmd` +tests refuse any shortcut through that sequence. + +## Hard external dependencies + +### An enforcing ruleset behind the scaffold gate + +`shell-analysis` proves what the commit under test *says* about +`cutover-scaffold-lint.yml`, and it cannot prove that a run of that workflow +happened or that a run reporting success ran this analyzer — the check lives +behind the invocation it checks, and the job producing the check is defined by +the same head commit. The reading detailed under "Cutover rehearsal scaffold" +narrows the shapes a green conclusion can hide (an `env:` at any level, an +assignment on the invocation, a `working-directory:`, a job `container:`, a +preceding `run:` step); it does not close the boundary. A preceding `uses:` +step is accepted while reaching `$GITHUB_ENV` and the checkout just as +directly, and so is an ordinary command ahead of the invocation inside the +analysis step's own body: a `cp` over the entrypoint is neither a shell +construct nor a builtin, and that is all the words ahead of the invocation are +read for. Both shapes are pinned as accepted in `test-source-binding.sh`, so a +later reading of the refusals cannot quietly grow into a claim of closure. + +Only a control defined where the pull request cannot edit it closes this. +GitHub's is a **ruleset rule requiring a workflow** — "Require workflows to +pass before merging", spelled `"type": "workflows"` in the API, settable at +the organisation or enterprise level. Its `parameters.workflows` is a list of +entries, each requiring a `repository_id` and a `path`, and each carrying two +optional pin fields: `ref`, documented as "the ref (branch or tag) of the +workflow file to use", and `sha`, "the commit SHA of the workflow file to +use". It replaced Actions Required Workflows, which stopped being configurable +on 2023-09-20 and became unreachable on 2023-10-18: +`/orgs/{org}/actions/required_workflows` is not the control to look for, and +whatever it answers settles nothing about the present +one. A branch-protection rule requiring the `scaffold-lint` check is not a +substitute either — it requires a conclusion under a job name that the commit +under test defines. + +Because an entry names one repository and one path, the rule cannot be pointed +at `cutover-scaffold-lint.yml` and be beyond this repository's reach at the +same time: requiring that path requires a file every pull request here can +rewrite, and the run it demands is the run the commit under test defines. **The +required workflow is a different file from the gate checked in here**, and the +gate checked in here stays advisory however the ruleset is configured. + +Three properties of that entry are what close the boundary, and any one +missing reopens it: + +- **The source repository is not this one.** `repository_id` must resolve to a + repository no pull request into `keep-core` can write to. The integer alone + settles nothing a reader can check, so the record names the repository it + resolves to. +- **The pin is immutable.** `ref` names a branch or a tag and both move — a + push to the branch, or a tag re-pointed at another commit, changes what runs + without changing the rule, so a `ref`-only entry pins a name and not the + bytes behind it. `sha` is what binds bytes, and is what the record carries. A + tag is admissible only with evidence that it cannot move: an `active` ruleset + on the source repository whose `target` is `tag`, whose conditions select + that tag rather than some other one, whose `bypass_actors` do not hand it + back to the maintainers the pin exists to bind, and whose rules include + `deletion`, `update` and `non_fast_forward`, recorded the way this one is — + the same exact-condition reading the ruleset behind the gate gets below, + because a tag ruleset aimed elsewhere holds nothing here either. +- **The analysis is carried by that pinned source.** A pinned workflow that + merely invokes the head commit's `scripts/release/pr4109/rehearse.sh` + re-inherits everything above: the analyzer it runs is still the one the + commit under test supplies. The SHA has to bind the checker implementation + too, and the record names the analyzer that SHA binds. + +Enforcement state belongs in the record rather than being assumed from the +ruleset's existence: `enforcement` is one of `disabled`, `active` and +`evaluate`, and `evaluate` is a dry run that reports without blocking a merge. +Only `active` gates anything. The carve-out that can leave an `active` ruleset +gating nothing for the merge that matters is recorded with it: +`bypass_actors` names actors holding permission to set the ruleset's rules +aside, each under a `bypass_mode` of `always`, `exempt` or `pull_request` that +governs when that permission is available and whether the actor has to reach +for it, and `pull_request` is not the narrow one it reads as. It confines that +actor's bypass to pull requests, and a merge into `main` goes through one, so +an actor listed that way can choose to bypass on exactly the event this gate +exists for. What the record carries is that capability, not a prediction it +gets exercised: a bypass declined on one merge is still available on the next, +so each actor's type, identity and mode belong in the record whether or not +one has ever been taken. `exempt` is the mode to read hardest — the rules are +not run for that actor and no bypass audit entry is written, so it is the +carve-out that leaves no trace on the merge it lets through. `pull_request` is +applicable only to branch rulesets, which is why it cannot appear on the `tag` +ruleset the pin above leans on: a bypass actor there holds `always` or +`exempt`, and both are unconditional. +The `workflows` rule's own `do_not_enforce_on_create` is recorded beside it +but is not a second such carve-out, and reading it as one waives a gate that +is in fact still standing: it is documented as allowing repositories and +branches to be *created* when a check would otherwise prohibit it, so it +waives the rule for the creation of a ref and not for an update to one that +exists. A merge into an existing `main` is an update, and this field leaves it +gated. What it does reach is a `main` deleted and created again, which is why +the record carries it rather than dropping it. + +`enforcement`, `target` and the entry's own three properties still say nothing +about *what* the ruleset is aimed at. `target` is one of `branch`, `tag`, +`push` and `repository`: it names a kind of ref, not an instance, and the +instances come from `conditions`. An organisation-level branch ruleset pairs +`ref_name` with exactly one repository selector — `repository_name`, +`repository_id` or `repository_property` — and those three do not read alike, +so a record resolving "the conditions" generically resolves nothing: + +- `ref_name` is an `include`/`exclude` pair of ref names or patterns, on all + three variants. `include` accepts `~ALL` and `~DEFAULT_BRANCH` alongside an + explicit `refs/heads/main` and one entry matching is enough; `exclude` fails + the condition when any entry matches, so it takes back what `include` + matched. +- `repository_name` is that same shape over repository names and patterns, + `~ALL` accepted, plus a `protected` flag that governs renaming the targets + and says nothing about what the ruleset is aimed at. +- `repository_id` is not that shape at all. It carries a `repository_ids` + array of integers and one of them matching is the entire test: there is no + repository-level `exclude` to take that back, and no `~ALL`. What it needs + is the resolution the integer withholds — the ID read back as + `threshold-network/keep-core`. +- `repository_property` is an `include`/`exclude` pair over `name` / + `property_values` / `source` objects rather than strings, and its `include` + is conjunctive where the others are disjunctive: *all* listed properties + must match, while `exclude` still fails on any. It therefore aims at + whatever set of repositories currently carries those values — a set this + one can enter or leave without the ruleset changing — so the record has to + name the properties and values, not merely which selector was used. + +So an `active`, unbypassed, externally SHA-pinned entry carrying its own +analyzer can hold every property above and gate nothing here, by being aimed +at another repository or at every branch except this one — and it reads, in a +record naming only the target, as though it closed the boundary. The record +therefore resolves the conditions instead of reproducing them: the exact +`conditions` object, the repository selector in use carrying the evidence that +resolves it to `threshold-network/keep-core` — the ID read back to a +repository, or the property names and values read back to this repository's — +and that `ref_name` matches `refs/heads/main`, by pattern, by `~ALL`, or by +`~DEFAULT_BRANCH` while `main` is the default branch, with `ref_name.exclude` +not removing it again. + +Standing, checked empirically on 2026-07-28: +`GET /repos/threshold-network/keep-core/rulesets?includes_parents=true` +returns an empty list. That is the informative probe — `includes_parents` +defaults to `true` and pulls in rulesets configured at higher levels that +apply to this repository, and GitHub filters only `bypass_actors` by the +caller's permission — so from outside the organisation this is the strongest +available signal, and it is a negative one: no ruleset, repository-level or +inherited, applies here. `GET /orgs/threshold-network/rulesets` returns 404 +without `admin:org`, and GitHub answers 404 rather than 403 for organisation +resources a caller cannot see, so on its own it distinguishes an absent +ruleset from an invisible one not at all. +`GET /repos/threshold-network/keep-core/branches/main/protection` returns 404, +which for that endpoint likewise means either no protection or no admin rights +and so settles nothing either way. + +Until an organisation admin configures such a ruleset, **the gate is +advisory**: a commit deleting `cutover-scaffold-lint.yml` deletes its own +enforcement silently, and a green `scaffold-lint` conclusion is evidence that +something under that name succeeded, not that this analyzer ran. Evidence that +rests on the scaffold's own checkers having judged a change should be read +with that in mind. Unblocking is a configuration change plus the record, not a +code change here, and the record has to name the ruleset's id and name, its +target, its exact `conditions` object together with the evidence resolving +that object's repository selector to this repository and its `ref_name` to +`refs/heads/main`, its `enforcement` — which must read `active` — its +`bypass_actors` with each actor's type, identity and `bypass_mode`, and the +rule's `do_not_enforce_on_create`, and, for the `workflows` entry, the +`repository_id` together with the repository it resolves to, the `path`, the +`sha` pinning it — or, for a tag, the tag together with the ruleset holding +that tag immutable — and the analyzer that pin binds. A record naming this +repository as the source, carrying a `ref` where the `sha` belongs, or leaving +the conditions unresolved, records something that does not close the boundary. + +### A non-atomic storage backend under the quarantine handoff + +Section 6.4's preservation is the last thing standing between a refused +ceremony and a key share that exists nowhere else, and the write it depends on +cannot be made crash-atomic from this repository. + +The backend is not here. `pkg/tbtc/quarantine.go` and +`pkg/beacon/registry/quarantine.go` write through +`github.com/keep-network/keep-common/pkg/persistence`, resolved by `go.mod`'s +replace directive to `github.com/threshold-network/keep-common +v1.7.1-tlabs.1`. That module's `Write` is `os.Create`, `Write`, `Sync` — no +temporary file renamed into place — so a crash during the write leaves a +truncated document, and `os.Create` truncates any record already at that name +before the new bytes are written, which means a rewrite of a landed record is +itself a window where the good copy is gone. + +Nor can it be worked around through the interface. `ProtectedHandle` is +`Save`, `ReadAll`, `Archive` and `Snapshot`: no rename and no delete, so there +is no primitive out of which record-level atomicity can be built. What the +release has instead is detection — a torn document fails the encrypted +handle's authentication, so the offline state audit reads it as an unreadable +record and blocks rather than any reader taking it for a preserved output. +Detection is not preservation: the audit reports the loss it cannot undo. + +A namespace that refuses every membership, metadata, and combined-handoff +write is the still narrower failure: no document survives for the offline +audit to discover, and the tBTC quarantined-signer gauge correctly remains +zero because the namespace retained no signer. The node now publishes that +attempt separately while the preservation retry and client-info endpoint are +both still live. Every tBTC preservation that remains incomplete after its +write-grace rounds increments +`performance_participation_tbtc_quarantine_preservation_failures_total`, and +the beacon path mirrors it with +`performance_participation_beacon_quarantine_preservation_failures_total`. +Both counters are pre-registered at zero and retain the incomplete episode as +history. The accompanying +`performance_participation_tbtc_quarantine_incomplete_outputs` and +`performance_participation_beacon_quarantine_incomplete_outputs` gauges are +also pre-registered at zero, rise for the whole live retry after grace is +exhausted, and clear only when the full output becomes durable. Both +exact-image gates sample all four signals while their candidates still answer. +All four are process-local and reset when a node restarts. The +`single_release` gate therefore opens its account before the cross-C restart, +takes and archives the retained numeric account under +`.pre_stop.*` together with +`.pre_stop.sample_readable`, and decides it before sending any stop +signal. A zero freshness bit states that the numeric fields are retained +history rather than a successful current scrape and blocks the control; a +fresh account with a nonzero live incomplete-output gauge fails it. In either +case the stop is not issued and the old process remains live to finish +preservation and to serve the later, independent controls. + +Only a readable pre-stop account whose two live incomplete-output gauges are +zero authorizes the split, watched stop. The stop derives its timeout from the +reviewed manifest — the same 20,160-second service-manager grace used by +rollback and the Compose deployment — so this recovery control cannot silently +reintroduce a shorter SIGKILL ceiling. While the endpoint answers, the gate +samples the old process throughout that stop; once the endpoint disappears it +stops scraping, waits for Compose, and retains the last node-authored values. +After the stop and before any start, it archives that watched account under +`.pre_restart.*` and inspects that exact stopped container's exit +status as `.pre_restart.container_exit_code`. Thus `pre_stop` means +the guard before a signal and `pre_restart` means the final account after the +guard authorized a stop; a record from one phase cannot stand in for the +other. Each watched value also carries +`.pre_restart..read_in_final_watched_sample`. The harness +fetches `/metrics` once per sampling attempt and parses all four values from +that one response, so a process stopping between independent HTTP requests +cannot manufacture an account whose fields never coexisted. A readable +exposition that genuinely omits one field leaves exactly that field's mask bit +clear. The harness replaces the four-field provenance mask only when the +response carried at least one signal; it does not combine freshness from +different attempts, and an all-unreadable fetch — whether `/diagnostics` still +answers or the endpoint has disappeared — retains the last useful mask. Both +incomplete-output fields must be marked `1` before restart, while a retained +historical counter marked `0` remains an advisory. This lets a normal endpoint +shutdown retain its last useful account without allowing one field's +successful read to make a different, carried zero look fresh. + +A missing watched-stop reading or exit status blocks the control; a +provenance marker, a carried incomplete-output field, a preservation gauge +that rises during the drain, or a nonzero exit status refuses the control. A +truncated stop is a refusal, not a pass, and a new process cannot retroactively +supply either fact with initialized zeros. On a clean account and zero exit +status, Compose starts the same container as the restart under test. If a +confirmed-stopped path instead refuses, the harness recovery-starts that same +container only to keep the remaining clock and quiescence controls +interpretable; the restart step remains refused and its old-process evidence +remains in the record. If that recovery cannot restore the client-info +endpoint, the rehearsal emits the partial record immediately and states that +the later controls were not evaluated instead of manufacturing failures +against a dead node. Pre-stop refusals never take that partial-record exit +because they issue no stop and leave the original process live. That early +partial-record exit occurs before the final fleet-stop stage; it leaves any +still-running rehearsal containers in place, and the operator must tear down +the `pr4109-single_release` Compose project manually after preserving the +record and any needed container evidence. Once any new process answers, the +gate re-seeds that node's live account, refreshes it through the chain-clock +cancellation and both quiescence controls, and records the fleet verdict +before its final fleet-stop stage. The rollback gate uses the same accumulator +and refreshes it while every candidate drains. The accumulator retains each +service's final useful four-field mask beside that service's values; a sample +from another service and an information-free post-exit scrape cannot replace +it. The fleet verdict archives those bits as +`..read_in_final_watched_sample` for every authoritative R1 +service in both gates. Both incomplete-output bits must be `1` for that +service. Historical counter bits may be `0` and remain advisory, but a carried +incomplete-output zero leaves the gate unrehearsed. Per-service masks can +differ and are classified independently; the process-global result of the +sampler's most recent call never supplies provenance for the rest of the +fleet. + +An unreadable signal, an account that does not cover every configured R1 +service exactly once, unreadable final-attempt provenance, a carried live +field, or a nonzero freshly read live gauge refuses either gate because the +node has not proved that the output became fully durable. A nonzero counter +paired with a freshly read zero live gauge is retained as a distinct, +non-fatal recovery finding: preservation exhausted its write-grace rounds and +later completed. +For the restart reading this finding remains in the archive after the new +process resets its counters. The finding is printed even when an unrelated +requirement refuses the same archive. It does not bypass the offline state +audit, which must still prove the durable namespace before the prior release +starts. All four zero means the node reported neither a grace-exhausting +episode nor an output currently incomplete; it does not turn a missing reading +into an empty quarantine. + +`performance_participation_quarantined_tbtc_signers` is deliberately not a +live preservation verdict. A nonzero value means the protected namespace +retained signer material that is not active — state a rollback must reconcile, +not a preservation failure — while zero can mean either no quarantined signer +or an output the namespace failed to retain at all. The offline state audit is +its authoritative consumer: it enumerates the protected namespace, binds each +record to chain settlement and permit evidence, and refuses rollback on any +unreadable or unreconciled output. The exact-image stages still snapshot the +gauge for diagnosis, but neither treats its value as a substitute for the +four preservation signals or for the offline audit. + +Two things would close it, and both are decisions rather than oversights. A +`keep-common` change making `Write` write-temp-sync-rename-sync-dir is the +real fix and lands in another repository and another release. Inside this one, +the available primitive is `Snapshot`, which writes under a generated name +`Save` never truncates: writing each output's key material through both would +leave a complete copy behind whatever a crash tore, at the cost of a second +copy of the material on disk and of teaching the audit which copy to prefer. +Neither is done here, and the release is entitled to know that what it has is +a detected loss and not a recoverable one. + +### Dual-mode tss-lib fork and independent-review gate + +R1's per-ceremony compatibility bundle covers all four wire- and +transcript-sensitive decisions (`pkg/protocol/compatibility`): the +announcement session-ID formats, the ECDH symmetric-key derivation, the G1 +hash-to-point mapping, and the tECDSA proof-transcript configuration. The +bundle travels from the participation permit into every tECDSA DKG and +signing party (`pkg/tecdsa/dkg`, `pkg/tecdsa/signing` take the bundle +explicitly — there is no default), and a repository check +(`pkg/protocol/compatibility/transcript_ownership_test.go`) fails the build +tests if a call site bypasses it. + +The build resolves exactly one `github.com/bnb-chain/tss-lib` replacement: +the immutable `threshold-network/tss-lib` revision +`d847ce0030193ccf5dbec0097571dcce5a2a5cf6` in `go.mod`. That revision adds an +explicit per-party `legacy`/`security-v2` transcript mode with no default and +freezes it when an ECDSA local party is constructed. Legacy reproduces the +historical untagged challenge formulas; security-v2 retains the +session-bound, domain-tagged transcript and requires a ceremony nonce. The +mode-independent validation and memory-safety guards remain active on both +paths. + +The dependency suite pins the historical DLN, range-proof, and Bob-proof +challenge formulas directly and completes homogeneous keygen and signing in +both modes. Its full test suite, focused race suite, and `go vet ./...` pass +for the pinned commit. In keep-core, the compatibility bundle is now the sole +owner of `SetProtocolMode` and session-nonce configuration, the temporary +legacy refusals are removed, and the formerly skipped homogeneous legacy DKG +and signing cases run complete real transcripts after a pre-cutover anchor. + +The homogeneous suites say nothing about a committee that disagrees, which is +the case the crossing actually creates. That case is held in keep-core rather +than in the dependency, because what has to stay true is a property of whatever +revision `go.mod` resolves: +`pkg/protocol/compatibility/mixed_transcript_test.go` drives real tECDSA +key-generation *and* signing ceremonies with each member's transcript +configured by the bundle its mode selects, and requires that no key share and +no signature is produced in either mixed direction — a member that has not +crossed joining a hardened group, and a member that has crossed refusing to +work with a group that has not. Keygen is covered alongside signing because it +is the half that leaves something behind: a refused signing ceremony costs an +attempt, while a tolerated mixed keygen would mint a wallet whose members +disagree about the transcript its shares were proved under, and that wallet +then signs for as long as it exists. + +A refusal does not end either ceremony under test. The driver keeps routing and +keeps watching the result channel until a refused ceremony has fallen silent, +so what is asserted is that nothing was output — not merely that a complaint +was raised, which a group that errored and then finished anyway would also +satisfy. Partial output fails too: the crossing is all-or-nothing, so some +members finishing what others refused is the split state the property exists to +prevent. Both homogeneous committees run through the same driver and are +required to output for every member, so the refusals cannot be an artifact of a +driver that never finishes anything. A replacement dependency revision that +tolerated a mixed committee fails these tests rather than surfacing during a +rehearsal. + +Independent cryptographic review is still a release gate, and it is a gate on +*evidence acceptance* rather than on execution. The commit is published on the +dependency repository's `codex/dual-mode-transcript` branch and proposed for +review in `threshold-network/tss-lib#9`, but this repository does not contain +an archived independent review record for it. + +Whether that review exists changes nothing about what a rehearsal can run or +observe — the immutable images and the work driver decide that — so the four +mixed-prior/R1 legacy-image steps of the `single-release` gate execute +whenever those artifacts are supplied, and record what they observed. What the +review decides is whether the resulting record is release-authoritative, and +that is settled once, at acceptance: + +- `PR4109_TSSLIB_REVIEW` names an archived review record. It is never + executed. It is bound twice — its bytes must hash to the `tsslib-review` + digest reviewed in `chain-inputs.sha256`, and the document must name the + exact dependency revision `go.mod` resolves — because either binding alone + admits a review of other code. +- The `single_release` acceptance contract requires `tsslib_review_sha256` in + the emitted record's `chain_inputs`. A rehearsal that ran every mandatory + step without a review record produces a complete, admissible record that + acceptance still refuses. + +`chain-inputs.sha256` pins the all-zero placeholder for `tsslib-review`, which +matches no file, so supplying a review record blocks until a real digest is +recorded there in a reviewed commit. The implementation, immutable dependency +pin, and in-repository acceptance evidence exist; independent review and +exact-image execution remain. + +## clientInfo.port 9601 compatibility smoke matrix + +### What is proven where + +| Layer | Proof | Runnable | +|---|---|---| +| Port resolution (flag/TOML precedence, both explicit-zero paths, custom port) | Go unit/config tests | ✅ locally, no Docker/chain | +| Port → listener decision (`0` disables, nonzero enables) | `pkg/clientinfo` unit tests | ✅ locally, no Docker/chain | +| Runtime image bakes the 9601 default | `clientinfo-port-smoke.sh image-default-check` | ✅ Docker only, no chain | +| Container listens on 9601 / custom, serves meaningful `/metrics` | `clientinfo-port-smoke.sh listener-matrix` | ⚙️ needs Docker **and** a chain endpoint + operator key | +| Testnet scrape from the real monitoring host, 3 consecutive intervals, current revision/epoch | — | 🔲 **manual / ops follow-up** | +| External untrusted-network probe: raw `9601` / `/diagnostics` unreachable unless an authenticated proxy is in front | — | 🔲 **manual / ops follow-up** | + +### Unit/config acceptance (fully runnable locally) + +``` +go test ./cmd/... ./config/... ./pkg/clientinfo/... \ + -run 'ClientInfoPort|TestReadConfig_ClientInfoPortZero|Initialize_' +``` + +Proves: no flag/TOML resolves to 9601 by default binding while an explicit +`--clientInfo.port 0` (CLI) and `[clientInfo] Port = 0` (TOML) both resolve to +zero; an explicit 9601 and a custom port enable; and `clientinfo.Initialize` +returns `(nil, false)` for port 0 and a registry for a nonzero port. + +### Container matrix (this harness) + +| Case | Configuration | Expected result | +|---|---|---| +| compatibility default | omit all client-info settings | TCP 9601 listens internally; `GET /metrics` succeeds | +| explicit compatibility | TOML `Port = 9601` | same | +| CLI compatibility | `--clientInfo.port 9601` | same | +| custom | `--clientInfo.port 9137` | only 9137 responds | +| CLI disabled | `--clientInfo.port 0` | no listener; node still starts | +| TOML disabled | `[clientInfo] Port = 0` | no listener; node still starts | + +`clientinfo.Initialize` runs only after `ethereum.Connect`, so the listener +cases require a node that can actually start against a chain (developer network +or a testnet RPC + operator key). Provide those and run: + +``` +IMAGE=keep-client@sha256: ETH_RPC=... KEY_FILE=... KEY_PASSWORD=... \ + ./clientinfo-port-smoke.sh listener-matrix +``` + +The harness's `require_digest` rejects a mutable tag: `IMAGE` (and `PROBE_IMAGE`) +MUST be pinned by an immutable `@sha256:` digest so a smoke run tests exactly the +bytes operators will deploy. + +The harness runs each case as a node container on a **private user-defined +bridge network** and probes the client-info port from a sibling `curl` container +— never via a published host port. The network is not made Docker `--internal` +because the node must still reach its Ethereum/Electrum backends to start; the +security property this harness proves is container-to-container reachability with +**no host publication of 9601**. Proving that raw `9601`/`/diagnostics` are +unreachable from a genuinely untrusted external network is a separate manual / +ops follow-up (see the matrix above). `compose.yaml` shows the same +private-network topology for the compatibility-default case. + +## Guardrails + +- 9601 is a **temporary** compatibility default; the follow-up R2 release flips + it back to `0` after the monitoring migration. Do not treat this harness as + permission to publish raw `9601`/`/diagnostics` publicly — always reach it over + a trusted path (firewall/VPN or an authenticated proxy). +- Do not add an unconditional `-p 9601:9601` to any operator-facing Docker + sample; the listener stays internal to the container unless explicitly + disabled with `--clientInfo.port 0`. diff --git a/scripts/release/pr4109/capture-cutover-evidence-window.sh b/scripts/release/pr4109/capture-cutover-evidence-window.sh new file mode 100755 index 0000000000..442d934456 --- /dev/null +++ b/scripts/release/pr4109/capture-cutover-evidence-window.sh @@ -0,0 +1,557 @@ +#!/usr/bin/env bash +# +# Open, capture, and close the node-local cutover roster evidence window for an +# exact set of running release-candidate containers. +# +# Each service is signaled with SIGUSR1. The capture is accepted only after +# every process logs that it opened the window and authors two clock-healthy +# empty roster snapshots with advancing blocks, separated by the production +# five-minute cadence. Relevant log lines are archived before SIGUSR2 is +# delivered; closing the window is then verified on every process. A delivery +# error, ignored signal, unreadable log, missing empty snapshot, stale clock, +# or missing cadence is a hard failure. + +set -euo pipefail + +readonly POLL_SECONDS=5 +readonly ACTIVATION_POLL_ATTEMPTS=6 +# A window opened just after a process's five-minute roster boundary may need +# almost ten minutes to produce two snapshots. Eleven minutes leaves one extra +# 30-second roster sweep plus scheduling margin. +readonly EVIDENCE_POLL_ATTEMPTS=132 +readonly MIN_CADENCE_SECONDS=270 +readonly MAX_CADENCE_SECONDS=360 + +usage() { + printf 'usage: %s [...]\n' \ + "$(basename "$0")" >&2 +} + +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +if (($# < 3)); then + usage + exit 2 +fi + +command -v docker >/dev/null 2>&1 || + fail "docker is required to signal and read the candidate containers" +command -v node >/dev/null 2>&1 || + fail "Node.js is required to validate timestamped roster evidence" + +ARCHIVE_DIR="$1" +shift +CAPTURE_CONTEXT="$1" +shift + +if [[ -e "${ARCHIVE_DIR}" ]]; then + fail "archive directory [${ARCHIVE_DIR}] already exists; refusing to overwrite evidence" +fi + +declare -a SERVICES=() +declare -a CONTAINERS=() +declare -a SIGNAL_DELIVERED=() +declare -a ACTIVATION_SEEN=() +declare -a CADENCE_SEEN=() +declare -a CLOSE_DELIVERED=() +declare -a CLOSE_SEEN=() +declare -a FAILURES=() +FAILURE_COUNT=0 +SEEN_SERVICES="|" +SEEN_CONTAINERS="|" + +for binding in "$@"; do + if [[ "${binding}" != *=* ]]; then + fail "container binding [${binding}] must have the form service=container" + fi + + service="${binding%%=*}" + container="${binding#*=}" + if [[ ! "${service}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then + fail "service [${service}] is not a safe evidence identifier" + fi + if [[ ! "${container}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then + fail "container [${container}] is not a safe Docker identifier" + fi + [[ "${SEEN_SERVICES}" != *"|${service}|"* ]] || + fail "service [${service}] appears more than once" + [[ "${SEEN_CONTAINERS}" != *"|${container}|"* ]] || + fail "container [${container}] is bound to more than one service" + SEEN_SERVICES="${SEEN_SERVICES}${service}|" + SEEN_CONTAINERS="${SEEN_CONTAINERS}${container}|" + + SERVICES+=("${service}") + CONTAINERS+=("${container}") + SIGNAL_DELIVERED+=(0) + ACTIVATION_SEEN+=(0) + CADENCE_SEEN+=(0) + CLOSE_DELIVERED+=(0) + CLOSE_SEEN+=(0) +done + +# The caller builds this context independently from the running fleet and +# later writes the same values into the rehearsal record. Validate it before +# delivering a signal: a malformed or differently bound capture must not open +# a logging window whose bytes could later be mistaken for release evidence. +CAPTURE_CONTEXT="$({ + node - "${CAPTURE_CONTEXT}" "${ARCHIVE_DIR##*/}" "$@" <<'NODE' +const [rawContext, archiveID, ...bindings] = process.argv.slice(2); +const fail = (message) => { + console.error(message); + process.exit(1); +}; +const isObject = (value) => + value !== null && typeof value === "object" && !Array.isArray(value); +const exactKeys = (value, expected) => { + if (!isObject(value)) return false; + const actual = Object.keys(value).sort(); + return actual.length === expected.length && + actual.every((key, index) => key === [...expected].sort()[index]); +}; + +let context; +try { + context = JSON.parse(rawContext); +} catch (_) { + fail("capture context is not valid JSON"); +} + +const contextKeys = [ + "schema_version", + "run_id", + "capture_id", + "archive_id", + "gate", + "source_sha", + "r1_image_digests", + "revision", + "protocol_epoch", + "chain_id", + "cutover_block", + "r1_fleet", +]; +if (!exactKeys(context, contextKeys) || context.schema_version !== 1) { + fail("capture context has no supported exact shape"); +} +if (!/^[0-9a-f]{32}$/.test(context.run_id || "")) { + fail("capture context has no valid run_id"); +} +if (!/^[0-9a-f]{32}$/.test(context.capture_id || "")) { + fail("capture context has no valid capture_id"); +} +const expectedArchiveID = + "cutover-roster-window-" + context.capture_id; +if ( + context.archive_id !== archiveID || + context.archive_id !== expectedArchiveID +) { + fail("capture context archive_id does not identify the archive directory"); +} +if (context.gate !== "single_release") { + fail("capture context gate is not single_release"); +} +if (!/^[0-9a-f]{40}$/.test(context.source_sha || "")) { + fail("capture context has no exact source revision"); +} +if (context.revision !== context.source_sha) { + fail("capture context runtime revision differs from its source revision"); +} +if (context.protocol_epoch !== "security_v2_cutover") { + fail("capture context has no security_v2_cutover epoch"); +} +if (!/^[0-9]+$/.test(context.chain_id || "")) { + fail("capture context has no numeric chain_id"); +} +if (!Number.isSafeInteger(context.cutover_block) || context.cutover_block < 1) { + fail("capture context has no positive safe cutover_block"); +} +if (!isObject(context.r1_image_digests)) { + fail("capture context has no executed R1 image map"); +} +const imagePlatforms = Object.keys(context.r1_image_digests); +if ( + imagePlatforms.length !== 1 || + !/^[A-Za-z0-9][A-Za-z0-9._/-]*$/.test(imagePlatforms[0]) || + !/@sha256:[0-9a-f]{64}$/.test( + String(context.r1_image_digests[imagePlatforms[0]] || "") + ) +) { + fail("capture context does not name exactly one immutable executed R1 image"); +} + +const bindingMap = new Map(); +for (const binding of bindings) { + const separator = binding.indexOf("="); + if (separator < 1) fail("capture context received a malformed fleet binding"); + const service = binding.slice(0, separator); + const containerID = binding.slice(separator + 1); + if (bindingMap.has(service)) { + fail("capture context received a duplicate fleet service"); + } + bindingMap.set(service, containerID); +} +if (!Array.isArray(context.r1_fleet) || context.r1_fleet.length < 1) { + fail("capture context has no authoritative R1 fleet"); +} +const fleetServices = new Set(); +for (const instance of context.r1_fleet) { + if (!exactKeys(instance, ["service", "container_id", "operator_address"])) { + fail("capture context carries a malformed R1 fleet entry"); + } + if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(instance.service || "")) { + fail("capture context carries an unsafe R1 service identity"); + } + if (!/^[0-9a-f]{64}$/.test(instance.container_id || "")) { + fail("capture context carries no immutable R1 container identity"); + } + if (!/^0x[0-9a-f]{40}$/.test(instance.operator_address || "")) { + fail("capture context carries no normalized R1 operator identity"); + } + if (fleetServices.has(instance.service)) { + fail("capture context carries a duplicate R1 service identity"); + } + fleetServices.add(instance.service); + if (bindingMap.get(instance.service) !== instance.container_id) { + fail("capture context R1 fleet differs from the signaled container set"); + } +} +if ( + fleetServices.size !== bindingMap.size || + Array.from(bindingMap.keys()).some((service) => !fleetServices.has(service)) +) { + fail("capture context R1 fleet differs from the signaled service set"); +} + +process.stdout.write(JSON.stringify(context)); +NODE +} 2>&1)" || fail "invalid capture context: ${CAPTURE_CONTEXT}" + +WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/cutover-evidence-window.XXXXXX")" || + fail "cannot create a temporary evidence directory" +trap 'rm -rf "${WORK_DIR}"' EXIT + +# Millisecond precision prevents an activation emitted just before this run in +# the same wall-clock second from being mistaken for this run's acknowledgement. +OPENED_AT="$(node -e 'process.stdout.write(new Date().toISOString())')" + +relevant_log_path() { + printf '%s/%s.log\n' "${WORK_DIR}" "$1" +} + +refresh_relevant_log() { + local index="$1" + local service="${SERVICES[${index}]}" + local container="${CONTAINERS[${index}]}" + local raw="${WORK_DIR}/${service}.raw" + local relevant + relevant="$(relevant_log_path "${service}")" + + if ! docker logs --timestamps --since "${OPENED_AT}" \ + "${container}" >"${raw}" 2>"${WORK_DIR}/${service}.docker-error"; then + return 1 + fi + + LC_ALL=C awk ' + index($0, "protocol cutover evidence window changed") || + index($0, "protocol cutover peer roster snapshot") + ' "${raw}" >"${relevant}" +} + +has_activation() { + LC_ALL=C grep -Fq \ + 'protocol cutover evidence window changed [active=true]' "$1" +} + +has_close() { + LC_ALL=C grep -Fq \ + 'protocol cutover evidence window changed [active=false]' "$1" +} + +has_periodic_empty_evidence() { + local candidate_count + candidate_count="$( + LC_ALL=C awk ' + index($0, "protocol cutover peer roster snapshot") && + index($0, "[clockAvailable=true]") && + index($0, "[legacyPeers=0]") { + count++ + } + END { + print count + 0 + } + ' "$1" + )" + ((candidate_count >= 2)) || return 1 + + node - "$1" "${MIN_CADENCE_SECONDS}" "${MAX_CADENCE_SECONDS}" <<'NODE' +const fs = require("fs"); +const [path, minimumText, maximumText] = process.argv.slice(2); +const minimum = Number(minimumText) * 1000; +const maximum = Number(maximumText) * 1000; +const lines = fs.readFileSync(path, "utf8").split(/\r?\n/); + +const snapshots = lines + .filter((line) => + line.includes("protocol cutover peer roster snapshot") && + line.includes("[clockAvailable=true]") && + line.includes("[legacyPeers=0]") + ) + .map((line) => { + const token = (line.match(/^(\S+)\s/) || [])[1] || ""; + const currentBlock = Number( + (line.match(/\[currentBlock=(\d+)\]/) || [])[1] + ); + // Docker uses RFC3339Nano. Date.parse implementations commonly accept + // milliseconds only, so truncate (never round) a longer fractional part. + const normalized = token.replace( + /(\.\d{3})\d+(Z|[+-]\d{2}:\d{2})$/, + "$1$2" + ); + return { timestamp: Date.parse(normalized), currentBlock }; + }) + .filter((snapshot) => + Number.isFinite(snapshot.timestamp) && + Number.isSafeInteger(snapshot.currentBlock) + ); + +for (let index = 1; index < snapshots.length; index++) { + const previous = snapshots[index - 1]; + const current = snapshots[index]; + const elapsed = current.timestamp - previous.timestamp; + if ( + elapsed >= minimum && + elapsed <= maximum && + current.currentBlock > previous.currentBlock + ) process.exit(0); +} +process.exit(1); +NODE +} + +all_marked() { + local mark + for mark in "$@"; do + [[ "${mark}" == "1" ]] || return 1 + done +} + +add_failure() { + FAILURES+=("$1") + FAILURE_COUNT=$((FAILURE_COUNT + 1)) +} + +for index in "${!SERVICES[@]}"; do + if docker kill --signal=SIGUSR1 "${CONTAINERS[${index}]}" >/dev/null; then + SIGNAL_DELIVERED[index]=1 + else + add_failure "SIGUSR1 delivery failed for ${SERVICES[${index}]}" + fi +done + +for ((attempt = 1; attempt <= ACTIVATION_POLL_ATTEMPTS; attempt++)); do + for index in "${!SERVICES[@]}"; do + if refresh_relevant_log "${index}" && + has_activation "$(relevant_log_path "${SERVICES[${index}]}")"; then + ACTIVATION_SEEN[index]=1 + fi + done + + all_marked "${ACTIVATION_SEEN[@]}" && break + ((attempt == ACTIVATION_POLL_ATTEMPTS)) || sleep "${POLL_SECONDS}" +done + +for index in "${!SERVICES[@]}"; do + if [[ "${ACTIVATION_SEEN[${index}]}" != "1" ]]; then + add_failure \ + "${SERVICES[${index}]} did not author an evidence-window activation" + fi +done + +if all_marked "${ACTIVATION_SEEN[@]}"; then + for ((attempt = 1; attempt <= EVIDENCE_POLL_ATTEMPTS; attempt++)); do + for index in "${!SERVICES[@]}"; do + if refresh_relevant_log "${index}" && + has_periodic_empty_evidence \ + "$(relevant_log_path "${SERVICES[${index}]}")"; then + CADENCE_SEEN[index]=1 + fi + done + + all_marked "${CADENCE_SEEN[@]}" && break + ((attempt == EVIDENCE_POLL_ATTEMPTS)) || sleep "${POLL_SECONDS}" + done +fi + +for index in "${!SERVICES[@]}"; do + if [[ "${CADENCE_SEEN[${index}]}" != "1" ]]; then + add_failure \ + "${SERVICES[${index}]} did not author two clock-healthy empty roster snapshots with advancing blocks at the five-minute cadence" + fi +done + +mkdir -p "$(dirname "${ARCHIVE_DIR}")" || + fail "cannot create the evidence archive parent" +mkdir "${ARCHIVE_DIR}" || + fail "cannot create evidence archive [${ARCHIVE_DIR}]" + +STATUS_FILE="${WORK_DIR}/status.tsv" +: >"${STATUS_FILE}" +for index in "${!SERVICES[@]}"; do + log="$(relevant_log_path "${SERVICES[${index}]}")" + if [[ ! -f "${log}" ]]; then + : >"${log}" + fi + cp "${log}" "${ARCHIVE_DIR}/${SERVICES[${index}]}.log" || + fail "cannot archive the log for ${SERVICES[${index}]}" + printf '%s\t%s\t%s\t%s\n' \ + "${SERVICES[${index}]}" \ + "${SIGNAL_DELIVERED[${index}]}" \ + "${ACTIVATION_SEEN[${index}]}" \ + "${CADENCE_SEEN[${index}]}" >>"${STATUS_FILE}" +done + +ARCHIVED_AT="$(node -e 'process.stdout.write(new Date().toISOString())')" +node - "${STATUS_FILE}" "${OPENED_AT}" "${ARCHIVED_AT}" \ + "${CAPTURE_CONTEXT}" <<'NODE' \ + >"${ARCHIVE_DIR}/window-open.json" +const fs = require("fs"); +const [statusPath, openedAt, archivedAt, captureContextJSON] = + process.argv.slice(2); +const services = fs.readFileSync(statusPath, "utf8").trim().split(/\r?\n/) + .filter(Boolean) + .map((line) => { + const [service, signalDelivered, activationSeen, cadenceSeen] = + line.split("\t"); + return { + service, + signal_delivered: signalDelivered === "1", + activation_seen: activationSeen === "1", + periodic_empty_snapshots_seen: cadenceSeen === "1", + }; + }); +process.stdout.write(JSON.stringify({ + schema_version: 2, + capture_context: JSON.parse(captureContextJSON), + opened_at: openedAt, + archived_at: archivedAt, + services, + complete: services.length > 0 && services.every((service) => + service.signal_delivered && + service.activation_seen && + service.periodic_empty_snapshots_seen + ), +}, null, 2) + "\n"); +NODE + +# The periodic evidence is now present in its final archive path. Only after +# that point may the operator signal close the window. +for index in "${!SERVICES[@]}"; do + if docker kill --signal=SIGUSR2 "${CONTAINERS[${index}]}" >/dev/null; then + CLOSE_DELIVERED[index]=1 + else + add_failure "SIGUSR2 delivery failed for ${SERVICES[${index}]}" + fi +done + +for ((attempt = 1; attempt <= ACTIVATION_POLL_ATTEMPTS; attempt++)); do + for index in "${!SERVICES[@]}"; do + if refresh_relevant_log "${index}" && + has_close "$(relevant_log_path "${SERVICES[${index}]}")"; then + CLOSE_SEEN[index]=1 + fi + done + + all_marked "${CLOSE_SEEN[@]}" && break + ((attempt == ACTIVATION_POLL_ATTEMPTS)) || sleep "${POLL_SECONDS}" +done + +for index in "${!SERVICES[@]}"; do + if [[ "${CLOSE_SEEN[${index}]}" != "1" ]]; then + add_failure "${SERVICES[${index}]} did not author an evidence-window close" + fi + + log="$(relevant_log_path "${SERVICES[${index}]}")" + if [[ -f "${log}" ]]; then + cp "${log}" "${ARCHIVE_DIR}/${SERVICES[${index}]}.log" || + fail "cannot finalize the archived log for ${SERVICES[${index}]}" + fi +done + +FAILURE_FILE="${WORK_DIR}/failures.txt" +: >"${FAILURE_FILE}" +if ((FAILURE_COUNT > 0)); then + printf '%s\n' "${FAILURES[@]}" >"${FAILURE_FILE}" +fi +: >"${STATUS_FILE}" +for index in "${!SERVICES[@]}"; do + printf '%s\t%s\t%s\t%s\t%s\t%s\n' \ + "${SERVICES[${index}]}" \ + "${SIGNAL_DELIVERED[${index}]}" \ + "${ACTIVATION_SEEN[${index}]}" \ + "${CADENCE_SEEN[${index}]}" \ + "${CLOSE_DELIVERED[${index}]}" \ + "${CLOSE_SEEN[${index}]}" >>"${STATUS_FILE}" +done + +CLOSED_AT="$(node -e 'process.stdout.write(new Date().toISOString())')" +node - "${STATUS_FILE}" "${FAILURE_FILE}" "${ARCHIVE_DIR}" \ + "${OPENED_AT}" "${ARCHIVED_AT}" "${CLOSED_AT}" \ + "${CAPTURE_CONTEXT}" <<'NODE' \ + >"${ARCHIVE_DIR}/result.json" +const crypto = require("crypto"); +const fs = require("fs"); +const path = require("path"); +const [ + statusPath, failurePath, archivePath, openedAt, archivedAt, closedAt, + captureContextJSON, +] = process.argv.slice(2); +const failures = fs.readFileSync(failurePath, "utf8").split(/\r?\n/) + .filter(Boolean); +const services = fs.readFileSync(statusPath, "utf8").trim().split(/\r?\n/) + .filter(Boolean) + .map((line) => { + const [ + service, signalDelivered, activationSeen, cadenceSeen, + closeDelivered, closeSeen, + ] = line.split("\t"); + const log = fs.readFileSync(path.join(archivePath, service + ".log")); + const text = log.toString("utf8"); + return { + service, + signal_delivered: signalDelivered === "1", + activation_seen: activationSeen === "1", + periodic_empty_snapshots_seen: cadenceSeen === "1", + close_delivered: closeDelivered === "1", + close_seen: closeSeen === "1", + empty_snapshot_lines: text.split(/\r?\n/).filter((line) => + line.includes("protocol cutover peer roster snapshot") && + line.includes("[legacyPeers=0]") + ).length, + relevant_log_sha256: crypto.createHash("sha256").update(log).digest("hex"), + }; + }); +const windowOpen = fs.readFileSync(path.join(archivePath, "window-open.json")); +process.stdout.write(JSON.stringify({ + schema_version: 2, + capture_context: JSON.parse(captureContextJSON), + opened_at: openedAt, + archived_before_close_at: archivedAt, + closed_at: closedAt, + window_open_sha256: + crypto.createHash("sha256").update(windowOpen).digest("hex"), + complete: failures.length === 0, + failures, + services, +}, null, 2) + "\n"); +NODE + +if ((FAILURE_COUNT > 0)); then + printf 'FAIL: cutover evidence window was not complete:\n' >&2 + printf ' - %s\n' "${FAILURES[@]}" >&2 + exit 1 +fi + +printf 'cutover roster evidence archived at %s\n' "${ARCHIVE_DIR}" diff --git a/scripts/release/pr4109/chain-inputs.sha256 b/scripts/release/pr4109/chain-inputs.sha256 new file mode 100644 index 0000000000..e550a41eff --- /dev/null +++ b/scripts/release/pr4109/chain-inputs.sha256 @@ -0,0 +1,40 @@ +# Reviewed SHA-256 digests of the external inputs this rehearsal depends on but +# does not contain. +# +# Two of them are programs it executes: the work driver that originates +# ceremonies on the rehearsal chain, and the generator that produces each +# node's external rollback evidence. Both arrive from a mutable secret bundle +# at dispatch time, and both produce readings that become release evidence — +# the driver's account of what it originated and what became of it is the +# entire terminal half of the drain, quiescence, straggler, and homogeneous +# controls. +# +# The third is not executed at all: the archived independent cryptographic +# review of the dual-mode dependency revision `go.mod` resolves. It gates +# acceptance of a single-release record rather than execution of any step, +# because whether that review exists changes nothing about what a rehearsal +# can run or observe — only whether the mixed prior/R1 legacy transcripts it +# exercised are release-authoritative. A supplied record must additionally +# name the exact revision `go.mod` resolves, so a review of some other +# revision cannot stand in for one of the code under test. +# +# An executable bit is not provenance, and neither is a document asserting its +# own approval. Without a reviewed digest here, a stale, replaced, or simply +# wrong input manufactures an internally consistent passing account while every +# check in this repository stays green, which is the one failure a dispatch +# cannot diagnose from the log it archives. +# +# Format: "<64 hex digits> ", one per line, over exactly the three +# names below. Comments and blank lines are ignored. +# +# STATUS: no driver or generator has been written and reviewed, and no +# independent cryptographic review of the pinned dependency revision has been +# archived. The digests below are the all-zero placeholder, which matches no +# file, so every rehearsal that supplies one of these inputs blocks until the +# reviewed digest is recorded here in a reviewed commit. That is deliberate: +# this file is the release control, and an unpinned control that admits +# anything is worse than an absent one because it reads as having been +# exercised. +0000000000000000000000000000000000000000000000000000000000000000 work-driver +0000000000000000000000000000000000000000000000000000000000000000 rollback-evidence-generator +0000000000000000000000000000000000000000000000000000000000000000 tsslib-review diff --git a/scripts/release/pr4109/clientinfo-port-smoke.sh b/scripts/release/pr4109/clientinfo-port-smoke.sh new file mode 100755 index 0000000000..0a0e0f91e2 --- /dev/null +++ b/scripts/release/pr4109/clientinfo-port-smoke.sh @@ -0,0 +1,336 @@ +#!/usr/bin/env bash +# +# clientinfo-port-smoke.sh — container smoke matrix for the temporary +# clientInfo.port 9601 compatibility default. +# +# This harness proves, against an immutable runtime image, that: +# - with no client-info setting the container listens on 9601 internally; +# - an explicit 9601 (TOML or CLI) also listens; +# - a custom port listens only on that port; +# - explicit 0 (TOML or CLI) starts no client-info listener while the node +# otherwise starts normally; +# and that every positive /metrics and /diagnostics response carries meaningful +# content (not just HTTP 200), including the stranded-peer observability signals +# added by this release. +# +# The unit/config half of the acceptance is proven by the Go tests and does +# NOT need this harness: +# go test ./cmd/... ./config/... ./pkg/clientinfo/... -run \ +# 'ClientInfoPort|TestReadConfig_ClientInfoPortZero' +# +# SCOPE NOTE: this build contains the block-height cutover gate, whose fixed +# gauges (performance_participation_gate_state, _cutover_block, +# _active_ceremonies, ...) are all registered at construction, so a positive +# /metrics response must carry them alongside the stranded-peer observability +# metrics. The one-value schedule exposes a single cutover height; there are +# no drain/stop metrics to assert. +# +# Two sub-steps CANNOT be exercised by this harness and are explicit manual / +# ops follow-up (do not fake them): +# - a real testnet run scraped from the actual monitoring host for three +# consecutive intervals with current revision/epoch; +# - an external untrusted-network probe proving raw 9601 / /diagnostics are +# unreachable unless an authenticated proxy is intentionally in front. +# +# Usage: +# # Docker-only, no chain: confirm the image bakes the 9601 compatibility +# # default into `keep-client start --help`. +# IMAGE=keep-client@sha256: ./clientinfo-port-smoke.sh image-default-check +# +# # Full listener matrix. Starts each of the six cases itself as a node +# # container on a private network and probes the internal endpoints from a +# # sibling container. A chain endpoint and an operator key are required +# # because a node only brings up the client-info listener after it connects +# # to Ethereum (cmd/start.go), so these are inherent inputs, not a scaffold. +# IMAGE=keep-client@sha256: \ +# ETH_RPC=wss://... \ +# BTC_ELECTRUM_URL=tcp://electrum:50001 \ +# KEY_FILE=/abs/path/to/keyfile.json \ +# KEY_PASSWORD=... \ +# ./clientinfo-port-smoke.sh listener-matrix +# +set -euo pipefail + +# Immutable-digest requirement: both the candidate and the probe image MUST be +# pinned by @sha256: digest, not a mutable tag, so a smoke run tests exactly the +# reviewed artifact and cannot be silently repointed between checks. Supply +# digest-pinned references via IMAGE / PROBE_IMAGE. The placeholders below are not +# valid digests and are rejected by require_digest until replaced; the live Docker +# run itself remains manual/ops follow-up (see the SCOPE NOTE above). +IMAGE="${IMAGE:-keep-client@sha256:REPLACE_WITH_CANDIDATE_IMAGE_DIGEST}" +PROBE_IMAGE="${PROBE_IMAGE:-curlimages/curl@sha256:REPLACE_WITH_CURL_IMAGE_DIGEST}" + +# Network mode: start the node in an explicit non-mainnet network so the harness +# never resolves the mainnet default (config.go). Override to --developer if the +# candidate image is built for developer mode. +NETWORK_MODE="${NETWORK_MODE:---testnet}" + +# Unique per-run suffix so a failed setup only ever force-removes THIS run's +# containers/network, never unrelated resources that happen to share a fixed name. +RUN_ID="${RUN_ID:-$$-${RANDOM}}" +NETWORK="cutover-port-smoke-net-${RUN_ID}" + +# The six case container names, uniquely suffixed per run. +CASES=(default toml9601 cli9601 custom cli0 toml0) +cname() { printf 'case-%s-%s' "$1" "${RUN_ID}"; } + +READY_TIMEOUT="${READY_TIMEOUT:-180}" +# The endpoint answering is the definitive readiness signal, so the positive +# probe retries with a bounded backoff instead of assuming the listener is up +# the instant a log line appears (which would race listener initialization). +PROBE_RETRIES="${PROBE_RETRIES:-20}" +PROBE_INTERVAL="${PROBE_INTERVAL:-3}" +# The negative (no-listener) probe re-checks over a short settling window so a +# listener that binds slightly after startup cannot false-pass a "disabled" case. +NEGATIVE_PROBE_ATTEMPTS="${NEGATIVE_PROBE_ATTEMPTS:-5}" +NEGATIVE_PROBE_INTERVAL="${NEGATIVE_PROBE_INTERVAL:-3}" +CUSTOM_PORT="${CUSTOM_PORT:-9137}" + +WORKDIR="" + +# require_digest fails unless ref is pinned by an immutable @sha256: digest. +require_digest() { + local ref="$1" what="$2" + case "${ref}" in + *@sha256:REPLACE_*|*REPLACE_*) + fail "${what} is a placeholder; set ${what} to an immutable @sha256: digest" ;; + *@sha256:[0-9a-f]*) + [[ "${#ref}" -ge 80 ]] || fail "${what} digest looks malformed: ${ref}" ;; + *) + fail "${what} must be pinned by an immutable @sha256: digest, not a mutable tag (${ref})" ;; + esac +} + +# Metric names every positive /metrics response must contain. The first six are +# backed by the current performance constants; the rest are the participation +# gate and stranded-peer / roster observability metrics added by this release +# (all registered at zero or their startup value, so they appear before any +# event). +REQUIRED_METRICS=( + "client_info" + "performance_signing_operations_total" + "performance_signing_success_total" + "performance_signing_failed_total" + "performance_signing_timeouts_total" + "performance_dkg_failed_total" + "performance_participation_gate_state" + "performance_participation_current_block" + "performance_participation_cutover_block" + "performance_participation_allowed" + "performance_participation_active_ceremonies" + "performance_announcer_session_id_mismatch_total" + "performance_announcer_cross_format_peer_total" + "performance_announcer_legacy_peers_current" + "performance_announcer_legacy_peer_additions_total" + "performance_announcer_legacy_peer_evictions_total" +) + +# Substrings every positive /diagnostics response must contain. +REQUIRED_DIAGNOSTICS=( + "client_info" + "cutover_legacy_peers" + "protocol_participation" +) + +log() { printf '[port-smoke] %s\n' "$*"; } +fail() { printf '[port-smoke][FAIL] %s\n' "$*" >&2; exit 1; } + +# image-default-check: Docker-only, no chain. Proves the runtime image bakes the +# 9601 compatibility default and the trusted-network help text. +image_default_check() { + require_digest "${IMAGE}" "IMAGE" + log "checking that ${IMAGE} bakes the 9601 compatibility default" + local help + help="$(docker run --rm --entrypoint keep-client "${IMAGE}" start --help)" + + grep -Eq -- '--clientInfo\.port int .* \(default 9601\)' <<<"${help}" \ + || fail "start --help does not show '(default 9601)' for --clientInfo.port" + grep -q -- 'Set to 0 to disable; expose only on a trusted network' <<<"${help}" \ + || fail "start --help is missing the trusted-network / zero-disable text" + + log "OK: image advertises the 9601 compatibility default with trusted-network guidance" +} + +# write_config — render a minimal, valid start config +# with the operator-supplied chain/key/electrum values and the given client-info +# section (which may be empty to omit the section entirely). +write_config() { + local file="$1" clientinfo="$2" + cat >"${file}" < [extra cli args...] — start a node +# container on the private network with the rendered config and key mounted. +start_node_case() { + local name="$1" config="$2" + shift 2 + # NETWORK_MODE forces an explicit non-mainnet network so the node never + # resolves mainnet defaults. It is a flag list, not one word — a caller can + # override it with several flags, so it is split deliberately. + # shellcheck disable=SC2086 + docker run -d --name "${name}" --network "${NETWORK}" \ + -e KEEP_ETHEREUM_PASSWORD="${KEY_PASSWORD}" \ + -v "${config}:/config/config.toml:ro" \ + -v "${KEY_FILE}:/keys/operator.json:ro" \ + "${IMAGE}" start ${NETWORK_MODE} --config /config/config.toml "$@" >/dev/null \ + || fail "case ${name}: container failed to start" +} + +# wait_ready — block until the node logs that it has initialized the +# client info registry (or reached the point past which no listener will appear), +# bounded by READY_TIMEOUT. +wait_ready() { + local container="$1" waited=0 + while (( waited < READY_TIMEOUT )); do + if ! docker ps --filter "name=${container}" --filter "status=running" \ + --format '{{.Names}}' | grep -q "${container}"; then + docker logs "${container}" 2>&1 | tail -40 >&2 + fail "case ${container}: node container exited before becoming ready" + fi + if docker logs "${container}" 2>&1 | grep -Eq \ + 'clientinfo|client info|initialized tbtc|Bootstrapping|started tbtc'; then + return 0 + fi + sleep 3 + waited=$(( waited + 3 )) + done + docker logs "${container}" 2>&1 | tail -40 >&2 + fail "case ${container}: node did not reach readiness within ${READY_TIMEOUT}s" +} + +# assert_listens — probe /metrics and /diagnostics from a +# sibling on the private network and require meaningful content on both. +assert_listens() { + local container="$1" port="$2" body diag metric substr attempt=0 + # Retry the /metrics probe with a bounded backoff: wait_ready only proves the + # process is up, so the listener may bind a moment later. The endpoint + # answering is the real readiness signal. + while :; do + if body="$(docker run --rm --network "${NETWORK}" "${PROBE_IMAGE}" \ + -fsS --max-time 10 "http://${container}:${port}/metrics")"; then + break + fi + attempt=$(( attempt + 1 )) + if (( attempt >= PROBE_RETRIES )); then + docker logs "${container}" 2>&1 | tail -40 >&2 + fail "case ${container}: expected a listener on ${port}, got none after ${attempt} attempts" + fi + sleep "${PROBE_INTERVAL}" + done + for metric in "${REQUIRED_METRICS[@]}"; do + grep -q "${metric}" <<<"${body}" \ + || fail "case ${container}: /metrics missing required metric ${metric}" + done + + diag="$(docker run --rm --network "${NETWORK}" "${PROBE_IMAGE}" \ + -fsS --max-time 10 "http://${container}:${port}/diagnostics")" \ + || fail "case ${container}: /diagnostics did not respond on ${port}" + for substr in "${REQUIRED_DIAGNOSTICS[@]}"; do + grep -q "${substr}" <<<"${diag}" \ + || fail "case ${container}: /diagnostics missing required content ${substr}" + done + log "OK: ${container} listens on ${port} with meaningful /metrics and /diagnostics content" +} + +# assert_no_listener — require the port to STAY closed across a +# short settling window while the node process itself keeps running. A single +# immediate probe would false-pass if the listener binds slightly after startup, +# so re-probe NEGATIVE_PROBE_ATTEMPTS times: if a listener EVER answers, fail. +assert_no_listener() { + local container="$1" port="$2" attempt + for (( attempt = 1; attempt <= NEGATIVE_PROBE_ATTEMPTS; attempt++ )); do + if docker run --rm --network "${NETWORK}" "${PROBE_IMAGE}" \ + -fsS --max-time 5 "http://${container}:${port}/metrics" >/dev/null 2>&1; then + fail "case ${container}: expected NO listener on ${port}, but one answered on attempt ${attempt}" + fi + # The node must stay up throughout — a disabled listener must not mean a dead + # node. + docker ps --filter "name=${container}" --filter "status=running" \ + --format '{{.Names}}' | grep -q "${container}" \ + || fail "case ${container}: node container is not running" + sleep "${NEGATIVE_PROBE_INTERVAL}" + done + log "OK: ${container} has no client-info listener across ${NEGATIVE_PROBE_ATTEMPTS} probes but the node is still running" +} + +cleanup() { + # Only this run's uniquely-named containers/network are ever removed. + local base + for base in "${CASES[@]}"; do + docker rm -f "$(cname "${base}")" >/dev/null 2>&1 || true + done + docker network rm "${NETWORK}" >/dev/null 2>&1 || true + [[ -n "${WORKDIR}" ]] && rm -rf "${WORKDIR}" +} + +listener_matrix() { + : "${ETH_RPC:?set ETH_RPC to a chain endpoint the node can start against}" + : "${BTC_ELECTRUM_URL:?set BTC_ELECTRUM_URL to a reachable Electrum endpoint}" + : "${KEY_FILE:?set KEY_FILE to an operator key file the node can start with}" + : "${KEY_PASSWORD:?set KEY_PASSWORD for the operator key file}" + + # Enforce immutable digests before doing anything destructive. + require_digest "${IMAGE}" "IMAGE" + require_digest "${PROBE_IMAGE}" "PROBE_IMAGE" + + WORKDIR="$(mktemp -d)" + docker network create "${NETWORK}" >/dev/null 2>&1 || true + trap cleanup EXIT + + # Render one config per case. The disabled cases and the explicit cases differ + # only in the [clientInfo] section / CLI flag; the compatibility-default case + # omits the section entirely. + write_config "${WORKDIR}/default.toml" "" + write_config "${WORKDIR}/toml9601.toml" $'[clientInfo]\nPort = 9601' + write_config "${WORKDIR}/cli.toml" "" + write_config "${WORKDIR}/custom.toml" "[clientInfo]"$'\n'"Port = ${CUSTOM_PORT}" + write_config "${WORKDIR}/toml0.toml" $'[clientInfo]\nPort = 0' + + log "starting the six client-info port cases (network mode: ${NETWORK_MODE})" + start_node_case "$(cname default)" "${WORKDIR}/default.toml" + start_node_case "$(cname toml9601)" "${WORKDIR}/toml9601.toml" + start_node_case "$(cname cli9601)" "${WORKDIR}/cli.toml" --clientInfo.port 9601 + start_node_case "$(cname custom)" "${WORKDIR}/custom.toml" + start_node_case "$(cname cli0)" "${WORKDIR}/cli.toml" --clientInfo.port 0 + start_node_case "$(cname toml0)" "${WORKDIR}/toml0.toml" + + local base + for base in "${CASES[@]}"; do + wait_ready "$(cname "${base}")" + done + + assert_listens "$(cname default)" 9601 + assert_listens "$(cname toml9601)" 9601 + assert_listens "$(cname cli9601)" 9601 + assert_listens "$(cname custom)" "${CUSTOM_PORT}" + # The custom-port case must NOT also answer on 9601. + assert_no_listener "$(cname custom)" 9601 + assert_no_listener "$(cname cli0)" 9601 + assert_no_listener "$(cname toml0)" 9601 + + log "OK: full client-info port listener matrix passed" +} + +case "${1:-}" in + image-default-check) image_default_check ;; + listener-matrix) listener_matrix ;; + *) + echo "usage: IMAGE=... $0 {image-default-check|listener-matrix}" >&2 + exit 2 + ;; +esac diff --git a/scripts/release/pr4109/compose.rehearsal.yaml b/scripts/release/pr4109/compose.rehearsal.yaml new file mode 100644 index 0000000000..386755e3d1 --- /dev/null +++ b/scripts/release/pr4109/compose.rehearsal.yaml @@ -0,0 +1,108 @@ +# Exact-image rehearsal fleet shell for the single-release cutover and +# homogeneous rollback rehearsals: an immutable prior-production node and two +# immutable R1 nodes sharing one rehearsal chain, each with a persistent +# keystore/work volume so restarts and rollback state audits are meaningful. +# It deliberately contains no chain service: the rehearsals run against a +# dedicated rehearsal chain with deployed beacon/tBTC contracts, supplied via +# ETH_WS_URL, because a throwaway in-compose chain without those contracts +# cannot produce release evidence. +# +# Required environment (validated by rehearse.sh preflight): +# PRIOR_IMAGE_DIGEST immutable prior-production runtime digest +# R1_IMAGE_DIGEST immutable R1 candidate runtime digest +# ETH_WS_URL rehearsal chain websocket endpoint +# CUTOVER_BLOCK rehearsed cutover block C (non-mainnet override) +# KEYSTORE_DIR per-node rehearsal inputs, one subdirectory per +# service; each holds that node's config.toml (with +# the rehearsal contract addresses, the key file +# path under /mnt/keystore, and storage directory +# /mnt/storage) plus the operator key file +# KEEP_ETHEREUM_PASSWORD operator key file password for the fleet +# +# Two networks separate the two reachability concerns. `rehearsal` is the +# internal inter-node protocol network — evidence probes attach here, and no +# node port is ever published to the host, including each node's client-info +# port. `chain-egress` exists only because the rehearsal chain endpoint lives +# outside this compose project; an internal-only topology would leave every +# node unable to reach ETH_WS_URL. +# +# The prior node receives no cutover configuration: the prior binary has no +# gate, which is exactly the straggler behavior the rehearsal must observe. +# It also keeps the default stop grace: without a lifecycle controller it +# exits on the first SIGTERM, so a long grace would only imply drain +# semantics the prior binary does not have. +# +# Both R1 nodes carry the release manifest's service-manager termination +# grace as their stop_grace_period: the rollback rehearsal stops R1 nodes +# mid-protocol, and Docker's 10-second default would SIGKILL a draining node +# long before its in-process backstop, so no rehearsal could ever evidence +# natural completion or the audited forced-cancellation path. The value is +# pinned to release-manifest.json by `go test ./cmd/ -run TestReleaseManifest`. + +services: + prior-node: + image: "${PRIOR_IMAGE_DIGEST}" + command: + - "start" + - "--config" + - "/mnt/keystore/config.toml" + - "--ethereum.url" + - "${ETH_WS_URL}" + environment: + KEEP_ETHEREUM_PASSWORD: "${KEEP_ETHEREUM_PASSWORD}" + volumes: + - "${KEYSTORE_DIR}/prior-node:/mnt/keystore:ro" + - "prior-node-storage:/mnt/storage" + networks: + - rehearsal + - chain-egress + + r1-node-1: + image: "${R1_IMAGE_DIGEST}" + stop_grace_period: 20160s + command: + - "start" + - "--config" + - "/mnt/keystore/config.toml" + - "--ethereum.url" + - "${ETH_WS_URL}" + - "--protocolParticipation.cutoverBlock" + - "${CUTOVER_BLOCK}" + environment: + KEEP_ETHEREUM_PASSWORD: "${KEEP_ETHEREUM_PASSWORD}" + volumes: + - "${KEYSTORE_DIR}/r1-node-1:/mnt/keystore:ro" + - "r1-node-1-storage:/mnt/storage" + networks: + - rehearsal + - chain-egress + + r1-node-2: + image: "${R1_IMAGE_DIGEST}" + stop_grace_period: 20160s + command: + - "start" + - "--config" + - "/mnt/keystore/config.toml" + - "--ethereum.url" + - "${ETH_WS_URL}" + - "--protocolParticipation.cutoverBlock" + - "${CUTOVER_BLOCK}" + environment: + KEEP_ETHEREUM_PASSWORD: "${KEEP_ETHEREUM_PASSWORD}" + volumes: + - "${KEYSTORE_DIR}/r1-node-2:/mnt/keystore:ro" + - "r1-node-2-storage:/mnt/storage" + networks: + - rehearsal + - chain-egress + +volumes: + prior-node-storage: + r1-node-1-storage: + r1-node-2-storage: + +networks: + rehearsal: + internal: true + chain-egress: {} diff --git a/scripts/release/pr4109/compose.yaml b/scripts/release/pr4109/compose.yaml new file mode 100644 index 0000000000..30c713ecd3 --- /dev/null +++ b/scripts/release/pr4109/compose.yaml @@ -0,0 +1,70 @@ +# compose.yaml — client-info port private-network smoke scaffold. +# +# Demonstrates the intended topology for the client-info port matrix: a +# keep-client node and a probe sit on a private Docker network, and the probe +# reaches the client-info listener over that private network only. Port 9601 is +# deliberately NOT published to the host (`ports:` is intentionally omitted) so +# the unauthenticated endpoint is never exposed on a public interface. +# +# This is the "compatibility default" case (no client-info settings). Fill in the +# chain endpoint, key file, and network flags for your environment, then: +# docker compose -f compose.yaml up -d node +# docker compose -f compose.yaml run --rm probe +# +# For the other matrix cases, override `command:` accordingly: +# explicit 9601 : add `--clientInfo.port 9601` +# custom : add `--clientInfo.port 9137` (and probe that port) +# CLI disabled : add `--clientInfo.port 0` (probe must get connection refused) +# TOML variants : mount a config file with `[clientInfo] Port = 9601` or `= 0` + +services: + node: + # Pin IMAGE by an immutable @sha256: digest, not a mutable tag, so the smoke + # run tests exactly the bytes operators will deploy. + image: ${IMAGE:-keep-client@sha256:REPLACE_WITH_CANDIDATE_IMAGE_DIGEST} + container_name: cutover-port-smoke-node + # No `ports:` mapping — 9601 stays internal to the private network. + networks: + - smoke + environment: + KEEP_ETHEREUM_PASSWORD: ${KEY_PASSWORD:?set KEY_PASSWORD} + volumes: + - ${KEY_FILE:?set KEY_FILE}:/mnt/keep/config/keyfile.json:ro + - ${STORAGE_DIR:-./storage}:/mnt/keep/storage + command: + - start + - --ethereum.url + - ${ETH_RPC:?set ETH_RPC} + - --ethereum.keyFile + - /mnt/keep/config/keyfile.json + - --storage.dir + - /mnt/keep/storage + # (compatibility default: no --clientInfo.port flag; 9601 is the default) + + probe: + image: curlimages/curl:8.10.1 + container_name: cutover-port-smoke-probe + depends_on: + - node + networks: + - smoke + # Succeeds only if the internal listener answers on 9601. + command: + - -fsS + - --retry + - "30" + - --retry-delay + - "2" + - --retry-connrefused + - http://cutover-port-smoke-node:9601/metrics + +networks: + smoke: + driver: bridge + # `internal: true` is the strictest isolation (no host publication AND no + # external egress) and proves 9601 is reachable only container-to-container. + # It requires the node's Ethereum/Electrum backends to also sit on this + # network; if you point the node at an EXTERNAL chain endpoint, drop this + # line (the runnable clientinfo-port-smoke.sh uses a plain bridge for exactly + # this reason) — 9601 still stays private because no `ports:` is published. + internal: true diff --git a/scripts/release/pr4109/deploy/keep-client-termination-grace.k8s-patch.yaml b/scripts/release/pr4109/deploy/keep-client-termination-grace.k8s-patch.yaml new file mode 100644 index 0000000000..114c63026a --- /dev/null +++ b/scripts/release/pr4109/deploy/keep-client-termination-grace.k8s-patch.yaml @@ -0,0 +1,33 @@ +# Kubernetes strategic-merge patch configuring the service-manager +# termination grace for a keep-client pod running the cutover release. +# +# The value is termination_grace_period_seconds from +# ../release-manifest.json: the client's in-process quiesce backstop +# (19800 s — the compiled maximum legacy completion bound plus the reviewed +# margin, converted at the reviewed upper block interval, plus the +# RPC/processing allowance) plus the reviewed forced-cancellation allowance +# (300 s) plus the process-exit headroom (60 s) for the controller +# scheduling, quiesce and close calls, logging, and teardown that run +# outside both in-process timers — Kubernetes counts this grace from SIGTERM +# delivery, before either timer arms. Without this patch the Kubernetes +# default of 30 s SIGKILLs a draining node hours before already-started +# protocol work can complete and before the audited forced-cancellation +# path runs. +# +# The grace is a ceiling, not a wait: a node whose drain completes exits +# immediately. Kubernetes delivers SIGTERM to the container's PID 1 at +# deletion, which is what starts the drain, so the container MUST run +# keep-client as PID 1 (exec-form entrypoint, no wrapping shell). +# +# Apply to each keep-client workload, for example: +# kubectl patch statefulset \ +# --patch-file keep-client-termination-grace.k8s-patch.yaml +# +# `go test ./cmd/ -run TestReleaseManifest` rejects this file whenever its +# value stops matching the validated release manifest; regenerate the +# manifest with `keep-client release-manifest derive` and re-review both +# together. +spec: + template: + spec: + terminationGracePeriodSeconds: 20160 diff --git a/scripts/release/pr4109/deploy/keep-client-termination-grace.systemd-dropin.conf b/scripts/release/pr4109/deploy/keep-client-termination-grace.systemd-dropin.conf new file mode 100644 index 0000000000..fa3c981009 --- /dev/null +++ b/scripts/release/pr4109/deploy/keep-client-termination-grace.systemd-dropin.conf @@ -0,0 +1,33 @@ +# systemd drop-in configuring the service-manager termination grace for a +# keep-client unit running the cutover release. +# +# TimeoutStopSec is termination_grace_period_seconds from +# ../release-manifest.json: the client's in-process quiesce backstop +# (19800 s — the compiled maximum legacy completion bound plus the reviewed +# margin, converted at the reviewed upper block interval, plus the +# RPC/processing allowance) plus the reviewed forced-cancellation allowance +# (300 s) plus the process-exit headroom (60 s) for the controller +# scheduling, quiesce and close calls, logging, and teardown that run +# outside both in-process timers — systemd counts this timeout from SIGTERM +# delivery, before either timer arms. Without it the systemd default +# (typically 90 s) SIGKILLs a draining node hours before already-started +# protocol work can complete and before the audited forced-cancellation +# path runs. +# +# The grace is a ceiling, not a wait: a node whose drain completes exits +# immediately. KillSignal stays SIGTERM because that is the signal the +# client's lifecycle controller quiesces on; systemd escalates to SIGKILL +# only after TimeoutStopSec. +# +# Install as: +# /etc/systemd/system/.service.d/50-termination-grace.conf +# then reload: +# systemctl daemon-reload +# +# `go test ./cmd/ -run TestReleaseManifest` rejects this file whenever its +# value stops matching the validated release manifest; regenerate the +# manifest with `keep-client release-manifest derive` and re-review both +# together. +[Service] +TimeoutStopSec=20160 +KillSignal=SIGTERM diff --git a/scripts/release/pr4109/rehearsal-evidence.schema.json b/scripts/release/pr4109/rehearsal-evidence.schema.json new file mode 100644 index 0000000000..bdb95a9da2 --- /dev/null +++ b/scripts/release/pr4109/rehearsal-evidence.schema.json @@ -0,0 +1,166 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "Cutover rehearsal evidence record", + "description": "One record per run of the exact-image single-release cutover rehearsal or the homogeneous rollback rehearsal. Screenshots alone are insufficient: every assertion must reference recorded values in this document.", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "run_id", + "gate", + "generated_at", + "source_sha", + "artifacts", + "r1_fleet", + "chain", + "release_manifest", + "stages", + "assertions" + ], + "properties": { + "schema_version": { "const": 1 }, + "run_id": { + "description": "Unique identity generated before this rehearsal touches the fleet. Supporting captures repeat it so evidence from another run cannot be substituted wholesale.", + "type": "string", + "pattern": "^[0-9a-f]{32}$" + }, + "gate": { + "description": "Which mandatory rehearsal this record evidences.", + "enum": ["single_release", "rollback"] + }, + "generated_at": { "type": "string", "format": "date-time" }, + "source_sha": { + "description": "Exact keep-core commit the R1 image was built from.", + "type": "string", + "pattern": "^[0-9a-f]{40}$" + }, + "artifacts": { + "type": "object", + "additionalProperties": false, + "required": ["r1_image_digests", "prior_image_digests", "version", "revision", "protocol_epoch"], + "properties": { + "r1_image_digests": { + "description": "The immutable R1 image this rehearsal ran, keyed by the platform its runner resolved. One runner executes one child of a published digest, so a record names one platform; covering everything the release publishes is checked across the record set.", + "type": "object", + "additionalProperties": { "type": "string", "pattern": "@sha256:[0-9a-f]{64}$" } + }, + "prior_image_digests": { + "type": "object", + "additionalProperties": { "type": "string", "pattern": "@sha256:[0-9a-f]{64}$" } + }, + "version": { "type": "string" }, + "revision": { "type": "string", "pattern": "^[0-9a-f]{40}$" }, + "protocol_epoch": { "const": "security_v2_cutover" } + } + }, + "r1_fleet": { + "description": "Exact authoritative R1 processes this run observed, without hostnames or network coordinates. Container IDs distinguish restarts/runs and operator addresses bind each process to its chain identity.", + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["service", "container_id", "operator_address"], + "properties": { + "service": { + "type": "string", + "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$" + }, + "container_id": { + "type": "string", + "pattern": "^[0-9a-f]{64}$" + }, + "operator_address": { + "type": "string", + "pattern": "^0x[0-9a-f]{40}$" + } + } + } + }, + "chain": { + "type": "object", + "additionalProperties": false, + "required": ["chain_id", "cutover_block"], + "properties": { + "chain_id": { "type": "string" }, + "cutover_block": { "type": "integer", "minimum": 1 } + } + }, + "release_manifest": { + "description": "Content binding to the reviewed release manifest whose termination grace the rehearsal fleet ran under. The sha256 is over the exact release-manifest.json bytes; validate-evidence cross-checks it against the checked-in manifest, so a record links the grace record to the source SHA, image digests, and chain identity recorded above.", + "type": "object", + "additionalProperties": false, + "required": ["sha256", "termination_grace_period_seconds"], + "properties": { + "sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "termination_grace_period_seconds": { "type": "integer", "minimum": 1 } + } + }, + "chain_inputs": { + "description": "SHA-256 digests of the external inputs the rehearsal depends on but does not contain. Two are programs it executes: the work driver that originates ceremonies on the rehearsal chain, and the generator that produces each node's external rollback evidence. Both arrive from outside the repository and both produce readings that become release evidence, so the rehearsal refuses to run either unless its bytes hash to the digest reviewed in scripts/release/pr4109/chain-inputs.sha256. The third is never executed: the archived independent cryptographic review of the dual-mode dependency revision go.mod resolves, which gates acceptance of a single-release record rather than execution of any step. A record naming no digest for an input is a run that was never handed one.", + "type": "object", + "additionalProperties": false, + "properties": { + "work_driver_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "rollback_evidence_generator_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" }, + "tsslib_review_sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" } + } + }, + "stages": { + "description": "One entry per executed rehearsal step, in execution order, with the canonical and callback blocks, permit modes, and gauge snapshots observed at that step.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["name", "outcome"], + "properties": { + "name": { "type": "string" }, + "outcome": { "enum": ["pass", "fail", "blocked"] }, + "canonical_blocks": { "type": "array", "items": { "type": "integer" } }, + "callback_blocks": { "type": "array", "items": { "type": "integer" } }, + "permit_modes": { + "type": "array", + "items": { "enum": ["legacy", "security_v2"] } + }, + "gauges": { + "description": "participation gate gauge snapshot at the step.", + "type": "object", + "additionalProperties": { "type": "number" } + }, + "transaction_hashes": { + "type": "array", + "items": { "type": "string", "pattern": "^0x[0-9a-f]{64}$" } + }, + "state_checksums": { + "description": "Non-secret checksums of persisted state snapshots (active and quarantine namespaces).", + "type": "object", + "additionalProperties": { "type": "string" } + }, + "evidence_refs": { + "description": "Safe relative identifiers for supporting artifacts stored beneath the rehearsal evidence directory.", + "type": "object", + "additionalProperties": { "type": "string" } + }, + "notes": { "type": "string" } + } + } + }, + "assertions": { + "description": "The gate's acceptance assertions with their observed values; every one must reference stage evidence above.", + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["assertion", "holds", "evidence_stage"], + "properties": { + "assertion": { "type": "string" }, + "holds": { "type": "boolean" }, + "evidence_stage": { "type": "string" } + } + } + } + } +} diff --git a/scripts/release/pr4109/rehearse.sh b/scripts/release/pr4109/rehearse.sh new file mode 100755 index 0000000000..4338c3716d --- /dev/null +++ b/scripts/release/pr4109/rehearse.sh @@ -0,0 +1,15367 @@ +#!/usr/bin/env bash +# +# Single-release cutover rehearsal driver. +# +# This driver structures the two mandatory container rehearsals — the +# exact-image single-release cutover rehearsal and the homogeneous rollback +# rehearsal — as explicit, individually reportable stages. Stages that are +# provable from this repository alone run real Go tests. Stages that require +# the immutable prior-production and R1 runtime images, a rehearsal chain, and +# persistent volumes refuse to run until those inputs are supplied: a +# rehearsal stage that cannot execute reports BLOCKED with its exact missing +# inputs instead of pretending to pass. +# +# Required environment for the container stages: +# +# PRIOR_IMAGE_DIGEST immutable prior-production runtime image digest +# (repo@sha256:...); a mutable tag is not evidence +# R1_IMAGE_DIGEST immutable R1 candidate runtime image digest +# PROBE_IMAGE_DIGEST immutable digest of the image every evidence probe +# runs in; it reads the numbers that become the record, +# so a mutable tag would leave the reading instrument +# outside the record's own provenance +# ETH_WS_URL rehearsal chain websocket endpoint +# ETH_RPC_URL the same chain's JSON-RPC endpoint, questioned to +# confirm every transaction a work driver reports; +# an unconfirmed report is the driver's own account +# of itself +# CUTOVER_BLOCK rehearsed cutover block C on that chain +# CHAIN_ID that chain's numeric id, recorded in the evidence +# KEYSTORE_DIR per-node rehearsal inputs, one subdirectory per +# compose service holding that node's config.toml and +# operator key file; each config must declare a nonzero +# clientInfo.port, which is the only surface the +# rehearsal can read that node's evidence from +# KEEP_ETHEREUM_PASSWORD operator key file password for the fleet +# STORAGE_SNAPSHOT_DIR rollback only: one storage snapshot per R1 service +# for the offline state audit +# PR4109_EVIDENCE_RECORD_SUFFIX +# optional safe filename component identifying the +# native runner (for example amd64 or arm64-v8). The +# dispatched workflow sets it so records from separate +# platform jobs remain unique when they are aggregated +# +# Rollback only — the audit inputs no storage snapshot can supply. Every one +# is required before the offline state audit can authorize anything, and a +# missing one blocks the barrier that releases the prior binary rather than +# being skipped: +# +# PR4109_ROLLBACK_EVIDENCE_GENERATOR +# executable called once per drained node as +# , +# after that node's state has been captured and +# audited for identity. It must write +# chain-reconciliation.json, +# bitcoin-reconciliation.json, quiescence-report.json, +# and prior-reader-compatibility.json into the output +# directory, each naming the identity manifest's +# snapshot_aggregate_sha256. The chain record must also +# contain signed successful receipt/log projections +# from the independently trusted collector named below. +# It is run rather than supplied as files because every +# record has to speak for the exact snapshot this run +# captured, and that snapshot does not exist until the +# fleet has drained +# PR4109_WALLET_REGISTRY_ADDRESS +# exact WalletRegistry address whose logs establish DKG +# settlement +# PR4109_RANDOM_BEACON_ADDRESS +# exact RandomBeacon address whose logs establish relay +# entry request, delivery, and timeout settlement +# PR4109_FINALIZED_ETHEREUM_BLOCK_NUMBER +# PR4109_FINALIZED_ETHEREUM_BLOCK_HASH +# independently obtained finalized-chain anchor the +# collector's canonical block set must end at +# PR4109_CHAIN_EVIDENCE_PUBLIC_KEY +# lowercase hexadecimal Ed25519 public key provisioned +# independently from the evidence generator; its +# signature authenticates the complete chain record +# PR4109_BITCOIN_NETWORK the Bitcoin network the rollback targets +# PR4109_PRIOR_VERSION exact version of the prior release restored +# PR4109_PRIOR_REVISION exact revision of the prior release restored +# PR4109_WORK_DRIVER executable that originates protocol work on the +# rehearsal chain, called with the phase name. The +# fleet only reacts to chain events, so without it no +# ceremony exists to observe and the steps that need +# one record themselves blocked. On stdout it may +# report what it originated, as a JSON object whose +# optional transaction_hashes array carries +# 0x-prefixed 32-byte hashes and whose optional +# ceremony_results array carries {ceremony, outcome} +# objects: the terminal result of each ceremony those +# transactions started, which no fleet counter can +# supply because a permit says a node was allowed to +# begin and a positive control is about one finishing. +# A report that cannot be read stops the step rather +# than passing for nothing having happened +# PR4109_TSSLIB_REVIEW archived independent cryptographic review of the +# dual-mode dependency revision go.mod resolves. It is +# never executed and gates no step: a rehearsal runs +# every mixed prior/R1 stage without it and records +# what it observed. What it decides is acceptance — +# whether those transcripts are release-authoritative. +# Its bytes must hash to the reviewed tsslib-review +# digest and the document must name the exact revision +# go.mod resolves +# +# Fail-closed source binding (every proof stage): +# +# PR4109_EXPECTED_SOURCE_COMMIT +# when set, a proof stage refuses to run unless the +# tree under test is exactly this commit: readable +# git metadata, HEAD equal to the value, and no +# divergence — untracked files included +# PR4109_SOURCE_BINDING_MODE +# exact (default) tolerates no divergence at all; +# build-image accepts only what the CI build image +# produces by design: context-excluded paths absent +# from the image, untracked files classified under +# the commit's own restored .gitignore rules, and +# the regenerated gen/ binding and _address families +# — never the committed protobuf code — restored +# byte-exact from the dispatched commit before any +# test compiles them, with a post-restore re-check +# that fails on anything left beyond the +# context-excluded absences +# +# Which absences build-image mode may explain away is decided by a +# classification written out in this script, so it is held to the commit's +# own .dockerignore rather than trusted: local-proofs and shell-analysis both +# compare the two over every tracked path and refuse to go on once they +# disagree in any direction the image build does not account for. Which +# ignore file that is comes out of the rehearsal workflow's build step, read +# from the commit rather than restated here, and the scaffold lint's path +# filters are held to the same resolution — otherwise a build moved onto +# another Dockerfile would take its ignore rules somewhere nothing checks. +# +# Evidence is written under EVIDENCE_DIR (default: ./rehearsal-evidence). +# Every accepted rehearsal run must produce a record conforming to +# rehearsal-evidence.schema.json and binding the checked-in release +# manifest — its exact hash and its termination grace; the validate-evidence +# stage enforces both, self-testing its own checker first. It also requires +# exactly one record for each single_release/rollback and published-platform +# pair; per-run emission validates only its own record, because no native +# runner can see another runner's workspace. Those comparisons only speak for +# the release while that manifest still matches the compiled +# bounds, so local-proofs attests it under EVIDENCE_DIR/attestation and +# validate-evidence refuses to measure a record without that receipt. The +# receipt belongs to one run at one commit: local-proofs destroys the +# inherited one before it proves anything and publishes its own by atomic +# rename only after every proof passed, stamping the commit the binding +# check proved, and validate-evidence requires that stamp to equal both its +# own binding and every record's source_sha. +# +# All of that decides whether a record is admissible, which is not whether it +# accepts anything. A record is where a rehearsal says a mandatory step +# failed or an acceptance assertion does not hold, so a correctly bound, +# schema-valid record can be exactly the evidence that a gate must not be +# accepted. Both the rehearsal's own exit and validate-evidence therefore +# read the recorded outcomes as the verdict: a failed step or a refused +# assertion refuses the gate, a step that never executed leaves it +# unrehearsed, and only a run with none of the three reports success. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" +EVIDENCE_DIR="${EVIDENCE_DIR:-${SCRIPT_DIR}/rehearsal-evidence}" + +# Where this scaffold lives inside the repository, and what invokes it, both +# resolved from the paths above rather than restated. The gate below has to +# run on every change to the scaffold's own code and has to be checked for +# still invoking it, and naming either a second time is exactly the +# restatement that would go stale the first time the scaffold moved. +SCAFFOLD_DIR="${SCRIPT_DIR#"${REPO_ROOT}/"}" +SCAFFOLD_ENTRYPOINT="$(basename "${BASH_SOURCE[0]}")" + +# The stage that gate exists to run — this script's own analysis verb, which +# is the one thing about the invocation it cannot read off its own identity. +SCAFFOLD_LINT_STAGE="shell-analysis" + +# The environment names the invocation may carry, which is the one this +# entrypoint documents itself as reading (EVIDENCE_DIR, above). Everything +# else is refused rather than dropped, because what bash does with a script it +# is handed is decided in that environment and not in the command line the +# reading below can see: BASH_ENV names a file bash sources before the +# script's first line, and a file that exits there ends the run at status zero +# without a line of the analysis having run; SHELLOPTS carrying `noexec` has +# bash parse the whole script and execute none of it. An assignment silently +# dropped is a command word read out of a command that is not the one that +# would run. +SCAFFOLD_LINT_ENV_NAMES="EVIDENCE_DIR" + +# The commit verify_source_binding proved the tree under test to be, empty +# until it has proved one. Only a caller-supplied binding can establish an +# identity a stage may stamp into evidence; an unbound run leaves this empty +# and falls back to the tree's own (possibly -dirty) stamp. +VERIFIED_SOURCE_COMMIT="" + +usage() { + cat <<'EOF' +usage: rehearse.sh + +stages: + local-proofs run the repository-local Go proofs of the cutover gate: + boundary modes, pre-C permit surviving C, quiescence and + the signal lifecycle, forced shutdown and clock-failure + quarantine, penalty suppression, forwarding lifecycle, + held-wait cancellation, the offline state audit, and the + tBTC cutover ceremony suites — real security-v2 + transcripts, the ten-misbehaved-seat real result, the + production-scale 90/10 split, heartbeat bands, and + roster wiring — under the race detector, plus the + integration-tag compile proof; self-tests source binding, + evidence acceptance, native-runner dispatch, and the + fleet evidence-window capture first (node/npx required), + reports every skipped case explicitly, holds the + verifier's build-context + classification to the commit's own .dockerignore, + discards any inherited EVIDENCE_DIR/attestation before + proving anything, and + ends by attesting the checked-in release manifest + against the compiled bounds — stamped with the commit + the binding check proved — into that directory by + atomic rename (runs today, no Docker) + static-analysis run the static analyzers CI enforces on the Go tree, + every tool at an immutable version: gofmt, go vet + over ./... (strictly wider than CI's root-only vet), + staticcheck 2025.1.1 (-SA1019), gosec v2.28.0 with + the CI flag set (G115/G118 and generated bindings + excluded; CI's own gosec action floats on master, the + pin keeps this evidence reproducible), and + golangci-lint v2.12.2 (network needed on first run to + fetch the pinned tools) + shell-analysis analyze the rehearsal scaffold itself: bash -n and + ShellCheck over every script here, actionlint v1.7.12 + over the scaffold's own workflows, the build-context + classification checked against the commit's own + .dockerignore over every tracked path — the file + selected by the Dockerfile the rehearsal workflow's + build step really compiles, with that workflow's own + path filters held to the same resolution — and both + validator self-tests: the gate the scaffold's CI job + runs on every change to these files and to the build + inputs they mirror, so the checkers and fleet-window + capture that admit rehearsal evidence are never proved + only by a manual dispatch + solidity-proofs build and test the changed ECDSA contracts surface + exactly as the contracts workflow's build-and-test job + does: the exact Node release that job pins — read out + of it, not restated here, so the stage blocks rather + than claims a parity CI has moved away from — the + Corepack-managed yarn from packageManager, and a + never-skipped 'yarn install --immutable' before + yarn build and yarn test + preflight validate the container-rehearsal inputs and image digests + single-release exact-image cutover rehearsal: prior+R1 mixed fleet + before C, work across C without restart, straggler + negative control, clock failure, quiesce with in-flight + permits. Runs every step this release can execute, + records each step's own outcome, and emits an evidence + record naming the steps that could not run and why; + exits FAIL if any mandatory step failed or any + acceptance assertion does not hold, and BLOCKED if any + step could not execute + rollback homogeneous rollback rehearsal: quiesce all R1, + all-candidate-down barrier, offline state audit, staged + prior redeploy, forbidden partial-rollback attempt. + Same per-step ledger and verdict as single-release; + additionally needs STORAGE_SNAPSHOT_DIR — the directory + this stage captures each drained node's state into, + straight out of the container the drain stopped, so the + audit's verdict is over the state this fleet left and + not over a tree supplied under the same name + verify-source-binding + run only the fail-closed source binding check on this + tree and record it; inside the CI build image set + PR4109_SOURCE_BINDING_MODE=build-image + validate-evidence validate every evidence record under EVIDENCE_DIR + against rehearsal-evidence.schema.json and require + each record's release-manifest binding — the exact + manifest hash and the termination grace the fleet ran + under — to match the checked-in reviewed manifest; + requires the local-proofs attestation proving that + manifest still matches the compiled bounds, requires + the attestation, every record, and this run's own + binding to name one commit, verifies its own source + binding like any proof stage, and self-tests its + checker first. Requires exactly one record for every + single_release/rollback and published-platform pair, + rejecting a wholly missing gate and duplicate accounts. + Then asks the separate question the binding checks + cannot: a correctly bound record still says whether its + gate held, so the stage exits FAIL on any recorded + failed step or refused acceptance assertion and BLOCKED + on any step that never executed + +environment (every proof stage): + PR4109_EXPECTED_SOURCE_COMMIT + fail closed: refuse to run unless the tree under test + is exactly this commit (clean, untracked included) + PR4109_SOURCE_BINDING_MODE + exact (default) | build-image (accept only the CI + build image's designed divergence: context-excluded + absences, with every regenerated gen/ file restored + byte-exact from the dispatched commit before testing) + +environment (preflight, single-release, rollback): + PRIOR_IMAGE_DIGEST immutable prior-production runtime digest + R1_IMAGE_DIGEST immutable R1 candidate runtime digest + PROBE_IMAGE_DIGEST immutable digest of the wget-carrying image every + evidence reading is scraped with + ETH_WS_URL rehearsal chain websocket endpoint + ETH_RPC_URL the same chain's JSON-RPC endpoint, questioned to + confirm every reported transaction + CUTOVER_BLOCK rehearsed cutover block C on that chain + CHAIN_ID that chain's numeric chain id + KEYSTORE_DIR per-node inputs, one / directory each holding + that node's config.toml and key material + PR4109_EVIDENCE_RECORD_SUFFIX + optional safe filename component identifying this + native runner; the workflow sets one per platform + KEEP_ETHEREUM_PASSWORD + the key files' password + PR4109_WORK_DRIVER executable called with the phase name to originate + protocol work on the rehearsal chain; may report the + transactions it submitted as a JSON object with a + transaction_hashes array. The fleet only reacts to + chain events, so the steps that need a ceremony record + themselves blocked without one + PR4109_TSSLIB_REVIEW + archived independent cryptographic review of the + dual-mode dependency revision go.mod resolves. Gates + acceptance of the emitted record, not execution of any + step + +environment (rollback, additionally): + STORAGE_SNAPSHOT_DIR + where this stage captures each drained node's state + from the container it stopped, for the offline audit + PR4109_ROLLBACK_EVIDENCE_GENERATOR + executable run once per drained node as + , after that + node's state is captured and audited for identity. It + writes the reconciliation, quiescence, and prior-reader + records the audit binds its verdict to, each naming the + snapshot the manifest identifies. From a snapshot alone + the audit reports namespace consistency and nothing + about rollback safety, and a record produced before the + drain could not name the snapshot the drain left + PR4109_WALLET_REGISTRY_ADDRESS + exact WalletRegistry whose raw logs establish DKG state + PR4109_RANDOM_BEACON_ADDRESS + exact RandomBeacon whose raw logs establish relay entry + request, delivery, and timeout settlement + PR4109_FINALIZED_ETHEREUM_BLOCK_NUMBER + PR4109_FINALIZED_ETHEREUM_BLOCK_HASH + independently obtained finalized block anchoring the + authenticated canonical block set + PR4109_CHAIN_EVIDENCE_PUBLIC_KEY + lowercase hexadecimal Ed25519 public key of the trusted + finalized-chain evidence collector + PR4109_BITCOIN_NETWORK + PR4109_PRIOR_VERSION + PR4109_PRIOR_REVISION + the operational identities the audit requires the + snapshot and the restored artifact to agree with +EOF +} + +note() { printf '>> %s\n' "$*"; } +blocked() { + printf 'BLOCKED: %s\n' "$*" >&2 + exit 3 +} +fail() { + printf 'FAIL: %s\n' "$*" >&2 + exit 1 +} + +# Working-tree divergence from HEAD as porcelain lines, untracked files +# included: a file git does not track can still change what a go or yarn +# invocation tests, so only ignored paths (evidence logs, build output) are +# exempt. If git itself cannot answer, a sentinel line keeps every consumer +# fail-closed instead of mistaking an error for a clean tree. +source_divergence() { + git -C "${REPO_ROOT}" status --porcelain 2>/dev/null || + printf '!! git status failed; divergence unknown\n' +} + +# The exact source commit every stage stamps into its log. A working tree +# that differs from HEAD — untracked files included — is marked -dirty so a +# local log can never pass for evidence of the clean commit; outside a git +# checkout the stamp degrades to "unknown" instead of failing the stage. +# Refusing to run on divergence is verify_source_binding's job. +source_commit() { + local commit + if ! commit="$(git -C "${REPO_ROOT}" rev-parse HEAD 2>/dev/null)"; then + printf 'unknown' + return + fi + if [[ -n "$(source_divergence)" ]]; then + commit="${commit}-dirty" + fi + printf '%s' "${commit}" +} + +# sha256 of stdin, portable across the CI build image (busybox sha256sum) +# and a macOS workstation (shasum). +hash_stdin() { + if command -v sha256sum >/dev/null 2>&1; then + sha256sum | awk '{print $1}' + else + shasum -a 256 | awk '{print $1}' + fi +} + +# The CI build image drops every root dotfile from its build context (the +# .dockerignore `.*` rule), so inside the image git sees a tree without the +# repository's own ignore rules and every gitignored build output — the +# keep-client binary, the tmp/contracts artifact trees — as untracked +# divergence. Restore the committed .gitignore files, and only where they +# are absent: the restored bytes come from the commit under verification +# itself, so restoration cannot mask anything, while a present-but-modified +# .gitignore keeps its modified status and still fails the stamp. +restore_committed_gitignores() { + local path + git -C "${REPO_ROOT}" ls-tree -r --name-only HEAD | + { grep -E '(^|/)\.gitignore$' || true; } | + while IFS= read -r path; do + if [[ ! -e "${REPO_ROOT}/${path}" ]]; then + mkdir -p "${REPO_ROOT}/$(dirname "${path}")" + git -C "${REPO_ROOT}" show "HEAD:${path}" >"${REPO_ROOT}/${path}" + fi + done +} + +# True when .dockerignore keeps this committed path out of the build context +# entirely, so its absence inside the image is the image's construction and +# not divergence. Mirrors .dockerignore rule by rule, negations included: +# .clusterfuzzlite MUST reach the context, so its absence is never explained +# away, and the regenerated gen/ families are deliberately not listed here — +# the image is supposed to recreate them, so their absence is drift. +dockerignore_excluded_path() { + local path="$1" + if [[ "${path}" =~ ^\.clusterfuzzlite(/|$) ]]; then + return 1 + fi + # The files .dockerignore negates back out of docs/ and scripts/ for the + # tests that open them, which run inside the image: an absence here is the + # missing input those tests fail on, never the image's construction. Listed + # one for one with the negations so the two can be read against each other. + if [[ "${path}" =~ ^docs/performance-metrics\.adoc$ ]] || + [[ "${path}" =~ ^scripts/release/pr4109/(compose\.rehearsal\.yaml|rehearsal-evidence\.schema\.json|release-manifest\.json|release-manifest\.schema\.json|release-provenance\.schema\.json)$ ]] || + [[ "${path}" =~ ^scripts/release/pr4109/deploy/keep-client-termination-grace\.(k8s-patch\.yaml|systemd-dropin\.conf)$ ]]; then + return 1 + fi + [[ "${path}" =~ ^\.[^/]*(/|$) ]] && return 0 + [[ "${path}" =~ ^docs[^/]*/ ]] && return 0 + [[ "${path}" =~ ^(infrastructure|scripts|tmp|solidity|token-stakedrop|token-tracker)/ ]] && + return 0 + [[ "${path}" =~ ^(CODEOWNERS|Dockerfile)$ ]] && return 0 + [[ "${path}" =~ ^[^/]+\.adoc$ ]] && return 0 + [[ "${path}" =~ (^|/)node_modules/ ]] && return 0 + [[ "${path}" =~ (^|/)gen/_contracts(/|$) ]] && return 0 + return 1 +} + +# True for the tracked files the image legitimately rewrites: .dockerignore +# keeps **/gen/**/*.go and **/gen/_address/* out of the context, and +# `make get_artifacts` + `make generate` recreate them from the published +# contract artifacts before the final COPY. The negated families — +# gen/pb/*.go, gen/gen.go, gen/cmd/cmd.go — DO reach the context and are +# overwritten with committed bytes by that COPY, so a difference there is +# tampering, never regeneration: the committed protobuf message code the +# tests compile stays byte-bound to the dispatched commit. A match here +# never accepts the found bytes — it only marks the path for byte-exact +# restoration from the dispatched commit before anything compiles it. +regenerated_by_design_path() { + local path="$1" + if [[ "${path}" =~ (^|/)gen/pb/[^/]+\.go$ ]] || + [[ "${path}" =~ (^|/)gen/gen\.go$ ]] || + [[ "${path}" =~ (^|/)gen/cmd/cmd\.go$ ]]; then + return 1 + fi + [[ "${path}" =~ (^|/)gen/.+\.go$ ]] && return 0 + [[ "${path}" =~ (^|/)gen/_address/[^/]+$ ]] && return 0 + return 1 +} + +# The workflow whose build step decides what the classification below has to +# be checked against, and the unconditional lint that has to run whenever any +# of those inputs changes. Both are paths inside the commit under test rather +# than on disk: the build context of a dispatched commit is that commit's tree. +REHEARSAL_WORKFLOW=".github/workflows/cutover-rehearsal.yml" +SCAFFOLD_LINT_WORKFLOW=".github/workflows/cutover-scaffold-lint.yml" + +# The action that workflow builds the proof image with. Its `context` and +# `file` inputs are the whole of what selects the build's ignore rules. +BUILD_ACTION="docker/build-push-action" + +# Read out of that step by resolve_build_step_identity: the build context root, +# and the Dockerfile the builder compiles relative to it. The Dockerfile name +# matters beyond the build itself, because it is what selects the ignore rules +# below — which is exactly why neither is restated here as a constant. +BUILD_CONTEXT="" +BUILD_DOCKERFILE="" + +# The ignore rules the two classifications above mirror, compiled once per +# tree into one extended regular expression per pattern with a parallel flag +# marking the negations, alongside the context-relative path they were read +# from. They are read from the commit, not from disk: the build context of a +# dispatched commit is that commit's own tree, and inside the build image the +# file itself is one of the paths its own `.*` rule kept out. +DOCKERIGNORE_SOURCE="" +DOCKERIGNORE_REGEX=() +DOCKERIGNORE_NEGATED=() + +# Go's path/filepath.Clean over a slash-separated path, which the builder +# applies to every ignore line before compiling it: a `.` segment drops out, +# a `..` pops the segment before it, repeated separators collapse, a rooted +# path keeps exactly one leading separator, and a relative path cleaned away +# to nothing becomes `.`. +# +# Without it, a rule written `./scripts` or `docs/../docs` would compile here +# into an expression matching nothing at all, and every path the build really +# removes under that rule would read as still in the build context — the +# dangerous direction, where an absence gets explained away. +dockerignore_clean_path() { + local path="$1" + if [[ -z "${path}" ]]; then + printf '.' + return + fi + + local rooted=0 + [[ "${path}" == /* ]] && rooted=1 + + local segments=() kept=() segment last cleaned="" i + IFS='/' read -r -a segments <<<"${path}" + for ((i = 0; i < ${#segments[@]}; i++)); do + segment="${segments[i]}" + case "${segment}" in + '' | '.') ;; + '..') + if ((${#kept[@]} > 0)); then + last="${kept[$((${#kept[@]} - 1))]}" + if [[ "${last}" != '..' ]]; then + unset "kept[$((${#kept[@]} - 1))]" + continue + fi + fi + # A rooted path has nothing above its root to climb to, so a `..` it + # cannot pop is dropped rather than kept. + ((rooted == 1)) || kept+=('..') + ;; + *) kept+=("${segment}") ;; + esac + done + + for ((i = 0; i < ${#kept[@]}; i++)); do + [[ -n "${cleaned}" ]] && cleaned+='/' + cleaned+="${kept[i]}" + done + + if ((rooted == 1)); then + printf '/%s' "${cleaned}" + elif [[ -z "${cleaned}" ]]; then + printf '.' + else + printf '%s' "${cleaned}" + fi +} + +# Translate one normalized ignore pattern into an extended regular expression +# over a whole context-relative path, following the build daemon's own +# compilation: `*` stops at a path separator, `?` is a single non-separator +# character, `**` spans any number of whole segments (`.*` when it ends the +# pattern), and every other character is literal. +# +# The daemon compiles to a regular expression too, and escapes exactly the +# five characters escaped below on the way — every other character reaches +# its expression engine carrying whatever meaning that engine gives it. So +# this translation is the daemon's only for patterns that carry none of the +# remaining metacharacters, and load_dockerignore_patterns refuses those, +# backslash escapes included, before this ever sees them: a refusal raised +# here would run inside a command substitution and exit nothing but its own +# subshell. +dockerignore_pattern_regex() { + local pattern="$1" out="^" i ch + for ((i = 0; i < ${#pattern}; i++)); do + ch="${pattern:i:1}" + if [[ "${ch}" == '*' ]]; then + if [[ "${pattern:i+1:1}" == '*' ]]; then + i=$((i + 1)) + # A `**/` prefix spans whole segments, so the separator belongs to it. + [[ "${pattern:i+1:1}" == '/' ]] && i=$((i + 1)) + if ((i + 1 == ${#pattern})); then + out+='.*' + else + out+='(.*/)?' + fi + else + out+='[^/]*' + fi + elif [[ "${ch}" == '?' ]]; then + out+='[^/]' + elif [[ '.+()$' == *"${ch}"* ]]; then + out+="\\${ch}" + else + out+="${ch}" + fi + done + printf '%s$' "${out}" +} + +# The characters the daemon hands to its expression engine unescaped and this +# script has no translation for: a character class (`[`…`]`, whose negation +# form the engine reads back to front from the glob grammar the rules are +# documented in), a repetition (`{`…`}`), an alternation (`|`), a class +# negation (`^`), and a backslash escape. Naming them one by one keeps the +# refusal specific enough to act on. +dockerignore_unmodelled_construct() { + local pattern="$1" i ch + for ((i = 0; i < ${#pattern}; i++)); do + ch="${pattern:i:1}" + if [[ '[]{}|^' == *"${ch}"* || "${ch}" == $'\\' ]]; then + printf '%s' "${ch}" + return 0 + fi + done + return 1 +} + +# The value a `key:` line carries, with one layer of matching quotes taken off +# and a trailing comment dropped the way the workflow parser drops it. Refuses +# — non-zero, no output — any quoting that would need escape processing to +# read, because a value carrying its own escapes is a value this parser and the +# workflow parser could disagree about. +yaml_scalar_value() { + local raw="$1" quote body rest + raw="${raw#"${raw%%[![:space:]]*}"}" + raw="${raw%"${raw##*[![:space:]]}"}" + case "${raw}" in + '"'* | "'"*) + quote="${raw:0:1}" + body="${raw:1}" + [[ "${body}" == *"${quote}"* ]] || return 1 + rest="${body#*"${quote}"}" + body="${body%%"${quote}"*}" + rest="${rest#"${rest%%[![:space:]]*}"}" + [[ -z "${rest}" || "${rest}" == '#'* ]] || return 1 + [[ "${body}" == *$'\\'* ]] && return 1 + printf '%s' "${body}" + ;; + *) + if [[ "${raw}" == *' #'* ]]; then + raw="${raw%% #*}" + raw="${raw%"${raw##*[![:space:]]}"}" + fi + printf '%s' "${raw}" + ;; + esac +} + +# The raw spellings of a value this parser refuses rather than guesses at: +# every one of them means something to the workflow parser that reading the +# characters literally would get wrong. Returns the reason, like +# dockerignore_unmodelled_construct, so the refusal is raised by a caller that +# can still stop the run rather than inside a command substitution. +# +# The expression opener is matched as the literal characters the workflow +# parser reads there, so it is deliberately never expanded here. +# shellcheck disable=SC2016 +yaml_unmodelled_value() { + local raw="$1" + case "${raw}" in + '') printf 'no value at all' ;; + '|'* | '>'*) printf 'a block scalar' ;; + '&'*) printf 'an anchor' ;; + '*'*) printf 'an alias' ;; + '['* | '{'*) printf 'a flow collection' ;; + *'${{'*) printf 'a workflow expression' ;; + *) return 1 ;; + esac + return 0 +} + +# Split the workflow into per-line indentation widths and leading-whitespace- +# stripped bodies, with -1 marking a line a parser has nothing to place — a +# blank line, or a comment at any column. Populates YAML_INDENTS and +# YAML_BODIES because a command substitution could not raise the tab refusal. +YAML_INDENTS=() +YAML_BODIES=() +yaml_index_lines() { + local source="$1" content="$2" line trimmed i + YAML_INDENTS=() + YAML_BODIES=() + + local -a lines=() + while IFS= read -r line; do lines+=("${line}"); done <<<"${content}" + + for ((i = 0; i < ${#lines[@]}; i++)); do + line="${lines[i]}" + trimmed="${line#"${line%%[![:space:]]*}"}" + if [[ -z "${trimmed}" || "${trimmed}" == '#'* ]]; then + YAML_INDENTS+=(-1) + YAML_BODIES+=("") + continue + fi + # YAML forbids a tab in indentation outright, so a width measured over one + # would not be the width the workflow parser sees. + if [[ "${line%%[![:space:]]*}" == *$'\t'* ]]; then + fail "${source} line $((i + 1)) indents with a tab, which YAML does not \ +allow as indentation and this parser cannot place" + fi + YAML_INDENTS+=("$((${#line} - ${#trimmed}))") + YAML_BODIES+=("${trimmed}") + done +} + +# The column a sequence item's own mapping keys sit at — past the dash and the +# whitespace after it — or nothing when the line does not open one. +yaml_item_key_indent() { + local index="$1" body value stripped + body="${YAML_BODIES[index]}" + [[ "${body}" == '-'[[:space:]]* ]] || return 1 + value="${body#-}" + stripped="${value#"${value%%[![:space:]]*}"}" + printf '%s' "$((YAML_INDENTS[index] + 1 + ${#value} - ${#stripped}))" +} + +# The index one past the last line belonging to a block whose content sits at +# `indent`, starting the scan at `from`. A block ends at the first line placed +# shallower than its own content, which is also how the next sequence item +# ends the one before it. +yaml_block_end() { + local from="$1" indent="$2" i + for ((i = from; i < ${#YAML_INDENTS[@]}; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + if ((YAML_INDENTS[i] < indent)); then + printf '%s' "${i}" + return + fi + done + printf '%s' "${#YAML_INDENTS[@]}" +} + +# The inputs of the step yaml_locate_action_step last placed, split into keys +# and their still-raw values. Parallel arrays because the shapes below have to +# stay bash-3 portable, and globals because the placement refuses unmodelled +# shapes as it goes and a refusal inside a command substitution would exit +# nothing but its own subshell. +YAML_STEP_INPUT_KEYS=() +YAML_STEP_INPUT_VALUES=() + +# Place the one step in [from, to) whose `uses:` names the given action and +# read its inputs. Placing a step is the same problem wherever the step lives, +# and every shape this parser does not read the way the workflow parser does is +# refused by name: a value resolved on a guess is worse than no value, because +# the guess is what every claim built on it would then be measured against. +yaml_locate_action_step() { + local source="$1" action="$2" from="$3" to="$4" + YAML_STEP_INPUT_KEYS=() + YAML_STEP_INPUT_VALUES=() + + # Every step using the action, whichever of the two spellings its `uses:` + # line takes — opening the sequence item or following one. + local -a hits=() + local i body value + for ((i = from; i < to; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + body="${YAML_BODIES[i]}" + if [[ "${body}" == '-'[[:space:]]* ]]; then + body="${body#-}" + body="${body#"${body%%[![:space:]]*}"}" + fi + [[ "${body}" == 'uses:'* ]] || continue + value="$(yaml_scalar_value "${body#uses:}")" || continue + [[ "${value}" == "${action}@"* ]] || continue + hits+=("${i}") + done + + ((${#hits[@]} != 0)) || + fail "${source} has no ${action} step; the values this script would \ +otherwise be restating are read out of that step, and there is nothing left \ +to read them from" + ((${#hits[@]} == 1)) || + fail "${source} has ${#hits[@]} ${action} steps; this script cannot tell \ +which one the values it reads belong to" + + # The step's mapping keys sit at the sequence item's content column: on the + # `uses:` line itself when that line opens the item, and otherwise at the + # column the item's own dash line opened. + local hit="${hits[0]}" start key_indent opened + if key_indent="$(yaml_item_key_indent "${hit}")"; then + start="${hit}" + else + key_indent="${YAML_INDENTS[hit]}" + start=-1 + for ((i = hit - 1; i >= from; i--)); do + ((YAML_INDENTS[i] < 0)) && continue + ((YAML_INDENTS[i] < key_indent)) || continue + start="${i}" + break + done + ((start >= 0)) || + fail "${source}: the ${action} step on line $((hit + 1)) opens no \ +sequence item this parser can place" + opened="$(yaml_item_key_indent "${start}")" || opened="" + [[ "${opened}" == "${key_indent}" ]] || + fail "${source} line $((start + 1)) is not the sequence item opening the \ +${action} step; this parser cannot place that step's inputs" + fi + + local end with_line=-1 + end="$(yaml_block_end "$((start + 1))" "${key_indent}")" + ((end > to)) && end="${to}" + + # The `with:` mapping, and nothing else read as one: a key line this parser + # cannot split is a step shape it is not reading the way the workflow parser + # does, wherever in the step it sits. + for ((i = start + 1; i < end; i++)); do + ((YAML_INDENTS[i] == key_indent)) || continue + body="${YAML_BODIES[i]}" + [[ "${body}" == *:* && "${body%%:*}" =~ ^[A-Za-z_][A-Za-z0-9_-]*$ ]] || + fail "${source} line $((i + 1)) is not a key this parser can read inside \ +the ${action} step" + [[ "${body%%:*}" == 'with' ]] || continue + value="${body#with:}" + value="${value#"${value%%[![:space:]]*}"}" + [[ -z "${value}" || "${value}" == '#'* ]] || + fail "${source} line $((i + 1)) writes the ${action} step's inputs as \ +[${value}]; this parser reads only a block mapping" + with_line="${i}" + done + + # A step passing no inputs at all is a legible shape. Whether it is an + # acceptable one is the caller's question, not this parser's. + ((with_line >= 0)) || return 0 + + local input_indent=-1 + for ((i = with_line + 1; i < end; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + # The step's own next key closes the mapping. + ((YAML_INDENTS[i] <= key_indent)) && break + if ((input_indent < 0)); then + input_indent="${YAML_INDENTS[i]}" + fi + ((YAML_INDENTS[i] > input_indent)) && continue + ((YAML_INDENTS[i] == input_indent)) || + fail "${source} line $((i + 1)) is indented under the ${action} step's \ +inputs at a column this parser cannot place" + body="${YAML_BODIES[i]}" + [[ "${body}" == *:* && "${body%%:*}" =~ ^[A-Za-z_][A-Za-z0-9_-]*$ ]] || + fail "${source} line $((i + 1)) is not an input this parser can read \ +inside the ${action} step" + YAML_STEP_INPUT_KEYS+=("${body%%:*}") + YAML_STEP_INPUT_VALUES+=("${body#*:}") + done +} + +# The still-raw value the placed step passes for an input, or non-zero when it +# passes none — which is a different thing from passing an empty one, and the +# callers below tell the two apart. +yaml_step_input() { + local key="$1" i + for ((i = 0; i < ${#YAML_STEP_INPUT_KEYS[@]}; i++)); do + [[ "${YAML_STEP_INPUT_KEYS[i]}" == "${key}" ]] || continue + printf '%s' "${YAML_STEP_INPUT_VALUES[i]}" + return 0 + done + return 1 +} + +# The line range of one job in the workflow currently indexed, so a step search +# can be scoped to it: a workflow runs the same action in several jobs, and +# only one of them is the job a claim of parity names. +YAML_JOB_START=-1 +YAML_JOB_END=-1 +yaml_locate_job() { + local source="$1" job="$2" i jobs_line=-1 jobs_end job_indent=-1 + YAML_JOB_START=-1 + YAML_JOB_END=-1 + + for ((i = 0; i < ${#YAML_BODIES[@]}; i++)); do + ((YAML_INDENTS[i] == 0)) || continue + [[ "${YAML_BODIES[i]}" == 'jobs:' ]] || continue + jobs_line="${i}" + break + done + ((jobs_line >= 0)) || + fail "${source} declares no jobs this parser can read" + + jobs_end="$(yaml_block_end "$((jobs_line + 1))" 1)" + for ((i = jobs_line + 1; i < jobs_end; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + ((job_indent < 0)) && job_indent="${YAML_INDENTS[i]}" + ((YAML_INDENTS[i] == job_indent)) || continue + [[ "${YAML_BODIES[i]}" == "${job}:" ]] || continue + YAML_JOB_START="${i}" + YAML_JOB_END="$(yaml_block_end "$((i + 1))" "$((job_indent + 1))")" + return 0 + done + + fail "${source} has no ${job} job; the values this script reads out of that \ +job have nowhere left to come from" +} + +# The CI job the contracts stage reproduces, and the rehearsal job that has to +# run it on the same toolchain. Naming a job is a claim about what a stage's +# evidence is evidence of, and the release is entitled to have that claim +# checked rather than restated. +CONTRACTS_WORKFLOW=".github/workflows/contracts-ecdsa.yml" +CONTRACTS_JOB="contracts-build-and-test" +SOLIDITY_PROOFS_JOB="solidity-proofs" +SETUP_NODE_ACTION="actions/setup-node" + +# Read by resolve_setup_node_version: the exact Node release a job pins. +SETUP_NODE_VERSION="" + +# The Node release one workflow job pins, read out of that job's setup-node +# step. The contracts stage claims to reproduce a named CI job, and a claim of +# parity restated as a constant beside the claim stops being a claim about +# anything the moment the job moves: the stage would go on producing green +# evidence whose log says it ran what CI runs while running something else. +resolve_setup_node_version() { + local workflow="$1" job="$2" content raw unmodelled version + SETUP_NODE_VERSION="" + + content="$(git -C "${REPO_ROOT}" show "HEAD:${workflow}" 2>/dev/null)" || + fail "the commit under test carries no ${workflow}; the toolchain this \ +scaffold reproduces is pinned there, and this script has nothing left to read \ +it from" + + yaml_index_lines "${workflow}" "${content}" + yaml_locate_job "${workflow}" "${job}" + yaml_locate_action_step "${workflow}" "${SETUP_NODE_ACTION}" \ + "${YAML_JOB_START}" "${YAML_JOB_END}" + + raw="$(yaml_step_input node-version)" || + fail "the ${SETUP_NODE_ACTION} step in ${workflow}'s ${job} job pins no \ +node-version, so it takes whatever the runner image ships; evidence from a \ +toolchain nobody named is not that job's evidence" + raw="${raw#"${raw%%[![:space:]]*}"}" + raw="${raw%"${raw##*[![:space:]]}"}" + + if unmodelled="$(yaml_unmodelled_value "${raw}")"; then + fail "the ${SETUP_NODE_ACTION} step in ${workflow}'s ${job} job writes its \ +node-version as ${unmodelled}, which this parser does not resolve" + fi + version="$(yaml_scalar_value "${raw}")" || + fail "the ${SETUP_NODE_ACTION} step in ${workflow}'s ${job} job quotes its \ +node-version in a form this parser does not read" + + # A range or a major line lets the runner choose the release, and the + # contracts build is pinned precisely because one it chose broke compile + # artifacts. Reproducing "whatever 18.x resolved to today" reproduces + # nothing. + [[ "${version}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || + fail "the ${SETUP_NODE_ACTION} step in ${workflow}'s ${job} job pins \ +node-version [${version}], which is not one exact release; the contracts \ +build is pinned exactly because a release the runner chose broke its compile \ +artifacts" + + SETUP_NODE_VERSION="${version}" +} + +# Both halves of the contracts stage's parity claim held to the job it names: +# the dispatch that provisions the toolchain, and — through the stage itself — +# the interpreter the proofs actually run on. Checked here, in the gate that +# runs on every change to either workflow, so a bump in CI is caught by a lint +# rather than by a dispatch that blocks on the wrong version. +verify_contracts_toolchain_pin() { + local ci_version + resolve_setup_node_version "${CONTRACTS_WORKFLOW}" "${CONTRACTS_JOB}" + ci_version="${SETUP_NODE_VERSION}" + + resolve_setup_node_version "${REHEARSAL_WORKFLOW}" "${SOLIDITY_PROOFS_JOB}" + [[ "${SETUP_NODE_VERSION}" == "${ci_version}" ]] || + fail "${REHEARSAL_WORKFLOW}'s ${SOLIDITY_PROOFS_JOB} job provisions Node \ +${SETUP_NODE_VERSION} while ${CONTRACTS_WORKFLOW}'s ${CONTRACTS_JOB} job pins \ +${ci_version}; the contracts stage reproduces that job, and evidence produced \ +on another toolchain is not its evidence" + + note "contracts toolchain: ${CONTRACTS_WORKFLOW}'s ${CONTRACTS_JOB} job and \ +${REHEARSAL_WORKFLOW}'s ${SOLIDITY_PROOFS_JOB} job both pin Node ${ci_version}" +} + +# The workflow that builds the artifact a cutover record binds its identity to. +RELEASE_WORKFLOW=".github/workflows/release.yml" +RELEASE_BUILD_JOB="build-and-release" +RELEASE_PUBLISH_JOB="publish-docker-images" +RELEASE_GITHUB_RELEASE_ACTION="softprops/action-gh-release" +RELEASE_DOCKER_TAG_STEP="docker-tags" +RELEASE_DOCKER_TAG_SELECTOR="scripts/release/pr4109/release-docker-tags.sh" +RELEASE_TRIGGER_TAG_RESOLVER="scripts/release/pr4109/release-trigger-tag.sh" + +# The released artifact must name its source commit exactly. +# +# capture_r1_release_identity requires every R1 node's reported revision to +# equal the commit this run is bound to, and that requirement is only +# satisfiable while the workflow that builds the artifact stamps the whole SHA +# into it. An abbreviation names a commit only as far as it goes, and the +# record would bind a rehearsal's every observation to a prefix. So the stamp +# is read out of the release workflow rather than assumed: a bump back to +# `--short` is caught by this lint, on the commit that makes it, instead of by +# a rehearsal that refuses every artifact the release pipeline can produce. +verify_release_revision_stamp() { + local content stamps abbreviated + content="$(git -C "${REPO_ROOT}" show "HEAD:${RELEASE_WORKFLOW}" \ + 2>/dev/null)" || + fail "the commit under test carries no ${RELEASE_WORKFLOW}; the source \ +stamp every cutover record binds its artifact identity to is written there, \ +and this script has nothing left to read it from" + + # Every assignment of the revision the build is stamped with, whatever job + # or step it sits in: a release building two images from two jobs stamps it + # twice, and one of them reverting is the whole failure this catches. + stamps="$(printf '%s\n' "${content}" | + { grep -nE '(^|[^[:alnum:]_])revision=\$\(' || true; })" + if [[ -z "${stamps}" ]]; then + fail "${RELEASE_WORKFLOW} assigns no revision from a command; the \ +artifact identity a cutover record is measured against comes from that \ +assignment, and a release that stopped making it stamps nothing this scaffold \ +can bind to" + fi + + abbreviated="$(printf '%s\n' "${stamps}" | + { grep -vE 'git rev-parse HEAD\)' || true; })" + if [[ -n "${abbreviated}" ]]; then + printf '%s\n' "${abbreviated}" >&2 + fail "${RELEASE_WORKFLOW} stamps the released artifact with a revision \ +this scaffold cannot bind to (lines above); every assignment must be \ +\$(git rev-parse HEAD), because a rehearsal record names one commit and an \ +abbreviation is not that commit" + fi + + note "release stamp: ${RELEASE_WORKFLOW} writes the full source SHA into \ +every artifact it builds ($(printf '%s\n' "${stamps}" | wc -l | tr -d ' ') \ +assignment(s))" +} + +# Verify one release job derives its version only from the exact tag ref that +# triggered the workflow. A commit may carry both a stable tag and a release- +# candidate tag, so repository tag discovery cannot identify the release the +# runner is processing. Both jobs must resolve the same GitHub event fields +# through the fail-closed helper and publish that exact result as `version`. +verify_release_job_trigger_identity() { + local content="$1" job="$2" i body + local trigger_ref_bindings=0 trigger_tag_bindings=0 + local resolver_invocations=0 version_exports=0 version_env_writes=0 + local expected_ref_binding="RELEASE_TRIGGER_REF: \${{ github.ref }}" + local expected_tag_binding="RELEASE_TRIGGER_TAG: \${{ github.ref_name }}" + local expected_version_export="echo \"version=\${version}\" >> \"\${GITHUB_ENV}\"" + local expected_image_label="version=\${{ env.version }}" + local resolver_assignment="version=\"\$(" + local trigger_ref_argument="\"\${RELEASE_TRIGGER_REF}\"" + local trigger_tag_argument="\"\${RELEASE_TRIGGER_TAG}\"" + + yaml_index_lines "${RELEASE_WORKFLOW}" "${content}" + yaml_locate_job "${RELEASE_WORKFLOW}" "${job}" + + for ((i = YAML_JOB_START; i < YAML_JOB_END; i++)); do + body="${YAML_BODIES[i]}" + + [[ "${body}" == "${expected_ref_binding}" ]] && + trigger_ref_bindings=$((trigger_ref_bindings + 1)) + [[ "${body}" == "${expected_tag_binding}" ]] && + trigger_tag_bindings=$((trigger_tag_bindings + 1)) + + if [[ "${body}" == *"version="* && "${body}" == *'git describe'* ]]; then + fail "${RELEASE_WORKFLOW}'s ${job} job derives the release version with \ +git describe on line $((i + 1)); a commit can carry stable and prerelease \ +tags simultaneously, so the release identity must come from the exact \ +triggering tag" + fi + + if [[ "${body}" == *'version='* && "${body}" == *'GITHUB_ENV'* ]]; then + version_env_writes=$((version_env_writes + 1)) + if [[ "${body}" == "${expected_version_export}" ]]; then + version_exports=$((version_exports + 1)) + else + fail "${RELEASE_WORKFLOW}'s ${job} job writes a release version to \ +GITHUB_ENV outside the exact triggering-tag export on line $((i + 1)); a \ +later write can replace the identity the resolver proved" + fi + fi + + if [[ "${body}" == version:* ]]; then + fail "${RELEASE_WORKFLOW}'s ${job} job declares a separate version \ +environment value on line $((i + 1)); the triggering-tag resolver must be the \ +only release-identity source" + fi + if [[ "${body}" == version=* && + "${body}" != *"./${RELEASE_TRIGGER_TAG_RESOLVER}"* && + "${body}" != "${expected_image_label}" ]]; then + fail "${RELEASE_WORKFLOW}'s ${job} job reassigns version outside the \ +triggering-tag resolver on line $((i + 1)); the selector and release action \ +must consume the identity the resolver returned" + fi + + if [[ "${body}" == *"./${RELEASE_TRIGGER_TAG_RESOLVER}"* ]]; then + resolver_invocations=$((resolver_invocations + 1)) + if [[ "${body}" != *"${resolver_assignment}"* || + "${body}" != *"${trigger_ref_argument}"* || + "${body}" != *"${trigger_tag_argument}"* ]]; then + fail "${RELEASE_WORKFLOW}'s ${job} job does not assign version from \ +${RELEASE_TRIGGER_TAG_RESOLVER} with both exact triggering-ref inputs on line \ +$((i + 1))" + fi + fi + done + + ((trigger_ref_bindings == 1)) || + fail "${RELEASE_WORKFLOW}'s ${job} job binds RELEASE_TRIGGER_REF to \ +github.ref [${trigger_ref_bindings}] times; exactly one binding is required" + ((trigger_tag_bindings == 1)) || + fail "${RELEASE_WORKFLOW}'s ${job} job binds RELEASE_TRIGGER_TAG to \ +github.ref_name [${trigger_tag_bindings}] times; exactly one binding is \ +required" + ((resolver_invocations == 1)) || + fail "${RELEASE_WORKFLOW}'s ${job} job invokes \ +${RELEASE_TRIGGER_TAG_RESOLVER} [${resolver_invocations}] times; exactly one \ +invocation must derive its release identity" + ((version_exports == 1)) || + fail "${RELEASE_WORKFLOW}'s ${job} job exports the resolved triggering tag \ +as version [${version_exports}] times; exactly one GITHUB_ENV export is \ +required" + ((version_env_writes == 1)) || + fail "${RELEASE_WORKFLOW}'s ${job} job writes version to GITHUB_ENV \ +[${version_env_writes}] times; exactly one triggering-tag export is required" +} + +# A prerelease image must be available by its versioned tag without moving the +# mutable aliases operators use for stable production releases. The release +# workflow therefore derives one identity from the triggering tag in both +# release jobs and delegates its complete Docker tag set to one tested +# selector: exact vMAJOR.MINOR.PATCH tags receive the stable aliases and every +# other accepted Docker tag receives only its versioned name. The GitHub +# release prerelease decision must consume that same identity. +verify_release_candidate_tag_isolation() { + local content i body selector_steps selector_invocations + local raw_tags expected_tags raw_prerelease expected_prerelease + local version_argument="\"\${version}\"" + content="$(git -C "${REPO_ROOT}" show "HEAD:${RELEASE_WORKFLOW}" \ + 2>/dev/null)" || + fail "the commit under test carries no ${RELEASE_WORKFLOW}; there is no \ +release publication path to verify for prerelease alias isolation" + + verify_release_job_trigger_identity "${content}" "${RELEASE_BUILD_JOB}" + verify_release_job_trigger_identity "${content}" "${RELEASE_PUBLISH_JOB}" + + yaml_index_lines "${RELEASE_WORKFLOW}" "${content}" + yaml_locate_job "${RELEASE_WORKFLOW}" "${RELEASE_PUBLISH_JOB}" + + selector_steps=0 + selector_invocations=0 + for ((i = YAML_JOB_START; i < YAML_JOB_END; i++)); do + body="${YAML_BODIES[i]}" + [[ "${body}" == "id: ${RELEASE_DOCKER_TAG_STEP}" ]] && + selector_steps=$((selector_steps + 1)) + + if [[ "${body}" == *"./${RELEASE_DOCKER_TAG_SELECTOR}"* ]]; then + selector_invocations=$((selector_invocations + 1)) + [[ "${body}" == *"${version_argument}"* ]] || + fail "${RELEASE_WORKFLOW}'s ${RELEASE_PUBLISH_JOB} job does not pass \ +the exact triggering-tag version to ${RELEASE_DOCKER_TAG_SELECTOR} on line \ +$((i + 1))" + fi + + if [[ "${body}" == *':latest'* || "${body}" == *':mainnet'* ]]; then + fail "${RELEASE_WORKFLOW}'s ${RELEASE_PUBLISH_JOB} job hard-codes a \ +mutable Docker alias on line $((i + 1)); all tags must come from \ +${RELEASE_DOCKER_TAG_SELECTOR}, whose stable-release test keeps prereleases \ +off latest and mainnet" + fi + done + + ((selector_steps == 1)) || + fail "${RELEASE_WORKFLOW}'s ${RELEASE_PUBLISH_JOB} job has \ +${selector_steps} steps with id ${RELEASE_DOCKER_TAG_STEP}; exactly one step \ +must resolve the complete tag set" + + ((selector_invocations == 1)) || + fail "${RELEASE_WORKFLOW}'s ${RELEASE_PUBLISH_JOB} job invokes \ +${RELEASE_DOCKER_TAG_SELECTOR} [${selector_invocations}] times; exactly one \ +invocation must decide the complete published tag set" + + yaml_body_carries "${YAML_JOB_START}" "${YAML_JOB_END}" 'GITHUB_OUTPUT' || + fail "${RELEASE_WORKFLOW}'s ${RELEASE_PUBLISH_JOB} job does not write the \ +selected Docker tags to GITHUB_OUTPUT" + + yaml_locate_action_step "${RELEASE_WORKFLOW}" "${BUILD_ACTION}" \ + "${YAML_JOB_START}" "${YAML_JOB_END}" + raw_tags="$(yaml_step_input tags)" || + fail "the ${BUILD_ACTION} step in ${RELEASE_WORKFLOW}'s \ +${RELEASE_PUBLISH_JOB} job publishes no tag set" + raw_tags="${raw_tags#"${raw_tags%%[![:space:]]*}"}" + raw_tags="${raw_tags%"${raw_tags##*[![:space:]]}"}" + + # The expression is workflow syntax, not a shell expansion. + # shellcheck disable=SC2016 + expected_tags='${{ steps.docker-tags.outputs.tags }}' + [[ "${raw_tags}" == "${expected_tags}" ]] || + fail "the ${BUILD_ACTION} step in ${RELEASE_WORKFLOW}'s \ +${RELEASE_PUBLISH_JOB} job publishes [${raw_tags}] instead of the complete \ +output of ${RELEASE_DOCKER_TAG_STEP}; another tag source could move stable \ +aliases during a prerelease" + + yaml_index_lines "${RELEASE_WORKFLOW}" "${content}" + yaml_locate_job "${RELEASE_WORKFLOW}" "${RELEASE_BUILD_JOB}" + yaml_locate_action_step "${RELEASE_WORKFLOW}" \ + "${RELEASE_GITHUB_RELEASE_ACTION}" "${YAML_JOB_START}" "${YAML_JOB_END}" + raw_prerelease="$(yaml_step_input prerelease)" || + fail "the ${RELEASE_GITHUB_RELEASE_ACTION} step in ${RELEASE_WORKFLOW}'s \ +${RELEASE_BUILD_JOB} job makes no prerelease decision" + raw_prerelease="${raw_prerelease#"${raw_prerelease%%[![:space:]]*}"}" + raw_prerelease="${raw_prerelease%"${raw_prerelease##*[![:space:]]}"}" + + # The expression is workflow syntax, not a shell expansion. + # shellcheck disable=SC2016 + expected_prerelease="\${{ contains(env.version, '-') }}" + [[ "${raw_prerelease}" == "${expected_prerelease}" ]] || + fail "the ${RELEASE_GITHUB_RELEASE_ACTION} step in ${RELEASE_WORKFLOW}'s \ +${RELEASE_BUILD_JOB} job derives prerelease from [${raw_prerelease}] instead \ +of the exact triggering-tag identity [${expected_prerelease}]" + + note "release tags: both release jobs derive identity from the exact \ +triggering tag; ${RELEASE_WORKFLOW} publishes exactly the tag set from \ +${RELEASE_DOCKER_TAG_SELECTOR}, and prereleases remain version-only" +} + +# The dispatch input a release run hands the detached provenance in, the +# variable attest_release_provenance reads it from, and the member it writes +# into the receipt. The producer and the acceptance consumer are what define +# these three; they are named here so the check below can hold the dispatched +# workflow to them. +RELEASE_PROVENANCE_INPUT="release_provenance_b64" +RELEASE_PROVENANCE_ENV="PR4109_RELEASE_PROVENANCE" +RELEASE_PROVENANCE_MEMBER="release-provenance.json" + +# The job that produces the receipt, and the job that is judged by it. +REHEARSAL_PROOF_JOB="local-proofs" +REHEARSAL_CONTAINER_JOB="container-rehearsal" + +# Does any content line in [from, to) carry the given text? Comments and blank +# lines are blanked by the indexer, so a check written on this can never be +# satisfied by a line the workflow parser does not read — which matters here, +# where every needle is a name a comment would naturally mention. +yaml_body_carries() { + local from="$1" to="$2" needle="$3" i + for ((i = from; i < to; i++)); do + if [[ "${YAML_BODIES[i]}" == *"${needle}"* ]]; then + return 0 + fi + done + return 1 +} + +# The dispatched rehearsal must be able to supply the half of the release +# identity the reviewed manifest cannot state about itself. +# +# A reviewed manifest names the cutover. It cannot name the commit finally +# built or the immutable image digests, because those are outputs of a build +# over its own bytes, so acceptance requires them from a detached document +# instead — and refuses, unconditionally, any release-ready receipt that +# carries none. Every one of those requirements lives in this script, and none +# of them is satisfiable by a dispatch with no way to hand the document in. +# +# That is a failure with no local symptom: every proof passes, the receipt is +# archived, and the refusal arrives only on the one dispatch that matters, at +# the end of a rehearsal that has already run every mandatory step. So the +# wiring is read out of the workflow here, on the commit that changes it, +# rather than discovered by the release it would block. +verify_release_provenance_wiring() { + local content i + content="$(git -C "${REPO_ROOT}" show "HEAD:${REHEARSAL_WORKFLOW}" \ + 2>/dev/null)" || + fail "the commit under test carries no ${REHEARSAL_WORKFLOW}; the \ +dispatch that supplies the detached release provenance is declared there, and \ +this script has nothing left to read it from" + + yaml_index_lines "${REHEARSAL_WORKFLOW}" "${content}" + local total="${#YAML_BODIES[@]}" + + # A mapping key of its own, matched whole: a step body mentioning the input + # is a use of it and not a declaration, and only the declaration puts the + # field in front of whoever dispatches the release. + local declared="" + for ((i = 0; i < total; i++)); do + if [[ "${YAML_BODIES[i]}" == "${RELEASE_PROVENANCE_INPUT}:" ]]; then + declared="yes" + break + fi + done + [[ -n "${declared}" ]] || + fail "${REHEARSAL_WORKFLOW} declares no ${RELEASE_PROVENANCE_INPUT} \ +input; acceptance refuses every record measured against a release-ready \ +manifest whose receipt names no artifact, so a dispatch with nowhere to put \ +the detached provenance cannot produce admissible release evidence" + + yaml_locate_job "${REHEARSAL_WORKFLOW}" "${REHEARSAL_PROOF_JOB}" + local proof_start="${YAML_JOB_START}" proof_end="${YAML_JOB_END}" + + yaml_body_carries "${proof_start}" "${proof_end}" \ + "inputs.${RELEASE_PROVENANCE_INPUT}" || + fail "${REHEARSAL_WORKFLOW}'s ${REHEARSAL_PROOF_JOB} job never reads the \ +${RELEASE_PROVENANCE_INPUT} input; a declared input nothing consumes is a \ +field an operator fills in and a release nobody can identify" + + yaml_body_carries "${proof_start}" "${proof_end}" \ + "${RELEASE_PROVENANCE_ENV}" || + fail "${REHEARSAL_WORKFLOW}'s ${REHEARSAL_PROOF_JOB} job never passes \ +${RELEASE_PROVENANCE_ENV} to the stage that writes the receipt; the producer \ +reads the document from that variable and records none without it, whatever \ +the dispatch supplied" + + yaml_body_carries "${proof_start}" "${proof_end}" \ + "${RELEASE_PROVENANCE_MEMBER}" || + fail "${REHEARSAL_WORKFLOW}'s ${REHEARSAL_PROOF_JOB} job never requires \ +${RELEASE_PROVENANCE_MEMBER} of the receipt it archives; a supplied document \ +that did not reach the receipt leaves the archive unable to admit container \ +evidence, and a green job saying otherwise" + + yaml_locate_job "${REHEARSAL_WORKFLOW}" "${REHEARSAL_CONTAINER_JOB}" + yaml_body_carries "${YAML_JOB_START}" "${YAML_JOB_END}" \ + "${RELEASE_PROVENANCE_MEMBER}" || + fail "${REHEARSAL_WORKFLOW}'s ${REHEARSAL_CONTAINER_JOB} job never checks \ +the receipt it downloads for ${RELEASE_PROVENANCE_MEMBER}; without it a \ +rehearsal drives a fleet through every mandatory step before its own emitter \ +refuses the run for an input the dispatch could have been given" + + note "release provenance: ${REHEARSAL_WORKFLOW} offers \ +${RELEASE_PROVENANCE_INPUT}, hands it to the proof stage as \ +${RELEASE_PROVENANCE_ENV}, and requires ${RELEASE_PROVENANCE_MEMBER} of the \ +receipt in both the producing and the consuming job" +} + +# The Dockerfile the rehearsal dispatch compiles and the context root it +# compiles from, read out of the workflow that does the building rather than +# restated here. The pair decides which ignore file the build applies, so a +# constant restating it goes stale the moment the build step changes — +# silently, and in the direction where this script keeps checking itself +# against rules the build has stopped reading. +# +# The workflow is read from the commit under test, like the ignore rules +# themselves. Every step shape this parser does not model is refused by name: +# resolving a real build's Dockerfile on a guess is how the whole classification +# below ends up measured against the wrong file. +resolve_build_step_identity() { + BUILD_CONTEXT="" + BUILD_DOCKERFILE="" + + local content + content="$(git -C "${REPO_ROOT}" show "HEAD:${REHEARSAL_WORKFLOW}" \ + 2>/dev/null)" || + fail "the commit under test carries no ${REHEARSAL_WORKFLOW}; that \ +workflow's build step is what decides which Dockerfile the proof image is \ +compiled from, and so which ignore rules the build-context classification in \ +this script has to be checked against" + + yaml_index_lines "${REHEARSAL_WORKFLOW}" "${content}" + yaml_locate_action_step "${REHEARSAL_WORKFLOW}" "${BUILD_ACTION}" 0 \ + "${#YAML_BODIES[@]}" + + local raw_context raw_file seen_context=1 seen_file=1 unmodelled + raw_context="$(yaml_step_input context)" || seen_context=0 + raw_file="$(yaml_step_input file)" || seen_file=0 + + # An unset `context` is the action's Git context — a build of the repository + # URL, not of this checkout — under which nothing the classification below + # says about a tracked path holds. + ((seen_context == 1)) || + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} sets no context, \ +so it builds the default Git context rather than the dispatched checkout; the \ +build-context classification in this script describes the checkout's tree" + + raw_context="${raw_context#"${raw_context%%[![:space:]]*}"}" + raw_context="${raw_context%"${raw_context##*[![:space:]]}"}" + if unmodelled="$(yaml_unmodelled_value "${raw_context}")"; then + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} writes its context \ +as ${unmodelled}, which this parser does not resolve; the build-context \ +classification below would be checked against a guess" + fi + local build_context + build_context="$(yaml_scalar_value "${raw_context}")" || + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} quotes its context \ +in a form this parser does not read" + build_context="$(dockerignore_clean_path "${build_context}")" + [[ "${build_context}" == '.' ]] || + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} builds from \ +context [${build_context}], but the build-context classification in this \ +script is written over repository-relative paths and holds only for a context \ +rooted at the repository; re-derive it before this scaffold admits any \ +further evidence" + + # buildx defaults `file` to /Dockerfile, and resolves a given one + # against the working directory — the same directory the context is rooted + # at, which is what makes the two readings agree here at all. + local build_dockerfile="Dockerfile" + if ((seen_file == 1)); then + raw_file="${raw_file#"${raw_file%%[![:space:]]*}"}" + raw_file="${raw_file%"${raw_file##*[![:space:]]}"}" + if unmodelled="$(yaml_unmodelled_value "${raw_file}")"; then + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} writes its \ +Dockerfile as ${unmodelled}, which this parser does not resolve; the ignore \ +rules the classification below is checked against are selected by that name" + fi + build_dockerfile="$(yaml_scalar_value "${raw_file}")" || + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} quotes its \ +Dockerfile in a form this parser does not read" + build_dockerfile="$(dockerignore_clean_path "${build_dockerfile}")" + [[ "${build_dockerfile}" == /* || "${build_dockerfile}" == '.' || + "${build_dockerfile}" == '..' || "${build_dockerfile}" == '../'* ]] && + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} builds \ +Dockerfile [${build_dockerfile}], which does not resolve to a path inside the \ +build context; this script cannot name the ignore file that selects" + fi + + git -C "${REPO_ROOT}" cat-file -e "HEAD:${build_dockerfile}" 2>/dev/null || + fail "the ${BUILD_ACTION} step in ${REHEARSAL_WORKFLOW} builds Dockerfile \ +[${build_dockerfile}], which the commit under test does not carry" + + BUILD_CONTEXT="${build_context}" + BUILD_DOCKERFILE="${build_dockerfile}" + note "build step: ${REHEARSAL_WORKFLOW} compiles ${BUILD_DOCKERFILE} from \ +context ${BUILD_CONTEXT}" +} + +# The build inputs the ignore-file selection above depends on decide what this +# scaffold accepts as evidence just as directly as its own code does, and the +# gate holding the two together only ever runs on the events and paths its own +# workflow names. So both are held to the resolved identity: a build step moved +# to another Dockerfile takes its ignore file with it, and a filter list left +# behind would leave every later change to that file ungated — the mirror check +# would keep passing, on a file nobody was told had changed. +# +# A trigger carrying no filter at all runs on every change and so covers +# everything; what this refuses is a gate that some class of change can get +# past — reachable only by remembering to dispatch it, restricted away from +# the merges it exists to hold, or listing an input it later negates again. +verify_scaffold_lint_path_filters() { + local content + content="$(git -C "${REPO_ROOT}" show "HEAD:${SCAFFOLD_LINT_WORKFLOW}" \ + 2>/dev/null)" || + fail "the commit under test carries no ${SCAFFOLD_LINT_WORKFLOW}; nothing \ +holds the build-context classification in this script to the build inputs it \ +mirrors" + + yaml_index_lines "${SCAFFOLD_LINT_WORKFLOW}" "${content}" + + load_lint_required_inputs + LINT_FILTER_MISSING="" + + local i on_line=-1 + for ((i = 0; i < ${#YAML_BODIES[@]}; i++)); do + ((YAML_INDENTS[i] == 0)) || continue + [[ "${YAML_BODIES[i]}" == 'on:' ]] || continue + on_line="${i}" + break + done + ((on_line >= 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} declares no triggers this parser can read, \ +so nothing says when the gate holding this script to the build inputs runs" + + local on_end trigger_indent=-1 covered=0 merges=0 + on_end="$(yaml_block_end "$((on_line + 1))" 1)" + for ((i = on_line + 1; i < on_end; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + ((trigger_indent < 0)) && trigger_indent="${YAML_INDENTS[i]}" + ((YAML_INDENTS[i] == trigger_indent)) || continue + case "${YAML_BODIES[i]}" in + 'push:' | 'pull_request:') + verify_lint_trigger_filters "${i}" "${trigger_indent}" + covered=$((covered + 1)) + [[ "${YAML_BODIES[i]}" == 'pull_request:' ]] && merges=1 + ;; + esac + done + + # A push trigger is not the merge gate: it fires after the branch already + # moved, and on a repository that merges by pull request it never fires on + # the release branch at all. Only a pull_request trigger can stop a change + # to these inputs from landing unchecked, so its absence is refused however + # many other events the workflow names. + ((merges > 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} runs on no pull request, so a change to \ +the build inputs the classification in this script mirrors can merge without \ +the gate that holds the two together ever having run" + + if [[ -n "${LINT_FILTER_MISSING}" ]]; then + printf '%s' "${LINT_FILTER_MISSING}" >&2 + fail "${SCAFFOLD_LINT_WORKFLOW} no longer runs on every input this \ +scaffold's trust model is derived from (listing above); a change to an \ +uncovered one would retire rules this scaffold never rechecks" + fi + + note "scaffold lint: ${SCAFFOLD_LINT_WORKFLOW} runs on every change to the \ +${#LINT_REQUIRED_INPUTS[@]} tracked input(s) this scaffold's trust model is \ +derived from, on all ${covered} push/pull-request trigger(s)" +} + +# The inputs a filter list has to cover and the ones a run found uncovered. +# Globals rather than arguments because the check below appends to the second +# from inside a loop, and reports every uncovered input at once rather than +# failing on the first: a filter list left behind by a moved build step is +# usually missing more than one entry, and the listing is what the fix needs. +LINT_REQUIRED_INPUTS=() +LINT_FILTER_MISSING="" + +# Every path a change to which can move what this scaffold accepts, read out +# of the commit under test rather than listed by hand — a hand-kept list is +# trusted by inspection, and the whole point of this gate is that nothing +# here is. +# +# Four classes, each one something this script really reads: +# +# the three workflows one names the build step every classification +# below is resolved from, one is this gate itself, +# and one pins the toolchain the contracts stage +# claims to reproduce +# the build's ignore rules the resolved Dockerfile, the ignore file its name +# selects, and the root .dockerignore that applies +# only while no such file exists — the last two are +# required whether or not the commit carries them, +# because adding one retires the other's every rule +# the scaffold itself the checkers deciding what may be accepted as +# release evidence, all of them, not just the ones +# written in shell +# ignore and build rules every committed .gitignore, root and nested, +# because build-image mode classifies untracked +# paths under the restored ones; and every +# committed Makefile, because the regeneration the +# gen/ classification models is what they run +load_lint_required_inputs() { + local path + LINT_REQUIRED_INPUTS=() + while IFS= read -r path; do + [[ -n "${path}" ]] && LINT_REQUIRED_INPUTS+=("${path}") + done < <( + { + printf '%s\n' \ + "${REHEARSAL_WORKFLOW}" \ + "${SCAFFOLD_LINT_WORKFLOW}" \ + "${CONTRACTS_WORKFLOW}" \ + "${RELEASE_WORKFLOW}" \ + "${BUILD_DOCKERFILE}" \ + "${BUILD_DOCKERFILE}.dockerignore" \ + '.dockerignore' + git -C "${REPO_ROOT}" ls-tree -r --name-only HEAD | + { + grep -E "^${SCAFFOLD_DIR}/|(^|/)\.gitignore$|(^|/)Makefile$" || true + } + } | sort -u + ) +} + +# GitHub's filter-pattern grammar, which is not the glob grammar the build's +# ignore rules are written in: `*` stops at a separator and `**` does not, and +# a leading `!` is handled by the caller because it negates the patterns +# before it rather than anything inside its own. +lint_pattern_regex() { + local pattern="$1" out="^" i ch + for ((i = 0; i < ${#pattern}; i++)); do + ch="${pattern:i:1}" + if [[ "${ch}" == '*' ]]; then + if [[ "${pattern:i+1:1}" == '*' ]]; then + i=$((i + 1)) + out+='.*' + else + out+='[^/]*' + fi + elif [[ '.(){}|^$' == *"${ch}"* ]]; then + out+="\\${ch}" + else + out+="${ch}" + fi + done + printf '%s$' "${out}" +} + +# The characters this grammar gives a meaning that reading them literally +# would get wrong, and that this script has no translation for. `?` and `+` +# quantify the character before them here rather than standing for one of any +# character — the reading the build's ignore rules would give them — so a +# required path measured against either would be measured wrong. +lint_pattern_unmodelled_construct() { + local pattern="$1" i ch + for ((i = 0; i < ${#pattern}; i++)); do + ch="${pattern:i:1}" + if [[ '?+[]' == *"${ch}"* || "${ch}" == $'\\' ]]; then + printf '%s' "${ch}" + return 0 + fi + done + return 1 +} + +# One trigger's compiled filter list, in the order it was written: the verdict +# a pattern carries when it matches (0 covers, 1 excludes again) travels beside +# it because order is what decides, and a later negation of an earlier listing +# is exactly the shape a coverage check reading membership cannot see. +LINT_FILTER_REGEX=() +LINT_FILTER_VERDICT=() + +# Whether the compiled list above runs on a change to one path. GitHub reads +# the whole list and lets the last matching entry decide, so this does too; a +# path no entry matches at all is not covered. +lint_filter_covers() { + local path="$1" i verdict=1 + for ((i = 0; i < ${#LINT_FILTER_REGEX[@]}; i++)); do + [[ "${path}" =~ ${LINT_FILTER_REGEX[i]} ]] || continue + verdict="${LINT_FILTER_VERDICT[i]}" + done + return "${verdict}" +} + +# One push or pull_request trigger: the events it really fires on, and the +# paths it really runs for. +verify_lint_trigger_filters() { + local line="$1" trigger_indent="$2" + local trigger="${YAML_BODIES[line]%:}" end key_indent=-1 + local i j body paths_line=-1 entry entries=0 bad negated + + end="$(yaml_block_end "$((line + 1))" "$((trigger_indent + 1))")" + for ((i = line + 1; i < end; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + ((key_indent < 0)) && key_indent="${YAML_INDENTS[i]}" + ((YAML_INDENTS[i] == key_indent)) || continue + body="${YAML_BODIES[i]}" + # An exclusion list says which changes are exempt rather than which are + # covered, so a trigger carrying one cannot be read as coverage at all. + [[ "${body}" == 'paths-ignore:'* ]] && + fail "${SCAFFOLD_LINT_WORKFLOW} filters its ${trigger} trigger with \ +paths-ignore, which this check cannot read as coverage of the build inputs the \ +classification in this script mirrors" + [[ "${body}" == 'paths:' ]] && paths_line="${i}" + if [[ "${trigger}" == 'pull_request' ]]; then + verify_lint_pull_request_reach "${i}" "${key_indent}" "${body}" + fi + done + + # No filter at all is the whole repository: every required input is covered. + ((paths_line >= 0)) || return 0 + + LINT_FILTER_REGEX=() + LINT_FILTER_VERDICT=() + end="$(yaml_block_end "$((paths_line + 1))" "$((key_indent + 1))")" + for ((j = paths_line + 1; j < end; j++)); do + ((YAML_INDENTS[j] < 0)) && continue + body="${YAML_BODIES[j]}" + [[ "${body}" == '-'[[:space:]]* ]] || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((j + 1)) is not a path filter \ +entry this parser can read" + entry="$(yaml_scalar_value "${body#-}")" || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((j + 1)) quotes its path filter \ +in a form this parser does not read" + negated=0 + if [[ "${entry}" == '!'* ]]; then + negated=1 + entry="${entry#!}" + fi + if bad="$(lint_pattern_unmodelled_construct "${entry}")"; then + fail "${SCAFFOLD_LINT_WORKFLOW} line $((j + 1)) filters on [${entry}], \ +whose [${bad}] this script has no reading for; a required input measured \ +against a guess would be reported covered on a guess" + fi + LINT_FILTER_REGEX+=("$(lint_pattern_regex "${entry}")") + LINT_FILTER_VERDICT+=("${negated}") + entries=$((entries + 1)) + done + + ((entries > 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} filters its ${trigger} trigger to an empty \ +path list, which no change matches; the gate holding this script to the build \ +inputs would never run" + + for entry in "${LINT_REQUIRED_INPUTS[@]}"; do + lint_filter_covers "${entry}" || + LINT_FILTER_MISSING+="${SCAFFOLD_LINT_WORKFLOW} line \ +$((paths_line + 1)): the ${trigger} filter list does not cover ${entry}"$'\n' + done +} + +# The lines a `run:` key hands to the shell, and the workflow line each of them +# sits on, or nothing when the line opens no `run:` at all. Both spellings +# these workflows use are read — the key on a sequence item's own line and the +# key opening one — because a step runs what its `run:` carries and nothing +# else: the same text in a step name, an `env:` value or a `with:` input names +# something, and a step that only names a command runs none of it. +# +# A shape whose lines are not what the shell receives is reported in +# YAML_RUN_UNMODELLED rather than read, with the lines still returned, so a +# caller can tell whether the shape nothing here reads is the one it was +# looking for. Refusing every folded scalar in the file would refuse steps this +# gate has no interest in. +YAML_RUN_LINES=() +YAML_RUN_LINENOS=() +YAML_RUN_UNMODELLED="" +yaml_run_lines() { + local index="$1" body key_indent raw value end i + YAML_RUN_LINES=() + YAML_RUN_LINENOS=() + YAML_RUN_UNMODELLED="" + + body="${YAML_BODIES[index]}" + if key_indent="$(yaml_item_key_indent "${index}")"; then + body="${body#-}" + body="${body#"${body%%[![:space:]]*}"}" + else + key_indent="${YAML_INDENTS[index]}" + fi + [[ "${body}" == 'run:'* ]] || return 1 + + raw="${body#run:}" + raw="${raw#"${raw%%[![:space:]]*}"}" + raw="${raw%"${raw##*[![:space:]]}"}" + + case "${raw}" in + '') return 1 ;; + # The literal block scalar, whose lines are the shell's lines. + '|' | '|-' | '|+') ;; + # A folded scalar joins its lines before the shell ever sees them, and an + # explicit indentation indicator moves where its content begins; either way + # what runs is not what these lines say. + '|'* | '>'*) + YAML_RUN_UNMODELLED="the block scalar header [${raw}]" + ;; + *) + # Kept as written when the quoting is one yaml_scalar_value refuses, so the + # invocation is still found in it and refused for the reason it really has + # rather than reported missing. + if value="$(yaml_scalar_value "${raw}")"; then + raw="${value}" + else + YAML_RUN_UNMODELLED="a quoted value needing escape processing to read" + fi + YAML_RUN_LINES=("${raw}") + YAML_RUN_LINENOS=("${index}") + return 0 + ;; + esac + + end="$(yaml_block_end "$((index + 1))" "$((key_indent + 1))")" + for ((i = index + 1; i < end; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + YAML_RUN_LINES+=("${YAML_BODIES[i]}") + YAML_RUN_LINENOS+=("${i}") + done +} + +# One entry per logical command in the lines above: a line ending in a +# backslash joins the one after it, which is the only shape here that spreads a +# command across lines. SHELL_COMMAND_LINES keeps the workflow line each +# command opened on, because that is the line a refusal has to name. +SHELL_COMMANDS=() +SHELL_COMMAND_LINES=() +shell_logical_commands() { + local i line acc="" start=-1 + local join=$'\\' joined=$'\\\\' + SHELL_COMMANDS=() + SHELL_COMMAND_LINES=() + for ((i = 0; i < ${#YAML_RUN_LINES[@]}; i++)); do + line="${YAML_RUN_LINES[i]}" + if ((start < 0)); then start="${YAML_RUN_LINENOS[i]}"; fi + if [[ "${line}" == *"${join}" && "${line}" != *"${joined}" ]]; then + acc+="${line%"${join}"} " + continue + fi + SHELL_COMMANDS+=("${acc}${line}") + SHELL_COMMAND_LINES+=("${start}") + acc="" + start=-1 + done + if [[ -n "${acc}" ]]; then + SHELL_COMMANDS+=("${acc}") + SHELL_COMMAND_LINES+=("${start}") + fi +} + +# The runner substitutes a workflow expression into this shell before the shell +# parses it, so a value carrying an operator writes a command nothing here ever +# saw. The contexts below are the runner's own — none of them can carry text +# from a pull request — and every other one is refused by name rather than read +# through, the way every other value this scaffold cannot model is. +SHELL_RUNNER_CONTEXTS="github.workspace github.repository github.sha \ +github.run_id github.run_number github.run_attempt runner.temp \ +runner.workspace runner.os runner.arch" + +SHELL_EXPANDED="" +SHELL_EXPRESSION_REFUSAL="" +# The expression opener and closer are the literal characters the workflow +# parser reads there, so they are deliberately never expanded here. +# shellcheck disable=SC2016 +shell_expand_expressions() { + local raw="$1" head rest context + SHELL_EXPANDED="" + SHELL_EXPRESSION_REFUSAL="" + while [[ "${raw}" == *'${{'* ]]; do + head="${raw%%'${{'*}" + rest="${raw#*'${{'}" + if [[ "${rest}" != *'}}'* ]]; then + SHELL_EXPRESSION_REFUSAL="an unterminated workflow expression" + return 1 + fi + context="${rest%%'}}'*}" + raw="${rest#*'}}'}" + context="${context#"${context%%[![:space:]]*}"}" + context="${context%"${context##*[![:space:]]}"}" + case " ${SHELL_RUNNER_CONTEXTS} " in + *" ${context} "*) ;; + *) + SHELL_EXPRESSION_REFUSAL="the workflow expression [${context}]" + return 1 + ;; + esac + # A value with no operator in it, so the command around it reads the same + # before and after the runner writes the real one in. + SHELL_EXPANDED+="${head}RUNNER_VALUE" + done + SHELL_EXPANDED+="${raw}" +} + +# The first thing in a command this parser has no reading for, named one by one +# the way dockerignore_unmodelled_construct names one. Every entry decides +# either whether the command runs or whose exit status the shell reports back, +# which are the only two questions asked of this body; reading past one of them +# would be answering both on a guess. +# +# Quoting is tracked because an operator inside quotes is not an operator, and +# a word-initial `#` outside them ends the command the way the shell ends it. +shell_unmodelled_construct() { + local cmd="$1" quote="" i ch next prev="" + for ((i = 0; i < ${#cmd}; i++)); do + ch="${cmd:i:1}" + next="${cmd:i+1:1}" + if [[ "${quote}" == "'" ]]; then + [[ "${ch}" == "'" ]] && quote="" + prev="${ch}" + continue + fi + if [[ "${ch}" == $'\\' ]]; then + i=$((i + 1)) + prev="" + continue + fi + if [[ "${ch}" == '`' ]] || [[ "${ch}" == '$' && "${next}" == '(' ]]; then + printf 'a command substitution' + return 0 + fi + if [[ "${quote}" == '"' ]]; then + [[ "${ch}" == '"' ]] && quote="" + prev="${ch}" + continue + fi + if [[ "${ch}" == '#' && -z "${prev}" ]]; then + return 1 + fi + case "${ch}" in + "'" | '"') quote="${ch}" ;; + '|') + if [[ "${next}" == '|' ]]; then + printf 'a conditional chain' + return 0 + fi + printf 'a pipeline' + return 0 + ;; + '&') + if [[ "${next}" == '&' ]]; then + printf 'a conditional chain' + return 0 + fi + printf 'a backgrounded command' + return 0 + ;; + ';') + printf 'a command list' + return 0 + ;; + '<' | '>') + printf 'a redirection' + return 0 + ;; + '(' | ')') + printf 'a subshell' + return 0 + ;; + esac + if [[ "${ch}" == [[:space:]] ]]; then prev=""; else prev="${ch}"; fi + done + if [[ -n "${quote}" ]]; then + printf 'an unterminated quote' + return 0 + fi + return 1 +} + +# A command's words, with the leading `NAME=value` assignments taken off into +# SHELL_ASSIGNMENTS and anything from a word-initial `#` onwards dropped. +# Placing the command word is the same problem for the invocation and for +# everything beside it, and both readings below start from it. The assignments +# are kept rather than discarded because they are part of what would run: they +# name the environment the command word resolves and executes under. +SHELL_WORDS=() +SHELL_ASSIGNMENTS=() +shell_command_words() { + local cmd="$1" word + local -a raw=() + SHELL_WORDS=() + SHELL_ASSIGNMENTS=() + read -ra raw <<<"${cmd}" + local i=0 + while ((i < ${#raw[@]})); do + [[ "${raw[i]}" =~ ^[A-Za-z_][A-Za-z_0-9]*= ]] || break + SHELL_ASSIGNMENTS+=("${raw[i]}") + i=$((i + 1)) + done + for ((; i < ${#raw[@]}; i++)); do + word="${raw[i]}" + [[ "${word}" == '#'* ]] && break + SHELL_WORDS+=("${word}") + done +} + +# The command words that decide something about the commands around them rather +# than doing work of their own: a compound statement's keywords, and the +# builtins that change what the shell does with the lines after them. One of +# these ahead of the invocation can stop it running — `set -n` reads the rest +# of the body without executing any of it — or replace the shell that would +# have run it, and neither leaves a mark on the step's exit status. +SHELL_COMPOUND_WORDS="if then elif else fi for while until do done case esac \ +select function coproc time in { } ! [" +SHELL_EXECUTION_WORDS="set shopt eval exec exit return source . trap" +shell_unmodelled_word() { + shell_command_words "$1" + ((${#SHELL_WORDS[@]} > 0)) || return 1 + case " ${SHELL_COMPOUND_WORDS} " in + *" ${SHELL_WORDS[0]} "*) + printf 'the compound-statement word [%s]' "${SHELL_WORDS[0]}" + return 0 + ;; + esac + case " ${SHELL_EXECUTION_WORDS} " in + *" ${SHELL_WORDS[0]} "*) + printf 'the shell builtin [%s]' "${SHELL_WORDS[0]}" + return 0 + ;; + esac + return 1 +} + +# The one command shape read as running the analysis: the entrypoint, the +# stage, nothing after it, and no `NAME=value` assignment ahead of it beyond +# the one name this entrypoint reads. The invocation as an argument to +# something else — an `echo`, a runner, a command substitution's subject — is a +# mention of the analysis rather than a run of it, and the exit status the step +# reports is that other command's. An assignment is the same substitution made +# without touching a character of the command: the words read here stay exactly +# as they are while the environment they run under decides whether bash +# executes a line of the file they name. +shell_invocation_shape() { + local entrypoint="${SCAFFOLD_DIR}/${SCAFFOLD_ENTRYPOINT}" word + shell_command_words "$1" + if ((${#SHELL_WORDS[@]} == 0)); then + printf 'it carries no command word at all' + return 0 + fi + + word="${SHELL_WORDS[0]//\"/}" + word="${word//\'/}" + case "${word}" in + "${entrypoint}" | */"${entrypoint}") ;; + *) + printf 'its command word is [%s]' "${SHELL_WORDS[0]}" + return 0 + ;; + esac + + if ((${#SHELL_WORDS[@]} < 2)); then + printf 'it names no stage to run' + return 0 + fi + word="${SHELL_WORDS[1]//\"/}" + word="${word//\'/}" + if [[ "${word}" != "${SCAFFOLD_LINT_STAGE}" ]]; then + printf 'its argument is [%s] rather than %s' \ + "${SHELL_WORDS[1]}" "${SCAFFOLD_LINT_STAGE}" + return 0 + fi + + if ((${#SHELL_WORDS[@]} > 2)); then + printf 'it carries the further argument [%s]' "${SHELL_WORDS[2]}" + return 0 + fi + + local name + for word in ${SHELL_ASSIGNMENTS[@]+"${SHELL_ASSIGNMENTS[@]}"}; do + name="${word%%=*}" + case " ${SCAFFOLD_LINT_ENV_NAMES} " in + *" ${name} "*) continue ;; + esac + printf 'it sets [%s] in the environment the entrypoint runs under' "${name}" + return 0 + done + return 1 +} + +# Triggers and filters say when the gate runs. They say nothing about what it +# runs, and a workflow firing on every change to every input while its job no +# longer invokes this script — or invokes it behind a condition, or with its +# failure declared survivable — is the same ungated state written a different +# way. So the invocation is placed and read. +# +# Placed in the only thing that runs anything, a step's `run:` body, and read +# there down to the shape of the command: exactly one of it, the last command +# of its body so that the step's exit status is the analysis's whatever the +# shell's error handling is set to, nothing around it that could condition it +# or swallow its status, and no condition on the step or the job holding it. +# +# What this cannot prove is that a run of that workflow happened, or that a run +# reporting success ran this file. Everything read here is the head commit's — +# the workflow, the steps around the invocation, the entrypoint itself — so a +# commit can drop the invocation along with this reading of it, and a commit +# whose job keeps the name a branch-protection rule requires can report success +# having run something else under it. A rule naming a job the head commit +# defines therefore holds the name, not the analysis. Nor is the reading below +# a closure within the one body it reads: the commands ahead of the invocation +# are read for the shell they open, so a `cp` over the entrypoint is accepted +# and the accepted final command runs the copy. What makes the absence of a run +# of *this* analysis block a merge is a ruleset requiring a workflow this +# repository does not supply: an entry naming the gate checked in here names a +# file every pull request here can rewrite, so the entry has to name an outside +# repository, pin it by commit SHA rather than by a branch or tag ref, and have +# that pinned source carry the analysis itself. That control and its current +# standing are recorded beside this scaffold rather than claimed here. +verify_scaffold_lint_runs_analysis() { + local content + content="$(git -C "${REPO_ROOT}" show "HEAD:${SCAFFOLD_LINT_WORKFLOW}" \ + 2>/dev/null)" || + fail "the commit under test carries no ${SCAFFOLD_LINT_WORKFLOW}; nothing \ +runs the analysis the evidence this scaffold admits rests on" + + yaml_index_lines "${SCAFFOLD_LINT_WORKFLOW}" "${content}" + + # The stage ends where a stage name can no longer continue, rather than at + # whitespace: an invocation the shell has wrapped in something — a + # substitution, a quote, a pipeline — is exactly the case the reading below + # exists to refuse, and one that never matched here would be refused for the + # wrong reason, as an invocation nobody could find. + local invocation="${SCAFFOLD_DIR}/${SCAFFOLD_ENTRYPOINT//./\\.}" + invocation+="[[:space:]]+${SCAFFOLD_LINT_STAGE}([^-.[:alnum:]_]|$)" + + # Every `run:` body in the file, searched over the logical commands it hands + # the shell rather than over its raw lines, so an invocation continued across + # two of them counts once and counts here. That there is exactly one of it is + # this question; what shape it has is the next. + local -a run_keys=() hit_lines=() + local i j invocations=0 found + for ((i = 0; i < ${#YAML_BODIES[@]}; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + yaml_run_lines "${i}" || continue + shell_logical_commands + found=0 + for ((j = 0; j < ${#SHELL_COMMANDS[@]}; j++)); do + [[ "${SHELL_COMMANDS[j]}" =~ ${invocation} ]] || continue + found=$((found + 1)) + hit_lines+=("${SHELL_COMMAND_LINES[j]}") + done + if ((found > 0)); then + invocations=$((invocations + found)) + run_keys+=("${i}") + fi + done + + ((invocations != 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} no longer runs ${SCAFFOLD_ENTRYPOINT} \ +${SCAFFOLD_LINT_STAGE} from any step's run: body; it would go on firing on \ +every change to the inputs this scaffold's trust model is derived from and \ +checking none of them" + ((invocations == 1)) || + fail "${SCAFFOLD_LINT_WORKFLOW} runs ${SCAFFOLD_ENTRYPOINT} \ +${SCAFFOLD_LINT_STAGE} ${invocations} times; this parser cannot tell which of \ +them the conditions it reads below belong to" + + # The shell that one body carries, read as the shell would take it: what the + # runner writes into it, what the commands around the invocation could do to + # it, and whether the status the step reports is the analysis's at all. + local run_key="${run_keys[0]}" run_line="${hit_lines[0]}" + yaml_run_lines "${run_key}" + [[ -z "${YAML_RUN_UNMODELLED}" ]] || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((run_key + 1)) hands the \ +${SCAFFOLD_LINT_STAGE} run to the shell through ${YAML_RUN_UNMODELLED}, which \ +this parser has no reading for; what would run there is not what these lines \ +say, and a run nothing here can read is not one this scaffold has proved" + shell_logical_commands + + local k reason invocation_at=-1 + for ((k = 0; k < ${#SHELL_COMMANDS[@]}; k++)); do + if ! shell_expand_expressions "${SHELL_COMMANDS[k]}"; then + fail "${SCAFFOLD_LINT_WORKFLOW} line $((SHELL_COMMAND_LINES[k] + 1)) \ +carries ${SHELL_EXPRESSION_REFUSAL} in the step running \ +${SCAFFOLD_LINT_STAGE}; the runner writes that value into this shell before \ +the shell parses it, so the command it would make is not one read here" + fi + SHELL_COMMANDS[k]="${SHELL_EXPANDED}" + + if reason="$(shell_unmodelled_construct "${SHELL_COMMANDS[k]}")"; then + fail "${SCAFFOLD_LINT_WORKFLOW} line $((SHELL_COMMAND_LINES[k] + 1)) \ +runs ${SCAFFOLD_LINT_STAGE} in a body carrying ${reason}; the status the step \ +reports would be decided by something other than the analysis, and a check \ +nothing depends on gates nothing" + fi + if reason="$(shell_unmodelled_word "${SHELL_COMMANDS[k]}")"; then + fail "${SCAFFOLD_LINT_WORKFLOW} line $((SHELL_COMMAND_LINES[k] + 1)) \ +opens ${reason} in the step running ${SCAFFOLD_LINT_STAGE}; whether the \ +analysis runs at all then rests on shell this parser does not read" + fi + [[ "${SHELL_COMMANDS[k]}" =~ ${invocation} ]] && invocation_at="${k}" + done + + ((invocation_at == ${#SHELL_COMMANDS[@]} - 1)) || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((run_line + 1)) runs \ +${SCAFFOLD_LINT_STAGE} with $((${#SHELL_COMMANDS[@]} - invocation_at - 1)) \ +command(s) after it; a step reports its last command's exit status, so a \ +failing analysis would be reported as whatever ran after it" + + if reason="$(shell_invocation_shape "${SHELL_COMMANDS[invocation_at]}")"; then + fail "${SCAFFOLD_LINT_WORKFLOW} line $((run_line + 1)) does not run \ +${SCAFFOLD_LINT_STAGE} as a command of its own: ${reason}; a mention of the \ +analysis is not a run of it, and the step would report whatever did run" + fi + + local step=-1 step_keys="" + if step_keys="$(yaml_item_key_indent "${run_key}")"; then + step="${run_key}" + else + # A `run:` key that did not open its own step belongs to the nearest + # sequence item opened shallower than it. + for ((i = run_key - 1; i >= 0; i--)); do + ((YAML_INDENTS[i] < 0)) && continue + ((YAML_INDENTS[i] < YAML_INDENTS[run_key])) || continue + step_keys="$(yaml_item_key_indent "${i}")" || continue + step="${i}" + break + done + fi + ((step >= 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((run_key + 1)) runs \ +${SCAFFOLD_LINT_STAGE} outside any step this parser can place, so nothing \ +here can say whether that run is conditioned away" + + local step_end + step_end="$(yaml_block_end "$((step + 1))" "${step_keys}")" + verify_scaffold_lint_unconditional "${step}" "${step_end}" "${step_keys}" \ + "step" + + local jobs_line=-1 job_indent=-1 job=-1 + for ((i = 0; i < ${#YAML_BODIES[@]}; i++)); do + ((YAML_INDENTS[i] == 0)) || continue + [[ "${YAML_BODIES[i]}" == 'jobs:' ]] || continue + jobs_line="${i}" + break + done + ((jobs_line >= 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} declares no jobs this parser can read, so \ +nothing here can say what surrounds the ${SCAFFOLD_LINT_STAGE} run" + + # The last job opened before the step is the one the step belongs to; a line + # shallower than a job name means the step sits outside jobs: altogether. + for ((i = jobs_line + 1; i <= step; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + ((job_indent < 0)) && job_indent="${YAML_INDENTS[i]}" + if ((YAML_INDENTS[i] < job_indent)); then + job=-1 + break + fi + ((YAML_INDENTS[i] == job_indent)) && job="${i}" + done + ((job >= 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((step + 1)) runs \ +${SCAFFOLD_LINT_STAGE} in no job this parser can place, so nothing here can \ +say whether that job is conditioned away" + + local job_end job_keys=-1 + job_end="$(yaml_block_end "$((job + 1))" "$((job_indent + 1))")" + for ((i = job + 1; i < job_end; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + job_keys="${YAML_INDENTS[i]}" + break + done + ((job_keys >= 0)) || + fail "${SCAFFOLD_LINT_WORKFLOW} job [${YAML_BODIES[job]%:}] carries \ +nothing this parser can read, so nothing here can say whether the \ +${SCAFFOLD_LINT_STAGE} run inside it is conditioned away" + verify_scaffold_lint_unconditional "${job}" "${job_end}" "${job_keys}" \ + "job [${YAML_BODIES[job]%:}]" + + verify_scaffold_lint_preceding_steps "${job}" "${step}" "${step_keys}" + + # The same two substitutions one level further out, where neither block above + # would show them. + for ((i = 0; i < ${#YAML_BODIES[@]}; i++)); do + ((YAML_INDENTS[i] == 0)) || continue + case "${YAML_BODIES[i]}" in + 'defaults:') + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) sets workflow-wide \ +defaults; what runs the ${SCAFFOLD_LINT_STAGE} body is then decided somewhere \ +this parser does not read, which is the same as not knowing" + ;; + 'env:'*) + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) writes a workflow-wide \ +environment; every step inherits it, so the shell running \ +${SCAFFOLD_LINT_STAGE} would be handed names that decide what the entrypoint's \ +own name resolves to before it parses the command read here" + ;; + esac + done + + note "scaffold lint: ${SCAFFOLD_LINT_WORKFLOW} runs ${SCAFFOLD_ENTRYPOINT} \ +${SCAFFOLD_LINT_STAGE} unconditionally, on line $((run_line + 1)), as its \ +step's last command, under the runner's own shell, with no environment or \ +working directory written around it and no earlier step in its job running a \ +shell of its own; that this commit says so is the whole of what is proved here \ +— a command ahead of the invocation in that same body is read for the shell it \ +opens and not for what it writes, and that a run happened at all, and that the \ +run reporting success ran this file, rests on a ruleset requiring a workflow \ +this repository does not supply, SHA-pinned outside it and carrying this \ +analysis itself, whose standing is recorded in ${SCAFFOLD_DIR}/README.md" +} + +# The keys that turn a step or the job around it into something a change can +# get past without this analysis having judged it: one deciding whether it runs +# at all, one deciding that its failure does not fail the run, and the rest +# deciding what the accepted command word would actually reach. A condition is +# refused rather than evaluated — this parser cannot tell which runs it would +# hold for, and a gate whose reachability rests on a condition nothing here +# reads is not a gate this scaffold has proved reachable. +# +# `shell:` is the one that leaves no mark at all on the shell it retires: the +# body reads exactly as it did while an interpreter that never runs a line of +# it — or never reports what running it said — takes the step's place. Only the +# runner's own default is accepted, spelled out or left out. `defaults:` sets +# the same thing a level or two away, and is refused outright rather than +# followed, because a body run by something this parser never saw named is the +# same unread state either way. +# +# The three added beside them retire the invocation without touching the line +# that carries it, which is why reading the body alone was never enough: +# +# `env:` — the runner writes these names into the step's own +# shell before it parses a line, and a `BASH_ENV` +# there names a file that shell sources first. A +# function defined in it can carry the entrypoint's +# own name; the exact command word accepted above then +# runs that function and returns whatever it says. +# `working-directory:` — the invocation is a relative path. Resolved from +# another directory it names another file, and the +# text proving the analysis runs proves it of +# something else entirely. +# `container:` — the job's steps run inside an image this parser +# never reads, which decides both what bash is and +# what stands at the entrypoint's path. +# +# All three are refused outright rather than followed: a value read here would +# have to be resolved against a filesystem and an environment that exist only +# on the runner, and a resolution guessed at is worse than a refusal. +verify_scaffold_lint_unconditional() { + local from="$1" to="$2" key_indent="$3" where="$4" i body value + for ((i = from + 1; i < to; i++)); do + ((YAML_INDENTS[i] == key_indent)) || continue + body="${YAML_BODIES[i]}" + case "${body}" in + 'if:'*) + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) conditions the ${where} \ +running ${SCAFFOLD_LINT_STAGE} on [${body#if:}]; a gate that runs only when a \ +condition holds is not the unconditional one this scaffold's evidence rests on" + ;; + 'continue-on-error:'*) + value="$(yaml_scalar_value "${body#continue-on-error:}")" || value="" + [[ "${value}" == 'false' ]] && continue + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) lets the ${where} \ +running ${SCAFFOLD_LINT_STAGE} fail without failing the run; a check nothing \ +depends on gates nothing" + ;; + 'shell:'*) + value="$(yaml_scalar_value "${body#shell:}")" || value="" + [[ "${value}" == 'bash' ]] && continue + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) hands the ${where} \ +running ${SCAFFOLD_LINT_STAGE} to [${value}]; the body would read the same \ +while an interpreter this parser never saw decided whether any of it runs and \ +what its failing said" + ;; + 'defaults:') + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) sets defaults on the \ +${where} running ${SCAFFOLD_LINT_STAGE}; what runs that body is then decided \ +somewhere this parser does not read, which is the same as not knowing" + ;; + 'env:'*) + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) writes an environment \ +into the ${where} running ${SCAFFOLD_LINT_STAGE}; the runner sets those names \ +before the shell parses a line, and one of them naming a file that shell \ +sources first can define the entrypoint's own name as a function — the \ +command read here would then be exactly as written and run none of the analysis" + ;; + 'working-directory:'*) + value="$(yaml_scalar_value "${body#working-directory:}")" || value="" + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) runs the ${where} \ +carrying ${SCAFFOLD_LINT_STAGE} from [${value}]; the invocation is a relative \ +path, so what it names is decided by a directory this parser cannot resolve, \ +and the analysis proved to run there is an analysis in some other file" + ;; + 'container:'*) + fail "${SCAFFOLD_LINT_WORKFLOW} line $((i + 1)) runs the ${where} \ +carrying ${SCAFFOLD_LINT_STAGE} inside a container image; what bash is there \ +and what stands at the entrypoint's path are decided by an image this parser \ +never reads, which is the same as not knowing what ran" + ;; + esac + done +} + +# The steps the job runs before the analysis one. +# +# Everything above reads a single step, and a step's body is only as good as +# the tree and the environment it meets. A step ahead of it can write over the +# entrypoint in the checkout, or append a name to $GITHUB_ENV that every step +# after it inherits. Either one leaves each line read above exactly as it was +# while the run they describe becomes a different run entirely. +# +# So a preceding step carrying a `run:` body is refused rather than read: what +# that shell would do is the whole question, and answering it would mean +# modelling a filesystem and an environment that exist only on the runner. +# +# A preceding `uses:` step is not proved harmless by this — an action runs code +# out of another repository and reaches $GITHUB_ENV and the checkout just as +# directly. It is accepted because refusing it would refuse the checkout the +# analysis needs in order to read anything at all. What that leaves open is not +# closed here and is not claimed to be; it is recorded with the rest of this +# reading's boundary in ${SCAFFOLD_DIR}/README.md. +verify_scaffold_lint_preceding_steps() { + local job="$1" step="$2" step_keys="$3" + local i j item_end opened body first + for ((i = job + 1; i < step; i++)); do + ((YAML_INDENTS[i] < 0)) && continue + opened="$(yaml_item_key_indent "${i}")" || continue + [[ "${opened}" == "${step_keys}" ]] || continue + + # A sequence item's first key sits on the dash line itself; the rest sit at + # the item's own key column, and the item ends where the next one opens. + item_end="$(yaml_block_end "$((i + 1))" "${step_keys}")" + first="${YAML_BODIES[i]#-}" + first="${first#"${first%%[![:space:]]*}"}" + for ((j = i; j < item_end; j++)); do + if ((j == i)); then + body="${first}" + else + ((YAML_INDENTS[j] == step_keys)) || continue + body="${YAML_BODIES[j]}" + fi + [[ "${body}" == 'run:'* ]] || continue + fail "${SCAFFOLD_LINT_WORKFLOW} line $((j + 1)) runs shell in the job \ +carrying ${SCAFFOLD_LINT_STAGE}, ahead of the step that carries it; what that \ +shell leaves behind — the entrypoint's own file in the checkout, a name \ +written into \$GITHUB_ENV for the steps after it — decides what the invocation \ +read here would reach, and this parser reads none of it" + done + done +} + +# The pull_request event states and base branches a run really covers. +# +# A restriction on either is invisible to a check that reads only paths, and +# both leave the same hole: a change to these inputs that merges without this +# gate having run on it. `branches` is refused outright rather than compared +# against a branch name — naming the branch here would be one more restated +# constant, and every restriction of it exempts some merge. `types` is read, +# because narrowing it is the subtler hole: a list without `synchronize` runs +# once when the pull request opens and never again on what is pushed into it +# afterwards, which is to say never on the change that actually merges. +verify_lint_pull_request_reach() { + local line="$1" key_indent="$2" body="$3" + local end j entry seen="" want required="" + + case "${body}" in + 'branches:' | 'branches-ignore:') + fail "${SCAFFOLD_LINT_WORKFLOW} restricts its pull_request trigger with \ +${body%:}, so a pull request into any branch that restriction leaves out \ +merges a change to these inputs without this gate having run" + ;; + 'types:') ;; + *) return 0 ;; + esac + + end="$(yaml_block_end "$((line + 1))" "$((key_indent + 1))")" + for ((j = line + 1; j < end; j++)); do + ((YAML_INDENTS[j] < 0)) && continue + body="${YAML_BODIES[j]}" + [[ "${body}" == '-'[[:space:]]* ]] || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((j + 1)) is not a pull_request \ +activity type this parser can read" + entry="$(yaml_scalar_value "${body#-}")" || + fail "${SCAFFOLD_LINT_WORKFLOW} line $((j + 1)) quotes its pull_request \ +activity type in a form this parser does not read" + seen+="${entry}"$'\n' + done + + # The three the event carries when nothing narrows it. A list may widen past + # them; dropping one is what leaves a pull request state this never runs in. + for want in opened synchronize reopened; do + grep -qxF -- "${want}" <<<"${seen}" || required+=" ${want}" + done + [[ -z "${required}" ]] || + fail "${SCAFFOLD_LINT_WORKFLOW} narrows its pull_request trigger to \ +activity types missing${required}, leaving pull request states in which a \ +change to these inputs is never rechecked" +} + +# Compile the ignore rules the build itself reads, from the commit under +# test. Which file that is, the builder decides by Dockerfile: +# `.dockerignore` beside the context root wins whenever the +# commit carries one, and the root `.dockerignore` applies only otherwise. So +# a commit adding the Dockerfile-specific file silently retires every rule in +# the root one, and a mirror that read the root file regardless would go on +# checking itself against rules the build has stopped applying. +# +# Each line is then normalized the way the builder normalizes it: a line +# opening with `#` is a comment before anything else touches it, surrounding +# whitespace is trimmed, a leading `!` splits off as a negation and what +# follows it is trimmed again, and the remainder is path-cleaned and stripped +# of a single leading separator. +load_dockerignore_patterns() { + DOCKERIGNORE_SOURCE="" + DOCKERIGNORE_REGEX=() + DOCKERIGNORE_NEGATED=() + + local content + if content="$(git -C "${REPO_ROOT}" show \ + "HEAD:${BUILD_DOCKERFILE}.dockerignore" 2>/dev/null)"; then + DOCKERIGNORE_SOURCE="${BUILD_DOCKERFILE}.dockerignore" + elif content="$(git -C "${REPO_ROOT}" show HEAD:.dockerignore 2>/dev/null)"; then + DOCKERIGNORE_SOURCE=".dockerignore" + else + fail "the commit under test carries neither \ +${BUILD_DOCKERFILE}.dockerignore nor .dockerignore; the build-context \ +classification in this script has nothing left to be checked against" + fi + + local line negated unmodelled + while IFS= read -r line; do + [[ "${line}" == '#'* ]] && continue + line="${line#"${line%%[![:space:]]*}"}" + line="${line%"${line##*[![:space:]]}"}" + [[ -n "${line}" ]] || continue + negated=0 + if [[ "${line}" == '!'* ]]; then + negated=1 + line="${line#!}" + line="${line#"${line%%[![:space:]]*}"}" + line="${line%"${line##*[![:space:]]}"}" + # The builder refuses a bare `!` as an illegal exclusion pattern and + # fails the whole build with it, rather than carrying on with the + # rules around it. + [[ -n "${line}" ]] || + fail "${DOCKERIGNORE_SOURCE} carries a bare [!] line, which the build \ +daemon refuses outright as an illegal exclusion pattern" + fi + line="$(dockerignore_clean_path "${line}")" + ((${#line} > 1)) && line="${line#/}" + # Checked here, in the shell that can still stop the run: the compiler + # below runs inside a command substitution, where refusing would exit + # only the subshell and leave the unmodelled pattern silently empty. + if unmodelled="$(dockerignore_unmodelled_construct "${line}")"; then + fail "the ${DOCKERIGNORE_SOURCE} pattern [${line}] carries \ +[${unmodelled}], which the build daemon gives to its expression engine and \ +the build-context classification in this script reads as a literal; extend \ +dockerignore_pattern_regex before relying on it" + fi + DOCKERIGNORE_REGEX+=("$(dockerignore_pattern_regex "${line}")") + DOCKERIGNORE_NEGATED+=("${negated}") + done <<<"${content}" + + ((${#DOCKERIGNORE_REGEX[@]} > 0)) || + fail "${DOCKERIGNORE_SOURCE} carries no pattern at all; the \ +build-context classification in this script has nothing to be checked against" +} + +# True when the build daemon would keep this committed path out of the build +# context. Patterns apply in file order with the last one to match deciding, +# and a path matches when it or any of its ancestor directories does — the +# daemon's own rule, and the reason a bare directory pattern like `solidity` +# removes everything beneath it. +dockerignore_context_excluded() { + local path="$1" excluded=1 i regex negated matched prefix rest + for ((i = 0; i < ${#DOCKERIGNORE_REGEX[@]}; i++)); do + negated="${DOCKERIGNORE_NEGATED[i]}" + # A negation has nothing to re-include while the path is still in the + # context, and an exclusion has nothing to add once it is already out. + if [[ "${negated}" == 1 ]]; then + [[ "${excluded}" == 0 ]] || continue + else + [[ "${excluded}" == 1 ]] || continue + fi + + regex="${DOCKERIGNORE_REGEX[i]}" + matched=1 + if [[ "${path}" =~ ${regex} ]]; then + matched=0 + else + prefix="" + rest="${path}" + while [[ "${rest}" == */* ]]; do + prefix+="${rest%%/*}" + rest="${rest#*/}" + if [[ "${prefix}" =~ ${regex} ]]; then + matched=0 + break + fi + prefix+="/" + done + fi + + if [[ "${matched}" == 0 ]]; then + if [[ "${negated}" == 1 ]]; then excluded=1; else excluded=0; fi + fi + done + return "${excluded}" +} + +# The two classifications above are hand-written mirrors of build inputs that +# live elsewhere, and a mirror is only ever as good as its last +# synchronization. This walks every path the commit tracks and compares each +# mirror's verdict against the rules the build's own ignore file carries. +# +# A path the mirror calls context-excluded while the context in fact holds it +# is the dangerous direction: build-image mode would explain that file's +# absence from the image as the image's own construction and accept a tree +# missing it. The opposite direction is safe — the mirror would report a +# legitimate absence as unexplained divergence and refuse to produce evidence +# — but it is still drift, so it is tolerated only for the families the image +# regenerates by design, which verify_build_image_tree restores byte-exact +# rather than explains away. +verify_build_context_mirror() { + # Which rules those are is itself a build input: the builder picks its + # ignore file by Dockerfile, and which Dockerfile it compiles is written in + # the workflow that does the building. So the identity is read from that + # workflow, and the gate that reruns this check is held to it, before a + # single pattern is compiled. + resolve_build_step_identity + verify_scaffold_lint_path_filters + verify_scaffold_lint_runs_analysis + load_dockerignore_patterns + note "build-context mirror: checking this script's classification against \ +the ${#DOCKERIGNORE_REGEX[@]} pattern(s) the build reads from \ +${DOCKERIGNORE_SOURCE}" + + local path mirror context tracked=0 excluded=0 regenerated=0 drift="" + while IFS= read -r -d '' path; do + [[ -n "${path}" ]] || continue + tracked=$((tracked + 1)) + if dockerignore_excluded_path "${path}"; then mirror=0; else mirror=1; fi + if dockerignore_context_excluded "${path}"; then context=0; else context=1; fi + + if [[ "${mirror}" == 0 && "${context}" == 0 ]]; then + excluded=$((excluded + 1)) + elif [[ "${mirror}" == 0 ]]; then + drift+="${path}: this script explains an absence here as a \ +context-excluded path, but ${DOCKERIGNORE_SOURCE} keeps it in the build \ +context"$'\n' + elif [[ "${context}" == 0 ]]; then + if regenerated_by_design_path "${path}"; then + regenerated=$((regenerated + 1)) + else + drift+="${path}: ${DOCKERIGNORE_SOURCE} keeps this path out of the \ +build context, but this script neither excludes it nor treats it as \ +regenerated"$'\n' + fi + fi + done < <(git -C "${REPO_ROOT}" ls-tree -r -z --name-only HEAD) + + if [[ -n "${drift}" ]]; then + printf '%s' "${drift}" >&2 + fail "the build-context classification in this script no longer mirrors \ +${DOCKERIGNORE_SOURCE} (listing above); re-derive dockerignore_excluded_path \ +and regenerated_by_design_path from the current build inputs before this \ +scaffold admits any further evidence" + fi + + note "build-context mirror: ${tracked} tracked path(s) classified \ +identically by ${DOCKERIGNORE_SOURCE} and this script (${excluded} kept out \ +of the build context; ${regenerated} excluded from the context but \ +regenerated into the image by design)" +} + +# The artifact input identity behind the image build: get_artifacts leaves +# each resolved npm tarball — name and exact version — under tmp/contracts. +# The digests are forensic context, not trust: the bytes the proof stages +# compile are restored from the dispatched commit regardless of what these +# artifacts contained, but recording them lets an evidence consumer verify +# against the registry what the image build itself consumed. +record_artifact_identity() { + local tarball + if [[ ! -d "${REPO_ROOT}/tmp/contracts" ]]; then + note "artifact identity: no tmp/contracts artifact tree in this image" + return + fi + note "artifact identity: resolved contract artifact tarballs:" + find "${REPO_ROOT}/tmp/contracts" -name '*.tgz' -type f | + LC_ALL=C sort | + while IFS= read -r tarball; do + printf '>> %s sha256 %s\n' "${tarball#"${REPO_ROOT}"/}" \ + "$(hash_stdin <"${tarball}")" + done +} + +# Restore one tracked path byte-exact from the commit under verification. +# `git show` only reads the object store, so it works against the read-only +# .git mount the workflow uses (a `git checkout` would have to write the +# index). HEAD equality with the dispatched SHA is proven before any +# restoration, so the restored bytes are the dispatched bytes by +# construction; a path that cannot be restored fails the stage. +restore_tracked_file_from_head() { + local path="$1" + mkdir -p "${REPO_ROOT}/$(dirname "${path}")" || + fail "could not create the directory to restore ${path}; the image \ +tree cannot be bound to the dispatched commit" + if ! git -C "${REPO_ROOT}" show "HEAD:${path}" >"${REPO_ROOT}/${path}"; then + fail "could not restore ${path} byte-exact from the dispatched commit; \ +the image tree cannot be bound to the dispatched SHA" + fi +} + +# Build-image verification: every porcelain line must be explained by the +# image's documented construction, and no regenerated byte is ever accepted +# into evidence. Deletions are accepted only for context-excluded paths — +# none of which holds Go code the proof stages compile. The families the +# image rewrites by design (the regenerated gen/ bindings and the +# gen/_address values, including placeholders the generator dropped) are +# never trusted as found: each one is restored byte-exact from the +# dispatched commit before any test compiles it, with the pre-restore image +# hash recorded for forensics, so whatever the generator — or anything +# else — put there can never become the tested bytes. Untracked files are +# always fatal once the committed ignore rules are restored; every other +# status — a modified or deleted committed file outside those families, +# index-side changes, renames, typechanges, an unreadable tree — is fatal. +# Restoration itself is not trusted either: the whole tree is re-checked +# afterwards and anything left beyond the context-excluded absences fails. +verify_build_image_tree() { + local expected="$1" + restore_committed_gitignores + + local divergence unexplained="" restorable="" absences=0 restores=0 + local line status path prior + divergence="$(source_divergence)" + while IFS= read -r line; do + [[ -n "${line}" ]] || continue + status="${line:0:2}" + path="${line:3}" + case "${status}" in + " D") + if dockerignore_excluded_path "${path}"; then + absences=$((absences + 1)) + elif [[ "${path}" =~ (^|/)gen/_address/[^/]+$ ]]; then + restorable+="${path}"$'\t'"absent from the image"$'\n' + else + unexplained+="${line}"$'\n' + fi + ;; + " M") + if regenerated_by_design_path "${path}"; then + restorable+="${path}"$'\t'"pre-restore image sha256 \ +$(hash_stdin <"${REPO_ROOT}/${path}")"$'\n' + else + unexplained+="${line}"$'\n' + fi + ;; + *) + unexplained+="${line}"$'\n' + ;; + esac + done <<<"${divergence}" + + if [[ -n "${unexplained}" ]]; then + printf '%s' "${unexplained}" >&2 + fail "source binding to ${expected} requested, but the build-image tree \ +diverges from that commit beyond what the image build produces by design \ +(listing above); refusing to produce evidence" + fi + + if [[ -n "${restorable}" ]]; then + note "regenerated tracked files restored byte-exact from the dispatched \ +commit before testing:" + while IFS=$'\t' read -r path prior; do + [[ -n "${path}" ]] || continue + restore_tracked_file_from_head "${path}" + restores=$((restores + 1)) + printf '>> %s committed sha256 %s (%s)\n' "${path}" \ + "$(hash_stdin <"${REPO_ROOT}/${path}")" "${prior}" + done <<<"${restorable}" + fi + + local residual="" + absences=0 + divergence="$(source_divergence)" + while IFS= read -r line; do + [[ -n "${line}" ]] || continue + status="${line:0:2}" + path="${line:3}" + if [[ "${status}" == " D" ]] && dockerignore_excluded_path "${path}"; then + absences=$((absences + 1)) + continue + fi + residual+="${line}"$'\n' + done <<<"${divergence}" + if [[ -n "${residual}" ]]; then + printf '%s' "${residual}" >&2 + fail "source binding to ${expected} requested, but restoration left the \ +build-image tree diverging from that commit (listing above); refusing to \ +produce evidence" + fi + + record_artifact_identity + + note "source commit: ${expected} (verified against the dispatched SHA \ +inside the build image; ${absences} context-excluded absence(s); \ +${restores} regenerated tracked file(s) restored byte-exact from that \ +commit before testing)" +} + +# Fail-closed source binding. When PR4109_EXPECTED_SOURCE_COMMIT is set — +# the workflow passes the dispatched SHA to every proof stage, mounting the +# checkout's .git and scripts/ read-only into the build image so even the +# container run can be held to it — the stage refuses to run unless the +# tree under test is exactly that commit. Without the variable the stage +# stamps its log via source_commit and runs anyway: a local iteration loop +# may test a dirty tree, it just can never produce evidence claiming to be +# a clean commit. +verify_source_binding() { + local expected="${PR4109_EXPECTED_SOURCE_COMMIT:-}" + if [[ -z "${expected}" ]]; then + note "source commit: $(source_commit) (unbound run; set \ +PR4109_EXPECTED_SOURCE_COMMIT to fail closed on divergence)" + return + fi + + local head + if ! head="$(git -C "${REPO_ROOT}" rev-parse HEAD 2>/dev/null)"; then + fail "source binding to ${expected} requested, but the tree under test \ +has no readable git metadata; mount the dispatched checkout's .git \ +(read-only) next to the source so the tested bytes can be verified" + fi + if [[ "${head}" != "${expected}" ]]; then + fail "source binding mismatch: the tree under test is at ${head}, the \ +dispatch expects ${expected}" + fi + + local mode="${PR4109_SOURCE_BINDING_MODE:-exact}" divergence + case "${mode}" in + exact) + divergence="$(source_divergence)" + if [[ -n "${divergence}" ]]; then + printf '%s\n' "${divergence}" >&2 + fail "source binding to ${expected} requested, but the tree diverges \ +from that commit (listing above; untracked files count); refusing to \ +produce evidence for bytes that are not the dispatched commit" + fi + note "source commit: ${expected} (verified against the dispatched SHA)" + ;; + build-image) + verify_build_image_tree "${expected}" + ;; + *) + fail "unknown PR4109_SOURCE_BINDING_MODE [${mode}]; use exact or \ +build-image" + ;; + esac + + # Reaching here means the tested bytes were proved to be this commit's: + # fail and blocked both exit. Later steps in the same stage stamp their + # output with this identity rather than re-deriving it, because the raw + # stamp cannot express what was proved — build-image mode verifies a tree + # that legitimately diverges from HEAD, so source_commit would call the + # very tree this function just accepted -dirty. + VERIFIED_SOURCE_COMMIT="${expected}" +} + +require_env() { + local missing=() + for name in "$@"; do + [[ -n "${!name:-}" ]] || missing+=("${name}") + done + if ((${#missing[@]} > 0)); then + blocked "missing required rehearsal inputs: ${missing[*]}" + fi +} + +require_immutable_digest() { + local name="$1" value="$2" + if [[ ! "${value}" =~ @sha256:[0-9a-f]{64}$ ]]; then + blocked "${name} must be an immutable repo@sha256:... digest, got [${value}]" + fi +} + +# The reviewed digests of the programs this rehearsal executes but does not +# contain, read out of the checked-in control file. Empty when the file names +# no digest for that program. +reviewed_input_digest() { + local program="$1" file digest="" hash name + file="${2:-${SCRIPT_DIR}/chain-inputs.sha256}" + [[ -f "${file}" ]] || { + printf '' + return 0 + } + while read -r hash name; do + [[ -n "${hash}" && "${hash#\#}" == "${hash}" ]] || continue + [[ "${name}" == "${program}" ]] || continue + [[ "${hash}" =~ ^[0-9a-f]{64}$ ]] || continue + digest="${hash}" + break + done <"${file}" + printf '%s' "${digest}" +} + +# The digests of the programs this run was actually handed, recorded so the +# evidence names the instruments it was produced with. +WORK_DRIVER_DIGEST="" +ROLLBACK_GENERATOR_DIGEST="" + +# The digest of the archived independent cryptographic review of the pinned +# dual-mode dependency, when one was supplied. Unlike the two above this is not +# an instrument the rehearsal runs: it is a release input the recorded evidence +# is accepted against, so it may be absent from an execution that still runs +# every step. +TSSLIB_REVIEW_DIGEST="" + +# The immutable dependency revision the build actually resolves. +# +# Read out of go.mod rather than restated, so a review record can only ever be +# bound to the revision this tree compiles against. Restating the commit here +# would let the pin move under a review record that still names the old one, +# which is the exact substitution the binding exists to prevent. +pinned_tsslib_commit() { + local line commit="" + while read -r line; do + case "${line}" in + *"github.com/bnb-chain/tss-lib =>"*) + # The pseudo-version's trailing revision, e.g. + # v0.0.0-20260729021955-d847ce003019 -> d847ce003019. + commit="${line##*-}" + break + ;; + esac + done <"${REPO_ROOT}/go.mod" + printf '%s' "${commit}" +} + +# Bind one supplied review record to its reviewed digest and to the exact +# dependency revision it reviews, or refuse to accept it. +# +# A review record is an external document, and a document asserting that some +# revision was reviewed says nothing about the revision this tree builds. Two +# separate bindings are therefore required: the bytes must hash to a digest +# reviewed in a commit of this repository, and the document must name the +# commit go.mod resolves. Either alone admits a review of other code. +require_reviewed_record() { + local variable="$1" program="$2" path="$3" control="${4:-}" + local reviewed actual commit + [[ -n "${path}" ]] || { + printf '' + return 0 + } + [[ -f "${path}" && -r "${path}" ]] || + blocked "${variable} points at ${path}, which is not a readable file; a \ +review record that cannot be read cannot be bound to anything" + reviewed="$(reviewed_input_digest "${program}" "${control}")" + [[ -n "${reviewed}" ]] || + blocked "no reviewed SHA-256 for ${program} is recorded in \ +${SCAFFOLD_DIR}/chain-inputs.sha256, so the record supplied through \ +${variable} is unbound; an unreviewed document asserting that the dependency \ +was reviewed is the assertion this gate exists to check, not evidence for it" + actual="$(hash_stdin <"${path}")" + [[ "${actual}" == "${reviewed}" ]] || + blocked "the record supplied through ${variable} hashes to ${actual}, and \ +${SCAFFOLD_DIR}/chain-inputs.sha256 pins ${program} at ${reviewed}; a \ +rehearsal cannot accept a review record other than the reviewed one" + commit="$(pinned_tsslib_commit)" + [[ -n "${commit}" ]] || + blocked "go.mod resolves no github.com/bnb-chain/tss-lib replacement, so \ +there is no dependency revision for the record supplied through ${variable} \ +to be bound to" + grep -qF "${commit}" "${path}" || + blocked "the record supplied through ${variable} does not name the \ +dependency revision [${commit}] that go.mod resolves; a review of another \ +revision is not a review of the code this rehearsal runs" + printf '%s' "${actual}" +} + +# Bind one supplied program to its reviewed digest, or refuse to run it. +# +# Both of these arrive from a mutable secret bundle, and both produce readings +# that become release evidence: the driver's account of what it originated and +# what became of it is the entire terminal half of every control that watches +# work settle. An executable bit is not provenance, and an internally +# consistent report from the wrong program passes every check in this +# repository. So the bytes are hashed here and compared against a digest +# reviewed in a commit, and a mismatch stops the rehearsal rather than +# producing a record naming an instrument nobody reviewed. +require_reviewed_input() { + local variable="$1" program="$2" path="$3" control="${4:-}" reviewed actual + [[ -n "${path}" ]] || { + printf '' + return 0 + } + [[ -x "${path}" ]] || + blocked "${variable} points at ${path}, which is not an executable \ +program; the rehearsal cannot drive work with it and cannot record what it did" + reviewed="$(reviewed_input_digest "${program}" "${control}")" + [[ -n "${reviewed}" ]] || + blocked "no reviewed SHA-256 for ${program} is recorded in \ +${SCAFFOLD_DIR}/chain-inputs.sha256, so the program supplied through \ +${variable} is unbound; its report is the terminal half of every control that \ +watches work settle, and an unreviewed program produces an internally \ +consistent passing account that every check in this repository accepts" + actual="$(hash_stdin <"${path}")" + [[ "${actual}" == "${reviewed}" ]] || + blocked "the program supplied through ${variable} hashes to ${actual}, \ +and ${SCAFFOLD_DIR}/chain-inputs.sha256 pins ${program} at ${reviewed}; a \ +rehearsal cannot record evidence produced by a program other than the \ +reviewed one" + printf '%s' "${actual}" +} + +# Directory holding the release-manifest attestation: the receipt proving the +# checked-in manifest still matches the compiled bounds of the source under +# test. It is a subdirectory on purpose. Both the record glob below and the +# workflow's record probe look at EVIDENCE_DIR's top level only, so producing +# this receipt never makes a record-free dispatch look like it produced a +# rehearsal record. +attestation_dir() { printf '%s\n' "${EVIDENCE_DIR}/attestation"; } + +# The source identity a receipt written now may claim: what the binding +# check proved, or — for an unbound run — the tree's own stamp, which carries +# its -dirty marker and its outside-a-checkout "unknown" with it. The +# acceptance stage refuses anything but a clean commit id, so an unbound or +# divergent run still produces a receipt; it just produces one that cannot +# launder bytes into release evidence. +attested_source_identity() { + if [[ -n "${VERIFIED_SOURCE_COMMIT}" ]]; then + printf '%s' "${VERIFIED_SOURCE_COMMIT}" + return + fi + source_commit +} + +# A receipt speaks for the run that wrote it and for no other, so every proof +# run destroys the receipt it inherits before it proves anything. Evidence +# directories get reused — a re-dispatch into the same workspace, a local +# iteration loop — and without this a run failing anywhere before the +# attestation step would leave its predecessor's receipt standing for the +# acceptance stage to find and accept. Interrupted staging directories go the +# same way, so no fragment of an older run survives into this one. +invalidate_release_manifest_attestation() { + local dir + dir="$(attestation_dir)" + if [[ -e "${dir}" ]]; then + note "discarding the release-manifest attestation inherited in ${dir}" + fi + rm -rf "${dir}" "${dir}".staging.* +} + +# The acceptance stage judges a rehearsal record by comparing it against the +# checked-in release manifest, but that manifest only speaks for the release +# while it still matches this binary's compiled bounds. The Go proofs pin that +# identity inside their own log; this turns it into a machine-checkable +# receipt, produced here — inside the source-bound tree, where the Go +# toolchain is — so the acceptance stage can require the proof without +# carrying a toolchain of its own. +attest_release_manifest() { + local manifest="${SCRIPT_DIR}/release-manifest.json" + local dir staging + dir="$(attestation_dir)" + # Build the receipt beside its destination and publish it with a single + # rename, so a reader sees this run's complete receipt or no receipt at + # all. Writing the files straight into the destination would publish a + # half-built receipt while it is being written, and would let files from + # two different runs end up sitting in one directory. + staging="${dir}.staging.$$" + rm -rf "${staging}" + mkdir -p "${staging}" + + note "attesting the release manifest against the compiled bounds" + # validate is the binary's own reviewed check: it rejects a manifest whose + # numbers differ from the compiled derivation in any field, the cleanup + # allowance the runtime actually waits included. + go run . release-manifest validate --manifest "${manifest}" + + # Validity is not readiness. A manifest is valid throughout development, but + # it cannot anchor a release-acceptance decision until the values that do not + # exist during development — the cutover block, the commit finally built, the + # image digests acceptance ran against — have been reviewed and recorded. The + # binary's own --release-ready mode is what answers that, and the answer is + # recorded here, inside the source-bound tree where the toolchain is, so the + # acceptance stage can refuse a placeholder without carrying one of its own. + # + # A manifest that is not ready does not fail this stage. Development runs + # legitimately have one, and everything proved above is about the code rather + # than about the release identity. The verdict is written down instead, and + # refusing on it is the acceptance stage's decision to take. + if go run . release-manifest validate --manifest "${manifest}" \ + --release-ready >"${staging}/release-ready.log" 2>&1; then + printf 'yes\n' >"${staging}/release-ready.txt" + else + printf 'no\n' >"${staging}/release-ready.txt" + note "ATTENTION: the reviewed release manifest is not release-ready;" \ + "no release-acceptance decision may be taken against it:" + # Only the violations. The whole log stays in the receipt, but the + # command-line usage the failing subcommand prints after them would + # bury the lines an operator is reading this for. + sed '/^Usage:/,$d' "${staging}/release-ready.log" | + sed 's/^/>> /' + fi + + # derive emits the manifest the compiled bounds produce, so the receipt + # carries those bounds themselves rather than an assertion about them, and + # the hash names the exact reviewed bytes validate just accepted. + go run . release-manifest derive >"${staging}/derived-manifest.json" + hash_stdin <"${manifest}" >"${staging}/reviewed-manifest.sha256" + + # The commit these bounds were compiled from. The acceptance stage requires + # every record it measures to name this same commit, so a receipt can never + # vouch for records built from other bytes — the case the manifest hash + # alone misses entirely, since a manifest that did not change between two + # commits hashes the same at both. + attested_source_identity >"${staging}/source-commit.txt" + printf '\n' >>"${staging}/source-commit.txt" + + attest_release_provenance "${manifest}" "${staging}" + + rm -rf "${dir}" + mv "${staging}" "${dir}" + + note "release-manifest attestation written to ${dir} for source \ +$(tr -d '[:space:]' <"${dir}/source-commit.txt")" +} + +# Take the detached provenance into the receipt, when the operator supplied +# one. +# +# The reviewed manifest names the cutover; it cannot name what was built to run +# it. Those values are outputs of a build over the manifest's own bytes, so +# recording them in the tree would require the tree to contain a hash of itself +# — write the commit and the commit changes. They live in a document generated +# after the build instead, and PR4109_RELEASE_PROVENANCE is where a release run +# points at it. +# +# Copied into the receipt rather than read from its original path at acceptance +# time: the receipt is the run's own sealed account, and a path re-read later +# is a file that may have been rewritten in between. The hash goes in beside it +# so the acceptance stage can say which document this was. +# +# A run without provenance writes none and is not refused here. Development +# runs legitimately have no build to describe, and the acceptance stage is +# where the absence becomes a refusal — for the same reason the readiness +# verdict is recorded rather than enforced here. +attest_release_provenance() { + local manifest="$1" staging="$2" + local provenance="${PR4109_RELEASE_PROVENANCE:-}" + + if [[ -z "${provenance}" ]]; then + note "no detached release provenance supplied \ +(PR4109_RELEASE_PROVENANCE); the receipt will carry none, and the acceptance \ +stage refuses release evidence without it" + return + fi + + [[ -f "${provenance}" ]] || + fail "PR4109_RELEASE_PROVENANCE names [${provenance}], which is not a \ +readable file" + + # The whole point of the document is that it is not in the tree it + # describes. A tracked file would put the source commit back inside the + # commit it names, which is the impossibility this split exists to remove — + # and it would do it quietly, since every check downstream would still pass + # against whatever stale hash the tree happened to carry. + if git -C "${REPO_ROOT}" ls-files --error-unmatch "${provenance}" \ + >/dev/null 2>&1; then + fail "the detached release provenance [${provenance}] is tracked in this \ +repository; it records the commit built from this tree and the images built \ +out of it, so committing it would require the tree to contain a hash of \ +itself. Generate it after the build, outside the checkout" + fi + + # The binary's own reviewed check: the manifest against the compiled bounds + # and against readiness, the provenance against its shape, and the pair + # against the manifest hash recorded inside the provenance. + go run . release-manifest verify-provenance \ + --manifest "${manifest}" --provenance "${provenance}" || + fail "the detached release provenance [${provenance}] does not verify \ +against ${manifest}" + + cp "${provenance}" "${staging}/release-provenance.json" + hash_stdin <"${provenance}" >"${staging}/release-provenance.sha256" + + note "detached release provenance recorded in the receipt \ +(sha256 $(tr -d '[:space:]' <"${staging}/release-provenance.sha256"))" +} + +# Everything stage_local_proofs proves, in one seam. The stage around it owns +# the receipt lifecycle — destroy the inherited one, prove, publish this run's +# — and that ordering is the whole reason a failed proof run cannot leave a +# usable receipt behind, so the self-test drives the stage with this function +# replaced by a failing stub to hold the ordering in place. +run_local_proof_suite() { + # The verifier gates every piece of evidence below, so it proves itself + # first: the self-test builds throwaway repositories shaped like the + # dispatched checkout and like the build image's tree and checks the + # verifier accepts exactly the image's documented construction. + "${SCRIPT_DIR}/test-source-binding.sh" + # The evidence-record validator gates the acceptance of every rehearsal + # record the same way, so it proves itself on every proof run — not only + # on the dispatches that happen to produce records for validate-evidence + # — and its verdicts land in this stage's archived log. + "${SCRIPT_DIR}/test-validate-evidence.sh" + # The workflow's own dispatch validator is extracted and driven over valid + # and hostile provenance/chain mappings. This keeps an invalid dispatch from + # reaching the expensive platform jobs merely because no container + # rehearsal happened to exercise that input shape. + "${SCRIPT_DIR}/test-rehearsal-matrix.sh" + # The go/no-go roster path is process-signaled and log-authored rather than + # exposed through the diagnostics API. Its capture helper is driven against + # a fake two-node Docker boundary so an ignored signal, failed delivery, or + # missing empty/cadenced evidence can never look like an empty ready fleet. + "${SCRIPT_DIR}/test-cutover-evidence-window.sh" + # The readiness verdict this stage is about to write is the one thing in the + # receipt the validator suite cannot prove: that suite hand-authors the + # verdict file, so it holds the refusal without ever running the producer. + # This proves the seam instead — the recorded verdict against the binary's + # own answer, and the produced receipt through the consumer that gates on it. + # It runs here rather than in the shell-analysis gate because it needs the Go + # toolchain to ask the binary, which is the whole point of the assertion. + "${SCRIPT_DIR}/test-attest-release-manifest.sh" + verify_source_binding + # The verifier's own build-context classification is what the binding check + # just used to explain away every absence from the image, so its agreement + # with the committed build inputs is proved here, where evidence is + # produced, and not only in the scaffold's static-analysis gate. + verify_build_context_mirror + go test -count=1 -v \ + -run 'TestJoinDKGIfEligible|TestMonitorRelayEntry|TestForwardSignatureShares' \ + ./pkg/beacon/ + go test -count=1 ./pkg/protocol/participation/... ./pkg/protocol/state/... + go test -count=1 -race \ + ./pkg/protocol/participation/... ./pkg/protocol/state/... + go test -count=1 \ + -run 'TestSubmitDKGResult|TestSyncExecute' \ + ./pkg/beacon/dkg/result/ ./pkg/protocol/state/ + go test -count=1 -race \ + -run 'TestAwaitQuiesce|TestQuiesceBackstop|TestSignalLifecycle|TestMaximumLegacyCompletionBlocks|TestReleaseManifest' \ + ./cmd/ + go test -count=1 -race ./cmd/participation-state-audit/ + go test -count=1 -run 'TestDecodeSignerAuditRecord' ./pkg/tbtc/ + # The inactivity claim lifecycle publishes from one goroutine per controlled + # member and shares a call-wide chain subscription and an atomic submission + # record between them, so its tests only have teeth under the race detector. + # The filter is deliberate rather than a whole-package run: pkg/tbtc carries + # race warnings and a load-dependent block counter flake that reproduce at + # this branch's merge base, and a gate that is red before the release changes + # anything cannot report on them. Everything outside this filter is covered + # by the ordinary suite and by CI's scheduled whole-tree race job. + go test -count=1 -race -timeout 900s -v \ + -run 'Cutover|HandleAnnouncerSessionMismatch|InactivityClaim|SubmitClaim' \ + ./pkg/tbtc/ + # The integration-tagged test files are not compiled by the ordinary + # suite; type-check them so a signature drift cannot hide behind the + # build tag. Their execution needs live Bitcoin/Ethereum endpoints and + # stays with the CI integration job. + go vet -tags=integration ./pkg/bitcoin/electrum/ ./pkg/chain/ethereum/ +} + +stage_local_proofs() { + note "running the repository-local cutover gate proofs" + mkdir -p "${EVIDENCE_DIR}" + local log="${EVIDENCE_DIR}/local-proofs.log" + + ( + # Before anything is proved, so no proof below can fail while an earlier + # run's receipt stays behind to be accepted in this run's name. Runs + # ahead of the cd because EVIDENCE_DIR may be relative to the caller's + # directory. + invalidate_release_manifest_attestation + + cd "${REPO_ROOT}" + run_local_proof_suite + + # Last, so the receipt exists only for a tree whose proofs all passed. + attest_release_manifest + ) 2>&1 | tee "${log}" + + # Skips are part of the evidence, not noise: every mandatory acceptance + # case that cannot run yet must be visible in the proof output. + local skips + skips=$(grep -c '^--- SKIP' "${log}" || true) + if [[ "${skips}" -gt 0 ]]; then + note "ATTENTION: ${skips} skipped case(s) inside the local proofs:" + grep '^--- SKIP' "${log}" | sed 's/^/>> /' + note "each skip above is a mandatory acceptance case still blocked on" \ + "an external dependency; see the hard-dependency record in README.md" + else + note "no skipped cases inside the local proofs" + fi + + note "local proofs recorded in ${log}" +} + +stage_static_analysis() { + note "running the CI-enforced Go static analyzers at immutable versions" + mkdir -p "${EVIDENCE_DIR}" + local log="${EVIDENCE_DIR}/static-analysis.log" + + ( + cd "${REPO_ROOT}" + verify_source_binding + + note "gofmt" + if [[ "$(gofmt -l . | wc -l)" -gt 0 ]]; then + gofmt -d -e . + exit 1 + fi + + # CI's client-vet job vets the root package only; the rehearsal vets + # the whole tree so a finding in any changed package blocks evidence. + note "go vet ./..." + go vet ./... + + note "staticcheck 2025.1.1 (checks: -SA1019)" + go run honnef.co/go/tools/cmd/staticcheck@2025.1.1 \ + -checks=-SA1019 ./... + + # CI's gosec job floats on securego/gosec@master; a rehearsal log must + # be reproducible, so the same flag set runs at a pinned release. + note "gosec v2.28.0 (CI flag set)" + go run github.com/securego/gosec/v2/cmd/gosec@v2.28.0 \ + -exclude=G115,G118 \ + -exclude-dir=pkg/chain/ethereum/beacon/gen \ + -exclude-dir=pkg/chain/ethereum/ecdsa/gen \ + -exclude-dir=pkg/chain/ethereum/threshold/gen \ + -exclude-dir=pkg/chain/ethereum/tbtc/gen \ + ./... + + note "golangci-lint v2.12.2" + go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run + ) 2>&1 | tee "${log}" + + note "static analysis recorded in ${log}" +} + +# The scaffold's own workflow files. actionlint is deliberately not pointed +# at the whole workflow directory: the unrelated workflows carry pre-existing +# findings, and a gate that is red for reasons outside its scope stops being +# read. +cutover_workflow_files() { + printf '%s\n' \ + "${REPO_ROOT}/.github/workflows/cutover-rehearsal.yml" \ + "${REPO_ROOT}/.github/workflows/cutover-scaffold-lint.yml" +} + +stage_shell_analysis() { + note "analyzing the rehearsal scaffold's shell scripts and workflows" + mkdir -p "${EVIDENCE_DIR}" + local log="${EVIDENCE_DIR}/shell-analysis.log" + + command -v shellcheck >/dev/null 2>&1 || + blocked "shellcheck is required to analyze the rehearsal scripts" + command -v node >/dev/null 2>&1 || + blocked "node (Node.js) is required by the evidence-validator self-test" + command -v npx >/dev/null 2>&1 || + blocked "npx (Node.js) is required by the evidence-validator self-test" + command -v git >/dev/null 2>&1 || + blocked "git is required by the source-binding and evidence-record \ +validator self-tests" + + ( + cd "${REPO_ROOT}" + verify_source_binding + + local script + note "bash -n" + for script in "${SCRIPT_DIR}"/*.sh; do + bash -n "${script}" + done + + note "shellcheck $(shellcheck --version | awk '/^version:/ {print $2}')" + for script in "${SCRIPT_DIR}"/*.sh; do + shellcheck "${script}" + done + + # Pinned like every other analyzer here: a floating version must never + # change what this gate accepts. + note "actionlint v1.7.12" + local workflow + while IFS= read -r workflow; do + go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.12 "${workflow}" + done < <(cutover_workflow_files) + + # The verifier's build-context classification is a hand-written mirror of + # .dockerignore, so it drifts silently whenever the real build inputs + # change. This gate runs on every change to those inputs, which is why it + # is where the mirror is held to them. + verify_build_context_mirror + + # The contracts stage's evidence is only the named CI job's evidence while + # both run the toolchain that job pins, and a bump there touches no line + # of this scaffold. Same reason, same gate. + verify_contracts_toolchain_pin + verify_release_revision_stamp + verify_release_candidate_tag_isolation + # And the one piece of the release path whose absence has no local + # symptom: a dispatch that cannot hand in the detached provenance passes + # every proof and produces evidence acceptance refuses. + verify_release_provenance_wiring + + # The two validators gate every piece of rehearsal evidence, so the gate + # that runs on every change to them runs their self-tests too — without + # this they are proved only by the manually dispatched proof stages, + # which is to say only when somebody remembers. + note "source-binding verifier self-test" + "${SCRIPT_DIR}/test-source-binding.sh" + note "release Docker-tag selector self-test" + "${SCRIPT_DIR}/test-release-docker-tags.sh" + note "evidence-record validator self-test" + "${SCRIPT_DIR}/test-validate-evidence.sh" + note "native-runner matrix validator self-test" + "${SCRIPT_DIR}/test-rehearsal-matrix.sh" + note "cutover evidence-window capture self-test" + "${SCRIPT_DIR}/test-cutover-evidence-window.sh" + ) 2>&1 | tee "${log}" + + note "shell and workflow analysis recorded in ${log}" +} + +stage_solidity_proofs() { + note "building and testing the ECDSA contracts surface" + mkdir -p "${EVIDENCE_DIR}" + local log="${EVIDENCE_DIR}/solidity-ecdsa-proofs.log" + + command -v node >/dev/null 2>&1 || blocked "Node.js is required" + command -v corepack >/dev/null 2>&1 || + blocked "corepack is required (bundled with Node >= 16.9)" + + # The contracts workflow pins one exact Node release because newer ones have + # produced broken hardhat compile artifacts, and evidence from any other + # release is not that workflow's evidence. Which release that is comes out + # of the job this stage reproduces rather than out of a constant here: a + # constant would go on claiming parity after CI moved. + resolve_setup_node_version "${CONTRACTS_WORKFLOW}" "${CONTRACTS_JOB}" + local ci_node_version="${SETUP_NODE_VERSION}" + local node_version + node_version=$(node -p 'process.versions.node') + if [[ "${node_version}" != "${ci_node_version}" ]]; then + blocked "${CONTRACTS_WORKFLOW}'s ${CONTRACTS_JOB} job runs on Node \ +${ci_node_version} (found $(node -v)); switch with 'nvm install \ +${ci_node_version} && nvm use ${ci_node_version}' before running \ +solidity-proofs" + fi + + ( + cd "${REPO_ROOT}/solidity/ecdsa" + verify_source_binding + + # Reproduce the contracts workflow's install exactly: the + # Corepack-managed yarn release pinned in package.json's packageManager + # field and an immutable install on every run — never skipped, so a + # stale node_modules cannot masquerade as CI parity. Hardened mode is + # opted out for the same reason CI opts out: the lockfile carries + # legitimate npm-descriptor -> git-URL remaps that hardened mode + # rejects, while lockfile checksums stay enforced either way. + export YARN_ENABLE_HARDENED_MODE=0 + corepack enable + note "yarn $(yarn --version)" + yarn install --immutable + yarn build + yarn test + ) 2>&1 | tee "${log}" + + note "solidity proofs recorded in ${log}" +} + +# The compose services the rehearsals drive, and the two roles that decide +# what each one may be asked to prove. The prior node carries no gate, so it +# is the straggler negative control and — after rollback — the only binary +# allowed to run a homogeneous legacy ceremony; the R1 nodes are the release +# under test. +REHEARSAL_PRIOR_SERVICE="prior-node" +REHEARSAL_R1_SERVICES=("r1-node-1" "r1-node-2") + +# This bounds only the client-info readiness probe after Compose starts a +# process. It is not a service-manager termination grace: every R1 stop derives +# that independently reviewed bound from release-manifest.json. +NODE_REACHABILITY_TIMEOUT_SECONDS=600 + +# One compose project per rehearsal so `docker compose` resolves the fleet, +# its volumes, and its two networks by name from any working directory, and +# so a rollback rehearsal never adopts a cutover rehearsal's containers. +compose_project() { printf 'pr4109-%s\n' "${REHEARSAL_GATE}"; } + +compose() { + docker compose --project-name "$(compose_project)" \ + --file "${SCRIPT_DIR}/compose.rehearsal.yaml" "$@" +} + +# The internal protocol network, which is where every evidence probe attaches. +# The compose file publishes no node port to the host on purpose, so a probe +# reaching a node from outside this network would be reading something the +# rehearsal topology says is unreachable. +rehearsal_network() { printf '%s_rehearsal\n' "$(compose_project)"; } + +# The client-info port a node serves its evidence on, read out of that node's +# own config rather than assumed. The parser is section-aware because `port` +# is not a unique key in this config format — the Bitcoin and network sections +# carry their own — so a scan for the first `port =` would scrape whichever +# section happened to come first. +clientinfo_port() { + local service="$1" config="${KEYSTORE_DIR}/$1/config.toml" port + port="$(awk ' + /^[[:space:]]*\[/ { + section = $0 + sub(/^[[:space:]]*\[/, "", section) + sub(/\].*$/, "", section) + next + } + section == "clientInfo" && /^[[:space:]]*port[[:space:]]*=/ { + value = $0 + sub(/^[^=]*=[[:space:]]*/, "", value) + sub(/[[:space:]]*(#.*)?$/, "", value) + print value + exit + } + ' "${config}")" + if [[ ! "${port}" =~ ^[0-9]+$ ]] || ((port == 0)); then + blocked "${config} declares no nonzero clientInfo.port; the rehearsal \ +reads every gauge, gate state, and roster snapshot from that port and the \ +fleet publishes none of them to the host, so a node without one can be \ +started but never evidenced" + fi + printf '%s\n' "${port}" +} + +# Read one node's client-info endpoint from inside the internal protocol +# network. Attaching the probe there rather than publishing a host port keeps +# the reachability the rehearsal evidences identical to the one the compose +# topology defines, and is what lets the rollback gate's network-quarantine +# steps mean anything: a quarantined node becomes unreachable to this probe +# because it is genuinely off the network, not because a flag was flipped. +probe_get() { + local service="$1" path="$2" port + port="$(clientinfo_port "${service}")" + docker run --rm --network "$(rehearsal_network)" "${PROBE_IMAGE_DIGEST}" \ + wget --quiet --output-document=- --timeout=10 \ + "http://${service}:${port}${path}" 2>/dev/null +} + +probe_diagnostics() { probe_get "$1" /diagnostics; } +probe_metrics() { probe_get "$1" /metrics; } + +# One JSON-RPC call against the rehearsal chain, from the egress network the +# fleet reaches the chain over — the same reachability the nodes have, so an +# endpoint this rehearsal can question is one they could act on. +chain_rpc() { + local method="$1" params="$2" + docker run --rm --network "$(compose_project)_chain-egress" \ + "${PROBE_IMAGE_DIGEST}" \ + wget --quiet --output-document=- --timeout=10 \ + --header='Content-Type: application/json' \ + --post-data="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"${method}\",\ +\"params\":${params}}" \ + "${ETH_RPC_URL}" 2>/dev/null +} + +# What the chain says became of one transaction: "succeeded ", +# "reverted ", "pending" while no receipt exists yet, or "unreadable" +# when the endpoint could not be questioned or did not answer in a form this +# rehearsal can read. +transaction_receipt() { + local tx="$1" response + response="$(chain_rpc eth_getTransactionReceipt "[\"${tx}\"]")" || { + printf 'unreadable' + return 0 + } + printf '%s' "${response}" | node -e ' + let raw = ""; + process.stdin.on("data", (d) => (raw += d)); + process.stdin.on("end", () => { + let body; + try { + body = JSON.parse(raw); + } catch (e) { + process.stdout.write("unreadable"); + return; + } + if (body === null || typeof body !== "object" || body.error) { + process.stdout.write("unreadable"); + return; + } + const receipt = body.result; + if (receipt === null || receipt === undefined) { + process.stdout.write("pending"); + return; + } + const status = receipt.status; + const block = receipt.blockNumber; + if (typeof status !== "string" || typeof block !== "string" || + !/^0x[0-9a-f]+$/.test(block)) { + process.stdout.write("unreadable"); + return; + } + process.stdout.write((status === "0x1" ? "succeeded" : "reverted") + + " " + String(parseInt(block, 16))); + }); + ' 2>/dev/null || printf 'unreadable' +} + +# The chain id the questioned endpoint reports, or "unreadable". +endpoint_chain_id() { + local response + response="$(chain_rpc eth_chainId '[]')" || { + printf 'unreadable' + return 0 + } + printf '%s' "${response}" | node -e ' + let raw = ""; + process.stdin.on("data", (d) => (raw += d)); + process.stdin.on("end", () => { + let body; + try { + body = JSON.parse(raw); + } catch (e) { + process.stdout.write("unreadable"); + return; + } + const id = (body || {}).result; + if (typeof id !== "string" || !/^0x[0-9a-f]+$/.test(id)) { + process.stdout.write("unreadable"); + return; + } + process.stdout.write(String(parseInt(id, 16))); + }); + ' 2>/dev/null || printf 'unreadable' +} + +# True when a node answers its client-info port at all. Used both ways: to +# wait for a node to come up, and to prove a quarantined one has gone. +# +# This is one node's own HTTP surface and nothing more. It says a node answers +# or does not answer, which is weaker than the barrier below needs: a candidate +# whose client-info listener died while its protocol stack kept running answers +# nothing and is still on the network. +node_reachable() { probe_get "$1" /diagnostics >/dev/null 2>&1; } + +# The compose project prefix every rehearsal gate of this scaffold runs under. +# A gate's own project name is compose_project; this is what makes another +# gate's leftovers recognizable as this scaffold's rather than as some +# unrelated container that happens to share the daemon. +REHEARSAL_PROJECT_PREFIX="pr4109-" + +# Every container on this daemon that a rollback barrier has to account for, +# one per line as "