diff --git a/docs/adr-004-pin-only-mutation-testing-contract.md b/docs/adr-004-pin-only-mutation-testing-contract.md new file mode 100644 index 00000000..29cc2a21 --- /dev/null +++ b/docs/adr-004-pin-only-mutation-testing-contract.md @@ -0,0 +1,68 @@ +# Architectural decision record (ADR) 004: pin-only mutation-testing contract + +## Status + +Accepted (2026-07-22): Keep Whitaker's mutation-testing adoption pin-only and +contract-test its declared configuration rather than claiming parity with the +continuous integration (CI) test baseline. + +## Date + +2026-07-22. + +## Context and problem statement + +Whitaker delegates mutation testing to the shared +`leynos/shared-actions/.github/workflows/mutation-cargo.yml` workflow. The +caller supplies workspace paths, exclusion globs, and `--all-features`, but the +shared workflow cannot reproduce the per-invocation `RUSTFLAGS` used by the +repository's `test` and `typecheck` Makefile targets. + +Those flags are required by the `whitaker`, `function_attrs_follow_docs`, +`module_max_lines`, and `no_expect_outside_tests` crates when the +`dylint-driver` feature enables `feature(rustc_private)`. Applying the flags +workspace-wide is not viable because, as documented in `.cargo/config.toml`, +they break `cargo install` for `whitaker-installer`. + +The decision is whether the mutation workflow should claim a workspace baseline +equivalent to `make test`, or expose the closest safe mutation scope and test +only that declared configuration. + +## Decision drivers + +- Keep the mutation workflow usable without breaking installer builds. +- Describe the mutation scope honestly rather than imply CI parity. +- Detect accidental caller drift while permitting automated commit-pin updates. +- Keep mutation testing informational and independent of pull-request gates. + +## Options considered + +| Option | Shared-workflow feasibility | Mutation coverage | Operational risk | CI-baseline claim / verification | Rationale | +| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | +| A: require full CI-baseline parity | Unsafe and unimplementable: the current shared workflow cannot inject the required flags per crate. | Aims for full workspace coverage equivalent to `make test`. | Applying the flags workspace-wide breaks `cargo install` for `whitaker-installer`. | Would claim parity, but cannot verify it through the shared workflow. | Missing per-crate flags make this option unsafe and unimplementable. | +| B: exclude every crate that needs dynamic-linking flags | Runs reliably with the shared workflow by removing the affected crates. | Omits important production code, making the reported scope less useful. | Lower run risk, but accepts the risk of untested affected crates. | Makes no full-baseline claim; verification covers only the reduced scope. | Reliable execution deliberately omits important affected crates. | +| C: adopt a pin-only declared-configuration contract | Closest safe approximation supported by the shared workflow. | Retains useful informational coverage without claiming full workspace coverage. | Keeps mutation testing informational and independent of pull-request gates. | Verifies the caller's declared security and configuration shape, not a full workspace assertion. | A declared-shape contract is the closest safe approximation, not full workspace parity. | + +_Table 1: Comparison of mutation-testing options._ + +## Decision outcome / proposed direction + +Adopt Option C. + +In the context of shared mutation testing, facing incompatible per-crate +compiler-flag requirements, Whitaker chooses a pin-only declared-configuration +contract over either claimed CI parity or excluding every affected crate, to +retain useful informational mutation coverage while accepting that the run +cannot reproduce `make test` crate-for-crate. + +The caller remains pinned to a full commit SHA. Its contract test validates the +pin shape, permissions, concurrency, triggers, and declared inputs. It does not +hard-code the pin value or assert successful full-workspace mutation testing. + +## Known risks and limitations + +- Mutation results do not certify the same crate scope as `make test`. +- Crates needing the dynamic-linking flags may fail or provide incomplete + mutation results under the shared workflow. +- The decision must be revisited if the shared workflow gains a safe per-crate + mechanism for supplying the required flags. diff --git a/docs/contents.md b/docs/contents.md index 446be541..5f3cb3b8 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -79,6 +79,10 @@ pipeline](adr-003-formal-proof-strategy-for-clone-detector-pipeline.md) records the formal verification direction for the clone detector pipeline and its proof boundaries. +- [Architectural decision record (ADR) 004: pin-only mutation-testing + contract](adr-004-pin-only-mutation-testing-contract.md) records why the + shared mutation workflow tests its declared configuration without claiming + parity with the continuous integration test baseline. ## Requests for comments diff --git a/docs/developers-guide.md b/docs/developers-guide.md index fb6add6d..92b1938d 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -291,6 +291,91 @@ make check-fmt # Verify formatting make fmt # Apply formatting ``` +## Mutation-testing workflow contract tests + +Whitaker runs scheduled, informational mutation testing through a thin caller +workflow, [`.github/workflows/mutation-testing.yml`](../.github/workflows/mutation-testing.yml), +which delegates to the shared reusable workflow +`leynos/shared-actions/.github/workflows/mutation-cargo.yml`. The heavy +lifting — running `cargo-mutants`, sharding, and summarizing survivors — lives +in `shared-actions`; this repository carries only declarative configuration. +The run is **informational only**: it never gates a pull request. Survivors +are reported through the job summary and downloadable artefacts so they can be +triaged into tests, not enforced as a blocking check. + +The workflow runs in two modes. A **daily schedule** (04:50 UTC) fires a +change-scoped run that mutates only the source files touched within the +detection window, so quiet days are cheap no-ops. A **manual dispatch** (the +Actions "Run workflow" control) mutates the whole workspace; select a branch +in that control to exercise a feature branch. + +The caller passes a small set of configuration inputs, each carrying intent: + +- `paths` — `src/,common/,crates/,installer/,suite/`, the workspace member + prefixes rooted at the repository root; there are no top-level `examples/` + or `benches/` directories to add. +- `exclude-globs` — scaffolding whose surviving mutants would be noise rather + than genuine test gaps: the `rustc_*` proxy crates and the `clippy_utils` + stub (both re-export compiler internals), each lint's `ui/` and `examples/` + dylint fixtures, and the shared test infrastructure in `src/testing` and + `common/src/test_support`. +- `extra-args` — `--all-features`, matching the feature baseline the + Makefile's `CARGO_FLAGS` uses for `make test`, so feature-gated code is not + reported as untested. + +Unlike the excludes applied by `make test`'s `TEST_EXCLUDES`, the mutation +caller does **not** exclude the `whitaker` root crate or the +`function_attrs_follow_docs`, `module_max_lines`, and `no_expect_outside_tests` +lint crates from mutation scope (only their `ui/`/`examples/` fixtures are +excluded). Those crates enable `feature(rustc_private)` under the +`dylint-driver` feature and need the dynamic-linking `RUSTFLAGS` +(`-C prefer-dynamic -Z force-unstable-if-unmarked`) that the `test` and +`typecheck` Makefile targets inject per invocation — see the note in +[`.cargo/config.toml`](../.cargo/config.toml), which deliberately keeps that +flag out of workspace-wide configuration because it would break `cargo +install` for `whitaker-installer`. The shared mutation workflow has no +equivalent per-crate RUSTFLAGS step, so it cannot reproduce +`TEST_CARGO_FLAGS` faithfully across the whole workspace. Consequently this +adoption is **pin-only**: the caller declares the best approximation of the CI +scope it safely can, rather than a `--test-workspace` run that mirrors +`make test` crate-for-crate, and the contract test asserts only that this +declared configuration holds — not that a full workspace mutation baseline +passes. [ADR 004](adr-004-pin-only-mutation-testing-contract.md) records this +decision, its alternatives, and the accepted limitations. + +The `uses:` reference pins the shared workflow to a full 40-character commit +SHA rather than a branch or tag, so a force-push upstream cannot silently +change what runs here. The contract test asserts only that the pin is a full +commit SHA, not a particular value, so Dependabot bumps it automatically +without any accompanying test edit. + +### Workflow contract tests + +Because the caller is configuration rather than code, a contract test suite, +[`tests/workflow_contracts/mutation_testing_test.py`](../tests/workflow_contracts/mutation_testing_test.py), +pins the shape it must uphold, failing the pull request when the caller +drifts — repointing the pin at a branch, widening the token scope, or +dropping a configuration input — rather than letting the breakage surface +only in a scheduled run. Run it locally with: + +```sh +make test-workflow-contracts +``` + +which wraps `uv run --with 'pytest>=8' --with 'pyyaml>=6' pytest +tests/workflow_contracts -q`. The suite validates: + +- the `uses:` reference targets `mutation-cargo.yml` pinned to a full commit + SHA; +- the `with:` block carries exactly the expected `paths`, `exclude-globs`, + and `extra-args` configuration described above; +- job permissions are least-privilege (`contents: read`, `id-token: write`) + and the workflow-level default token scope is empty; +- `concurrency` serializes runs per ref without cancelling one in progress; + and +- the triggers keep the daily schedule and a plain `workflow_dispatch` with no + legacy branch input. + ## Proof workflows Whitaker now ships repository-managed proof tooling for the formal verification