Scoped (schedule-triggered) mutation runs in mutation-cargo.yml are pinned
to a single shard regardless of the shard-count input — the input's own
description says "Shard fan-out for full (workflow_dispatch) runs; scoped
scheduled runs always use a single shard." When the window-hours
change-detection accumulates a large batch of changed files, that single
shard can outgrow the timeout-minutes ceiling before finishing, and the
run is cancelled with the remaining mutants never evaluated.
Evidence
Run 29475461743
(leynos/rstest-bdd, schedule trigger, 2026-07-16):
- Job
mutation / mutants (., crates/cargo-bdd/src/registry.rs …) ran from
06:01:08 to 11:01:29 UTC — almost exactly the configured
timeout-minutes: 300, then GitHub Actions cancelled it ("The operation
was canceled").
INPUT_SHARD_COUNT: 1 (single shard, despite shard-count: 6 being
passed as the caller's fan-out input for full runs).- The changed-file window matched 65 files, and cargo-mutants reported
"Found 661 mutants to test".
- Unmutated baseline succeeded in 154s build + 171s test.
- Only 281 of the 661 mutants were evaluated before cancellation (81
caught, 200 unviable, 0 missed, 0 timeout — outcomes.json from the
uploaded mutation-report-root-0 artefact). That is roughly 63
seconds/mutant post-baseline; completing all 661 mutants at that rate
would need on the order of 11–12 hours, well over twice the 300-minute
budget.
- No survivors were reported, but that is an artefact of the run being cut
short at 42% completion, not a clean pass — 58% of mutants in that
window were never tested.
A related, less clear-cut case: run
29075480770
(schedule, 2026-07-10) ran a single scoped shard covering 1,088 mutants
for 3h36m before being cancelled by "The runner has received a shutdown
signal", i.e. under the 300-minute ceiling but still a very large,
un-sharded scoped batch. This may be an unrelated infrastructure blip, but
it is consistent with the same underlying fragility: large scoped batches
run as one long, un-fanned job.
Proposed next step
Extend shard fan-out to scoped (schedule-triggered) runs when the detected
batch is large, rather than always using a single shard. Concretely:
- Add a scoped-run shard count (e.g. a
scoped-shard-count input, default
1 to preserve current behaviour for small windows) so callers with a
wide window-hours and a busy repository can opt into fan-out for
scheduled runs too.
- Alternatively, compute the shard count dynamically in the
detect job
from the matched file count (or from a quick cargo mutants --list
count), sizing each shard so its expected duration stays comfortably
under timeout-minutes. At the ~60–65s/mutant throughput observed here,
capping each shard at roughly 200–250 mutants (leaving headroom under
the 300-minute ceiling) would need at least 3 shards for the 661-mutant
case above.
Either approach avoids scoped runs silently truncating coverage when a
detection window happens to catch a large batch of changed files.
Scoped (schedule-triggered) mutation runs in
mutation-cargo.ymlare pinnedto a single shard regardless of the
shard-countinput — the input's owndescription says "Shard fan-out for full (workflow_dispatch) runs; scoped
scheduled runs always use a single shard." When the
window-hourschange-detection accumulates a large batch of changed files, that single
shard can outgrow the
timeout-minutesceiling before finishing, and therun is cancelled with the remaining mutants never evaluated.
Evidence
Run 29475461743
(leynos/rstest-bdd,
scheduletrigger, 2026-07-16):mutation / mutants (., crates/cargo-bdd/src/registry.rs …)ran from06:01:08 to 11:01:29 UTC — almost exactly the configured
timeout-minutes: 300, then GitHub Actions cancelled it ("The operationwas canceled").
INPUT_SHARD_COUNT: 1(single shard, despiteshard-count: 6beingpassed as the caller's fan-out input for full runs).
"Found 661 mutants to test".
caught, 200 unviable, 0 missed, 0 timeout —
outcomes.jsonfrom theuploaded
mutation-report-root-0artefact). That is roughly 63seconds/mutant post-baseline; completing all 661 mutants at that rate
would need on the order of 11–12 hours, well over twice the 300-minute
budget.
short at 42% completion, not a clean pass — 58% of mutants in that
window were never tested.
A related, less clear-cut case: run
29075480770
(
schedule, 2026-07-10) ran a single scoped shard covering 1,088 mutantsfor 3h36m before being cancelled by "The runner has received a shutdown
signal", i.e. under the 300-minute ceiling but still a very large,
un-sharded scoped batch. This may be an unrelated infrastructure blip, but
it is consistent with the same underlying fragility: large scoped batches
run as one long, un-fanned job.
Proposed next step
Extend shard fan-out to scoped (schedule-triggered) runs when the detected
batch is large, rather than always using a single shard. Concretely:
scoped-shard-countinput, default1 to preserve current behaviour for small windows) so callers with a
wide
window-hoursand a busy repository can opt into fan-out forscheduled runs too.
detectjobfrom the matched file count (or from a quick
cargo mutants --listcount), sizing each shard so its expected duration stays comfortably
under
timeout-minutes. At the ~60–65s/mutant throughput observed here,capping each shard at roughly 200–250 mutants (leaving headroom under
the 300-minute ceiling) would need at least 3 shards for the 661-mutant
case above.
Either approach avoids scoped runs silently truncating coverage when a
detection window happens to catch a large batch of changed files.