Summary
A scheduled mutation-testing run hit the 300-minute step timeout on a single shard after a large diff (many files touched within the 25-hour detection window) landed in one of the fixed eight shards, leaving mutation testing incomplete for that run.
Run details
- Run 29685703301, scheduled, 2026-07-19, FAILURE, ~311.6 minutes.
- Shard scope:
src/agent/commands/system.rs src/app.rs src/app/tests.rs src/boot_screen/tests.rs ... (a long file list, not a single-file diff like other scheduled runs).
- Failing step:
Run mutation testing, ran from 11:48 to 16:48:41 UTC — exactly the configured timeout-minutes: 300 — then was killed by the runner as timed out.
- Configured inputs:
window-hours: 25, shard-count: 8, timeout-multiplier: 3, timeout-minutes: 300 (from the mutation / detect job log).
- Partial survivors were still recorded before the timeout, for example:
src/import/mod.rs:48/51-53 (ImportStats::is_empty), src/cli/mod.rs:241 (Cli::should_run_agent), src/bootstrap.rs:277 (restrict_file_permissions) — these are incidental and not the focus of this issue since the run did not complete.
Root cause
The scheduled workflow always uses a fixed shard-count: 8 regardless of how many files changed within the 25-hour detection window. On a day when an unusually large diff landed (multiple substantial files, including src/app.rs), one shard received a disproportionate share of mutants and could not finish within the fixed 300-minute step timeout. This differs from both other failures in this sweep: it is neither a setup/toolchain failure nor a survivor-harvest question, but a scheduling/sizing problem specific to how the changed-files window interacts with a fixed shard count.
Separately (and not the primary cause of this run's failure, since mutation testing had already timed out by then), the job's Post Setup Rust step also failed with the same symptom as leynos/shared-actions#366 (local composite action resolved against the relocated workflow-src path); a comment citing this run has been added to that issue.
Proposed next step
Make shard sizing responsive to diff size for scheduled runs: either scale shard-count (or timeout-multiplier) with the number of changed files/mutants reported by the detect job, or cap the per-shard file count so no single shard can exceed the step timeout regardless of how much changed since the last successful window.
Summary
A scheduled mutation-testing run hit the 300-minute step timeout on a single shard after a large diff (many files touched within the 25-hour detection window) landed in one of the fixed eight shards, leaving mutation testing incomplete for that run.
Run details
src/agent/commands/system.rs src/app.rs src/app/tests.rs src/boot_screen/tests.rs ...(a long file list, not a single-file diff like other scheduled runs).Run mutation testing, ran from 11:48 to 16:48:41 UTC — exactly the configuredtimeout-minutes: 300— then was killed by the runner as timed out.window-hours: 25,shard-count: 8,timeout-multiplier: 3,timeout-minutes: 300(from themutation / detectjob log).src/import/mod.rs:48/51-53(ImportStats::is_empty),src/cli/mod.rs:241(Cli::should_run_agent),src/bootstrap.rs:277(restrict_file_permissions) — these are incidental and not the focus of this issue since the run did not complete.Root cause
The scheduled workflow always uses a fixed
shard-count: 8regardless of how many files changed within the 25-hour detection window. On a day when an unusually large diff landed (multiple substantial files, includingsrc/app.rs), one shard received a disproportionate share of mutants and could not finish within the fixed 300-minute step timeout. This differs from both other failures in this sweep: it is neither a setup/toolchain failure nor a survivor-harvest question, but a scheduling/sizing problem specific to how the changed-files window interacts with a fixed shard count.Separately (and not the primary cause of this run's failure, since mutation testing had already timed out by then), the job's
Post Setup Ruststep also failed with the same symptom as leynos/shared-actions#366 (local composite action resolved against the relocatedworkflow-srcpath); a comment citing this run has been added to that issue.Proposed next step
Make shard sizing responsive to diff size for scheduled runs: either scale
shard-count(ortimeout-multiplier) with the number of changed files/mutants reported by thedetectjob, or cap the per-shard file count so no single shard can exceed the step timeout regardless of how much changed since the last successful window.