diff --git a/CHANGELOG.md b/CHANGELOG.md index c81d031..1bfb1dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- Clarified inclusive `--fail-on-category` thresholds in CLI help and docs, including the `permissions:0` always-fail pitfall (#217). + ## [0.1.4] - 2026-06-12 ### Security diff --git a/docs/platform/cli.md b/docs/platform/cli.md index 407af2e..94b8bfb 100644 --- a/docs/platform/cli.md +++ b/docs/platform/cli.md @@ -87,7 +87,7 @@ When `-o` is set, format determines serialization. SARIF uses `reporting/sarif.p | `--fail-on-critical` | false | Exit **1** if any critical finding | | `--min-score` | — | Exit **1** if legacy `score.overall` < N (0–100) | | `--max-critical` | — | Exit **1** if critical count > N | -| `--fail-on-category` | — | Repeatable. Format: `category:limit`. Exit **1** when **legacy** category score ≥ limit | +| `--fail-on-category` | — | Repeatable. Format: `category:limit`. Exit **1** when **legacy** category score ≥ limit. Inclusive: `permissions:0` fails even at score 0; use `permissions:1` to allow a zero-point pass | | `--scoring` | `both` | `legacy`, `v2`, or `both` — enable multi-factor scoring | | `--min-security-score` | — | Exit **1** if v2 benchmark security score < N (requires `--scoring v2` or `both`) | | `--max-absolute-risk` | — | Exit **1** if v2 `absolute_risk` > N (requires `--scoring v2` or `both`) | @@ -99,6 +99,12 @@ When `-o` is set, format determines serialization. SARIF uses `reporting/sarif.p Valid **legacy** category keys: `permissions`, `injection`, `execution`, `data_leakage`, `attack_chains`, `shadowing`, `jailbreak`. Category gates apply to v1 tiles only — not `category_scores_v2`. See [Scoring developer guide](../reporting/scoring-guide.md). +`--fail-on-category` budgets legacy category **risk points**, not finding counts. The threshold is inclusive: + +- `permissions:10` fails when the permissions score is 10 or higher. +- `permissions:1` allows a zero-point permissions category to pass and fails once it reaches 1 or more. +- `permissions:0` is an always-fail gate because score 0 still meets `>= 0`. + ### Terminal UI flags | Flag | Default | Description | @@ -222,6 +228,10 @@ mcts scan ./repo/ \ --fail-on-category permissions:10 \ --fail-on-category injection:15 +# Strict zero-risk category gate +mcts scan ./repo/ \ + --fail-on-category permissions:1 + # Fuzz telemetry replay mcts scan ./server.py --runtime-events fuzz.json diff --git a/docs/reporting/scoring-spec.md b/docs/reporting/scoring-spec.md index e7c3560..1279801 100644 --- a/docs/reporting/scoring-spec.md +++ b/docs/reporting/scoring-spec.md @@ -212,7 +212,7 @@ Exit code **1** when a gate fails; **2** for usage/consent errors. | `--max-critical N` | `summary.critical > N` | | `--fail-on-category KEY:LIMIT` | Legacy category score ≥ LIMIT | -Category gates are **inclusive** at the limit: `--fail-on-category permissions:10` fails when permissions category score is **10 or higher**. +Category gates are **inclusive** at the limit: `--fail-on-category permissions:10` fails when permissions category score is **10 or higher**. To require a zero-risk permissions category, use `permissions:1`; a limit of `permissions:0` fails even when the score is 0 because `0 >= 0`. ### v2 gates (shipped) diff --git a/src/mcts/cli/main.py b/src/mcts/cli/main.py index 8c92d7c..0107df4 100644 --- a/src/mcts/cli/main.py +++ b/src/mcts/cli/main.py @@ -318,7 +318,8 @@ def scan( help=( "Exit 1 when legacy category risk score meets or exceeds threshold (inclusive). " "Legacy v1 tiles only — not category_scores_v2. " - "e.g. permissions:0 fails when score is 0 or more. Repeatable." + "Example: permissions:10 fails at 10+ risk points; permissions:0 always fails, " + "even when the category score is 0. Use permissions:1 to allow a zero-point pass. Repeatable." ), ), ] = None,