Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed

- Clarified inclusive `--fail-on-category` thresholds in CLI help and docs, including the `permissions:0` always-fail pitfall (#217).

## [0.1.4] - 2026-06-12

### Security
Expand Down
12 changes: 11 additions & 1 deletion docs/platform/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ When `-o` is set, format determines serialization. SARIF uses `reporting/sarif.p
| `--fail-on-critical` | false | Exit **1** if any critical finding |
| `--min-score` | — | Exit **1** if legacy `score.overall` < N (0–100) |
| `--max-critical` | — | Exit **1** if critical count > N |
| `--fail-on-category` | — | Repeatable. Format: `category:limit`. Exit **1** when **legacy** category score ≥ limit |
| `--fail-on-category` | — | Repeatable. Format: `category:limit`. Exit **1** when **legacy** category score ≥ limit. Inclusive: `permissions:0` fails even at score 0; use `permissions:1` to allow a zero-point pass |
| `--scoring` | `both` | `legacy`, `v2`, or `both` — enable multi-factor scoring |
| `--min-security-score` | — | Exit **1** if v2 benchmark security score < N (requires `--scoring v2` or `both`) |
| `--max-absolute-risk` | — | Exit **1** if v2 `absolute_risk` > N (requires `--scoring v2` or `both`) |
Expand All @@ -99,6 +99,12 @@ When `-o` is set, format determines serialization. SARIF uses `reporting/sarif.p

Valid **legacy** category keys: `permissions`, `injection`, `execution`, `data_leakage`, `attack_chains`, `shadowing`, `jailbreak`. Category gates apply to v1 tiles only — not `category_scores_v2`. See [Scoring developer guide](../reporting/scoring-guide.md).

`--fail-on-category` budgets legacy category **risk points**, not finding counts. The threshold is inclusive:

- `permissions:10` fails when the permissions score is 10 or higher.
- `permissions:1` allows a zero-point permissions category to pass and fails once it reaches 1 or more.
- `permissions:0` is an always-fail gate because score 0 still meets `>= 0`.

### Terminal UI flags

| Flag | Default | Description |
Expand Down Expand Up @@ -222,6 +228,10 @@ mcts scan ./repo/ \
--fail-on-category permissions:10 \
--fail-on-category injection:15

# Strict zero-risk category gate
mcts scan ./repo/ \
--fail-on-category permissions:1

# Fuzz telemetry replay
mcts scan ./server.py --runtime-events fuzz.json

Expand Down
2 changes: 1 addition & 1 deletion docs/reporting/scoring-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ Exit code **1** when a gate fails; **2** for usage/consent errors.
| `--max-critical N` | `summary.critical > N` |
| `--fail-on-category KEY:LIMIT` | Legacy category score ≥ LIMIT |

Category gates are **inclusive** at the limit: `--fail-on-category permissions:10` fails when permissions category score is **10 or higher**.
Category gates are **inclusive** at the limit: `--fail-on-category permissions:10` fails when permissions category score is **10 or higher**. To require a zero-risk permissions category, use `permissions:1`; a limit of `permissions:0` fails even when the score is 0 because `0 >= 0`.

### v2 gates (shipped)

Expand Down
3 changes: 2 additions & 1 deletion src/mcts/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,8 @@ def scan(
help=(
"Exit 1 when legacy category risk score meets or exceeds threshold (inclusive). "
"Legacy v1 tiles only — not category_scores_v2. "
"e.g. permissions:0 fails when score is 0 or more. Repeatable."
"Example: permissions:10 fails at 10+ risk points; permissions:0 always fails, "
"even when the category score is 0. Use permissions:1 to allow a zero-point pass. Repeatable."
),
),
] = None,
Expand Down
Loading