fix(cli,supply-chain): parse package.json as JSON, and send fatal errors to stderr

[Mark2Mac]

Two correctness fixes, both surfaced while driving the CLI from automation. They are independent of each other but small enough to review together; happy to split if you prefer.

Rebased onto main at 27fd962 (2.9.5). The second half changed shape since the last review — see "What changed in this rebase" at the end.

1. package.json was scanned line by line

_extract_packages_from_package_json walked the file looking for a line containing "dependencies", then read entries until a line starting with }. A manifest written on a single line — valid JSON, and what several generators emit — never enters the section:

$ cat package.json
{"name":"x","dependencies":{"express":"^4.18.0","lodash":"4.17.21"}}

$ skillspector scan .   # before
# no dependencies extracted, no supply-chain findings, no warning

That is not a noisy result, it is a blind one: the output is indistinguishable from a clean manifest.

It is now parsed with json.loads. Specifically:

• Version extraction is unchanged, including the caret handling. Only the parsing of the file changes.
• Line numbers survive. JSON parsing loses positions, so the entry is located in the raw text starting from the section header, which also means a package name that appears in "scripts" does not steal the line.
• A manifest that does not parse falls back to the previous line scan, so a truncated or templated file keeps today's behaviour instead of going silent.

2. Fatal diagnostics were written to stdout

console = Console() writes to stdout, and every Error: went through it. Any caller that separates the two streams — which is what automation does — throws the diagnosis away:

$ skillspector scan . --baseline old.yaml 2>err.log >out.json
$ cat err.log        # before: empty
$ head -c 60 out.json
Error: unsupported baseline version 1; expected 2. Version 1 ...

The message lands in the file that was supposed to hold the report, and the error log is empty. It is lost as a diagnostic and corrupting as output.

The mechanism now already exists. err_console = Console(stderr=True) arrived with the author-shipped baseline notices in #286, and those correctly go to stderr. This PR no longer introduces anything — it moves the diagnostics onto the console that is already there. Thirteen call sites: every message that prints and then raises typer.Exit, the two print_exception() calls in the --verbose branches, and the per-skill error inside the multi-skill loop.

--version deliberately stays on stdout: that is program output, not a diagnostic. Two Warning: messages also stay where they are — same stream, but a different question, and I would rather not widen this PR to decide it.

Tests

Seven cases for the parser (one-line manifest, compact manifest, line numbers preserved, name shadowed by scripts, invalid JSON falling back, non-object manifest, non-string specs ignored).

For the CLI, thirteen parametrized cases, one per fatal path, each asserting the message reaches stderr and never stdout. Verified red against the unmodified module: fourteen failures.

The fourteenth is a guard, and it is the part I would keep even if you drop the rest. When this PR was first opened there were twelve such sites and it moved eight of them — I missed four. A thirteenth arrived afterwards, in 2d198ab, the same commit that introduced err_console: the local-target gate prints its ValueError to stdout. The invariant has no enforcement, so it regenerates in both directions. test_cli_writes_no_error_styled_output_to_stdout parses cli.py and fails when error-styled output is written to the default console.

Gates, run locally

Gate	Result
make lint (ruff check src/ tests/)	All checks passed
make format-check (ruff format --check)	175 files already formatted
make test-unit	2210 passed, 14 skipped, 4 xfailed
tests/docker/smoke.sh (image built from this branch)	exit 0, including the GitHub URL scan
DCO	both commits signed off

Interaction with #344 measured rather than assumed: merged the two branches, no conflict, 2225 passed.

What changed in this rebase

• Split into two commits along the two fixes, so the CLI change can be read on its own.
• The err_console definition is gone from the diff — main has it now, and that was the only line this PR was colliding on.
• Five call sites added: the four this PR should have covered from the start (both --mcp-registry argument checks, the registry scan handler, and --baseline with --recursive) and the one that arrived with the local-target gate in 2.9.4.
• The two example tests became a table over every fatal path, plus the source guard.

[Mark2Mac]

Cross-PR note: this PR conflicts with #302, which also touches _extract_packages_from_package_json (it replaces the caret-stripping with a _pinned_npm_version predicate).

The reconciliation is one line — the JSON parser here calls that predicate instead of its own _npm_version_or_none — and with both applied the full suite is green (1585 passed, 14 skipped, 6 xfailed). I kept version extraction untouched in this PR precisely so the two stay orthogonal; happy to rebase onto #302 or the other way round, whichever you prefer to land first.

[rng1995]

[Automated SkillSpector Review]

Requesting changes. Parsing package.json as JSON fixes the compact-manifest blind spot and the fallback preserves best-effort handling of malformed manifests. The stderr change is incomplete, however: both generic --verbose exception branches still call console.print_exception(), so a fatal traceback is written to stdout. I reproduced this at the current head with a forced graph exception: exit 2, full traceback in stdout, empty stderr. Please route those tracebacks through err_console and add a stream-separation regression for the verbose path.

[rng1995]

Blocking: the generic --verbose handlers in both scan() and baseline() still call console.print_exception(), which writes the fatal traceback to stdout. I reproduced exit 2 with the full traceback in stdout and empty stderr. Please use err_console.print_exception() for those branches and add a regression that asserts stdout/stderr separation under --verbose.

[Mark2Mac]

Fixed in f617861 — the objection was correct, and it was the same defect this PR set out to fix, left in the one branch where it is hardest to notice.

Both generic handlers now print through err_console, so scan --verbose and baseline --verbose behave like the one-line error paths. Reproduced your case first: with graph.invoke raising, the traceback was in stdout and result.stderr was empty.

Regression covers both commands (tests/unit/test_cli.py): exit code 2, RuntimeError present in stderr and absent from stdout. Verified it fails on the previous commit.

grep -rn print_exception src/ now returns only those two lines, both on err_console. Full suite: 1572 passed, 12 skipped, 6 xfailed.

[Mark2Mac]

Rebased onto main (past #319) and updated. Both review points are addressed.

The blocker. You were right, and I had reproduced it the same way: console.print_exception() in the two generic --verbose branches wrote the traceback to stdout while stderr stayed empty. Both now go through err_console, and test_scan_verbose_traceback_goes_to_stderr / test_baseline_verbose_traceback_goes_to_stderr assert the separation by forcing a failure on the verbose path — they fail on the pre-fix code.

The reconciliation with #319, which is the reason for the rebase. #319 landed _pinned_npm_version, so this PR no longer carries its own _npm_version_or_none: it is deleted, and both the JSON path and the line-oriented fallback call the shared predicate. Version resolution is therefore identical to main — the only thing this PR changes is how the manifest is read.

One test moved with it: test_package_json_compact_keeps_versions asserted "^7.5.0" -> "7.5.0", which was correct against the old base and is wrong now. It asserts None, and its point is unchanged — the compact manifest must resolve exactly like the indented one.

The defect is still live on main today:

>>> _extract_packages_from_package_json('{"name":"x","dependencies":{"express":"^4.18.0","lodash":"4.17.21"}}')
[]

A valid one-line manifest yields no dependencies at all, silently — no error, no warning, just an empty supply-chain surface for that unit.

Full suite on the rebased branch: 1741 passed, 13 skipped, 4 xfailed. ruff check and ruff format --check clean at 0.15.19.

[rng1995]

[Automated SkillSpector Review]

Re-review: the previous stderr blocker is resolved—both verbose fatal traceback paths now use err_console, with stream-separation regressions—and the JSON package parser is well covered. However, the exact head fails ruff format --check: tests/unit/test_patterns_new.py would be reformatted. Please run Ruff format and update the PR; the focused functional suites otherwise pass (289 tests).

[rng1995]

Blocking CI issue: the exact head fails ruff format --check; Ruff reports this test file would be reformatted. Please run ruff format tests/unit/test_patterns_new.py and commit the result. The prior stderr blocker is resolved and the focused tests pass.

[rng1995]

@Mark2Mac - Please address review comments and resolve merge conflicts.

[Mark2Mac]

Both points addressed. New head bf5792e.

The lint failure. You were right, and the CI run on 01a92a4 names it precisely: lint was the only failing job (docker-smoke, changes, DCO Check, test-unit all passed). ruff format --check reported tests/unit/test_patterns_new.py, and the cause was mine: the previous rebase left two test methods with no blank line between them, and one manifest fixture written as an implicit string concatenation that Ruff collapses to a single line. Reproduced at 0.15.19 before fixing, so the failure was the code and not a version difference.

The formatting is squashed into the commit that introduces those tests rather than added as a separate style: commit — the file is only untidy because of that commit.

The conflicts. Rebased onto main at 2b408ee (2.9.3). One conflict, in the import block of static_patterns_supply_chain.py: #357 added os there, this PR adds json. Resolved as the union of the two. Nothing else in the module conflicted, and no behaviour changed.

Verified on the exact head:

• ruff check src/ tests/ — clean
• ruff format --check src/ tests/ — 165 files already formatted
• pytest -m "not integration and not provider" tests/ — 2045 passed, 13 skipped, 38 deselected, 4 xfailed
• both commits carry Signed-off-by

The defect this PR fixes is still live on main today:

>>> _extract_packages_from_package_json('{"name":"x","dependencies":{"express":"^4.18.0","lodash":"4.17.21"}}')
[]

A valid single-line manifest yields no dependencies at all — no error, no warning, just an empty supply-chain surface for that unit.

[Mark2Mac]

Both points addressed, and the second half of the PR is smaller than it was. New head 232a226, rebased onto main at 27fd962 (2.9.5).

The blocking comment. ruff format --check is clean on this head — 175 files already formatted. That was fixed on the previous head and survives the rebase.

The conflict. It was the err_console = Console(stderr=True) line, and main owns it now: #286 added exactly that console for the author-shipped baseline notices. So the definition is gone from this diff. What remains is only the move of the diagnostics onto a console that already exists — which makes the PR both smaller and, I think, easier to justify.

Two things I got wrong, both found while rebasing.

When this PR was first opened there were twelve console.print calls that print a diagnostic and then exit, and it moved eight of them. I missed four: both --mcp-registry argument checks, the registry scan handler, and --baseline with --recursive. They are covered now.

A thirteenth arrived afterwards — in 2d198ab, the same commit that introduced err_console. The local-target gate reports its ValueError on stdout:

        except ValueError as e:
            console.print(f"[red]Error:[/red] {e}")
            raise typer.Exit(code=2) from e

That is the argument for the last test better than anything in the PR body. The invariant regenerates in both directions because nothing enforces it, so test_cli_writes_no_error_styled_output_to_stdout parses cli.py and fails when error-styled output goes to the default console. If you would rather not carry a source-level guard, say so and I will drop it — but then this will come back.

The thirteen behavioural cases were verified red against the unmodified module: fourteen failures, one per path plus the guard.

Gates, run locally before pushing:

Gate	Result
make lint	All checks passed
make format-check	175 files already formatted
make test-unit	2210 passed, 14 skipped, 4 xfailed
tests/docker/smoke.sh, image built from this branch	exit 0, GitHub URL scan included
DCO	both commits signed off

Also merged with #344 to check the two do not fight over static_patterns_supply_chain.py: no conflict, 2225 passed.

Split into two commits along the two fixes, so the CLI change can be read on its own. CI has not run on a head of this PR since 01a92a4 — the workflow needs authorising again.