NOTICKET Add CLI-vs-MCP token efficiency benchmark harness

[alexander-rykhlitskiap]

Why

Andy asked in Slack whether we
have proof we can point to that the CLI helps users spend ~25% fewer tokens than MCP for
high-volume agent pipelines. We don't — so this adds the instrument that produces that number
instead of the number itself.

What

benchmarks/token-efficiency/ — runs the same natural-language Apollo task through headless Claude
twice per case and reports the token delta:

• cli arm — no MCP servers (--strict-mcp-config with an empty server map), Bash plus the
  shipped apollo-cli skill copied into the run's scratch cwd.
• mcp arm — only apollo-work (https://mcp.apollo.io/mcp), Bash denied so it can't
  shell out to the CLI and win on its behalf.

Everything else is pinned: same prompt, same model, same turn cap, a fresh mkdtemp cwd per run
(no CLAUDE.md, no project settings) and --settings '{}' so the operator's global settings,
plugins and enabled MCP servers don't leak into either arm.

node benchmarks/token-efficiency/run.ts            # Node >= 23.6

Three cases, spanning the shapes that drive the difference:

Case	Shape
single-enrich	1 call, small payload — floor case, isolates fixed per-session overhead
filtered-search	1 call, compact 11-field payload — intended as the CLI's projection win; measurement showed there is little to project away
chained-pipeline	~11–21 calls with per-company fan-out — the "high-volume pipeline" the claim is about

Reports raw totalTokens, a cost-weighted billableEquivalent (cache reads at 0.1x, cache writes
at 1.25x) and billed USD; median of N reps per arm; per-case percentages plus both an
equal-weighted mean and a pooled average, because those differ and the difference matters.

Results

Executed. results/2026-08-21-sonnet-5.md —
claude-sonnet-5, Claude Code 2.1.238, 3 reps/arm/case, medians:

Case	MCP tokens	CLI tokens	CLI saves
Single company enrichment	262,979	120,786	+54.1%
Filtered people search	405,040	566,471	−39.9%
Chained multi-entity pipeline	2,748,188	1,061,312	+61.4%

Equal-weighted mean +25.2%; pooled +48.8%. All 18 runs passed the completion gate.

The mean lands on the claimed "25% fewer tokens" almost exactly — but it is the average of +54%,
−40% and +61%, so no real workload is described by "about a quarter". The direction flips with
workload shape. The CLI wins where it can chain calls and jq-project before results reach the
context window.

On filtered-search, a traced rep of each arm shows projection working as advertised — MCP put
15,730 B into context vs the CLI's jq-projected 2,401 B (55,022 B vs 9,419 B across all tool
results) — but it doesn't matter: every model call re-reads the whole conversation, so ~11k tokens
saved is ~2% of a 400k-token run while turn count is ~98%. And Apollo's people search returns only
11 fields per person (465 B each), so there was little fat to trim in the first place.

One finding worth separate attention: in this client version the ~90 apollo-work tools are
deferred behind ToolSearch, not loaded up front — so the usual "MCP pays for every schema every
session" justification does not apply here. Any CLI advantage should be attributed to payload
projection and call chaining instead.

Two defects worth reviewing, because they produced confident wrong numbers

The first sweep reported +33.9% / −68.7% / −50.3% (mean −28.4%). Both causes are in the results doc:

1. No completion gate. The MCP server's credit-cost annotations made the agent stop and ask
   "this will cost 10 credits, proceed?" — cheap, and not the task. Scored naively, bailing out
   looks like efficiency. Both arms now get an identical pre-approval prompt suffix and every run
   must pass gradeComplete().
2. A $1 budget cap that culled only the expensive arm — 3 errored runs, all MCP, on the two
   costly cases.

Verification

benchmarks/ sits outside tsconfig.json's include (src/**/*.ts) and outside package.json's
files, so it is neither typechecked by npm run typecheck nor published. It has, however, now been
run end to end 20+ times against the live API. Known rough edge: num_turns from the CLI is
unreliable (one 2.75M-token run reported num_turns: 1 while returning a complete answer), so turn
counts in the output are indicative only.

[orca-security-us]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[kodem-security]

Kodem Security Scan Summary

Status	Check	Issues by Severity	Link
✅ Passed	Code Weakness Security Report	🟧 M: 1   🟨 L: 1	Link to Check

Note: Kai marked all findings as likely false positives.