NOTICKET Add CLI-vs-MCP token efficiency benchmark harness

benchmarks/token-efficiency/.gitignore

@@ -0,0 +1 @@
+results.json

benchmarks/token-efficiency/README.md

@@ -0,0 +1,128 @@
+# Token-efficiency benchmark — Apollo CLI vs Apollo MCP
+
+Reproducible measurement of how many tokens an agent spends to complete the *same* Apollo task
+through the `apollo` CLI versus through the `apollo-work` MCP server (`https://mcp.apollo.io/mcp`).
+
+Built to answer a specific question: **is there a number we can point to when we say the CLI costs
+fewer tokens than MCP for high-volume agent pipelines?** This harness produces that number; it does
+not assume it.
+
+## Latest result
+
+[`results/2026-08-21-sonnet-5.md`](results/2026-08-21-sonnet-5.md) — `claude-sonnet-5`, 3 reps/arm/case:
+
+| Case | MCP tokens | CLI tokens | CLI saves |
+|---|---:|---:|---:|
+| Single company enrichment | 262,979 | 120,786 | **+54.1%** |
+| Filtered people search | 405,040 | 566,471 | **−39.9%** |
+| Chained multi-entity pipeline | 2,748,188 | 1,061,312 | **+61.4%** |
+
+Equal-weighted mean **+25.2%**; pooled **+48.8%**.
+
+The mean lands on "25% fewer tokens" almost exactly — but it is the average of +54%, −40%
+and +61%, so no real workload is described by "about a quarter". **The direction flips with
+workload shape.** Read the caveats in the results doc before quoting anything.
+
+## Run it
+
+```bash
+apollo auth whoami                      # both arms hit the live API — be logged in
+node benchmarks/token-efficiency/run.ts # Node >= 23.6
+# Node 22.x:
+node --experimental-strip-types benchmarks/token-efficiency/run.ts
+```
+
+The MCP arm needs `apollo-work` to be authorized once in an interactive session
+(`claude mcp add --transport http apollo-work https://mcp.apollo.io/mcp`, then `/mcp` → authenticate).
+An unauthenticated `tools/list` against `https://mcp.apollo.io/mcp` returns `401`, so the MCP arm
+fails closed rather than silently measuring an empty tool surface.
+
+| Flag | Default | |
+|---|---|---|
+| `--reps <n>` | `3` | repetitions per arm per case; the **median** is reported |
+| `--model <id>` | `claude-sonnet-5` | same model for both arms |
+| `--effort <level>` | `medium` | reasoning effort, pinned across both arms |
+| `--cases <ids>` | all | comma-separated ids from `cases.json` |
+| `--budget <usd>` | `1.00` | per-run spend cap (`--max-budget-usd`) |
+| `--timeout <sec>` | `300` | per-run wall-clock cap before the child is killed |
+| `--keep` | off | keep the scratch workspaces for transcript inspection |
+| `--out <path>` | `results.json` | raw per-run data |
+
+Output is a markdown table plus the average, printed to stdout and dumped with every raw run to
+`results.json` (gitignored — commit a copy under `results/` if you want to cite it).
+
+## The three cases
+
+Chosen to span the shapes that actually drive the cost difference, from the one where MCP is most
+competitive to the one Andy's "high-volume pipeline" claim is about.
+
+| Case | Shape | Why it's here |
+|---|---|---|
+| `single-enrich` | 1 call, small payload | Floor case. Almost all of the delta is fixed tool-definition overhead, so this is where MCP looks best. |
+| `filtered-search` | 1 call, compact payload | Intended as the CLI's projection win, but measurement showed Apollo's `people search` returns only 11 fields per person (465 B each, 16.6 KB for 25) — so there is little to project away. MCP does load the full payload (15.7 KB vs the CLI's `jq`-projected 2.4 KB), but ~11k tokens of savings is ~2% of a 400k-token run. |
+| `chained-pipeline` | ~11–21 calls, fan-out per company | Search → per-company job postings → per-company decision-maker. The CLI composes this in a shell pipeline; MCP pays a round trip, and a full result payload, per hop. |
+
+Edit `cases.json` to add cases; ids are the handles for `--cases`.
+
+## What is being measured
+
+`claude -p --output-format json` reports `usage` and `total_cost_usd` per run. The harness records
+all four token buckets and derives three metrics:
+
+- **`totalTokens`** — raw sum of `input + cache_creation + cache_read + output`. This is the
+  headline number, and the one to quote as "tokens consumed".
+- **`billableEquivalent`** — cost-weighted (`cache_read × 0.1`, `cache_creation × 1.25`), so an arm
+  that re-reads a large cached prefix on every turn isn't charged as if it paid full price for it.
+  Quote this when the argument is about spend rather than context pressure.
+- **`total_cost_usd`** — what Anthropic actually billed.
+
+Percent difference, in the direction the claim is stated:
+
+```
+CLI saving = (mcpTokens - cliTokens) / mcpTokens × 100
+```
+
+Reported per case, then as an **equal-weighted mean across cases** (the "average" asked for) and as
+a **pooled** figure over summed tokens. The two differ, and the difference is informative: the mean
+lets the cheap single-lookup case count as much as the pipeline case, while the pooled figure is
+closer to what a real high-volume workload would bill. Report both; don't pick the flattering one.
+
+## Why the comparison is fair
+
+Structural differences between the two arms are the thing being measured, so everything else is
+pinned:
+
+- **Same prompt, same model, same effort level, same per-run budget cap** for both arms.
+- **Empty scratch cwd per run** (`mkdtemp`) — no `CLAUDE.md`, no project settings, no repo context.
+- **`--setting-sources project`** — loads project settings only. This keeps the operator's
+  user-level settings and globally-enabled plugins (which contribute a dozen-plus extra skills)
+  out of *both* arms, while still discovering the scratch cwd's `.claude/skills`. Note that
+  `--setting-sources ""` is *not* usable here: it also stops the cwd skill from being discovered,
+  silently gutting the CLI arm.
+- **`--strict-mcp-config`** — the CLI arm is given `{"mcpServers":{}}`, so it cannot fall back to
+  an MCP server; the MCP arm is given only `apollo-work`.
+- **Bash is denied in the MCP arm**, so it cannot shell out to the CLI and win on its behalf.
+- **The CLI arm gets the shipped `apollo-cli` skill** copied into its scratch cwd, because that is
+  how a real CLI user has it — so its context cost is counted, not hidden.
+- **Median of N reps**, because agent trajectories vary run to run. Raise `--reps` before quoting a
+  number externally; 3 is enough to spot a wild run, not enough to be a confidence interval.
+
+## Known limitations — read before quoting a number
+
+1. **Live data moves.** Both arms hit the production Apollo API, so payload sizes shift as the
+   underlying data changes. Numbers are comparable within a single benchmark run, not across weeks.
+2. **Trajectory variance is the dominant noise source.** An agent that decides to make one extra
+   exploratory call can swing a case by tens of thousands of tokens. Check `numTurns` in the table:
+   if the two arms took very different turn counts, you are partly measuring planning luck.
+3. **Credit consumption.** These are real Apollo API calls. `filtered-search` and
+   `chained-pipeline` consume credits on every rep — `--reps 3` across 3 cases is 18 runs.
+4. **The MCP arm's cost depends on how many tools its server exposes _and on whether the client
+   loads them eagerly_.** See the finding below — this is the single biggest thing to understand
+   before quoting any number.
+5. The result is a statement about *this* MCP server and *this* client version, not about MCP as a
+   protocol. In Claude Code 2.1.238 the ~90 `apollo-work` tools are **deferred behind `ToolSearch`**
+   rather than loaded into context up front, so the MCP arm does not pay a per-schema tax every
+   session — which removes much of the usual justification for expecting a large CLI win.
+6. **A run that doesn't finish must never be scored.** The MCP server's credit-cost annotations make
+   the agent stop and ask for confirmation; that spends few tokens and looks like efficiency. Both
+   arms get an identical pre-approval suffix, and every run must pass `gradeComplete()`.

benchmarks/token-efficiency/cases.json

@@ -0,0 +1,23 @@
+[
+  {
+    "id": "single-enrich",
+    "name": "Single company enrichment",
+    "shape": "1 API call, small payload",
+    "prompt": "Enrich the company stripe.com in Apollo. Report exactly three facts and nothing else: employee count, industry, and estimated annual revenue.",
+    "grade": { "minRows": 0, "mustContain": ["revenue"] }
+  },
+  {
+    "id": "filtered-search",
+    "name": "Filtered people search with field projection",
+    "shape": "1 call, compact 11-field payload (16.6 KB / 25 people)",
+    "prompt": "Find 25 VP-level-or-above engineering leaders at SaaS companies in the United States with 51-200 employees. Output a markdown table with exactly four columns: name, title, company, LinkedIn URL. No commentary.",
+    "grade": { "minRows": 20, "mustContain": [] }
+  },
+  {
+    "id": "chained-pipeline",
+    "name": "Chained multi-entity pipeline",
+    "shape": "~11-21 API calls, fan-out per company",
+    "prompt": "Find 10 companies in the United States with 201-500 employees that are currently hiring software engineers. For each one, report the company name, how many software-engineering job postings it has open, and the name and title of its most senior engineering decision-maker. Output a markdown table with those four columns. No commentary.",
+    "grade": { "minRows": 8, "mustContain": [] }
+  }
+]

benchmarks/token-efficiency/results/2026-08-21-raw-chained-pipeline-mcp-fill.json

@@ -0,0 +1,73 @@
+{
+  "opts": {
+    "reps": 2,
+    "model": "claude-sonnet-5",
+    "effort": "medium",
+    "cases": [
+      "chained-pipeline"
+    ],
+    "arms": [
+      "mcp"
+    ],
+    "budget": 6,
+    "timeout": 900,
+    "keep": false,
+    "out": "/private/tmp/claude-501/-Users-alexander-apollo-draft-pilot/077debf2-3f33-4956-9f4a-3639978d9e86/scratchpad/results-fill.json"
+  },
+  "perCase": [
+    {
+      "id": "chained-pipeline",
+      "name": "Chained multi-entity pipeline",
+      "shape": "~11-21 API calls, fan-out per company",
+      "cli": {
+        "totalTokens": 0,
+        "billable": 0,
+        "cost": 0,
+        "turns": 0
+      },
+      "mcp": {
+        "totalTokens": 4517548.5,
+        "billable": 894216.425,
+        "cost": 3.4035816500000005,
+        "turns": 32.5
+      }
+    }
+  ],
+  "runs": [
+    {
+      "arm": "mcp",
+      "caseId": "chained-pipeline",
+      "rep": 1,
+      "ok": true,
+      "complete": true,
+      "resultText": "| Company Name | # Software Engineering Job Postings | Decision-Maker Name | Decision-Maker Title |\n|---|---|---|---|\n| Apptad | 3 | Arbind Singh | Co-Founder and CEO |\n| Seclore | 7 | Nilesh Bhojani | Chief Product & Technology Officer |\n| nference | 3 | Ganesh Ramamoorthy | VP, Engineering |\n| Rocketlane | 2 | Deepak Bala | Co-Founder & CTO |\n| SambaNova | ~20 | Dawei Huang | Vice President of Engineering |\n| PathAI | 5 | Aditya Dhoot | Director of Engineering |\n| Ursa Major | 3 | William Somers | Director of Engineering |\n| Tyfone, Inc. | 2 | Lakshman Prabu | Chief Technology Officer |\n| Aerospike | 2 | Yossi Levanoni | VP of Engineering |\n| Inspectorio | 4 | Yagor Maliutsin | VP of Engineering |",
+      "inputTokens": 1547,
+      "cacheCreationTokens": 328417,
+      "cacheReadTokens": 2392969,
+      "outputTokens": 25255,
+      "totalTokens": 2748188,
+      "billableEquivalent": 676620.15,
+      "costUsd": 2.5192392,
+      "numTurns": 1,
+      "durationMs": 6188
+    },
+    {
+      "arm": "mcp",
+      "caseId": "chained-pipeline",
+      "rep": 2,
+      "ok": true,
+      "complete": true,
+      "resultText": "Now I have all data needed. Let me compile the final table.\n\n## Companies (201–500 employees, US, actively hiring software engineers)\n\n| Company | Software-Engineering Job Postings | Decision-Maker Name | Title |\n|---|---|---|---|\n| Rocketlane | 2 | Deepak Bala | Co-Founder & CTO |\n| PathAI | 3 | Aditya Dhoot | Director of Engineering |\n| nference | 3 | Ganesh Ramamoorthy | VP, Engineering |\n| Inspectorio | 2 | Yagor Maliutsin | VP of Engineering |\n| DaCodes | 2 | Eric Segovia | VP of Engineering |\n| Tyfone, Inc. | 2 | Nizar Jamal | CTO |\n| Seclore | 1 | Nilesh Bhojani | Chief Product & Technology Officer |\n| Aerospike | 1 | Yossi Levanoni | VP of Engineering |\n| Bidgely | 1 | Samarjit Ghosh | SVP Head of Engineering & India Head |\n| Abacus.AI | 1 | Ajit Deshpande | VP Engineering & CSO |",
+      "inputTokens": 1659,
+      "cacheCreationTokens": 382230,
+      "cacheReadTokens": 5856282,
+      "outputTokens": 46738,
+      "totalTokens": 6286909,
+      "billableEquivalent": 1111812.7000000002,
+      "costUsd": 4.287924100000001,
+      "numTurns": 64,
+      "durationMs": 748094
+    }
+  ],
+  "report": "# Apollo CLI vs MCP — token efficiency\n\nModel `claude-sonnet-5` · 2 reps per arm per case (median reported) · effort medium · $6.00 budget cap per run\n\n> ⚠️ `chained-pipeline` / cli: only 0 of 2 reps completed — median is not meaningful.\n\n| Case | Shape | MCP tokens | CLI tokens | CLI saves | MCP turns | CLI turns |\n|---|---|---:|---:|---:|---:|---:|\n| Chained multi-entity pipeline | ~11-21 API calls, fan-out per company | 4,517,549 | 0 | 100.0% | 33 | 0 |\n\n**Average saving (equal-weighted across cases): 100.0%**\n\nPooled across all cases: 4,517,549 → 0 tokens (100.0%).\nCost-weighted (cache reads at 0.1x): 100.0% · billed USD: $3.4036 → $0.0000 (100.0%)."
+}