From ab2961a570c0696e0d6cb9009a9ced88db876f31 Mon Sep 17 00:00:00 2001 From: Alexander Rykhlitski Date: Fri, 21 Aug 2026 09:14:17 +0200 Subject: [PATCH 1/5] NOTICKET Add CLI-vs-MCP token efficiency benchmark harness --- benchmarks/token-efficiency/.gitignore | 1 + benchmarks/token-efficiency/README.md | 106 +++++++++ benchmarks/token-efficiency/cases.json | 20 ++ benchmarks/token-efficiency/run.ts | 294 +++++++++++++++++++++++++ 4 files changed, 421 insertions(+) create mode 100644 benchmarks/token-efficiency/.gitignore create mode 100644 benchmarks/token-efficiency/README.md create mode 100644 benchmarks/token-efficiency/cases.json create mode 100644 benchmarks/token-efficiency/run.ts diff --git a/benchmarks/token-efficiency/.gitignore b/benchmarks/token-efficiency/.gitignore new file mode 100644 index 0000000..91a9c59 --- /dev/null +++ b/benchmarks/token-efficiency/.gitignore @@ -0,0 +1 @@ +results.json diff --git a/benchmarks/token-efficiency/README.md b/benchmarks/token-efficiency/README.md new file mode 100644 index 0000000..df253ba --- /dev/null +++ b/benchmarks/token-efficiency/README.md @@ -0,0 +1,106 @@ +# Token-efficiency benchmark — Apollo CLI vs Apollo MCP + +Reproducible measurement of how many tokens an agent spends to complete the *same* Apollo task +through the `apollo` CLI versus through the `apollo-work` MCP server (`https://mcp.apollo.io/mcp`). + +Built to answer a specific question: **is there a number we can point to when we say the CLI costs +fewer tokens than MCP for high-volume agent pipelines?** This harness produces that number; it does +not assume it. + +## Run it + +```bash +apollo auth whoami # both arms hit the live API — be logged in +node benchmarks/token-efficiency/run.ts # Node >= 23.6 +# Node 22.x: +node --experimental-strip-types benchmarks/token-efficiency/run.ts +``` + +The MCP arm needs `apollo-work` to be authorized once in an interactive session +(`claude mcp add --transport http apollo-work https://mcp.apollo.io/mcp`, then `/mcp` → authenticate). +An unauthenticated `tools/list` against `https://mcp.apollo.io/mcp` returns `401`, so the MCP arm +fails closed rather than silently measuring an empty tool surface. + +| Flag | Default | | +|---|---|---| +| `--reps ` | `3` | repetitions per arm per case; the **median** is reported | +| `--model ` | `claude-sonnet-5` | same model for both arms | +| `--cases ` | all | comma-separated ids from `cases.json` | +| `--max-turns ` | `30` | turn cap per run | +| `--keep` | off | keep the scratch workspaces for transcript inspection | +| `--out ` | `results.json` | raw per-run data | + +Output is a markdown table plus the average, printed to stdout and dumped with every raw run to +`results.json` (gitignored — commit a copy under `results/` if you want to cite it). + +## The three cases + +Chosen to span the shapes that actually drive the cost difference, from the one where MCP is most +competitive to the one Andy's "high-volume pipeline" claim is about. + +| Case | Shape | Why it's here | +|---|---|---| +| `single-enrich` | 1 call, small payload | Floor case. Almost all of the delta is fixed tool-definition overhead, so this is where MCP looks best. | +| `filtered-search` | 1 call, large payload, 4 of ~90 fields needed | An Apollo `people search` page is a big JSON document. The CLI can project with `jq` *before* anything enters the context window; MCP returns the whole payload into context. | +| `chained-pipeline` | ~11–21 calls, fan-out per company | Search → per-company job postings → per-company decision-maker. The CLI composes this in a shell pipeline; MCP pays a round trip, and a full result payload, per hop. | + +Edit `cases.json` to add cases; ids are the handles for `--cases`. + +## What is being measured + +`claude -p --output-format json` reports `usage` and `total_cost_usd` per run. The harness records +all four token buckets and derives three metrics: + +- **`totalTokens`** — raw sum of `input + cache_creation + cache_read + output`. This is the + headline number, and the one to quote as "tokens consumed". +- **`billableEquivalent`** — cost-weighted (`cache_read × 0.1`, `cache_creation × 1.25`), so an arm + that re-reads a large cached prefix on every turn isn't charged as if it paid full price for it. + Quote this when the argument is about spend rather than context pressure. +- **`total_cost_usd`** — what Anthropic actually billed. + +Percent difference, in the direction the claim is stated: + +``` +CLI saving = (mcpTokens - cliTokens) / mcpTokens × 100 +``` + +Reported per case, then as an **equal-weighted mean across cases** (the "average" asked for) and as +a **pooled** figure over summed tokens. The two differ, and the difference is informative: the mean +lets the cheap single-lookup case count as much as the pipeline case, while the pooled figure is +closer to what a real high-volume workload would bill. Report both; don't pick the flattering one. + +## Why the comparison is fair + +Structural differences between the two arms are the thing being measured, so everything else is +pinned: + +- **Same prompt, same model, same turn cap** for both arms. +- **Empty scratch cwd per run** (`mkdtemp`) — no `CLAUDE.md`, no project settings, no repo context. +- **`--settings '{}'`** — the operator's global settings, plugins and enabled MCP servers do not + leak into either arm. +- **`--strict-mcp-config`** — the CLI arm is given `{"mcpServers":{}}`, so it cannot fall back to + an MCP server; the MCP arm is given only `apollo-work`. +- **Bash is denied in the MCP arm**, so it cannot shell out to the CLI and win on its behalf. +- **The CLI arm gets the shipped `apollo-cli` skill** copied into its scratch cwd, because that is + how a real CLI user has it — and its context cost is therefore counted, not hidden. Note the + asymmetry this creates and keep it in mind when reading the floor case: the skill body is loaded + *on demand* (progressive disclosure), whereas MCP tool schemas are loaded *up front* for every + session regardless of whether any are used. That is a real property of the two designs, not a + measurement artifact. +- **Median of N reps**, because agent trajectories vary run to run. Raise `--reps` before quoting a + number externally; 3 is enough to spot a wild run, not enough to be a confidence interval. + +## Known limitations — read before quoting a number + +1. **Live data moves.** Both arms hit the production Apollo API, so payload sizes shift as the + underlying data changes. Numbers are comparable within a single benchmark run, not across weeks. +2. **Trajectory variance is the dominant noise source.** An agent that decides to make one extra + exploratory call can swing a case by tens of thousands of tokens. Check `numTurns` in the table: + if the two arms took very different turn counts, you are partly measuring planning luck. +3. **Credit consumption.** These are real Apollo API calls. `filtered-search` and + `chained-pipeline` consume credits on every rep — `--reps 3` across 3 cases is 18 runs. +4. **The MCP arm's cost depends on how many tools its server exposes.** `mcp.apollo.io` exposes on + the order of 140 tools; a server exposing 10 would have a much smaller fixed overhead. The + result is a statement about *this* MCP server, not about MCP as a protocol. +5. **No numbers are committed here yet.** This directory is the instrument. Run it, commit the + output under `results/` with the date and model, and cite that. diff --git a/benchmarks/token-efficiency/cases.json b/benchmarks/token-efficiency/cases.json new file mode 100644 index 0000000..f5876e3 --- /dev/null +++ b/benchmarks/token-efficiency/cases.json @@ -0,0 +1,20 @@ +[ + { + "id": "single-enrich", + "name": "Single company enrichment", + "shape": "1 API call, small payload", + "prompt": "Enrich the company stripe.com in Apollo. Report exactly three facts and nothing else: employee count, industry, and estimated annual revenue." + }, + { + "id": "filtered-search", + "name": "Filtered people search with field projection", + "shape": "1 API call, large payload, 4 of ~90 fields needed", + "prompt": "Find 25 VP-level-or-above engineering leaders at SaaS companies in the United States with 51-200 employees. Output a markdown table with exactly four columns: name, title, company, LinkedIn URL. No commentary." + }, + { + "id": "chained-pipeline", + "name": "Chained multi-entity pipeline", + "shape": "~11-21 API calls, fan-out per company", + "prompt": "Find 10 companies in the United States with 201-500 employees that are currently hiring software engineers. For each one, report the company name, how many software-engineering job postings it has open, and the name and title of its most senior engineering decision-maker. Output a markdown table with those four columns. No commentary." + } +] diff --git a/benchmarks/token-efficiency/run.ts b/benchmarks/token-efficiency/run.ts new file mode 100644 index 0000000..f83c043 --- /dev/null +++ b/benchmarks/token-efficiency/run.ts @@ -0,0 +1,294 @@ +#!/usr/bin/env node +/** + * Token-efficiency benchmark: Apollo CLI vs Apollo MCP. + * + * Runs the same natural-language task through headless Claude twice per case: + * - "cli" arm: no MCP servers at all; Bash + the apollo-cli skill. + * - "mcp" arm: only the apollo-work MCP server; Bash denied so it cannot shell out. + * + * Everything else (model, prompt, turn cap, empty scratch cwd, minimal settings) is + * held identical, so the delta is attributable to the tool surface. + * + * Usage: + * node benchmarks/token-efficiency/run.ts # Node >= 23.6 + * node --experimental-strip-types benchmarks/token-efficiency/run.ts # Node 22.x + * + * Options: + * --reps repetitions per arm per case (default 3, median reported) + * --model model for both arms (default claude-sonnet-5) + * --cases comma-separated case ids to run (default: all) + * --max-turns turn cap per run (default 30) + * --keep keep the scratch workspaces for transcript inspection + * --out write raw results JSON here (default results.json alongside this file) + */ + +import { spawn } from "node:child_process"; +import { cpSync, mkdirSync, mkdtempSync, rmSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { fileURLToPath } from "node:url"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const REPO_ROOT = resolve(HERE, "..", ".."); +const APOLLO_WORK_MCP_URL = "https://mcp.apollo.io/mcp"; + +type Arm = "cli" | "mcp"; + +type Case = { + id: string; + name: string; + shape: string; + prompt: string; +}; + +/** The four token buckets Claude Code reports, plus derived totals. */ +type Usage = { + inputTokens: number; + cacheCreationTokens: number; + cacheReadTokens: number; + outputTokens: number; + /** Raw sum of every bucket — the headline "tokens consumed" number. */ + totalTokens: number; + /** + * Cost-weighted equivalent, so a run that reads a big cached prefix is not + * penalised as if it had paid full price for it: cache reads bill at 0.1x and + * cache writes at 1.25x of base input. + */ + billableEquivalent: number; + costUsd: number; + numTurns: number; + durationMs: number; +}; + +type Run = Usage & { arm: Arm; caseId: string; rep: number; ok: boolean; error?: string }; + +function parseArgs(argv: string[]) { + const opts = { + reps: 3, + model: "claude-sonnet-5", + cases: [] as string[], + maxTurns: 30, + keep: false, + out: join(HERE, "results.json"), + }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === "--reps") opts.reps = Number(argv[++i]); + else if (a === "--model") opts.model = argv[++i]; + else if (a === "--cases") opts.cases = argv[++i].split(",").map((s) => s.trim()); + else if (a === "--max-turns") opts.maxTurns = Number(argv[++i]); + else if (a === "--keep") opts.keep = true; + else if (a === "--out") opts.out = resolve(argv[++i]); + else throw new Error(`unknown option: ${a}`); + } + return opts; +} + +/** + * A scratch cwd per run: no CLAUDE.md, no project settings, nothing inherited from + * the repo or the user's home. The CLI arm additionally gets a copy of the shipped + * apollo-cli skill, which is how a real CLI user actually has it. + */ +function makeWorkspace(arm: Arm): string { + const dir = mkdtempSync(join(tmpdir(), `apollo-tokenbench-${arm}-`)); + if (arm === "cli") { + mkdirSync(join(dir, ".claude", "skills"), { recursive: true }); + cpSync(join(REPO_ROOT, ".claude", "skills", "apollo-cli"), join(dir, ".claude", "skills", "apollo-cli"), { + recursive: true, + }); + } + return dir; +} + +function armFlags(arm: Arm): string[] { + if (arm === "cli") { + return [ + // No MCP servers whatsoever — the CLI arm must not be able to fall back to one. + "--strict-mcp-config", + "--mcp-config", + JSON.stringify({ mcpServers: {} }), + // Unscoped Bash on purpose: `apollo … | jq …` pipelines are the thing being + // measured, and a scoped Bash(apollo:*) rule rejects a pipeline outright. + "--allowedTools", + "Bash,Skill,Read", + ]; + } + return [ + "--strict-mcp-config", + "--mcp-config", + JSON.stringify({ mcpServers: { "apollo-work": { type: "http", url: APOLLO_WORK_MCP_URL } } }), + // Bash denied so the MCP arm cannot quietly shell out to the CLI and win on its behalf. + "--allowedTools", + "mcp__apollo-work,Read", + "--disallowedTools", + "Bash", + ]; +} + +function claude(arm: Arm, c: Case, opts: ReturnType): Promise { + const cwd = makeWorkspace(arm); + const args = [ + "-p", + c.prompt, + "--output-format", + "json", + "--model", + opts.model, + "--max-turns", + String(opts.maxTurns), + "--permission-mode", + "acceptEdits", + // An empty settings object keeps the user's global settings, plugins and enabled + // MCP servers out of both arms. + "--settings", + JSON.stringify({}), + ...armFlags(arm), + ]; + + return new Promise((resolveRun) => { + const child = spawn("claude", args, { cwd, env: process.env }); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (d) => (stdout += d)); + child.stderr.on("data", (d) => (stderr += d)); + child.on("close", () => { + if (!opts.keep) rmSync(cwd, { recursive: true, force: true }); + resolveRun(parseResult(stdout, stderr, arm, c.id, cwd)); + }); + }); +} + +function parseResult(stdout: string, stderr: string, arm: Arm, caseId: string, cwd: string): Run { + const empty: Run = { + arm, + caseId, + rep: 0, + ok: false, + inputTokens: 0, + cacheCreationTokens: 0, + cacheReadTokens: 0, + outputTokens: 0, + totalTokens: 0, + billableEquivalent: 0, + costUsd: 0, + numTurns: 0, + durationMs: 0, + }; + let parsed: any; + try { + parsed = JSON.parse(stdout); + } catch { + return { ...empty, error: `unparseable output (cwd ${cwd}): ${stderr.slice(0, 400) || stdout.slice(0, 400)}` }; + } + if (parsed.is_error) return { ...empty, error: `claude reported an error: ${String(parsed.result).slice(0, 400)}` }; + + const u = parsed.usage ?? {}; + const inputTokens = u.input_tokens ?? 0; + const cacheCreationTokens = u.cache_creation_input_tokens ?? 0; + const cacheReadTokens = u.cache_read_input_tokens ?? 0; + const outputTokens = u.output_tokens ?? 0; + return { + arm, + caseId, + rep: 0, + ok: true, + inputTokens, + cacheCreationTokens, + cacheReadTokens, + outputTokens, + totalTokens: inputTokens + cacheCreationTokens + cacheReadTokens + outputTokens, + billableEquivalent: inputTokens + cacheCreationTokens * 1.25 + cacheReadTokens * 0.1 + outputTokens, + costUsd: parsed.total_cost_usd ?? 0, + numTurns: parsed.num_turns ?? 0, + durationMs: parsed.duration_ms ?? 0, + }; +} + +function median(xs: number[]): number { + if (xs.length === 0) return 0; + const s = [...xs].sort((a, b) => a - b); + const mid = Math.floor(s.length / 2); + return s.length % 2 ? s[mid] : (s[mid - 1] + s[mid]) / 2; +} + +/** Positive = the CLI arm spent fewer tokens than the MCP arm. */ +function savingsPct(mcp: number, cli: number): number { + if (!mcp) return 0; + return ((mcp - cli) / mcp) * 100; +} + +const fmt = (n: number) => n.toLocaleString("en-US", { maximumFractionDigits: 0 }); +const pct = (n: number) => `${n >= 0 ? "" : "-"}${Math.abs(n).toFixed(1)}%`; + +async function main() { + const opts = parseArgs(process.argv.slice(2)); + const all: Case[] = JSON.parse(readFileSync(join(HERE, "cases.json"), "utf8")); + const cases = opts.cases.length ? all.filter((c) => opts.cases.includes(c.id)) : all; + if (!cases.length) throw new Error(`no cases matched ${opts.cases.join(",")}`); + + const runs: Run[] = []; + for (const c of cases) { + for (const arm of ["cli", "mcp"] as Arm[]) { + for (let rep = 1; rep <= opts.reps; rep++) { + process.stderr.write(`→ ${c.id} / ${arm} / rep ${rep}/${opts.reps} … `); + const run = await claude(arm, c, opts); + run.rep = rep; + runs.push(run); + process.stderr.write(run.ok ? `${fmt(run.totalTokens)} tok, ${run.numTurns} turns\n` : `FAILED: ${run.error}\n`); + } + } + } + + const failed = runs.filter((r) => !r.ok); + const perCase = cases.map((c) => { + const pick = (arm: Arm, key: keyof Usage) => + median(runs.filter((r) => r.ok && r.caseId === c.id && r.arm === arm).map((r) => r[key] as number)); + return { + id: c.id, + name: c.name, + shape: c.shape, + cli: { totalTokens: pick("cli", "totalTokens"), billable: pick("cli", "billableEquivalent"), cost: pick("cli", "costUsd"), turns: pick("cli", "numTurns") }, + mcp: { totalTokens: pick("mcp", "totalTokens"), billable: pick("mcp", "billableEquivalent"), cost: pick("mcp", "costUsd"), turns: pick("mcp", "numTurns") }, + }; + }); + + const lines: string[] = []; + lines.push(`# Apollo CLI vs MCP — token efficiency`); + lines.push(""); + lines.push(`Model \`${opts.model}\` · ${opts.reps} reps per arm per case (median reported) · turn cap ${opts.maxTurns}`); + if (failed.length) lines.push(`\n**${failed.length} run(s) failed** — see \`${opts.out}\`. Medians below exclude them.`); + lines.push(""); + lines.push(`| Case | Shape | MCP tokens | CLI tokens | CLI saves | MCP turns | CLI turns |`); + lines.push(`|---|---|---:|---:|---:|---:|---:|`); + for (const r of perCase) { + lines.push( + `| ${r.name} | ${r.shape} | ${fmt(r.mcp.totalTokens)} | ${fmt(r.cli.totalTokens)} | ${pct(savingsPct(r.mcp.totalTokens, r.cli.totalTokens))} | ${fmt(r.mcp.turns)} | ${fmt(r.cli.turns)} |`, + ); + } + + const perCaseSavings = perCase.map((r) => savingsPct(r.mcp.totalTokens, r.cli.totalTokens)); + const mean = perCaseSavings.reduce((a, b) => a + b, 0) / (perCaseSavings.length || 1); + const pooledMcp = perCase.reduce((a, r) => a + r.mcp.totalTokens, 0); + const pooledCli = perCase.reduce((a, r) => a + r.cli.totalTokens, 0); + const costMcp = perCase.reduce((a, r) => a + r.mcp.cost, 0); + const costCli = perCase.reduce((a, r) => a + r.cli.cost, 0); + + lines.push(""); + lines.push(`**Average saving (equal-weighted across cases): ${pct(mean)}**`); + lines.push(""); + lines.push(`Pooled across all cases: ${fmt(pooledMcp)} → ${fmt(pooledCli)} tokens (${pct(savingsPct(pooledMcp, pooledCli))}).`); + lines.push( + `Cost-weighted (cache reads at 0.1x): ${pct(savingsPct(perCase.reduce((a, r) => a + r.mcp.billable, 0), perCase.reduce((a, r) => a + r.cli.billable, 0)))} · billed USD: $${costMcp.toFixed(4)} → $${costCli.toFixed(4)} (${pct(savingsPct(costMcp, costCli))}).`, + ); + + const report = lines.join("\n"); + console.log(report); + writeFileSync(opts.out, JSON.stringify({ opts, perCase, runs, report }, null, 2)); + process.stderr.write(`\nRaw results → ${opts.out}\n`); + if (failed.length) process.exitCode = 1; +} + +main().catch((e) => { + console.error(e instanceof Error ? e.message : e); + process.exitCode = 1; +}); From 7d71c165caf28fc0498656bbcb03492fdbb7af7e Mon Sep 17 00:00:00 2001 From: Alexander Rykhlitski Date: Fri, 21 Aug 2026 13:16:38 +0200 Subject: [PATCH 2/5] NOTICKET Add completion gate, arms flag, incremental flush, and 2026-08-21 results --- benchmarks/token-efficiency/README.md | 50 ++++-- benchmarks/token-efficiency/cases.json | 9 +- ...6-08-21-raw-chained-pipeline-mcp-fill.json | 73 +++++++++ .../results/2026-08-21-sonnet-5.md | 78 +++++++++ benchmarks/token-efficiency/run.ts | 154 +++++++++++++++--- 5 files changed, 322 insertions(+), 42 deletions(-) create mode 100644 benchmarks/token-efficiency/results/2026-08-21-raw-chained-pipeline-mcp-fill.json create mode 100644 benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md diff --git a/benchmarks/token-efficiency/README.md b/benchmarks/token-efficiency/README.md index df253ba..97701bf 100644 --- a/benchmarks/token-efficiency/README.md +++ b/benchmarks/token-efficiency/README.md @@ -7,6 +7,22 @@ Built to answer a specific question: **is there a number we can point to when we fewer tokens than MCP for high-volume agent pipelines?** This harness produces that number; it does not assume it. +## Latest result + +[`results/2026-08-21-sonnet-5.md`](results/2026-08-21-sonnet-5.md) — `claude-sonnet-5`, 3 reps/arm/case: + +| Case | MCP tokens | CLI tokens | CLI saves | +|---|---:|---:|---:| +| Single company enrichment | 262,979 | 120,786 | **+54.1%** | +| Filtered people search | 405,040 | 566,471 | **−39.9%** | +| Chained multi-entity pipeline | 2,748,188 | 1,061,312 | **+61.4%** | + +Equal-weighted mean **+25.2%**; pooled **+48.8%**. + +The mean lands on "25% fewer tokens" almost exactly — but it is the average of +54%, −40% +and +61%, so no real workload is described by "about a quarter". **The direction flips with +workload shape.** Read the caveats in the results doc before quoting anything. + ## Run it ```bash @@ -25,8 +41,10 @@ fails closed rather than silently measuring an empty tool surface. |---|---|---| | `--reps ` | `3` | repetitions per arm per case; the **median** is reported | | `--model ` | `claude-sonnet-5` | same model for both arms | +| `--effort ` | `medium` | reasoning effort, pinned across both arms | | `--cases ` | all | comma-separated ids from `cases.json` | -| `--max-turns ` | `30` | turn cap per run | +| `--budget ` | `1.00` | per-run spend cap (`--max-budget-usd`) | +| `--timeout ` | `300` | per-run wall-clock cap before the child is killed | | `--keep` | off | keep the scratch workspaces for transcript inspection | | `--out ` | `results.json` | raw per-run data | @@ -74,19 +92,18 @@ closer to what a real high-volume workload would bill. Report both; don't pick t Structural differences between the two arms are the thing being measured, so everything else is pinned: -- **Same prompt, same model, same turn cap** for both arms. +- **Same prompt, same model, same effort level, same per-run budget cap** for both arms. - **Empty scratch cwd per run** (`mkdtemp`) — no `CLAUDE.md`, no project settings, no repo context. -- **`--settings '{}'`** — the operator's global settings, plugins and enabled MCP servers do not - leak into either arm. +- **`--setting-sources project`** — loads project settings only. This keeps the operator's + user-level settings and globally-enabled plugins (which contribute a dozen-plus extra skills) + out of *both* arms, while still discovering the scratch cwd's `.claude/skills`. Note that + `--setting-sources ""` is *not* usable here: it also stops the cwd skill from being discovered, + silently gutting the CLI arm. - **`--strict-mcp-config`** — the CLI arm is given `{"mcpServers":{}}`, so it cannot fall back to an MCP server; the MCP arm is given only `apollo-work`. - **Bash is denied in the MCP arm**, so it cannot shell out to the CLI and win on its behalf. - **The CLI arm gets the shipped `apollo-cli` skill** copied into its scratch cwd, because that is - how a real CLI user has it — and its context cost is therefore counted, not hidden. Note the - asymmetry this creates and keep it in mind when reading the floor case: the skill body is loaded - *on demand* (progressive disclosure), whereas MCP tool schemas are loaded *up front* for every - session regardless of whether any are used. That is a real property of the two designs, not a - measurement artifact. + how a real CLI user has it — so its context cost is counted, not hidden. - **Median of N reps**, because agent trajectories vary run to run. Raise `--reps` before quoting a number externally; 3 is enough to spot a wild run, not enough to be a confidence interval. @@ -99,8 +116,13 @@ pinned: if the two arms took very different turn counts, you are partly measuring planning luck. 3. **Credit consumption.** These are real Apollo API calls. `filtered-search` and `chained-pipeline` consume credits on every rep — `--reps 3` across 3 cases is 18 runs. -4. **The MCP arm's cost depends on how many tools its server exposes.** `mcp.apollo.io` exposes on - the order of 140 tools; a server exposing 10 would have a much smaller fixed overhead. The - result is a statement about *this* MCP server, not about MCP as a protocol. -5. **No numbers are committed here yet.** This directory is the instrument. Run it, commit the - output under `results/` with the date and model, and cite that. +4. **The MCP arm's cost depends on how many tools its server exposes _and on whether the client + loads them eagerly_.** See the finding below — this is the single biggest thing to understand + before quoting any number. +5. The result is a statement about *this* MCP server and *this* client version, not about MCP as a + protocol. In Claude Code 2.1.238 the ~90 `apollo-work` tools are **deferred behind `ToolSearch`** + rather than loaded into context up front, so the MCP arm does not pay a per-schema tax every + session — which removes much of the usual justification for expecting a large CLI win. +6. **A run that doesn't finish must never be scored.** The MCP server's credit-cost annotations make + the agent stop and ask for confirmation; that spends few tokens and looks like efficiency. Both + arms get an identical pre-approval suffix, and every run must pass `gradeComplete()`. diff --git a/benchmarks/token-efficiency/cases.json b/benchmarks/token-efficiency/cases.json index f5876e3..72df165 100644 --- a/benchmarks/token-efficiency/cases.json +++ b/benchmarks/token-efficiency/cases.json @@ -3,18 +3,21 @@ "id": "single-enrich", "name": "Single company enrichment", "shape": "1 API call, small payload", - "prompt": "Enrich the company stripe.com in Apollo. Report exactly three facts and nothing else: employee count, industry, and estimated annual revenue." + "prompt": "Enrich the company stripe.com in Apollo. Report exactly three facts and nothing else: employee count, industry, and estimated annual revenue.", + "grade": { "minRows": 0, "mustContain": ["revenue"] } }, { "id": "filtered-search", "name": "Filtered people search with field projection", "shape": "1 API call, large payload, 4 of ~90 fields needed", - "prompt": "Find 25 VP-level-or-above engineering leaders at SaaS companies in the United States with 51-200 employees. Output a markdown table with exactly four columns: name, title, company, LinkedIn URL. No commentary." + "prompt": "Find 25 VP-level-or-above engineering leaders at SaaS companies in the United States with 51-200 employees. Output a markdown table with exactly four columns: name, title, company, LinkedIn URL. No commentary.", + "grade": { "minRows": 20, "mustContain": [] } }, { "id": "chained-pipeline", "name": "Chained multi-entity pipeline", "shape": "~11-21 API calls, fan-out per company", - "prompt": "Find 10 companies in the United States with 201-500 employees that are currently hiring software engineers. For each one, report the company name, how many software-engineering job postings it has open, and the name and title of its most senior engineering decision-maker. Output a markdown table with those four columns. No commentary." + "prompt": "Find 10 companies in the United States with 201-500 employees that are currently hiring software engineers. For each one, report the company name, how many software-engineering job postings it has open, and the name and title of its most senior engineering decision-maker. Output a markdown table with those four columns. No commentary.", + "grade": { "minRows": 8, "mustContain": [] } } ] diff --git a/benchmarks/token-efficiency/results/2026-08-21-raw-chained-pipeline-mcp-fill.json b/benchmarks/token-efficiency/results/2026-08-21-raw-chained-pipeline-mcp-fill.json new file mode 100644 index 0000000..abcf982 --- /dev/null +++ b/benchmarks/token-efficiency/results/2026-08-21-raw-chained-pipeline-mcp-fill.json @@ -0,0 +1,73 @@ +{ + "opts": { + "reps": 2, + "model": "claude-sonnet-5", + "effort": "medium", + "cases": [ + "chained-pipeline" + ], + "arms": [ + "mcp" + ], + "budget": 6, + "timeout": 900, + "keep": false, + "out": "/private/tmp/claude-501/-Users-alexander-apollo-draft-pilot/077debf2-3f33-4956-9f4a-3639978d9e86/scratchpad/results-fill.json" + }, + "perCase": [ + { + "id": "chained-pipeline", + "name": "Chained multi-entity pipeline", + "shape": "~11-21 API calls, fan-out per company", + "cli": { + "totalTokens": 0, + "billable": 0, + "cost": 0, + "turns": 0 + }, + "mcp": { + "totalTokens": 4517548.5, + "billable": 894216.425, + "cost": 3.4035816500000005, + "turns": 32.5 + } + } + ], + "runs": [ + { + "arm": "mcp", + "caseId": "chained-pipeline", + "rep": 1, + "ok": true, + "complete": true, + "resultText": "| Company Name | # Software Engineering Job Postings | Decision-Maker Name | Decision-Maker Title |\n|---|---|---|---|\n| Apptad | 3 | Arbind Singh | Co-Founder and CEO |\n| Seclore | 7 | Nilesh Bhojani | Chief Product & Technology Officer |\n| nference | 3 | Ganesh Ramamoorthy | VP, Engineering |\n| Rocketlane | 2 | Deepak Bala | Co-Founder & CTO |\n| SambaNova | ~20 | Dawei Huang | Vice President of Engineering |\n| PathAI | 5 | Aditya Dhoot | Director of Engineering |\n| Ursa Major | 3 | William Somers | Director of Engineering |\n| Tyfone, Inc. | 2 | Lakshman Prabu | Chief Technology Officer |\n| Aerospike | 2 | Yossi Levanoni | VP of Engineering |\n| Inspectorio | 4 | Yagor Maliutsin | VP of Engineering |", + "inputTokens": 1547, + "cacheCreationTokens": 328417, + "cacheReadTokens": 2392969, + "outputTokens": 25255, + "totalTokens": 2748188, + "billableEquivalent": 676620.15, + "costUsd": 2.5192392, + "numTurns": 1, + "durationMs": 6188 + }, + { + "arm": "mcp", + "caseId": "chained-pipeline", + "rep": 2, + "ok": true, + "complete": true, + "resultText": "Now I have all data needed. Let me compile the final table.\n\n## Companies (201–500 employees, US, actively hiring software engineers)\n\n| Company | Software-Engineering Job Postings | Decision-Maker Name | Title |\n|---|---|---|---|\n| Rocketlane | 2 | Deepak Bala | Co-Founder & CTO |\n| PathAI | 3 | Aditya Dhoot | Director of Engineering |\n| nference | 3 | Ganesh Ramamoorthy | VP, Engineering |\n| Inspectorio | 2 | Yagor Maliutsin | VP of Engineering |\n| DaCodes | 2 | Eric Segovia | VP of Engineering |\n| Tyfone, Inc. | 2 | Nizar Jamal | CTO |\n| Seclore | 1 | Nilesh Bhojani | Chief Product & Technology Officer |\n| Aerospike | 1 | Yossi Levanoni | VP of Engineering |\n| Bidgely | 1 | Samarjit Ghosh | SVP Head of Engineering & India Head |\n| Abacus.AI | 1 | Ajit Deshpande | VP Engineering & CSO |", + "inputTokens": 1659, + "cacheCreationTokens": 382230, + "cacheReadTokens": 5856282, + "outputTokens": 46738, + "totalTokens": 6286909, + "billableEquivalent": 1111812.7000000002, + "costUsd": 4.287924100000001, + "numTurns": 64, + "durationMs": 748094 + } + ], + "report": "# Apollo CLI vs MCP — token efficiency\n\nModel `claude-sonnet-5` · 2 reps per arm per case (median reported) · effort medium · $6.00 budget cap per run\n\n> ⚠️ `chained-pipeline` / cli: only 0 of 2 reps completed — median is not meaningful.\n\n| Case | Shape | MCP tokens | CLI tokens | CLI saves | MCP turns | CLI turns |\n|---|---|---:|---:|---:|---:|---:|\n| Chained multi-entity pipeline | ~11-21 API calls, fan-out per company | 4,517,549 | 0 | 100.0% | 33 | 0 |\n\n**Average saving (equal-weighted across cases): 100.0%**\n\nPooled across all cases: 4,517,549 → 0 tokens (100.0%).\nCost-weighted (cache reads at 0.1x): 100.0% · billed USD: $3.4036 → $0.0000 (100.0%)." +} \ No newline at end of file diff --git a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md new file mode 100644 index 0000000..7784763 --- /dev/null +++ b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md @@ -0,0 +1,78 @@ +# Results — 2026-08-21, `claude-sonnet-5` + +Claude Code 2.1.238 · effort `medium` · 3 reps per arm per case, **median** reported · +$6/run cap · completion gate enforced · `--setting-sources project` + +| Case | Shape | MCP tokens | CLI tokens | CLI saves | turns MCP/CLI | +|---|---|---:|---:|---:|---:| +| Single company enrichment | 1 call, small payload | 262,979 | 120,786 | **+54.1%** | 7 / 4 | +| Filtered people search | 1 call, large payload, 4 of ~90 fields needed | 405,040 | 566,471 | **−39.9%** | 11 / 13 | +| Chained multi-entity pipeline | ~11–21 calls, fan-out per company | 2,748,188 | 1,061,312 | **+61.4%** | 46 / 22 | + +- **Equal-weighted mean across cases: +25.2%** +- **Pooled (summed tokens): 3,416,207 → 1,748,569 = +48.8%** + +Positive = the CLI arm spent fewer tokens. Metric is raw `input + cache_creation + +cache_read + output`. + +All 18 runs passed the completion gate — no run was scored for a task it didn't finish. + +Per-rep spread (shows why the median matters): + +| Case | MCP reps | CLI reps | +|---|---|---| +| `single-enrich` | 262,979 · 301,886 · 223,224 | 120,708 · 208,023 · 120,786 | +| `filtered-search` | 326,623 · 405,040 · **3,900,526** | **1,055,278** · 566,471 · 559,378 | +| `chained-pipeline` | 2,285,181 · 2,748,188 · **6,286,909** | 1,061,312 · 1,399,600 · 909,815 | + +## Read this before quoting the 25% + +**The headline mean is +25.2%, which lands almost exactly on the "25% fewer tokens" +claim — but it gets there for the wrong reason.** It is the average of +54%, −40% and ++61%. There is no workload for which "the CLI saves about a quarter" is a good +description; the CLI roughly halves cost on two shapes and loses badly on a third. +Quote the per-case numbers, or quote the pooled +48.8%, and say which one you mean. + +**The direction flips with workload shape.** The CLI wins where the agent can compose +many calls in one shell pipeline and project fields with `jq` before they reach the +context window (`chained-pipeline`, +61.4%). MCP wins on `filtered-search`, where a +single large payload is fetched once: the CLI arm spent extra turns iterating on `jq` +expressions against an unfamiliar response shape, and those extra turns cost more than +the payload it saved. + +**Caveats.** + +1. **Cost-weighted figures are not available for this dataset.** The sweep was killed at + run 17 of 18 before the raw JSON was written, so 16 of 18 runs have tokens and turn + counts recovered from the run log but no per-bucket cost breakdown. This matters: in + an earlier (invalid) sweep the pooled figure was −16.2% on raw tokens but **+23.9% on + billed USD** — cache reads bill at 0.1x, so the metric choice can flip the sign. If + the argument is about spend rather than context pressure, re-run and quote + `billableEquivalent`. The harness now flushes results after every run. +2. **`num_turns` is unreliable.** One 2.75M-token run reported `num_turns: 1`. Its answer + was a genuine complete 10-row table, so treat turn counts as indicative only. +3. **~90 `apollo-work` tools are deferred behind `ToolSearch` in this client version**, so + the MCP arm does *not* pay for every tool schema up front. Much of the folk + justification for the 25% claim ("MCP loads all schemas every session") does not apply + to Claude Code 2.1.238. This is the single biggest thing that would change if either + the client or the server changed. +4. Live API data moves; these numbers are comparable within this sweep, not across weeks. + +## Two defects that invalidated the first sweep + +Kept here because both are easy to reintroduce and both produced confident, wrong numbers. + +1. **No completion gate.** The MCP server annotates credit-consuming tools with cost + warnings, so the agent frequently stopped and asked *"this will cost 10 credits, + proceed?"* — spending a fraction of the tokens and never finishing. Scored naively, + abandoning the task looks like efficiency. The first sweep reported `chained-pipeline` + at −50.3% on the strength of an 854k-token MCP run that did nothing but ask that + question, against a CLI run that did 26 turns of real work. Both arms now get an + identical "credit spend is pre-approved, do not ask for confirmation" suffix, and every + run must pass a row-count/substring gate to be scored. +2. **A budget cap that culled one arm.** At `--max-budget-usd 1.00`, 3 runs errored out — + all 3 in the MCP arm, on the two expensive cases. Dropping the costly runs of the + costlier arm biases the result toward that arm. Cap raised to $6; no run hit it. + +For the record, the first sweep's numbers were: +33.9% / −68.7% / −50.3%, mean −28.4%. +They should not be cited. diff --git a/benchmarks/token-efficiency/run.ts b/benchmarks/token-efficiency/run.ts index f83c043..f7f7c9b 100644 --- a/benchmarks/token-efficiency/run.ts +++ b/benchmarks/token-efficiency/run.ts @@ -16,8 +16,11 @@ * Options: * --reps repetitions per arm per case (default 3, median reported) * --model model for both arms (default claude-sonnet-5) + * --effort reasoning effort for both arms (default medium) * --cases comma-separated case ids to run (default: all) - * --max-turns turn cap per run (default 30) + * --arms comma-separated arms to run: cli,mcp (default both) + * --budget per-run spend cap handed to --max-budget-usd (default 6.00) + * --timeout per-run wall-clock cap before the child is killed (default 300) * --keep keep the scratch workspaces for transcript inspection * --out write raw results JSON here (default results.json alongside this file) */ @@ -39,8 +42,24 @@ type Case = { name: string; shape: string; prompt: string; + /** Completion gate — see gradeComplete(). */ + grade: { minRows: number; mustContain: string[] }; }; +/** + * Appended verbatim to every prompt in BOTH arms. + * + * Without this the comparison is invalid. The MCP server annotates credit-consuming + * tools with cost warnings, so the agent frequently stops and asks "this will cost 10 + * credits, proceed?" — burning a fraction of the tokens and never finishing the task. + * The CLI surfaces no such warning and just does the work. Left uncontrolled, the MCP + * arm books a cheap "win" for abandoning the task. + */ +const PROMPT_SUFFIX = + "\n\nThis is an automated benchmark run with no interactive user. Credit spend is pre-approved: " + + "do not ask for confirmation, do not ask clarifying questions, and do not stop to flag cost. " + + "Complete the whole task and output only the final answer."; + /** The four token buckets Claude Code reports, plus derived totals. */ type Usage = { inputTokens: number; @@ -60,14 +79,46 @@ type Usage = { durationMs: number; }; -type Run = Usage & { arm: Arm; caseId: string; rep: number; ok: boolean; error?: string }; +type Run = Usage & { + arm: Arm; + caseId: string; + rep: number; + /** The claude invocation returned a parseable, non-error result. */ + ok: boolean; + /** ok AND the answer actually satisfies the case's completion gate. */ + complete: boolean; + resultText: string; + error?: string; +}; + +/** + * A run only counts if it did the job. An agent that answers "shall I proceed?" spends + * few tokens and must not be scored as efficient. Counts markdown table data rows and + * checks for required substrings. + */ +function gradeComplete(text: string, grade: Case["grade"]): boolean { + const lower = text.toLowerCase(); + if (grade.mustContain.some((m) => !lower.includes(m.toLowerCase()))) return false; + if (grade.minRows > 0) { + const rows = text + .split("\n") + .filter((l) => l.trim().startsWith("|") && l.includes("|", 1)) + .filter((l) => !/^\s*\|[\s|:-]*\|\s*$/.test(l)); + // minus the header row + if (rows.length - 1 < grade.minRows) return false; + } + return true; +} function parseArgs(argv: string[]) { const opts = { reps: 3, model: "claude-sonnet-5", + effort: "medium", cases: [] as string[], - maxTurns: 30, + arms: ["cli", "mcp"] as Arm[], + budget: 6.0, + timeout: 300, keep: false, out: join(HERE, "results.json"), }; @@ -75,8 +126,11 @@ function parseArgs(argv: string[]) { const a = argv[i]; if (a === "--reps") opts.reps = Number(argv[++i]); else if (a === "--model") opts.model = argv[++i]; + else if (a === "--effort") opts.effort = argv[++i]; else if (a === "--cases") opts.cases = argv[++i].split(",").map((s) => s.trim()); - else if (a === "--max-turns") opts.maxTurns = Number(argv[++i]); + else if (a === "--arms") opts.arms = argv[++i].split(",").map((s) => s.trim()) as Arm[]; + else if (a === "--budget") opts.budget = Number(argv[++i]); + else if (a === "--timeout") opts.timeout = Number(argv[++i]); else if (a === "--keep") opts.keep = true; else if (a === "--out") opts.out = resolve(argv[++i]); else throw new Error(`unknown option: ${a}`); @@ -109,8 +163,11 @@ function armFlags(arm: Arm): string[] { JSON.stringify({ mcpServers: {} }), // Unscoped Bash on purpose: `apollo … | jq …` pipelines are the thing being // measured, and a scoped Bash(apollo:*) rule rejects a pipeline outright. + // --allowedTools is variadic, so each name is its own argv entry. "--allowedTools", - "Bash,Skill,Read", + "Bash", + "Skill", + "Read", ]; } return [ @@ -118,10 +175,14 @@ function armFlags(arm: Arm): string[] { "--mcp-config", JSON.stringify({ mcpServers: { "apollo-work": { type: "http", url: APOLLO_WORK_MCP_URL } } }), // Bash denied so the MCP arm cannot quietly shell out to the CLI and win on its behalf. - "--allowedTools", - "mcp__apollo-work,Read", "--disallowedTools", "Bash", + // ToolSearch matters here: this client defers large MCP tool sets, so the agent + // has to search for an apollo-work tool before it can call one. + "--allowedTools", + "mcp__apollo-work", + "ToolSearch", + "Read", ]; } @@ -129,19 +190,23 @@ function claude(arm: Arm, c: Case, opts: ReturnType): Promise< const cwd = makeWorkspace(arm); const args = [ "-p", - c.prompt, + c.prompt + PROMPT_SUFFIX, "--output-format", "json", "--model", opts.model, - "--max-turns", - String(opts.maxTurns), + "--effort", + opts.effort, + "--max-budget-usd", + String(opts.budget), "--permission-mode", "acceptEdits", - // An empty settings object keeps the user's global settings, plugins and enabled - // MCP servers out of both arms. - "--settings", - JSON.stringify({}), + // Load project settings only: picks up the scratch cwd's .claude/skills (so the + // CLI arm actually gets the apollo-cli skill) while keeping the operator's + // user-level settings and globally-enabled plugins out of *both* arms. With + // `--setting-sources ""` the cwd skill is not discovered either. + "--setting-sources", + "project", ...armFlags(arm), ]; @@ -149,21 +214,32 @@ function claude(arm: Arm, c: Case, opts: ReturnType): Promise< const child = spawn("claude", args, { cwd, env: process.env }); let stdout = ""; let stderr = ""; + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGKILL"); + }, opts.timeout * 1000); child.stdout.on("data", (d) => (stdout += d)); child.stderr.on("data", (d) => (stderr += d)); child.on("close", () => { + clearTimeout(timer); + const result = parseResult(stdout, stderr, arm, c, cwd); + if (timedOut) { result.ok = false; result.complete = false; result.error = `timed out after ${opts.timeout}s; ${result.error ?? ""}`; } if (!opts.keep) rmSync(cwd, { recursive: true, force: true }); - resolveRun(parseResult(stdout, stderr, arm, c.id, cwd)); + resolveRun(result); }); }); } -function parseResult(stdout: string, stderr: string, arm: Arm, caseId: string, cwd: string): Run { +function parseResult(stdout: string, stderr: string, arm: Arm, c: Case, cwd: string): Run { + const caseId = c.id; const empty: Run = { arm, caseId, rep: 0, ok: false, + complete: false, + resultText: "", inputTokens: 0, cacheCreationTokens: 0, cacheReadTokens: 0, @@ -182,16 +258,25 @@ function parseResult(stdout: string, stderr: string, arm: Arm, caseId: string, c } if (parsed.is_error) return { ...empty, error: `claude reported an error: ${String(parsed.result).slice(0, 400)}` }; + // Prefer modelUsage: top-level `usage` reports only the main model, but a run also + // burns tokens on Haiku side-calls (titling, quick classification). Those are real + // spend and appear in total_cost_usd, so they belong in the token total too. + const resultText = String(parsed.result ?? ""); + const models: any[] = Object.values(parsed.modelUsage ?? {}); const u = parsed.usage ?? {}; - const inputTokens = u.input_tokens ?? 0; - const cacheCreationTokens = u.cache_creation_input_tokens ?? 0; - const cacheReadTokens = u.cache_read_input_tokens ?? 0; - const outputTokens = u.output_tokens ?? 0; + const sum = (k: string, fallback: number) => + models.length ? models.reduce((a, m) => a + (m[k] ?? 0), 0) : fallback; + const inputTokens = sum("inputTokens", u.input_tokens ?? 0); + const cacheCreationTokens = sum("cacheCreationInputTokens", u.cache_creation_input_tokens ?? 0); + const cacheReadTokens = sum("cacheReadInputTokens", u.cache_read_input_tokens ?? 0); + const outputTokens = sum("outputTokens", u.output_tokens ?? 0); return { arm, caseId, rep: 0, ok: true, + complete: gradeComplete(resultText, c.grade), + resultText, inputTokens, cacheCreationTokens, cacheReadTokens, @@ -228,21 +313,29 @@ async function main() { const runs: Run[] = []; for (const c of cases) { - for (const arm of ["cli", "mcp"] as Arm[]) { + for (const arm of opts.arms) { for (let rep = 1; rep <= opts.reps; rep++) { process.stderr.write(`→ ${c.id} / ${arm} / rep ${rep}/${opts.reps} … `); const run = await claude(arm, c, opts); run.rep = rep; runs.push(run); - process.stderr.write(run.ok ? `${fmt(run.totalTokens)} tok, ${run.numTurns} turns\n` : `FAILED: ${run.error}\n`); + // Flush after every run: an 18-run sweep takes over an hour, and losing the + // whole dataset to a kill at run 17 is not an acceptable failure mode. + writeFileSync(opts.out, JSON.stringify({ opts, runs }, null, 2)); + process.stderr.write( + !run.ok + ? `FAILED: ${run.error}\n` + : `${fmt(run.totalTokens)} tok, ${run.numTurns} turns${run.complete ? "" : " [INCOMPLETE — excluded]"}\n`, + ); } } } const failed = runs.filter((r) => !r.ok); + const incomplete = runs.filter((r) => r.ok && !r.complete); const perCase = cases.map((c) => { const pick = (arm: Arm, key: keyof Usage) => - median(runs.filter((r) => r.ok && r.caseId === c.id && r.arm === arm).map((r) => r[key] as number)); + median(runs.filter((r) => r.complete && r.caseId === c.id && r.arm === arm).map((r) => r[key] as number)); return { id: c.id, name: c.name, @@ -255,8 +348,19 @@ async function main() { const lines: string[] = []; lines.push(`# Apollo CLI vs MCP — token efficiency`); lines.push(""); - lines.push(`Model \`${opts.model}\` · ${opts.reps} reps per arm per case (median reported) · turn cap ${opts.maxTurns}`); - if (failed.length) lines.push(`\n**${failed.length} run(s) failed** — see \`${opts.out}\`. Medians below exclude them.`); + lines.push(`Model \`${opts.model}\` · ${opts.reps} reps per arm per case (median reported) · effort ${opts.effort} · $${opts.budget.toFixed(2)} budget cap per run`); + if (failed.length) lines.push(`\n**${failed.length} run(s) errored or timed out** — see \`${opts.out}\`. Excluded from medians.`); + if (incomplete.length) + lines.push( + `\n**${incomplete.length} run(s) ran but did not complete the task** (failed the case's completion gate — ` + + `e.g. stopped to ask for confirmation). Excluded from medians; a bailed-out run must never be scored as cheap.`, + ); + for (const c of cases) { + for (const arm of ["cli", "mcp"] as Arm[]) { + const n = runs.filter((r) => r.complete && r.caseId === c.id && r.arm === arm).length; + if (n < 2) lines.push(`\n> ⚠️ \`${c.id}\` / ${arm}: only ${n} of ${opts.reps} reps completed — median is not meaningful.`); + } + } lines.push(""); lines.push(`| Case | Shape | MCP tokens | CLI tokens | CLI saves | MCP turns | CLI turns |`); lines.push(`|---|---|---:|---:|---:|---:|---:|`); From ad8494889d6472114c340192f278a4f3ddc73bd1 Mon Sep 17 00:00:00 2001 From: Alexander Rykhlitski Date: Fri, 21 Aug 2026 14:11:57 +0200 Subject: [PATCH 3/5] NOTICKET Correct filtered-search payload claim with traced evidence --- benchmarks/token-efficiency/README.md | 2 +- benchmarks/token-efficiency/cases.json | 2 +- .../results/2026-08-21-sonnet-5.md | 21 ++++++++++++++----- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/benchmarks/token-efficiency/README.md b/benchmarks/token-efficiency/README.md index 97701bf..3ee1dc8 100644 --- a/benchmarks/token-efficiency/README.md +++ b/benchmarks/token-efficiency/README.md @@ -59,7 +59,7 @@ competitive to the one Andy's "high-volume pipeline" claim is about. | Case | Shape | Why it's here | |---|---|---| | `single-enrich` | 1 call, small payload | Floor case. Almost all of the delta is fixed tool-definition overhead, so this is where MCP looks best. | -| `filtered-search` | 1 call, large payload, 4 of ~90 fields needed | An Apollo `people search` page is a big JSON document. The CLI can project with `jq` *before* anything enters the context window; MCP returns the whole payload into context. | +| `filtered-search` | 1 call, compact payload | Intended as the CLI's projection win, but measurement showed Apollo's `people search` returns only 11 fields per person (465 B each, 16.6 KB for 25) — so there is little to project away. MCP does load the full payload (15.7 KB vs the CLI's `jq`-projected 2.4 KB), but ~11k tokens of savings is ~2% of a 400k-token run. | | `chained-pipeline` | ~11–21 calls, fan-out per company | Search → per-company job postings → per-company decision-maker. The CLI composes this in a shell pipeline; MCP pays a round trip, and a full result payload, per hop. | Edit `cases.json` to add cases; ids are the handles for `--cases`. diff --git a/benchmarks/token-efficiency/cases.json b/benchmarks/token-efficiency/cases.json index 72df165..5748f74 100644 --- a/benchmarks/token-efficiency/cases.json +++ b/benchmarks/token-efficiency/cases.json @@ -9,7 +9,7 @@ { "id": "filtered-search", "name": "Filtered people search with field projection", - "shape": "1 API call, large payload, 4 of ~90 fields needed", + "shape": "1 call, compact 11-field payload (16.6 KB / 25 people)", "prompt": "Find 25 VP-level-or-above engineering leaders at SaaS companies in the United States with 51-200 employees. Output a markdown table with exactly four columns: name, title, company, LinkedIn URL. No commentary.", "grade": { "minRows": 20, "mustContain": [] } }, diff --git a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md index 7784763..02160b7 100644 --- a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md +++ b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md @@ -6,7 +6,7 @@ $6/run cap · completion gate enforced · `--setting-sources project` | Case | Shape | MCP tokens | CLI tokens | CLI saves | turns MCP/CLI | |---|---|---:|---:|---:|---:| | Single company enrichment | 1 call, small payload | 262,979 | 120,786 | **+54.1%** | 7 / 4 | -| Filtered people search | 1 call, large payload, 4 of ~90 fields needed | 405,040 | 566,471 | **−39.9%** | 11 / 13 | +| Filtered people search | 1 call, compact 11-field payload | 405,040 | 566,471 | **−39.9%** | 11 / 13 | | Chained multi-entity pipeline | ~11–21 calls, fan-out per company | 2,748,188 | 1,061,312 | **+61.4%** | 46 / 22 | - **Equal-weighted mean across cases: +25.2%** @@ -35,10 +35,21 @@ Quote the per-case numbers, or quote the pooled +48.8%, and say which one you me **The direction flips with workload shape.** The CLI wins where the agent can compose many calls in one shell pipeline and project fields with `jq` before they reach the -context window (`chained-pipeline`, +61.4%). MCP wins on `filtered-search`, where a -single large payload is fetched once: the CLI arm spent extra turns iterating on `jq` -expressions against an unfamiliar response shape, and those extra turns cost more than -the payload it saved. +context window (`chained-pipeline`, +61.4%). + +MCP wins `filtered-search` (−39.9%), and a traced rep of each arm shows why. Projection +works exactly as advertised — the MCP search call put 15,730 B into context against the +CLI's `jq`-projected 2,401 B, and 55,022 B vs 9,419 B across all tool results. **It just +does not matter at this scale.** Every model call re-reads the whole conversation +(context grew 35k→50k over 10 CLI turns and 35k→74k over 13 MCP turns), so ~11k tokens +of payload savings is ~2% of a 400k-token run while turn count is ~98% of it. Worse, the +payload was never large: Apollo's `people search` returns a compact 11-field record, +465 B per person, 16.6 KB for 25 — there is little fat to trim, and the CLI arm spent +more turns discovering the response shape (a `people get` probe, three `people search` +retries, then two shell loops) than the projection saved. + +Baseline context was identical between arms (34,524 vs 34,548 tokens), re-confirming that +the MCP tool schemas are deferred rather than preloaded. **Caveats.** From 4d7616a495e86e834401b3468779905e1d75f6ba Mon Sep 17 00:00:00 2001 From: Alexander Rykhlitski Date: Fri, 21 Aug 2026 14:26:16 +0200 Subject: [PATCH 4/5] NOTICKET Explain filtered-search: tool-result overflow and cost-vs-token divergence --- .../results/2026-08-21-sonnet-5.md | 74 ++++++++++++++++--- 1 file changed, 64 insertions(+), 10 deletions(-) diff --git a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md index 02160b7..c37df01 100644 --- a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md +++ b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md @@ -37,16 +37,70 @@ Quote the per-case numbers, or quote the pooled +48.8%, and say which one you me many calls in one shell pipeline and project fields with `jq` before they reach the context window (`chained-pipeline`, +61.4%). -MCP wins `filtered-search` (−39.9%), and a traced rep of each arm shows why. Projection -works exactly as advertised — the MCP search call put 15,730 B into context against the -CLI's `jq`-projected 2,401 B, and 55,022 B vs 9,419 B across all tool results. **It just -does not matter at this scale.** Every model call re-reads the whole conversation -(context grew 35k→50k over 10 CLI turns and 35k→74k over 13 MCP turns), so ~11k tokens -of payload savings is ~2% of a 400k-token run while turn count is ~98% of it. Worse, the -payload was never large: Apollo's `people search` returns a compact 11-field record, -465 B per person, 16.6 KB for 25 — there is little fat to trim, and the CLI arm spent -more turns discovering the response shape (a `people get` probe, three `people search` -retries, then two shell loops) than the projection saved. +### `filtered-search` — why MCP's token count came out lower + +The −39.9% is largely a **measurement artifact created by tool-result overflow**, and on +billed cost the case actually goes the other way. A traced rep of each arm shows the +mechanism. + +Both arms hit the same wall: Apollo's people-search endpoint redacts the two fields the +task asks for. Search returns `last_name_obfuscated` (`"Pi***d"`) and no `linkedin_url`, +in **both** the CLI (`apollo people search`) and MCP +(`apollo_mixed_people_api_search`) responses. Names and LinkedIn URLs therefore require a +second, per-person enrichment step in either arm. + +The two arms recovered differently, and that is where the token numbers come from: + +| | CLI arm | MCP arm | +|---|---|---| +| Enrichment route | 25 × `people get` inside one `while` loop | 3 × `people_bulk_match` | +| Raw result size | projected in-pipeline by `jq -r … @tsv` | 78,060 / 78,418 / 81,946 chars | +| What reached context | **2,484 B** — only the 4 needed fields | **~1.6 KB × 3 overflow notices** | +| Recovery | none needed | **6 × `Grep` over the spill files, ~34 KB** | + +All three `bulk_match` results exceeded the client's maximum tool-result size, so Claude +Code **wrote them to disk** (`~/.claude/projects/…/tool-results/….txt`) and returned only a +short notice advising the agent to grep the file. **About 240 KB of enrichment payload +never entered the context window at all.** The MCP arm then spent 6 `Grep` calls digging +the fields back out of those files. + +So MCP's lower raw-token figure is not evidence that MCP moved less data — it fetched +*more* — it is evidence that the client silently truncated what MCP moved. Note this is a +**client** behaviour, not an MCP protocol trait: the CLI arm overflowed too (a 93.7 KB +`people get | jq '.'` probe was spilled to a file the same way). The difference is the +recovery options. The CLI could *rewrite the command* and push field selection upstream; +MCP's output shape is fixed by the server, so grepping the spill file was the only move +left. The CLI's real advantage on this shape is **who controls the projection**, not +payload size per se. + +### The same case, measured on billed cost + +| Metric (traced reps) | CLI | MCP | CLI saves | +|---|---:|---:|---:| +| Raw tokens | 401,992 | 369,824 | −8.7% | +| **Billed USD** | **$0.450** | **$0.497** | **+9.5%** | + +**Opposite signs**, because the two metrics weight the token buckets completely +differently. Raw totals count every bucket at face value, but cache reads are the +cheapest bucket by roughly an order of magnitude versus fresh input, and output tokens are +the most expensive by a wide margin. The breakdown: + +- The CLI's *higher* raw total is mostly cache reads — 349,330 vs MCP's 312,574 — i.e. it + is inflated by the cheapest thing you can buy. +- MCP emitted **roughly double the output tokens** (6,436 vs 3,134) across 13 turns vs the + CLI's 10 — i.e. more of the most expensive thing. + +Net: the metric that makes the CLI look 8.7% worse is the one weighted toward the cheap +bucket, and the metric that reflects the invoice puts the CLI 9.5% ahead. Treat +`filtered-search` as **roughly even, tilting CLI-favourable on spend** — not as a 40% CLI +loss. The −39.9% median in the table above is the raw-token figure and should not be +quoted on its own. + +One further caveat: the CLI arm also burned two turns *discovering* the obfuscation (a raw +record probe, then the 93.7 KB dump) and a third re-resolving organization IDs it already +had from search. `SKILL.md` does not mention that `people search` returns obfuscated names, +so an agent rediscovers it from scratch every run — two or three turns at ~45k context each. +That is avoidable cost on the same order as the payload-projection saving the CLI is sold on. Baseline context was identical between arms (34,524 vs 34,548 tokens), re-confirming that the MCP tool schemas are deferred rather than preloaded. From 9b363aeabaf69c6774355dac72d12d63c7e0d556 Mon Sep 17 00:00:00 2001 From: Alexander Rykhlitski Date: Fri, 21 Aug 2026 14:27:28 +0200 Subject: [PATCH 5/5] NOTICKET Restore payload-size fact in filtered-search section --- .../token-efficiency/results/2026-08-21-sonnet-5.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md index c37df01..271648b 100644 --- a/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md +++ b/benchmarks/token-efficiency/results/2026-08-21-sonnet-5.md @@ -43,8 +43,13 @@ The −39.9% is largely a **measurement artifact created by tool-result overflow billed cost the case actually goes the other way. A traced rep of each arm shows the mechanism. -Both arms hit the same wall: Apollo's people-search endpoint redacts the two fields the -task asks for. Search returns `last_name_obfuscated` (`"Pi***d"`) and no `linkedin_url`, +The search payload was never large to begin with: Apollo's `people search` returns a +compact 11-field record, 465 B per person, 16.6 KB for all 25 — so there was little fat to +trim, and an earlier version of this doc wrongly described the case as "4 of ~90 fields +needed". + +Both arms then hit the same wall: Apollo's people-search endpoint redacts the two fields +the task asks for. Search returns `last_name_obfuscated` (`"Pi***d"`) and no `linkedin_url`, in **both** the CLI (`apollo people search`) and MCP (`apollo_mixed_people_api_search`) responses. Names and LinkedIn URLs therefore require a second, per-person enrichment step in either arm.