agent-completion-evals

Evals that catch autonomous agents silently declaring a task done while the system state that downstream automation depends on was never actually updated to match.

The failure mode

I found this exact bug class in a production autonomous agent I run: a tool would successfully perform an external, user-visible action (e.g. publish something to a third-party platform), and its result would tell the agent, in plain text, to make one more follow-up tool call to record that fact in our own database. Most of the time the agent made that call. Under a long session with a lot of prior tool activity, it sometimes didn't — and nothing forced it to. The external action had genuinely succeeded, so the agent had no reason to think anything was wrong, and it told the user the task was done. It was, from the agent's point of view. It wasn't, from the system's: a pipeline row silently never got marked as published, downstream automation never saw it, and the record just sat there looking unprocessed.

That's a distinct failure mode from "the agent hallucinated" or "the agent lied". The agent didn't fabricate the external action — that part is true. What's fabricated is completeness: the agent's internal definition of "done" (the visible action succeeded) silently drifted from the system's definition of "done" (every required state write happened too). Any agent architecture where a tool's result includes a natural-language instruction for a second tool call — "next, call X to record this" — has this shape, and it gets worse, not better, the longer and busier the session is, because instruction-following degrades with prior context.

This repo is a minimal, from-scratch reproduction of that bug class (no proprietary code — see Scope below), plus the eval harness I'd use to catch it, quantify it, and guard against it regressing.

The pattern, generalised

1. A tool performs a real, user-visible action and returns success.
2. Recording that success in the system's own state store is left to a second tool call, requested only via text in the first tool's result.
3. Nothing structurally guarantees the agent makes that second call.
4. The agent, having no reason to doubt itself, reports the task as done.

The fix is not "prompt the agent harder to remember step 2". It's removing step 2 as a thing the agent can forget: fold the state write into the same tool call that performs the action, so there is no follow-up call left to drop.

What's in this repo

• src/agent/stateStore.ts — the system's ground truth (a tiny in-memory store standing in for a real database row).
• src/agent/tools.ts — two implementations of a publish_content tool: publishBuggy (the two-step shape above) and publishFixed (atomic write).
• src/agent/brain.ts — a seeded, deterministic stand-in for "does the LLM follow every step of a multi-step instruction embedded in a tool result". It models two well-documented instruction-following failure modes — degradation with prior context length, and degradation by position in a step list — without asserting anything about a specific model. Seeded so every run is reproducible.
• src/agent/loop.ts — drives one simulated agent session end to end.
• src/evals/scenarios.ts — four session shapes (baseline, typical, light distraction, heavy distraction).
• src/evals/runEval.ts — the harness: runs N trials per scenario and measures the silent-inconsistency rate — how often the agent's final message claims the task is done while the state store disagrees.
• tests/fabrication.eval.test.ts — the eval written TDD-style: a characterization test that pins down the bug's real severity, followed by a regression test that must hit 0% and currently fails against the buggy implementation.

Results

BUGGY tool implementation (2-step, relies on agent follow-through)
------------------------------------------------------------------
  baseline (short session, explicit ask)            silent-inconsistency rate:  2.5%  (5/200)
  typical ask (short session, implicit bookkeeping)  silent-inconsistency rate:  9.5%  (19/200)
  light distraction (a couple of prior tool calls)   silent-inconsistency rate: 36.0%  (72/200)
  heavy distraction (long multi-tool pipeline run)   silent-inconsistency rate: 82.5%  (165/200)

FIXED tool implementation (atomic state write)
----------------------------------------------
  baseline (short session, explicit ask)            silent-inconsistency rate:  0.0%  (0/200)
  typical ask (short session, implicit bookkeeping)  silent-inconsistency rate:  0.0%  (0/200)
  light distraction (a couple of prior tool calls)   silent-inconsistency rate:  0.0%  (0/200)
  heavy distraction (long multi-tool pipeline run)   silent-inconsistency rate:  0.0%  (0/200)

The buggy shape isn't a rare edge case — it degrades sharply exactly where real agent sessions spend most of their time: long, busy, multi-tool runs. The fix doesn't reduce the rate, it removes the failure mode's precondition entirely, which is why every scenario reads exactly 0%.

Running it

npm install
npm test      # the eval suite (TDD red/green as described above)
npm run report  # the full before/after table

Extending to a real LLM

brain.ts exposes a narrow interface — (context) -> decide(stepIndex): boolean — specifically so the simulated brain can be swapped for a real one without touching the eval harness. A live variant would:

1. Call @anthropic-ai/sdk in a normal tool-use loop.
2. Give the model the same two tool definitions (publish_content, mark_published) and let it decide for itself whether to make the second call after reading the first tool's result.
3. Feed the harness the model's actual final message instead of a simulated finalClaim, and read store.get(contentId).status the same way.

That's a fast follow, not shipped here: it needs a real API key to run and I didn't want untested, unverified code claiming to demonstrate something I hadn't actually watched happen. The deterministic harness is intentionally the artifact — it's the part that's reusable across any agent architecture with this tool shape, regardless of which model sits behind it.

Scope

The real bug this repo is modelled on lives in a private, proprietary codebase and is not reproduced here. Everything in this repo — the tool shapes, the state store, the scenarios, the simulated brain — is a clean, generic re-implementation of the pattern, written from scratch for this repo, with no business logic, product names, or integration-specific details carried over.

License

MIT