From f2918cfb3e32ca8633536d4b7e0f44783f639bdb Mon Sep 17 00:00:00 2001 From: Hunter B Date: Tue, 7 Jul 2026 22:24:27 -0700 Subject: [PATCH] docs: test-time-compute design (verify tool + sub-agent reasoning) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Approved TTC design: one shared CriticEngine behind three triggers — the agent-invoked verify tool (#4196, MVP #4199), advisor watcher (#3982), and post-turn verification gates (#4013) — plus clamp->floor sub-agent reasoning (#4137). Synthesized from three independent reviews (verify-tool contributor, GLM 5.2, internal analysis). Design only, no code; v0.8.69, behind stopship. Drafted with agent assistance. --- docs/TTC_DESIGN.md | 75 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 docs/TTC_DESIGN.md diff --git a/docs/TTC_DESIGN.md b/docs/TTC_DESIGN.md new file mode 100644 index 000000000..637500913 --- /dev/null +++ b/docs/TTC_DESIGN.md @@ -0,0 +1,75 @@ +# Test-Time Compute (TTC) in CodeWhale — design + +Status: **approved direction** (maintainer greenlit). Synthesized from three independent reviews — the verify-tool implementation contributor, GLM 5.2, and an internal analysis — which all converged. This doc is the spec; implementation lands post-stopship (v0.8.69) and is split so nothing here blocks the v0.8.68 release. + +## What TTC means here + +Spend *more inference at decision time* for a better answer, on the **agent's own judgment** — not an always-on tax. Two capabilities: + +- **(A) An agent-invoked `verify` / critic pass** — the model chooses to adversarially review its own recent work before claiming done, to catch "green-but-wrong" (motivating case: a fix passed 16/16 CI but only covered a CLI path, not the interactive TUI; a critic pass caught it, deterministic CI did not). +- **(B) Reasoning-effort escalation for sub-agents** — un-cap the hard `Low` clamp so a Fleet role can think at the tier its job needs. + +## Core principle: one `CriticEngine`, three triggers + +Factor a single `CriticEngine` that owns: the **target-context snapshot** (recent tool calls + the claimed-done state + a diff/evidence gather), the **prompt-template family** (adversarial "refute it"), the **reasoning effort** (Max), the **tools-disabled** flag, and the **structured verdict schema** (`verdict: pass|fail|uncertain`, findings `[{severity, issue, evidence, suggested_fix}]`, `unresolved_risk`). + +Then three *distinct* entry points share that engine but **keep their own invocation contracts** — do NOT merge the triggers: + +| Trigger | Contract | Issue | +|---|---|---| +| `verify` **tool** | sync, model-chosen, default-on | #4196 (MVP in PR #4199) | +| advisor **watcher** | async, rate-limited, off-by-default | #3982 | +| verification **gates** | post-turn, deterministic (compile/test/lint/review) | #4013 | + +> Unify the **engine**, never the **trigger**. Merging sync/model-chosen + async/throttled + post-turn/deterministic produces a Frankenstein. Sharing the engine keeps drift at zero while each entry point keeps its character. + +## (A) The `verify` tool + +**Why a tool, not a critic sub-agent:** a `verify` tool is *structurally isomorphic to the existing `review` tool* — same `ToolSpec` trait, same `ToolRegistryBuilder` path, same `Feature` gate, same `MessageRequest` reasoning normalization. It inherits every existing guarantee for almost no new surface. A critic *sub-agent* would be a **second runtime with a second policy surface** (spawn-depth, allowlist, sub-agent tier resolution) — the textbook bolted-on smell. (A sub-agent critic that autonomously explores may return as an opt-in follow-up *behind the same tool contract* once #4193's spawn work has settled — but it is NOT the default.) + +**Interface:** registered via `ToolRegistryBuilder::with_verify(critic)`, gated by a `Feature` flag. Input: `claim` (required) + optional `requirement`, `scope` (`diff|staged|none`), `base`, `files[]`, `focus`. It snapshots evidence deterministically (diff by scope — **including uncommitted working-tree changes when a base is given**, per PR #4199 fix — plus named files), builds ONE `MessageRequest` at `ReasoningEffort::Max` with **tools disabled**, and returns the structured verdict as a tool result. + +**Where it plugs in:** the standard tool loop. No new control plane. The model invokes it like `read`/`edit`/`review`. + +**How the model decides (and abuse is bounded):** +- **Constitution rule** (harness-enforced, not prose): verify before claiming done when debugging, on multi-file changes, security-sensitive edits, or changes touching *divergent surfaces* (CLI vs TUI, sync vs async). The green-CI-but-wrong case is the canonical trigger. +- **Engine-level rate limit / `TtcBudget`**: per-turn (one verify) and per-session budget, held in the **engine**, not in `MessageRequest`. +- **Feedback loop**: the verdict returns to the model; several consecutive clean verdicts should, via the Constitution, discourage further calls that session. The model sees its own hit rate and self-corrects. + +**Verdict semantics** (PR #4199, already hardened): any finding at **medium severity or above** forces `unresolved_risk = true` and downgrades an `upheld` verdict to `uncertain`; only `low` nits are exempt. Advisory by default; a Constitution rule may make a `fail`/`unresolved_risk` verdict something the agent must address before claiming done (soft-block in the harness, not hard-coded in the tool). + +**Recursion / cost bounding:** +1. *By construction*: tools disabled inside the critic call ⇒ no further tool calls. +2. *By registry*: `verify` is **structurally refused** when building a `SubAgentRuntime` allowlist — via a `Feature::CriticProducerOnly` (or equivalent) the `ToolRegistryBuilder` checks. The spawn-depth guard is only the *backup* line, not the primary. +3. *By budget*: the per-session `TtcBudget` consulted by the engine. + +## (B) Sub-agent reasoning — replace the clamp with a floor + +The bug in `auto_reasoning.rs` was never "Low is wrong" — it's that **Low is a *ceiling*** for sub-agents. Fix: **Low stays the default *floor*; remove the ceiling.** + +**Tier resolution order:** `Profile (#4137) → explicit task override → session default → Low`. +- `SubAgentRuntime.reasoning_effort` continues to be forwarded verbatim. +- `Auto` inside a sub-agent resolves through a **Fleet-role-aware resolver** (a `review`-role profile pins High, a `search`-role pins Low, a `planner` pins Max) — NOT the global keyword resolver. +- The `agent` tool's `reasoning_effort` becomes "inherit from the Fleet profile unless explicitly overridden at spawn." + +**Do not change the default floor from Low** — sub-agent traffic is majority search/lookup, and raising the floor silently raises cost on every existing fleet. Non-surprising > clever. This composes with #4137 (profile carries the tier alongside provider/model) rather than competing with it. + +## Anti-patterns (what would read as bolted-on *in CodeWhale specifically*) + +1. A **second critic implementation** — if `verify`/`review`/#3982/#4013 each roll their own prompt+call+parse, four paths diverge on the first bug. The single `CriticEngine` is the whole game. +2. A **non-tool control plane for reasoning escalation** — CodeWhale's model contract is tool-shaped; a side-channel breaks symmetry and bleeds into every provider adapter. The `verify` call *is* the escalation (Max internally). One vocabulary. +3. **Recursion policy in Constitution prose** — enforce it in the registry builder; the depth guard is secondary. +4. **Cost accounting leaking into `MessageRequest`** — budget belongs in the engine + a session `TtcBudget`. Don't make every tool cost-aware. +5. **`auto_reasoning.rs` becoming TTC-aware** — Auto resolves *effort per turn*; the model decides verify. Keep them separate or you get non-determinism the user can't reason about. +6. **Gate-ordering ambiguity between #4013 and `verify`** — different lifecycle points (mid-turn/model-chosen vs post-turn/deterministic). Document in the Constitution so contributors don't merge them. +7. **Conflating the watcher's contract with verify's** — #3982 is async/throttled/off; `verify` is sync/chosen/on. Share the engine; never the trigger. + +## Issue map & sequencing + +- **#4196** — `verify` tool. MVP in **PR #4199** (direct-critic, Max, tools-disabled, recursion-guarded, verdict-hardened). Refactor to sit on the extracted `CriticEngine` before merge. *Config-disjoint (`crates/tui/src/tools/`).* +- **CriticEngine extraction** — new; refactor `review`'s call/parse into the shared engine, then have `verify` consume it. Prereq for wiring #3982/#4013 to it later. +- **#4137** — Fleet profile carries a `reasoning` tier; drives (B). *Touches `crates/config` — sequence with the config work + after #4136 (canonical AgentProfile) and #4193 (landed).* +- **(B) resolver** — `auto_reasoning.rs` clamp→floor + Fleet-role-aware Auto resolution. +- **#3982 / #4013** — rebase onto the `CriticEngine` as additional triggers (later). + +All of the above is **v0.8.69, behind the v0.8.68 stopship** (which is green).