feat(v1): dialect-routed interception — byte relay on matching protocols, typed translate otherwise

[xeophon]

Summary

Stacked on #1651 (native provider clients); merges into feat/nano-as-v1 once that lands.

This PR combines the two halves discussed in #1654: route-detected wire dialects on ingress (from that PR's design) and the native provider clients on egress (#1651), reconciled by one request-time rule:

• Relay — when the rollout's client natively speaks the request's dialect (Client.dialect == Dialect.name), the program's request bytes are forwarded verbatim and the provider's response (JSON or SSE) is relayed back untouched. The dialect parses a copy only to record the trace. No field is lost to a typed round-trip — not reasoning, not cache_control, not any future provider field.
• Translate — otherwise (training via the renderer or chat→vLLM, or any cross-protocol pairing such as a claude-code harness against a chat endpoint), the request is parsed into typed messages, the client runs it in its own wire format, and the typed Response is serialized back in the dialect the program spoke.

The selection is a string compare per request: no model→client tables, no Harness.DIALECT, no body sniffing. The harness only chooses env vars, as before.

What's new

• verifiers/v1/dialects/ — one module per native format (chat, anthropic, responses), each owning its route, auth carrier (Bearer / x-api-key), wire→vf parsers (parse_request / parse_response / parse_stream), and vf→wire serializers (serialize_response / serialize_stream, translate path only). Response parsing reuses the clients' response_from_wire, so provider_state (thinking blocks, Responses output items, reasoning_details) and vLLM token ids land on the trace on both arms.
• InterceptionServer mounts every dialect's routes; the SDK's URL picks the codec. Streaming relays pass SSE bytes through chunk-by-chunk and tee-parse the assembled final message for the trace; streaming translate fake-streams a minimal valid SSE. Anthropic's count_tokens is relayed on the relay arm and estimated (~4 chars/token) on translate. Errors come back in each dialect's native error shape.
• Clients keep the typed get_response (judges and in-env calls unaffected) and gain dialect + relay(body, route); RetryingClient retries relays the same way (relay raises before any byte is returned). Renderer and Google stay translate-only (dialect = None) — which is exactly how training keeps typed token data.
• Harnesses now receive the server's root endpoint and append what their SDK expects ({endpoint}/v1 for OpenAI SDKs; bare root for ANTHROPIC_BASE_URL). The default harness carries the eval's sampling into the program (OPENAI_SAMPLING env → extra_body) since the relay no longer injects it.

A claude-code harness now needs only ANTHROPIC_BASE_URL={endpoint} + ANTHROPIC_API_KEY={secret}: ingress (incl. streaming + count_tokens) is already served, relay applies against any anthropic-speaking endpoint (api.anthropic.com, vLLM's native /v1/messages), and translate covers everything else.

Verification

Unit (tests/v1/test_dialects.py, 17 tests + full v1 suite, 63 passed): codec round-trips validated against the provider SDK models (anthropic.types.Message, openai.types.responses.Response), stream assembly/fake-stream round-trips, byte-verbatim relay assertions (unknown request fields survive), per-dialect auth + error shapes, aux-route relay/estimate, refusals in dialect error shape.

Live, through the real eval machinery (default harness × subprocess):

scenario	arm	result
chat program → OpenAI chat endpoint	relay	reward 1.0
chat program → OpenAI Responses client	translate (cross-protocol)	reward 1.0
chat program → Google client	translate (cross-protocol)	reward 1.0
multi-turn user-sim (extend_request over relay)	relay	reward 1.0, 2 turns
agentic bash tool calls	relay	reward 1.0

Live testing caught two real relay bugs, both fixed: newer openai/anthropic SDKs exclude auth from default_headers (now merged from auth_headers explicitly), and duplicate Content-Type headers made OpenAI reject relayed bodies (header keys now lowercased/deduped).

Not live-tested: anthropic/responses ingress (no claude-code/codex harness exists yet — covered by unit tests against the SDK models) and the renderer translate path (needs a vLLM engine).

Design choices reviewers may want to veto

• Bytes mean bytes: the relay arm does not inject model or sampling. The model id already reaches the program via harness env; the default harness now carries sampling itself. On rlm's relay path, eval --sampling.* does not apply until rlm reads an equivalent env (translate/training keeps ctx.sampling pinned).
• User-sim over relay is chat-only (Dialect.extend_request); on other dialects a simulator works via the translate arm. Streamed turns never drive a simulator.
• Anthropic translate emits reasoning as an unsigned thinking block so the program displays it and echoes it back, where parse_request recovers it — the reasoning-passback some models hard-require, carried across protocols through the program itself.
• Responses statefulness is not emulated on translate (previous_response_id ignored; the trace graph could emulate it later); on relay the endpoint owns it.
• count_tokens translate-arm estimate is deliberately crude (compaction-trigger fidelity, not billing).

Breaking

• parse_message / parse_tools / serialize_completion moved from interception.server to verifiers.v1.dialects (chat module).
• Rollout/InterceptionPool hand harnesses the endpoint root (no /v1 suffix); out-of-tree harnesses must append their SDK's path (in-tree default/rlm/compact updated).
• Client gains dialect/relay; custom clients are unaffected unless they want relay.

Follow-up (out of scope)

• claude-code / codex harnesses (env vars only).
• Egress auto-detection sugar (-m anthropic:claude-… scheme + host claims) so direct-API runs don't need --client.type.
• Optional: vf→wire request serializers for non-chat user-sim relay, previous_response_id emulation from the trace graph.

---

Note

High Risk
Core rollout path changes (interception routing, verbatim upstream relay, endpoint root contract) affect every harness model call; mistakes could break auth, streaming, or cross-protocol evals.

Overview
Adds polyglot interception: the server mounts chat, anthropic, and responses routes and picks relay (verbatim request/response bytes when Client.dialect matches) vs translate (parse → typed client → serialize back) per request.

New verifiers/v1/dialects/ holds per-format codecs (parse/serialize, streaming SSE, dialect-shaped errors, Anthropic count_tokens aux). Chat wire helpers move here from interception.server.

Clients gain dialect + relay() via shared relay_post/relay_headers; RetryingClient retries relays. Shared classify_model_error replaces duplicated context-length checks in OpenAI client code.

Harness/rollout expose the interception root (no /v1 suffix); in-tree harnesses set OPENAI_BASE_URL to {endpoint}/v1. Default harness passes eval sampling as OPENAI_SAMPLING merged into program extra_body (relay no longer injects sampling).

Tests: new test_dialects.py for codecs and relay/translate server behavior; test_clients imports from dialects.

^{Reviewed by Cursor Bugbot for commit 302dcee. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add dialect-routed interception server supporting Anthropic, OpenAI Chat, and OpenAI Responses

• Refactors the interception server into a multi-dialect polyglot proxy that registers routes for Anthropic (/v1/messages), OpenAI Chat (/v1/chat/completions), and OpenAI Responses (/v1/responses) concurrently.
• Introduces a Dialect abstraction in verifiers/v1/dialects/ with per-dialect parse/serialize, SSE handling, auth extraction, error shaping, and aux route support.
• Routes requests through two arms: relay (raw byte pass-through when the client's dialect matches the ingress dialect) and translate (typed conversion through the existing middle layer otherwise).
• Adds relay() to Client and all three client implementations, backed by a shared relay_post helper with retry logic, streaming, and error classification via classify_model_error.
• Harnesses now receive root endpoints and must append their SDK-specific path (e.g. /v1); all harness OPENAI_BASE_URL values updated accordingly.
• Risk: Behavioral change — existing callers of InterceptionPool._entry and Rollout._serve_interception now receive root endpoints (no /v1 suffix); harnesses must be updated or requests will 404.

📊 Macroscope summarized 302dcee. 20 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 302dcee. Configure here.}

[cursor]

Compact harness drops eval sampling

Medium Severity

On the relay path the interception server forwards request bodies unchanged, so eval ctx.sampling must reach the program via env (as the default harness does with OPENAI_SAMPLING and extra_body). The compact harness and program were not updated, so temperature, max tokens, and other sampling settings from the eval are omitted from compact harness model calls when the client uses the chat dialect.

 

^{Reviewed by Cursor Bugbot for commit 302dcee. Configure here.}

[cursor]

Missing docs for breaking changes

Medium Severity

This PR changes core v1 interception and harness contracts (endpoint root vs /v1, relay vs translate, moved parse_message / serialize_completion, new Client.dialect / relay), but the diff includes no updates under docs/ or affected skills/ for evaluation and harness setup.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 302dcee. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 302dceeff8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve sampling for compact harness requests

When this harness points the OpenAI SDK at the new /v1 interception endpoint, a matching chat client now takes the relay path, so InterceptionServer no longer injects ctx.sampling into the model call. The default harness compensates by passing OPENAI_SAMPLING, but compact only sets base URL/key/model and its program calls chat.completions.create without any sampling arguments; compact evals that configure max_tokens, temperature, top_p, etc. will silently run with provider defaults instead of the eval config.

Useful? React with 👍 / 👎.

[macroscopeapp]

🟡 Medium dialects/responses.py:126

When an input item with role "system", "developer", or user-role lacks a "content" key, item.get("content") returns None and passes it to parse_content(). Since isinstance(None, str) is false, the code falls through to for part in content:, which raises TypeError: 'NoneType' object is not iterable. Consider adding a default empty string or list to item.get("content") before calling parse_content(), or handle None inside parse_content().

-            elif item.get("role") in ("system", "developer"):
-                prompt.append(SystemMessage(content=parse_content(item.get("content"))))
+            elif item.get("role") in ("system", "developer"):
+                prompt.append(SystemMessage(content=parse_content(item.get("content") or "")))
             else:
-                prompt.append(UserMessage(content=parse_content(item.get("content"))))
+                prompt.append(UserMessage(content=parse_content(item.get("content") or "")))

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/dialects/responses.py around lines 126-129:

When an input item with `role` `"system"`, `"developer"`, or user-role lacks a `"content"` key, `item.get("content")` returns `None` and passes it to `parse_content()`. Since `isinstance(None, str)` is false, the code falls through to `for part in content:`, which raises `TypeError: 'NoneType' object is not iterable`. Consider adding a default empty string or list to `item.get("content")` before calling `parse_content()`, or handle `None` inside `parse_content()`.

Evidence trail:
verifiers/v1/dialects/responses.py lines 36-55 (parse_content definition, no None guard), lines 102-128 (caller passes item.get('content') which can be None)

[macroscopeapp]

🟢 Low interception/server.py:185

handle_aux calls json.loads(raw) on line 202 without exception handling. If the request body is empty or malformed, this raises json.JSONDecodeError and propagates as a 500 Internal Server Error instead of 400 Bad Request. This leaves the non-relay path unprotected while the relay path (lines 196-201) already catches client errors, creating inconsistent error handling.

-        return web.json_response(dialect.handle_aux(route, json.loads(raw)))
+        try:
+            body = json.loads(raw)
+        except json.JSONDecodeError as e:
+            return web.json_response(dialect.error_body(str(e)), status=400)
+        return web.json_response(dialect.handle_aux(route, body))

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/interception/server.py around lines 185-207:

`handle_aux` calls `json.loads(raw)` on line 202 without exception handling. If the request body is empty or malformed, this raises `json.JSONDecodeError` and propagates as a 500 Internal Server Error instead of 400 Bad Request. This leaves the non-relay path unprotected while the relay path (lines 196-201) already catches client errors, creating inconsistent error handling.

Evidence trail:
verifiers/v1/interception/server.py lines 185-202 (REVIEWED_COMMIT): `handle_aux` method, line 202 has `json.loads(raw)` without try/except. Lines 195-201 show the relay path with try/except around `session.ctx.client.relay(raw, route)`. Line 163: `web.Application(client_max_size=_MAX_REQUEST_BODY)` with no middleware parameter. git_grep for 'middleware|error_middleware' in the interception package returned no results, confirming no error-handling middleware exists.

[macroscopeapp]

Approvability

Verdict: Needs human review

1 blocking correctness issue found. This PR introduces a significant new feature (dialect-routed interception) with ~1000+ lines of new code, breaking changes to endpoint contracts, and major runtime behavior changes to the interception server. Multiple unresolved review comments identify functional bugs including compact harness dropping eval sampling on the relay path.

^{You can customize Macroscope's approvability policy. Learn more.}