diff --git a/CONTEXT.md b/CONTEXT.md index 0a294c6..c31b991 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -20,6 +20,22 @@ _Avoid_: Trajectory, session export A caller-facing Run Output that exposes a stable, selected sequence of Agent Run events as they occur. _Avoid_: Internal event bus, execution trace +**Source Record**: +An agent-specific record observed at a CLI, SDK, or protocol boundary and consumed by an adapter; it may already be a projection of the agent's internal state. +_Avoid_: Native event, journal entry + +**Native Event**: +An agent-independent runtime fact accepted by the core directly or normalized from a Source Record. +_Avoid_: Source record, public event record, trajectory step + +**Journal Entry**: +A Native Event together with the identity, ordering, and recording metadata required for durable history. +_Avoid_: Persistence envelope, trajectory step + +**Event Journal**: +A durable append-only sequence of Journal Entries for an Agent Run from which derived artifacts can be rebuilt. +_Avoid_: Trajectory, session, public event stream + **Partial Content**: An unfinished representation of a message, reasoning block, or tool input that is exposed before the logical content is complete. _Avoid_: Final message, completed event @@ -40,6 +56,10 @@ _Avoid_: Run output, trajectory A task-scoped record of observations, actions, results, and outcome prepared for evaluation or offline analysis. _Avoid_: Execution trace, session, transcript +**ATIF Trajectory**: +A Trajectory represented by the Agent Trajectory Interchange Format and exposed as the runtime's public trajectory contract. +_Avoid_: Event journal, native trajectory + **Session**: Durable agent state that can span multiple runs and supports continuing or branching prior work. _Avoid_: Agent run, trajectory, transcript diff --git a/docs/dev_notes/en/0.8.x.md b/docs/dev_notes/en/0.8.x.md index a9ba095..3d26101 100644 --- a/docs/dev_notes/en/0.8.x.md +++ b/docs/dev_notes/en/0.8.x.md @@ -105,7 +105,7 @@ The next work therefore has three independent parts: - Add `httpx` as a direct dependency so container installation does not rely on a transitive relationship. - Add the Harbor adapter to the project, fixing and versioning its installation command, headless CLI invocation, environment-variable forwarding, and version probing so future benchmark runs can reuse it instead of rewriting a temporary adapter each time. -- Design canonical internal events, `stream-json`, and a trajectory writer so the adapter can convert the native trajectory to ATIF and reliably populate token, cost, and step statistics. These changes improve the installability, reproducibility, and observability of the working benchmark path. +- Design canonical internal events and a trajectory writer so the agent produces ATIF directly and the adapter can read it to populate token, cost, and step statistics reliably; implement `stream-json` later as separate run output. These changes improve the installability, reproducibility, and observability of the working benchmark path. ### Adding the Harbor adapter to the project @@ -119,4 +119,56 @@ The configuration boundary is fixed as well. `ANTHROPIC_API_KEY`, `ANTHROPIC_BAS The formal adapter's container boundary is first covered by five contract tests built on Harbor 0.21.0's real base class: published-release pins, Git revision pins, mutually exclusive source arguments, instruction piping and logging, and both environment-variable mappings. The same task was then rerun once through the formal adapter in a real Docker environment. Harbor completed the full path — starting the container, installing the specified Git revision, injecting model configuration, invoking the agent, collecting logs, and running the official verifier — with zero agent or verifier infrastructure exceptions, demonstrating that the adapter itself works end to end. The task received a reward of 0, with 5 of 6 official verifier tests passing. The only failure was that the agent's `check_cert.py` depended on `cryptography`, which was unavailable in the verifier's Python environment. This was a portability defect in the task solution, not an adapter infrastructure failure. -In addition, the Harbor CLI version command and dynamic import in the isolated workspace verify the host-side integration, while a build of the root project confirms that its wheel contains neither the adapter nor a Harbor dependency. The first two follow-up items — owning `httpx` directly and adding the formal adapter — are therefore complete. The remaining work is to design canonical internal events, `stream-json`, and a trajectory writer so that the adapter can convert its native trajectory to ATIF and reliably populate token, cost, and step statistics. +In addition, the Harbor CLI version command and dynamic import in the isolated workspace verify the host-side integration, while a build of the root project confirms that its wheel contains neither the adapter nor a Harbor dependency. The first two follow-up items — owning `httpx` directly and adding the formal adapter — are therefore complete. The remaining trajectory work is to design canonical internal events and an Event Journal, have the agent produce ATIF directly, and let the adapter read it to populate token, cost, and step statistics reliably. `stream-json` is a separate form of run output and will be implemented in a later PR. + +### Implementing Trajectory + +Based on the research ([the boundary between agent output and trajectory](../../research/en/agent_output_and_trajectory.md), [mapping agent events to ATIF](../../research/en/agent_events_to_atif_examples.md), [OpenRouter cost accounting](../../research/en/openrouter_cost_accounting.md), and [the unified OpenRouter model protocol](../../research/en/openrouter_unified_protocol.md)), the implementation path has converged on a replayable internal **Event Journal**, while the public `--trajectory` option produces only **ATIF-v1.7**. `stream-json` remains a separate form of run output that shares the same runtime facts with trajectory; it and the corresponding `--output-format` CLI interface are outside this trajectory implementation and will be delivered in a later PR. + +One completed `read` tool execution illustrates the distinction between a `Native Event` and a `Journal Entry`. + +The agent loop first produces a Native Event that describes only what happened: + +```json +{ + "type": "tool.completed", + "payload": { + "tool_call_id": "call-read-1", + "tool_name": "read", + "result": "file contents", + "is_error": false, + "duration_ms": 330 + } +} +``` + +After receiving it, the journal writer adds the identity, order, and record time needed for persistence, producing a Journal Entry: + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 17, + "recorded_at": "2026-08-23T08:00:01.420Z", + "type": "tool.completed", + "payload": { + "tool_call_id": "call-read-1", + "tool_name": "read", + "result": "file contents", + "is_error": false, + "duration_ms": 330 + } +} +``` + +These describe the same occurrence rather than two events. A `Native Event` is the runtime fact produced by core; a `Journal Entry` is the persistable record of that fact after it enters the Event Journal, additionally answering which run it belongs to, where it appears in the sequence, and when it was recorded. The ATIF projector consumes Journal Entries ordered by `seq` and folds multiple facts into trajectory steps. + +The implementation order is: + +1. **Define runtime facts.** Establish a versioned contract for Native Events and Journal Entries. The first Native Event version covers `run.started`, `user.message`, `model.started/completed`, `tool.started/completed`, and `run.completed/failed`, carrying message and tool-call IDs, reliable `source_timestamp` values, and precise durations. Journal Entries then uniformly add `schema_version`, `run_id`, a strictly increasing `seq`, and UTC `recorded_at`. +2. **Make the agent loop emit facts only.** Emit events around model calls, tool execution, and run finalization. `model.completed` retains the complete response, actual model, stop reason, token and cache usage, provider response ID, and OpenRouter generation ID. Project the existing text output from those facts while preserving current user-visible behavior, leaving one shared data source for a future `stream-json` implementation. +3. **Append to the internal Event Journal.** The journal writer wraps every event in a Journal Entry and appends it incrementally as JSONL under restricted file permissions, preserving the last complete fact even if the process is interrupted. It also records truncation metadata for large output and treats the log explicitly as sensitive. The journal is an internal reconstruction source, is not exposed through `--trajectory`, and does not define ATIF steps. +4. **Complete usage and actual cost.** Record `usage.cost` directly when present. On the current OpenRouter Messages path, preserve `X-Generation-Id` and use bounded retries during run finalization to query the generation's `total_cost`, then append `model.cost_resolved`. Query failure does not change the task result; unknown cost is not written as `0`, and summaries explicitly indicate partial data. +5. **Implement a one-way ATIF projector.** Replay the Event Journal, mapping `user.message` to a user step and folding one model call plus its tool calls and results into an agent step. Map timestamps, tokens, cache, cost, terminal state, and `final_metrics`; only values with reliable sources and complete attribution enter standard fields, with other information under `extra`. At each checkpoint or run end, update the complete ATIF snapshot through a temporary file and atomic rename. +6. **Connect the CLI and Harbor.** Add an independent `--trajectory PATH`, preserve stdout's current text behavior, and reject `-` to avoid competing for stdout. The Harbor adapter only passes the log path, declares and reads ATIF, and then populates steps, tokens, and cost; it no longer converts a native trajectory. `--output-format` and `stream-json` remain for a later independent PR. +7. **Validate in layers.** Start with unit tests for event order, ID associations, failed tools, pending and resolved cost, Journal replay, and the ATIF schema. Then test stdout independence, atomic CLI writes, and interruption recovery. Finally, use Harbor contract tests and one real trial to confirm that the trajectory is collected and that step, token, and cost statistics are populated. diff --git a/docs/dev_notes/zh-CN/0.8.x.md b/docs/dev_notes/zh-CN/0.8.x.md index 3641f9f..f3bcab4 100644 --- a/docs/dev_notes/zh-CN/0.8.x.md +++ b/docs/dev_notes/zh-CN/0.8.x.md @@ -105,7 +105,7 @@ harbor run \ - 把 `httpx` 补成直接依赖，保证容器安装不依赖传递关系 - 把 Harbor adapter 正式纳入项目，固定并版本化安装命令、headless CLI 调用、环境变量传递和版本探测逻辑，使后续 benchmark 可以直接复用，不必每次重新编写临时适配器 -- 设计内部标准事件、`stream-json` 和 trajectory writer，让这个 adapter 可以把 native trajectory 转成 ATIF，并可靠回填 token、cost 和步骤统计。这些属于对已跑通链路的可安装性、可复现性与可观测性增强。 +- 设计内部标准事件和 trajectory writer，让 agent 直接产出 ATIF，再由 adapter 读取并可靠回填 token、cost 和步骤统计；`stream-json` 作为独立的 run output 后续实现。这些属于对已跑通链路的可安装性、可复现性与可观测性增强。 ### Harbor adapter 正式纳入项目 @@ -119,4 +119,56 @@ harbor run \ 正式 adapter 的容器边界先由五个基于 Harbor 0.21.0 真实基类的契约测试覆盖，分别验证发布版 pin、Git revision pin、互斥参数、instruction 管道与日志、两种环境变量映射；随后又通过正式 adapter 在真实 Docker 环境中重跑了一次同一道题。Harbor 完成了启动容器、安装指定 Git revision、注入模型配置、调用 agent、收集日志和执行官方 verifier 的完整链路，agent / verifier 基础设施异常为 0，说明 adapter 本身已经端到端接通。这次任务 reward 为 0，官方 verifier 通过 5/6 项；唯一失败是 agent 编写的 `check_cert.py` 依赖 `cryptography`，而 verifier 的 Python 环境中没有该依赖。这属于任务解答的可移植性问题，不是 adapter 基础设施故障。 -此外，独立 workspace 中 Harbor CLI 的版本命令和动态导入检查了宿主侧集成，根项目 wheel 的构建则确认其中不包含 adapter 或 Harbor 依赖。原先三件后续工作中的前两件——`httpx` 直接依赖与正式 adapter——至此完成。剩余工作是设计内部标准事件、`stream-json` 和 trajectory writer，让 adapter 能把 native trajectory 转成 ATIF，并可靠回填 token、cost 和步骤统计。 +此外，独立 workspace 中 Harbor CLI 的版本命令和动态导入检查了宿主侧集成，根项目 wheel 的构建则确认其中不包含 adapter 或 Harbor 依赖。原先三件后续工作中的前两件——`httpx` 直接依赖与正式 adapter——至此完成。剩余的 trajectory 工作是设计内部标准事件与 Event Journal，由 agent 直接产出 ATIF，再让 adapter 读取并可靠回填 token、cost 和步骤统计。`stream-json` 属于独立的 run output，将在后续 PR 中实现。 + +### 实现 Trajectory + +根据调研（[agent output 与 trajectory 边界](../../research/zh-CN/agent_output_and_trajectory.md)、[agent 事件到 ATIF 的映射](../../research/zh-CN/agent_events_to_atif_examples.md)、[OpenRouter cost 记账](../../research/zh-CN/openrouter_cost_accounting.md)、[OpenRouter 统一模型协议](../../research/zh-CN/openrouter_unified_protocol.md)），实现路线收敛为：内部保留可重放的 **Event Journal**，对外的 `--trajectory` 只产出 **ATIF-v1.7**。`stream-json` 仍是独立的 run output，只与 trajectory 复用同一组运行事实；它和对应的 `--output-format` CLI 接口不在本轮 trajectory 实现范围内，后续单开 PR。 + +先用一次 `read` 工具执行完成来说明 `Native Event` 与 `Journal Entry` 的概念。 + +agent loop 先产生一条只描述“发生了什么”的 `Native Event`： + +```json +{ + "type": "tool.completed", + "payload": { + "tool_call_id": "call-read-1", + "tool_name": "read", + "result": "file contents", + "is_error": false, + "duration_ms": 330 + } +} +``` + +journal writer 接收它以后，补上持久化所需的身份、顺序和记录时间，形成一条 `Journal Entry`： + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 17, + "recorded_at": "2026-08-23T08:00:01.420Z", + "type": "tool.completed", + "payload": { + "tool_call_id": "call-read-1", + "tool_name": "read", + "result": "file contents", + "is_error": false, + "duration_ms": 330 + } +} +``` + +两者描述的是同一件事，不是两个事件：`Native Event` 是 core 产生的运行事实；`Journal Entry` 是这条事实进入 Event Journal 后的可持久化记录，额外回答“属于哪次 run、排在第几、何时被记录”。ATIF projector 消费的是按 `seq` 排列的 Journal Entry，再把多条事实折叠成 trajectory step。 + +整体按以下顺序实现： + +1. **定义运行事实。** 建立 `Native Event` 与 `Journal Entry` 的版本化契约。Native Event 第一版覆盖 `run.started`、`user.message`、`model.started/completed`、`tool.started/completed`、`run.completed/failed`，并携带 message/tool call ID、可信的 `source_timestamp` 和精确 duration；Journal Entry 再统一添加 `schema_version`、`run_id`、严格递增的 `seq` 和 UTC `recorded_at`。 +2. **让 agent loop 只产生事实。** 在模型调用、工具执行和 run 收尾处发出事件；`model.completed` 保留完整回复、实际 model、stop reason、token/cache usage、provider response ID 和 OpenRouter generation ID。现有文本输出改为这些事实的投影，保持当前用户可见行为不变，也为后续 `stream-json` 留出同一数据源。 +3. **追加写入内部 Event Journal。** journal writer 把每个事件包装成 Journal Entry，以受限文件权限和 JSONL 增量落盘，使进程中断后仍保留最后一条完整事实；同时为大输出保留截断元数据，并把日志明确视为敏感文件。它是内部重建来源，不由 `--trajectory` 暴露，也不定义 ATIF step。 +4. **补齐 usage 与真实 cost。** 有 `usage.cost` 时直接记录；当前 OpenRouter Messages 路径则保存 `X-Generation-Id`，在 run 收尾时用有界重试查询 generation 的 `total_cost`，再追加 `model.cost_resolved`。查询失败不改变任务结果，未知 cost 不写成 `0`，汇总时明确标记 partial。 +5. **实现单向 ATIF projector。** 重放 Event Journal，将 `user.message` 映射为 user step，将一次模型调用及其 tool calls/results 折叠成 agent step，并映射 timestamp、token、cache、cost、终态和 `final_metrics`；只有来源可靠、归因完整的值才进入标准字段，其余信息放入 `extra`。每次 checkpoint 或 run 结束时通过临时文件加原子 rename 更新完整 ATIF 快照。 +6. **接通 CLI 与 Harbor。** 增加独立的 `--trajectory PATH`，保持 stdout 当前的文本行为，并拒绝 `-` 与 stdout 争用；Harbor adapter 只负责传入日志路径、声明并读取 ATIF，再回填 steps、tokens 和 cost，不再做 native trajectory 转换。`--output-format` 与 `stream-json` 留给后续独立 PR。 +7. **分层验收。** 先用单元测试覆盖事件顺序、ID 关联、异常工具、pending/resolved cost、Journal 重放和 ATIF schema；再验证 CLI 的 stdout 独立性、原子写入和中断恢复；最后用 Harbor 契约测试与一次真实 trial 确认 trajectory 被采集，步骤与 token/cost 统计能够回填。 diff --git a/docs/research/en/agent_events_to_atif_examples.md b/docs/research/en/agent_events_to_atif_examples.md new file mode 100644 index 0000000..452e443 --- /dev/null +++ b/docs/research/en/agent_events_to_atif_examples.md @@ -0,0 +1,728 @@ +# How Source Records, Internal Events, and ATIF Correspond Across Pi, Claude Code, Codex, OpenCode, and Grok + +> Generated from the Chinese source [`../zh-CN/agent_events_to_atif_examples.md`](../zh-CN/agent_events_to_atif_examples.md). Do not edit by hand. + +Surveyed on 2026-08-23. + +## Question + +> Can you show realistic examples of what agent-internal events look like in Pi, Codex, Claude Code, OpenCode, and Grok, and what the corresponding ATIF looks like? +> +> Why do these native-event examples have no timestamps? + +## Conclusions first + +Yes, but both “real” and “internal” need precise boundaries. This document uses two evidence layers. The earlier “Local runtime evidence” section preserves IDs, token counts, and timestamps from four actual Pi, Codex, Claude Code, and OpenCode stdout runs. Those records are **Source Records** visible at CLI or SDK boundaries and must not all be treated as raw events from an agent core. The five later comparative examples are **schema-faithful reconstructions** based on real type definitions, converters, and tests; their task content, IDs, token counts, and timestamps are illustrative values chosen for comparison. Grok was not installed locally, so it has only a source-based reconstruction. + +All five projects follow the same broad pattern: **the agent loop first produces its own messages, events, or state; a CLI or SDK projects them into Source Records; ATIF is an evaluation view assembled afterward, not the object model in which the loop itself runs.** The main differences are: + +1. Whether Source Records are organized around model messages, tool states, or CLI items. +2. Whether token and cost data is available for every model call or only aggregated at the end of a run or turn. +3. Whether source time is present on the Source Record, nested in its payload, or available only as receiver-captured time. + +## Concepts and conversion relationships + +This document uses the terminology in the repository-root [`CONTEXT.md`](../../../CONTEXT.md): + +| Concept | Definition | Must not be conflated with | +|---|---|---| +| **Source Record** | An agent-specific record that an adapter actually observes at a third-party CLI, SDK, or protocol boundary; it may already be a reduced projection of internal state | nano Native Event, Journal Entry | +| **Native Event** | An agent-independent runtime fact produced directly by nano core or normalized by an adapter from a Source Record | A raw third-party stdout line, ATIF step | +| **Journal Entry** | A Native Event plus identity, ordering, and recording-time metadata required for persistence | A separate “persistence envelope” artifact, trajectory step | +| **Event Journal** | The internal fact log formed by one Agent Run's Journal Entries in append order | Public Event Stream, Trajectory, Session | +| **ATIF Trajectory** | The public completed evaluation and analysis view projected from the Event Journal | Event Journal, native JSONL | + +For a third-party agent, the complete boundary is: + +```text +Third-party internal state/events + ↓ third-party CLI/SDK projector +Source Record + ↓ nano adapter normalization +Native Event + ↓ journal writer adds run_id / seq / recorded_at +Journal Entry + ↓ append-only persistence (JSONL is one possible encoding) +Event Journal + ↓ ATIF projector +ATIF Trajectory +``` + +nano's own core can produce Native Events directly, skipping the first two steps. JSONL here is only a physical encoding with one Journal Entry per line, and append-only is only the write policy; neither defines a new trajectory semantic model. The earlier phrase “persistence envelope” is likewise not a separate domain concept and is standardized here as **Journal Entry**. + +The observed stdout maps to these concepts as follows: + +| Agent | Source Record captured here | Distance from internal events | Journal Entry? | +|---|---|---|---| +| Pi | Session, message, and tool lines from `--mode json` | Most event lines are direct public serializations of `AgentEvent`; the session header is a CLI protocol record | No; a separate, unshown persistent harness entry is closer to a Journal Entry | +| Claude Code | System, assistant, user, and result messages from SDK `stream-json` | Already a public SDK protocol; it cannot be assumed identical to the internal event bus | No; session persistence was also disabled for this run | +| Codex | `ThreadEvent` from `exec --json` | A reduced projection of a lower-level protocol | No; the local rollout JSONL is another persistence artifact and is not the stdout source used here | +| OpenCode | Step, tool, and text lines from `run --format json` | Projected again from internal `message.part.updated` events | No; the CLI's top-level timestamp is not journal ordering metadata | +| Grok | `AcpLine` from `streaming-json` | Reduced from ACP updates | No; it has NDJSON line order but no Journal Entry envelope | + +Timestamps in simplified examples must not be used to infer the source protocol. The actual situation is: + +| Agent | Uniform time on Source Records | Other available time | Recommended ATIF time source | +|---|---|---|---| +| Pi | `AgentEvent` has none | The session header and `AgentMessage.timestamp` have time; a persistent harness entry has a separate storage timestamp | Prefer message time; let the collector add receive time for tool events | +| Claude Code | No uniform time on the SDK stream | In local 2.1.237, `assistant` and `user` messages have ISO timestamps; `system` and `result` do not, while `result` has duration | Use message time when available, otherwise receive time | +| Codex `exec --json` | `ThreadEvent` has none | Some lower-level protocol objects have start/end times, but public `ThreadEvent` omits them | Add receive time in the JSONL receiver | +| OpenCode | Yes | `message.part.updated.properties.time`; message and tool state also have create/start/end time | Use native millisecond time directly | +| Grok `streaming-json` | None | The terminal event has usage; the internal persistence layer has other sequencing data, but the public reducer omits it | Add receive time in the JSONL receiver | + +nanoPyCodeAgent Journal Entries should therefore not reproduce gaps in any one public stream. Every Journal Entry should have at least a strictly increasing `seq` and UTC `recorded_at`: the former is authoritative ordering within a run, and the latter is when nano accepted or recorded the fact. If a Source Record contains reliable time, preserve it on the Native Event as `source_timestamp`. Wall clocks can move backward or assign identical times to concurrent events, so neither kind of timestamp can be the only ordering mechanism. + +## Research scope + +This document uses fixed snapshots under the repository's `references/` directory: + +- Pi: `c49906ec77788625aacbdc53ebca6fbe65bd20f5` +- Codex: `4f39251a010a8bd7d692d25fb33832ff06f1635a` +- Unofficial Claude Code source snapshot: `a371abbe75ffa0d0a3c92290e2bbf56a7ef54367` +- OpenCode: `e00890c67261a435cee6409366a68999a93393fd` +- Grok Build: `19d42e35c07a9c9244f03f6df0c4c353f970d4f9` +- Harbor: 0.21.0 as pinned by [`benchmarks/harbor/pyproject.toml`](../../../benchmarks/harbor/pyproject.toml), whose trajectory schema is ATIF-v1.7 + +### Local runtime evidence + +In addition to source research, this document records actual runs of four locally installed CLIs on 2026-08-23: + +| CLI | Version | Result | +|---|---:|---| +| Pi | 0.84.2 | Success; `openai-codex/gpt-5.6-sol`, two model responses, and one `read` | +| Codex | 0.149.0 | Success; two command-item states and one agent message | +| Claude Code | 2.1.237 | Success; two Claude Opus messages, one `Read`, plus auxiliary Haiku usage | +| OpenCode | 1.18.21 | Success; two steps and one `read` | + +To avoid sending repository contents to external models, all four tests ran in a temporary directory under `/tmp` and read only a purpose-built, non-sensitive file. The following preparation creates a new random directory; `$probe_dir` in later commands refers to it: + +```bash +probe_dir="$(mktemp -d /tmp/nanopy-agent-event-probe.XXXXXX)" +printf 'NANOPY_EVENT_PROBE_20260823\n' > "$probe_dir/probe.txt" +cd "$probe_dir" +``` + +The successful captures used these commands. A rerun will naturally produce new session and message IDs, token counts, and timestamps. + +Pi: + +```bash +pi --mode json --print --no-session \ + --tools read \ + --no-extensions \ + --no-skills \ + --no-prompt-templates \ + --no-themes \ + --no-context-files \ + --approve \ + "Use the read tool exactly once to read probe.txt. Do not write or modify anything. Then reply with only the exact file content." +``` + +Codex: + +```bash +codex exec --json \ + --sandbox read-only \ + --ephemeral \ + --skip-git-repo-check \ + --ignore-user-config \ + "Read probe.txt using a shell command exactly once. Do not write or modify anything. Then reply with only the exact file content." +``` + +Claude Code: + +```bash +claude -p \ + --output-format stream-json \ + --verbose \ + --no-session-persistence \ + --safe-mode \ + --tools Read \ + --permission-mode dontAsk \ + "Use the Read tool exactly once to read probe.txt. Do not write or modify anything. Then reply with only the exact file content." +``` + +OpenCode; its local installation path is added by `.zshrc`, so the observed procedure loads it first: + +```bash +source ~/.zshrc +opencode run \ + --format json \ + --pure \ + --auto \ + --dir "$probe_dir" \ + "Use the read tool exactly once to read probe.txt. Do not write or modify anything. Then reply with only the exact file content." +``` + +The following records extract the relevant parts of actual stdout. Each excerpt selects only events or fields needed for structural comparison: complete Pi and Claude Code lines contain large encrypted-reasoning, initialization, and plugin-list fields, while OpenCode tool output and metadata are also reduced. Event names, IDs, numeric values, and times all come from the actual runs. These are **key-field projections of real stdout**, not untouched line-by-line captures. + +#### Pi 0.84.2 + +```jsonl +{"type":"session","version":3,"id":"01a02e3c-e606-7060-baf2-9a1ded866776","timestamp":"2026-08-23T10:48:58.118Z","cwd":"/tmp/nanopy-agent-event-probe.ktOr9V"} +{"type":"message_end","message":{"role":"assistant","model":"gpt-5.6-sol","provider":"openai-codex","responseId":"resp_0f6a0eea8b7401c0016a8ad01c304087d0a882c3eee1686e09","usage":{"input":597,"output":30,"reasoning":10,"totalTokens":627,"cost":{"total":0.0038850000000000004}},"stopReason":"toolUse","timestamp":1787482138161}} +{"type":"tool_execution_start","toolCallId":"call_j2gAtuUWhalLvXqYLKLke6Yk|fc_0f6a0eea8b7401c0016a8ad01e705c87d0a0c8b9b64c05762e","toolName":"read","args":{"path":"probe.txt"}} +{"type":"tool_execution_end","toolCallId":"call_j2gAtuUWhalLvXqYLKLke6Yk|fc_0f6a0eea8b7401c0016a8ad01e705c87d0a0c8b9b64c05762e","toolName":"read","result":{"content":[{"type":"text","text":"NANOPY_EVENT_PROBE_20260823\n"}]},"isError":false} +{"type":"message_end","message":{"role":"assistant","model":"gpt-5.6-sol","provider":"openai-codex","responseId":"resp_0f6a0eea8b7401c0016a8ad01f3c5c87d0910ad017f42f5588","content":[{"type":"text","text":"NANOPY_EVENT_PROBE_20260823"}],"usage":{"input":648,"output":14,"reasoning":0,"totalTokens":662,"cost":{"total":0.00366}},"stopReason":"stop","timestamp":1787482143410}} +``` + +This directly verifies that tool-lifecycle events have no timestamp, while assistant and tool-result payloads do; per-response usage is rich. Pi's source [`calculateCost`](../../../references/pi/packages/ai/src/models.ts) multiplies tokens by model-catalog rates, so `usage.cost` is agent-calculated and must not automatically be treated as a provider charge. + +#### Codex 0.149.0 + +```jsonl +{"type":"thread.started","thread_id":"01a02e3d-3743-7ac1-8c30-6c7254135573"} +{"type":"turn.started"} +{"type":"item.started","item":{"id":"item_0","type":"command_execution","command":"/usr/bin/zsh -lc 'cat probe.txt'","aggregated_output":"","exit_code":null,"status":"in_progress"}} +{"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"/usr/bin/zsh -lc 'cat probe.txt'","aggregated_output":"NANOPY_EVENT_PROBE_20260823\n","exit_code":0,"status":"completed"}} +{"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"NANOPY_EVENT_PROBE_20260823"}} +{"type":"turn.completed","usage":{"input_tokens":31003,"cached_input_tokens":19968,"cache_write_input_tokens":0,"output_tokens":135,"reasoning_output_tokens":60}} +``` + +This output confirms two gaps completely: it has no timestamps and no cost, while usage appears only as a `turn.completed` aggregate. + +#### Claude Code 2.1.237 + +```jsonl +{"type":"system","subtype":"api_retry","attempt":1,"max_retries":10,"retry_delay_ms":524,"error_status":401,"error":"authentication_failed","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"assistant","message":{"model":"claude-opus-5","id":"msg_011CeKYuCnCMFvi24JSsJLvS","content":[{"type":"tool_use","id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","name":"Read","input":{"file_path":"/tmp/nanopy-agent-event-probe.ktOr9V/probe.txt"}}],"usage":{"input_tokens":2,"cache_creation_input_tokens":2521,"cache_read_input_tokens":1247,"output_tokens":14}},"timestamp":"2026-08-23T10:50:01.254Z","request_id":"req_011CeKYuC24Db1xnVkk2y6Ai","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","type":"tool_result","content":"1\tNANOPY_EVENT_PROBE_20260823\n2\t"}]},"timestamp":"2026-08-23T10:50:01.307Z","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"assistant","message":{"model":"claude-opus-5","id":"msg_011CeKYuMzLUNjC8x7bv2GJD","content":[{"type":"text","text":"NANOPY_EVENT_PROBE_20260823"}],"usage":{"input_tokens":2,"cache_creation_input_tokens":130,"cache_read_input_tokens":3768,"output_tokens":1}},"timestamp":"2026-08-23T10:50:02.486Z","request_id":"req_011CeKYuMBxrDUiGKVmhMFzJ","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"result","subtype":"success","num_turns":2,"total_cost_usd":0.032467499999999996,"modelUsage":{"claude-haiku-4-5-20251001":{"inputTokens":920,"outputTokens":12,"costUSD":0.00098},"claude-opus-5[1m]":{"inputTokens":4,"outputTokens":98,"cacheReadInputTokens":5015,"cacheCreationInputTokens":2651,"costUSD":0.031487499999999995}},"result":"NANOPY_EVENT_PROBE_20260823","duration_ms":7920,"session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +``` + +This corrects a difference between the fixed source snapshot and the newer local version: assistant and user messages in 2.1.237 now carry timestamps and request IDs, but `system.api_retry` and `result` still have no uniform timestamp. `result.total_cost_usd` also includes a Haiku auxiliary call beyond the visible Opus turns, so run total cannot be reconstructed by summing only visible assistant messages. + +#### OpenCode 1.18.21 + +```jsonl +{"type":"step_start","timestamp":1787482258116,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"messageID":"msg_02e3eaf250017z8BF8zCT2CQ3f","type":"step-start"}} +{"type":"tool_use","timestamp":1787482259124,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"type":"tool","tool":"read","callID":"call_Kap3tANM5Uu9U5raX0ogDuYu","state":{"status":"completed","input":{"filePath":"/tmp/nanopy-agent-event-probe.ktOr9V/probe.txt"},"output":"NANOPY_EVENT_PROBE_20260823","time":{"start":1787482259101,"end":1787482259121}},"messageID":"msg_02e3eaf250017z8BF8zCT2CQ3f"}} +{"type":"step_finish","timestamp":1787482259168,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"reason":"tool-calls","messageID":"msg_02e3eaf250017z8BF8zCT2CQ3f","type":"step-finish","tokens":{"total":8011,"input":7952,"output":35,"reasoning":24,"cache":{"write":0,"read":0}},"cost":0}} +{"type":"step_start","timestamp":1787482261701,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"messageID":"msg_02e3ebedc001joGNk0E3u0NSp7","type":"step-start"}} +{"type":"text","timestamp":1787482261701,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"messageID":"msg_02e3ebedc001joGNk0E3u0NSp7","type":"text","text":"NANOPY_EVENT_PROBE_20260823","time":{"start":1787482261684,"end":1787482261691}}} +{"type":"step_finish","timestamp":1787482261701,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"reason":"stop","messageID":"msg_02e3ebedc001joGNk0E3u0NSp7","type":"step-finish","tokens":{"total":8113,"input":397,"output":16,"reasoning":20,"cache":{"write":0,"read":7680}},"cost":0}} +``` + +The OpenCode CLI does not print the internal SDK `message.part.updated` unchanged. [`run.ts`](../../../references/opencode/packages/opencode/src/cli/cmd/run.ts) projects it into the `step_start/tool_use/step_finish/text` records above and adds a top-level timestamp to every line with `Date.now()`. The observed `cost: 0` is only the agent-reported value OpenCode produced for the configured model; without a provider billing source, it cannot prove that the call was actually free. + +### Projecting the observed events into ATIF + +The four observed runs use these mapping strategies: + +| CLI | ATIF step boundary | Timestamp | Step metrics | Run cost | +|---|---|---|---|---| +| Pi | Every completed assistant message | Message Unix milliseconds | Per-response tokens; locally calculated cost goes under `metrics.extra.estimated_cost_usd` | Not treated as actual `total_cost_usd` | +| Codex | Command-item step plus final-message step | Receiver-added | Turn aggregate belongs only in `final_metrics` | Missing | +| Claude Code | Every assistant `message.id` | Assistant timestamp | Visible message usage; prompt tokens must sum input + cache create + cache read | `result.total_cost_usd` can be an agent-reported run cost, annotated as including auxiliary calls | +| OpenCode | Every `messageID`, closed by `step_finish` | CLI top-level timestamp | Per-step tokens; cache is a prompt subset | Mark cost agent-reported; do not present it as provider-reported | + +For the observed Claude Code run, the key ATIF fields cannot simply copy `input_tokens`. ATIF defines `prompt_tokens` to include both cached and non-cached tokens, so the first visible Opus step is `2 + 2521 + 1247 = 3770`, and the second is `2 + 130 + 3768 = 3900`: + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T10:50:01.254Z", + "source": "agent", + "model_name": "claude-opus-5", + "message": "", + "tool_calls": [{"tool_call_id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","function_name":"Read","arguments":{"file_path":"/tmp/nanopy-agent-event-probe.ktOr9V/probe.txt"}}], + "observation": {"results":[{"source_call_id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","content":"NANOPY_EVENT_PROBE_20260823"}]}, + "metrics": {"prompt_tokens":3770,"completion_tokens":14,"cached_tokens":1247}, + "llm_call_count": 1 + }, + { + "step_id": 3, + "timestamp": "2026-08-23T10:50:02.486Z", + "source": "agent", + "model_name": "claude-opus-5", + "message": "NANOPY_EVENT_PROBE_20260823", + "metrics": {"prompt_tokens":3900,"completion_tokens":1,"cached_tokens":3768}, + "llm_call_count": 1 + } + ], + "final_metrics": { + "total_prompt_tokens": 8590, + "total_completion_tokens": 110, + "total_cached_tokens": 5015, + "total_cost_usd": 0.032467499999999996, + "extra": {"cost_source":"claude_code_result","includes_auxiliary_model_calls":true,"auxiliary_model":"claude-haiku-4-5-20251001"} + } +} +``` + +Here `final_metrics` aggregates Opus and auxiliary Haiku from `modelUsage`, so it exceeds the sum of the two visible steps. That discrepancy must remain documented; hidden calls must not be discarded or Haiku tokens forced onto a visible step merely to make the totals equal. + +For easier comparison, all five reconstructed examples use the same task: + +```text +Read src/main.rs, then summarize what it does. +``` + +The following ATIF snippets expand only mapping-relevant fields under `steps`. A complete document also requires root fields such as `schema_version` and `agent`. ATIF-v1.7 defines `Step.timestamp` as an optional ISO 8601 string. nano should nevertheless populate it whenever a reliable source exists and record that source under `extra.timestamp_source`. + +--- + +## 1. Pi: events around turns, messages, and tool execution + +Pi's real `AgentEvent` union is defined in [`packages/agent/src/types.ts`](../../../references/pi/packages/agent/src/types.ts), with JSONL documented in [`packages/coding-agent/docs/json.md`](../../../references/pi/packages/coding-agent/docs/json.md). Event types include: + +```typescript +type AgentEvent = + | { type: "agent_start" } + | { type: "agent_end"; messages: AgentMessage[] } + | { type: "turn_start" } + | { type: "turn_end"; message: AgentMessage; toolResults: ToolResultMessage[] } + | { type: "message_start"; message: AgentMessage } + | { type: "message_update"; message: AgentMessage; assistantMessageEvent: AssistantMessageEvent } + | { type: "message_end"; message: AgentMessage } + | { type: "tool_execution_start"; toolCallId: string; toolName: string; args: any } + | { type: "tool_execution_update"; toolCallId: string; toolName: string; args: any; partialResult: any } + | { type: "tool_execution_end"; toolCallId: string; toolName: string; result: any; isError: boolean }; +``` + +### 1.1 What Pi's public AgentEvent serialization looks like + +The example omits text deltas and keeps only authoritative completion events for one tool call and two model responses: + +```jsonl +{"type":"session","version":3,"id":"pi-session-1","timestamp":"2026-08-23T08:00:00.000Z","cwd":"/workspace"} +{"type":"agent_start"} +{"type":"turn_start"} +{"type":"message_start","message":{"role":"user","content":"Read src/main.rs, then summarize what it does.","timestamp":1787472000100}} +{"type":"message_end","message":{"role":"user","content":"Read src/main.rs, then summarize what it does.","timestamp":1787472000100}} +{"type":"tool_execution_start","toolCallId":"call_read_1","toolName":"read","args":{"path":"src/main.rs"}} +{"type":"tool_execution_end","toolCallId":"call_read_1","toolName":"read","result":{"content":"fn main() { println!(\"hello\"); }"},"isError":false} +{"type":"turn_end","message":{"role":"assistant","content":[{"type":"text","text":"I will read the entry-point file first."},{"type":"toolCall","id":"call_read_1","name":"read","arguments":{"path":"src/main.rs"}}],"api":"anthropic-messages","provider":"anthropic","model":"claude-sonnet-4","usage":{"input":120,"output":24,"cacheRead":0,"cacheWrite":0,"totalTokens":144,"cost":{"input":0.00036,"output":0.00036,"cacheRead":0,"cacheWrite":0,"total":0.00072}},"stopReason":"toolUse","timestamp":1787472001250},"toolResults":[{"role":"toolResult","toolCallId":"call_read_1","toolName":"read","content":[{"type":"text","text":"fn main() { println!(\"hello\"); }"}],"isError":false,"timestamp":1787472001420}]} +{"type":"turn_start"} +{"type":"turn_end","message":{"role":"assistant","content":[{"type":"text","text":"This is a minimal Rust program whose main function prints hello to standard output."}],"api":"anthropic-messages","provider":"anthropic","model":"claude-sonnet-4","usage":{"input":168,"output":31,"cacheRead":0,"cacheWrite":0,"totalTokens":199,"cost":{"input":0.000504,"output":0.000465,"cacheRead":0,"cacheWrite":0,"total":0.000969}},"stopReason":"stop","timestamp":1787472002100},"toolResults":[]} +``` + +The time semantics are mixed: + +- The session header has an ISO timestamp. +- The `AgentEvent` envelopes for `turn_start` and `tool_execution_start/end` have no timestamp. +- The real `UserMessage`, `AssistantMessage`, and `ToolResultMessage` types have Unix-millisecond `timestamp` fields; see [`packages/ai/src/types.ts`](../../../references/pi/packages/ai/src/types.ts). +- Pi's newer persistent harness entries also have storage-assigned timestamps, but that is another journal layer, not the Source Records above. + +### 1.2 Corresponding ATIF + +```json +{ + "steps": [ + { + "step_id": 1, + "timestamp": "2026-08-23T08:00:00.100Z", + "source": "user", + "message": "Read src/main.rs, then summarize what it does.", + "extra": {"timestamp_source": "user_message"} + }, + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.250Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "I will read the entry-point file first.", + "tool_calls": [ + { + "tool_call_id": "call_read_1", + "function_name": "read", + "arguments": {"path": "src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "call_read_1", + "content": "fn main() { println!(\"hello\"); }", + "extra": {"is_error": false} + } + ] + }, + "metrics": { + "prompt_tokens": 120, + "completion_tokens": 24, + "cached_tokens": 0, + "extra": {"estimated_cost_usd": 0.00072, "cost_source": "pi_model_catalog"} + }, + "llm_call_count": 1, + "extra": {"timestamp_source": "assistant_message"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.100Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "This is a minimal Rust program whose main function prints hello to standard output.", + "metrics": { + "prompt_tokens": 168, + "completion_tokens": 31, + "cached_tokens": 0, + "extra": {"estimated_cost_usd": 0.000969, "cost_source": "pi_model_catalog"} + }, + "llm_call_count": 1, + "extra": {"timestamp_source": "assistant_message"} + } + ] +} +``` + +Pi comes closest of the five to a lossless per-model-call mapping: `turn_end.message` already carries model, content, tool calls, usage, and timestamp, while `toolResults` preserves call association. Cost still needs provenance: mainstream provider adapters currently calculate it from model-catalog rates, so structural completeness does not make it a provider charge. + +--- + +## 2. Claude Code: the SDK message stream must be joined by message ID and tool_use ID + +Claude Code's SDK output types are defined in [`src/entrypoints/sdk/coreSchemas.ts`](../../../references/claude-code/src/entrypoints/sdk/coreSchemas.ts). Harbor's Claude Code adapter uses `--output-format=stream-json --print`, then organizes multiple SDK messages into ATIF. + +### 2.1 What Claude Code SDK Source Records look like + +```jsonl +{"type":"system","subtype":"init","cwd":"/workspace","tools":["Read"],"model":"claude-sonnet-4","permissionMode":"bypassPermissions","uuid":"sys-1","session_id":"claude-session-1","apiKeySource":"ANTHROPIC_API_KEY","mcp_servers":[],"slash_commands":[],"output_style":"default","skills":[],"plugins":[],"claude_code_version":"2.x"} +{"type":"assistant","message":{"id":"msg_tool_1","type":"message","role":"assistant","model":"claude-sonnet-4","content":[{"type":"text","text":"I will read the entry-point file first."},{"type":"tool_use","id":"toolu_read_1","name":"Read","input":{"file_path":"/workspace/src/main.rs"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":120,"output_tokens":24,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}},"parent_tool_use_id":null,"uuid":"assistant-1","session_id":"claude-session-1"} +{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_read_1","content":"fn main() { println!(\"hello\"); }","is_error":false}]},"parent_tool_use_id":null,"tool_use_result":{"type":"text","file":{"filePath":"/workspace/src/main.rs","content":"fn main() { println!(\"hello\"); }"}},"timestamp":"2026-08-23T08:00:01.420Z","uuid":"user-tool-result-1","session_id":"claude-session-1"} +{"type":"assistant","message":{"id":"msg_final_1","type":"message","role":"assistant","model":"claude-sonnet-4","content":[{"type":"text","text":"This is a minimal Rust program whose main function prints hello to standard output."}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":168,"output_tokens":31,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}},"parent_tool_use_id":null,"uuid":"assistant-2","session_id":"claude-session-1"} +{"type":"result","subtype":"success","duration_ms":2100,"duration_api_ms":1700,"is_error":false,"num_turns":2,"result":"This is a minimal Rust program whose main function prints hello to standard output.","stop_reason":"end_turn","total_cost_usd":0.001689,"usage":{"input_tokens":288,"output_tokens":55},"modelUsage":{"claude-sonnet-4":{"inputTokens":288,"outputTokens":55,"cacheReadInputTokens":0,"cacheCreationInputTokens":0,"costUSD":0.001689,"contextWindow":200000,"maxOutputTokens":64000}},"permission_denials":[],"uuid":"result-1","session_id":"claude-session-1"} +``` + +The SDK stream does not give every event a uniform timestamp. Local 2.1.237 adds ISO timestamps to `assistant` and `user` messages, but `system`, `api_retry`, and `result` can still omit them. In the fixed source snapshot, the `user` timestamp is optional and consumers are explicitly told to fall back to receive time for older emitters. + +### 2.2 Corresponding ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.250Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "I will read the entry-point file first.", + "tool_calls": [ + { + "tool_call_id": "toolu_read_1", + "function_name": "Read", + "arguments": {"file_path": "/workspace/src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "toolu_read_1", + "content": "fn main() { println!(\"hello\"); }", + "extra": {"is_error": false} + } + ] + }, + "metrics": {"prompt_tokens": 120, "completion_tokens": 24, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg_tool_1", "timestamp_source": "assistant_message_or_receiver"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.100Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "This is a minimal Rust program whose main function prints hello to standard output.", + "metrics": {"prompt_tokens": 168, "completion_tokens": 31, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg_final_1", "timestamp_source": "assistant_message_or_receiver"} + } + ], + "final_metrics": { + "total_prompt_tokens": 288, + "total_completion_tokens": 55, + "total_cached_tokens": 0, + "total_cost_usd": 0.001689, + "total_steps": 3, + "extra": {"cost_source": "claude_code_result", "cost_kind": "agent_reported"} + } +} +``` + +The mapping requires two key operations: + +1. Build one model step from each assistant `message.id`. +2. Use `tool_result.tool_use_id` to find the earlier `tool_use.id` and attach the environment result to that ATIF step's `observation`. + +`result.total_cost_usd` is Claude Code's whole-run aggregate. It may include auxiliary model calls and is not a billing field directly returned by an Anthropic Messages response. It cannot be proportionally fabricated across the two visible steps. The example therefore writes only `final_metrics.total_cost_usd`, records its source, and leaves per-step `metrics.cost_usd` absent. + +--- + +## 3. Codex: public JSONL is a thread/turn/item projection, with usage aggregated only at turn end + +The actual event definitions for Codex `exec --json` are in [`codex-rs/exec/src/exec_events.rs`](../../../references/codex/codex-rs/exec/src/exec_events.rs). The top-level `ThreadEvent` variants are `thread.started`, `turn.started`, `item.started/updated/completed`, `turn.completed/failed`, and `error`. + +### 3.1 What Codex ThreadEvent Source Records look like + +```jsonl +{"type":"thread.started","thread_id":"codex-thread-1"} +{"type":"turn.started"} +{"type":"item.started","item":{"id":"item_cmd_1","type":"command_execution","command":"sed -n '1,200p' src/main.rs","aggregated_output":"","exit_code":null,"status":"in_progress"}} +{"type":"item.completed","item":{"id":"item_cmd_1","type":"command_execution","command":"sed -n '1,200p' src/main.rs","aggregated_output":"fn main() { println!(\"hello\"); }\n","exit_code":0,"status":"completed"}} +{"type":"item.completed","item":{"id":"item_msg_1","type":"agent_message","text":"This is a minimal Rust program whose main function prints hello to standard output."}} +{"type":"turn.completed","usage":{"input_tokens":288,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":55,"reasoning_output_tokens":12}} +``` + +These public `ThreadEvent` values contain no timestamps. Some begin/end objects in the lower-level Codex protocol have `started_at_ms` and `completed_at_ms`, but the normalized `exec --json` events drop them. If Harbor consumes only stdout JSONL, it can only add receive time while reading each line. + +### 3.2 Corresponding ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.200Z", + "source": "agent", + "message": "", + "tool_calls": [ + { + "tool_call_id": "item_cmd_1", + "function_name": "command_execution", + "arguments": {"command": "sed -n '1,200p' src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "item_cmd_1", + "content": "fn main() { println!(\"hello\"); }\n", + "extra": {"exit_code": 0, "status": "completed"} + } + ] + }, + "llm_call_count": 1, + "extra": {"timestamp_source": "receiver", "usage_attribution": "unavailable"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.050Z", + "source": "agent", + "message": "This is a minimal Rust program whose main function prints hello to standard output.", + "llm_call_count": 1, + "extra": {"source_item_id": "item_msg_1", "timestamp_source": "receiver", "usage_attribution": "unavailable"} + } + ], + "final_metrics": { + "total_prompt_tokens": 288, + "total_completion_tokens": 55, + "total_cached_tokens": 0, + "total_steps": 3, + "extra": {"reasoning_output_tokens": 12} + } +} +``` + +This mapping contains genuine information loss: `turn.completed.usage` covers the whole user turn, potentially including multiple model inferences. The public stream provides no per-inference usage that can be assigned reliably to ATIF steps 2 and 3. The correct mapping fills only `final_metrics` and explicitly says attribution is unavailable instead of dividing the aggregate evenly. + +--- + +## 4. OpenCode: events update message parts, and a tool is a state machine + +OpenCode's current V2 SDK types are defined in [`packages/sdk/js/src/v2/gen/types.gen.ts`](../../../references/opencode/packages/sdk/js/src/v2/gen/types.gen.ts), while the CLI projection of those events is in [`packages/opencode/src/cli/cmd/run.ts`](../../../references/opencode/packages/opencode/src/cli/cmd/run.ts). The internal core event `message.part.updated` is also exposed through the V2 SDK event protocol, so it is both an internal event type and a Source Record observable at the SDK boundary. Parts include text, tool, step-start, step-finish, and more. `opencode run --format json` projects them again into `step_start/tool_use/step_finish/text` and adds a top-level timestamp. + +### 4.1 What OpenCode SDK Source Records (`message.part.updated`) look like + +```jsonl +{"id":"evt-1","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472001100,"part":{"id":"part-tool-1","sessionID":"oc-session-1","messageID":"msg-1","type":"tool","callID":"call-read-1","tool":"read","state":{"status":"running","input":{"filePath":"src/main.rs"},"title":"Read src/main.rs","metadata":{},"time":{"start":1787472001090}}}}} +{"id":"evt-2","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472001420,"part":{"id":"part-tool-1","sessionID":"oc-session-1","messageID":"msg-1","type":"tool","callID":"call-read-1","tool":"read","state":{"status":"completed","input":{"filePath":"src/main.rs"},"output":"fn main() { println!(\"hello\"); }","title":"Read src/main.rs","metadata":{},"time":{"start":1787472001090,"end":1787472001418}}}}} +{"id":"evt-3","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472001500,"part":{"id":"part-finish-1","sessionID":"oc-session-1","messageID":"msg-1","type":"step-finish","reason":"tool-calls","cost":0.00072,"tokens":{"input":120,"output":24,"reasoning":0,"cache":{"read":0,"write":0}}}}} +{"id":"evt-4","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472002100,"part":{"id":"part-text-2","sessionID":"oc-session-1","messageID":"msg-2","type":"text","text":"This is a minimal Rust program whose main function prints hello to standard output.","time":{"start":1787472001800,"end":1787472002090}}}} +{"id":"evt-5","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472002120,"part":{"id":"part-finish-2","sessionID":"oc-session-1","messageID":"msg-2","type":"step-finish","reason":"stop","cost":0.000969,"tokens":{"input":168,"output":31,"reasoning":0,"cache":{"read":0,"write":0}}}}} +``` + +Reducing this kind of example to only `type` and `part` would hide an important fact: the real current `message.part.updated` envelope has `properties.time`. Assistant messages also have `time.created/completed`, while tool states have `time.start/end`. + +### 4.2 Corresponding ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.500Z", + "source": "agent", + "message": "", + "tool_calls": [ + { + "tool_call_id": "call-read-1", + "function_name": "read", + "arguments": {"filePath": "src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "call-read-1", + "content": "fn main() { println!(\"hello\"); }", + "extra": {"tool_started_at_ms": 1787472001090, "tool_ended_at_ms": 1787472001418} + } + ] + }, + "metrics": {"prompt_tokens": 120, "completion_tokens": 24, "cached_tokens": 0, "extra":{"estimated_cost_usd":0.00072,"cost_source":"opencode_model_catalog"}}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg-1", "timestamp_source": "event"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.120Z", + "source": "agent", + "message": "This is a minimal Rust program whose main function prints hello to standard output.", + "metrics": {"prompt_tokens": 168, "completion_tokens": 31, "cached_tokens": 0, "extra":{"estimated_cost_usd":0.000969,"cost_source":"opencode_model_catalog"}}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg-2", "timestamp_source": "event"} + } + ] +} +``` + +The converter must aggregate parts by `messageID` and associate running and completed tool states by `callID`. `step-finish` provides per-model-step tokens and cost, making the structural projection straightforward. OpenCode currently calculates that cost from model prices and tokens, however; it is not evidence of a provider-returned charge. + +--- + +## 5. Grok: `streaming-json` is a lightweight state stream reduced from ACP updates + +Grok's actual `streaming-json` wire definitions are in [`xai-grok-pager/src/headless/reducer/acp.rs`](../../../references/grok-build/crates/codegen/xai-grok-pager/src/headless/reducer/acp.rs), with upstream unified events in the neighboring [`mod.rs`](../../../references/grok-build/crates/codegen/xai-grok-pager/src/headless/reducer/mod.rs). The reducer turns ACP updates into events such as `text`, `thought`, `tool_call`, `tool_call_update`, `usage`, and terminal `end`. + +### 5.1 What Grok streaming-json Source Records look like + +```jsonl +{"type":"thought","data":"I need to inspect the Rust entry point."} +{"type":"tool_call","toolCallId":"grok-call-1","title":"Read src/main.rs","kind":"read","status":"pending","toolName":"read_file","rawInput":{"path":"src/main.rs"},"content":[],"locations":[{"path":"src/main.rs"}]} +{"type":"tool_call_update","toolCallId":"grok-call-1","status":"completed","content":[{"type":"content","content":{"type":"text","text":"fn main() { println!(\"hello\"); }"}}],"rawOutput":{"text":"fn main() { println!(\"hello\"); }"},"locations":[{"path":"src/main.rs"}]} +{"type":"usage","messageId":"grok-msg-1","stopReason":"tool_use","usage":{"inputTokens":120,"outputTokens":24,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}} +{"type":"text","data":"This is a minimal Rust program whose main function prints hello to standard output."} +{"type":"usage","messageId":"grok-msg-2","stopReason":"end_turn","usage":{"inputTokens":168,"outputTokens":31,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}} +{"type":"end","stopReason":"EndTurn","sessionId":"grok-session-1","requestId":"grok-request-1","usage":{"input_tokens":288,"output_tokens":55,"total_tokens":343,"num_turns":2}} +``` + +Neither `AcpLine` nor `AcpUsageLine` has a timestamp field. Event order comes from NDJSON line order, and terminal `end` provides only a run aggregate. Grok internally maintains more session and usage state, but a third party integrating through `streaming-json` cannot assume those internal fields appear on stdout. + +### 5.2 Corresponding ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.500Z", + "source": "agent", + "message": "", + "reasoning_content": "I need to inspect the Rust entry point.", + "tool_calls": [ + { + "tool_call_id": "grok-call-1", + "function_name": "read_file", + "arguments": {"path": "src/main.rs"} + } + ], + "observation": { + "results": [ + {"source_call_id": "grok-call-1", "content": "fn main() { println!(\"hello\"); }"} + ] + }, + "metrics": {"prompt_tokens": 120, "completion_tokens": 24, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "grok-msg-1", "timestamp_source": "receiver"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.100Z", + "source": "agent", + "message": "This is a minimal Rust program whose main function prints hello to standard output.", + "metrics": {"prompt_tokens": 168, "completion_tokens": 31, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "grok-msg-2", "timestamp_source": "receiver"} + } + ] +} +``` + +Grok conversion is a small state machine: accumulate thought and text deltas; create a call on `tool_call`; complete its observation with a `tool_call_update` bearing the same `toolCallId`; then close one model step with `usage.messageId`. + +--- + +## 6. Content differences among the five Source Record formats and ATIF + +The examples show that Source Records and ATIF do not preserve wholly different facts. Their content overlaps heavily; the differences lie in organization and completion timing. An adapter should first normalize Source Records faithfully into Native Events rather than fabricate information absent from the source protocol: + +| Observable fact | Pi | Claude Code | Codex | OpenCode | Grok | ATIF target | +|---|---|---|---|---|---|---| +| Model-response boundary | `turn_end.message` | Assistant `message.id` | Not directly exposed; inferred from item/turn | `messageID` + `step-finish` | `usage.messageId` | One `source: agent` step | +| Tool-call ID | `toolCallId` | `tool_use.id` | Item `id` | `callID` | `toolCallId` | `tool_calls[].tool_call_id` | +| Tool result | `toolResults` / execution end | `tool_result` | Completed item | Completed ToolPart | Tool-call update | `observation.results[]` | +| Per-call tokens | Yes | Yes | No; turn aggregate only | Yes | Yes | `step.metrics` | +| Per-call cost | Agent-calculated value, usually not provider charge | None on message; result has an agent-reported run aggregate | None | Agent-calculated value | Depends on backend usage | Write `step.metrics.cost_usd` only with reliable provenance; otherwise use `extra` | +| Time | Messages have it; events are inconsistent | Inconsistent | Absent from public stream | Present on events, messages, and tools | Absent from public stream | Optional `step.timestamp` | + +ATIF is a **completed-state snapshot**: it expects each step to already contain a complete message, tool calls, observation, and metrics. Native Events are **facts as they happen over time**: a tool starts and later completes; model text may arrive as deltas before completion; cost may even resolve asynchronously after the run. Journal Entries add reliable order and record time to those facts, while the Event Journal preserves the replayable append history. ATIF and the Event Journal can derive from the same facts, but they should not share one write model. + +## 7. Concrete recommendations for nanoPyCodeAgent + +### 7.1 Journal Entries must provide record time and ordering + +Native Events do not need to allocate persistence order themselves. After accepting one, the journal writer wraps it in a uniform Journal Entry: + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 17, + "recorded_at": "2026-08-23T08:00:01.420Z", + "type": "tool.completed", + "payload": { + "tool_call_id": "call-read-1", + "tool_name": "read", + "result": "fn main() { println!(\"hello\"); }", + "is_error": false, + "duration_ms": 330, + "source_timestamp": null, + "timestamp_source": "receiver" + } +} +``` + +The semantic constraints should be: + +- `schema_version` describes the internal Journal Entry and Native Event contract, not the ATIF schema version. +- `seq` is allocated by the journal writer, strictly increases, and is authoritative ordering within a run. +- `recorded_at` is the UTC wall-clock time when nano accepted or recorded the event, uses RFC 3339/ISO 8601, and is required on every Journal Entry. +- `source_timestamp` belongs to the Native Event payload and is set only when a Source Record provides reliable source time. When absent, keep it `null`; never present `recorded_at` as source occurrence time. +- Precise duration uses `duration_ms` or paired start/end events rather than relying only on subtraction between two wall-clock values. +- ATIF `step.timestamp` prefers reliable `source_timestamp`, otherwise falls back to `recorded_at`, and records the choice in `extra.timestamp_source`. + +### 7.2 The minimum Native Event set should cover model, tool, and run terminal states + +At minimum, preserve: + +```text +run.started +user.message +model.started +model.completed +tool.started +tool.completed +run.completed +run.failed +``` + +`model.completed` should retain `message_id`, complete content and tool calls, actual model, stop reason, usage, and provider response or generation ID. Only then can an ATIF converter map each inference without reproducing the information loss of Codex public JSONL, where only turn-level aggregates are available. + +### 7.3 The internal Event Journal and the public ATIF Trajectory are different artifacts + +Keep an internal Event Journal. The journal writer appends each Journal Entry; the first implementation may encode one Journal Entry per JSONL line. This JSONL is an internal fact log. It defines no steps, observations, or final metrics and is not exposed by `--trajectory`. + +Use only ATIF-v1.7 for the public trajectory. The ATIF projector consumes the Event Journal live or replays it later, folding multiple Native Events into completed steps. `--trajectory PATH` points to the ATIF file, not the internal Event Journal. This avoids maintaining both a “native trajectory” and ATIF as separate public trajectory semantics; it does not eliminate internal schemas. Native Events and Journal Entries still have an internal schema, and a one-way Native Event → ATIF projector still requires maintenance. + +This is a **later architecture recommendation** derived from the Source Record-to-ATIF comparison. It explicitly supersedes the older proposal in section 8.4 of [`agent_output_and_trajectory.md`](agent_output_and_trajectory.md), where `--trajectory` wrote native trajectory JSONL and the Harbor adapter converted it to ATIF. The stdout `--output-format` contract in that earlier document remains valid; only the public trajectory persistence boundary has changed. + +In implementation, the Event Journal can append Journal Entries during the run. At run end or a checkpoint, fold the current facts into complete ATIF and atomically update the `--trajectory` target using a temporary file plus rename. If the process is interrupted, the Event Journal remains complete through its final intact Journal Entry, while ATIF remains at its latest complete snapshot. + +## Final answer + +Real projects expose no single uniform Source Record. Pi has turn/message/tool records close to internal `AgentEvent`; Claude Code exposes SDK messages; Codex exposes thread items; the OpenCode CLI exposes a public projection of message parts; and Grok exposes a public reduction of ACP updates. An adapter first normalizes these different shapes into nano Native Events, the journal writer then creates Journal Entries, and the ATIF projector finally folds the Event Journal into uniform user and agent steps, tool calls, observations, and metrics. + +When a Source Record example has no timestamp, that does not make time unimportant. More precisely, some source protocols have no uniform timestamp, some place time inside message or tool payloads, and the OpenCode CLI adds time at the top level of its public records. nano's journal writer should assign `seq + recorded_at` to every Journal Entry while preserving any available `source_timestamp`. ATIF projection should state which time source it selected. It must neither depend on clocks hidden behind stdout line order nor fabricate source occurrence times merely to make ATIF look complete. diff --git a/docs/research/en/agent_output_and_trajectory.md b/docs/research/en/agent_output_and_trajectory.md index 81954c4..9f45b3a 100644 --- a/docs/research/en/agent_output_and_trajectory.md +++ b/docs/research/en/agent_output_and_trajectory.md @@ -2,6 +2,8 @@ > Generated from the Chinese source [`../zh-CN/agent_output_and_trajectory.md`](../zh-CN/agent_output_and_trajectory.md). Do not edit by hand. +> **Later decision (2026-08-23):** This document's recommendation that `--trajectory` write a native trajectory JSONL for a Harbor adapter to convert into ATIF has been superseded by [section 7.3 of `agent_events_to_atif_examples.md`](agent_events_to_atif_examples.md#73-the-internal-event-journal-and-the-public-atif-trajectory-are-different-artifacts). The current recommendation keeps an internal append-only Event Journal, which may use JSONL encoding, while `--trajectory` writes only ATIF-v1.7; the Event Journal is not another trajectory. The `--output-format` stdout contract in this document remains unchanged. + Surveyed on 2026-08-22. [`benchmark_headless_interface.md`](benchmark_headless_interface.md) previously proposed the following interface, but did not explain whether the two parameters control the same kind of artifact: diff --git a/docs/research/en/openrouter_cost_accounting.md b/docs/research/en/openrouter_cost_accounting.md new file mode 100644 index 0000000..0315f4e --- /dev/null +++ b/docs/research/en/openrouter_cost_accounting.md @@ -0,0 +1,335 @@ +# OpenRouter Actual Cost, Pricing APIs, and Trajectory Accounting + +> Generated from the Chinese source [`../zh-CN/openrouter_cost_accounting.md`](../zh-CN/openrouter_cost_accounting.md). Do not edit by hand. + +Surveyed on 2026-08-23. + +## Question + +> I use OpenRouter. Does the response contain the actual cost? If not, is there an API for retrieving model prices? +> +> Can cost accounting be implemented together with the native storage format or trajectory? + +## Related research + +For the protocol shape, agent capabilities, endpoint choice, and nano integration boundary of OpenRouter's model-independent API, see [OpenRouter's Unified Model Protocol: Capabilities, Cost, and the nanoPyCodeAgent Integration Boundary](openrouter_unified_protocol.md). This document focuses only on cost sources, delayed reconciliation, and trajectory mapping. + +## Conclusions first + +OpenRouter **does provide actual billed cost**, and it is more reliable than an estimate based on “tokens × the model's current list price.” The unified OpenRouter Chat Completions API returns `usage.cost` in the complete response or the final SSE event. nanoPyCodeAgent currently uses the Anthropic Messages compatibility endpoint, however, and that endpoint's Anthropic-shaped `usage` object does not publicly promise a `cost` field. As a result, `stream.get_final_message().usage` will usually expose only tokens. + +The most reliable implementation for nano has two paths: + +1. When a future Chat Completions transport is in use, write `usage.cost` directly to the `model.completed` Native Event as resolved, `provider_reported` cost. +2. When the current Anthropic Messages transport does not return cost directly, capture `X-Generation-Id` from the HTTP headers and initially mark cost as pending. +3. Call `GET /api/v1/generation?id=...` to retrieve that request's `total_cost`, then append a `model.cost_resolved` Native Event. +4. In both paths, let the journal writer persist the event as a Journal Entry and project it into ATIF `step.metrics.cost_usd`. +5. Preserve the generation ID for every OpenRouter request so missing cost can be reconciled and existing cost audited. +6. Treat the price catalog from `GET /api/v1/model/:author/:slug` or `GET /api/v1/models` only as a budgeting and estimation fallback, never as the historical bill of record. + +## 1. Why OpenRouter documents cost but nano does not see it in the response + +OpenRouter's [Usage Accounting](https://openrouter.ai/docs/cookbook/administration/usage-accounting) documentation says that a complete Chat Completions/Responses response, or the final SSE chunk, includes: + +```json +{ + "usage": { + "prompt_tokens": 194, + "completion_tokens": 2, + "total_tokens": 196, + "cost": 0.00095, + "cost_details": { + "upstream_inference_cost": 0.00090 + } + } +} +``` + +Here: + +- `usage.cost` is the total charged to the current OpenRouter account. +- `cost_details.upstream_inference_cost` is the upstream provider's inference cost. +- With streaming, usage arrives in the final SSE event; without streaming, it is part of the complete response. +- The old `usage: {include: true}` and `stream_options.include_usage` parameters are no longer required. + +OpenRouter's [FAQ](https://openrouter.ai/docs/faq) says that credits use US dollars as their base currency and that both site and API pricing are denominated in dollars. For ordinary credits requests, provider-reported `usage.cost` or `data.total_cost` can therefore map to ATIF `cost_usd`. Internal events should still preserve `currency: "USD"` and the source explicitly rather than letting a target field name imply both currency and provenance. + +The current nano code in [`src/nanopycodeagent/agent.py`](../../../src/nanopycodeagent/agent.py) uses: + +```text +anthropic.Anthropic + -> ANTHROPIC_BASE_URL + -> OpenRouter /api/v1/messages + -> client.messages.stream(...) + -> stream.get_final_message() +``` + +OpenRouter's [Anthropic Messages endpoint](https://openrouter.ai/docs/api/api-reference/anthropic-messages/create-messages) returns Anthropic-compatible `usage.input_tokens`, `usage.output_tokens`, and cache fields. Its public response schema does not list `usage.cost`. The `usage.cost` guarantee from Chat Completions therefore cannot be applied directly to the Messages skin. + +The base models in the Anthropic Python SDK installed by this repository permit extra fields. If the server actually includes `cost` under `usage`, the SDK will not necessarily discard it. The problem is that OpenRouter's public Messages protocol does not promise to send the field, so an implementation cannot depend on that undeclared extension. + +### 1.1 Evidence from four local CLI runs + +On the same day, tests using a non-sensitive sentinel file ran Pi 0.84.2, Codex 0.149.0, Claude Code 2.1.237, and OpenCode 1.18.21 and observed: + +| CLI | Cost in its output | Source assessment | +|---|---|---| +| Pi | Every assistant response has `usage.cost.total` | The source explicitly calls `calculateCost(model, usage)` using model-catalog rates | +| Codex | No cost in `exec --json` | Only turn-aggregate tokens | +| Claude Code | Terminal `result.total_cost_usd` and `modelUsage.*.costUSD` | Agent-reported aggregate; Anthropic Messages usage itself has no cost, and the result may include auxiliary model calls | +| OpenCode | Each `step_finish.cost`; the observed value was `0` | The source calculates from model-catalog prices and tokens; `0` does not necessarily prove the provider charged nothing | + +This shows that “the agent emitted a cost number” and “the provider returned the actual bill” are different claims. A trajectory should distinguish at least: + +- `provider_reported`: for example, OpenRouter generation `total_cost`. +- `agent_calculated`: for example, Pi or OpenCode calculating from a price catalog. +- `agent_reported`: for example, a Claude Code terminal result whose calculation details are encapsulated by the agent. +- `unknown`: for example, Codex's public JSONL, which provides no cost. + +Only `provider_reported` directly answers “what did OpenRouter actually charge for this request?” The other values remain useful observations, but they must not be mixed into one unattributed total. + +## 2. The Messages compatibility path and audit fallback: Generation API + +OpenRouter creates a generation record for every request. Its official [Get a Generation](https://openrouter.ai/docs/api/api-reference/generations/get-generation) API is: + +```http +GET https://openrouter.ai/api/v1/generation?id=gen-1234567890 +Authorization: Bearer +``` + +The important response fields are: + +```json +{ + "data": { + "id": "gen-1234567890", + "model": "anthropic/claude-sonnet-4", + "provider_name": "Anthropic", + "streamed": true, + "native_tokens_prompt": 120, + "native_tokens_completion": 24, + "native_tokens_cached": 0, + "native_tokens_reasoning": 0, + "total_cost": 0.00072, + "usage": 0.00072, + "upstream_inference_cost": null + } +} +``` + +When the response body does not contain `usage.cost`, the trajectory should use `data.total_cost`, which represents the cost recorded against the OpenRouter account for that generation. If Chat Completions already returned `usage.cost`, the Generation API instead serves reconciliation and audit. `upstream_inference_cost` should not replace the amount actually charged to the account. The Usage Accounting documentation explicitly notes that, when looking up a Generation ID, this field is available only for BYOK requests; for non-BYOK requests it is normally `0` or `null`. + +### 2.1 Where the generation ID comes from + +Do not blindly reuse the Anthropic `message.id`. A Messages API message ID may be `msg_...`, while an OpenRouter generation ID is `gen-...`. OpenRouter supplies `X-Generation-Id` in the HTTP response headers. + +The current Anthropic SDK stream object exposes the underlying response headers, so the existing `with client.messages.stream(...) as stream:` block can read: + +```python +generation_id = stream.response.headers.get("x-generation-id") +``` + +After the response completes, save this value alongside `message.id`, model, and usage. If the header is missing, cost status should be unknown—not a synthetic zero. + +### 2.2 Why this is more reliable than price-catalog arithmetic + +The generation record already knows the final: + +- model and provider; +- fallback and routing result; +- native-tokenizer input, output, cache, and reasoning tokens; +- billing rules in effect at the time; +- `total_cost` actually recorded against the OpenRouter account. + +A price-catalog estimate is vulnerable to provider routing, fallback, cache reads and writes, reasoning, images, search, per-request charges, service tiers, and price changes. For the historical question “what did this run actually cost?”, the generation record is the more appropriate source of truth. + +## 3. A model pricing API does exist, but it should be treated as an estimation tool + +OpenRouter provides: + +```http +GET https://openrouter.ai/api/v1/model/anthropic/claude-sonnet-4 +GET https://openrouter.ai/api/v1/models +Authorization: Bearer +``` + +The official [Models API](https://openrouter.ai/docs/api/api-reference/models/get-models) returns data such as: + +```json +{ + "data": { + "id": "openai/gpt-4", + "pricing": { + "prompt": "0.00003", + "completion": "0.00006", + "request": "0", + "image": "0" + } + } +} +``` + +These string prices are dollar amounts per token, per request, or per corresponding unit. For example, `prompt = 0.00003` means `$30 / 1M tokens`. The website often displays prices per million tokens; do not divide the API value by another million. + +The simplest text-only estimate is: + +```text +estimated_cost = + prompt_tokens × pricing.prompt + + completion_tokens × pricing.completion + + request_count × pricing.request +``` + +A real implementation must also handle any present `input_cache_read`, `input_cache_write`, `internal_reasoning`, image, web-search, and other billing fields, and must use `Decimal` rather than binary floating point for accounting. + +The pricing API is appropriate for: + +- pre-request budgeting and max-cost guards; +- showing approximate unit prices in a UI; +- clearly labeled estimates when a provider has no actual-cost API; +- offline model-price comparisons. + +The pricing API is not appropriate for: + +- backfilling OpenRouter's historical actual bill; +- guessing the final provider after automatic routing; +- applying current prices to past runs; +- presenting an estimate as reported cost when fields are missing. + +## 4. Implementing this with Native Events, the Event Journal, and ATIF + +### 4.1 Native Events express facts; Journal Entries are persisted immediately + +If a Chat Completions response or terminal SSE event directly returns `usage.cost`, `model.completed` can immediately record: + +```json +{ + "cost": { + "status": "resolved", + "amount": "0.00072", + "currency": "USD", + "source": "openrouter_response.usage.cost", + "kind": "provider_reported" + } +} +``` + +This case does not require another Generation API query solely to obtain cost, though the generation ID should still be retained for audit. + +If the current Anthropic Messages response has no cost, core first emits a `model.completed` Native Event when the model response completes. The journal writer adds persistence metadata and immediately appends the following Journal Entry without waiting for the cost lookup: + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 12, + "recorded_at": "2026-08-23T08:00:01.250Z", + "type": "model.completed", + "payload": { + "provider": "openrouter", + "model": "anthropic/claude-sonnet-4", + "message_id": "msg_abc123", + "generation_id": "gen-1234567890", + "usage": { + "input_tokens": 120, + "output_tokens": 24, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 0 + }, + "cost": { + "status": "pending", + "source": "openrouter_generation" + } + } +} +``` + +After the Generation API succeeds, append: + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 13, + "recorded_at": "2026-08-23T08:00:01.520Z", + "type": "model.cost_resolved", + "payload": { + "generation_id": "gen-1234567890", + "amount": "0.00072", + "currency": "USD", + "source": "openrouter_generation.total_cost", + "model": "anthropic/claude-sonnet-4", + "provider_name": "Anthropic" + } +} +``` + +This has three benefits: model output is not lost when the cost API is temporarily unavailable; the append-only Event Journal never has to rewrite an old Journal Entry; and the ATIF projector can join the two Journal Entry payloads on `generation_id`. + +### 4.2 ATIF mapping rules + +When the generation lookup for a model call succeeds: + +```json +{ + "metrics": { + "prompt_tokens": 120, + "completion_tokens": 24, + "cached_tokens": 0, + "cost_usd": 0.00072, + "extra": { + "cost_source": "openrouter_generation.total_cost", + "generation_id": "gen-1234567890" + } + } +} +``` + +Populate run-level `final_metrics.total_cost_usd` with a sum only when every billable model call has resolved. If any generation is missing or its query fails: + +- Known steps may still contain their own `metrics.cost_usd`. +- Do not fill an unknown step with `0`. +- Prefer omitting `final_metrics.total_cost_usd`. +- Record `known_cost_usd`, `cost_is_partial: true`, and the missing generation IDs under `final_metrics.extra`. + +Zero is valid only when OpenRouter explicitly reports zero cost for the request, such as a genuinely free request or certain cache hits. It must never mean “no data was available.” + +### 4.3 Query timing + +The recommended sequence is: + +1. Persist `model.completed` immediately when the model finishes. +2. If the response contains `usage.cost`, mark it resolved directly. +3. If cost is absent, query actual cost by generation ID during run finalization. +4. Use bounded retries because generation metadata becomes queryable asynchronously. +5. A failed query must not change the agent task's success or failure state; it only marks cost completeness as partial. +6. Finally, write the complete ATIF snapshot atomically. + +If the CLI later needs to return as quickly as possible, cost enrichment can also happen offline. The Event Journal already contains the generation ID, so the ability to reconcile the bill is preserved. + +## 5. Effect on the 0.8.x trajectory decision + +Cost can ship in the same development series as trajectory, but it should be a provider-aware enrichment rather than OpenRouter HTTP logic embedded in the ATIF serializer: + +```text +OpenRouter Chat response ─ usage.cost ─────────────┐ + ↓ +Anthropic Messages response ─ generation_id ─ cost resolver + │ + ↓ + Native Event + Journal Entry + │ + ↓ + Event Journal + │ + ↓ + ATIF projector +``` + +This also shows why Native Events plus an Event Journal are more robust than assembling ATIF JSON directly during execution: actual cost may arrive after the model response, ATIF is a completed-state document, and the Event Journal naturally represents pending → resolved. + +## Final answer + +When you use OpenRouter, actual cost is available. The unified OpenRouter Chat Completions API can return `usage.cost` directly, and OpenRouter also exposes generation-specific `total_cost`. nano does not currently see those values mainly because it uses the Anthropic Messages compatibility protocol, whose `usage` schema does not promise to carry cost. See the [OpenRouter unified protocol survey](openrouter_unified_protocol.md) for the protocol choice and migration rationale. + +In a future Chat Completions transport, prefer `usage.cost`. For the current Messages transport—or cases such as a stream ending before final usage—capture `X-Generation-Id` and call `/api/v1/generation` to backfill `total_cost`. Both values enter Native Events as provider-reported cost, are persisted by the journal writer as Journal Entries, and are eventually projected to ATIF. A model pricing API also exists, but it is better suited to budgeting and fallback estimation; it must not overwrite actual OpenRouter billing or present a current-price estimate as a historical bill. diff --git a/docs/research/en/openrouter_unified_protocol.md b/docs/research/en/openrouter_unified_protocol.md new file mode 100644 index 0000000..56c0eff --- /dev/null +++ b/docs/research/en/openrouter_unified_protocol.md @@ -0,0 +1,392 @@ +# OpenRouter's Unified Model Protocol: Capabilities, Cost, and the nanoPyCodeAgent Integration Boundary + +> Generated from the Chinese source [`../zh-CN/openrouter_unified_protocol.md`](../zh-CN/openrouter_unified_protocol.md). Do not edit by hand. + +Surveyed on 2026-08-23. + +## Question + +> Does OpenRouter have its own model-independent protocol that both returns actual cost directly and provides the LLM and agent capabilities commonly used through the OpenAI and Anthropic protocols? +> +> If so, should nanoPyCodeAgent use it as its model-independent provider protocol going forward? + +## Conclusions first + +OpenRouter does provide a unified model-access surface, but the more accurate name is the **OpenRouter unified API**, not a completely separate “native OpenRouter messaging protocol.” Its primary entry point is the OpenAI-compatible Chat Completions endpoint: + +```http +POST https://openrouter.ai/api/v1/chat/completions +``` + +The same request and response shape can select models from different vendors, while OpenRouter adds unified provider routing, fallback, usage accounting, and related capabilities. It supports familiar features including streaming, tool calling, structured outputs, reasoning, multimodal input, and prompt caching, though individual models and providers still differ in which parameters they support. + +For nanoPyCodeAgent, this should become the preferred protocol for a future **model-independent OpenRouter transport**: + +1. Use Chat Completions by default rather than continuing to treat Anthropic Messages as OpenRouter's general-purpose protocol. +2. Read `usage.cost` directly from a complete non-streaming response or the final SSE event of a streaming response. +3. Preserve the generation ID as well, using the Generation API to retrieve or audit actual cost when necessary. +4. Set `provider.require_parameters: true` to avoid routing to a provider that would ignore required parameters. +5. Keep the OpenRouter Responses API as a peer candidate. Its item/event model is richer, but nano does not currently depend on those additional capabilities. +6. Treat the wire response as a Source Record that a transport adapter must convert into nano Native Events. It neither replaces the Event Journal nor directly equals ATIF. + +## 1. First, clarify the phrase “native OpenRouter protocol” + +OpenRouter's [FAQ](https://openrouter.ai/docs/faq) and [Quickstart](https://openrouter.ai/docs/quickstart) describe `/api/v1/chat/completions` as an OpenAI-compatible API. An OpenAI SDK can be used by pointing its base URL and API key at OpenRouter; OpenRouter's own SDK is another option. + +The following concepts should therefore remain distinct: + +| Concept | Meaning | Model-independent? | +|---|---|---| +| OpenAI Chat Completions | The base messages/choices/tool_calls shape defined by OpenAI | The base protocol itself is not tied to one model | +| OpenRouter unified API | An OpenAI-compatible shape plus OpenRouter extensions for cross-model routing, provider constraints, unified usage/cost, and more | Yes; suitable as nano's OpenRouter transport | +| OpenRouter Client SDK | A lightweight, type-safe wrapper around that HTTP API | It is a client implementation, not another wire protocol | +| OpenRouter Agent SDK | Agent loop, tool execution, and state management layered above model calls | It is an agent runtime, not an LLM wire protocol | + +The rest of this document therefore uses “OpenRouter unified API” or “OpenRouter Chat Completions transport,” avoiding “native OpenRouter protocol,” which could incorrectly suggest a fourth, wholly new set of message semantics. nano's purpose is to implement its own agent core. Even if the OpenRouter Agent SDK can provide an agent loop directly, it should not replace nano's tool loop, Native Events, or trajectory. If an official SDK is adopted, the thin Client SDK is the better fit. + +## 2. Roles of the three relevant endpoints + +| Endpoint | Protocol shape | Status and capabilities | Recommendation for nano | +|---|---|---|---| +| `/api/v1/chat/completions` | OpenAI-compatible messages/choices | OpenRouter's main entry point; supports streaming, tools, structured output, and unified usage | **Default for a model-independent OpenRouter transport** | +| `/api/v1/responses` | Item/event-oriented Responses | OpenAI-compatible; supports reasoning, tools, and web search, but currently only stateless requests | Evaluate as a peer candidate, not the default for this phase | +| `/api/v1/messages` | Anthropic Messages-compatible | Convenient for reusing the Anthropic SDK and content blocks; the public usage schema does not promise `cost` | Retain as a compatibility transport, not the default OpenRouter abstraction | + +OpenRouter's [Chat Completions API](https://openrouter.ai/docs/api/api-reference/chat/create-a-chat-completion) supports model, messages, tools, tool choice, response format, reasoning, provider routing, fallback models, streaming, and other parameters. + +The [Responses API](https://openrouter.ai/docs/api/reference/responses/overview) uses a data model closer to an event/item stream and also supports reasoning, tool calling, and web search. OpenRouter currently supports it only in stateless form: every request must carry the full history, and `store: true` or a non-empty `previous_response_id` is rejected. It does not lack the basic capabilities needed by nano's agent loop; it merely adds another input/output item and streaming-event mapping that nano does not currently require. + +Chat Completions is the preferred choice not because Responses “cannot build an agent,” but because the official Quickstart still presents Chat Completions as the most direct entry point and nano's current message/tool loop is closer to that shape. A transport interface should avoid embedding `choices[]` into core so a Responses adapter can be added later without rewriting Native Events again. + +The [Anthropic Messages endpoint](https://openrouter.ai/docs/api/api-reference/anthropic-messages/create-messages) is a compatibility surface. It allows the current nano agent loop to access OpenRouter with almost no changes, but it also constrains OpenRouter-specific extensions to the Anthropic response schema. Cost is the clearest current example. + +## 3. Agent fundamentals covered by Chat Completions + +### 3.1 Multi-turn messages and streaming + +A request sends the complete user/assistant/tool history through `messages`. With `stream: true`, the response uses SSE. Ordinary text appears in incremental `delta.content`, while tool calls appear in incremental `delta.tool_calls`. + +The transport must assemble these increments by choice index, tool-call index, and call ID, producing a complete tool name and JSON arguments before executing the tool. A single SSE frame must not be treated as a complete Tool Call Native Event. + +### 3.2 Tool calling + +OpenRouter's [Tool Calling](https://openrouter.ai/docs/guides/features/tool-calling) uses the OpenAI function-calling shape: + +- The request declares a name, description, and JSON Schema under `tools[].function`. +- The assistant requests calls through `message.tool_calls[]`. +- The client executes local tools. +- The next request returns each result with `role: "tool"` and `tool_call_id`. +- `tool_choice` is supported, and some models support parallel tool calls. + +“The protocol supports tools” does not mean “every model supports tools.” The model catalog declares supported parameters, and routing should require the selected provider to support parameters on which the task depends. + +### 3.3 Structured outputs + +[Structured Outputs](https://openrouter.ai/docs/guides/features/structured-outputs) uses `response_format.type = "json_schema"` with a JSON Schema constraint and can be combined with streaming. Model support varies. If a task depends on a strict schema, pair it with: + +```json +{ + "provider": { + "require_parameters": true + } +} +``` + +OpenRouter's [Provider Routing](https://openrouter.ai/docs/guides/routing/provider-selection) documentation explains that providers may otherwise ignore optional parameters they do not support. `require_parameters` restricts candidates to providers that support the requested parameters, which matters for portable tool calling and structured output. + +### 3.4 Reasoning, multimodal input, and caching + +OpenRouter also provides: + +- [Reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens): a partially unified set of controls and response fields for reasoning; models still vary in whether they expose reasoning content. +- [Multimodal requests](https://openrouter.ai/docs/guides/overview/multimodal/overview): images and other content continue to travel in Chat Completions messages/content blocks. +- [Prompt caching](https://openrouter.ai/docs/guides/best-practices/prompt-caching): unified cache accounting across supported models and providers, though automatic caching, explicit breakpoints, and TTL capabilities are not identical. +- Model and provider fallback: OpenRouter can route among candidate models or providers; the final response model and generation metadata are the actual result for that call. + +“Model-independent” therefore means that **one base request/response contract can access multiple models**, not that every model has identical capabilities, parameter semantics, or quality. + +## 4. A realistically shaped tool-call round trip + +The IDs, token counts, and amounts below are illustrative, but the field shapes follow the OpenRouter Chat Completions, Tool Calling, and Usage Accounting documentation. + +### 4.1 Initial request + +```http +POST /api/v1/chat/completions +Authorization: Bearer +Content-Type: application/json +``` + +```json +{ + "model": "anthropic/claude-sonnet-4", + "messages": [ + { + "role": "user", + "content": "Read pyproject.toml and tell me the project name" + } + ], + "tools": [ + { + "type": "function", + "function": { + "name": "read_file", + "description": "Read a UTF-8 text file", + "parameters": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"], + "additionalProperties": false + } + } + } + ], + "tool_choice": "auto", + "provider": { + "require_parameters": true + }, + "stream": false +} +``` + +### 4.2 The model requests a tool call + +```json +{ + "id": "gen-abc123", + "model": "anthropic/claude-sonnet-4", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_read_1", + "type": "function", + "function": { + "name": "read_file", + "arguments": "{\"path\":\"pyproject.toml\"}" + } + } + ] + }, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 205, + "completion_tokens": 18, + "total_tokens": 223, + "cost": 0.00071, + "cost_details": { + "upstream_inference_cost": 0.00066 + } + } +} +``` + +`usage.cost` is the total OpenRouter charged the current account for this call; it is not an estimate nano calculated from a public price catalog. + +### 4.3 Execute the tool and continue the request + +After the client executes `read_file` locally, it places both the original assistant tool call and the tool result back into the message history: + +```json +{ + "model": "anthropic/claude-sonnet-4", + "messages": [ + { + "role": "user", + "content": "Read pyproject.toml and tell me the project name" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_read_1", + "type": "function", + "function": { + "name": "read_file", + "arguments": "{\"path\":\"pyproject.toml\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_read_1", + "content": "[project]\nname = \"nanoPyCodeAgent\"" + } + ], + "tools": [ + { + "type": "function", + "function": { + "name": "read_file", + "description": "Read a UTF-8 text file", + "parameters": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"], + "additionalProperties": false + } + } + } + ], + "provider": { + "require_parameters": true + } +} +``` + +The final response's `choices[0].message.content` is the textual answer for the user and again carries usage/cost for that model call. One agent turn may contain multiple model calls, so run cost must account for each actual call separately and sum them only when completeness is known. + +## 5. Cost: direct responses, streaming, and the Generation API + +OpenRouter's [Usage Accounting](https://openrouter.ai/docs/cookbook/administration/usage-accounting) documentation says every complete response contains detailed usage, including prompt, completion, reasoning, and cache tokens, total cost, and cost details. + +### 5.1 Non-streaming + +Read these fields directly from the complete JSON response: + +```text +usage.cost +usage.cost_details.upstream_inference_cost +``` + +For ordinary OpenRouter credits calls, `usage.cost` is the provider-reported amount nano should use. `upstream_inference_cost` is provider cost detail and must not replace the total actually charged to the OpenRouter account. + +OpenRouter's [FAQ](https://openrouter.ai/docs/faq) says its credits use US dollars as their base currency and that site and API prices are denominated in dollars. `usage.cost` from ordinary credits calls can therefore map to ATIF `cost_usd`. Native Events should still explicitly preserve `currency: "USD"` and the field source rather than inferring currency from the target field name. + +### 5.2 Streaming + +`usage` appears in the final SSE event. The transport must consume through the terminal event before marking a model call's cost resolved. If a stream ends early, cost must not become `0`; retain an unknown or pending state and the generation ID. + +The old: + +```json +{"usage": {"include": true}} +``` + +and: + +```json +{"stream_options": {"include_usage": true}} +``` + +are no longer required to obtain usage; the official documentation marks them deprecated or without effect. + +### 5.3 The Generation API is a fallback and audit path + +Even when the main path receives `usage.cost` directly, preserve the response header `X-Generation-Id` or an equivalent generation ID. Query `data.total_cost` through: + +```http +GET /api/v1/generation?id= +``` + +when: + +- the Anthropic Messages compatibility endpoint is in use and the response body contains no cost; +- streaming disconnects before the final usage event; +- the actual routed model or provider needs to be verified; +- offline cost reconciliation or audit is needed. + +See [OpenRouter Actual Cost, Pricing APIs, and Trajectory Accounting](openrouter_cost_accounting.md) for detailed accounting and ATIF mapping. Price fields from the model catalog are suitable only for budgets and explicitly labeled estimates; they must not overwrite provider-reported cost. + +## 6. Relationship between OpenRouter wire events and nano internal events + +The unified API solves the **provider transport** problem, not trajectory storage. The recommended data flow is: + +```text +OpenRouter HTTP response / SSE frames + │ + │ Source Records + ▼ +OpenRouter Chat transport + - assemble content deltas + - assemble tool-call arguments + - normalize usage / cost / errors + │ + │ Native Events + ▼ + journal writer + │ + │ Journal Entries + ▼ + Event Journal + │ + ▼ + ATIF projector + │ + ▼ + ATIF Trajectory +``` + +The mapping is: + +| OpenRouter data | nano concept | Reason | +|---|---|---| +| Raw JSON response, response headers, or one SSE frame | Source Record | It is a raw observation from an external protocol, not a core domain event | +| Fully assembled assistant output | `model.completed` Native Event | Core now understands that one model call has completed | +| Fully assembled `tool_calls[]` | `model.completed.payload.tool_calls` | Part of the model-completion fact; streaming arguments must be assembled before any tool executes | +| `usage.cost` | Resolved provider-reported cost in `model.completed` | Cost arrives with the model call | +| `total_cost` retrieved later from the Generation API | `model.cost_resolved` Native Event | Cost arrives late, so an append-only event supplements rather than rewrites an old Journal Entry | +| `run_id`, `seq`, and `recorded_at` added by the journal writer | Journal Entry persistence metadata | Part of a complete Journal Entry, not part of the OpenRouter protocol | + +OpenRouter JSONL/SSE should therefore not be renamed “native trajectory,” and the ATIF serializer should not directly understand every provider wire shape. The transport adapter owns protocol differences; the semantic boundaries among Native Events, the Event Journal, and ATIF remain stable. + +## 7. nano cannot currently switch by changing only the base URL + +The current repository is tightly coupled to the Anthropic SDK transport: + +- [`agent.py`](../../../src/nanopycodeagent/agent.py) directly creates `anthropic.Anthropic` and calls `client.messages.stream(...)`. +- Conversation history uses `anthropic.types.MessageParam`. +- Tool calls and results use `ToolUseBlock` and `ToolResultBlockParam`. +- All four tool definitions are typed as `anthropic.types.ToolParam`. +- Test fake clients and exception types also model Anthropic Messages. + +OpenRouter Chat Completions tool declarations are close to the existing JSON Schemas, but assistant tool calls, tool results, streaming deltas, and finish reasons have different shapes. Migration is therefore not a matter of changing `ANTHROPIC_BASE_URL`; it requires extracting a transport boundary. + +### 7.1 Recommended minimum implementation sequence + +1. Define provider-neutral internal message, content, tool-call, tool-result, and usage types. +2. Wrap the existing logic in `AnthropicMessagesTransport` without changing behavior. +3. Add `OpenRouterChatTransport` to assemble Chat Completions requests, responses, and SSE streams. +4. Have both transports emit only the same Native Events to core. +5. Let the OpenRouter transport prefer `usage.cost` and delegate to the generation cost resolver when it is missing. +6. Keep tool scheduling in the agent loop without directly depending on either SDK's block classes. +7. Test text streaming, fragmented tool arguments, multiple tool calls, usage/cost, reasoning and cache tokens, early disconnects, and API errors. + +The purpose of this abstraction is to unify the **semantics visible to core**, not to erase all provider-specific functionality. Provider extensions can remain in Source Records or a namespaced `extra` field on Native Events. + +## 8. Recommended architecture decision + +Adopt the following decisions when implementing a model-independent protocol: + +1. **Default OpenRouter protocol:** `/api/v1/chat/completions`. +2. **Primary OpenRouter cost source:** `usage.cost` from the response or final SSE event. +3. **Cost fallback:** generation ID plus `total_cost` from `/api/v1/generation`. +4. **Capability constraints:** pair required capabilities such as tool calling and structured outputs with `provider.require_parameters: true`. +5. **Compatibility path:** retain Anthropic Messages as a separate transport; it no longer represents core's internal message model. +6. **Peer candidate:** do not make Responses API the default transport yet; add an adapter when nano needs richer item/event semantics, web search, or Responses SDK compatibility. +7. **Storage boundary:** wire responses are Source Records; Native Events, Journal Entries, and the Event Journal remain nano's own runtime-fact layer. +8. **Export boundary:** continue projecting ATIF from the Event Journal rather than constructing it directly from any provider response. + +## 9. Validation scope of this research + +The protocol conclusions come from OpenRouter's official API reference, feature guides, and usage-accounting documentation as of 2026-08-23, combined with static integration analysis of the current nano code. This research did not make an online request that would incur OpenRouter charges, so the example IDs, token counts, and amounts are illustrative rather than actual billing records from this project's account. + +That limitation does not affect the protocol fields or architecture decision, but the implementation should add an opt-in live integration test using the least expensive available model to verify, for the current account: + +- non-streaming `usage.cost`; +- the terminal usage event from streaming; +- reconciliation between `X-Generation-Id` and the Generation API; +- tool-call delta assembly; +- consistency among the response model, provider, and cost after fallback. diff --git a/docs/research/zh-CN/agent_events_to_atif_examples.md b/docs/research/zh-CN/agent_events_to_atif_examples.md new file mode 100644 index 0000000..079ae08 --- /dev/null +++ b/docs/research/zh-CN/agent_events_to_atif_examples.md @@ -0,0 +1,728 @@ +# Pi、Claude Code、Codex、OpenCode 与 Grok 的 Source Record、内部事件与 ATIF 如何对应 + +> 本文件为**中文源文件**（source of truth）；英文版 [`../en/agent_events_to_atif_examples.md`](../en/agent_events_to_atif_examples.md) 由其生成。 + +调研时间：2026-08-23。 + +## 问题 + +> 能不能给一些真实的例子，比如 Pi、Codex、Claude Code、OpenCode、Grok 的 agent 内部事件长什么样，对应的 ATIF 是什么？ +> +> 这些 native event 示例为什么没有时间戳？ + +## 结论先行 + +可以，但“真实”和“内部”都需要先限定含义。本文分两层证据：前面的“本机实测补充”保留 Pi、Codex、Claude Code 和 OpenCode 四次真实运行 stdout 的 ID、token 与时间；这些记录是 CLI/SDK 边界可见的 **Source Record**，不能一律视为 agent core 的原始内部事件。后面的五组横向示例依据真实类型定义、转换器和测试做 **schema-faithful 重建**，其中任务内容、ID、token 数和时间值是便于比较的示意值。Grok 因本机未安装，只提供源码重建示例。 + +五个项目的共同规律是：**agent loop 先产生自身的 message/event/state，CLI 或 SDK 再将其投影成 Source Record；ATIF 是之后整理出的评测视图，并不是 loop 内部直接运行的对象模型。** 差异主要在三个地方： + +1. Source Record 按模型消息、工具状态还是 CLI item 组织； +2. token/cost 是逐次模型调用提供，还是只在 run/turn 末尾聚合； +3. 原始时间是 Source Record 自带、藏在 payload 中，还是只能由接收器补采。 + +## 概念与转换关系 + +本文采用根目录 [`CONTEXT.md`](../../../CONTEXT.md) 中的术语： + +| 概念 | 定义 | 不应混称为 | +|---|---|---| +| **Source Record** | adapter 在第三方 CLI、SDK 或协议边界实际观察到的 agent-specific 记录；它可能已经是内部状态的裁剪投影 | nano Native Event、Journal Entry | +| **Native Event** | nano core 直接产生，或 adapter 从 Source Record 归一化得到的 agent-independent 运行事实 | 第三方 stdout 原始行、ATIF step | +| **Journal Entry** | Native Event 加上持久化所需的身份、顺序和记录时间元数据 | “持久化 envelope”这个独立产物、trajectory step | +| **Event Journal** | 一个 Agent Run 的 Journal Entry 按追加顺序组成的内部事实日志 | Public Event Stream、Trajectory、Session | +| **ATIF Trajectory** | Event Journal 投影出的公开评测/分析完成态 | Event Journal、native JSONL | + +对第三方 agent，完整边界是： + +```text +第三方内部状态/事件 + ↓ 第三方 CLI/SDK projector +Source Record + ↓ nano adapter 归一化 +Native Event + ↓ journal writer 添加 run_id / seq / recorded_at +Journal Entry + ↓ append-only 持久化（可采用 JSONL 编码） +Event Journal + ↓ ATIF projector +ATIF Trajectory +``` + +nano 自己的 core 可以直接产生 Native Event，因此跳过前两步。这里的 JSONL 只是一种“每行一个 Journal Entry”的物理编码，append-only 只是写入策略；两者都不是新的 trajectory 语义模型。本文此前使用的“持久化 envelope”也不是独立领域概念，统一称为 **Journal Entry**。 + +实测 stdout 与这些概念的对应关系如下： + +| Agent | 本文抓到的 Source Record | 与内部事件的距离 | 是 Journal Entry 吗 | +|---|---|---|---| +| Pi | `--mode json` 的 session/message/tool 行 | 大部分 event 行是 `AgentEvent` 的直接公开序列化；session header 是 CLI 协议记录 | 不是；另有未展示的持久化 harness entry 更接近 Journal Entry | +| Claude Code | SDK `stream-json` 的 system/assistant/user/result message | 已是 SDK 公共协议，不能假定等同内部 event bus | 不是；实测还关闭了 session persistence | +| Codex | `exec --json` 的 `ThreadEvent` | 是更底层 protocol 的裁剪投影 | 不是；本机 rollout JSONL 是另一种持久化产物，不是本文 stdout 来源 | +| OpenCode | `run --format json` 的 step/tool/text 行 | 由内部 `message.part.updated` 再投影一次 | 不是；CLI 顶层 timestamp 不等于 journal 顺序元数据 | +| Grok | `streaming-json` 的 `AcpLine` | 由 ACP update 归约而来 | 不是；只有 NDJSON 行序，没有 Journal Entry envelope | + +时间戳尤其不能从简化示例反推原始协议。实际情况是： + +| Agent | Source Record 的统一时间 | 其他可用时间 | 本文建议的 ATIF 时间来源 | +|---|---|---|---| +| Pi | `AgentEvent` 没有 | session header 和 `AgentMessage.timestamp` 有；持久化 harness entry 另有 storage timestamp | 优先 message timestamp；工具事件由采集器补 receive time | +| Claude Code | SDK stream 没有统一时间 | 本机 2.1.237 的 `assistant`/`user` message 有 ISO timestamp；`system`/`result` 没有，`result` 有 duration | 有 message timestamp 时采用，否则补 receive time | +| Codex `exec --json` | `ThreadEvent` 没有 | 更底层 protocol 的部分对象有 start/end，但公开 `ThreadEvent` 未保留 | 由 JSONL 接收器补 receive time | +| OpenCode | 有 | `message.part.updated.properties.time`，message/tool state 也有 create/start/end | 直接使用原生毫秒时间 | +| Grok `streaming-json` | 没有 | terminal event 有 usage，内部持久化层另有时序信息，但公开 reducer 未输出 | 由 JSONL 接收器补 receive time | + +因此 nanoPyCodeAgent 的 Journal Entry 不应照抄任何一家公开流的缺口。每条 Journal Entry 至少应有严格递增的 `seq` 和 UTC `recorded_at`：前者是 run 内排序的权威，后者表示 nano 接受/记录事实的时间。若 Source Record 另有可信时间，则在 Native Event 中保留为 `source_timestamp`；墙上时钟可能回拨或让并发事件得到相同时间，不能只靠任一种 timestamp 排序。 + +## 调研范围 + +本文以仓库 `references/` 下的固定快照为依据： + +- Pi：`c49906ec77788625aacbdc53ebca6fbe65bd20f5` +- Codex：`4f39251a010a8bd7d692d25fb33832ff06f1635a` +- Claude Code 非官方源码快照：`a371abbe75ffa0d0a3c92290e2bbf56a7ef54367` +- OpenCode：`e00890c67261a435cee6409366a68999a93393fd` +- Grok Build：`19d42e35c07a9c9244f03f6df0c4c353f970d4f9` +- Harbor：[`benchmarks/harbor/pyproject.toml`](../../../benchmarks/harbor/pyproject.toml) 固定的 0.21.0，其 trajectory schema 为 ATIF-v1.7 + +### 本机实测补充 + +在源码调研之外，本文还于 2026-08-23 实际运行了本机安装的四个 CLI： + +| CLI | 版本 | 结果 | +|---|---:|---| +| Pi | 0.84.2 | 成功，`openai-codex/gpt-5.6-sol`，两次模型响应和一次 `read` | +| Codex | 0.149.0 | 成功，两次 command item 状态和一个 agent message | +| Claude Code | 2.1.237 | 成功，两次 Claude Opus message、一次 `Read`，另有辅助 Haiku usage | +| OpenCode | 1.18.21 | 成功，两组 step 和一次 `read` | + +为避免把仓库内容发送给外部模型，四次测试均在 `/tmp` 的临时目录中，只读取一个专门创建的无敏感文件。下面的准备命令会创建新的随机目录；后续命令中的 `$probe_dir` 都指向它： + +```bash +probe_dir="$(mktemp -d /tmp/nanopy-agent-event-probe.XXXXXX)" +printf 'NANOPY_EVENT_PROBE_20260823\n' > "$probe_dir/probe.txt" +cd "$probe_dir" +``` + +本次成功抓包实际使用的命令如下。重新运行会生成新的 session/message ID、token 数和 timestamp，这是正常现象。 + +Pi： + +```bash +pi --mode json --print --no-session \ + --tools read \ + --no-extensions \ + --no-skills \ + --no-prompt-templates \ + --no-themes \ + --no-context-files \ + --approve \ + "Use the read tool exactly once to read probe.txt. Do not write or modify anything. Then reply with only the exact file content." +``` + +Codex： + +```bash +codex exec --json \ + --sandbox read-only \ + --ephemeral \ + --skip-git-repo-check \ + --ignore-user-config \ + "Read probe.txt using a shell command exactly once. Do not write or modify anything. Then reply with only the exact file content." +``` + +Claude Code： + +```bash +claude -p \ + --output-format stream-json \ + --verbose \ + --no-session-persistence \ + --safe-mode \ + --tools Read \ + --permission-mode dontAsk \ + "Use the Read tool exactly once to read probe.txt. Do not write or modify anything. Then reply with only the exact file content." +``` + +OpenCode；本机安装路径由 `.zshrc` 加入 `PATH`，因此按实测过程先加载它： + +```bash +source ~/.zshrc +opencode run \ + --format json \ + --pure \ + --auto \ + --dir "$probe_dir" \ + "Use the read tool exactly once to read probe.txt. Do not write or modify anything. Then reply with only the exact file content." +``` + +下面是从真实 stdout 中抽取的关键记录。四组片段都只选择与结构比较有关的事件或字段：Pi 与 Claude Code 的完整行含大量 encrypted reasoning、初始化信息和插件清单，OpenCode 的工具输出和 metadata 也做了压缩。事件名、ID、数值和时间均来自真实运行；这些片段是**真实 stdout 的关键字段投影**，不是未经处理的逐行抓包。 + +#### Pi 0.84.2 + +```jsonl +{"type":"session","version":3,"id":"01a02e3c-e606-7060-baf2-9a1ded866776","timestamp":"2026-08-23T10:48:58.118Z","cwd":"/tmp/nanopy-agent-event-probe.ktOr9V"} +{"type":"message_end","message":{"role":"assistant","model":"gpt-5.6-sol","provider":"openai-codex","responseId":"resp_0f6a0eea8b7401c0016a8ad01c304087d0a882c3eee1686e09","usage":{"input":597,"output":30,"reasoning":10,"totalTokens":627,"cost":{"total":0.0038850000000000004}},"stopReason":"toolUse","timestamp":1787482138161}} +{"type":"tool_execution_start","toolCallId":"call_j2gAtuUWhalLvXqYLKLke6Yk|fc_0f6a0eea8b7401c0016a8ad01e705c87d0a0c8b9b64c05762e","toolName":"read","args":{"path":"probe.txt"}} +{"type":"tool_execution_end","toolCallId":"call_j2gAtuUWhalLvXqYLKLke6Yk|fc_0f6a0eea8b7401c0016a8ad01e705c87d0a0c8b9b64c05762e","toolName":"read","result":{"content":[{"type":"text","text":"NANOPY_EVENT_PROBE_20260823\n"}]},"isError":false} +{"type":"message_end","message":{"role":"assistant","model":"gpt-5.6-sol","provider":"openai-codex","responseId":"resp_0f6a0eea8b7401c0016a8ad01f3c5c87d0910ad017f42f5588","content":[{"type":"text","text":"NANOPY_EVENT_PROBE_20260823"}],"usage":{"input":648,"output":14,"reasoning":0,"totalTokens":662,"cost":{"total":0.00366}},"stopReason":"stop","timestamp":1787482143410}} +``` + +它直接验证了：tool lifecycle event 没 timestamp，assistant/toolResult payload 有 timestamp；逐 response usage 很完整。Pi 源码的 [`calculateCost`](../../../references/pi/packages/ai/src/models.ts) 使用模型目录费率乘 tokens，所以 `usage.cost` 是 agent 计算值，不能自动视为 provider 实扣。 + +#### Codex 0.149.0 + +```jsonl +{"type":"thread.started","thread_id":"01a02e3d-3743-7ac1-8c30-6c7254135573"} +{"type":"turn.started"} +{"type":"item.started","item":{"id":"item_0","type":"command_execution","command":"/usr/bin/zsh -lc 'cat probe.txt'","aggregated_output":"","exit_code":null,"status":"in_progress"}} +{"type":"item.completed","item":{"id":"item_0","type":"command_execution","command":"/usr/bin/zsh -lc 'cat probe.txt'","aggregated_output":"NANOPY_EVENT_PROBE_20260823\n","exit_code":0,"status":"completed"}} +{"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"NANOPY_EVENT_PROBE_20260823"}} +{"type":"turn.completed","usage":{"input_tokens":31003,"cached_input_tokens":19968,"cache_write_input_tokens":0,"output_tokens":135,"reasoning_output_tokens":60}} +``` + +该输出完整印证了两个缺口：没有任何 timestamp，也没有 cost；usage 只在 `turn.completed` 聚合。 + +#### Claude Code 2.1.237 + +```jsonl +{"type":"system","subtype":"api_retry","attempt":1,"max_retries":10,"retry_delay_ms":524,"error_status":401,"error":"authentication_failed","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"assistant","message":{"model":"claude-opus-5","id":"msg_011CeKYuCnCMFvi24JSsJLvS","content":[{"type":"tool_use","id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","name":"Read","input":{"file_path":"/tmp/nanopy-agent-event-probe.ktOr9V/probe.txt"}}],"usage":{"input_tokens":2,"cache_creation_input_tokens":2521,"cache_read_input_tokens":1247,"output_tokens":14}},"timestamp":"2026-08-23T10:50:01.254Z","request_id":"req_011CeKYuC24Db1xnVkk2y6Ai","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"user","message":{"role":"user","content":[{"tool_use_id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","type":"tool_result","content":"1\tNANOPY_EVENT_PROBE_20260823\n2\t"}]},"timestamp":"2026-08-23T10:50:01.307Z","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"assistant","message":{"model":"claude-opus-5","id":"msg_011CeKYuMzLUNjC8x7bv2GJD","content":[{"type":"text","text":"NANOPY_EVENT_PROBE_20260823"}],"usage":{"input_tokens":2,"cache_creation_input_tokens":130,"cache_read_input_tokens":3768,"output_tokens":1}},"timestamp":"2026-08-23T10:50:02.486Z","request_id":"req_011CeKYuMBxrDUiGKVmhMFzJ","session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +{"type":"result","subtype":"success","num_turns":2,"total_cost_usd":0.032467499999999996,"modelUsage":{"claude-haiku-4-5-20251001":{"inputTokens":920,"outputTokens":12,"costUSD":0.00098},"claude-opus-5[1m]":{"inputTokens":4,"outputTokens":98,"cacheReadInputTokens":5015,"cacheCreationInputTokens":2651,"costUSD":0.031487499999999995}},"result":"NANOPY_EVENT_PROBE_20260823","duration_ms":7920,"session_id":"9408bea7-c831-4edd-9a00-e7475e63fb4b"} +``` + +这纠正了固定源码快照与本机新版本之间的差异：2.1.237 的 assistant/user message 已带 timestamp 和 request ID；但 `system.api_retry`、`result` 仍没有统一 timestamp。`result.total_cost_usd` 还包含可见 Opus 回合之外的 Haiku 辅助调用，因此不能只对可见 assistant message 做加法来重建 run 总额。 + +#### OpenCode 1.18.21 + +```jsonl +{"type":"step_start","timestamp":1787482258116,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"messageID":"msg_02e3eaf250017z8BF8zCT2CQ3f","type":"step-start"}} +{"type":"tool_use","timestamp":1787482259124,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"type":"tool","tool":"read","callID":"call_Kap3tANM5Uu9U5raX0ogDuYu","state":{"status":"completed","input":{"filePath":"/tmp/nanopy-agent-event-probe.ktOr9V/probe.txt"},"output":"NANOPY_EVENT_PROBE_20260823","time":{"start":1787482259101,"end":1787482259121}},"messageID":"msg_02e3eaf250017z8BF8zCT2CQ3f"}} +{"type":"step_finish","timestamp":1787482259168,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"reason":"tool-calls","messageID":"msg_02e3eaf250017z8BF8zCT2CQ3f","type":"step-finish","tokens":{"total":8011,"input":7952,"output":35,"reasoning":24,"cache":{"write":0,"read":0}},"cost":0}} +{"type":"step_start","timestamp":1787482261701,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"messageID":"msg_02e3ebedc001joGNk0E3u0NSp7","type":"step-start"}} +{"type":"text","timestamp":1787482261701,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"messageID":"msg_02e3ebedc001joGNk0E3u0NSp7","type":"text","text":"NANOPY_EVENT_PROBE_20260823","time":{"start":1787482261684,"end":1787482261691}}} +{"type":"step_finish","timestamp":1787482261701,"sessionID":"ses_fd1c15233ffeiwqf92JsdPfahU","part":{"reason":"stop","messageID":"msg_02e3ebedc001joGNk0E3u0NSp7","type":"step-finish","tokens":{"total":8113,"input":397,"output":16,"reasoning":20,"cache":{"write":0,"read":7680}},"cost":0}} +``` + +OpenCode CLI 并没有把 SDK 内部的 `message.part.updated` 原样打印，而是在 [`run.ts`](../../../references/opencode/packages/opencode/src/cli/cmd/run.ts) 中投影成上述 `step_start/tool_use/step_finish/text`，并在每行用 `Date.now()` 添加顶层 timestamp。本次 `cost: 0` 只能解释为 OpenCode 对当前模型给出的 agent-reported value；没有 provider 账单来源时，不能进一步断言该调用真实免费。 + +### 实测事件怎样投影 ATIF + +四次实测的映射策略如下： + +| CLI | ATIF step 边界 | timestamp | step metrics | run cost | +|---|---|---|---|---| +| Pi | 每个 completed assistant message | message Unix ms | 逐 response tokens；本地计算 cost 放 `metrics.extra.estimated_cost_usd` | 不作为真实 `total_cost_usd` | +| Codex | command item step + final message step | receiver 补采 | turn 聚合只能进 `final_metrics` | 缺失 | +| Claude Code | 每个 assistant `message.id` | assistant timestamp | 可见 message usage；prompt tokens 需加 input + cache create + cache read | `result.total_cost_usd` 可作为 agent-reported run cost，并注明含辅助调用 | +| OpenCode | 每个 `messageID`，由 `step_finish` 封口 | CLI 顶层 timestamp | tokens 逐 step；cache 是 prompt 子集 | cost 标明 agent-reported，不能冒充 provider-reported | + +以实测 Claude Code 为例，对应的关键 ATIF 不是简单复制 `input_tokens`：ATIF `prompt_tokens` 定义包含 cached 与 non-cached，所以第一次可见 Opus step 是 `2 + 2521 + 1247 = 3770`，第二次是 `2 + 130 + 3768 = 3900`： + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T10:50:01.254Z", + "source": "agent", + "model_name": "claude-opus-5", + "message": "", + "tool_calls": [{"tool_call_id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","function_name":"Read","arguments":{"file_path":"/tmp/nanopy-agent-event-probe.ktOr9V/probe.txt"}}], + "observation": {"results":[{"source_call_id":"toolu_01BdirHMfK8eKMQgjDn2z7KA","content":"NANOPY_EVENT_PROBE_20260823"}]}, + "metrics": {"prompt_tokens":3770,"completion_tokens":14,"cached_tokens":1247}, + "llm_call_count": 1 + }, + { + "step_id": 3, + "timestamp": "2026-08-23T10:50:02.486Z", + "source": "agent", + "model_name": "claude-opus-5", + "message": "NANOPY_EVENT_PROBE_20260823", + "metrics": {"prompt_tokens":3900,"completion_tokens":1,"cached_tokens":3768}, + "llm_call_count": 1 + } + ], + "final_metrics": { + "total_prompt_tokens": 8590, + "total_completion_tokens": 110, + "total_cached_tokens": 5015, + "total_cost_usd": 0.032467499999999996, + "extra": {"cost_source":"claude_code_result","includes_auxiliary_model_calls":true,"auxiliary_model":"claude-haiku-4-5-20251001"} + } +} +``` + +这里 `final_metrics` 按 `modelUsage` 汇总了 Opus 与辅助 Haiku，故大于两个可见 step 的 metrics 总和。该差异必须保留说明，不能为了让数字相等而丢掉隐藏调用或把 Haiku tokens 硬分摊给某个可见 step。 + +为便于比较，五组例子都采用同一个任务： + +```text +读取 src/main.rs，然后总结它的作用。 +``` + +下面的 ATIF 片段只展开 `steps` 中与映射有关的字段。完整文档还需要根级 `schema_version`、`agent` 等字段。ATIF-v1.7 的 `Step.timestamp` 是可选 ISO 8601 字符串；本文仍建议 nano 在有可靠来源时总是填写，并在 `extra.timestamp_source` 标明来源。 + +--- + +## 一、Pi：围绕 turn、message 和 tool execution 发事件 + +Pi 的真实 `AgentEvent` 联合类型见 [`packages/agent/src/types.ts`](../../../references/pi/packages/agent/src/types.ts)，JSONL 说明见 [`packages/coding-agent/docs/json.md`](../../../references/pi/packages/coding-agent/docs/json.md)。事件类型包括： + +```typescript +type AgentEvent = + | { type: "agent_start" } + | { type: "agent_end"; messages: AgentMessage[] } + | { type: "turn_start" } + | { type: "turn_end"; message: AgentMessage; toolResults: ToolResultMessage[] } + | { type: "message_start"; message: AgentMessage } + | { type: "message_update"; message: AgentMessage; assistantMessageEvent: AssistantMessageEvent } + | { type: "message_end"; message: AgentMessage } + | { type: "tool_execution_start"; toolCallId: string; toolName: string; args: any } + | { type: "tool_execution_update"; toolCallId: string; toolName: string; args: any; partialResult: any } + | { type: "tool_execution_end"; toolCallId: string; toolName: string; result: any; isError: boolean }; +``` + +### 1.1 Pi AgentEvent 的公开序列化长什么样 + +下面省略文本 delta，只保留一次工具调用和两次模型响应的权威完成事件： + +```jsonl +{"type":"session","version":3,"id":"pi-session-1","timestamp":"2026-08-23T08:00:00.000Z","cwd":"/workspace"} +{"type":"agent_start"} +{"type":"turn_start"} +{"type":"message_start","message":{"role":"user","content":"读取 src/main.rs，然后总结它的作用。","timestamp":1787472000100}} +{"type":"message_end","message":{"role":"user","content":"读取 src/main.rs，然后总结它的作用。","timestamp":1787472000100}} +{"type":"tool_execution_start","toolCallId":"call_read_1","toolName":"read","args":{"path":"src/main.rs"}} +{"type":"tool_execution_end","toolCallId":"call_read_1","toolName":"read","result":{"content":"fn main() { println!(\"hello\"); }"},"isError":false} +{"type":"turn_end","message":{"role":"assistant","content":[{"type":"text","text":"我先读取入口文件。"},{"type":"toolCall","id":"call_read_1","name":"read","arguments":{"path":"src/main.rs"}}],"api":"anthropic-messages","provider":"anthropic","model":"claude-sonnet-4","usage":{"input":120,"output":24,"cacheRead":0,"cacheWrite":0,"totalTokens":144,"cost":{"input":0.00036,"output":0.00036,"cacheRead":0,"cacheWrite":0,"total":0.00072}},"stopReason":"toolUse","timestamp":1787472001250},"toolResults":[{"role":"toolResult","toolCallId":"call_read_1","toolName":"read","content":[{"type":"text","text":"fn main() { println!(\"hello\"); }"}],"isError":false,"timestamp":1787472001420}]} +{"type":"turn_start"} +{"type":"turn_end","message":{"role":"assistant","content":[{"type":"text","text":"这是一个最小 Rust 程序，main 函数向标准输出打印 hello。"}],"api":"anthropic-messages","provider":"anthropic","model":"claude-sonnet-4","usage":{"input":168,"output":31,"cacheRead":0,"cacheWrite":0,"totalTokens":199,"cost":{"input":0.000504,"output":0.000465,"cacheRead":0,"cacheWrite":0,"total":0.000969}},"stopReason":"stop","timestamp":1787472002100},"toolResults":[]} +``` + +这里的时间语义是混合的： + +- session header 有 ISO timestamp； +- `turn_start`、`tool_execution_start/end` 的 `AgentEvent` envelope 没有 timestamp； +- `UserMessage`、`AssistantMessage` 和 `ToolResultMessage` 的真实类型有 Unix 毫秒 `timestamp`，见 [`packages/ai/src/types.ts`](../../../references/pi/packages/ai/src/types.ts)； +- Pi 新的持久化 harness entry 还有 storage-assigned timestamp，但那是另一层 journal，不是上述 Source Record。 + +### 1.2 对应 ATIF + +```json +{ + "steps": [ + { + "step_id": 1, + "timestamp": "2026-08-23T08:00:00.100Z", + "source": "user", + "message": "读取 src/main.rs，然后总结它的作用。", + "extra": {"timestamp_source": "user_message"} + }, + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.250Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "我先读取入口文件。", + "tool_calls": [ + { + "tool_call_id": "call_read_1", + "function_name": "read", + "arguments": {"path": "src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "call_read_1", + "content": "fn main() { println!(\"hello\"); }", + "extra": {"is_error": false} + } + ] + }, + "metrics": { + "prompt_tokens": 120, + "completion_tokens": 24, + "cached_tokens": 0, + "extra": {"estimated_cost_usd": 0.00072, "cost_source": "pi_model_catalog"} + }, + "llm_call_count": 1, + "extra": {"timestamp_source": "assistant_message"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.100Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "这是一个最小 Rust 程序，main 函数向标准输出打印 hello。", + "metrics": { + "prompt_tokens": 168, + "completion_tokens": 31, + "cached_tokens": 0, + "extra": {"estimated_cost_usd": 0.000969, "cost_source": "pi_model_catalog"} + }, + "llm_call_count": 1, + "extra": {"timestamp_source": "assistant_message"} + } + ] +} +``` + +Pi 是五者中最接近逐模型调用无损映射的一种：`turn_end.message` 已经有模型、内容、tool call、usage 和 timestamp，`toolResults` 又保留调用关联。其 cost 还需标注来源：当前主流 provider adapter 调用模型目录费率计算，结构完整不等于 provider 实扣。 + +--- + +## 二、Claude Code：SDK message stream 需要按 message ID 和 tool_use ID 归并 + +Claude Code 的 SDK 输出类型见 [`src/entrypoints/sdk/coreSchemas.ts`](../../../references/claude-code/src/entrypoints/sdk/coreSchemas.ts)。Harbor 的 Claude Code adapter 使用 `--output-format=stream-json --print`，再将多行 SDK message 整理为 ATIF。 + +### 2.1 Claude Code SDK Source Record 长什么样 + +```jsonl +{"type":"system","subtype":"init","cwd":"/workspace","tools":["Read"],"model":"claude-sonnet-4","permissionMode":"bypassPermissions","uuid":"sys-1","session_id":"claude-session-1","apiKeySource":"ANTHROPIC_API_KEY","mcp_servers":[],"slash_commands":[],"output_style":"default","skills":[],"plugins":[],"claude_code_version":"2.x"} +{"type":"assistant","message":{"id":"msg_tool_1","type":"message","role":"assistant","model":"claude-sonnet-4","content":[{"type":"text","text":"我先读取入口文件。"},{"type":"tool_use","id":"toolu_read_1","name":"Read","input":{"file_path":"/workspace/src/main.rs"}}],"stop_reason":"tool_use","stop_sequence":null,"usage":{"input_tokens":120,"output_tokens":24,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}},"parent_tool_use_id":null,"uuid":"assistant-1","session_id":"claude-session-1"} +{"type":"user","message":{"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_read_1","content":"fn main() { println!(\"hello\"); }","is_error":false}]},"parent_tool_use_id":null,"tool_use_result":{"type":"text","file":{"filePath":"/workspace/src/main.rs","content":"fn main() { println!(\"hello\"); }"}},"timestamp":"2026-08-23T08:00:01.420Z","uuid":"user-tool-result-1","session_id":"claude-session-1"} +{"type":"assistant","message":{"id":"msg_final_1","type":"message","role":"assistant","model":"claude-sonnet-4","content":[{"type":"text","text":"这是一个最小 Rust 程序，main 函数向标准输出打印 hello。"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":168,"output_tokens":31,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}},"parent_tool_use_id":null,"uuid":"assistant-2","session_id":"claude-session-1"} +{"type":"result","subtype":"success","duration_ms":2100,"duration_api_ms":1700,"is_error":false,"num_turns":2,"result":"这是一个最小 Rust 程序，main 函数向标准输出打印 hello。","stop_reason":"end_turn","total_cost_usd":0.001689,"usage":{"input_tokens":288,"output_tokens":55},"modelUsage":{"claude-sonnet-4":{"inputTokens":288,"outputTokens":55,"cacheReadInputTokens":0,"cacheCreationInputTokens":0,"costUSD":0.001689,"contextWindow":200000,"maxOutputTokens":64000}},"permission_denials":[],"uuid":"result-1","session_id":"claude-session-1"} +``` + +SDK stream 没有所有事件统一具备的 timestamp。本机 2.1.237 的 `assistant` 和 `user` message 已有 ISO timestamp，但 `system`、`api_retry` 和 `result` 仍可缺失；固定源码快照中的 `user` timestamp 还是可选字段，并明确要求旧 emitter 缺失时消费者回退到接收时间。 + +### 2.2 对应 ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.250Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "我先读取入口文件。", + "tool_calls": [ + { + "tool_call_id": "toolu_read_1", + "function_name": "Read", + "arguments": {"file_path": "/workspace/src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "toolu_read_1", + "content": "fn main() { println!(\"hello\"); }", + "extra": {"is_error": false} + } + ] + }, + "metrics": {"prompt_tokens": 120, "completion_tokens": 24, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg_tool_1", "timestamp_source": "assistant_message_or_receiver"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.100Z", + "source": "agent", + "model_name": "claude-sonnet-4", + "message": "这是一个最小 Rust 程序，main 函数向标准输出打印 hello。", + "metrics": {"prompt_tokens": 168, "completion_tokens": 31, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg_final_1", "timestamp_source": "assistant_message_or_receiver"} + } + ], + "final_metrics": { + "total_prompt_tokens": 288, + "total_completion_tokens": 55, + "total_cached_tokens": 0, + "total_cost_usd": 0.001689, + "total_steps": 3, + "extra": {"cost_source": "claude_code_result", "cost_kind": "agent_reported"} + } +} +``` + +映射时有两个关键动作： + +1. 以 assistant `message.id` 形成一次模型 step； +2. 以 `tool_result.tool_use_id` 找回此前 `tool_use.id`，把环境结果附到相同 ATIF step 的 `observation`。 + +`result.total_cost_usd` 是 Claude Code 报告的整个 run 汇总，可能包含辅助模型调用，且不是 Anthropic Messages 响应直接回传的账单字段；不能凭比例伪分摊到两个可见 step。因此上例只写 `final_metrics.total_cost_usd` 并记录来源，不填写逐 step 的 `metrics.cost_usd`。 + +--- + +## 三、Codex：公开 JSONL 是 thread/turn/item 投影，usage 只在 turn 末汇总 + +Codex `exec --json` 的真实事件定义见 [`codex-rs/exec/src/exec_events.rs`](../../../references/codex/codex-rs/exec/src/exec_events.rs)。顶层是 `ThreadEvent`：`thread.started`、`turn.started`、`item.started/updated/completed`、`turn.completed/failed` 和 `error`。 + +### 3.1 Codex ThreadEvent Source Record 长什么样 + +```jsonl +{"type":"thread.started","thread_id":"codex-thread-1"} +{"type":"turn.started"} +{"type":"item.started","item":{"id":"item_cmd_1","type":"command_execution","command":"sed -n '1,200p' src/main.rs","aggregated_output":"","exit_code":null,"status":"in_progress"}} +{"type":"item.completed","item":{"id":"item_cmd_1","type":"command_execution","command":"sed -n '1,200p' src/main.rs","aggregated_output":"fn main() { println!(\"hello\"); }\n","exit_code":0,"status":"completed"}} +{"type":"item.completed","item":{"id":"item_msg_1","type":"agent_message","text":"这是一个最小 Rust 程序，main 函数向标准输出打印 hello。"}} +{"type":"turn.completed","usage":{"input_tokens":288,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":55,"reasoning_output_tokens":12}} +``` + +这些公开 `ThreadEvent` 本身没有 timestamp。Codex 更底层 protocol 的某些 begin/end 对象有 `started_at_ms`、`completed_at_ms`，但 `exec --json` 的规范化事件把它们裁掉了。若 Harbor 只消费 stdout JSONL，就只能在读取每行时补 receive time。 + +### 3.2 对应 ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.200Z", + "source": "agent", + "message": "", + "tool_calls": [ + { + "tool_call_id": "item_cmd_1", + "function_name": "command_execution", + "arguments": {"command": "sed -n '1,200p' src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "item_cmd_1", + "content": "fn main() { println!(\"hello\"); }\n", + "extra": {"exit_code": 0, "status": "completed"} + } + ] + }, + "llm_call_count": 1, + "extra": {"timestamp_source": "receiver", "usage_attribution": "unavailable"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.050Z", + "source": "agent", + "message": "这是一个最小 Rust 程序，main 函数向标准输出打印 hello。", + "llm_call_count": 1, + "extra": {"source_item_id": "item_msg_1", "timestamp_source": "receiver", "usage_attribution": "unavailable"} + } + ], + "final_metrics": { + "total_prompt_tokens": 288, + "total_completion_tokens": 55, + "total_cached_tokens": 0, + "total_steps": 3, + "extra": {"reasoning_output_tokens": 12} + } +} +``` + +这里发生了真实的信息损失：`turn.completed.usage` 覆盖整个用户 turn，其中可能有多次模型推理；公开流没有逐 inference usage，不能可靠地分给 ATIF step 2 和 step 3。正确做法是只填 `final_metrics`，并显式说明 attribution unavailable，而不是平均拆分。 + +--- + +## 四、OpenCode：事件更新的是 message part，工具本身是状态机 + +OpenCode 当前 V2 SDK 类型见 [`packages/sdk/js/src/v2/gen/types.gen.ts`](../../../references/opencode/packages/sdk/js/src/v2/gen/types.gen.ts)，CLI 对这些事件的投影见 [`packages/opencode/src/cli/cmd/run.ts`](../../../references/opencode/packages/opencode/src/cli/cmd/run.ts)。内部核心事件 `message.part.updated` 也通过 V2 SDK 事件协议暴露，因此它既是内部事件类型，也是 SDK 边界可观察的 Source Record；part 可以是 text、tool、step-start、step-finish 等。`opencode run --format json` 又把它投影为 `step_start/tool_use/step_finish/text` 并加顶层 timestamp。 + +### 4.1 OpenCode SDK Source Record（message.part.updated）长什么样 + +```jsonl +{"id":"evt-1","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472001100,"part":{"id":"part-tool-1","sessionID":"oc-session-1","messageID":"msg-1","type":"tool","callID":"call-read-1","tool":"read","state":{"status":"running","input":{"filePath":"src/main.rs"},"title":"Read src/main.rs","metadata":{},"time":{"start":1787472001090}}}}} +{"id":"evt-2","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472001420,"part":{"id":"part-tool-1","sessionID":"oc-session-1","messageID":"msg-1","type":"tool","callID":"call-read-1","tool":"read","state":{"status":"completed","input":{"filePath":"src/main.rs"},"output":"fn main() { println!(\"hello\"); }","title":"Read src/main.rs","metadata":{},"time":{"start":1787472001090,"end":1787472001418}}}}} +{"id":"evt-3","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472001500,"part":{"id":"part-finish-1","sessionID":"oc-session-1","messageID":"msg-1","type":"step-finish","reason":"tool-calls","cost":0.00072,"tokens":{"input":120,"output":24,"reasoning":0,"cache":{"read":0,"write":0}}}}} +{"id":"evt-4","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472002100,"part":{"id":"part-text-2","sessionID":"oc-session-1","messageID":"msg-2","type":"text","text":"这是一个最小 Rust 程序，main 函数向标准输出打印 hello。","time":{"start":1787472001800,"end":1787472002090}}}} +{"id":"evt-5","type":"message.part.updated","properties":{"sessionID":"oc-session-1","time":1787472002120,"part":{"id":"part-finish-2","sessionID":"oc-session-1","messageID":"msg-2","type":"step-finish","reason":"stop","cost":0.000969,"tokens":{"input":168,"output":31,"reasoning":0,"cache":{"read":0,"write":0}}}}} +``` + +此前若把这类示例写成只有 `type` 和 `part`，会错误隐藏一个重要事实：当前 `message.part.updated` 的真实 envelope 有 `properties.time`。此外 assistant message 有 `time.created/completed`，tool state 有 `time.start/end`。 + +### 4.2 对应 ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.500Z", + "source": "agent", + "message": "", + "tool_calls": [ + { + "tool_call_id": "call-read-1", + "function_name": "read", + "arguments": {"filePath": "src/main.rs"} + } + ], + "observation": { + "results": [ + { + "source_call_id": "call-read-1", + "content": "fn main() { println!(\"hello\"); }", + "extra": {"tool_started_at_ms": 1787472001090, "tool_ended_at_ms": 1787472001418} + } + ] + }, + "metrics": {"prompt_tokens": 120, "completion_tokens": 24, "cached_tokens": 0, "extra":{"estimated_cost_usd":0.00072,"cost_source":"opencode_model_catalog"}}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg-1", "timestamp_source": "event"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.120Z", + "source": "agent", + "message": "这是一个最小 Rust 程序，main 函数向标准输出打印 hello。", + "metrics": {"prompt_tokens": 168, "completion_tokens": 31, "cached_tokens": 0, "extra":{"estimated_cost_usd":0.000969,"cost_source":"opencode_model_catalog"}}, + "llm_call_count": 1, + "extra": {"source_message_id": "msg-2", "timestamp_source": "event"} + } + ] +} +``` + +转换器必须以 `messageID` 聚合 part，并以 `callID` 关联工具的 running/completed 状态。`step-finish` 提供逐模型 step 的 tokens 和 cost，所以结构上很容易投影；但 OpenCode 当前 cost 是根据模型价格表和 tokens 计算出的值，不等同于 provider 回传的实际扣费凭据。 + +--- + +## 五、Grok：`streaming-json` 是从 ACP update 归约出的轻量状态流 + +Grok 的真实 `streaming-json` wire 定义见 [`xai-grok-pager/src/headless/reducer/acp.rs`](../../../references/grok-build/crates/codegen/xai-grok-pager/src/headless/reducer/acp.rs)，上游统一事件见同目录 [`mod.rs`](../../../references/grok-build/crates/codegen/xai-grok-pager/src/headless/reducer/mod.rs)。它把 ACP update 归约为 `text`、`thought`、`tool_call`、`tool_call_update`、`usage` 和 terminal `end` 等事件。 + +### 5.1 Grok streaming-json Source Record 长什么样 + +```jsonl +{"type":"thought","data":"I need to inspect the Rust entry point."} +{"type":"tool_call","toolCallId":"grok-call-1","title":"Read src/main.rs","kind":"read","status":"pending","toolName":"read_file","rawInput":{"path":"src/main.rs"},"content":[],"locations":[{"path":"src/main.rs"}]} +{"type":"tool_call_update","toolCallId":"grok-call-1","status":"completed","content":[{"type":"content","content":{"type":"text","text":"fn main() { println!(\"hello\"); }"}}],"rawOutput":{"text":"fn main() { println!(\"hello\"); }"},"locations":[{"path":"src/main.rs"}]} +{"type":"usage","messageId":"grok-msg-1","stopReason":"tool_use","usage":{"inputTokens":120,"outputTokens":24,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}} +{"type":"text","data":"这是一个最小 Rust 程序，main 函数向标准输出打印 hello。"} +{"type":"usage","messageId":"grok-msg-2","stopReason":"end_turn","usage":{"inputTokens":168,"outputTokens":31,"cacheReadInputTokens":0,"cacheCreationInputTokens":0}} +{"type":"end","stopReason":"EndTurn","sessionId":"grok-session-1","requestId":"grok-request-1","usage":{"input_tokens":288,"output_tokens":55,"total_tokens":343,"num_turns":2}} +``` + +`AcpLine` 和 `AcpUsageLine` 都没有 timestamp 字段。事件顺序由 NDJSON 行序给出，terminal `end` 只提供 run 汇总。Grok 内部会维护更多 session/usage 状态，但从 `streaming-json` 接入的第三方不能假定那些内部字段也出现在 stdout。 + +### 5.2 对应 ATIF + +```json +{ + "steps": [ + { + "step_id": 2, + "timestamp": "2026-08-23T08:00:01.500Z", + "source": "agent", + "message": "", + "reasoning_content": "I need to inspect the Rust entry point.", + "tool_calls": [ + { + "tool_call_id": "grok-call-1", + "function_name": "read_file", + "arguments": {"path": "src/main.rs"} + } + ], + "observation": { + "results": [ + {"source_call_id": "grok-call-1", "content": "fn main() { println!(\"hello\"); }"} + ] + }, + "metrics": {"prompt_tokens": 120, "completion_tokens": 24, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "grok-msg-1", "timestamp_source": "receiver"} + }, + { + "step_id": 3, + "timestamp": "2026-08-23T08:00:02.100Z", + "source": "agent", + "message": "这是一个最小 Rust 程序，main 函数向标准输出打印 hello。", + "metrics": {"prompt_tokens": 168, "completion_tokens": 31, "cached_tokens": 0}, + "llm_call_count": 1, + "extra": {"source_message_id": "grok-msg-2", "timestamp_source": "receiver"} + } + ] +} +``` + +Grok 的转换是一个小型状态机：累积 thought/text delta；看到 `tool_call` 建立调用；用同一 `toolCallId` 的 `tool_call_update` 完成 observation；最后用 `usage.messageId` 封口一个模型 step。 + +--- + +## 六、五种 Source Record 与 ATIF 的内容差异 + +从例子可以看到，Source Record 和 ATIF 并非保存完全不同的事实。二者内容高度重叠，差异在组织方式和完成时机；adapter 的职责是先将 Source Record 忠实归一化为 Native Event，而不是直接补造源协议没有提供的信息： + +| 可观测事实 | Pi | Claude Code | Codex | OpenCode | Grok | ATIF 目标 | +|---|---|---|---|---|---|---| +| 模型响应边界 | `turn_end.message` | assistant `message.id` | 未直接暴露，只能从 item/turn 推断 | `messageID` + `step-finish` | `usage.messageId` | 一个 `source: agent` step | +| 工具调用 ID | `toolCallId` | `tool_use.id` | item `id` | `callID` | `toolCallId` | `tool_calls[].tool_call_id` | +| 工具结果 | `toolResults` / execution end | `tool_result` | completed item | completed ToolPart | tool call update | `observation.results[]` | +| 逐调用 token | 有 | 有 | 无，只有 turn 聚合 | 有 | 有 | `step.metrics` | +| 逐调用 cost | 有 agent 计算值，通常非 provider 实扣 | message 无，result 有 agent-reported run 汇总 | 无 | 有 agent 计算值 | 视后端 usage 而定 | 只有来源可靠时写 `step.metrics.cost_usd`，否则进 `extra` | +| 时间 | message 有，event 不统一 | 不统一 | 公开流无 | event/message/tool 均有 | 公开流无 | `step.timestamp`，可选 | + +ATIF 是**完成态快照**：它希望每个 step 已经有完整 message、tool calls、observation 和 metrics。Native Event 则是**按时间发生的事实**：tool 先 started，后 completed；模型文本可能先 delta，后 completed；cost 甚至可能在 run 后异步补齐。Journal Entry 为这些事实增加可靠顺序和记录时间，Event Journal 保存可重放的追加历史。ATIF 与 Event Journal 可以来自同一组事实，但不适合共用一个写入模型。 + +## 七、对 nanoPyCodeAgent 的具体建议 + +### 7.1 Journal Entry 必须提供记录时间和顺序 + +Native Event 不必自行分配持久化顺序。journal writer 接受 Native Event 后，将它包装成统一的 Journal Entry： + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 17, + "recorded_at": "2026-08-23T08:00:01.420Z", + "type": "tool.completed", + "payload": { + "tool_call_id": "call-read-1", + "tool_name": "read", + "result": "fn main() { println!(\"hello\"); }", + "is_error": false, + "duration_ms": 330, + "source_timestamp": null, + "timestamp_source": "receiver" + } +} +``` + +语义约束应是： + +- `schema_version` 描述内部 Journal Entry/Native Event 契约，不是 ATIF schema version； +- `seq` 由 journal writer 分配，严格递增，是 run 内排序的权威； +- `recorded_at` 是 nano 接受/记录事件时的 UTC wall-clock，使用 RFC 3339/ISO 8601，并且每条 Journal Entry 都必须有； +- `source_timestamp` 属于 Native Event payload，仅在 Source Record 提供可信原始时间时填写；缺失时保持 `null`，不能把 `recorded_at` 冒充成源端发生时间； +- 精确耗时使用 `duration_ms` 或成对 start/end 事件，不用两个 wall-clock 相减作为唯一依据； +- ATIF `step.timestamp` 优先采用可信 `source_timestamp`，否则回退到 `recorded_at`，并在 `extra.timestamp_source` 记录来源。 + +### 7.2 Native Event 最小集合应覆盖模型、工具和 run 终态 + +建议至少保留： + +```text +run.started +user.message +model.started +model.completed +tool.started +tool.completed +run.completed +run.failed +``` + +其中 `model.completed` 应保留 `message_id`、完整 content/tool calls、实际 model、stop reason、usage 和 provider response/generation ID。这样 ATIF converter 才能做到逐 inference 映射，避免 Codex 公开 JSONL 那种只能拿到 turn 汇总的损失。 + +### 7.3 内部 Event Journal 与对外 ATIF Trajectory 是不同产物 + +建议保留内部 Event Journal：journal writer 将每个 Journal Entry 追加保存，第一版可以采用“一行一个 Journal Entry”的 JSONL 编码。这个 JSONL 是内部事实日志，不定义 step、observation 或 final metrics，也不由 `--trajectory` 暴露。 + +对外 trajectory 只采用 ATIF-v1.7：ATIF projector 实时消费或事后重放 Event Journal，将多个 Native Event 折叠成完成态 step。`--trajectory PATH` 指向 ATIF 文件；它不指向内部 Event Journal。这样避免的是同时维护“native trajectory”和 ATIF 两套公开 trajectory 语义，而不是消灭所有内部 schema：Native Event/Journal Entry 仍有内部 schema，也仍需维护 Native Event → ATIF 的单向 projector。 + +这是在看到 Source Record 与 ATIF 的对应关系后得到的**后续架构建议**，明确取代 [`agent_output_and_trajectory.md`](agent_output_and_trajectory.md) 第 8.4 节中“`--trajectory` 写 native trajectory JSONL、Harbor adapter 再转 ATIF”的旧建议。旧文档所列 stdout `--output-format` 契约仍然成立；被取代的只是 trajectory 的公开持久格式边界。 + +实现上，Event Journal 可以边运行边追加 Journal Entry；run 结束或需要 checkpoint 时，将当前事实折叠成完整 ATIF，并用临时文件加原子 rename 更新 `--trajectory` 目标。若进程中断，Event Journal 仍保留到最后一条完整 Journal Entry；ATIF 则保持最后一个完整快照。 + +## 最终回答 + +真实项目并没有一种统一的可观测 Source Record：Pi 是接近内部 `AgentEvent` 的 turn/message/tool 记录，Claude Code 是 SDK message，Codex 是 thread item，OpenCode CLI 是 message part 的公开投影，Grok 是 ACP update 的公开归约。adapter 先把这些不同形状归一化为 nano Native Event，journal writer 再生成 Journal Entry；ATIF projector 最后将 Event Journal 折叠成统一的 user/agent step、tool call、observation 和 metrics。 + +此前 Source Record 示例若没有时间戳，不能理解为时间不重要。准确说法是：部分源协议不带统一 timestamp，部分把时间放在 message/tool payload，OpenCode CLI 则在公开记录顶层补时间。nano 的 journal writer 应统一为 Journal Entry 分配 `seq + recorded_at`，同时保留可用的 `source_timestamp`；投影 ATIF 时记录选择了哪种时间，不能依赖 stdout 行序之外的隐含时钟，也不能为了让 ATIF 看起来完整而伪造源端发生时间。 diff --git a/docs/research/zh-CN/agent_output_and_trajectory.md b/docs/research/zh-CN/agent_output_and_trajectory.md index 1e30347..de6a356 100644 --- a/docs/research/zh-CN/agent_output_and_trajectory.md +++ b/docs/research/zh-CN/agent_output_and_trajectory.md @@ -2,6 +2,8 @@ > 本文件为**中文源文件**（source of truth）；英文版 [`../en/agent_output_and_trajectory.md`](../en/agent_output_and_trajectory.md) 由其生成。 +> **后续决策（2026-08-23）：** 本文关于“`--trajectory` 写 native trajectory JSONL，再由 Harbor adapter 转 ATIF”的建议已被 [`agent_events_to_atif_examples.md` 第 7.3 节](agent_events_to_atif_examples.md#73-内部-event-journal-与对外-atif-trajectory-是不同产物) 取代。当前建议是内部保留 append-only Event Journal（可用 JSONL 编码），`--trajectory` 对外只写 ATIF-v1.7；Event Journal 不是另一种 trajectory。本文的 `--output-format` stdout 契约不受影响。 + 调研时间：2026-08-22。 [`benchmark_headless_interface.md`](benchmark_headless_interface.md) 先提出了下面这组接口，但没有说明两个参数控制的是否是同一种产物： diff --git a/docs/research/zh-CN/openrouter_cost_accounting.md b/docs/research/zh-CN/openrouter_cost_accounting.md new file mode 100644 index 0000000..c44b8d8 --- /dev/null +++ b/docs/research/zh-CN/openrouter_cost_accounting.md @@ -0,0 +1,335 @@ +# OpenRouter 的真实 Cost、价格 API 与 Trajectory 记账方案 + +> 本文件为**中文源文件**（source of truth）；英文版 [`../en/openrouter_cost_accounting.md`](../en/openrouter_cost_accounting.md) 由其生成。 + +调研时间：2026-08-23。 + +## 问题 + +> 关于 cost，我用的是 OpenRouter，响应里没有真实 cost 吗？如果没有真实 cost，有 API 获取模型价格吗？ +> +> cost 能和原生存储格式或 trajectory 一起实现吗？ + +## 相关调研 + +OpenRouter 模型无关 API 的协议形状、agent 能力、endpoint 选型和 nano 接入边界，见 [OpenRouter 统一模型协议：能力、Cost 与 nanoPyCodeAgent 接入边界](openrouter_unified_protocol.md)。本文只展开 cost 的来源、补账和 trajectory 映射。 + +## 结论先行 + +OpenRouter **有真实扣费数据**，而且比“token × 当前模型标价”的估算更可靠。OpenRouter Chat Completions 统一 API 会在完整 response 或最后一个 SSE event 中直接返回 `usage.cost`；但 nanoPyCodeAgent 当前走的是 Anthropic Messages 兼容端点，这个端点的 Anthropic 形状 `usage` 没有公开承诺 `cost` 字段，所以 `stream.get_final_message().usage` 通常只能看到 tokens。 + +对 nano 最可靠的实现分为两条路径： + +1. 后续使用 Chat Completions transport 时，直接把 `usage.cost` 作为 resolved、`provider_reported` cost 写入 `model.completed` Native Event； +2. 当前 Anthropic Messages transport 没有直接 cost 时，从 HTTP header 捕获 `X-Generation-Id`，先把 cost 标为 pending； +3. 调用 `GET /api/v1/generation?id=...` 获取该次请求的 `total_cost`，再追加 `model.cost_resolved` Native Event； +4. 两条路径都由 journal writer 持久化为 Journal Entry，再投影到 ATIF `step.metrics.cost_usd`； +5. 所有 OpenRouter 请求都保留 generation ID，供缺失补账和对账； +6. `GET /api/v1/model/:author/:slug` 或 `GET /api/v1/models` 的价格表只作为预算/估算 fallback，不作为历史实际账单。 + +## 一、为什么 OpenRouter 文档说有 cost，nano 响应里却没有 + +OpenRouter 的 [Usage Accounting](https://openrouter.ai/docs/cookbook/administration/usage-accounting) 文档说明，Chat Completions/Responses 的完整响应或最后一个 SSE chunk 会带： + +```json +{ + "usage": { + "prompt_tokens": 194, + "completion_tokens": 2, + "total_tokens": 196, + "cost": 0.00095, + "cost_details": { + "upstream_inference_cost": 0.00090 + } + } +} +``` + +其中： + +- `usage.cost` 是向当前 OpenRouter 账户收取的总额； +- `cost_details.upstream_inference_cost` 是上游 provider 推理成本； +- streaming 时 usage 在最后一个 SSE event，非 streaming 时在完整响应； +- 不再需要旧的 `usage: {include: true}` 或 `stream_options.include_usage` 参数。 + +OpenRouter [FAQ](https://openrouter.ai/docs/faq) 说明 credits 的基础货币是美元，站点和 API 定价也以美元表示；因此普通 credits 请求可以把 provider-reported `usage.cost` 或 `data.total_cost` 映射为 ATIF `cost_usd`。内部事件仍应显式保存 `currency: "USD"` 与 source，不能仅靠目标字段名隐含币种和来源。 + +但 nano 当前代码在 [`src/nanopycodeagent/agent.py`](../../../src/nanopycodeagent/agent.py) 中使用： + +```text +anthropic.Anthropic + -> ANTHROPIC_BASE_URL + -> OpenRouter /api/v1/messages + -> client.messages.stream(...) + -> stream.get_final_message() +``` + +OpenRouter 的 [Anthropic Messages endpoint](https://openrouter.ai/docs/api/api-reference/anthropic-messages/create-messages) 返回 Anthropic 兼容的 `usage.input_tokens`、`usage.output_tokens` 和 cache 字段；该端点的公开 response schema 没有列 `usage.cost`。因此不能把 Chat Completions 的 `usage.cost` 承诺直接套到 Messages skin 上。 + +仓库当前安装的 Anthropic Python SDK 基础模型允许额外字段：如果服务器真的在 `usage` 中附加 `cost`，SDK 不会必然删除它；问题是 OpenRouter Messages 的公开协议没有承诺发送该字段。实现不能依赖一个未声明扩展。 + +### 1.1 本机四个 CLI 的实测旁证 + +同日使用无敏感哨兵文件实测 Pi 0.84.2、Codex 0.149.0、Claude Code 2.1.237 和 OpenCode 1.18.21，观察到： + +| CLI | 输出的 cost | 来源判断 | +|---|---|---| +| Pi | 每个 assistant response 都有 `usage.cost.total` | 源码明确由 `calculateCost(model, usage)` 用模型目录费率计算 | +| Codex | `exec --json` 无 cost | 只有 turn aggregate tokens | +| Claude Code | terminal `result.total_cost_usd` 与 `modelUsage.*.costUSD` | agent-reported 汇总；Anthropic Messages usage 本身没有 cost，且结果可含辅助模型调用 | +| OpenCode | 每个 `step_finish.cost`；本次实际为 `0` | 源码按模型价格表和 tokens 计算；`0` 未必能证明 provider 未扣费 | + +这说明“agent 输出了一个 cost 数字”与“provider 返回了实际账单”是两件事。trajectory 最好明确区分： + +- `provider_reported`：例如 OpenRouter generation `total_cost`； +- `agent_calculated`：例如 Pi/OpenCode 用价格目录计算； +- `agent_reported`：例如 Claude Code terminal result，计算细节由 agent 封装； +- `unknown`：例如 Codex 公开 JSONL 没有 cost。 + +只有 `provider_reported` 可以直接回答“OpenRouter 这次实际记账多少”。其他值仍有观测价值，但不能无来源地混入同一个 total。 + +## 二、Messages 兼容路径与审计 fallback：Generation API + +OpenRouter 为每次请求建立 generation record。官方 [Get a Generation](https://openrouter.ai/docs/api/api-reference/generations/get-generation) API 是： + +```http +GET https://openrouter.ai/api/v1/generation?id=gen-1234567890 +Authorization: Bearer +``` + +响应关键字段如下： + +```json +{ + "data": { + "id": "gen-1234567890", + "model": "anthropic/claude-sonnet-4", + "provider_name": "Anthropic", + "streamed": true, + "native_tokens_prompt": 120, + "native_tokens_completion": 24, + "native_tokens_cached": 0, + "native_tokens_reasoning": 0, + "total_cost": 0.00072, + "usage": 0.00072, + "upstream_inference_cost": null + } +} +``` + +当 response body 没有 `usage.cost` 时，trajectory 应使用 `data.total_cost`：它表示这次 generation 实际记到 OpenRouter 账户的成本。若 Chat Completions 已直接返回 `usage.cost`，Generation API 则用于缺失补账和审计。`upstream_inference_cost` 不应替代账户实际扣费；Usage Accounting 文档明确说明，通过 Generation ID 查询时该字段只对 BYOK 请求可用，非 BYOK 通常为 `0` 或 `null`。 + +### 2.1 generation ID 从哪里拿 + +不能盲目使用 Anthropic `message.id`。Messages API 的消息 ID 可以是 `msg_...`，OpenRouter generation ID 则是 `gen-...`。OpenRouter 在 HTTP response header 中提供 `X-Generation-Id`。 + +当前 Anthropic SDK 的 stream 对象暴露底层 response headers，所以在现有 `with client.messages.stream(...) as stream:` 块内即可读取： + +```python +generation_id = stream.response.headers.get("x-generation-id") +``` + +应在响应完成后将它与 `message.id`、model、usage 一起保存。若 header 缺失，cost 状态应是 unknown，而不是把 `0` 当作未知值。 + +### 2.2 为什么它比价格表计算可靠 + +Generation record 已经知道最终实际使用的： + +- model 与 provider； +- fallback/routing 结果； +- native tokenizer 的输入、输出、cache 和 reasoning tokens； +- 当时生效的计费规则； +- OpenRouter 实际计入账户的 `total_cost`。 + +价格表估算则容易受 provider routing、fallback、cache read/write、reasoning、图片、搜索、按请求收费、service tier 和价格变更影响。对“这次历史运行到底花了多少”这个问题，generation record 是更合适的事实来源。 + +## 三、确实有模型价格 API，但应定位为估算工具 + +OpenRouter 提供： + +```http +GET https://openrouter.ai/api/v1/model/anthropic/claude-sonnet-4 +GET https://openrouter.ai/api/v1/models +Authorization: Bearer +``` + +官方 [Models API](https://openrouter.ai/docs/api/api-reference/models/get-models) 返回类似： + +```json +{ + "data": { + "id": "openai/gpt-4", + "pricing": { + "prompt": "0.00003", + "completion": "0.00006", + "request": "0", + "image": "0" + } + } +} +``` + +API 中这些字符串价格是每 token/每 request/每相应单位的美元价格；例如 `prompt = 0.00003` 等于 `$30 / 1M tokens`。页面常以每百万 token 展示，不要再额外除一百万。 + +最简单的文本估算是： + +```text +estimated_cost = + prompt_tokens × pricing.prompt + + completion_tokens × pricing.completion + + request_count × pricing.request +``` + +实际实现还必须按存在的字段处理 `input_cache_read`、`input_cache_write`、`internal_reasoning`、image、web search 等计费项，并使用 `Decimal`，不要用二进制 float 做账。 + +价格 API 适合： + +- 请求前预算和 max-cost guard； +- UI 展示大致单价； +- provider 没有实际 cost API 时的明确标注估算； +- 离线比较模型价格。 + +价格 API 不适合： + +- 回填 OpenRouter 历史实际账单； +- 自动路由后猜最终 provider； +- 把当前价格套到过去运行； +- 在字段缺失时把估算冒充 reported cost。 + +## 四、怎样与 Native Event、Event Journal 和 ATIF 一起实现 + +### 4.1 Native Event 表达事实，Journal Entry 立即落盘 + +若 Chat Completions response 或 terminal SSE event 已直接返回 `usage.cost`，`model.completed` 可以立即记录： + +```json +{ + "cost": { + "status": "resolved", + "amount": "0.00072", + "currency": "USD", + "source": "openrouter_response.usage.cost", + "kind": "provider_reported" + } +} +``` + +这种情况不需要仅为取得 cost 再查询一次 Generation API，但仍应保存 generation ID 以便审计。 + +若当前 Anthropic Messages response 没有 cost，模型响应完成时，core 先产生 `model.completed` Native Event；journal writer 添加持久化元数据并立即追加下面的 Journal Entry，不等待价格查询： + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 12, + "recorded_at": "2026-08-23T08:00:01.250Z", + "type": "model.completed", + "payload": { + "provider": "openrouter", + "model": "anthropic/claude-sonnet-4", + "message_id": "msg_abc123", + "generation_id": "gen-1234567890", + "usage": { + "input_tokens": 120, + "output_tokens": 24, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 0 + }, + "cost": { + "status": "pending", + "source": "openrouter_generation" + } + } +} +``` + +Generation API 成功后再追加： + +```json +{ + "schema_version": 1, + "run_id": "run-123", + "seq": 13, + "recorded_at": "2026-08-23T08:00:01.520Z", + "type": "model.cost_resolved", + "payload": { + "generation_id": "gen-1234567890", + "amount": "0.00072", + "currency": "USD", + "source": "openrouter_generation.total_cost", + "model": "anthropic/claude-sonnet-4", + "provider_name": "Anthropic" + } +} +``` + +这样做有三个好处：模型输出不会因为 cost API 暂时不可用而丢失；append-only Event Journal 不需要回头改旧 Journal Entry；ATIF projector 可以用 `generation_id` join 两条 Journal Entry 的 payload。 + +### 4.2 ATIF 的映射规则 + +若一次模型调用的 generation 查询成功： + +```json +{ + "metrics": { + "prompt_tokens": 120, + "completion_tokens": 24, + "cached_tokens": 0, + "cost_usd": 0.00072, + "extra": { + "cost_source": "openrouter_generation.total_cost", + "generation_id": "gen-1234567890" + } + } +} +``` + +run 级 `final_metrics.total_cost_usd` 只有在所有应计费 model call 都 resolved 时才填写为总和。若任何 generation 缺失或查询失败： + +- 已知 step 仍可填写各自的 `metrics.cost_usd`； +- 不要把未知 step 填成 `0`； +- 建议省略 `final_metrics.total_cost_usd`； +- 在 `final_metrics.extra` 记录 `known_cost_usd`、`cost_is_partial: true` 和缺失 generation IDs。 + +`0` 只能表示 OpenRouter 明确报告此次 cost 为零，例如真正免费的请求或某些 cache hit；它不能表示“没拿到数据”。 + +### 4.3 查询时机 + +推荐流程是： + +1. 模型完成立即落 `model.completed`； +2. response 已带 `usage.cost` 时直接标为 resolved； +3. cost 缺失时在 run 收尾阶段按 generation ID 查询真实 cost； +4. 使用有界 retry，因为 generation metadata 是异步可查的； +5. 查询失败不改变 agent 任务成功/失败状态，只把 cost completeness 标成 partial； +6. 最后原子写入完整 ATIF 快照。 + +若将来只需要尽快返回 CLI 结果，也可以让 cost enrichment 离线进行；Event Journal 中已有 generation ID，不会失去补账能力。 + +## 五、对 0.8.x trajectory 的决策影响 + +cost 可以和 trajectory 同一期实现，但应拆成 provider-aware enrichment，而不是把 OpenRouter HTTP 逻辑写进 ATIF serializer： + +```text +OpenRouter Chat response ─ usage.cost ─────────────┐ + ↓ +Anthropic Messages response ─ generation_id ─ cost resolver + │ + ↓ + Native Event + Journal Entry + │ + ↓ + Event Journal + │ + ↓ + ATIF projector +``` + +这也说明为什么 Native Event + Event Journal 比“直接边跑边拼 ATIF JSON”更稳：实际 cost 可能晚于模型响应到达，ATIF 是完成态文档，而 Event Journal 能自然表达 pending → resolved。 + +## 最终回答 + +你用 OpenRouter 时，真实 cost 并不是算不出来。OpenRouter Chat Completions 统一 API 可以直接返回 `usage.cost`，OpenRouter 也有准确到 generation 的 `total_cost`。nano 当前看不到，主要因为它使用 Anthropic Messages 兼容协议，而该协议的 `usage` schema 没有承诺直接携带 cost。协议选型与迁移依据见 [OpenRouter 统一模型协议调研](openrouter_unified_protocol.md)。 + +实现上，后续 Chat Completions transport 应优先读取 `usage.cost`；当前 Messages transport 或提前断流等缺失场景则捕获 `X-Generation-Id`，调用 `/api/v1/generation` 回填 `total_cost`。两者都作为 provider-reported cost 进入 Native Event，再由 journal writer 持久化为 Journal Entry，最终投影到 ATIF。模型价格 API 也存在，但它更适合预算和 fallback estimation，不应覆盖 OpenRouter 返回的真实扣费，也不应把当前价格估算冒充历史账单。 diff --git a/docs/research/zh-CN/openrouter_unified_protocol.md b/docs/research/zh-CN/openrouter_unified_protocol.md new file mode 100644 index 0000000..bb67ebe --- /dev/null +++ b/docs/research/zh-CN/openrouter_unified_protocol.md @@ -0,0 +1,392 @@ +# OpenRouter 统一模型协议：能力、Cost 与 nanoPyCodeAgent 接入边界 + +> 本文件为**中文源文件**（source of truth）；英文版 [`../en/openrouter_unified_protocol.md`](../en/openrouter_unified_protocol.md) 由其生成。 + +调研时间：2026-08-23。 + +## 问题 + +> OpenRouter 有没有自己模型无关的协议，既直接返回真实 cost，又具备 OpenAI、Anthropic 协议常用的 LLM 与 agent 能力？ +> +> 如果有，nanoPyCodeAgent 后续是否应以它作为模型无关的 provider 协议？ + +## 结论先行 + +有统一的模型访问表面，但更准确的名称是 **OpenRouter 统一 API**，而不是一套完全独立的“OpenRouter 原生消息协议”。它的主入口是 OpenAI-compatible 的 Chat Completions： + +```http +POST https://openrouter.ai/api/v1/chat/completions +``` + +同一个 request/response 形状可以选择不同厂商的模型；OpenRouter 在其上统一了 provider routing、fallback、usage accounting 等能力。它支持 streaming、tool calling、structured outputs、reasoning、multimodal input 与 prompt caching 等常见功能，但具体模型和 provider 是否支持某项参数仍有差异。 + +对 nanoPyCodeAgent，建议把它作为后续 **OpenRouter 模型无关 transport 的首选协议**： + +1. 默认使用 Chat Completions，而不是继续把 Anthropic Messages 当成 OpenRouter 的通用协议； +2. 非 streaming 完整响应或 streaming 最后一个 SSE event 中直接读取 `usage.cost`； +3. 同时保存 generation ID，必要时用 Generation API 查询或审计真实 cost； +4. 使用 `provider.require_parameters: true`，避免被路由到会忽略必需参数的 provider； +5. OpenRouter Responses API 作为并列候选保留；它的 item/event 模型更丰富，但当前 nano 没有依赖这些额外能力； +6. wire response 只是 Source Record，仍需经过 transport adapter 转成 nano 的 Native Event；它不取代 Event Journal，也不直接等于 ATIF。 + +## 一、先澄清“OpenRouter 原生协议”这个说法 + +OpenRouter 的 [FAQ](https://openrouter.ai/docs/faq) 与 [Quickstart](https://openrouter.ai/docs/quickstart) 将 `/api/v1/chat/completions` 描述为 OpenAI-compatible API：可以使用 OpenAI SDK，只需把 base URL 和 API key 指向 OpenRouter；也可以使用 OpenRouter 自己的 SDK。 + +因此应区分三个概念： + +| 概念 | 含义 | 是否适合称为模型无关 | +|---|---|---| +| OpenAI Chat Completions | OpenAI 定义的 messages/choices/tool_calls 基础形状 | 基础协议本身不绑定具体模型 | +| OpenRouter 统一 API | OpenAI-compatible 形状，加上跨模型路由、provider 约束、统一 usage/cost 等 OpenRouter 扩展 | 是，适合作为 nano 的 OpenRouter transport | +| OpenRouter Client SDK | 对上述 HTTP API 的轻量、类型安全封装 | 是客户端实现，不是另一套 wire protocol | +| OpenRouter Agent SDK | 在模型调用之上增加 agent loop、tool execution 和 state management | 是 agent runtime，不是 LLM wire protocol | + +所以本文后续使用“OpenRouter 统一 API”或“OpenRouter Chat Completions transport”，不使用容易让人误以为存在第四套全新消息语义的“OpenRouter 原生协议”。nano 的目标是自己实现 agent core，因此即使 OpenRouter Agent SDK 能直接完成 agent loop，也不应让它取代 nano 的 tool loop、Native Event 和 trajectory；若采用官方 SDK，更合适的是薄层 Client SDK。 + +## 二、三个相关端点的定位 + +| 端点 | 协议形状 | 状态与能力 | 对 nano 的建议 | +|---|---|---|---| +| `/api/v1/chat/completions` | OpenAI-compatible messages/choices | OpenRouter 主入口；支持 streaming、tools、结构化输出及统一 usage | **作为 OpenRouter 模型无关 transport 的默认选择** | +| `/api/v1/responses` | item/event-oriented Responses | OpenAI-compatible；支持 reasoning、tools、web search，但当前仅支持 stateless 请求 | 作为并列候选评估，不作为本阶段默认 | +| `/api/v1/messages` | Anthropic Messages-compatible | 方便复用 Anthropic SDK 与 content blocks；公开 usage schema 未承诺 `cost` | 作为兼容 transport 保留，不作为 OpenRouter 默认抽象 | + +OpenRouter 的 [Chat Completions API](https://openrouter.ai/docs/api/api-reference/chat/create-a-chat-completion) 提供 model、messages、tools、tool choice、response format、reasoning、provider routing、fallback models 和 streaming 等参数。 + +[Responses API](https://openrouter.ai/docs/api/reference/responses/overview) 的数据模型更接近事件/item 流，也支持 reasoning、tool calling 和 web search；但 OpenRouter 当前只支持 stateless 使用，每次请求都要带完整历史，`store: true` 和非空 `previous_response_id` 会被拒绝。它并不缺少 nano agent loop 的基本能力，只是相对 Chat Completions 增加了一套 input/output item 与 streaming event 映射，而 nano 当前没有必须依赖这些额外语义的需求。 + +选择 Chat Completions 不是因为 Responses “不能做 agent”，而是因为官方 Quickstart 仍把 Chat Completions 作为最直接入口，当前 nano 的 message/tool loop 也更接近它。实现 transport interface 时应避免把 `choices[]` 固化进 core，以便未来增加 Responses adapter，而不必再次改写 Native Event。 + +[Anthropic Messages endpoint](https://openrouter.ai/docs/api/api-reference/anthropic-messages/create-messages) 则是兼容表面。它让当前 nano 几乎不改 agent loop 就能访问 OpenRouter，但也使 OpenRouter 自己的扩展字段受 Anthropic response schema 约束；cost 就是当前最明显的例子。 + +## 三、Chat Completions 覆盖哪些 agent 基本能力 + +### 3.1 多轮消息与 streaming + +请求通过 `messages` 传入完整的 user/assistant/tool 历史；`stream: true` 时响应使用 SSE。普通文本位于增量 `delta.content`，工具调用位于增量 `delta.tool_calls`。 + +transport 必须先按 choice index、tool-call index 和 call ID 组装增量，得到完整的工具名与 JSON arguments 后才能执行工具。不能把单个 SSE frame 当成完整 Tool Call Native Event。 + +### 3.2 Tool calling + +OpenRouter 的 [Tool Calling](https://openrouter.ai/docs/guides/features/tool-calling) 使用 OpenAI function-calling 形状： + +- request 在 `tools[].function` 中声明 name、description 和 JSON Schema； +- assistant 通过 `message.tool_calls[]` 发起调用； +- client 执行本地工具； +- 下一次 request 用 `role: "tool"` 和 `tool_call_id` 回传结果； +- 支持 `tool_choice`，部分模型支持并行 tool calls。 + +“协议支持 tools”不等于“每个模型都支持 tools”。模型目录会声明支持的参数，routing 时还应要求 provider 实际支持这些参数。 + +### 3.3 Structured outputs + +[Structured Outputs](https://openrouter.ai/docs/guides/features/structured-outputs) 使用 `response_format.type = "json_schema"` 和 JSON Schema 约束输出，可与 streaming 组合。模型支持情况并不一致；如果任务依赖严格 schema，应配合： + +```json +{ + "provider": { + "require_parameters": true + } +} +``` + +OpenRouter 的 [Provider Routing](https://openrouter.ai/docs/guides/routing/provider-selection) 说明，默认情况下 provider 可能忽略其不支持的可选参数；`require_parameters` 会把候选范围限制为支持请求参数的 provider。这对 tool calling 和 structured output 的可移植性很重要。 + +### 3.4 Reasoning、multimodal 与 caching + +OpenRouter 还提供： + +- [Reasoning tokens](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens)：统一一部分 reasoning 控制与返回字段；不同模型是否暴露 reasoning 内容仍有差异； +- [Multimodal requests](https://openrouter.ai/docs/guides/overview/multimodal/overview)：图片等内容继续通过 Chat Completions 的 messages/content blocks 发送； +- [Prompt caching](https://openrouter.ai/docs/guides/best-practices/prompt-caching)：在支持的模型/provider 上统一 cache accounting，但自动缓存、显式断点与 TTL 能力并非完全相同； +- model/provider fallback：OpenRouter 可以在候选模型或 provider 之间路由，最终 response 的 model 与 generation metadata 才是本次调用的实际结果。 + +所以“模型无关”应理解为 **同一个基础 request/response contract 可以访问多模型**，而不是所有模型的能力、参数语义和质量完全一致。 + +## 四、一次真实形状的 tool-call 往返 + +下面的 ID、tokens 与金额是说明性值，但字段形状对应 OpenRouter Chat Completions、Tool Calling 和 Usage Accounting 文档。 + +### 4.1 首次请求 + +```http +POST /api/v1/chat/completions +Authorization: Bearer +Content-Type: application/json +``` + +```json +{ + "model": "anthropic/claude-sonnet-4", + "messages": [ + { + "role": "user", + "content": "读取 pyproject.toml 并告诉我项目名" + } + ], + "tools": [ + { + "type": "function", + "function": { + "name": "read_file", + "description": "Read a UTF-8 text file", + "parameters": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"], + "additionalProperties": false + } + } + } + ], + "tool_choice": "auto", + "provider": { + "require_parameters": true + }, + "stream": false +} +``` + +### 4.2 模型请求调用工具 + +```json +{ + "id": "gen-abc123", + "model": "anthropic/claude-sonnet-4", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_read_1", + "type": "function", + "function": { + "name": "read_file", + "arguments": "{\"path\":\"pyproject.toml\"}" + } + } + ] + }, + "finish_reason": "tool_calls" + } + ], + "usage": { + "prompt_tokens": 205, + "completion_tokens": 18, + "total_tokens": 223, + "cost": 0.00071, + "cost_details": { + "upstream_inference_cost": 0.00066 + } + } +} +``` + +`usage.cost` 表示 OpenRouter 向当前账户收取的本次总额；它不是 nano 根据公开价格表自行计算的估值。 + +### 4.3 执行工具并继续请求 + +client 本地执行 `read_file` 后，把原 assistant tool call 和 tool result 都放回消息历史： + +```json +{ + "model": "anthropic/claude-sonnet-4", + "messages": [ + { + "role": "user", + "content": "读取 pyproject.toml 并告诉我项目名" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call_read_1", + "type": "function", + "function": { + "name": "read_file", + "arguments": "{\"path\":\"pyproject.toml\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call_read_1", + "content": "[project]\nname = \"nanoPyCodeAgent\"" + } + ], + "tools": [ + { + "type": "function", + "function": { + "name": "read_file", + "description": "Read a UTF-8 text file", + "parameters": { + "type": "object", + "properties": { + "path": {"type": "string"} + }, + "required": ["path"], + "additionalProperties": false + } + } + } + ], + "provider": { + "require_parameters": true + } +} +``` + +最终响应的 `choices[0].message.content` 是给用户的文字答案，并再次带本次 model call 自己的 usage/cost。一次 agent turn 可能包含多次 model call，因此 run cost 应对每次实际调用分别记账，再在完整性已知时汇总。 + +## 五、Cost：直接返回、streaming 与 Generation API + +OpenRouter 的 [Usage Accounting](https://openrouter.ai/docs/cookbook/administration/usage-accounting) 说明，每个完整响应都会包含详细 usage：包括 prompt/completion/reasoning/cache tokens、总 cost 和 cost details。 + +### 5.1 非 streaming + +直接读取完整 JSON response 的： + +```text +usage.cost +usage.cost_details.upstream_inference_cost +``` + +对普通 OpenRouter credits 调用，`usage.cost` 是 nano 应采用的 provider-reported amount。`upstream_inference_cost` 是 provider 成本明细，不应代替向 OpenRouter 账户实际收取的总额。 + +OpenRouter [FAQ](https://openrouter.ai/docs/faq) 说明其 credits 的基础货币是美元，站点和 API 的定价也以美元表示。因此普通 credits 调用的 `usage.cost` 可以映射到 ATIF `cost_usd`；Native Event 仍应显式保存 `currency: "USD"` 和字段来源，避免仅凭字段名猜测币种。 + +### 5.2 Streaming + +`usage` 位于最后一个 SSE event。transport 必须消费到 terminal event，才能把一次 model call 标成 cost resolved。提前断流时不能把 cost 写成 `0`；应保留 unknown/pending 状态和 generation ID。 + +旧的： + +```json +{"usage": {"include": true}} +``` + +以及： + +```json +{"stream_options": {"include_usage": true}} +``` + +已经不再是取得 usage 的必要条件；官方文档将它们标为 deprecated/no effect。 + +### 5.3 Generation API 是 fallback 与审计路径 + +即使主路径直接收到 `usage.cost`，仍应保存 response header `X-Generation-Id` 或等价 generation ID。遇到以下情况时，通过： + +```http +GET /api/v1/generation?id= +``` + +查询 `data.total_cost`： + +- 当前走的是 Anthropic Messages compatibility endpoint，response body 没有 cost； +- streaming 在最终 usage event 前断开； +- 需要核对路由后的实际 model/provider； +- 需要离线补账或审计。 + +具体记账与 ATIF 映射见 [OpenRouter 的真实 Cost、价格 API 与 Trajectory 记账方案](openrouter_cost_accounting.md)。模型目录的价格字段只适合预算和显式估算，不应覆盖 provider-reported cost。 + +## 六、OpenRouter wire event 与 nano 内部事件的关系 + +统一 API 解决的是 **provider transport**，不是 trajectory 存储。建议的数据流是： + +```text +OpenRouter HTTP response / SSE frames + │ + │ Source Records + ▼ +OpenRouter Chat transport + - assemble content deltas + - assemble tool-call arguments + - normalize usage / cost / errors + │ + │ Native Events + ▼ + journal writer + │ + │ Journal Entries + ▼ + Event Journal + │ + ▼ + ATIF projector + │ + ▼ + ATIF Trajectory +``` + +对应关系如下： + +| OpenRouter 数据 | nano 概念 | 原因 | +|---|---|---| +| 原始 JSON response、response headers、单个 SSE frame | Source Record | 是外部协议的原始观察，不是 core 领域事件 | +| 已完整组装的 assistant output | `model.completed` Native Event | core 已理解为一次模型调用完成 | +| 已完整组装的 `tool_calls[]` | `model.completed.payload.tool_calls` | 作为模型完成事实的一部分；必须先合并 streaming arguments，再执行工具 | +| `usage.cost` | `model.completed` 中已 resolved 的 provider-reported cost | cost 与本次 model call 同时到达 | +| Generation API 后补的 `total_cost` | `model.cost_resolved` Native Event | cost 晚到，用 append-only 事件补充，不改写旧 Journal Entry | +| journal writer 添加的 `run_id`、`seq`、`recorded_at` | Journal Entry 的持久化元数据 | 属于完整 Journal Entry，但不属于 OpenRouter 协议 | + +因此不应把 OpenRouter JSONL/SSE 原样命名为“native trajectory”，也不应让 ATIF serializer 直接理解所有 provider wire shapes。transport adapter 负责协议差异；Native Event、Event Journal 和 ATIF 的语义边界保持不变。 + +## 七、nano 当前并不能只改 base URL + +当前仓库的 transport 与 Anthropic SDK 紧密耦合： + +- [`agent.py`](../../../src/nanopycodeagent/agent.py) 直接创建 `anthropic.Anthropic` 并调用 `client.messages.stream(...)`； +- conversation history 使用 `anthropic.types.MessageParam`； +- tool call/result 使用 `ToolUseBlock` 与 `ToolResultBlockParam`； +- 四个工具定义都标注为 `anthropic.types.ToolParam`； +- tests 中的 fake client 和异常类型也模拟 Anthropic Messages。 + +OpenRouter Chat Completions 的 tool declaration 很接近现有 JSON Schema，但 assistant tool calls、tool results、streaming delta 和 finish reason 的形状不同。因此迁移不是把 `ANTHROPIC_BASE_URL` 换成另一个 URL，而是要抽出 transport 边界。 + +### 7.1 建议的最小实现顺序 + +1. 定义 provider-neutral 的内部 message、content、tool call、tool result 和 usage 类型； +2. 把现有逻辑包进 `AnthropicMessagesTransport`，保持兼容行为； +3. 新增 `OpenRouterChatTransport`，负责 Chat Completions request/response 和 SSE 组装； +4. 两个 transport 都只向 core 产出统一 Native Events； +5. OpenRouter transport 优先读取 `usage.cost`，缺失时交给 generation cost resolver； +6. agent loop 继续负责工具调度，不直接依赖任一 SDK 的 block class； +7. 测试文本 streaming、碎片化 tool arguments、多个 tool calls、usage/cost、reasoning/cache token、提前断流与 API error。 + +这层抽象的目标是统一 **core 看见的语义**，不是强行抹平所有 provider 功能。provider-specific extensions 可以保留在 Source Record 或 Native Event 的 namespaced `extra` 中。 + +## 八、建议形成的架构决策 + +后续实现模型无关协议时，采用以下决策： + +1. **OpenRouter 默认协议：** `/api/v1/chat/completions`； +2. **OpenRouter cost 主来源：** response/final SSE event 的 `usage.cost`； +3. **cost fallback：** generation ID + `/api/v1/generation` 的 `total_cost`； +4. **能力约束：** tool calling、structured outputs 等必需能力配合 `provider.require_parameters: true`； +5. **兼容路径：** Anthropic Messages 作为独立 transport 保留，不再代表 core 的内部消息模型； +6. **并列候选：** Responses API 暂不成为默认 transport；待 nano 需要 richer item/event、web search 或 Responses SDK 兼容时再实现 adapter； +7. **存储边界：** wire response 是 Source Record，Native Event/Journal Entry/Event Journal 仍是 nano 自己的运行事实层； +8. **导出边界：** ATIF 继续由 Event Journal 投影，不直接从任一 provider response 拼装。 + +## 九、本次调研的验证范围 + +本文的协议结论来自 2026-08-23 的 OpenRouter 官方 API reference、feature guides 与 usage accounting 文档，并结合当前 nano 源码做了静态接入分析。本次没有发起会产生 OpenRouter 费用的线上请求，因此示例中的 ID、token 数与金额是说明性值，不是本项目账户的真实账单记录。 + +这不影响协议字段与架构决策，但实际实现时仍应增加一个使用最便宜可用模型的 opt-in live integration test，验证当前账户下的： + +- non-streaming `usage.cost`； +- streaming terminal usage event； +- `X-Generation-Id` 与 Generation API 对账； +- tool-call delta 组装； +- fallback 后 response model/provider 与 cost 的一致性。