diff --git a/docs/dev_notes/en/0.8.x.md b/docs/dev_notes/en/0.8.x.md index d1768f8..7fc2492 100644 --- a/docs/dev_notes/en/0.8.x.md +++ b/docs/dev_notes/en/0.8.x.md @@ -10,7 +10,9 @@ I first surveyed the landscape in [code_agent_benchmark](../../research/en/code_ The first feature is therefore a **headless CLI**, the first step in the survey's minimum viable path. It is the only hard blocker: the first three requirements shared by all three benchmarks — accept task text in one command and run to completion, work in the process's current directory, and never prompt, ask questions, or wait for confirmation — all depend on it. The only existing entry point is the `input("You> ")` loop in `agent.py`; inside a container stdin is EOF, so the process immediately prints `Bye!` and exits without doing anything. Until this works, none of the later harness improvements can be verified programmatically, so there is no basis for deciding what to optimize. -### Headless CLI contract +### Basic headless CLI implementation + +Before worrying about diagnostics or optimization, the first goal is to implement the smallest feature that can complete the simplest benchmark task. **Command-line shape**: @@ -50,3 +52,57 @@ printf "%s" "create hello.py that prints hi" | nanoPyCodeAgent; echo $? If the task arrives through the pipe, the file is created, and the exit code is 0, the step is complete. This is the first point at which nanoPyCodeAgent can be called by a script, and only then can it be connected to a benchmark. After this, the next work is: `--output-format stream-json` and `--trajectory` (step 4 of the minimum viable path, once failures need attribution), API retry and backoff (step 2), context compaction (step 5), `--workdir` (downgraded from P0 by the survey because all three benchmarks pass their working directory through the container's default `WORKDIR`; the agent only needs to operate in the current process directory), and `--timeout` (the harness owns the wall-clock timeout — Harbor stores it in `[agent].timeout_sec` in `task.toml` — so an agent-side timeout is a safeguard rather than an integration prerequisite). + +### First end-to-end validation with Terminal-Bench + +The `hello.py` smoke test only proves that the CLI can accept a task and exit; it does not prove that the CLI can participate in a real benchmark. The goal of this run was not to produce a formal benchmark score, but to choose a task simple enough for an initial attempt while still exercising command execution, file writes, recovery from mistakes, and official verification, then run the full path for real. + +The selected task was `terminal-bench/openssl-selfsigned-cert` from Terminal-Bench 2.1. Harbor 0.21.0 and the nanoPyCodeAgent adapter were both placed temporarily under `/tmp`; no benchmark-specific code was added to the project. The adapter inherits Harbor's `BaseInstalledAgent`. The `--agent nanopy_harbor_agent:NanoPyCodeAgent` argument uses the dynamic import form `module:ClassName`, allowing Harbor on the host to execute this chain: + +```text +Harbor + -> dynamically load the temporary adapter + -> start the task's Docker environment + -> install and invoke the nanoPyCodeAgent headless CLI in /app + -> wait for the agent to exit + -> run the official verifier + -> save logs, test results, and the reward +``` + +The core run configuration was as follows. The API key, base URL, and model configuration were still injected through environment variables and were not written into the command or logs: + +```bash +harbor run \ + --task terminal-bench/openssl-selfsigned-cert \ + --agent nanopy_harbor_agent:NanoPyCodeAgent \ + --model openrouter/deepseek/deepseek-v4-flash-0731 \ + --env docker \ + --n-concurrent 1 \ + --n-attempts 1 +``` + +During the first environment preparation, the container's initial Python and dependency downloads exceeded the default setup timeout. The model had not been called yet, so this was not a benchmark result. After increasing only the installation-phase timeout while keeping the task, model, turn limit, and verifier unchanged, the final trial completed end to end: + +| Metric | Result | +| --- | --- | +| Reward | 1.0 | +| Verifier | 6/6 tests passed | +| Agent execution time | 2 minutes 9 seconds | +| Tool calls | 16 bash, 2 write, 1 edit | +| Agent / verifier exceptions | 0 | + +This result does not prove that nanoPyCodeAgent can run an entire leaderboard workload reliably. It proves that the minimum closed loop works: Harbor can deliver a real task to the headless CLI; the CLI can invoke the model and tools unattended in the isolated container's current working directory; the model can recover from mistakes; and after the process exits normally, the official verifier can independently produce a reward. At this point, the simplest benchmark can run all the way from task input to final scoring. + +### Dependency and observability boundaries exposed by the run + +One container installation reported a missing `httpx` dependency when starting the CLI. Adding `uv tool install --with httpx` to the temporary adapter allowed the evaluation to continue. Two later clean installations using the same uv version, Python version, and project revision did not reproduce the omission because `anthropic` still installs `httpx` as a transitive dependency. It would therefore be inaccurate to claim that every clean installation fails. The confirmed issue is that nanoPyCodeAgent directly imports `httpx` without declaring it as a direct dependency. The project currently works only by relying on the `anthropic -> httpx` transitive relationship, and that dependency ownership should still be corrected. + +Although trajectory output does not exist yet, the tool calls could still be counted from the plain-text log. Before every tool call, the CLI prints a prefix such as `[bash]`, `[write]`, or `[edit]`, so the completed evaluation log can be counted with text matching. This count has no turn boundaries, tool call IDs, input or output token counts, cost, duration, or structured error status, which is why Harbor still reports token and cost fields as `null`. + +The [agent_output_and_trajectory](../../research/en/agent_output_and_trajectory.md) survey makes clear that this text log must not simply be renamed a trajectory. Run output is what a single invocation publicly delivers, while a trajectory is a structured execution path bounded by one benchmark task or trial and intended for benchmark and offline analysis. In the future, `--output-format text|json|stream-json` should control only the public stdout representation, while `--trajectory PATH` independently stores analysis data such as observations, actions, tool results, outcomes, and usage. Both should be projected from the same canonical internal events, but they must remain separate interfaces. + +The next work therefore has three independent parts: + +- Add `httpx` as a direct dependency so container installation does not rely on a transitive relationship. +- Add the Harbor adapter to the project, fixing and versioning its installation command, headless CLI invocation, environment-variable forwarding, and version probing so future benchmark runs can reuse it instead of rewriting a temporary adapter each time. +- Design canonical internal events, `stream-json`, and a trajectory writer so the adapter can convert the native trajectory to ATIF and reliably populate token, cost, and step statistics. These changes improve the installability, reproducibility, and observability of the working benchmark path. diff --git a/docs/dev_notes/zh-CN/0.8.x.md b/docs/dev_notes/zh-CN/0.8.x.md index f308cbf..caac658 100644 --- a/docs/dev_notes/zh-CN/0.8.x.md +++ b/docs/dev_notes/zh-CN/0.8.x.md @@ -1,6 +1,6 @@ # 开发笔记 — 0.8.x -> 本文件为**手写中文源文件**(source of truth);英文版 [`../en/0.8.x.md`](../en/0.8.x.md) 由其生成。 +> 本文件为**中文源文件**(source of truth);英文版 [`../en/0.8.x.md`](../en/0.8.x.md) 由其生成。 ## 0.8.0 - YYYY.MM.DD @@ -10,7 +10,9 @@ 第一个功能，先实现 **headless 模式的 CLI**，也就是调研里「最小可行路径」的第一步。它是唯一的硬阻塞项：三个 benchmark 的最小公共契约的头三条——一条命令拿到任务文本跑完退出、在进程当前 cwd 里干活、不交互不提问不等确认——全部卡在这里。当前唯一入口是 `agent.py` 的 `input("You> ")` 循环，容器里 stdin 是 EOF，进程会立刻打印 `Bye!` 退出，一件事都没干。这一步不通，后面任何 harness 层面的改动都无法被程序化地验证，也就无从谈优化目标。 -### headless CLI 实现契约 +### headless CLI 的基本实现 + +先不考虑排查问题，优化效果等，先实现一个最小功能，能跑通一个最小的 benchmark 问题。 **命令行形态**： @@ -50,3 +52,56 @@ printf "%s" "create hello.py that prints hi" | nanoPyCodeAgent; echo $? 任务从管道输入、创建文件、退出码是 0，这一步就算完成。此后 nanoPyCodeAgent 才第一次具备「被脚本调用」的能力，也才谈得上接 benchmark。 上述完成后，后续要做的：`--output-format stream-json` 与 `--trajectory`（属最小可行路径第 4 步，等失败需要归因时再做）、API 重试与退避（第 2 步）、上下文压缩（第 5 步）、`--workdir`（调研已把它从 P0 降级：三家 benchmark 的工作目录都是靠容器默认 WORKDIR 传递的，agent 在进程当前 cwd 里干活即可）、`--timeout`（墙钟超时由 harness 侧管，Harbor 在 `task.toml` 的 `[agent].timeout_sec` 里，agent 自己的超时是保险而非接入前提）。 + +### 用 Terminal-Bench 完成第一次端到端验收 + +`hello.py` 的 smoke test 只能证明 CLI 能接收任务并退出，不能证明它已经接入真实 benchmark。这次的目标不是正式跑分，而是选择一条足够简单、但仍包含命令执行、文件写入、出错后自我修正和官方验证的任务，把完整链路真正跑通一次。 + +选择的是 Terminal-Bench 2.1 的 `terminal-bench/openssl-selfsigned-cert`。Harbor 0.21.0 和 nanoPyCodeAgent 的适配器都临时放在 `/tmp`，没有把 benchmark 专用代码加入项目。适配器继承 Harbor 的 `BaseInstalledAgent`；`--agent nanopy_harbor_agent:NanoPyCodeAgent` 采用 `module:ClassName` 动态导入语法，让宿主机上的 Harbor 完成下面这条调用链： + +```text +Harbor + -> 动态加载临时适配器 + -> 启动题目的 Docker 环境 + -> 在容器的 /app 中安装并调用 nanoPyCodeAgent headless CLI + -> 等待 agent 退出 + -> 运行官方 verifier + -> 保存日志、测试结果和 reward +``` + +实际运行的核心配置如下；密钥、base URL 和模型配置仍通过环境变量注入，没有写进命令或日志： + +```bash +harbor run \ + --task terminal-bench/openssl-selfsigned-cert \ + --agent nanopy_harbor_agent:NanoPyCodeAgent \ + --model openrouter/deepseek/deepseek-v4-flash-0731 \ + --env docker \ + --n-concurrent 1 \ + --n-attempts 1 +``` + +第一次准备环境时，容器首次下载 Python 和依赖超过了默认 setup timeout；这时尚未调用模型，不能算一次 benchmark 结果。只放大安装阶段的超时、保持题目、模型、轮数和 verifier 不变后，最终 trial 完整跑通： + +| 指标 | 结果 | +| --- | --- | +| Reward | 1.0 | +| Verifier | 6/6 测试通过 | +| Agent 执行时间 | 2 分 9 秒 | +| 工具调用 | 16 次 bash、2 次 write、1 次 edit | +| Agent / verifier 异常 | 0 | + +这次结果证明的不是 nanoPyCodeAgent 已经具备稳定跑整套榜单的能力，而是最小闭环成立了：Harbor 能把真实任务交给 headless CLI；CLI 能在隔离容器的当前工作目录中无人值守地调用模型和工具；模型中途出错后能继续修正；进程正常退出后，官方 verifier 能独立给出 reward。至此，最简单 benchmark 已经可以从任务输入一直运行到最终评分。 + +### 实跑暴露的依赖与可观测性边界 + +容器中有一次安装在启动 CLI 时报告缺少 `httpx`，临时适配器用 `uv tool install --with httpx` 后继续完成了评测。后续使用相同 uv、Python 和项目提交做的两次干净安装都没能复现这个遗漏，因为 `anthropic` 当前仍会把 `httpx` 作为传递依赖安装进来。因此，不能把它概括成「干净安装必然失败」；真正确定的问题是 nanoPyCodeAgent 自己直接 `import httpx`，却没有把它声明为直接依赖。当前能运行只是借用了 `anthropic -> httpx` 的传递关系，这个依赖所有权仍应修正。 + +虽然还没有 trajectory，这次仍能从普通文本日志里数出工具调用次数：每次调用工具前，CLI 都会打印 `[bash]`、`[write]`、`[edit]` 等前缀，评测结束后可以用文本匹配计数。但这种计数没有 turn 边界、tool call ID、输入输出 token、cost、duration 和结构化错误状态，所以 Harbor 结果中的 token 与 cost 仍然是 `null`。 + +根据 [agent_output_and_trajectory](../../research/zh-CN/agent_output_and_trajectory.md) 的调研，这份文本日志不能被重新命名成 trajectory：run output 是本次调用公开交付的内容，trajectory 才是以一次 task/trial 为边界、供 benchmark 和离线分析使用的结构化执行路径。后续实现时，`--output-format text|json|stream-json` 只控制 stdout 的公开表示；`--trajectory PATH` 独立保存 observation、action、tool result、outcome 和 usage 等分析数据，两者应由同一组内部标准事件投影，但不能混成一个接口。 + +所以下一步分成三件独立的工作： + - 把 `httpx` 补成直接依赖，保证容器安装不依赖传递关系 + - 把 Harbor adapter 正式纳入项目，固定并版本化安装命令、headless CLI 调用、环境变量传递和版本探测逻辑，使后续 benchmark 可以直接复用，不必每次重新编写临时适配器 + - 设计内部标准事件、`stream-json` 和 trajectory writer，让这个 adapter 可以把 native trajectory 转成 ATIF，并可靠回填 token、cost 和步骤统计。这些属于对已跑通链路的可安装性、可复现性与可观测性增强。