feat: scaffold SWE-bench-Pro baseline for the vero optimizer#50
Draft
shehabyasser-scale wants to merge 42 commits into
Draft
feat: scaffold SWE-bench-Pro baseline for the vero optimizer#50shehabyasser-scale wants to merge 42 commits into
shehabyasser-scale wants to merge 42 commits into
Conversation
varunursekar
force-pushed
the
pr3-harness-bench
branch
6 times, most recently
from
July 26, 2026 04:26
51c152d to
ced19f9
Compare
Bring the benchmark harness with each benchmark as a top-level sibling of gaia/ (ale-bench, officeqa, swe-atlas-qna, tau3), flattening the former candidates/ grouping.
The floor gates a ship on a validation comparison while gaia's reward is on the test partition (different distributions), so keep it opt-in / off. Now that the harbor default is false this is explicit-for-clarity.
Now that the sidecar image pre-warms the harness uv cache, the inner eval runs as the unprivileged harness user again, so the candidate harness cannot read the held-out records, budgets, or admin token.
Move each benchmark to harbor 0.20.0 in lockstep so the eval resolves: build.yaml harbor_requirement (the evaluator's --with harbor[modal]) and target/pyproject.toml (the candidate's own harbor dep) must match, else `uv run --project <target> --with harbor[modal]==0.20.0` is unsatisfiable. uv.lock regenerated with `uv lock` (not hand-edited). Covers gaia, officeqa, swe-atlas-qna, tau3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the pr2 rename: instructions, READMEs, and the ale-bench reference solution now teach `evals run/status/result/submit` and the .evals/tasks + .evals/results paths; baseline gitignores cover .evals/. Compiled trees are build outputs (not committed) and pick this up on rebuild. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All four vero-compiled benchmarks pass --ek app_name=harness-engineering-bench so their Modal sandboxes are grouped under one app instead of __harbor__. Docker inner environments ignore the kwarg.
The task's verifier runs inside the Modal task container and templates EVAL_BASE_URL from OPENAI_API_BASE, unreachable-from-Modal gateway URLs aside. Enable task_services_use_upstream so the judge gets the real upstream on OPENAI_*, and point the baseline agent at the metered gateway via VERO_AGENT_INFERENCE_* (falling back to OPENAI_* locally).
The Modal app-name commit added a second extra_harbor_args key; YAML last-key-wins silently dropped --no-force-build, which would have force-built the guard Dockerfile instead of pulling the prebuilt corpus image.
Killing the outer harbor process leaves the compose topology and any in-flight Modal sandboxes running (each layer's cleanup lives in the layer above). Cap sandbox idle lifetime at 1h via --ek, and add scripts/cleanup_orphans.sh to tear down task__* containers and terminate sandboxes in the dedicated Modal app.
Across gaia/officeqa/swe-atlas-qna/tau3: 100-run budgets and 4x-pass
case budgets on both agent partitions, submit-mode selection,
baseline_floor off, 16k feedback, parameterized inner environment
(${inner_env:-modal}), and verifier_timeout = 2x timeout_seconds.
swe-atlas-qna and tau3 get explicit case/task-agent timeouts derived
from their datasets' declared agent timeouts (enforced budgets 1800s
and 900s, provisional until empirical wall_seconds data). harness_user
is explicit on gaia/officeqa; tau3 and swe-atlas-qna stay unisolated
pending off-env delivery of task-service credentials. Also fix the
vendor script's stale pre-restructure path in its usage comment.
Root cause of every swe-atlas trial dying at setup: the task images set ENTRYPOINT ["/bin/bash"], Modal prepends it to the sandbox command, so harbor's default keepalive ["sh","-c","sleep infinity"] ran as 'bash sh -c ...' - bash interprets the sh binary as a script and exits 126, killing the sandbox before the first exec. Pass bash's own args via the JSON-decoded keepalive env kwarg instead. Validated end-to-end on Modal (3 dev trials, 379-602s wall). CONFIGURATION.md records the empirical probe timings behind the provisional case timeouts.
scripts/per_trial_tokens.py reads a session's gateway request log and attributes token/latency to Harbor trials: by the gateway's stamped thread_id when present (request_log.attribution builds → full coverage incl. stateful chains), else a legacy fallback that recovers each conversation's root turn (labelable via --tasks-dir) and residualizes unrecoverable follow-ups. Reports gateway-total vs attributed vs agent-reported per evaluation. Stdlib-only; tests cover both paths.
…r at turn cap Chat Completions is the only universally-supported inference surface — the OpenAI Responses API doesn't translate cleanly to non-OpenAI models via litellm (parallel_tool_calls rejected, output_text crashes on a null message part). Rewrite AtlasAgent's loop to stateless chat messages + tool_calls, validated cross-provider (gpt-5.4-mini and fireworks_ai/gpt-oss-120b both run the multi-turn tool loop). Also: at the turn cap, force one final tool-free answer instead of raising, so a budget-exhausted trial scores best-effort rather than losing the case.
Same port as swe-atlas: stateless chat messages + tool_calls, works across providers (validated cross-provider on gpt-5.4-mini and fireworks_ai/gpt-oss-120b). officeqa drops the hosted web_search tool (closed-corpus benchmark, and chat/completions has no hosted-tool equivalent) and moves read_image feedback to chat image_url content. tau3 keeps its host-side MCP loop; the stateless history also fixes the previous_response_id reset that dropped context after plain-text turns, and the MCP session id is now charset-validated before shell interpolation (greptile injection finding).
Mirrors the swe-atlas fix: at MAX_TURNS force one tool-free completion for the best answer and submit it, so a budget-exhausted case scores best-effort rather than being lost with no answer recorded.
A fifth harness-engineering benchmark: optimize a retrieval + reasoning agent over BrowseComp-Plus's fixed corpus and canonical BM25 index, graded by a semantic answer judge (gpt-4.1). Task packages are generated from the encrypted upstream dataset by scripts/build_tasks.py and are gitignored (they contain decrypted questions/answers); the upstream repo is pinned as a submodule and the 2.2 GB index is pulled at image-build time. Wired consistently with the normalized suite: 33/66/66 stratified split, 4-pass budgets, submit-mode selection, harness_user null + task_services_use_upstream for the in-container judge. Baseline agent uses Chat Completions (works across providers) and forces a best-effort answer at the turn cap. The task image installs a full JDK with JAVA_HOME resolved from it, so pyserini's Lucene BM25 searcher works (a headless JRE with JAVA_HOME unset left jnius unable to locate the JVM).
… succeeds pyserini 1.2.0 constructs an openai.OpenAI() client at import time for its unused OpenAI encoders; BM25/Lucene search never calls OpenAI, so set a placeholder key in search.py before the import.
The task.toml had no verifier.env, so the gpt-4.1 answer judge ran without OPENAI_API_KEY and scored every case 0 (Missing credentials). Template the upstream creds and judge model into the verifier phase.
…las) From a 10-trial per-benchmark probe of gpt-5.4-mini vs gpt-oss-120b vs deepseek-v4-flash: deepseek matches/beats both on tau3 (0.875) and is ~2-3x gpt-oss on the grounded-reasoning benchmarks (officeqa/browsecomp 0.60) at roughly gpt-oss cost and far below mini. It is weaker only on swe-atlas (0.30 vs gpt-oss 0.59 mean rubric), which is pinned to gpt-oss-120b — cheaper and stronger there. Updates each benchmark's target `model` and the evaluation-scope `allowed_models` (kept with the fireworks_ai/ prefix, since agents strip only the openai/ prefix and the gateway matches the sent model exactly). Producer scopes and tau3's user/assertion models are unchanged. Also hardens the swe-atlas atlas agent: a shell command emitting non-UTF-8 (binary) output raised UnicodeDecodeError and killed the whole trial; it now returns a tool error so the model can retry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… relock - .gitignore: remove the paper-planning benchmark-scoping.md ignore - README.md: correct candidate paths (candidates/ -> top level), add browsecomp-plus, drop the stale nested table - vero/uv.lock: relock Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fireworks' shared-org rate limit surfaces as a RateLimitError inside the agent's OpenAI client; harbor only retries infra/env failures, not LLM 429s, so a burst lost trials. The SDK's default 2 retries were exhausted under sustained load. Bump to 8 so transient 429s back off in-place instead of failing the trial (durable fix is gateway-level retry, tracked separately). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…line Full held-out (test) pass of the seed harness, 3 independent rounds, under the new targets (deepseek-v4-flash; gpt-oss-120b on swe-atlas): swe-atlas 0.097 ±0.011 (agg_score 0.632) [gpt-oss-120b] tau3 0.611 ±0.021 [deepseek-v4-flash] officeqa 0.360 ±0.042 [deepseek-v4-flash] browsecomp 0.449 ±0.007 [deepseek-v4-flash] Each pinned into its target's baseline_reward with score_baseline: false, so finalization uses the number instead of re-scoring the seed every run. The three deepseek benchmarks logged zero exceptions over 945 trials (max_retries absorbed the shared-quota 429s); swe-atlas lost 5/150 to gpt-oss 128k overflow. CONFIGURATION.md documents the numbers, method, and the swe-atlas binary-vs- agg_score note. gaia deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
It was the suite's only Family B task and the only one that bypassed the compiler: a hand-written task directory with no build.yaml, so it got none of the build-time validation the other five benchmarks get. Its optimizer was also unmetered -- no gateway container, raw upstream credentials in main, no model allow-list -- which is a fair-play hole in a benchmark whose whole point is controlled measurement. Its sidecar factory had additionally gone stale against SidecarEvaluationPolicy and would have died at startup. The nested-backend refactor now supports `evaluation_backend: command`, so a replacement Family B task can be built through the compiler and get the gateway for free. examples/harbor-circle-packing carries a worked version of exactly that shape, verified end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These tests moved here from pr2-add-vero, where they could only ever skip. They had been failing for a while against a pinned literal -- evaluation allowed_models == ["gpt-5.4-mini-2026-03-17"] -- that stopped matching when the benchmarks switched to fireworks-served targets. Rewrite the assertions so they cannot drift that way again: the evaluation scope must allow exactly [config.model], derived rather than pinned. That is the real invariant, and it now matches the build-time validator that rejects a model outside its scope's allow-list. Also drop the skip guard, so a missing harness-engineering-bench fails rather than passing vacuously, and extend coverage from three benchmarks to all five. 415 passed, 5 skipped -- the three long-standing failures are gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 26, 2026 05:02
ced19f9 to
20dd166
Compare
WIP scaffold: a working agent + build/config skeleton for a SWE-bench-Pro optimization target, mirroring the gaia baseline layout. Two things are stubbed and must be completed before it runs: (1) the task_source digest in baseline/build.yaml is a clearly-marked placeholder that must be pinned to the real SWE-bench-Pro Harbor task package, and (2) the partitions/ files are empty placeholders that must be regenerated from the real task list via scripts/partition_swe_bench_pro.py. The agent wraps responses.create in a retry-with-backoff helper from the start (the unguarded call is what scored the gaia baseline 0.0). See harness-engineering-bench/swe-bench-pro/README.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The agents strip only the `openai/` prefix before calling the gateway
(`removeprefix("openai/")`), and the gateway matches the allow-list exactly
(`model not in scope.allowed_models` -> 403 model_denied). So `model:
openai/gpt-4o` puts `gpt-4o` on the wire, and an allow-list carrying the
prefixed spelling never matches. Listing both spellings papered over that
and then fails the config invariant that the evaluation scope allow exactly
the target model.
Drop the prefix on both sides, and template the producer list the way every
other benchmark does so `--model` / `--param optimizer_model` keeps working
without a config edit.
This config still cannot be swept by tests/test_v05_benchmark_configs.py:
its partitions are deliberately empty pending real task ids, so it fails to
load. It should join BENCHMARKS there once the partitions and the task
source digest are pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A model that is configured but not deployed upstream is invisible until the run is over. The gateway forwards the request, the upstream 404s, the agent makes no progress, and every case is scored 0.0 and labelled `task_failure` — a run that looks honest and says the candidate failed. That is how swe-atlas-qna produced 469 zero-score cases across two runs (#51): `gpt-5.4-mini-2026-03-17` is not provisioned on the configured Azure resource, so 322/322 evaluation-scope and 79/79 finalization-scope requests returned 404 DeploymentNotFound while the producer scope ran clean. `vero harbor run` now sends one 16-token request per distinct allow-listed model before compiling the task, and aborts if any comes back 404. Only a definitive 404 is fatal. A timeout, a 429 or a 503 is inconclusive and is reported without blocking, so a transient upstream blip cannot stop a run that would have succeeded. Names are probed with the provider prefix stripped, which is the form the agents actually send. Verified against the live endpoint: gaia (which points at the undeployed model) now exits 1 in about two seconds with the DeploymentNotFound body quoted, instead of burning ~30 minutes and ~$2; a gpt-5.3-codex producer with a gpt-4o evaluation scope passes. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#53 stopped a dead sandbox being scored as an informative zero. The same masking survives one layer up, for a cause that is even easier to fix. In the 2026-07-25 swe-atlas-qna run, 145 of 469 cases ended on: NotFoundError: Error code: 404 - {'error': {'type': 'invalid_request_error', 'code': 'DeploymentNotFound', 'message': 'The API deployment for this resource does not exist. ...'}} The configured eval model is not provisioned on the Azure resource. The agent's every call 404s, it writes no answer, and `classify_case` — finding nothing in the signal it recognises — returns TASK_FAILURE, whose policy is `is_informative_sample=True`. So each case was recorded as a genuine score of 0.0 and the harness reported that the candidate failed the task. Add UPSTREAM_MODEL_UNAVAILABLE: never an informative sample, never retried, terminating (every remaining case fails identically, so there is nothing left to measure), and counted toward invalidity so the aggregate still comes out invalid if the terminating path is ever bypassed. It is ranked above TRANSIENT_INFRA, so a case that saw both a dead sandbox and a missing deployment reports the deterministic, operator-fixable cause. The pattern deliberately matches the provider error codes and the exact Azure sentence rather than a bare "does not exist": 102 cases in that same run failed with `FetchSpec failed: loading container: file does not exist`, which is genuine infrastructure and must stay TRANSIENT_INFRA. There is a test pinning both sides. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The harbor backend's error taxonomy records any unrecognized case failure as TASK_FAILURE: an informative, scoreable sample at the failure value with status SUCCESS. That is right for a candidate that crashes on its own, but Modal sandbox lifecycle failures and a missing held-out tests fixture are infrastructure the candidate did not cause. Seen on a swe-atlas-qna run (2026-07-24): all 396 cases returned status=success, error_category=task_failure, score=0.0. Terminal exceptions were AddTestsDirError x144, Modal sandbox NotFound x125, RuntimeError x67, BadRequestError x60. Classified as task failures, the aggregate reported error_rate 0.0 and objective feasible at 0.0, so a fully broken environment looked like a real, shippable zero and the optimizer ran with no gradient (baseline and every candidate 0.0). Add the harness/Modal environment-loss signals (sandbox lifecycle, AddTestsDirError / "failed to add tests directory", container fetch failures, StreamTerminatedError) to the TRANSIENT_INFRA patterns. Those cases are now excluded from the aggregate and counted toward invalidity, so a collapsed environment surfaces as an invalid run instead of a silent zero. A candidate's own bug (ValueError, KeyError, no answer) is still a task failure, unchanged. Covered by two new tests; existing behavior asserted unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The probe only spoke the Responses API and treated any 404 as proof the deployment was absent. A route-level 404 (the upstream does not implement that path) and a model-level 404 (the deployment does not exist) share a status code and mean opposite things, so the preflight would hard-block a run against a Chat-Completions-only upstream that would have succeeded. That is now a live risk in this repo: officeqa, tau3, swe-atlas-qna and browsecomp-plus were ported to Chat Completions, leaving gaia as the only agent still on Responses. Try both routes, and only call a model missing when the 404 body names the model (`DeploymentNotFound`, `model_not_found`, or "model ... does not exist") rather than the route. If every route 404s on the route itself, the result is inconclusive and the run proceeds. Same fix for the model name: the provider prefix routes on a proxy and is meaningless to a single-provider endpoint, so probe the configured spelling first and fall back to the bare one before reporting anything missing. Previously the prefix was stripped unconditionally, which is wrong for `fireworks_ai/...` names. Verified against the live Azure endpoint: `gpt-4o` 200 on both routes; `openai/gpt-5.3-codex` 404s prefixed and passes preflight via the bare fallback; `fireworks_ai/gpt-oss-120b` is still correctly blocked; and a model 404 body is distinguished from a bare route 404. Refs #51. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shehabyasser-scale
force-pushed
the
feat/swe-bench-pro-baseline-scaffold
branch
from
July 26, 2026 06:28
ccc7d07 to
ec708e9
Compare
Collaborator
Author
|
Rebased onto the current Added one commit on top, fixing a real defect the new invariants exposed. The config had: model: openai/gpt-4o
producer: allowed_models: ["gpt-5.3-codex", "openai/gpt-5.3-codex"]
evaluation: allowed_models: ["gpt-4o", "openai/gpt-4o"]The dual-spelling lists were a workaround for a mismatch that should not exist. Agents strip only Two things a reviewer should know:
Full suite matches |
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 26, 2026 06:57
20dd166 to
014eb72
Compare
fix: a missing upstream deployment is not a task failure
varunursekar
force-pushed
the
pr3-harness-bench
branch
2 times, most recently
from
July 26, 2026 16:52
5397fe4 to
ec0b444
Compare
feat: preflight configured models before launching an optimizer trial
fix: classify Modal sandbox / tests-dir loss as transient_infra
varunursekar
force-pushed
the
pr3-harness-bench
branch
from
July 27, 2026 00:51
505df1a to
2e11b14
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP scaffold (draft). Adds a SWE-bench-Pro baseline under
harness-engineering-bench/swe-bench-pro/, mirroring the gaia baseline layout, so the vero optimizer can hill-climb a patch-style SWE agent. Not runnable yet: two pieces are stubbed (see below).What's here (working skeleton)
baseline/target/src/swebench_pro_agent/agent.py: code-editing agent (SweBenchProAgent, Responses API, MAX_TURNS 50), toolsrun_shell/read_file/write_file/apply_patch/run_tests/submit. Edits the repo in place; the task-source verifier runs the test suite for reward (no self-grade, no answer.txt). Theresponses.createcall is wrapped in a retry-with-backoff helper from the start (the gaia baseline scored 0.0 because that call was unguarded and transient errors crashed it, the optimizer's winning fix was exactly this retry).baseline/build.yaml:agent_import_path,harbor[modal]==0.20.0, inference-gateway (producergpt-5.3-codex, evalgpt-4o, both prefixed and bare forms), timeouts bumped for repo builds + test suites, W&Bvero-swe-bench-pro.baseline/target/pyproject.toml:requires-python = ">=3.11,<3.14"(avoids the litellm/pyo3 3.14 build wall).secrets.env.example,.gitignore, andscripts/partition_swe_bench_pro.py(deterministic sha256 stratified split +--check).Stubbed (must finish before it runs)
task_source(placeholderscale-ai/swe-bench-pro@sha256:REPLACE_WITH_REAL_DIGEST). Get the SWE-bench-Pro Harbor task package (repos + failing tests + gold patches + grading) from the Harbor task registry / DEX-harness team.scripts/partition_swe_bench_pro.py, fillagent_access.total_cases, and runuv lockinbaseline/target/.How to run (once the stubs are filled)
--environment modalneeds a Modal-reachable model endpoint (e.g. the Azure…openai.azure.com/openai/v1endpoint), not a VPN-internal one.🤖 Generated with Claude Code