feat: make the SWE-bench-Pro baseline runnable for the vero optimizer - #50

Open

shehabyasser-scale wants to merge 45 commits into

pr3-harness-benchfrom

feat/swe-bench-pro-baseline-scaffold

Commits on Jul 26, 2026

Add harness-engineering-bench (benchmarks at top level)
varunursekar
committed
gaia: baseline_floor off (feedback #1)
varunursekar
committed
gaia: re-enable harness isolation (harness_user: harness)
varunursekar
committed
Bump harbor to 0.20.0 across benchmark configs and regenerate locks

varunursekar
and
claude
committed
Benchmarks: adopt .evals context naming and the evals CLI

varunursekar
and
claude
committed
Benchmarks: run inner eval sandboxes in a dedicated Modal app
varunursekar
committed
swe-atlas-qna: submit-mode selection, matching gaia
varunursekar
committed
Benchmarks: enable W&B reporting on officeqa, swe-atlas-qna, and tau3
varunursekar
committed
swe-atlas-qna: route the in-container rubric judge to the upstream
varunursekar
committed
officeqa: merge duplicate extra_harbor_args keys
varunursekar
committed
officeqa: submit-mode selection, matching gaia
varunursekar
committed
Benchmarks: bound orphaned infrastructure from killed runs
varunursekar
committed
Benchmarks: normalize optimization task configurations
varunursekar
committed
Benchmarks: document the normalized configuration
varunursekar
committed
swe-atlas-qna: keep Modal sandboxes alive under the images' ENTRYPOINT
varunursekar
committed
Add post-hoc per-trial token aggregator
varunursekar
committed
Docs: note infra retry/exclusion is trusted-only in benchmark config
varunursekar
committed
Docs: add target model as a per-benchmark field in CONFIGURATION.md
varunursekar
committed
swe-atlas: port baseline agent to Chat Completions; best-effort answer at turn cap
varunursekar
committed
browsecomp-plus: reduce partitions to 33/66/66 (in line with the suite)
varunursekar
committed
officeqa + tau3: port baseline agents to Chat Completions
varunursekar
committed
officeqa: gate parallel_tool_calls to non-fireworks models (as reasoning_effort)
varunursekar
committed
officeqa: best-effort answer at turn cap instead of raising
varunursekar
committed
Add BrowseComp-Plus benchmark (deep-research over a fixed BM25 corpus)
varunursekar
committed
browsecomp-plus: placeholder OPENAI_API_KEY so pyserini's BM25 import succeeds
varunursekar
committed
browsecomp-plus: pass judge credentials via [verifier.env]
varunursekar
committed
Set deepseek-v4-flash as default target model (gpt-oss-120b on swe-atlas)

varunursekar
and
claude
committed
Housekeeping: drop stale gitignore note, fix README candidates table, relock

varunursekar
and
claude
committed
Add in-place 429 retry (max_retries=8) to the four baseline agents

varunursekar
and
claude
committed
nit: style
varunursekar
committed
Pin held-out baselines (K=3) into baseline_reward; disable score_baseline

varunursekar
and
claude
committed
Drop the ALE-Bench benchmark

varunursekar
and
claude
committed
Assert benchmark-config invariants from the configs themselves

varunursekar
and
claude
committed
feat: scaffold SWE-bench-Pro baseline for the vero optimizer

shehabyasser-scale
and
claude
committed
fix: spell swe-bench-pro's models the way the gateway matches them

shehabyasser-scale
and
claude
committed
feat: preflight configured models before launching an optimizer trial

shehabyasser-scale
and
claude
committed
fix: a missing upstream deployment is not a task failure

shehabyasser-scale
and
claude
committed
fix: classify Modal sandbox / tests-dir loss as transient_infra

shehabyasser-scale
and
claude
committed
fix: do not read a missing route as a missing model in preflight

shehabyasser-scale
and
claude
committed
Merge pull request #60 from scaleapi/fix/classify-missing-deployment
varunursekar
authored