Speak to a Protein — reproduction repo

This repo reproduces the empirical results in the Speak to a Protein paper against the public deployment at https://open.playmolecule.org:

1. BixBench-Verified-50 — an automated, browser-driven harness that runs all 50 tasks through the live system, grades them with a deterministic verifier (mirroring upstream BixBench's str_verifier), and reports accuracy.
2. Case studies (CDK2, D3R) — the exact prompts to paste into the chat, with expected outcomes, for interactive reproduction. See docs/case-studies.md.

The model is fixed server-side on the open deployment, so every run uses the same agent the paper describes; there is nothing to configure on that front.

Install

uv sync
uv run playwright install chromium
cp .env.example .env                     # tweak if you want a non-default project name

You need an account on https://open.playmolecule.org (free sign-up). Grading is deterministic (see Grading below).

Reproduce BixBench-Verified-50

1. Authenticate (once)

uv run stap setup-auth

A browser window opens at the login page. Log in normally; the session is saved to .auth/storage_state.json and reused by later runs.

2. Fetch the task data (once)

The 50 tasks ship with input files ("capsules") that are too large to commit. Download them from Hugging Face:

uv sync --extra fetch
uv run python scripts/fetch_bixbench.py

This populates eval_sets/bixbench-verified/fixtures/ (~6 GB) and strips the answer-leaking notebooks so the agent never sees ground truth.

3. Run, grade, report

# Smoke-test a single case first.
uv run stap run --case bix-11-q1 --auto-grade
uv run stap report <run_id>

# Full benchmark. --repeats N reruns every case N times for error bars.
uv run stap run --repeats 3 --auto-grade
uv run stap report <run_id>

run executes the agent and writes one cell per (case, repeat); grade scores the answers; report rolls them up into accuracy (mean ± std across repeats). <run_id> is printed at the end of run. Re-grading is free; re-running the agent is not.

CLI

Command	Purpose
setup-auth	Open a browser, log in manually, save the session.
run	Run the (case × repeat) matrix; write artifacts under runs/<run_id>/.
grade	Grade an existing run. Re-runnable with --force.
report	Render an accuracy report from a graded run.

Grading

The harness uses a deterministic grader: it extracts the agent's <answer>...</answer>, alphanumeric-normalises both sides, and accepts an exact match or two-way substring containment. This mirrors upstream BixBench's str_verifier.

Every prediction is saved to runs/<run_id>/<case>/repeat-<k>/final_answer.md so you can re-grade externally with any tool you prefer.

Layout

src/stap/          harness code (browser, runner, grade, report, cli)
eval_sets/         bixbench-verified/cases.yaml + (fetched) fixtures
scripts/           fetch_bixbench.py
case_studies/      session traces (cdk2.json, d3r.json) + the files each session produced
docs/              how-it-works.md, case-studies.md
runs/              run artifacts (gitignored)

See docs/how-it-works.md for the pipeline tour.