diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index d3fe54d..5d4a52f 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -3,6 +3,20 @@ What advanced physics workflow does this task represent, and who performs this work? +## Task history + +This task must come from research you personally carried out, and which took +you two weeks or more. Give real dates and an honest hour count. + +| Field | Value | +|---|---| +| What the original work was | | +| Start date | YYYY-MM-DD | +| End date | YYYY-MM-DD | +| Working hours spent | | +| Did an LLM agent help, and where? | | +| Your background | PhD / PhD candidate / lab or industry experience | + ## Task | Field | Value | @@ -28,8 +42,13 @@ work? - [ ] No-skill and with-skill runs use the same task commit and model settings - [ ] At least one strong agent passes the with-skill solvability control - [ ] Trajectories and output artifacts were inspected +- [ ] The task comes from my own research and took two weeks or more +- [ ] This PR is from a fork and touches only `tasks//` + +## Local test results -## Results +Report multiple trials per condition, not a single run. If a trial set was cut +short, say so and report what finished. | Agent | Model | Reasoning | No skill (primary) | With skills (control) | Time | |---|---|---|---:|---:|---:| @@ -40,6 +59,11 @@ work? Explain whether failures came from scientific reasoning, environment/tooling, instructions, formatting, or verifier behavior. +## What you learned building it + +Anything a reviewer or future contributor should know — a leaky environment, a +brittle tolerance, a metric that turned out uninformative. + ## Artifacts Include oracle output, verifier logs, trajectories, and any visual or binary diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 44e34e1..a677779 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,22 +23,30 @@ once the task it reviewed is merged. Quality beats quantity — one excellent ta is worth more than many mediocre ones, and a submission that does not clear the bar in [What makes a good task](#what-makes-a-good-task) earns nothing. -# What makes a good task -A good task must satisfy three requirements: -1. Representative -It must come from workflows used in real Physics research. +# Who should contribute +A PhD or current PhD candidate in physics, EECS, or an adjacent field — or +someone with extensive hands-on experience in a physics lab or an equivalent +industry role. -2. Complex -Require substantial domain expertise and effort. An agent without mentoring skills have to use at least 80 tool calls and over 100 steps, and is very likely to fail the task. +# What makes an ideal task +Three things: -3. Verifiable -The task should have deliverables that can be deterministically graded. +1. **Your own work.** Real research you personally carried out, not a problem + invented for the benchmark. +2. **Two weeks or more.** It took you at least two weeks of genuine effort, + with or without an LLM agent helping. +3. **Verifiable.** The result is right or wrong, and a script can tell which. + +A task that misses any one of these will not merge. # How to contribute -1. **Ideate**: Pick a domain where you have real expertise and choose a project you have done before that satisfy the above three requirements for a good task. -2. **Create**: Implement a task package. You can refer to .. for examples of a good task, and you can refer to the `Task Package` section below for the strcture of a task. -3. **Test**: Run the oracle and at least one agent with and without skills. -4. **Submit**: Open a PR using the required checklist below. +1. **Ideate**: Pick a project that meets all three. Bring it to group chat or + confirm with a maintainer before you build. +2. **Create**: Implement the task package. See `Task Package` below. +3. **Test**: Run the oracle, then run at least one agent with and without + skills. +4. **Submit**: Fork this repository and open a PR against `main` here. See + [PR Requirements](#pr-requirements). # Task Package Technically, a task consists of: @@ -120,17 +128,63 @@ Reviewers look for: - **Verification**: deterministic, outcome-based, anti-cheat aware. - **Instructions**: concise, fair, no skill hints. - **Environment**: reproducible Docker image, pinned deps, no leaked skills. -- **Complexity**: -the agents use over 100 steps + over 80 tool calls to solve the task + have a high chance of failing agents without skills. +- **Complexity**: took the author two weeks or more, and agents without skills + are likely to fail it. + +# The final submission + +Every task submission consists of three things. + +## 1. A PR from your fork + +Fork this repository, push your task to a branch on your fork, and open a pull +request against `main` here. One task per PR, and the PR should touch only +files under `tasks//`. + +## 2. A detailed PR description + +The description is part of the submission, not a formality — it is the evidence +a reviewer uses to judge provenance and difficulty. Explain the history of the +task: where this work came from and what it cost you. + +| Field | Example | +|---|---| +| What the original work was | the analysis, measurement, or simulation, and why you were doing it | +| Start date | 2025-03-04 | +| End date | 2025-03-28 | +| Working hours spent | approximately 60 hours over three weeks | +| Whether an LLM agent helped | and if so, on which parts | + +Give real dates and an honest hour count. A task that took you two days is not +a fit, and saying so early saves everyone a review cycle. + +Also cover the scientific motivation: what physics the task exercises, who does +this kind of work, and where the data or model came from — with citations and +license provenance for anything you did not produce yourself. -# PR Requirements +## 3. A local test results report -Before opening a PR: +Before you open the PR, confirm all of these locally: 1. `bench tasks check tasks/` passes. 2. `bench eval run --tasks-dir tasks/ --agent oracle --sandbox docker` passes with reward 1.0. -3. At least one agent has been tested with and without skills. -4. The PR description includes pass rates, failure analysis, and artifacts for - multimodal outputs. -5. The task prompt, oracle, skills, tests, and metadata are ready for human review. +3. At least one agent has been run both with and without skills. +4. The task prompt, oracle, skills, tests, and metadata are ready for human + review. + +Then report what you actually ran: + +- oracle result, showing reward 1.0; +- a table of agent runs — agent, model, with-skill and no-skill pass rates over + multiple trials, not a single run; +- failure analysis: whether failures came from scientific reasoning, + environment or tooling, instructions, formatting, or verifier behaviour; +- artifacts for any multimodal or binary outputs; +- anything you discovered while building it that a reviewer or future + contributor should know — a leaky environment, a brittle tolerance, a metric + that turned out uninformative. + +Report the runs you completed. If you ran out of credits partway through a +trial set, say so and report what finished; partial evidence honestly labelled +is worth more than a padded table. diff --git a/website/src/app/contribute/page.tsx b/website/src/app/contribute/page.tsx index 9938453..10c58d0 100644 --- a/website/src/app/contribute/page.tsx +++ b/website/src/app/contribute/page.tsx @@ -4,8 +4,8 @@ import { ArrowUpRight, Award, Check, + Clock, FlaskConical, - Layers, ShieldCheck, X, } from "lucide-react"; @@ -20,51 +20,72 @@ export const metadata: Metadata = { const CRITERIA = [ { - title: "Representative", - body: "It comes from a workflow used in real physics research — something you or your group actually had to do.", - check: "Have I done this myself?", + title: "Your own work", + body: "Real research you carried out, not a problem invented for the benchmark.", + check: "Did I do this myself?", icon: FlaskConical, accent: "text-chart-1", tint: "bg-chart-1/10", }, { - title: "Complex", - body: "It needs substantial domain expertise. An agent without mentor skills should need 100+ steps and 80+ tool calls, and should be likely to fail.", - check: "Would this take a new student days?", - icon: Layers, + title: "Weeks of effort", + body: "At least two weeks of genuine effort, with or without an agent helping.", + check: "Did it really take weeks to finish?", + icon: Clock, accent: "text-chart-2", tint: "bg-chart-2/10", }, { title: "Verifiable", - body: "The deliverables can be graded deterministically — numbers, files, and artifacts a test can check without a human in the loop.", - check: "Can a script tell right from wrong?", + body: "The result is right or wrong, and a script can tell which.", + check: "Can a script grade it?", icon: ShieldCheck, accent: "text-chart-3", tint: "bg-chart-3/10", }, ]; +const ELIGIBILITY = [ + "A PhD or current PhD candidate in physics, EECS, or an adjacent field", + "Or extensive hands-on experience in a physics lab or an equivalent industry role", +]; + const STEPS = [ { step: "01", title: "Ideate", - body: "Pick a domain where you have real expertise and a project you have already done that meets the three criteria above.", + body: "Pick a project that meets all three. Bring it to group chat or confirm with a maintainer before you build.", }, { step: "02", title: "Create", - body: "Write the task package: the prompt and metadata in task.md, a pinned Docker environment, mentor skills, the oracle solution, and the verifier.", + body: "Write the task package: prompt and metadata, Docker environment, mentor skills, oracle, verifier.", }, { step: "03", title: "Test", - body: "Run the oracle, then run at least one agent both with and without skills so the PR carries real evidence.", + body: "Run the oracle, then at least one agent with and without skills, over multiple trials.", }, { step: "04", title: "Submit", - body: "Open a PR with pass rates, failure analysis, and artifacts for any multimodal outputs.", + body: "Fork the repository and open a pull request against main.", + }, +]; + +const SUBMISSION = [ + { + title: "A PR from your fork", + body: "Fork this repository and open a pull request against main here. One task per PR, touching only files under tasks//.", + }, + { + title: "A detailed PR description", + body: "The history of the task: what the original work was, its start and end dates, and how many working hours you spent. Plus the physics it exercises and where the data came from.", + }, + { + title: "A local test results report", + body: "What you ran and what happened: the oracle at reward 1.0, and agent pass rates with and without skills across multiple trials.", + example: { label: "Example task: PR #2.", href: `${site.repo}/pull/2` }, }, ]; @@ -97,39 +118,25 @@ export default function Contribute() {

Turn research you have already done into a benchmark task

-

- No AI background required. The hard part is the physics, and you have - already done that part. -

-
-
+

Earn {credit.authorship} points, become a co-author

- A task you authored is worth{" "} + A merged task you authored earns{" "} - {credit.task} points - {" "} - when it merges; a task you reviewed is worth{" "} + {credit.task} + + , one you reviewed earns{" "} {credit.review} - . At{" "} - - {credit.authorship} points - {" "} - you are a co-author on the FrontierPhysics paper and the released - dataset — {tasksForAuthorship} authored tasks, or any mix of - authoring and reviewing that adds up. -

-

- Merged is the operative word. Points land on merge, not on - submission, and each task has to clear the bar below. + . {tasksForAuthorship} authored tasks gets you there, as does any + mix that adds up. Points land on merge.

@@ -150,15 +157,29 @@ export default function Contribute() {
-
-

- What makes a good task -

-

- All three at once. A task that misses any one of them will not - merge. -

-
+

+ Who should contribute +

+
    + {ELIGIBILITY.map((item) => ( +
  • +
    • + ))} +
+
+ +
+

+ What makes an ideal task +

+

+ All three. A task that misses any one will not merge. +

{CRITERIA.map((item) => ( @@ -177,7 +198,9 @@ export default function Contribute() {

{item.body}

-

+ {/* Two lines are reserved so the rule sits at the same height in + every card, whether the question wraps or not. */} +

{item.check}

@@ -185,8 +208,7 @@ export default function Contribute() {

- Quality beats quantity — one excellent task is worth more than many - mediocre ones. + One excellent task is worth more than many mediocre ones.

@@ -215,9 +237,7 @@ export default function Contribute() { What you must write yourself

- You can use an AI assistant for the software plumbing. The science - has to be yours — a benchmark built from generated physics measures - nothing. + Use an AI assistant for the plumbing. The science has to be yours.

@@ -265,21 +285,53 @@ export default function Contribute() {

- Before you open the PR + The final submission

- Both commands have to pass, and the oracle has to come back with - reward 1.0. + Three things.

+ +
    + {SUBMISSION.map((item, index) => ( +
  1. + + {String(index + 1).padStart(2, "0")} + +
    +

    {item.title}

    +

    + {item.body} + {item.example ? ( + <> + {" "} + + {item.example.label} + + + ) : null} +

    +
    +
    2. + ))} +
+ +

+ Before you open it, both of these have to pass, with the oracle + returning reward 1.0: + 

           {CHECKS.join("\n")}
-

- Then run at least one agent with and without skills, and put the pass - rates and failure analysis in the PR description. The prompt must - never mention a skill by name, and the verifier must check the - science, not which tools the agent reached for. + +

+ Give real dates and an honest hour count, and report only the runs you + actually completed.

@@ -288,8 +340,7 @@ export default function Contribute() { Ten minutes of triage can save a weekend

- Bring your task idea to Discord before you build it. A maintainer will - tell you quickly whether it clears the bar. + Bring your idea to Discord before you build it.