Issues

• Tracking: agent use. A third question for the suite: when an AI agent simulates a model with each engine, how often is it right, and how often wrong without knowing?

  enhancement
  lanl/bngsim-benchmarks#17 · Filter by author wshlavacek opened on Nov 13, 2026

• Agent pilot: ten tasks, bngsim and libRoadRunner, two arms, one agent, three repetitions, then the same tasks on every bngsim release

  enhancement
  lanl/bngsim-benchmarks#16 · Filter by author wshlavacek opened on Nov 13, 2026

• Agent harness: a job is a task × an engine × a documentation arm × an agent × a repetition, run in a pinned container with the transcript kept

  enhancement
  lanl/bngsim-benchmarks#15 · Filter by author wshlavacek opened on Nov 13, 2026

• Agent scoring: an outcome vocabulary whose headline is the silently wrong answer, reported beside the completion rate, with causes coded from transcripts

  enhancement
  lanl/bngsim-benchmarks#14 · Filter by author wshlavacek opened on Nov 13, 2026

• Agent tasks: a problem type with a plain-language request and stored reference values, in seven categories, with traps and a held-out set

  enhancement
  lanl/bngsim-benchmarks#13 · Filter by author wshlavacek opened on Nov 13, 2026

• Agent comparison: fix the protocol before the first run (tasks, prompts, arms, engine versions, what is reported), so the result can be believed either way

  enhancement
  lanl/bngsim-benchmarks#12 · Filter by author wshlavacek opened on Nov 13, 2026