Add doctest validation stage to `Callable` decoding - #694

Draft

eb8680 wants to merge 167 commits into

eb-doctest-synthesis

Commits on Feb 8, 2026

add dummy tools
eb8680
committed
d65dc4e
Add Agent class
eb8680
committed
fa3fb6b
stash examples
eb8680
committed
eca5cb8

Commits on Apr 25, 2026

no changes to library code
eb8680
committed
310e3db
no agent test
eb8680
committed
136b3a8
Merge branch 'master' into eb-llm-examples
eb8680
committed
3b32604
updates
eb8680
committed
52dc30e
notebook sections as scripts
eb8680
committed
2cbb0e0
lint
eb8680
committed
ed3d16b
nit
eb8680
committed
3cac33d
retry
eb8680
committed
ee6fc1f

Commits on Jun 2, 2026

Add example using object handles (#597)

jfeser
and
eb8680
authored
c2f6bbc

Commits on Jun 19, 2026

Synthesis-based template handling
eb8680
committed
c3555c4

Commits on Jun 20, 2026

doctest
eb8680
committed
1eaa685
Add idiomatic codeadapt example (#695)
eb8680
authored
d5129c8

Commits on Jun 22, 2026

observability
eb8680
committed
70afc43

Commits on Jun 23, 2026

Support bound method Templates
eb8680
committed
598c687

Commits on Jun 29, 2026

cleanup
eb8680
committed
ad2e585
add name and signature to prompt_template and tool
eb8680
committed
ce03b30
add serialization-only schema for callable and add return schema to tool description
eb8680
committed
9c775f3

Commits on Jun 30, 2026

Assemble system prompt
eb8680
committed
00afd96

Commits on Jul 1, 2026

many simplifications
eb8680
committed
e17af14
Add terminal renderer
eb8680
committed
eecd8ec
rename system prompt block helpers to avoid renderer confusion
eb8680
committed
991be62
Tweak system prompt
eb8680
committed
cc0b742
Remove IsRecursive
eb8680
committed
d515a41
include docstring
eb8680
committed
0ba7274

Commits on Jul 2, 2026

move around
eb8680
committed
1a1f42f

Commits on Jul 11, 2026

Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
f5d0440

Commits on Jul 13, 2026

formatting nits
eb8680
committed
c26081f
formatting nits
eb8680
committed
600f058

Commits on Jul 15, 2026

Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
a44624b

Commits on Jul 19, 2026

fix failed merge
eb8680
committed
8c11615
split into more files
eb8680
committed
6b665aa
Add a library-level harness.py
eb8680
committed
395c51f
Merge branch 'eb-llm-examples' into eb-doctest-synthesis
eb8680
committed
3b0a01f
standardize harness
eb8680
committed
a3cf92b
cleanup examples
eb8680
committed
b0fd2cd
nits
eb8680
committed
234c059
fix docstring
eb8680
committed
9902b84
tool choice in harness
eb8680
committed
2cc4c35
contextvar
eb8680
committed
d0f0d4b
remove dumb example
eb8680
committed
ad8afa8
image
eb8680
committed
bcb5a0c

Commits on Jul 20, 2026

fix synthesisfinaltool tool collection bug
eb8680
committed
1c5cdc6
remove hanoi hints
eb8680
committed
969c703
fix bug
eb8680
committed
e6f5c71
nit
eb8680
committed
6e64028
comment about nasty bug
eb8680
committed
ed6de7d
Move Encodable to template.py and fix lint
eb8680
committed
bb67580
nit
eb8680
committed
8c7b9b0
revert
eb8680
committed
4cf864e
minor
eb8680
committed
958415d
Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
44c251a
repl_history
eb8680
committed
22bd549

Commits on Jul 23, 2026

Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
b244167
cleanup
eb8680
committed
a623637
force type checking of submit_solution
eb8680
committed
62ebd45
remove trivial majority_vote
eb8680
committed
fda9049

Commits on Jul 24, 2026

many fixes
eb8680
committed
867939d
fix no-tool-call error
eb8680
committed
7ecf823
fix prompts
eb8680
committed
998c59a
fix bug in method templates that synthesize code
eb8680
committed
2537af0
fix test bugs
eb8680
committed
4717f2f
add schema-harness example in world_model_agent.py
eb8680
committed
7f9476a
add pdb and reasoning-effort options to harness
eb8680
committed
a348f8a
help = doc
eb8680
committed
f349948
explore
eb8680
committed
7ba88ac
consolidate hanoi
eb8680
committed
9763421
add a summary comment
eb8680
committed
b7d859a

Commits on Jul 25, 2026

reorganize examples
eb8680
committed
53dea20
remove bad example
eb8680
committed
c944e7d
format
eb8680
committed
587fe8a
nit
eb8680
committed
96edcbe

Commits on Jul 26, 2026

Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
a44f5d8
fix nested agent history
eb8680
committed
23cb7e4

Commits on Jul 28, 2026

port persistent agent (#604)
eb8680
committed
61be1b6
add more aime examples and consolidate into one file
eb8680
committed
86d1c51
Port multi-agent choreography (#607) to asyncio

eb8680
and
claude
committed
6730683
Move multi-agent example under llm_examples

eb8680
and
claude
committed
cb069f1
Replace the TaskQueue backends with asyncio futures and a Queue

eb8680
and
claude
committed
cca3fb2
Rename multi.py to choreographies.py

eb8680
and
claude
committed
0f23524
Add StepLog, a write-ahead log of completed steps, for resume

eb8680
and
claude
committed
2000219
Give reviewers a run_tests tool, so the fix loop turns on a fact

eb8680
and
claude
committed
d05cf18
Drop Choreography's handlers argument

eb8680
and
claude
committed
bf500e2
Make step, call and scatter Operations handled by EndpointProjection

eb8680
and
claude
committed
8569080
Fold call into step, and inline _first_exception and replay

eb8680
and
claude
committed
b3fd65a
Remove Choreography.projection

eb8680
and
claude
committed
8635b9d
Absorb StepLog into _Steps; Choreography takes a log path

eb8680
and
claude
committed
8b2be03
Drop recorded_steps; fold _connect into _Steps; trim the docstring

eb8680
and
claude
committed
8e154c2
Make Choreography generic in its program's signature, and callable

eb8680
and
claude
committed
d894aac
Give run_doctests a default rule, and make RestrictedEvalProvider usable

eb8680
and
claude
committed
622979a
Let the harness choose which provider runs model-authored Python

eb8680
and
claude
committed
6094ecc
Stop re-validating a callable against synthesis constraints when encoding it

eb8680
and
claude
committed
4539b77
Fix two examples that could not run at all

eb8680
and
claude
committed
c728275
Merge branch 'eb-doctest-synthesis' into worktree-restricted-eval-doctests
eb8680
committed
1335319

Commits on Jul 29, 2026

Add test checking resolution of issue #576
eb8680
committed
4f40407
harness module
eb8680
committed
3485df4
Renaming, part 1
eb8680
committed
7fea052
nit
eb8680
committed
e1c26b2
nit
eb8680
committed
2a8d0c6
execution module
eb8680
committed
9a89558
persistence module
eb8680
committed
6a307d5
observability.py
eb8680
committed
a2d3272
tracing
eb8680
committed
aead502
completions
eb8680
committed
58eb532
retrying.py
eb8680
committed
3eaef0c
choreographies.py
eb8680
committed
190f0d6
TenacityRetryer
eb8680
committed
7b22406
rendering.py
eb8680
committed
9ca087d
encoding.py
eb8680
committed
524372b
persisting.py
eb8680
committed
6fe53c8
choreographing.py
eb8680
committed
3130fa7
scoping.py
eb8680
committed
f4a5189
coding.py
eb8680
committed
b1fb146
providing.py
eb8680
committed
b96d8ae
hooks.py
eb8680
committed
a1945e9
hooks.py
eb8680
committed
1a52e83
nit
eb8680
committed
0bfd814
scoping
eb8680
committed
7260ece
unsafe.py
eb8680
committed
09232b7
restricted.py
eb8680
committed
8b85965
unsafe -> builtin
eb8680
committed
563f442
restricted
eb8680
committed
6514222
nits
eb8680
committed
613a89f
executing
eb8680
committed
8fd3660
rename
eb8680
committed
a388657
fix tests
eb8680
committed
a0697f6
Add ClaimCheck llm_example: auditing proof against intent (#660)

eb8680
and
claude
committed
aa16590
Sharpen the ClaimCheck corpus; report that the ablation does not separate

eb8680
and
claude
committed
6ae402d
Add optimize_anything example: reflective Pareto search over typed artifacts

eb8680
and
claude
committed
4e38c73
__main__
eb8680
committed
cf0d3cb
Run the ablation on older models at temperature 0; record the result

eb8680
and
claude
committed
1562eac
Fix an answer-key leak into every agent's prompt; withdraw the old results

eb8680
and
claude
committed
92c22d6
Impoverish the naive arm to match upstream; restore the truncated taxonomy

eb8680
and
claude
committed
bb33b7a
remove task from investigation.py workspace
eb8680
committed
26aa158
Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
dc8ed00
fix failing doctest
eb8680
committed
2aed5de
remove obsolete notebook
eb8680
committed
b81172a
Rebuild the corpus around opaque-invariant projections; the gap still does not reproduce

eb8680
and
claude
committed
ce245cd
pack: follow the paper's circle-packing setup instead of approximating it

eb8680
and
claude
committed
01855e0
Split the optimize_anything example into a package of three scripts

eb8680
and
claude
committed
6ea0d28
Rebuild the ClaimCheck corpus as a 1:1 port of upstream's Dafny benchmark

eb8680
and
claude
committed
2e5af20
Fix port fidelity issues found in review, and isolate the batching mechanism

eb8680
and
claude
committed
ef58e66
Correct false claims and measurement defects in prompting.py and kernels.py

eb8680
and
claude
committed
c955d07
Fix two more measurement defects; retract the caveat ablation and its mechanism

eb8680
and
claude
committed
4448adf
Strip working history from the example; refresh numbers against shipped code

eb8680
and
claude
committed
2138d99

Commits on Jul 30, 2026

Add the missing control, and fix what the numbers were claiming

eb8680
and
claude
committed
e469905
Record the batching measurement; correct the matched-model comparison

eb8680
and
claude
committed
31b9d77
Neutralise single-pass module text; attribute the strong baselines to the model

eb8680
and
claude
committed
e0cf7c5
Withdraw the model-capability explanation for the strong baselines

eb8680
and
claude
committed
31a37de
Reconcile the two-pass deficit against upstream's same-model runs

eb8680
and
claude
committed
319a881
Remove a rule leak from the naive arm's prompt; verify all three from live capture

eb8680
and
claude
committed
b4694a1
Correct the claim that all three arms differ by signature

eb8680
and
claude
committed
c2067f8
Correct the single-pass attribution and the return-type mechanics

eb8680
and
claude
committed
4465ef8
Note that upstream measures the single-pass/naive contrast at zero

eb8680
and
claude
committed
d49acae
Drop the single-pass arm; the ablation is two-pass against naive

eb8680
and
claude
committed
c82aaca
Cut the example header to a description of the problem and results

eb8680
and
claude
committed
e726536
Measure the outstanding ablations; none of the three claims reproduce

eb8680
and
claude
committed
75c9896

Commits on Jul 31, 2026

optimize_anything example
eb8680
committed
dd6ab94
Move the formalization examples into llm_examples/autoformalization

eb8680
and
claude
committed
d057a18
claimcheck
eb8680
committed
fd52fc1
rename multi_agent -> multi_agent_choreography
eb8680
committed
fe99ee2
move system prompt builders to scoping
eb8680
committed
b1779ba
move choreographies to examples
eb8680
committed
77f1b74
rename files
eb8680
committed
e9ef53c
Merge branch 'master' into eb-doctest-synthesis
eb8680
committed
de773ab