Skip to content

optimizer: size the autobatcher by the GPU, not by the input - #40

Merged
niklashoelter merged 1 commit into
niklashoelter:masterfrom
avnoordenne:fix/autobatcher-memory-ceiling
Sep 25, 2026
Merged

niklashoelter merged 1 commit into
niklashoelter:masterfrom
avnoordenne:fix/autobatcher-memory-ceiling

Conversation

@avnoordenne

Copy link
Copy Markdown
Contributor

_optimize_batch capped the memory probe with
min(batched_state.n_atoms, max_atoms_to_try). max_atoms_to_try bounds how
much InFlightAutoBatcher may try to fit on the device, so tying it to the
job's atom count made the ceiling a function of how much work was submitted.
InFlightAutoBatcher then raises when any state's metric exceeds that
ceiling. Measured on a Quadro RTX 6000 with a Lennard-Jones stand-in so only
the batching varies:

single structure               ValueError: Max metric ... 18.0 > 17.1
small then large (10, 200)     ValueError: State metric=398.0 > 302.4
3 equal conformers             ok
mixed library (12/40/8/150)    ValueError: State metric=298.0 > 211.2

All four complete after. A single structure failed deterministically: the
probe was capped at one structure's worth of atoms and the 0.95 padding put the
ceiling just under that structure's own metric. Equal-sized conformers were the
shape that survived, which is why this looked intermittent.

Also drops the batcher.load_states() pre-load. torch_sim.optimize calls it
itself on the OptimState, which carries the optimizer's own tensors and so
reflects the real footprint. The pre-load was not just redundant:
_get_first_batch skips the estimate when max_memory_scaler is already set,
so the pre-load pinned the ceiling to a measurement on the bare SimState and
torch-sim never re-measured. Worst with LBFGS, which stores history per system.

Consequence: the "Memory estimation" timing phase disappears, since the probe
now runs inside torch_sim.optimize. TimingCapture.memory_estimation is kept
so examples/full_benchmark.py still runs; it reads 0.0 and the cost lands in
"Optimization". Documented on the property.

Full suite: 244 passed, 3 skipped (no new tests; the four-shape check above was
done by hand).

_optimize_batch capped the autobatcher's memory probe with

    effective_max_atoms = min(batched_state.n_atoms, max_atoms_to_try)

but max_atoms_to_try bounds how much InFlightAutoBatcher is allowed to try
fitting on the device. Tying it to the total atom count of the submitted job
made the memory ceiling a function of how much work was handed in, so a small
job probed a small ceiling. InFlightAutoBatcher then raises outright when any
state's metric exceeds that ceiling.

Checked against the real _optimize_batch path on a Quadro RTX 6000, with a
Lennard-Jones potential standing in for the MLIP so only the batching logic
varies. Before, three of four shapes fail:

    single structure               ValueError: Max metric ... 18.0 > 17.1
    small then large (10, 200)     ValueError: State metric=398.0 > 302.4
    3 equal conformers             ok
    mixed library (12/40/8/150)    ValueError: State metric=298.0 > 211.2

After, all four complete. A single structure in batch mode failed
deterministically: the probe is capped at exactly one structure's worth of
atoms, so max_memory_padding=0.95 scales the ceiling just below that same
structure's own metric. Equal-sized conformers were the shape that happened
to survive, which is why this reads as intermittent.

Also drop the batcher.load_states() call. torch_sim.optimize calls it itself,
on the OptimState from _chunked_apply -- which carries the optimizer's own
tensors and so reflects the real footprint. Pre-loading was not just
redundant: _get_first_batch skips the estimate when max_memory_scaler is
already set, so the pre-load pinned the ceiling to a measurement taken on the
bare SimState and torch_sim never re-measured, packing batches against an
underestimate. Worst with LBFGS, which stores history per system.

The "Memory estimation" phase disappears with it, since the probe now happens
inside torch_sim.optimize where gpuma cannot time it. TimingCapture keeps the
attribute so examples/full_benchmark.py still runs; it reads 0.0 and the cost
lands in "Optimization". Documented on the property.
@niklashoelter
niklashoelter merged commit 294b16d into niklashoelter:master Sep 25, 2026
1 check passed
@avnoordenne
avnoordenne deleted the fix/autobatcher-memory-ceiling branch October 1, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants