Repository navigation
optimizer: size the autobatcher by the GPU, not by the input - #40
Merged
niklashoelter merged 1 commit intoSep 25, 2026
Merged
niklashoelter merged 1 commit into
niklashoelter merged 1 commit into
Conversation
_optimize_batch capped the autobatcher's memory probe with
effective_max_atoms = min(batched_state.n_atoms, max_atoms_to_try)
but max_atoms_to_try bounds how much InFlightAutoBatcher is allowed to try
fitting on the device. Tying it to the total atom count of the submitted job
made the memory ceiling a function of how much work was handed in, so a small
job probed a small ceiling. InFlightAutoBatcher then raises outright when any
state's metric exceeds that ceiling.
Checked against the real _optimize_batch path on a Quadro RTX 6000, with a
Lennard-Jones potential standing in for the MLIP so only the batching logic
varies. Before, three of four shapes fail:
single structure ValueError: Max metric ... 18.0 > 17.1
small then large (10, 200) ValueError: State metric=398.0 > 302.4
3 equal conformers ok
mixed library (12/40/8/150) ValueError: State metric=298.0 > 211.2
After, all four complete. A single structure in batch mode failed
deterministically: the probe is capped at exactly one structure's worth of
atoms, so max_memory_padding=0.95 scales the ceiling just below that same
structure's own metric. Equal-sized conformers were the shape that happened
to survive, which is why this reads as intermittent.
Also drop the batcher.load_states() call. torch_sim.optimize calls it itself,
on the OptimState from _chunked_apply -- which carries the optimizer's own
tensors and so reflects the real footprint. Pre-loading was not just
redundant: _get_first_batch skips the estimate when max_memory_scaler is
already set, so the pre-load pinned the ceiling to a measurement taken on the
bare SimState and torch_sim never re-measured, packing batches against an
underestimate. Worst with LBFGS, which stores history per system.
The "Memory estimation" phase disappears with it, since the probe now happens
inside torch_sim.optimize where gpuma cannot time it. TimingCapture keeps the
attribute so examples/full_benchmark.py still runs; it reads 0.0 and the cost
lands in "Optimization". Documented on the property.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_optimize_batchcapped the memory probe withmin(batched_state.n_atoms, max_atoms_to_try).max_atoms_to_trybounds howmuch
InFlightAutoBatchermay try to fit on the device, so tying it to thejob's atom count made the ceiling a function of how much work was submitted.
InFlightAutoBatcherthen raises when any state's metric exceeds thatceiling. Measured on a Quadro RTX 6000 with a Lennard-Jones stand-in so only
the batching varies:
All four complete after. A single structure failed deterministically: the
probe was capped at one structure's worth of atoms and the 0.95 padding put the
ceiling just under that structure's own metric. Equal-sized conformers were the
shape that survived, which is why this looked intermittent.
Also drops the
batcher.load_states()pre-load.torch_sim.optimizecalls ititself on the
OptimState, which carries the optimizer's own tensors and soreflects the real footprint. The pre-load was not just redundant:
_get_first_batchskips the estimate whenmax_memory_scaleris already set,so the pre-load pinned the ceiling to a measurement on the bare
SimStateandtorch-sim never re-measured. Worst with LBFGS, which stores history per system.
Consequence: the "Memory estimation" timing phase disappears, since the probe
now runs inside
torch_sim.optimize.TimingCapture.memory_estimationis keptso
examples/full_benchmark.pystill runs; it reads 0.0 and the cost lands in"Optimization". Documented on the property.
Full suite: 244 passed, 3 skipped (no new tests; the four-shape check above was
done by hand).