Repository navigation
optimizer: skip the autobatcher probe when the whole job fits - #42
Merged
Merged
Conversation
#40 uncapped max_atoms_to_try so the probe ceiling can no longer land below the job's own largest structure. The probe then grew replicated copies of a single system towards 100k atoms, twice per call (chunked init + InFlight). For UMA that cost ~750 s on a 36-structure job whose optimization takes ~20 s. Run one forward pass over the job instead. If it fits within max_memory_padding of the GPU budget, set max_memory_scaler to the job's total metric: everything runs as one batch and neither batcher probes. If it OOMs or lacks headroom, leave the scaler unset and torch-sim probes on the OptimState as before, so #40's fix holds on both paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the slowdown introduced by #40 while keeping its fix.
Problem
#40 stopped capping
max_atoms_to_tryat the job size, so the probe ceiling can no longer land below the job's own largest structure. As a side effect, everyoptimize_structure_batchcall now runs five probe sequences, each growing replicated copies of one system towards 100k atoms or OOM: two in_chunked_apply'sBinningAutoBatcherand three inInFlightAutoBatcher._get_first_batch.Profiled on a Quadro RTX 6000 with UMA-s on 36 conformers (927 atoms): 838 s total, of which 754 s is probing and ~20 s is the optimization itself.
Fix
Before optimizing, run one forward pass over the whole job:
max_memory_paddingof the GPU budget: setmax_memory_scalerto the job's totaln_edgesmetric. Everything runs as one batch and neither batcher probes. The ceiling can't be below any structure in the job, so optimizer: size the autobatcher by the GPU, not by the input #40's fix holds.max_atoms_to_try: leave the scaler unset, and torch-sim probes on theOptimStateexactly as after optimizer: size the autobatcher by the GPU, not by the input #40.The "Memory estimation" timing phase is timed again and now covers this check.
Results
examples/full_workflow_benchmark.py, FIRE, fconv 5e-2, dedicated GPU (optimization time):*0.7.0 medium had 189 structures; 0.8.0 and this PR have 243/244 (#33 keeps more conformers).
max_atoms_to_try, so it takes the unchanged probe path.UMA medium vs 0.7.0: UMA gets slower per structure as batches grow. With the scaler forced to 1/2, 1/4 and 1/8 of the job, medium drops from 431 s to 316, 253 and 217 s. The cause is in fairchem: the MoLE layer (
uma/nn/mole.py) loops over systems once per activation-checkpoint chunk, and the backward pass takes 0.2 s at 30 systems vs 11 s at 240. 0.7.0's smaller batches were a side effect of its capped probe. This is left for an upstream fairchem fix rather than capped here.Tests
5 new GPU tests in
tests/optimization/test_batch.py::TestAutobatcherMemoryProbe:determine_max_batch_size. This fails on master with 5 probe calls.Full suite with
HF_TOKENset: 299 passed, 3 skipped, in 6 min 38 s (22 min 47 s on master, mostly probing).