Repository navigation
Commit 3b80932
[bug fix] ZeRO2: preserve fp32 grad accumulation precision in partition storage (deepspeedai#8749)
When training bf16 models with `data_types.grad_accum_dtype: fp32`,
ZeRO-2 could still store the reduced owned gradient partition in bf16
storage. That defeats the fp32 accumulation setting after communication:
values that are representable in fp32 but lie between bf16 values are
rounded before APIs such as `safe_get_full_grad()` read them back.
This matters for request accumulation and similar uneven-gradient cases.
If two ranks contribute gradients `1.0` and `1.0078125`, fp32
communication produces the exact average `1.00390625`. Storing that
result in a bf16 partition rounds it back to `1.0`, so the training step
loses the precision that `grad_accum_dtype: fp32` requested.
This PR makes ZeRO-1/2 use fp32 owned gradient partition storage when
fp32 gradient accumulation is active. The ZeRO-2 path now writes reduced
fp32 communication results into a compact fp32 partition buffer and
skips allocating the legacy bf16 `grads_in_partition` buffer for that
path. This keeps the memory behavior aligned with the feature: the old
bf16 owned partition is replaced rather than kept alongside the fp32
one, so bf16 + fp32 grad accumulation should use less GPU memory than
the previous mixed-storage path.
ZeRO-3 was checked separately. It already allocates
`grad_partitions_flat_buffer` with `gradient_accumulation_dtype` and
casts reduced partitions to that dtype before storing them, so it does
not have the same persistent bf16 partition-storage issue. A ZeRO-3
regression case is included to guard that behavior.
## Memory saved
Run with a 1B-parameter bf16 toy model on 2 H200 ranks.
Per rank:
```
metric before a77aeb6 after 87e384e change
━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━
peak allocated 14,336 MiB 14,336 MiB unchanged
─────────────────────────── ────────────────── ───────────────── ────────────
peak reserved 16,386 MiB 14,338 MiB -2,048 MiB
─────────────────────────── ────────────────── ───────────────── ────────────
reserved delta after init 12,290 MiB 10,242 MiB -2,048 MiB
```
Config was ZeRO-2, bf16 model, communication_data_type=fp32,
data_types.grad_accum_dtype=fp32, GAS=4, numel=1,073,741,824, 2 ranks.
PyTorch peak allocated is unchanged, but peak reserved GPU memory drops
by 2 GiB per rank.
## Implementation
- Adds compact fp32 owned gradient partition buffers for ZeRO-1/2 when
the model dtype differs from fp32 gradient accumulation dtype.
- Writes fp32 all-reduce results into that compact partition storage
instead of relying on the bf16-owned partition copy.
- Avoids allocating the legacy bf16 `grads_in_partition` buffer on the
compact fp32 path.
- Keeps the regression observable through `safe_get_full_grad()` rather
than depending on private storage layout, with the ZeRO-2 internal dtype
assertion kept only where it directly checks the fixed path.
---------
Signed-off-by: Stas Bekman <stas@stason.org>
Co-authored-by: Olatunji Ruwase <tunji.ruwase@snowflake.com>1 parent dcd8595 commit 3b80932
4 files changed
Lines changed: 445 additions & 46 deletions
File tree
- deepspeed
- runtime/zero
- utils
- tests/unit
- runtime/zero
- v1/zero
0 commit comments