fix(templates): align PyTorch CUDA to the build image (cu128) - #926
Closed
Aydin-ab wants to merge 2 commits into
Closed
fix(templates): align PyTorch CUDA to the build image (cu128)#926Aydin-ab wants to merge 2 commits into
Aydin-ab wants to merge 2 commits into
Conversation
Four templates shipped a torch CUDA that didn't match their cu128 build image.
ecommerce_batch_embeddings left torch unpinned (torch>=2.0), which drifted to
2.12.0+cu13 — a newer CUDA than the 12.8 image (the risky direction). The other
three pinned torch==2.5.1+cu121.
Pin torch==2.7.0 (the cu128 fleet version; torchvision==0.22.0 for creatives) and
interpolate the depset index as --index .../${CUDA_VARIANT} so it always tracks the
build image's CUDA and can't drift again. Recompiled locks resolve torch==2.7.0+cu128;
base frameworks (numpy/pyarrow/pandas) unchanged.
Templates: ecommerce_batch_embeddings, ecommerce_multi_model_serving,
creatives_diffusion_finetuning, biotech_boltz_screening.
Claude-Session: https://claude.ai/code/session_01QmLc4yWtmC3PX2NwWzPbzD
Signed-off-by: Aydin Abiar <aydin@anyscale.com>
Contributor
Author
|
/test-template ecommerce_batch_embeddings ecommerce_multi_model_serving creatives_diffusion_finetuning |
Contributor
Author
|
/test-template biotech_boltz_screening |
Aydin-ab
added a commit
that referenced
this pull request
Jul 21, 2026
…torch Reverts the earlier cu128->cu121 image move. That move assumed torch 2.5.1 (cu121-only), but #926 aligns this template's torch to 2.7.0+cu128 to match the build image, so the deployed service image must be cu128 too — otherwise torch (cu128) would be newer than the service image (cu121). Claude-Session: https://claude.ai/code/session_01QmLc4yWtmC3PX2NwWzPbzD Signed-off-by: Aydin Abiar <aydin@anyscale.com>
Contributor
Author
|
/test-template ecommerce_batch_embeddings ecommerce_multi_model_serving creatives_diffusion_finetuning |
Contributor
Author
|
/test-template biotech_boltz_screening |
These templates left pandas unpinned, so the lock resolved pandas==3.0.3 — a major version ahead of the base image's 2.3.3 (what worker nodes run). Ray Data pickles pandas blocks across nodes, so a head(lock)-vs-worker(base) mismatch risks unpickle failures — the same skew these templates already guard for numpy/pyarrow, just missed for pandas. Pin pandas==2.3.3 to match the base (added to creatives too, where pandas is transitive). Templates: ecommerce_batch_embeddings, ecommerce_multi_model_serving, biotech_boltz_screening, creatives_diffusion_finetuning. Claude-Session: https://claude.ai/code/session_01QmLc4yWtmC3PX2NwWzPbzD Signed-off-by: Aydin Abiar <aydin@anyscale.com>
Aydin-ab
force-pushed
the
fix/align-torch-cuda-to-image
branch
from
July 22, 2026 00:37
be20443 to
e51035d
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes cross-node dependency skew in four templates: the PyTorch CUDA didn't match the cu128 build image, and pandas was unpinned (drifting above the base image).
What changed
torch==2.7.0(+torchvision==0.22.0for creatives) and interpolate the depset index as--index .../${CUDA_VARIANT}so it tracks the build image's CUDA — root-cause fix (the flag was hardcodedcu121). Worst case:ecommerce_batch_embeddings, whose unpinnedtorch>=2.0drifted to2.12.0+cu13— a newer CUDA than the 12.8 image. Locks now resolvetorch==2.7.0+cu128.pandas>=2.0, resolvingpandas==3.0.3— a major version ahead of the base image's2.3.3(what workers run). Ray Data pickles pandas blocks cross-node, so pinpandas==2.3.3to match the base (completes the numpy/pyarrow pin group these templates already had).Testing
/test-templateon all four (GPU CI). numpy/pyarrow unchanged; verified no other base-framework drift.Caveats
text_embeddingsexcluded from the CUDA change: pinstorch<2.5(old optimum/transformers stack), no cu128 wheels — a separate modernization follow-up.ecommerce_multi_model_serving's service image returns to cu128 in fix(templates): pin prod configs to lock + fix broken compute refs (ecommerce, biotech x2) #917 (merge together).https://claude.ai/code/session_01QmLc4yWtmC3PX2NwWzPbzD