Repository navigation
Add GPU total-VRAM compatibility filtering to the native model catalog - #1295
Selena Yang (selenayang888) wants to merge 9 commits into
Conversation
- Preserve explicit minimum-VRAM requirements from catalog metadata - Query total GPU memory through the ORT EP device capacity API - Filter known-incompatible GPU variants while preserving CPU fallbacks and treating missing requirements or capacity as unknown - Hide incompatible retained variants after refresh without invalidating handles or removing cached and loaded inventory access - Detect actual ORT API support and offer strict configure-time validation - Add focused regressions and real-CUDA catalog end-to-end coverage Default packaged activation remains pending a post-merge ORT runtime and matching CUDA bundles. Existing dependencies retain unknown-capacity behavior; matching source builds enable the filtering path.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
🟡 Changes recommended
Capacity filtering makes cached variants inaccessible for cleanup and explicit-ID lookup, while per-alias GPU capacity queries repeat expensive work under the catalog mutex.
3 open issues
What changed in this PR
Adds total-GPU-memory compatibility filtering to the native model catalog while preserving unknown-capacity behavior. Activation remains dependent on matching ORT and provider builds; dependency pins are unchanged.
Changes:
- Preserve explicit VRAM requirements and compare them against individual GPUs’ total capacity.
- Add variant visibility controls and offline compatibility reevaluation.
- Detect API support at configure time and add unit tests, optional CUDA coverage, and documentation.
| File | Description |
|---|---|
| sdk_v2/cpp/test/internal_api/ep_detector_test.cc | Tests total-capacity queries and unsupported results. |
| sdk_v2/cpp/test/internal_api/capacity_compatibility_test.cc | Tests boundaries, unknowns, and individual-device checks. |
| sdk_v2/cpp/test/internal_api/azure_model_catalog_test.cc | Covers filtering, refresh visibility, offline snapshots, and real CUDA. |
| sdk_v2/cpp/test/internal_api/azure_catalog_test.cc | Tests VRAM metadata validation and serialization. |
| sdk_v2/cpp/test/CMakeLists.txt | Registers capacity compatibility tests. |
| sdk_v2/cpp/src/model.h | Declares variant visibility controls. |
| sdk_v2/cpp/src/model.cc | Implements visibility-aware variant enumeration. |
| sdk_v2/cpp/src/ep_detection/ep_detector.h | Adds the device-capacity query interface. |
| sdk_v2/cpp/src/ep_detection/ep_detector.cc | Queries total GPU memory through ORT. |
| sdk_v2/cpp/src/catalog/capacity_compatibility.h | Defines capacity data and compatibility checks. |
| sdk_v2/cpp/src/catalog/base_model_catalog.h | Adds the visibility predicate hook. |
| sdk_v2/cpp/src/catalog/base_model_catalog.cc | Applies visibility while retaining internal inventory access. |
| sdk_v2/cpp/src/catalog/azure_model_catalog.h | Declares capacity filtering helpers. |
| sdk_v2/cpp/src/catalog/azure_model_catalog.cc | Integrates capacity checks into catalog selection and fetching. |
| sdk_v2/cpp/src/catalog/azure_catalog_models.h | Adds optional VRAM requirement metadata. |
| sdk_v2/cpp/src/catalog/azure_catalog_models.cc | Parses and preserves valid VRAM requirements. |
| sdk_v2/cpp/README.md | Documents behavior, dependencies, and CUDA validation. |
| sdk_v2/cpp/include/foundry_local/foundry_local_c.h | Exposes the min_vram_bytes property key. |
| sdk_v2/cpp/cmake/FindOnnxRuntime.cmake | Detects API availability and supports strict configuration. |
Review effort: Balanced
Set the job timeout to 120 minutes, matching Windows x64. The ARM64 build was still compiling when Azure canceled it at the default 60-minute limit.
Baiju Meswani (baijumeswani)
left a comment
Baiju Meswani (baijumeswani)
left a comment
There was a problem hiding this comment.
This is a useful first step: use the catalog's memory requirement and ask ORT/each EP about the execution device. Keeping full catalog metadata and preserving cached/loaded model handles are also good choices.
Capacity-only filtering is enough for this first step. Driver, architecture, and feature checks do not need to be implemented now. However, we should establish a shared boundary so those checks can be added later without adding separate filtering paths throughout the catalog.
I suggest this flow:
flowchart TD
O["ORT / EP queries on the execution machine"] --> S["Target snapshot: provider, device identity, capacity"]
S --> T["Use that snapshot in local and hosted catalogs"]
T --> E["Evaluate a model against each execution target"]
M["Catalog model requirements"] --> E
F["Future driver / feature checks"] -.-> E
E --> R["Result: fits, does not fit, or unknown; reason; eligible targets"]
R --> C["Catalog visibility, version listings, default selection"]
R --> L["Load using a target that passed the checks"]
The key rule is that all checks must apply to the same provider/device. One GPU having enough memory and another supporting a required feature must not make the model pass. Catalog selection and loading must also agree on that target.
This can stay small: a target snapshot and one shared evaluator are enough; there is no need for a general plugin framework. Keep device facts separate from catalog policy. Unknown results can remain visible under the current policy, but should stay distinguishable from a known fit. Cached/loaded inventory should remain separate from discovery visibility.
Before merging, please address the routing and hosted-mode gaps noted inline, and complete the ORT 1.32.0 runtime/provider dependency update once those artifacts are released. The current GPU-only implementation is not yet the full cross-EP capacity solution, so the boundary should allow other resource types without treating CPU/NPU memory as GPU VRAM.
The inline comments distinguish concrete behavior problems from the small architectural changes that would make future criteria easier to add. The additional independent reviews were source reviews; no full SDK or GPU end-to-end validation was run in that pass.
| option(FOUNDRY_LOCAL_REQUIRE_DEVICE_MEMORY_CAPACITY | ||
| "Require an ORT build supporting EP device total-memory queries" OFF) |
There was a problem hiding this comment.
Please make the dependency rollout a merge requirement. sdk_v2/deps_versions.json still pins ORT 1.31.0. With this option off, normal builds only warn and compile out the capacity query, so the shipped catalog does not filter oversized models.
Please wait for the ORT 1.32.0 release, update the matching runtime and provider packages together, and require this API in shipping builds. Source builds can retain an explicit compatibility option. Validation should also exercise the enabled query path; a compiled-out or skipped test does not prove packaged filtering works.
Summary
Add GPU total-VRAM compatibility filtering to the native model catalog using the device-memory capacity API introduced by ONNX Runtime PR #33447.
GPU variants with explicit minimum-VRAM requirements are excluded when all candidate devices have known, insufficient total capacity. CPU variants retain existing compatibility behavior, and aliases remain visible when a compatible variant remains.
Changes
vRamFootprintBytesmetadata as themin_vram_bytesmodel property.EpDevice_GetMemoryCapacity; never use currently available VRAM.FOUNDRY_LOCAL_REQUIRE_DEVICE_MEMORY_CAPACITY=ONto reject dependencies that lack it.Capacity structures allow future extension, but this PR implements only GPU total-VRAM checks. CPU RAM and unified-memory checks are out of scope.
Validation
CUDA_VISIBLE_DEVICES=7,2.Draft Status / Dependency Rollout
Default packaged activation remains pending. The pinned ORT 1.31 runtime and CUDA bundles predate the API. The newest public nightly checked declares API 32 but does not contain
EpDevice_GetMemoryCapacity, so a version-only update is insufficient.This PR leaves dependency pins unchanged. Existing packaged builds retain unknown-capacity behavior and emit a configure warning; matching source builds enable the filtering path.
Before marking this ready, decide whether to land the guarded implementation ahead of dependency publication or require matching post-merge runtime and CUDA artifacts in this PR.