feat(podvm): opt-in NVIDIA 595.71.05 driver for B200 multi-GPU CC #29
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,10 +22,20 @@ on: | |
| type: string | ||
| default: "https://github.com/cohere-ai/guest-components.git" | ||
| guest_components_ref: | ||
| description: | | ||
| guest-components ref (branch, tag, or SHA). | ||
|
|
||
| Default: alhassankhedr/sync-main-to-cohere (head of PR #9). | ||
| That branch carries upstream main's nvidia-attester rewrite | ||
| (NVAT SDK based, no `count == 1` guard) and is required for | ||
| multi-GPU evidence to work end-to-end on 8x B200 hosts. The | ||
| plain `cohere` branch still has the old NVML-based attester | ||
| which silently produces empty evidence on 2+ GPU systems | ||
| (mod a sed `s/count == 1/count >= 1/` patch the podvm-mkosi | ||
| Dockerfile applies). Switch back to `cohere` after PR #9 merges. | ||
| required: false | ||
| type: string | ||
| default: "alhassankhedr/sync-main-to-cohere" | ||
| custom_gc_binaries: | ||
| description: "guest-components binaries to build from source" | ||
| required: false | ||
|
|
@@ -46,6 +56,11 @@ on: | |
| required: false | ||
| type: boolean | ||
| default: false | ||
| b200_cc_drivers: | ||
| description: "Install NVIDIA 595.71.05 open driver (enables Confidential Computing on multi-GPU B200). EXPERIMENTAL" | ||
| required: false | ||
| type: boolean | ||
| default: false | ||
|
|
||
| permissions: | ||
| id-token: write # OIDC token for build provenance attestation | ||
|
|
@@ -70,11 +85,13 @@ jobs: | |
| image_name_debug: ${{ steps.compute.outputs.image_name_debug }} | ||
| image_tag_release: ${{ steps.compute.outputs.image_tag_release }} | ||
| image_tag_debug: ${{ steps.compute.outputs.image_tag_debug }} | ||
| b200_cc_drivers: ${{ steps.compute.outputs.b200_cc_drivers }} | ||
| steps: | ||
| - name: Compute tags and image names | ||
| id: compute | ||
| env: | ||
| DISTRO: ${{ inputs.distro || 'ubuntu' }} | ||
| B200_CC_DRIVERS: ${{ inputs.b200_cc_drivers && 'true' || 'false' }} | ||
| run: | | ||
| if [[ "$GITHUB_REF" == refs/tags/podvm-v* ]]; then | ||
| TAG="${GITHUB_REF#refs/tags/podvm-}" | ||
|
|
@@ -84,10 +101,15 @@ jobs: | |
| REPLACE_IMAGE="true" | ||
| fi | ||
| TAG="${TAG//./-}" | ||
| # Suffix CC-driver builds so they never collide with standard images | ||
| if [ "$B200_CC_DRIVERS" = "true" ]; then | ||
| TAG="${TAG}-cc595" | ||
| fi | ||
| { | ||
| echo "tag=$TAG" | ||
| echo "distro=$DISTRO" | ||
| echo "replace_image=$REPLACE_IMAGE" | ||
| echo "b200_cc_drivers=$B200_CC_DRIVERS" | ||
| echo "image_name_release=podvm-${DISTRO}-${TEE_PLATFORM}-release-${TAG}" | ||
| echo "image_name_debug=podvm-${DISTRO}-${TEE_PLATFORM}-debug-${TAG}" | ||
| echo "image_tag_release=${TAG}-${DISTRO}-release" | ||
|
|
@@ -176,7 +198,7 @@ jobs: | |
| PODVM_DISTRO: ${{ needs.meta.outputs.distro }} | ||
| AA_FEATURES: ${{ inputs.aa_features || 'bin,ttrpc,kbs,coco_as,rust-crypto,tdx-attester,nvidia-attester' }} | ||
| GC_REPO: ${{ inputs.guest_components_repo || 'https://github.com/cohere-ai/guest-components.git' }} | ||
| GC_REF: ${{ inputs.guest_components_ref || 'alhassankhedr/sync-main-to-cohere' }} | ||
| GC_CUSTOM_BINARIES: ${{ inputs.custom_gc_binaries || 'attestation-agent,api-server-rest' }} | ||
| run: | | ||
| MAKE_ARGS=( | ||
|
|
@@ -195,6 +217,47 @@ jobs: | |
| echo "Disk after binaries build:" | ||
| df -h / | ||
|
|
||
| - name: Override NVIDIA driver to 595.71.05 (B200 multi-GPU CC) | ||
| if: needs.meta.outputs.b200_cc_drivers == 'true' | ||
| working-directory: src/cloud-api-adaptor/podvm-mkosi | ||
| run: | | ||
| set -euo pipefail | ||
| CONF=mkosi.presets/system/mkosi.conf.d/ubuntu.conf | ||
| # The 595 branch only ships the unversioned `nvidia-driver-open` | ||
| # metapackage in NVIDIA's CUDA repo (no `nvidia-driver-595-open`). | ||
| # Match by package name only so this survives future 580.x.y bumps. | ||
| sed -i -E \ | ||
| -e 's|^([[:space:]]*)nvidia-driver-580-open=.*|\1nvidia-driver-open=595.71.05-1ubuntu1|' \ | ||
| -e 's|^([[:space:]]*)nvidia-persistenced=.*|\1nvidia-persistenced=595.71.05-1ubuntu1|' \ | ||
| -e 's|^([[:space:]]*)nvidia-fabricmanager=.*|\1nvidia-fabricmanager=595.71.05-1ubuntu1|' \ | ||
| -e 's|^([[:space:]]*)libnvidia-nscq=.*|\1libnvidia-nscq=595.71.05-1ubuntu1|' \ | ||
| "$CONF" | ||
| echo "----- Updated NVIDIA package pins -----" | ||
| grep -E '^[[:space:]]*(nvidia|libnvidia)' "$CONF" | ||
|
|
||
| - name: Increase debug root partition for CC595 drivers | ||
| if: needs.meta.outputs.b200_cc_drivers == 'true' && matrix.profile == 'debug' | ||
| working-directory: src/cloud-api-adaptor/podvm-mkosi | ||
| run: | | ||
| set -euo pipefail | ||
| CONF=mkosi.presets/system/mkosi.repart-debug/10-root.conf | ||
| # NVIDIA 595 drivers make the root filesystem too large for | ||
| # systemd-repart's Minimize=guess estimation, causing mkfs.ext4 | ||
| # "No space left on device" during the build. | ||
| printf '[Partition]\nType=root\nFormat=ext4\nCopyFiles=/\nMinimize=off\nSizeMinBytes=12G\nSizeMaxBytes=12G\n' > "$CONF" | ||
| echo "----- Updated repart config -----" | ||
| cat "$CONF" | ||
|
There was a problem hiding this comment. Debug root partition override is redundant dead codeLow Severity The "Increase debug root partition for CC595 drivers" workflow step (conditional on Additional Locations (1)  Reviewed by Cursor Bugbot for commit de75f12. Configure here. |
||
|
|
||
| - name: Resolve installed NVIDIA driver version | ||
| working-directory: src/cloud-api-adaptor/podvm-mkosi | ||
| run: | | ||
| set -euo pipefail | ||
| CONF=mkosi.presets/system/mkosi.conf.d/ubuntu.conf | ||
| DRIVER_LINE=$(grep -E '^[[:space:]]*nvidia-driver(-580)?-open=' "$CONF" | head -n1) | ||
| DRIVER_VER=$(printf '%s' "$DRIVER_LINE" | sed -E 's|.*=([0-9]+\.[0-9]+\.[0-9]+).*|\1|') | ||
| echo "Resolved NVIDIA driver version: $DRIVER_VER" | ||
| echo "NVIDIA_DRIVER=$DRIVER_VER" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Build OS image | ||
| working-directory: src/cloud-api-adaptor/podvm-mkosi | ||
| env: | ||
|
|
@@ -265,6 +328,7 @@ jobs: | |
| --arg distro "$DISTRO" \ | ||
| --arg profile "$PROFILE" \ | ||
| --arg tee_platform "$TEE_PLATFORM" \ | ||
| --arg nvidia_driver "$NVIDIA_DRIVER" \ | ||
| --arg build_date "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ | ||
| '$ARGS.named' > /tmp/measurements.json | ||
|
|
||
|
|
@@ -307,6 +371,7 @@ jobs: | |
| --annotation "com.cohere.caa.commit=${CAA_COMMIT}" \ | ||
| --annotation "com.cohere.caa.version=${GITHUB_REF_NAME}" \ | ||
| --annotation "com.cohere.rtmr2=${RTMR2}" \ | ||
| --annotation "com.cohere.nvidia.driver=${NVIDIA_DRIVER}" \ | ||
| --format json > oras-output.json | ||
|
|
||
| cat oras-output.json | ||
|
There was a problem hiding this comment. 🔒 Agentic Security Review
Impact: If that branch is updated maliciously (or compromised), unreviewed guest-components code can be pulled into release artifacts and published as trusted PodVM images. |
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Opt-in flag is non-functional; config already hardcodes 595
High Severity
The
b200_cc_driversflag is supposed to opt-in to the 595 driver by sed-replacing 580 package pins, butubuntu.confwas directly committed with 595 packages (nvidia-driver-open=595.71.05-1ubuntu1), making the sed a no-op. Specifically, the first sed pattern looks fornvidia-driver-580-open=which doesn't exist in the committed file (it hasnvidia-driver-open=without580), and patterns 2–4 match but replace with the same already-present 595 values. This means ALL builds use the 595 driver regardless of the flag, contradicting the stated "Default behaviour is unchanged" and the PR's "opt-in" design. The flag's only actual effect is appending-cc595to the image tag.Additional Locations (1)
src/cloud-api-adaptor/podvm-mkosi/mkosi.presets/system/mkosi.conf.d/ubuntu.conf#L27-L31Reviewed by Cursor Bugbot for commit fd50ec5. Configure here.