Skip to content

arm64 controller - #549

Closed
cyanidium wants to merge 1 commit into
ROCm:mainfrom
cyanidium:feat/multi-arch
Closed

arm64 controller#549
cyanidium wants to merge 1 commit into
ROCm:mainfrom
cyanidium:feat/multi-arch

Conversation

@cyanidium

@cyanidium cyanidium commented May 10, 2026

Copy link
Copy Markdown
Contributor

Motivation

This allows control plane nodes running arm64/aarch64 CPUs to be able to manage clusters with AMD GPUs. Heterogeneous/multi-arch clusters do not need dedicated amd64/x86_64 control plane nodes just to control AMD GPU scheduling.

Technical Details

Builds the controller/manager container for both amd64 and arm64, in a single multi-platform image.

Not changed:

  • Build host still has to be amd64/x86_64
  • GPU node still has to be amd64/x86_64
  • e2e still only tests amd64/x86_64

Resolves #331

Test Plan

Tested working with a cluster like this:

control-plane   arm64/aarch64   no_gpu
control-plane   arm64/aarch64   no_gpu
control-plane   arm64/aarch64   no_gpu
worker          amd64/x86_64    no_gpu
worker          amd64/x86_64    no_gpu
worker          amd64/x86_64    amd_gpu
worker          amd64/x86_64    amd_gpu
worker          amd64/x86_64    amd_gpu

Helm values:

node-feature-discovery.enabled=false
deviceConfig.spec.driver.enabled=false

Test Result

gpu-operator pod schedules correctly and runs on the arm64/aarch64 control plane nodes. workflow pod schedules and runs correctly on an amd64/x86_64 worker node. GPU resources are available to pods that request them.

Submission Checklist

@yansun1996
yansun1996 self-requested a review May 10, 2026 02:20
@yansun1996

yansun1996 commented May 10, 2026

Copy link
Copy Markdown
Member

Hi @cyanidium , thanks for the PR, really appreciate the contribution, this commit needs to go through internal review and CI test on our side, will get back to you once we decide to move forward with this change.

@cyanidium
feat: arm64 controller
70bedc3 
Builds the controller/manager container for both amd64 and arm64, in a
single multi-platform image. This allows control plane nodes running
arm64/aarch64 CPUs to be able to manage clusters with AMD GPUs.

Not changes:
- Build host still has to be amd64/x86_64
- GPU node still has to be amd64/x86_64
- e2e still only tests amd64/x86_64

Benefits:
- Heterogeneous/multi-arch clusters do not need dedicated amd64/x86_64
  control plane nodes just to control AMD GPU scheduling.

Resolves ROCm#331
@cyanidium

cyanidium commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on v1.5.1

@yansun1996 yansun1996 mentioned this pull request Aug 4, 2026
Merged
6 tasks
@yansun1996

yansun1996 commented Aug 4, 2026

Copy link
Copy Markdown
Member

This work has been consolidated (with the ppc64le changes from #565) into a unified, configurable-arch controller manager image build in #623. Your arm64 commit was cherry-picked there with authorship preserved.

@yansun1996 yansun1996 mentioned this pull request Aug 4, 2026
Closed
1 task
yansun1996 added a commit to yansun1996/gpu-operator-rocm that referenced this pull request Aug 4, 2026
@yansun1996 @claude
build: unify controller manager image arch on single TARGETARCH ENV
6485eaf 
Reconcile the arm64 (ROCm#549) and ppc64le (ROCm#565) multi-arch approaches into
one configurable build for the controller manager image. The Dockerfile
now cross-compiles a single manager binary driven by TARGETARCH (default
amd64) with TARGETOS (default linux), instead of building both amd64 and
arm64 binaries unconditionally.

- Makefile `manager` target builds one binary honoring GOOS/GOARCH.
- `docker-build` builds a single arch selected by ARCH (default amd64),
  passing TARGETARCH/TARGETOS as build args.
- `docker-build-multiarch` retains the buildx multi-platform push path.
- Dockerfile ARGs default to amd64/linux so a plain `docker build`
  produces an amd64 image with no extra flags.

Co-Authored-By: Claude <noreply@anthropic.com>
@cyanidium cyanidium closed this Aug 4, 2026
yansun1996 added a commit to yansun1996/gpu-operator-rocm that referenced this pull request Aug 4, 2026
@yansun1996 @claude
build: make controller manager and utils images buildx PLATFORM-confi…
90156f6 
…gurable

Unify the arm64 (ROCm#549) and ppc64le (ROCm#565) multi-arch work into a single
configurable build for both the controller manager and utils container
images.

docker-build and docker-build-utils now run
`docker buildx build --platform "$(PLATFORM)"` with PLATFORM defaulting to
linux/amd64 (unchanged default behavior). Set PLATFORM to a single platform
(e.g. linux/arm64) or a comma-separated list (linux/amd64,linux/arm64) for a
multi-arch build. buildx auto-populates TARGETOS/TARGETARCH per platform,
which drive the Go cross-compile, the kubectl download, and the utils
container's oc/kubectl client download.

The Dockerfile and utils Dockerfile TARGETARCH/TARGETOS ARGs intentionally
carry no default: a default suppresses buildx's per-platform value and would
pin every build to that arch.

bundle build is left as upstream (amd64, no changes). Adds a developer-guide
section documenting the PLATFORM builds and the QEMU/buildx prerequisites for
cross-arch builds.

Co-Authored-By: Claude <noreply@anthropic.com>
yansun1996 added a commit to yansun1996/gpu-operator-rocm that referenced this pull request Aug 4, 2026
@yansun1996 @claude
build: make controller manager and utils images buildx PLATFORM-confi…
3fd286f 
…gurable

Unify the arm64 (ROCm#549) and ppc64le (ROCm#565) multi-arch work into a single
configurable build for both the controller manager and utils container
images.

docker-build and docker-build-utils run `$(CONTAINER_ENGINE) buildx build
--platform "$(PLATFORM)" $(BUILDX_OUTPUT)`, defaulting to
PLATFORM=linux/amd64 and BUILDX_OUTPUT=--load (single amd64 image loaded
locally, unchanged default). Set PLATFORM to a single platform
(linux/arm64) or a comma-separated list; for a multi-arch list set
BUILDX_OUTPUT=--push (a multi-platform result cannot be --load-ed into the
local store). Works with docker buildx and podman 4.0+.

buildx auto-populates TARGETOS/TARGETARCH per platform, which drive the Go
cross-compile, the kubectl download, and the utils container's oc/kubectl
client download. Dockerfile.build gains qemu-user-static + libc6-arm64-cross
for cross-arch runtime stages.

bundle build is left as upstream (amd64, no changes). Adds a developer-guide
section documenting PLATFORM/BUILDX_OUTPUT and the QEMU prerequisite.

Co-Authored-By: Claude <noreply@anthropic.com>
yansun1996 added a commit to yansun1996/gpu-operator-rocm that referenced this pull request Aug 4, 2026
@yansun1996 @claude
build: make controller manager and utils images buildx PLATFORM-confi…
d6bd900 
…gurable

Unify the arm64 (ROCm#549) and ppc64le (ROCm#565) multi-arch work into a single
configurable build for both the controller manager and utils container
images.

docker-build and docker-build-utils run `$(CONTAINER_ENGINE) buildx build
--platform "$(PLATFORM)"`, defaulting to PLATFORM=linux/amd64. A
single-platform build auto-loads into the local image store. Set PLATFORM to
a single platform (linux/arm64) or a comma-separated list; a multi-platform
list must be pushed via a direct `buildx --push`. Works with docker buildx
and podman 4.0+.

buildx auto-populates TARGETOS/TARGETARCH per platform, driving the Go
cross-compile, the kubectl download, and the utils container's oc/kubectl
client download. Dockerfile.build gains qemu-user-static + libc6-arm64-cross
for cross-arch runtime stages.

bundle build is left as upstream (amd64). Adds a developer-guide section.

Co-Authored-By: Claude <noreply@anthropic.com>
spraveenio pushed a commit that referenced this pull request Aug 6, 2026
@yansun1996 @cyanidium
@Amulyam24 @claude
build: configurable-arch controller manager image (unify arm64 + ppc6…
6df13c1 
…4le) (#623)

* feat: arm64 controller

Builds the controller/manager container for both amd64 and arm64, in a
single multi-platform image. This allows control plane nodes running
arm64/aarch64 CPUs to be able to manage clusters with AMD GPUs.

Not changes:
- Build host still has to be amd64/x86_64
- GPU node still has to be amd64/x86_64
- e2e still only tests amd64/x86_64

Benefits:
- Heterogeneous/multi-arch clusters do not need dedicated amd64/x86_64
  control plane nodes just to control AMD GPU scheduling.

Resolves #331

* Build AMD GPU operator images for ppc64le

Signed-off-by: Amulyam24 <amulmek1@in.ibm.com>

* build: make controller manager and utils images buildx PLATFORM-configurable

Unify the arm64 (#549) and ppc64le (#565) multi-arch work into a single
configurable build for both the controller manager and utils container
images.

docker-build and docker-build-utils run `$(CONTAINER_ENGINE) buildx build
--platform "$(PLATFORM)"`, defaulting to PLATFORM=linux/amd64. A
single-platform build auto-loads into the local image store. Set PLATFORM to
a single platform (linux/arm64) or a comma-separated list; a multi-platform
list must be pushed via a direct `buildx --push`. Works with docker buildx
and podman 4.0+.

buildx auto-populates TARGETOS/TARGETARCH per platform, driving the Go
cross-compile, the kubectl download, and the utils container's oc/kubectl
client download. Dockerfile.build gains qemu-user-static + libc6-arm64-cross
for cross-arch runtime stages.

bundle build is left as upstream (amd64). Adds a developer-guide section.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Signed-off-by: Amulyam24 <amulmek1@in.ibm.com>
Co-authored-by: Martin Rowe <martin.p.rowe@gmail.com>
Co-authored-by: Amulyam24 <amulmek1@in.ibm.com>
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yansun1996 yansun1996 Awaiting requested review from yansun1996

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Support arm64 for gpu-operator containers

2 participants

@cyanidium @yansun1996