DRA pod reconciler and node labeling by TomerNewman · Pull Request #1289 · kubernetes-sigs/kernel-module-management

TomerNewman · 2026-06-17T12:09:58Z

Add DRA node labeling via unified PodNodeLabel reconciler

Summary

Merge DevicePluginPodReconciler and the new DRA pod labeling logic into a single PodNodeLabelReconciler. The reconciler watches DaemonSet pods with a ModuleNameLabel, determines the role from the DaemonSetRole label, and manages the corresponding node label (device-plugin-ready or dra-ready) based on pod readiness. This provides a node-level readiness signal indicating which nodes have a functioning device-plugin or DRA driver for a Module.

Changes

Unified controller:

PodNodeLabelReconciler in internal/controllers/pod_node_label_reconciler.go — replaces DevicePluginPodReconciler
Single registration in cmd/manager/main.go
Determines label name based on DaemonSetRole label: dra → dra-ready, otherwise → device-plugin-ready

Label utilities:

GetDRANodeLabel in internal/utils/kmmlabels.go — generates the dra-ready node label
GetDRANodeLabel exposed in public API via pkg/labels/labels.go

Constants:

DRARoleLabelValue and DRAReadySuffix in internal/constants/constants.go

Testing

Unit tests: Tests organized in two Context blocks (device-plugin pods, DRA pods) covering label add/remove, finalizer, replacement check, error paths, empty-nodeName, and DRA exclusion
Live cluster validation: Deployed to minikube — verified labeling on ready, unlabeling on delete, switching between roles, and independent operation of both roles

Acceptance Criteria

AC-1: PodNodeLabelReconciler watches pods and sets dra-ready or device-plugin-ready label on node when ready
AC-2: Label removed when pod not ready or deleting
AC-3: Pod event filter requires ModuleNameLabel + DaemonSet owner
AC-4: Reconciler uses NodeLabelerFinalizer
AC-5: Device-plugin replacement check excludes DRA pods
AC-6: Single PodNodeLabelReconciler registered in cmd/manager/main.go
AC-7: Unit tests cover both roles — label add/remove, predicate filtering, replacement check, error paths

k8s-ci-robot · 2026-06-17T12:10:01Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2026-06-17T12:10:04Z

✅ Deploy Preview for kubernetes-sigs-kmm ready!

Name	Link
🔨 Latest commit	`d250135`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kmm/deploys/6a38fea95b6b400008365f72
😎 Deploy Preview	https://deploy-preview-1289--kubernetes-sigs-kmm.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2026-06-17T12:10:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: TomerNewman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [TomerNewman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

TomerNewman · 2026-06-17T12:13:08Z

/cc @yevgeny-shnaidman @ybettan
/assign @yevgeny-shnaidman @ybettan
/uncc @mresvanis

codecov-commenter · 2026-06-17T12:14:58Z

Codecov Report

❌ Patch coverage is 73.46939% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.52%. Comparing base (fa23a9b) to head (d250135).
⚠️ Report is 385 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/controllers/pod_node_label_reconciler.go	74.41%	6 Missing and 5 partials ⚠️
cmd/manager/main.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1289      +/-   ##
==========================================
- Coverage   79.09%   73.52%   -5.58%     
==========================================
  Files          51       67      +16     
  Lines        5109     5004     -105     
==========================================
- Hits         4041     3679     -362     
- Misses        882     1156     +274     
+ Partials      186      169      -17

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ybettan · 2026-06-21T07:32:57Z

providing downstream consumers a signal (dra-ready node label) indicating which nodes have a functioning DRA driver for a Module.

Why is d/s related here?

ybettan · 2026-06-21T07:42:33Z

+		return ctrl.Result{}, fmt.Errorf("pod %s/%s has no %q label", pod.Namespace, pod.Name, constants.ModuleNameLabel)
+	}
+
+	labelName := utils.GetDRANodeLabel(pod.Namespace, moduleName)


GetDRANodeLabel is a it confusing here since we are constructing/building the label pattern, not getting it from any object.

Can we rename this function while still staying consistent with how we construct version-ready/ready labels?

Also, the labelName receiver have a too generic name IMO.

This follows the same naming convention as the existing GetDevicePluginNodeLabel, GetKernelModuleReadyNodeLabel, GetDevicePluginTargetNodeLabel, etc. in kmmlabels.go. Renaming just this one would be inconsistent

labelName is the same variable name used in DevicePluginPodReconciler for the same concept. Changing it only here would be inconsistent.

labelName is the same variable name used in DevicePluginPodReconciler for the same concept. Changing it only here would be inconsistent.

Seems like this controller doesn't exist anymore. So can we change it only in one place in the converged controller?

ybettan · 2026-06-21T07:53:04Z

+	)
+
+	if !podutils.IsPodReady(pod) || !pod.DeletionTimestamp.IsZero() {
+		if nodeName != "" {


I would add a funcion here or anything else to make it clearer we are targeting pods that weren't scheduled yet.

Done — added the same comment as in DevicePluginPodReconciler.

kubernetes-prow · 2026-06-21T08:54:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: TomerNewman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [TomerNewman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

TomerNewman · 2026-06-21T08:56:31Z

providing downstream consumers a signal (dra-ready node label) indicating which nodes have a functioning DRA driver for a Module.

Why is d/s related here?

"Downstream consumers" here doesn't refer to hub-spoke — it means anything that reads node labels to make scheduling or readiness decisions (other controllers, custom schedulers, or user workloads with node selectors). Poor wording on my part — I've updated the PR description to say "provides a node-level readiness signal" instead.

yevgeny-shnaidman · 2026-06-21T09:32:22Z


 	WorkerPodVersionLabelPrefix    = "beta.kmm.node.kubernetes.io/version-worker-pod"
 	DevicePluginVersionLabelPrefix = "beta.kmm.node.kubernetes.io/version-device-plugin"
+	DRAVersionLabelPrefix          = "beta.kmm.node.kubernetes.io/version-dra"


I don't think this should be in this PR. This is related to ordered uprade.

Done — removed.

yevgeny-shnaidman · 2026-06-21T09:35:10Z

General question: dra_pod_reconciler.go is a copy of device_plugin_pod_reconciler.go, except for the label it uses. Since device plugin and dra are mutually exclusive, can't we just use the device-pod-reconcile (and rename it) for both device plugin and dra

yevgeny-shnaidman · 2026-06-21T13:25:15Z

-	labelName := utils.GetDevicePluginNodeLabel(pod.Namespace, moduleName)
+	isDRA := pod.Labels[constants.DaemonSetRole] == constants.DRARoleLabelValue
+
+	var labelName string


Suggested change

var labelName string

labelName := utils.GetDevicePluginNodeLabel(pod.Namespace, moduleName)

if isDRA {

labelName = utils.GetDRANodeLabel(pod.Namespace, moduleName)

}

yevgeny-shnaidman · 2026-06-21T13:26:44Z

-			// Make sure we don't already have a new running pod before unlabeling the node
+			// Making sure there is no other pod of the same role already running
 			labelSelector := client.MatchingLabels{constants.ModuleNameLabel: moduleName}
+			if isDRA {


have not we also added a role label for devicePlugin?

Existing device-plugin pods from older KMM versions may not have the role label. Adding it to the selector would miss those pods during the replacement check and incorrectly unlabel the node. That's why we only narrow the selector for DRA (which is new — all DRA pods will always have the label) and use a post-filter skip for device-plugin instead.

yevgeny-shnaidman · 2026-06-21T13:27:44Z

 			var foundRunningPod bool
-			for _, p := range modulePodsList.Items {
+			for _, p := range podsList.Items {
+				if !isDRA && p.Labels[constants.DaemonSetRole] == constants.DRARoleLabelValue {


don't we need to make the same check for DRA pod?

No — when isDRA is true, lines 69-71 already add DaemonSetRole=dra to the label selector, so the List only returns DRA pods. Device-plugin pods won't appear in the results. This skip (line 81) is only needed for the device-plugin path, where the query doesn't filter by role (for backwards compatibility), so DRA pods could appear in the list and need to be excluded.

yevgeny-shnaidman · 2026-06-21T13:28:37Z

 		)
 }
-
-func (dppr *DevicePluginPodReconciler) addLabel(ctx context.Context, nodeName string, labelName string) error {


why not keep this function here? this is the only code that is using it

Done — moved back as private methods on the reconciler.

TomerNewman · 2026-06-21T16:44:46Z

/retest

TomerNewman · 2026-06-21T16:52:56Z

/retest

ybettan · 2026-06-22T07:16:56Z

I am happy with this PR. @yevgeny-shnaidman can lgtm when he is happy too.

yevgeny-shnaidman · 2026-06-22T09:18:18Z

-			// Make sure we don't already have a new running pod before unlabeling the node
+			// Making sure there is no other pod of the same role already running
 			labelSelector := client.MatchingLabels{constants.ModuleNameLabel: moduleName}
+			if isDRA {


we already have a code in line 42-43 that check if this DRA or not. Lets define a variable there that will hold either constants.DRARoleLabelValue or constants.DevicePluginRoleLabelValue and here we just set it , without needing to check again if it is DRA

Merge DevicePluginPodReconciler and the new DRA pod labeling logic into a single PodNodeLabelReconciler that handles both roles. The reconciler watches DaemonSet pods with a ModuleNameLabel, determines the role from the DaemonSetRole label, and manages the corresponding node label (device-plugin-ready or dra-ready) based on pod readiness.

yevgeny-shnaidman · 2026-06-22T09:22:49Z

/lgtm

TomerNewman · 2026-06-22T09:40:58Z

/retest

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 17, 2026

k8s-ci-robot requested review from mresvanis and yevgeny-shnaidman June 17, 2026 12:10

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 17, 2026

k8s-ci-robot assigned ybettan and yevgeny-shnaidman Jun 17, 2026

k8s-ci-robot requested review from ybettan and removed request for mresvanis June 17, 2026 12:13

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from 999f13a to d6eaa21 Compare June 17, 2026 14:07

TomerNewman marked this pull request as ready for review June 17, 2026 14:08

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 17, 2026

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from d6eaa21 to 6417d60 Compare June 18, 2026 09:20

ybettan reviewed Jun 21, 2026

View reviewed changes

Comment thread cmd/manager/main.go Outdated

ybettan reviewed Jun 21, 2026

View reviewed changes

Comment thread internal/controllers/dra_pod_reconciler.go Outdated

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from 6417d60 to 740df9d Compare June 21, 2026 08:54

yevgeny-shnaidman reviewed Jun 21, 2026

View reviewed changes

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from 740df9d to cb93dc3 Compare June 21, 2026 11:55

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from cb93dc3 to fba8829 Compare June 21, 2026 12:07

yevgeny-shnaidman reviewed Jun 21, 2026

View reviewed changes

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from fba8829 to d2ff271 Compare June 21, 2026 15:58

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch 2 times, most recently from 44e8e24 to f4c9bd9 Compare June 22, 2026 05:26

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from f4c9bd9 to 4e001cb Compare June 22, 2026 08:59

yevgeny-shnaidman reviewed Jun 22, 2026

View reviewed changes

TomerNewman force-pushed the MGMT-24473-dra-pod-reconciler branch from 4e001cb to d250135 Compare June 22, 2026 09:21

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 22, 2026

kubernetes-prow Bot merged commit 9547ce6 into kubernetes-sigs:main Jun 22, 2026
23 checks passed

Conversation

TomerNewman commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add DRA node labeling via unified PodNodeLabel reconciler

Summary

Changes

Testing

Acceptance Criteria

Uh oh!

k8s-ci-robot commented Jun 17, 2026

Uh oh!

netlify Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kmm ready!

Uh oh!

k8s-ci-robot commented Jun 17, 2026

Uh oh!

TomerNewman commented Jun 17, 2026

Uh oh!

codecov-commenter commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

ybettan commented Jun 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomerNewman Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kubernetes-prow Bot commented Jun 21, 2026

Uh oh!

TomerNewman commented Jun 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yevgeny-shnaidman commented Jun 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomerNewman commented Jun 21, 2026

Uh oh!

TomerNewman commented Jun 21, 2026

Uh oh!

ybettan commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yevgeny-shnaidman commented Jun 22, 2026

Uh oh!

TomerNewman commented Jun 22, 2026

TomerNewman commented Jun 17, 2026 •

edited

Loading

netlify Bot commented Jun 17, 2026 •

edited

Loading

codecov-commenter commented Jun 17, 2026 •

edited

Loading

TomerNewman Jun 21, 2026 •

edited

Loading