CR Status Improvements by gazarenkov · Pull Request #3348 · redhat-developer/rhdh-operator

gazarenkov · 2026-08-07T13:07:54Z

Description

Two-level health monitoring: Added Runtime condition alongside existing Deployed condition
Deployed: Reflects Deployment/StatefulSet replica status
Runtime: Provides Pod/Container-level health (crash loops, image pull failures, OOMKilled, etc.)
Improved status messages
Smart crash loop detection: Only reports crash if container is NOT currently Ready - containers that recover after restart are not flagged as failed
Enabled plugins in status: status.plugins field lists enabled dynamic plugins when healthy, cleared when unhealthy
Unit tests: Added ~490 lines of tests covering all container state scenarios
Updated docs/design.md with full status condition reference tables and examples

Which issue(s) does this PR fix or relate to

https://redhat.atlassian.net/browse/RHIDP-15021

PR acceptance criteria

Tests
Documentation

Building Container Images for Testing

Need to test container images from this PR?

For Maintainers: To trigger a test image build, review the code and comment /build-images.
This always builds the HEAD of the PR branch.

For Contributors: Ask a maintainer to run /build-images.

Images will be built and pushed to Quay with links posted in comments.

rhdh-qodo-merge · 2026-08-07T13:09:15Z

PR Summary by Qodo

Add Runtime condition and plugin list to Backstage CR status

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🐞 Bug fix 🕐 40+ Minutes

AI Description

• Add Runtime and Config status conditions for pod/container health and spec validation feedback.
• Populate status.plugins with enabled dynamic plugins only when Deployed+Runtime are healthy.
• Add unit/integration coverage and update design docs/CRD schema for the new status fields.

Diagram

graph TD
  A["BackstageReconciler"] --> B["reconcileStatus"] --> C["setRuntimeCondition"] --> D{{"Pods"}} --> E[("Backstage CR status")]
  B --> F["setDeployedCondition"] --> G{{"Deployment/StatefulSet"}} --> E
  B --> H["setEnabledPlugins"] --> I["BackstageModel"] --> E
  subgraph Legend
    direction LR
    _proc["Controller logic"] ~~~ _k8s{{"K8s resource"}} ~~~ _cr[("CR status")]
  end

High-Level Assessment

The following are alternative approaches to this PR:

1. Single condition with richer reasons/messages

➕ Avoids introducing/teaching multiple condition types to consumers
➕ Simplifies integration tests and external tooling that expects one condition
➖ Conflates rollout readiness with runtime health; harder to distinguish 'rolled out but crash-looping' vs 'still progressing'
➖ Less extensible if future health signals (e.g., config validity) need separate lifecycles

2. Derive runtime health from Deployment/ReplicaSet conditions/events only

➕ Avoids per-pod inspection and reduces list calls during reconcile
➕ Leans on native workload status fields
➖ Misses container-granular failures (image pull, init container failures, OOMKilled details)
➖ Slower/less precise feedback than direct Pod/ContainerStatus inspection

3. Add a dedicated status sub-structure (e.g., status.runtime/status.deployment) instead of conditions

➕ More structured, strongly-typed status for clients; easier to evolve with new fields
➕ Avoids overloading Condition messages for machine parsing
➖ Moves away from standard Kubernetes Condition patterns; harder to integrate with generic tooling
➖ Requires CRD/schema expansion and client updates beyond condition watchers

Recommendation: Keep the PR’s two-condition approach. Separating Deployed (workload readiness) from Runtime (pod/container health) matches Kubernetes best practices for multi-signal health, prevents false positives during brief readiness windows, and enables future expansion (e.g., Config validity) without conflating concerns. The added unit tests make the container-state heuristics (notably the ‘only flag crash loops when not Ready’) maintainable.

Files changed (17) +808 / -47

Enhancement (6) +232 / -30

current-types.goExpose new condition types/reasons in current API constants +14/-1

Expose new condition types/reasons in current API constants

• Adds Runtime and Config condition types alongside Deployed, and defines dedicated reason constants for runtime health (Running/ContainerFailed/Pending) and config validation (Invalid). This centralizes status taxonomy for controllers and clients.

api/current-types.go

backstage_types.goAdd Runtime/Config condition types and status.plugins field +15/-0

Add Runtime/Config condition types and status.plugins field
• Extends v1alpha5 condition taxonomy with Runtime and Config types plus corresponding reasons. Adds BackstageStatus.Plugins to publish enabled dynamic plugin package names when healthy.
api/v1alpha5/backstage_types.go

zz_generated.deepcopy.goRegenerate deepcopy for plugins field support (v1alpha5) +6/-1

Regenerate deepcopy for plugins field support (v1alpha5)
• Adds DeepCopy logic for the new BackstageStatus.Plugins slice and normalizes import aliasing. Ensures status objects copy safely across reconciler updates.
api/v1alpha5/zz_generated.deepcopy.go

backstage_controller.goWire new status reconciliation and type-specific error conditions +13/-8

Wire new status reconciliation and type-specific error conditions

• Routes preprocessing/model init failures to Config/Invalid and runtime object failures to Deployed/Failed, improving signal specificity. Switches from setDeploymentStatus to reconcileStatus and requeues when runtime/deployment isn’t ready to re-check pod/container health.

internal/controller/backstage_controller.go

backstage_status.goImplement Runtime condition, improved Deployed messages, and plugin publication +172/-18

Implement Runtime condition, improved Deployed messages, and plugin publication

• Introduces reconcileStatus to set Runtime (pod/container) and Deployed (workload) conditions, requiring both for overall readiness. Adds detailed container state inspection (waiting/terminated/crash-loop with 'ready' recovery logic), improves rollout messages to 'X/Y replicas ready', and publishes a sorted enabled-plugin list only when healthy.

internal/controller/backstage_status.go

runtime.goExpose enabled dynamic plugins from the model +12/-2

Expose enabled dynamic plugins from the model

• Adds BackstageModel.GetEnabledPlugins to retrieve enabled dynamic plugin packages (or an empty slice when absent). Also slightly simplifies validation loop error handling.

pkg/model/runtime.go

Bug fix (1) +11 / -1

install_plugins.shPreserve xargs failures and emit termination message fallback +11/-1

Preserve xargs failures and emit termination message fallback

• Captures xargs exit status from parallel plugin downloads and fails the script with a clear message if xargs failed but no FAILURE_LOG was produced. Improves diagnosability and prevents silent success on download failures.

plugin-installer/install_plugins.sh

Refactor (4) +4 / -4

zz_generated.deepcopy.goFix import aliasing in generated deepcopy (v1alpha1) +1/-1

Fix import aliasing in generated deepcopy (v1alpha1)
• Updates the metav1 import to use an explicit alias, reflecting regeneration/gofmt normalization. No behavioral change expected.
api/v1alpha1/zz_generated.deepcopy.go

zz_generated.deepcopy.goFix import aliasing in generated deepcopy (v1alpha2) +1/-1

Fix import aliasing in generated deepcopy (v1alpha2)
• Updates the apiextensions v1 import to use an explicit alias, reflecting regeneration/gofmt normalization. No behavioral change expected.
api/v1alpha2/zz_generated.deepcopy.go

zz_generated.deepcopy.goFix import aliasing in generated deepcopy (v1alpha3) +1/-1

Fix import aliasing in generated deepcopy (v1alpha3)
• Updates the apiextensions v1 import to use an explicit alias, reflecting regeneration/gofmt normalization. No behavioral change expected.
api/v1alpha3/zz_generated.deepcopy.go

zz_generated.deepcopy.goFix import aliasing in generated deepcopy (v1alpha4) +1/-1

Fix import aliasing in generated deepcopy (v1alpha4)
• Updates the apiextensions v1 import to use an explicit alias, reflecting regeneration/gofmt normalization. No behavioral change expected.
api/v1alpha4/zz_generated.deepcopy.go

Tests (3) +500 / -6

default-config_test.goUpdate integration assertions for two status conditions +7/-6

Update integration assertions for two status conditions
• Adjusts tests to expect both Deployed and Runtime conditions and uses meta.FindStatusCondition to assert by condition type rather than slice order. Improves stability as conditions expand.
integration_tests/default-config_test.go

rhdh-config_test.goEnsure namespaces are cleaned up after integration tests +2/-0

Ensure namespaces are cleaned up after integration tests
• Adds deleteNamespace calls after specific test flows to reduce resource leakage and cross-test interference in the suite.
integration_tests/rhdh-config_test.go

backstage_status_test.goAdd comprehensive unit tests for container/deploy/statefulset status logic +491/-0

Add comprehensive unit tests for container/deploy/statefulset status logic
• Adds ~490 LOC of tests covering main vs init container states, crash-loop recovery behavior, deployment/statefulset message formatting, and condition upsert semantics. Provides regression coverage for the new Runtime condition heuristics.
internal/controller/backstage_status_test.go

Documentation (1) +55 / -5

design.mdDocument Deployed vs Runtime conditions and plugins behavior +55/-5

Document Deployed vs Runtime conditions and plugins behavior

• Replaces the single-condition description with tables defining Deployed and Runtime condition semantics, reasons, and example messages. Adds an end-to-end status YAML example and clarifies that plugins are only listed when fully healthy.

docs/design.md

Other (2) +6 / -1

rhdh.redhat.com_backstages.yamlExtend CRD schema with status.plugins +6/-0

Extend CRD schema with status.plugins
• Adds the plugins array under status to the CRD OpenAPI schema, documenting that it is sorted alphabetically. Enables kubectl/clients to validate and display the field.
config/crd/bases/rhdh.redhat.com_backstages.yaml

deployment.yamlTidy default deployment env var comment +0/-1

Tidy default deployment env var comment
• Removes an outdated inline comment above CATALOG_INDEX_IMAGE. No functional change to the rendered deployment.
config/profile/rhdh/default-config/deployment.yaml

rhdh-qodo-merge · 2026-08-07T13:18:51Z

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Startup waits flagged failed 🐞 Bug ≡ Correctness

Description

checkContainerState() treats any non-empty container Waiting reason (except PodInitializing) as
ContainerFailed, which will mark Runtime as failed during normal startup/transient waits that the
design explicitly categorizes as Pending.

Code

internal/controller/backstage_status.go[R149-153]

+	if cs.State.Waiting != nil && cs.State.Waiting.Reason != "" {
+		if cs.State.Waiting.Reason == "PodInitializing" {
+			return "", "" // Normal state, not an error
+		}
+		return api.BackstageConditionReasonContainerFailed, fmt.Sprintf("%s %q: %s", prefix, cs.Name, cs.State.Waiting.Reason)

Relevance

●● Moderate
Status semantics change (Waiting reasons) could be debated vs design; prior status-condition
behavioral tweaks were sometimes rejected.
PR-#1949

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The design doc defines Pending as covering containers still starting; however the new logic returns
ContainerFailed for any Waiting reason other than PodInitializing, which will violate that
definition for transient startup waits.

internal/controller/backstage_status.go[141-155]
docs/design.md[104-111]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`checkContainerState()` returns `ContainerFailed` for almost all Waiting reasons. This conflicts with the design doc which states Pending should cover containers "still starting"; many normal startup states are represented as Waiting with a reason.

## Issue Context
You already correctly classify `ImagePullBackOff` and `CrashLoopBackOff` as failure scenarios. The issue is the default behavior for Waiting reasons is too aggressive.

## Fix Focus Areas
- internal/controller/backstage_status.go[141-155]
- docs/design.md[104-111]
- internal/controller/backstage_status_test.go[14-147]

### Suggested fix
1. Replace the current Waiting handling with a small classifier:
  - Treat known transient reasons as `Pending` (example set: `PodInitializing`, `ContainerCreating`).
  - Treat known failure reasons as `ContainerFailed` (keep `ImagePullBackOff`, `ErrImagePull`, `CrashLoopBackOff`, and optionally config errors like `CreateContainerConfigError`).
  - For unknown Waiting reasons, prefer `Pending` unless you have strong evidence it's terminal.
2. Add unit tests for a transient Waiting reason (e.g., `ContainerCreating`) asserting Runtime reason `Pending`, not `ContainerFailed`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~Idling guidance contradicts status~~ ✓ Resolved 🐞 Bug ≡ Correctness ⭐ New

Description

The updated design and controller now report an idled instance as Deployed=True with reason
Deployed, but the admin guide still promises Deployed=False with reason Idled. Operators and
CI scripts following that guide will interpret idled instances incorrectly.

Code

docs/design.md[102]

+**Idled instances:** When the `rhdh.redhat.com/idle: true` annotation is set, the deployment scales to 0 replicas. The Deployed condition shows `Deployed` with message "0/0 replicas ready (Idled)".

Relevance

●●● Strong
Recent history accepts documentation corrections that align operator behavior with user-facing
guidance; this status contradiction is concrete.
PR-#3312
PR-#2000
PR-#3215

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed design states that idled workloads report a Deployed condition with a 0/0 message, and
the controller preserves the Deployed reason/status while appending (Idled). In contrast, the
existing admin guide explicitly tells users and CI scripts to expect Status: False and `Reason:
Idled`.

docs/design.md[102-102]
internal/controller/backstage_status.go[65-75]
docs/admin.md[195-204]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The idling status semantics changed to `Deployed=True`, reason `Deployed`, and message `0/0 replicas ready (Idled)`, but the admin guide still documents the removed `Deployed=False`/`Idled` behavior and gives CI guidance based on it.

## Issue Context
The controller now derives readiness from the zero-replica workload state and only appends `(Idled)` to its message. Update all user-facing idling guidance to describe the actual condition and revise the CI/monitoring note accordingly.

## Fix Focus Areas
- docs/admin.md[195-204]
- docs/design.md[102-102]
- internal/controller/backstage_status.go[65-75]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. ~~Pod List() missing RBAC~~ ✓ Resolved 📘 Rule violation ⛨ Security

Description

The controller now lists Pods to compute the Runtime status condition, but the operator/controller
RBAC does not grant access to the core pods resource, which can cause reconciliation and status
updates to fail with RBAC forbidden errors. This can leave the Runtime condition stuck in
Pending and trigger continuous requeues.

Code

internal/controller/backstage_status.go[R83-86]

+	labelSelector := client.MatchingLabels{
+		model.BackstageAppLabel: utils.BackstageAppLabelValue(backstage.Name),
+	}
+	if err := r.List(ctx, podList, client.InNamespace(backstage.Namespace), labelSelector); err != nil {

Relevance

●●● Strong
Missing RBAC for newly listed resource is a common runtime-breaker; team has accepted adding RBAC
for new kinds.
PR-#1651

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR-added runtime status computation calls r.List(...) on a corev1.PodList to derive the
Runtime condition, which requires Kubernetes RBAC permissions on the core pods resource.
However, the existing RBAC coverage (both the generated/annotated controller rules and the operator
ClusterRole in config/rbac/role.yaml) includes other core/app resources (e.g., configmaps,
secrets, services, PVCs, deployments, statefulsets, etc.) but omits pods, so the new Pod listing
operation is not authorized and would fail at runtime with forbidden errors, preventing the
condition from progressing beyond Pending.

Rule 8: Ensure operator RBAC includes required verbs for managed Kubernetes resource kinds
internal/controller/backstage_status.go[83-90]
config/rbac/role.yaml[7-21]
internal/controller/backstage_status.go[79-92]
internal/controller/backstage_controller.go[47-58]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Backstage controller now calls `r.List(...)` on `corev1.PodList` to compute/derive the `Runtime` status condition, but the operator/controller RBAC does not include permissions for the core `pods` resource. This can result in RBAC `forbidden` errors during reconciliation, leaving `Runtime` stuck in Pending and causing repeated requeues.

## Issue Context
The pod-listing behavior was introduced as part of the new Runtime condition computation, so RBAC must be updated to include at least `list` (and typically `get;list;watch`) on `pods`, and the repo’s manifest/RBAC generation pipeline (e.g., controller-gen/kustomize) should be rerun so the deployed Role/ClusterRole includes these permissions. Optionally, surfacing the underlying list error in the Runtime condition message can make RBAC/debugging failures easier to diagnose.

## Fix Focus Areas
- internal/controller/backstage_status.go[79-92]
- internal/controller/backstage_status.go[83-86]
- internal/controller/backstage_controller.go[47-58]
- config/rbac/role.yaml[7-21]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

4. Terminating pods skew Runtime 🐞 Bug ☼ Reliability

Description

checkPodStates() evaluates all pods matching the Backstage app label and can return Pending/Failed
based on old terminating pods, so Runtime may stay unhealthy during rollouts even when the newest
replacement pod is Running/Ready.

Code

internal/controller/backstage_status.go[R111-115]

+	for _, pod := range pods {
+		// Check pod phase for terminal failures
+		if pod.Status.Phase == corev1.PodFailed {
+			return api.BackstageConditionReasonContainerFailed, fmt.Sprintf("pod %q failed", pod.Name)
+		}

Relevance

●● Moderate
Pod selection during rollouts (terminating/stale pods) is subtle; no strong repo precedent for
excluding them.
PR-#1652

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Pods are selected solely by the Backstage app label (set on the pod template/selector), and the scan
iterates all matching pods without excluding terminating pods, making stale rollout pods able to
affect the Runtime result.

internal/controller/backstage_status.go[104-136]
pkg/model/deployment.go[160-165]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Runtime condition scans every pod that matches the Backstage app label. During rolling updates, this label matches both new and old pods. Because the scan does not skip terminating pods, an old pod with non-ready containers can incorrectly determine the Runtime condition.

## Issue Context
The Deployment model sets the Backstage app label on the pod template and uses it in the selector, so all ReplicaSet pods (including old ones) will match.

## Fix Focus Areas
- internal/controller/backstage_status.go[104-136]
- pkg/model/deployment.go[160-165]

### Suggested fix
1. Filter pods before evaluation:
  - Skip pods with `pod.DeletionTimestamp != nil` (terminating).
  - Optionally skip `pod.Status.Phase == Succeeded`.
2. If all matching pods are skipped and none remain, report `Pending` with a clear message (e.g., "no active pods found").
3. Add unit tests for `checkPodStates()` covering a mix of terminating + healthy pods to ensure terminating pods don't force Pending/Failed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

View medium (1)

5. Unbounded 10s status polling 🐞 Bug ➹ Performance

Description

Reconcile() requeues every 10 seconds for any non-healthy status (including ContainerFailed),
causing repeated apply+status loops and API churn during persistent failures because the controller
does not watch Pods for status changes.

Code

internal/controller/backstage_controller.go[R119-123]

+	isReady := r.reconcileStatus(ctx, &backstage, *bsModel)
+	if !isReady {
+		// Requeue to check pod status again (for init container failures, etc.)
+		return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+	}

Relevance

●● Moderate
Polling/requeue strategy is behavioral/perf tradeoff; no close precedent showing team will change it
now.
PR-#2084

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Reconcile explicitly requeues on any !isReady, and Runtime health is only true for reason Running;
combined with no Pod watches, this results in fixed-interval polling during persistent Runtime
failures.
internal/controller/backstage_controller.go[113-123]
internal/controller/backstage_status.go[76-102]
internal/controller/watchers.go[78-104]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The controller now requeues every 10 seconds whenever `reconcileStatus()` returns false. This includes terminal/persistent failure cases (e.g., Runtime `ContainerFailed`) and will continuously re-run reconciliation, including re-applying objects, which increases load.

## Issue Context
The controller watches Deployments/StatefulSets but not Pods, so polling is currently the only way Runtime can change. That makes it important to avoid tight polling for terminal failure modes.

## Fix Focus Areas
- internal/controller/backstage_controller.go[113-123]
- internal/controller/backstage_status.go[19-40]
- internal/controller/watchers.go[78-104]

### Suggested fix
1. Have `reconcileStatus()` (or `setRuntimeCondition`) return (ready, runtimeReason) so Reconcile can requeue selectively:
  - Requeue for `Pending` / `DeployInProgress`.
  - Do not requeue (or requeue with a much larger backoff) for `ContainerFailed` / `DeployFailed`.
2. Alternatively/additionally, add a Pod watch that enqueues the owning Backstage CR by label, which reduces the need for polling (requires pod RBAC as well).
3. If keeping polling, implement exponential backoff keyed by condition reason to avoid sustained 10s loops.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Context sources

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 18 rules

✅ Cross-repo context — repo relationships

Explored: repo: redhat-developer/rhdh (sha: 4462977e)

Explored: repo: redhat-developer/rhdh-plugins (sha: 5192aeff)

Tip of the day

💡 Did you know, you can hide the parts of a finding you never read, like the evidence or the agent prompt

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Previous reviews

Review updated until commit 9e508dc

Results up to commit 90db027 ⚖️ Balanced

🐞 Bugs (3) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Startup waits flagged failed 🐞 Bug ≡ Correctness

Description

checkContainerState() treats any non-empty container Waiting reason (except PodInitializing) as
ContainerFailed, which will mark Runtime as failed during normal startup/transient waits that the
design explicitly categorizes as Pending.

Code

internal/controller/backstage_status.go[R149-153]

+	if cs.State.Waiting != nil && cs.State.Waiting.Reason != "" {
+		if cs.State.Waiting.Reason == "PodInitializing" {
+			return "", "" // Normal state, not an error
+		}
+		return api.BackstageConditionReasonContainerFailed, fmt.Sprintf("%s %q: %s", prefix, cs.Name, cs.State.Waiting.Reason)

Relevance

●● Moderate
Status semantics change (Waiting reasons) could be debated vs design; prior status-condition
behavioral tweaks were sometimes rejected.
PR-#1949

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The design doc defines Pending as covering containers still starting; however the new logic returns
ContainerFailed for any Waiting reason other than PodInitializing, which will violate that
definition for transient startup waits.

internal/controller/backstage_status.go[141-155]
docs/design.md[104-111]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`checkContainerState()` returns `ContainerFailed` for almost all Waiting reasons. This conflicts with the design doc which states Pending should cover containers "still starting"; many normal startup states are represented as Waiting with a reason.

## Issue Context
You already correctly classify `ImagePullBackOff` and `CrashLoopBackOff` as failure scenarios. The issue is the default behavior for Waiting reasons is too aggressive.

## Fix Focus Areas
- internal/controller/backstage_status.go[141-155]
- docs/design.md[104-111]
- internal/controller/backstage_status_test.go[14-147]

### Suggested fix
1. Replace the current Waiting handling with a small classifier:
  - Treat known transient reasons as `Pending` (example set: `PodInitializing`, `ContainerCreating`).
  - Treat known failure reasons as `ContainerFailed` (keep `ImagePullBackOff`, `ErrImagePull`, `CrashLoopBackOff`, and optionally config errors like `CreateContainerConfigError`).
  - For unknown Waiting reasons, prefer `Pending` unless you have strong evidence it's terminal.
2. Add unit tests for a transient Waiting reason (e.g., `ContainerCreating`) asserting Runtime reason `Pending`, not `ContainerFailed`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~Pod List() missing RBAC~~ ✓ Resolved 📘 Rule violation ⛨ Security

Description

The controller now lists Pods to compute the Runtime status condition, but the operator/controller
RBAC does not grant access to the core pods resource, which can cause reconciliation and status
updates to fail with RBAC forbidden errors. This can leave the Runtime condition stuck in
Pending and trigger continuous requeues.

Code

internal/controller/backstage_status.go[R83-86]

+	labelSelector := client.MatchingLabels{
+		model.BackstageAppLabel: utils.BackstageAppLabelValue(backstage.Name),
+	}
+	if err := r.List(ctx, podList, client.InNamespace(backstage.Namespace), labelSelector); err != nil {

Relevance

●●● Strong
Missing RBAC for newly listed resource is a common runtime-breaker; team has accepted adding RBAC
for new kinds.
PR-#1651

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR-added runtime status computation calls r.List(...) on a corev1.PodList to derive the
Runtime condition, which requires Kubernetes RBAC permissions on the core pods resource.
However, the existing RBAC coverage (both the generated/annotated controller rules and the operator
ClusterRole in config/rbac/role.yaml) includes other core/app resources (e.g., configmaps,
secrets, services, PVCs, deployments, statefulsets, etc.) but omits pods, so the new Pod listing
operation is not authorized and would fail at runtime with forbidden errors, preventing the
condition from progressing beyond Pending.

Rule 8: Ensure operator RBAC includes required verbs for managed Kubernetes resource kinds
internal/controller/backstage_status.go[83-90]
config/rbac/role.yaml[7-21]
internal/controller/backstage_status.go[79-92]
internal/controller/backstage_controller.go[47-58]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Backstage controller now calls `r.List(...)` on `corev1.PodList` to compute/derive the `Runtime` status condition, but the operator/controller RBAC does not include permissions for the core `pods` resource. This can result in RBAC `forbidden` errors during reconciliation, leaving `Runtime` stuck in Pending and causing repeated requeues.

## Issue Context
The pod-listing behavior was introduced as part of the new Runtime condition computation, so RBAC must be updated to include at least `list` (and typically `get;list;watch`) on `pods`, and the repo’s manifest/RBAC generation pipeline (e.g., controller-gen/kustomize) should be rerun so the deployed Role/ClusterRole includes these permissions. Optionally, surfacing the underlying list error in the Runtime condition message can make RBAC/debugging failures easier to diagnose.

## Fix Focus Areas
- internal/controller/backstage_status.go[79-92]
- internal/controller/backstage_status.go[83-86]
- internal/controller/backstage_controller.go[47-58]
- config/rbac/role.yaml[7-21]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. Terminating pods skew Runtime 🐞 Bug ☼ Reliability

Description

checkPodStates() evaluates all pods matching the Backstage app label and can return Pending/Failed
based on old terminating pods, so Runtime may stay unhealthy during rollouts even when the newest
replacement pod is Running/Ready.

Code

internal/controller/backstage_status.go[R111-115]

+	for _, pod := range pods {
+		// Check pod phase for terminal failures
+		if pod.Status.Phase == corev1.PodFailed {
+			return api.BackstageConditionReasonContainerFailed, fmt.Sprintf("pod %q failed", pod.Name)
+		}

Relevance

●● Moderate
Pod selection during rollouts (terminating/stale pods) is subtle; no strong repo precedent for
excluding them.
PR-#1652

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Pods are selected solely by the Backstage app label (set on the pod template/selector), and the scan
iterates all matching pods without excluding terminating pods, making stale rollout pods able to
affect the Runtime result.

internal/controller/backstage_status.go[104-136]
pkg/model/deployment.go[160-165]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Runtime condition scans every pod that matches the Backstage app label. During rolling updates, this label matches both new and old pods. Because the scan does not skip terminating pods, an old pod with non-ready containers can incorrectly determine the Runtime condition.

## Issue Context
The Deployment model sets the Backstage app label on the pod template and uses it in the selector, so all ReplicaSet pods (including old ones) will match.

## Fix Focus Areas
- internal/controller/backstage_status.go[104-136]
- pkg/model/deployment.go[160-165]

### Suggested fix
1. Filter pods before evaluation:
  - Skip pods with `pod.DeletionTimestamp != nil` (terminating).
  - Optionally skip `pod.Status.Phase == Succeeded`.
2. If all matching pods are skipped and none remain, report `Pending` with a clear message (e.g., "no active pods found").
3. Add unit tests for `checkPodStates()` covering a mix of terminating + healthy pods to ensure terminating pods don't force Pending/Failed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

4. Unbounded 10s status polling 🐞 Bug ➹ Performance

Description

Reconcile() requeues every 10 seconds for any non-healthy status (including ContainerFailed),
causing repeated apply+status loops and API churn during persistent failures because the controller
does not watch Pods for status changes.

Code

internal/controller/backstage_controller.go[R119-123]

+	isReady := r.reconcileStatus(ctx, &backstage, *bsModel)
+	if !isReady {
+		// Requeue to check pod status again (for init container failures, etc.)
+		return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
+	}

Relevance

●● Moderate
Polling/requeue strategy is behavioral/perf tradeoff; no close precedent showing team will change it
now.
PR-#2084

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Reconcile explicitly requeues on any !isReady, and Runtime health is only true for reason Running;
combined with no Pod watches, this results in fixed-interval polling during persistent Runtime
failures.
internal/controller/backstage_controller.go[113-123]
internal/controller/backstage_status.go[76-102]
internal/controller/watchers.go[78-104]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The controller now requeues every 10 seconds whenever `reconcileStatus()` returns false. This includes terminal/persistent failure cases (e.g., Runtime `ContainerFailed`) and will continuously re-run reconciliation, including re-applying objects, which increases load.

## Issue Context
The controller watches Deployments/StatefulSets but not Pods, so polling is currently the only way Runtime can change. That makes it important to avoid tight polling for terminal failure modes.

## Fix Focus Areas
- internal/controller/backstage_controller.go[113-123]
- internal/controller/backstage_status.go[19-40]
- internal/controller/watchers.go[78-104]

### Suggested fix
1. Have `reconcileStatus()` (or `setRuntimeCondition`) return (ready, runtimeReason) so Reconcile can requeue selectively:
  - Requeue for `Pending` / `DeployInProgress`.
  - Do not requeue (or requeue with a much larger backoff) for `ContainerFailed` / `DeployFailed`.
2. Alternatively/additionally, add a Pod watch that enqueues the owning Backstage CR by label, which reduces the need for polling (requires pod RBAC as well).
3. If keeping polling, implement exponential backoff keyed by condition reason to avoid sustained 10s loops.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

# Conflicts: # api/current-types.go # api/v1alpha5/backstage_types.go # internal/controller/backstage_status.go

codecov · 2026-08-10T08:25:28Z

Codecov Report

❌ Patch coverage is 58.45070% with 59 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.31%. Comparing base (22ee1d4) to head (9e508dc).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/controller/backstage_status.go	65.60%	43 Missing ⚠️
internal/controller/backstage_controller.go	0.00%	11 Missing ⚠️
pkg/model/runtime.go	16.66%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3348      +/-   ##
==========================================
+ Coverage   63.97%   65.31%   +1.33%     
==========================================
  Files          38       38              
  Lines        2365     2465     +100     
==========================================
+ Hits         1513     1610      +97     
- Misses        702      705       +3     
  Partials      150      150

Flag	Coverage Δ
nightly	`?`
unittests	`65.31% <58.45%> (+1.33%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/model/runtime.go	`74.46% <16.66%> (-4.43%)`	⬇️
internal/controller/backstage_controller.go	`0.00% <0.00%> (ø)`
internal/controller/backstage_status.go	`63.63% <65.60%> (+63.63%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Fortune-Ndlovu · 2026-08-19T15:23:30Z

/agentic_review

rhdh-qodo-merge · 2026-08-19T15:26:26Z

Code review by qodo was updated up to the latest commit 95246b6

Fortune-Ndlovu

Great CR Status improvements. See inline comments for details and suggested fixes.

rm3l · 2026-08-23T20:14:30Z

/build-images

github-actions · 2026-08-23T20:22:09Z

❌ Image build failed

See workflow run for details: https://github.com/redhat-developer/rhdh-operator/actions/runs/32663805630

Triggered by @rm3l

gazarenkov · 2026-08-25T07:34:24Z

/build-images

github-actions · 2026-08-25T07:38:06Z

❌ Image build failed

See workflow run for details: https://github.com/redhat-developer/rhdh-operator/actions/runs/32822255509

Triggered by @gazarenkov

sonarqubecloud · 2026-08-25T07:50:04Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

gazarenkov · 2026-08-25T07:53:21Z

/build-images

github-actions · 2026-08-25T07:56:27Z

✅ PR images built successfully!

Images are available for testing:

Operator: quay.io/rhdh-community/operator:2.0.0-pr-3348-9e508dc
Bundle: quay.io/rhdh-community/operator-bundle:2.0.0-pr-3348-9e508dc
Catalog: quay.io/rhdh-community/operator-catalog:2.0.0-pr-3348-9e508dc

Also available with PR number tag:

quay.io/rhdh-community/operator:2.0.0-pr-3348
quay.io/rhdh-community/operator-bundle:2.0.0-pr-3348
quay.io/rhdh-community/operator-catalog:2.0.0-pr-3348

_{Triggered by @gazarenkov}

Fortune-Ndlovu

lgtm

gazarenkov added 6 commits August 3, 2026 19:07

added statuses and plugin list

b2d5650

add status_test

006ddc2

Merge remote-tracking branch 'upstream/main' into plugin-status

10dc7b7

Merge remote-tracking branch 'upstream/main' into plugin-status

52630f5

fix statuses, tests and doc

0c50014

Merge remote-tracking branch 'upstream/main' into plugin-status

90db027

gazarenkov requested a review from a team as a code owner August 7, 2026 13:07

openshift-ci Bot requested review from OpinionatedHeron and rm3l August 7, 2026 13:07

openshift-ci Bot added the needs-rebase label Aug 7, 2026

rhdh-qodo-merge Bot added documentation Improvements or additions to documentation enhancement New feature or request Tests Bug fix labels Aug 7, 2026

gazarenkov added 2 commits August 10, 2026 10:27

Merge remote-tracking branch 'upstream/main' into plugin-status

adb0eb6

# Conflicts: # api/current-types.go # api/v1alpha5/backstage_types.go # internal/controller/backstage_status.go

merge with latest changes

7f5f751

openshift-ci Bot removed the needs-rebase label Aug 10, 2026

lint fix

2e5dfd1

gazarenkov added 3 commits August 10, 2026 13:12

script fix and make bundle

060b83f

make bundle

c07a823

remove idled condition

95246b6

Fortune-Ndlovu requested changes Aug 19, 2026

View reviewed changes

openshift-ci Bot assigned Fortune-Ndlovu Aug 19, 2026

fixes

345bb93

gazarenkov requested a review from Fortune-Ndlovu August 20, 2026 12:08

gazarenkov added 3 commits August 20, 2026 15:14

fixes

2d2bc10

fixes

dd884e4

Merge remote-tracking branch 'upstream/main' into plugin-status

a77d7bc

redhat-developer deleted a comment from github-actions Bot Aug 23, 2026

rm3l reviewed Aug 23, 2026

View reviewed changes

Comment thread internal/controller/backstage_controller.go Outdated

Comment thread internal/controller/backstage_status.go

Comment thread api/current-types.go

gazarenkov added 2 commits August 24, 2026 12:49

RolloutStalled reason

562e015

idled doc

a6560ae

gazarenkov requested a review from rm3l August 24, 2026 10:14

Fortune-Ndlovu removed their assignment Aug 24, 2026

rm3l approved these changes Aug 24, 2026

View reviewed changes

openshift-ci Bot assigned rm3l Aug 24, 2026

openshift-ci Bot added the lgtm label Aug 24, 2026

Fortune-Ndlovu mentioned this pull request Aug 24, 2026

ADR-008: Reconciliation pause for Backstage CR redhat-developer/rhdh-adr#25

Closed

Merge remote-tracking branch 'upstream/main' into plugin-status

9e508dc

openshift-ci Bot removed the lgtm label Aug 25, 2026

Fortune-Ndlovu approved these changes Aug 25, 2026

View reviewed changes

openshift-ci Bot assigned Fortune-Ndlovu Aug 25, 2026

openshift-ci Bot added the lgtm label Aug 25, 2026

gazarenkov merged commit d90d743 into redhat-developer:main Aug 25, 2026
11 of 12 checks passed

Uh oh!

Conversation

gazarenkov commented Aug 7, 2026

Description

Which issue(s) does this PR fix or relate to

PR acceptance criteria

Building Container Images for Testing

Uh oh!

rhdh-qodo-merge Bot commented Aug 7, 2026

PR Summary by Qodo

Uh oh!

rhdh-qodo-merge Bot commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

codecov Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Fortune-Ndlovu commented Aug 19, 2026

Uh oh!

rhdh-qodo-merge Bot commented Aug 19, 2026

Uh oh!

Fortune-Ndlovu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rm3l commented Aug 23, 2026

Uh oh!

github-actions Bot commented Aug 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gazarenkov commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

sonarqubecloud Bot commented Aug 25, 2026

Quality Gate passed

Uh oh!

gazarenkov commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

Fortune-Ndlovu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rhdh-qodo-merge Bot commented Aug 7, 2026 •

edited

Loading

codecov Bot commented Aug 10, 2026 •

edited

Loading

Fortune-Ndlovu left a comment •

edited

Loading