Repo Radius: externalize control-plane and Terraform state (rad startup / rad shutdown) - #12213
Closed
sylvainsf wants to merge 4 commits into
Closed
Repo Radius: externalize control-plane and Terraform state (rad startup / rad shutdown)#12213sylvainsf wants to merge 4 commits into
sylvainsf wants to merge 4 commits into
Conversation
Fixes the pre-existing gaps that prevented database.enabled=true from producing a working PostgreSQL-backed control plane: - UCP, Applications RP, and Dynamic RP configmaps/deployments were hardcoded to the apiserver provider with no database.enabled conditional; they now switch to the postgresql provider and inject POSTGRES_PASSWORD from the database secret when database.enabled=true. - Add init-db ConfigMap (mounted at /docker-entrypoint-initdb.d) that creates the per-RP databases, users, and tables on first start (option 3 from radius-project#8398). - Fix POSTGRES_DB secret value (was the literal string POSTGRES_DB). - Pin the postgres image tag to 16-alpine. - Fix the databaseProvider URL env-var substitution in factory.go, which replaced the entire URL with the first captured variable name instead of expanding env-var references. Adds helm-unittest coverage for the conditional rendering and a unit test for the env-var expansion helper. Also adds the Repo Radius state-storage technical design note. Relates to radius-project#8096, radius-project#8398
Terraform recipes store their state in Kubernetes Secrets (the Terraform
"kubernetes" backend) in the radius-system namespace, not in the Radius
PostgreSQL databases. On an ephemeral control plane those Secrets are
destroyed on teardown, so a second deploy of the same Terraform-backed
resource in a later run plans against an empty backend and either fails
or orphans cloud resources.
Add a pkg/cli/tfstate package that exports the Secrets labelled
tfstate=true to a state directory and restores them into a fresh cluster
before any deploy runs. The label selector also captures the chunked
tfstate-{workspace}-{suffix}-{index} Secrets the backend creates for
large state. Server-managed fields are stripped on backup, and restore
is idempotent (create-or-update).
Covered by unit tests using the client-go fake clientset.
Relates to radius-project#8096
Adds the two commands that back up and restore all durable Radius state across an ephemeral control plane, plus the end-to-end lifecycle test. The commands operate on the current workspace's Kubernetes context like any other command. They do not create or delete clusters and do not install Radius; cluster lifecycle is the caller's responsibility. There is no dedicated workspace kind. - pkg/cli/pgbackup: control-plane PostgreSQL backup/restore via "kubectl exec pg_dump/psql". - pkg/cli/gitstate: persists the state directory to a git orphan branch (radius-state) in an isolated worktree. CommitAndPush fails loudly when a remote is configured (a failed backup push would otherwise be silent data loss) and tolerates the no-remote local/test case. - rad shutdown: backs up the control-plane databases and the Terraform state Secrets, then commits and pushes them. - rad startup: waits for the database, then restores the control-plane databases and the Terraform state Secrets. The lifecycle test (test/functional-portable/statestore) installs Radius with database.enabled=true, deploys a Terraform-backed resource, shuts down, uninstalls, reinstalls, starts up, and deploys an update to the same resource -- the cross-run path that fails when Terraform state is lost. It is destructive and requires a cluster, so it is skipped unless RADIUS_STATE_E2E is set. Relates to radius-project#8096
Radius functional test overviewClick here to see the test run details
Test Status⌛ Building Radius and pushing container images for functional tests... |
Contributor
Author
|
Closing to re-open from an upstream branch so workflow changes run on the PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Repo Radius state externalization support by introducing rad shutdown / rad startup to persist and restore durable control-plane state (PostgreSQL) and Terraform recipe state (Kubernetes Secrets) across ephemeral control planes, plus the Helm chart plumbing to make database.enabled=true actually functional.
Changes:
- Adds
rad startup/rad shutdowncommands (plus unit tests) that use a git orphan branch worktree (radius-state) as the persisted state location. - Introduces state backup/restore libraries:
pkg/cli/pgbackup(pg_dump/psql via kubectl) andpkg/cli/tfstate(export/restoretfstate=trueSecrets). - Updates the Helm chart to properly switch RPs/UCP to the PostgreSQL provider when
database.enabled=true, pins Postgres image to16-alpine, and adds init-db ConfigMap + helm-unittest coverage.
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| test/functional-portable/statestore/noncloud/statestore_lifecycle_test.go | Adds a gated destructive e2e lifecycle test for shutdown/startup restore across uninstall/reinstall. |
| pkg/components/database/databaseprovider/storageprovider_test.go | Adds unit tests for URL env-var expansion in PostgreSQL connection strings. |
| pkg/components/database/databaseprovider/factory.go | Fixes PostgreSQL URL env-var substitution by expanding ${VAR} occurrences from the environment. |
| pkg/cli/tfstate/tfstate.go | Implements backup/restore of Terraform backend state Secrets to/from a state directory. |
| pkg/cli/tfstate/tfstate_test.go | Adds fake-client round-trip tests and idempotent restore coverage for tfstate Secrets. |
| pkg/cli/pgbackup/pgbackup.go | Implements PostgreSQL dump/restore and readiness wait via kubectl exec/kubectl wait. |
| pkg/cli/gitstate/gitstate.go | Implements state persistence to a git orphan branch via isolated worktrees + commit/push. |
| pkg/cli/gitstate/gitstate_test.go | Adds tests covering orphan-branch creation, isolation, and push-to-remote behavior. |
| pkg/cli/cmd/startup/stateclient.go | Wraps pgbackup/tfstate for testable rad startup restore operations. |
| pkg/cli/cmd/startup/startup.go | Adds rad startup command and runner wiring (wait DB → restore DBs → restore tfstate). |
| pkg/cli/cmd/startup/startup_test.go | Adds runner unit tests validating order and error short-circuiting. |
| pkg/cli/cmd/shutdown/stateclient.go | Wraps pgbackup/tfstate for testable rad shutdown backup operations. |
| pkg/cli/cmd/shutdown/shutdown.go | Adds rad shutdown command and runner wiring (backup DBs + tfstate → commit/push). |
| pkg/cli/cmd/shutdown/shutdown_test.go | Adds runner unit tests validating backup + commit behavior and failures. |
| eng/design-notes/2026-06-repo-radius-state-storage.md | Adds the technical design note for Repo Radius state storage (Investment 2). |
| deploy/Chart/values.yaml | Pins Postgres image tag to 16-alpine. |
| deploy/Chart/tests/database_test.yaml | Adds helm-unittest coverage for database.enabled provider switching and initdb mount. |
| deploy/Chart/templates/ucp/deployment.yaml | Injects POSTGRES_PASSWORD env var into UCP when database.enabled=true. |
| deploy/Chart/templates/ucp/configmaps.yaml | Switches UCP database provider to PostgreSQL when enabled, otherwise apiserver. |
| deploy/Chart/templates/rp/deployment.yaml | Injects POSTGRES_PASSWORD env var into Applications RP when database.enabled=true. |
| deploy/Chart/templates/rp/configmaps.yaml | Switches Applications RP database provider to PostgreSQL when enabled. |
| deploy/Chart/templates/dynamic-rp/deployment.yaml | Injects POSTGRES_PASSWORD env var into Dynamic RP when database.enabled=true. |
| deploy/Chart/templates/dynamic-rp/configmaps.yaml | Switches Dynamic RP database provider to PostgreSQL when enabled. |
| deploy/Chart/templates/database/statefulset.yaml | Mounts init-db scripts into /docker-entrypoint-initdb.d. |
| deploy/Chart/templates/database/configmaps.yaml | Fixes POSTGRES_DB secret value to a literal "radius". |
| deploy/Chart/templates/database/configmap-initdb.yaml | Adds init-db script configmap to create per-RP DBs/users and a resources table. |
| cmd/rad/cmd/root.go | Wires startup and shutdown into the root rad CLI command tree. |
Comment on lines
+81
to
+82
| out, err := cli.RunCommand(ctx, []string{"uninstall", "kubernetes", "--purge"}) | ||
| require.NoErrorf(t, err, "rad uninstall failed: %s", out) |
Comment on lines
+13
to
+14
| #!/bin/bash | ||
| set -e |
Comment on lines
+16
to
+18
| RESOURCE_PROVIDERS=("ucp" "applications_rp" "dynamic_rp") | ||
|
|
||
| for RESOURCE_PROVIDER in "${RESOURCE_PROVIDERS[@]}"; do |
| EOSQL | ||
| done | ||
|
|
||
| for RESOURCE_PROVIDER in "${RESOURCE_PROVIDERS[@]}"; do |
Comment on lines
+102
to
+105
| outPath := filepath.Join(stateDir, db+".sql") | ||
| if err := os.WriteFile(outPath, stdout.Bytes(), 0o644); err != nil { | ||
| return fmt.Errorf("failed to write backup file %q: %w", outPath, err) | ||
| } |
Comment on lines
+115
to
+118
| outPath := filepath.Join(outDir, secret.Name+".json") | ||
| if err := os.WriteFile(outPath, data, 0o644); err != nil { | ||
| return fmt.Errorf("failed to write Terraform state file %q: %w", outPath, err) | ||
| } |
Comment on lines
+133
to
+138
| if err := gitExecIn(ctx, w.Path, "add", "-A"); err != nil { | ||
| return fmt.Errorf("failed to stage state files: %w", err) | ||
| } | ||
| if err := gitExecIn(ctx, w.Path, "commit", "-m", message, "--allow-empty"); err != nil { | ||
| return fmt.Errorf("failed to commit state: %w", err) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repo Radius: externalize control-plane and Terraform state (
rad startup/rad shutdown)Description
Implements Investment 2 of the Repo Radius feature spec — externalization of the Radius data
store, scoped to the state-storage aspects only (control-plane PostgreSQL state and Terraform
recipe state). It adds two kind-agnostic commands,
rad startupandrad shutdown, that back upand restore all durable Radius state across an ephemeral control plane, plus the Helm and
Terraform-state plumbing they depend on.
Design note:
eng/design-notes/2026-06-repo-radius-state-storage.md.There is no dedicated workspace kind. The commands operate on the current workspace's
Kubernetes context like any other command and do not create or delete clusters or install Radius —
cluster lifecycle is the caller's responsibility.
What's included
The change is organized as three logical commits:
PostgreSQL enablement (closes the chart gaps for
database.enabled=true)postgresqlprovider and inject
POSTGRES_PASSWORDwhendatabase.enabled=true(previously hardcoded toapiserver).tables before the servers start.
POSTGRES_DBsecret value and pins the postgres image to16-alpine.databaseProviderURL env-var substitution infactory.go.Terraform recipe state backup/restore (
pkg/cli/tfstate)tfstate=true), not in PostgreSQL, sothat state is lost on teardown. This package exports and restores those Secrets, including the
chunked
tfstate-{workspace}-{suffix}-{index}Secrets for large state.rad startup/rad shutdown+ end-to-end lifecycle testpkg/cli/pgbackup: control-plane PostgreSQL backup/restore viakubectl exec pg_dump/psql.pkg/cli/gitstate: persists the state directory to aradius-stategit orphan branch in anisolated worktree; the backup push fails loudly when a remote is configured (a failed push
would otherwise be silent data loss) and tolerates the no-remote local/test case.
rad shutdownbacks up both stores then commits and pushes;rad startupwaits for thedatabase then restores both stores.
Testing
Unit tests cover the chart rendering, the Terraform-state round-trip (fake Kubernetes client), the
git orphan-branch worktree behaviour (real temporary repos), and the command runners
(hand-written fakes). All pass, along with the existing 85 Helm chart tests.
End-to-end test dependency (please read)
The end-to-end lifecycle test lives at
test/functional-portable/statestoreand exercises thefull path that this work exists to protect: install → deploy a Terraform-backed resource →
rad shutdown→ tear down → reinstall →rad startup→ deploy an update to the same resource.The update is the path that fails when Terraform state is lost.
This test depends on the separate Repo Radius workflow code (in flight) that creates the
ephemeral cluster, installs Radius, and runs the deploy. Because
rad startup/rad shutdownare intentionally kind-agnostic and do not manage cluster lifecycle, the test needs that workflow
to stand up the environment. Until it lands, the test drives the install/uninstall itself and is
gated behind the
RADIUS_STATE_E2Eenvironment variable, so it does not run in the normalfunctional suite. Once the shared cluster-create + deploy workflow is merged, the test's
install/uninstall helpers should be re-pointed at that code rather than duplicating it.
Related
Type of change
This pull request adds new features (state externalization commands) for Radius.