From bed122aa133d914f8f7c3055ab561eb9c0e16d60 Mon Sep 17 00:00:00 2001 From: Karishma Chawla Date: Thu, 14 May 2026 03:25:37 -0700 Subject: [PATCH 1/2] Design: Unified Bicep extension publishing for default resource types Signed-off-by: Karishma Chawla --- ...6-05-unified-bicep-extension-publishing.md | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md diff --git a/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md b/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md new file mode 100644 index 00000000000..0b8d22f55e6 --- /dev/null +++ b/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md @@ -0,0 +1,216 @@ +# Unified Bicep Extension Publishing for Default Resource Types + +* **Author**: Karishma Chawla (@kachawla) + +## Overview + +Today, Bicep extensions for resource types from `resource-types-contrib` are published as separate per-namespace extensions (`radiusCompute`, `radiusData`, `radiusSecurity`), each at its own OCI registry path. This requires users to configure multiple extension entries in `bicepconfig.json` and authors to use different `extension` directives depending on the namespace. Meanwhile, the core Radius extension (`radius`) already bundles multiple namespaces (`Applications.Core`, `Applications.Dapr`, `Applications.Datastores`, `Applications.Messaging`, `Radius.Core`) in a single published artifact. + +This design proposes consolidating the contrib-sourced Bicep extensions into fewer (ideally one) published artifacts, and automating the publishing process so it stays in sync with the default resource type registration introduced in the [default registration design](2026-04-automated-default-resource-type-registration.md). + +## Terms and definitions + +| Term | Definition | +|---|---| +| **Bicep extension** | A published OCI artifact containing `index.json` and `types.json` files that enable Bicep's type system to provide autocompletion, validation, and compilation for resource types. | +| **`index.json`** | The entry point of a Bicep extension. Maps resource type names to their type definitions in `types.json` files. Supports multiple namespaces in a single file. | +| **`types.json`** | Contains the Bicep type definitions (ObjectType, ResourceType, StringType, etc.) for resource types in a namespace. | +| **`rad bicep publish-extension`** | CLI command that takes a YAML manifest, generates Bicep types via the Go-based `bicep-tools` converter, and publishes to an OCI registry. | +| **`bicep publish-extension`** | The Bicep CLI command that publishes a pre-generated `index.json` file to an OCI registry. Used by both the TypeScript and Go pipelines. | + +## Goals + +1. **Reduce the number of Bicep extension entries** in `bicepconfig.json` for resource types from `resource-types-contrib`. +2. **Automate Bicep extension publishing** so it stays in sync with the default resource type registration list (`defaults.yaml`). +3. **Publish both versioned and latest** extensions to support release consistency and edge development. + +## Non goals + +- Changing the Bicep extension format or the `bicep-types` schema. +- Modifying how users write `extension` directives in Bicep files (beyond changing extension names). + +## Design + +### Current state + +```json +// bicepconfig.json +{ + "extensions": { + "radius": "br:biceptypes.azurecr.io/radius:latest", + "radiusCompute": "br:biceptypes.azurecr.io/radiuscompute:latest", + "radiusData": "br:biceptypes.azurecr.io/radiusdata:latest", + "radiusSecurity": "br:biceptypes.azurecr.io/radiussecurity:latest", + "aws": "br:biceptypes.azurecr.io/aws:latest" + } +} +``` + +The `radius` extension is generated by a **TypeScript autorest pipeline** from Swagger/OpenAPI specs and bundles 5 namespaces. The contrib extensions (`radiusCompute`, `radiusData`, `radiusSecurity`) are each published separately using `rad bicep publish-extension` from individual YAML manifests via the **Go-based `bicep-tools` converter**. + +Both pipelines produce the same output format (`types.json` + `index.json` conforming to the `bicep-types` schema). The `index.json` format supports multiple namespaces in a single file, as demonstrated by the existing `radius` extension. + +### Option 1: Separate `radiusResources` extension (merging contrib namespaces only) + +Merge `Radius.Compute`, `Radius.Data`, and `Radius.Security` and any future default namespaces into a single new extension published at `br:biceptypes.azurecr.io/radiusresources:latest`. + +**Resulting `bicepconfig.json`:** +```json +{ + "extensions": { + "radius": "br:biceptypes.azurecr.io/radius:latest", + "radiusResources": "br:biceptypes.azurecr.io/radiusresources:latest", + "aws": "br:biceptypes.azurecr.io/aws:latest" + } +} +``` + +**How it works:** + +1. Add a `MergeConversionResults()` function to `bicep-tools/pkg/converter/` that takes multiple `ConversionResult`s and produces a single `index.json` referencing per-namespace `types.json` subdirectories. +2. Update `generator.RunGenerate()` to accept multiple manifest files, call `Convert()` per manifest, then merge the results. +3. Update `rad bicep publish-extension` to accept multiple `--from-file` flags, or add a new `--defaults-file` flag that reads `defaults.yaml` and processes all listed manifests. +4. Publish the merged output as one extension. + +**Implementation in `bicep-tools`:** + +```go +// converter/merge.go +func MergeConversionResults(results map[string]*ConversionResult) (*MergedResult, error) { + // Build unified index.json with entries from all namespaces + // Each namespace's types.json goes in a subdirectory: //types.json + // Returns merged index.json + directory of types.json files +} +``` + +```go +// generator/generate.go +func RunGenerateMultiple(manifestFiles []string, outputDir string) error { + results := map[string]*ConversionResult{} + for _, file := range manifestFiles { + result, err := generator.GenerateFromFile(file) + // ... + results[namespace] = result + } + return writeMergedOutput(results, outputDir) +} +``` + +**Advantages:** +- Reduces three entries in `bicepconfig.json` to one +- No changes to the existing TypeScript pipeline or `radius` extension +- Independent release cadence from core Radius types +- Simpler to implement (Go-only changes) + +**Disadvantages:** +- Users need two Radius-related extensions (`radius` and `radiusResources`). Not that bad as we can automatically generate the `radiusResources` entry in `bicepconfig.json` and document it as the extension for all non-core resource types. + +### Option 2: Merge into existing `radius` extension + +Add `Radius.Compute`, `Radius.Data`, and `Radius.Security` to the existing `radius` extension alongside `Applications.Core`, `Applications.Dapr`, etc. + +**Resulting `bicepconfig.json`:** +```json +{ + "extensions": { + "radius": "br:biceptypes.azurecr.io/radius:latest", + "aws": "br:biceptypes.azurecr.io/aws:latest" + } +} +``` + +**How it works:** + +Both pipelines produce `types.json` files in subdirectories. The merge step places them in the same directory tree and rebuilds the unified `index.json`. + +1. Run the TypeScript autorest pipeline as today, producing types in `hack/bicep-types-radius/generated/`. +2. Run the Go converter for each contrib manifest, outputting `types.json` files into the same `hack/bicep-types-radius/generated/` directory tree under namespace-specific subdirectories (e.g., `radius/radius.compute/2025-08-01-preview/types.json`). +3. Rebuild `index.json` to include all `types.json` files. The existing TypeScript `buildTypeIndex()` function already walks directories recursively, so it would pick up the new files automatically. +4. Publish the unified output as the `radius` extension. + +**Build flow:** +```make +generate-bicep-types: generate-node-installed generate-pnpm-installed + # Step 1: Generate types from Swagger/TypeSpec (existing) + pnpm -C hack/bicep-types-radius/src/autorest.bicep install && pnpm -C hack/bicep-types-radius/src/autorest.bicep run build + pnpm -C hack/bicep-types-radius/src/generator run generate --specs-dir ../../../../swagger + + # Step 2: Generate types from contrib manifests (new) + $(eval MODULE_DIR := $(shell go mod download -json github.com/radius-project/resource-types-contrib | jq -r '.Dir')) + @for entry in $$(yq '.defaultRegistration[]' $(MODULE_DIR)/defaults.yaml); do \ + NAMESPACE=$$(echo $$entry | cut -d/ -f1 | sed 's/Radius\.//'); \ + TYPE=$$(echo $$entry | cut -d/ -f2); \ + FILEPATH=$$(echo "$$NAMESPACE/$$TYPE/$$TYPE.yaml"); \ + go run ./bicep-tools/cmd/manifest-to-bicep \ + --input "$(MODULE_DIR)/$$FILEPATH" \ + --output-dir hack/bicep-types-radius/generated/radius/$$(echo $$NAMESPACE | tr '[:upper:]' '[:lower:]')/2025-08-01-preview/; \ + done + + # Step 3: Rebuild unified index.json (existing function picks up new files) + pnpm -C hack/bicep-types-radius/src/generator run rebuild-index +``` + +**Changes needed:** +- Add `--output-dir` flag to `bicep-tools/cmd/manifest-to-bicep` (or a new command) that outputs `types.json` without publishing, so the files can be placed in the right directory. +- Add a `rebuild-index` script (or modify the existing generator) to walk the full `generated/` directory and build `index.json`. The existing `buildTypeIndex()` may already do this. +- Update the `generate-bicep-types` Makefile target to include the contrib step. +- Remove `radiusCompute`, `radiusData`, `radiusSecurity` from `bicepconfig.json`. + +**Advantages:** +- Single extension for all Radius types; simplest user experience +- No new extension names to communicate to users +- Users upgrading just get new types automatically + +**Disadvantages:** +- Coupling contrib type publishing with the core Radius release pipeline +- Requires the TypeScript `buildTypeIndex()` to run after the Go converter (ordering dependency) +- If the contrib types have issues, it could block the core `radius` extension from being published +- More complex build pipeline with two generation steps feeding into one output + +### Proposed option + +**Option 1: Separate `radiusResources` extension.** This is simpler to implement, keeps the build pipelines independent, and makes automation for both versioned and latest publishing straightforward. The extra `extension radiusResources` directive is a minor user-facing cost compared to the build and operational complexity of merging two generation pipelines. Option 2 (merging into `radius`) can be a future enhancement if the user experience benefit justifies aligning the pipelines. + +### Decision + +**Option 2: Merge into the existing `radius` extension.** After review, we decided to consolidate all Radius-authored types (core + contrib) into the single `radius` extension rather than introducing a new `radiusResources` extension. + +Rationale: +- **Better end-user experience.** Users have one `extension radius` directive in their Bicep files for everything Radius ships. No need to teach users which extension owns which namespace. +- **No migration churn.** Existing users on the `radius` extension automatically pick up the new namespaces on upgrade. No update required to start using contrib types. + +### Automating extension publishing + +Extension publishing should be automated in two places: + +1. **Radius release pipeline (versioned).** During release, generate and publish the extension with the version tag (e.g., `br:biceptypes.azurecr.io/radiusresources:v0.58.0` for Option 1, or `br:biceptypes.azurecr.io/radius:v0.58.0` for Option 2). This ensures the published extension matches the schemas embedded in the binary. + +2. **`resource-types-contrib` CI (latest).** On merge to `main`, publish the latest extension so developers authoring Bicep against edge always have the latest type definitions. + +**Option 1 automation:** `resource-types-contrib` CI reads `defaults.yaml`, runs the merged type generation, and publishes to `br:biceptypes.azurecr.io/radiusresources:latest`. Since the `radiusResources` extension is Go-only and independent of the TypeScript autorest pipeline, this runs entirely within `resource-types-contrib` CI without any cross-pipeline coordination. + +**Option 2 automation:** Publishing the unified `radius` extension requires running both the TypeScript autorest pipeline (for `Applications.*` and `Radius.Core`) and the Go converter (for contrib types) before merging into a single `index.json`. Because both pipelines live in the Radius repo, the Radius repo owns publishing for both `latest` and versioned tags: + +- **Versioned tag (`v0.58.0`, etc.):** Published by the Radius release pipeline. Pins to the `resource-types-contrib` commit referenced in `go.mod` for that release. +- **`latest` tag:** Published by a Radius CI workflow on every merge to `main`. The same workflow is also triggered via `repository_dispatch` from `resource-types-contrib` CI on merges to its `main`, so contrib changes flow into `latest` without waiting for the next Radius commit. This requires a cross-repo PAT scoped to triggering the dispatch event. + +## Error Handling + +| Scenario | Behavior | +|---|---| +| Contrib manifest fails type conversion | `Convert()` returns error. Extension is not published. Build fails with the specific manifest identified. | +| Merged index.json is invalid | `bicep publish-extension` fails. Extension is not published. | +| OCI registry is unreachable | Publish step fails. Retried by CI. | + +## Compatibility + +- **Breaking change for existing users:** Users currently using `extension radiusCompute` / `extension radiusData` / `extension radiusSecurity` in their Bicep files will need to switch to `extension radius`. Since these types were recently made available as defaults, the user base is likely small. This will be communicated in release notes. +- **`bicepconfig.json` update:** Users will need to remove the `radiusCompute`, `radiusData`, and `radiusSecurity` entries from `bicepconfig.json`. The `radius` entry stays unchanged. A migration guide and an updated `rad init` template will handle this for new and existing projects. +- **No change for users already on the `radius` extension** for core types: contrib types become available automatically on upgrade. + +## Development plan + +1. **PR 1 (radius)**: Add `--output-dir` flag to `bicep-tools/cmd/manifest-to-bicep` for generating `types.json` without publishing. +2. **PR 2 (radius)**: Update `generate-bicep-types` Makefile target to include contrib manifest processing (driven by `defaults.yaml` from the `resource-types-contrib` Go module). Update or add `rebuild-index` step so `index.json` covers both TypeScript- and Go-generated types. Remove `radiusCompute`, `radiusData`, `radiusSecurity` from `bicepconfig.json` and from the install scripts (`deploy/install.sh`, `deploy/install.ps1`). +3. **PR 3 (radius)**: Update the release pipeline to publish the unified `radius` extension (versioned tag) and add a CI workflow that publishes the `latest` tag on merges to `main`. Wire up `repository_dispatch` from `resource-types-contrib` so contrib merges also trigger a `latest` republish. +4. **PR 4 (resource-types-contrib)**: Add a workflow that fires `repository_dispatch` to the Radius repo on merges to `main`. \ No newline at end of file From c56a079f6688b4292b38a41ca5a282c6ae4dd4a5 Mon Sep 17 00:00:00 2001 From: Karishma Chawla Date: Thu, 14 May 2026 23:37:07 -0700 Subject: [PATCH 2/2] Updates to reflect updates in automatic resource types sync design Signed-off-by: Karishma Chawla --- ...6-05-unified-bicep-extension-publishing.md | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md b/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md index 0b8d22f55e6..b9fab04fc89 100644 --- a/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md +++ b/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md @@ -21,7 +21,7 @@ This design proposes consolidating the contrib-sourced Bicep extensions into few ## Goals 1. **Reduce the number of Bicep extension entries** in `bicepconfig.json` for resource types from `resource-types-contrib`. -2. **Automate Bicep extension publishing** so it stays in sync with the default resource type registration list (`defaults.yaml`). +2. **Automate Bicep extension publishing** so it stays in sync with the default resource type registration list (`deploy/manifest/defaults.yaml` in the Radius repo). 3. **Publish both versioned and latest** extensions to support release consistency and edge development. ## Non goals @@ -136,13 +136,14 @@ generate-bicep-types: generate-node-installed generate-pnpm-installed pnpm -C hack/bicep-types-radius/src/generator run generate --specs-dir ../../../../swagger # Step 2: Generate types from contrib manifests (new) - $(eval MODULE_DIR := $(shell go mod download -json github.com/radius-project/resource-types-contrib | jq -r '.Dir')) - @for entry in $$(yq '.defaultRegistration[]' $(MODULE_DIR)/defaults.yaml); do \ + # Manifests are already copied into deploy/manifest/built-in-providers/ by `make update-resource-types` + # (see the default registration design). The generation step reads them directly from disk. + @for entry in $$(yq '.defaultRegistration[]' deploy/manifest/defaults.yaml); do \ NAMESPACE=$$(echo $$entry | cut -d/ -f1 | sed 's/Radius\.//'); \ TYPE=$$(echo $$entry | cut -d/ -f2); \ - FILEPATH=$$(echo "$$NAMESPACE/$$TYPE/$$TYPE.yaml"); \ + FILEPATH="deploy/manifest/built-in-providers/$$TYPE.yaml"; \ go run ./bicep-tools/cmd/manifest-to-bicep \ - --input "$(MODULE_DIR)/$$FILEPATH" \ + --input "$$FILEPATH" \ --output-dir hack/bicep-types-radius/generated/radius/$$(echo $$NAMESPACE | tr '[:upper:]' '[:lower:]')/2025-08-01-preview/; \ done @@ -176,9 +177,16 @@ generate-bicep-types: generate-node-installed generate-pnpm-installed **Option 2: Merge into the existing `radius` extension.** After review, we decided to consolidate all Radius-authored types (core + contrib) into the single `radius` extension rather than introducing a new `radiusResources` extension. Rationale: -- **Better end-user experience.** Users have one `extension radius` directive in their Bicep files for everything Radius ships. No need to teach users which extension owns which namespace. +- **Best end-user experience.** Users have one `extension radius` directive in their Bicep files for everything Radius ships. No need to teach users which extension owns which namespace. - **No migration churn.** Existing users on the `radius` extension automatically pick up the new namespaces on upgrade. No update required to start using contrib types. +The disadvantages noted under Option 2 (pipeline coupling, ordering dependency, build complexity) are accepted as engineering tradeoffs to optimize for the end-user experience. They are mitigated by: +- Keeping the contrib generation step idempotent and isolated so a failure surfaces clearly without partially corrupting the unified output. +- Failing the build fast if the contrib step errors, rather than publishing a partial extension. +- Documenting the ordering requirement in the Makefile and CI workflow. + +The remainder of this document focuses on Option 2 details. Option 1 is retained above as a documented alternative. + ### Automating extension publishing Extension publishing should be automated in two places: @@ -189,10 +197,12 @@ Extension publishing should be automated in two places: **Option 1 automation:** `resource-types-contrib` CI reads `defaults.yaml`, runs the merged type generation, and publishes to `br:biceptypes.azurecr.io/radiusresources:latest`. Since the `radiusResources` extension is Go-only and independent of the TypeScript autorest pipeline, this runs entirely within `resource-types-contrib` CI without any cross-pipeline coordination. -**Option 2 automation:** Publishing the unified `radius` extension requires running both the TypeScript autorest pipeline (for `Applications.*` and `Radius.Core`) and the Go converter (for contrib types) before merging into a single `index.json`. Because both pipelines live in the Radius repo, the Radius repo owns publishing for both `latest` and versioned tags: +**Option 2 automation:** Publishing the unified `radius` extension requires running both the TypeScript autorest pipeline (for `Applications.*` and `Radius.Core`) and the Go converter (for contrib types) before merging into a single `index.json`. Because both pipelines live in the Radius repo and the contrib manifest copies are also committed there (see the [default registration design](2026-04-automated-default-resource-type-registration.md) decision to use Alternative approach 1), publishing is fully self-contained in the Radius repo: + +- **Versioned tag (`v0.58.0`, etc.):** Published by the Radius release pipeline. Uses the manifest copies committed to `deploy/manifest/built-in-providers/`, which were last refreshed via `make update-resource-types` against the `resource-types-contrib` version pinned in `go.mod`. +- **`latest` tag:** Published by a Radius CI workflow on every merge to `main`. The same workflow is also triggered via `repository_dispatch` from `resource-types-contrib` CI on merges to its `main`. Important nuance: the dispatch alone does not refresh the manifest copies in Radius (those only update via the explicit `make update-resource-types` PR), so dispatching `latest` after a contrib merge would publish stale types. Instead, contrib's dispatch should open (or update) a Radius PR that runs `make update-resource-types`; the `latest` republish happens when that PR merges. This requires a cross-repo PAT scoped to triggering the dispatch event and opening PRs. -- **Versioned tag (`v0.58.0`, etc.):** Published by the Radius release pipeline. Pins to the `resource-types-contrib` commit referenced in `go.mod` for that release. -- **`latest` tag:** Published by a Radius CI workflow on every merge to `main`. The same workflow is also triggered via `repository_dispatch` from `resource-types-contrib` CI on merges to its `main`, so contrib changes flow into `latest` without waiting for the next Radius commit. This requires a cross-repo PAT scoped to triggering the dispatch event. +A staging-location approach (where `resource-types-contrib` CI publishes intermediate artifacts that the Radius pipeline later consumes) was considered but rejected: it adds an extra artifact and storage location without solving the core requirement that the unified extension must be assembled in one place by the pipeline that owns both generators. ## Error Handling @@ -211,6 +221,6 @@ Extension publishing should be automated in two places: ## Development plan 1. **PR 1 (radius)**: Add `--output-dir` flag to `bicep-tools/cmd/manifest-to-bicep` for generating `types.json` without publishing. -2. **PR 2 (radius)**: Update `generate-bicep-types` Makefile target to include contrib manifest processing (driven by `defaults.yaml` from the `resource-types-contrib` Go module). Update or add `rebuild-index` step so `index.json` covers both TypeScript- and Go-generated types. Remove `radiusCompute`, `radiusData`, `radiusSecurity` from `bicepconfig.json` and from the install scripts (`deploy/install.sh`, `deploy/install.ps1`). +2. **PR 2 (radius)**: Update `generate-bicep-types` Makefile target to include contrib manifest processing (driven by `deploy/manifest/defaults.yaml` and the manifest copies in `deploy/manifest/built-in-providers/`, both maintained by `make update-resource-types`; see the default registration design). Update or add `rebuild-index` step so `index.json` covers both TypeScript- and Go-generated types. Remove `radiusCompute`, `radiusData`, `radiusSecurity` from `bicepconfig.json` and from the install scripts (`deploy/install.sh`, `deploy/install.ps1`). 3. **PR 3 (radius)**: Update the release pipeline to publish the unified `radius` extension (versioned tag) and add a CI workflow that publishes the `latest` tag on merges to `main`. Wire up `repository_dispatch` from `resource-types-contrib` so contrib merges also trigger a `latest` republish. -4. **PR 4 (resource-types-contrib)**: Add a workflow that fires `repository_dispatch` to the Radius repo on merges to `main`. \ No newline at end of file +4. **PR 4 (resource-types-contrib)**: Add a workflow that fires `repository_dispatch` to the Radius repo on merges to `main`. The dispatch opens or updates a Radius PR running `make update-resource-types`; merging that PR refreshes the committed manifest copies and triggers the `latest` republish via PR 3's workflow. \ No newline at end of file