diff --git a/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md b/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md new file mode 100644 index 00000000000..b9fab04fc89 --- /dev/null +++ b/eng/design-notes/extensibility/2026-05-unified-bicep-extension-publishing.md @@ -0,0 +1,226 @@ +# Unified Bicep Extension Publishing for Default Resource Types + +* **Author**: Karishma Chawla (@kachawla) + +## Overview + +Today, Bicep extensions for resource types from `resource-types-contrib` are published as separate per-namespace extensions (`radiusCompute`, `radiusData`, `radiusSecurity`), each at its own OCI registry path. This requires users to configure multiple extension entries in `bicepconfig.json` and authors to use different `extension` directives depending on the namespace. Meanwhile, the core Radius extension (`radius`) already bundles multiple namespaces (`Applications.Core`, `Applications.Dapr`, `Applications.Datastores`, `Applications.Messaging`, `Radius.Core`) in a single published artifact. + +This design proposes consolidating the contrib-sourced Bicep extensions into fewer (ideally one) published artifacts, and automating the publishing process so it stays in sync with the default resource type registration introduced in the [default registration design](2026-04-automated-default-resource-type-registration.md). + +## Terms and definitions + +| Term | Definition | +|---|---| +| **Bicep extension** | A published OCI artifact containing `index.json` and `types.json` files that enable Bicep's type system to provide autocompletion, validation, and compilation for resource types. | +| **`index.json`** | The entry point of a Bicep extension. Maps resource type names to their type definitions in `types.json` files. Supports multiple namespaces in a single file. | +| **`types.json`** | Contains the Bicep type definitions (ObjectType, ResourceType, StringType, etc.) for resource types in a namespace. | +| **`rad bicep publish-extension`** | CLI command that takes a YAML manifest, generates Bicep types via the Go-based `bicep-tools` converter, and publishes to an OCI registry. | +| **`bicep publish-extension`** | The Bicep CLI command that publishes a pre-generated `index.json` file to an OCI registry. Used by both the TypeScript and Go pipelines. | + +## Goals + +1. **Reduce the number of Bicep extension entries** in `bicepconfig.json` for resource types from `resource-types-contrib`. +2. **Automate Bicep extension publishing** so it stays in sync with the default resource type registration list (`deploy/manifest/defaults.yaml` in the Radius repo). +3. **Publish both versioned and latest** extensions to support release consistency and edge development. + +## Non goals + +- Changing the Bicep extension format or the `bicep-types` schema. +- Modifying how users write `extension` directives in Bicep files (beyond changing extension names). + +## Design + +### Current state + +```json +// bicepconfig.json +{ + "extensions": { + "radius": "br:biceptypes.azurecr.io/radius:latest", + "radiusCompute": "br:biceptypes.azurecr.io/radiuscompute:latest", + "radiusData": "br:biceptypes.azurecr.io/radiusdata:latest", + "radiusSecurity": "br:biceptypes.azurecr.io/radiussecurity:latest", + "aws": "br:biceptypes.azurecr.io/aws:latest" + } +} +``` + +The `radius` extension is generated by a **TypeScript autorest pipeline** from Swagger/OpenAPI specs and bundles 5 namespaces. The contrib extensions (`radiusCompute`, `radiusData`, `radiusSecurity`) are each published separately using `rad bicep publish-extension` from individual YAML manifests via the **Go-based `bicep-tools` converter**. + +Both pipelines produce the same output format (`types.json` + `index.json` conforming to the `bicep-types` schema). The `index.json` format supports multiple namespaces in a single file, as demonstrated by the existing `radius` extension. + +### Option 1: Separate `radiusResources` extension (merging contrib namespaces only) + +Merge `Radius.Compute`, `Radius.Data`, and `Radius.Security` and any future default namespaces into a single new extension published at `br:biceptypes.azurecr.io/radiusresources:latest`. + +**Resulting `bicepconfig.json`:** +```json +{ + "extensions": { + "radius": "br:biceptypes.azurecr.io/radius:latest", + "radiusResources": "br:biceptypes.azurecr.io/radiusresources:latest", + "aws": "br:biceptypes.azurecr.io/aws:latest" + } +} +``` + +**How it works:** + +1. Add a `MergeConversionResults()` function to `bicep-tools/pkg/converter/` that takes multiple `ConversionResult`s and produces a single `index.json` referencing per-namespace `types.json` subdirectories. +2. Update `generator.RunGenerate()` to accept multiple manifest files, call `Convert()` per manifest, then merge the results. +3. Update `rad bicep publish-extension` to accept multiple `--from-file` flags, or add a new `--defaults-file` flag that reads `defaults.yaml` and processes all listed manifests. +4. Publish the merged output as one extension. + +**Implementation in `bicep-tools`:** + +```go +// converter/merge.go +func MergeConversionResults(results map[string]*ConversionResult) (*MergedResult, error) { + // Build unified index.json with entries from all namespaces + // Each namespace's types.json goes in a subdirectory: //types.json + // Returns merged index.json + directory of types.json files +} +``` + +```go +// generator/generate.go +func RunGenerateMultiple(manifestFiles []string, outputDir string) error { + results := map[string]*ConversionResult{} + for _, file := range manifestFiles { + result, err := generator.GenerateFromFile(file) + // ... + results[namespace] = result + } + return writeMergedOutput(results, outputDir) +} +``` + +**Advantages:** +- Reduces three entries in `bicepconfig.json` to one +- No changes to the existing TypeScript pipeline or `radius` extension +- Independent release cadence from core Radius types +- Simpler to implement (Go-only changes) + +**Disadvantages:** +- Users need two Radius-related extensions (`radius` and `radiusResources`). Not that bad as we can automatically generate the `radiusResources` entry in `bicepconfig.json` and document it as the extension for all non-core resource types. + +### Option 2: Merge into existing `radius` extension + +Add `Radius.Compute`, `Radius.Data`, and `Radius.Security` to the existing `radius` extension alongside `Applications.Core`, `Applications.Dapr`, etc. + +**Resulting `bicepconfig.json`:** +```json +{ + "extensions": { + "radius": "br:biceptypes.azurecr.io/radius:latest", + "aws": "br:biceptypes.azurecr.io/aws:latest" + } +} +``` + +**How it works:** + +Both pipelines produce `types.json` files in subdirectories. The merge step places them in the same directory tree and rebuilds the unified `index.json`. + +1. Run the TypeScript autorest pipeline as today, producing types in `hack/bicep-types-radius/generated/`. +2. Run the Go converter for each contrib manifest, outputting `types.json` files into the same `hack/bicep-types-radius/generated/` directory tree under namespace-specific subdirectories (e.g., `radius/radius.compute/2025-08-01-preview/types.json`). +3. Rebuild `index.json` to include all `types.json` files. The existing TypeScript `buildTypeIndex()` function already walks directories recursively, so it would pick up the new files automatically. +4. Publish the unified output as the `radius` extension. + +**Build flow:** +```make +generate-bicep-types: generate-node-installed generate-pnpm-installed + # Step 1: Generate types from Swagger/TypeSpec (existing) + pnpm -C hack/bicep-types-radius/src/autorest.bicep install && pnpm -C hack/bicep-types-radius/src/autorest.bicep run build + pnpm -C hack/bicep-types-radius/src/generator run generate --specs-dir ../../../../swagger + + # Step 2: Generate types from contrib manifests (new) + # Manifests are already copied into deploy/manifest/built-in-providers/ by `make update-resource-types` + # (see the default registration design). The generation step reads them directly from disk. + @for entry in $$(yq '.defaultRegistration[]' deploy/manifest/defaults.yaml); do \ + NAMESPACE=$$(echo $$entry | cut -d/ -f1 | sed 's/Radius\.//'); \ + TYPE=$$(echo $$entry | cut -d/ -f2); \ + FILEPATH="deploy/manifest/built-in-providers/$$TYPE.yaml"; \ + go run ./bicep-tools/cmd/manifest-to-bicep \ + --input "$$FILEPATH" \ + --output-dir hack/bicep-types-radius/generated/radius/$$(echo $$NAMESPACE | tr '[:upper:]' '[:lower:]')/2025-08-01-preview/; \ + done + + # Step 3: Rebuild unified index.json (existing function picks up new files) + pnpm -C hack/bicep-types-radius/src/generator run rebuild-index +``` + +**Changes needed:** +- Add `--output-dir` flag to `bicep-tools/cmd/manifest-to-bicep` (or a new command) that outputs `types.json` without publishing, so the files can be placed in the right directory. +- Add a `rebuild-index` script (or modify the existing generator) to walk the full `generated/` directory and build `index.json`. The existing `buildTypeIndex()` may already do this. +- Update the `generate-bicep-types` Makefile target to include the contrib step. +- Remove `radiusCompute`, `radiusData`, `radiusSecurity` from `bicepconfig.json`. + +**Advantages:** +- Single extension for all Radius types; simplest user experience +- No new extension names to communicate to users +- Users upgrading just get new types automatically + +**Disadvantages:** +- Coupling contrib type publishing with the core Radius release pipeline +- Requires the TypeScript `buildTypeIndex()` to run after the Go converter (ordering dependency) +- If the contrib types have issues, it could block the core `radius` extension from being published +- More complex build pipeline with two generation steps feeding into one output + +### Proposed option + +**Option 1: Separate `radiusResources` extension.** This is simpler to implement, keeps the build pipelines independent, and makes automation for both versioned and latest publishing straightforward. The extra `extension radiusResources` directive is a minor user-facing cost compared to the build and operational complexity of merging two generation pipelines. Option 2 (merging into `radius`) can be a future enhancement if the user experience benefit justifies aligning the pipelines. + +### Decision + +**Option 2: Merge into the existing `radius` extension.** After review, we decided to consolidate all Radius-authored types (core + contrib) into the single `radius` extension rather than introducing a new `radiusResources` extension. + +Rationale: +- **Best end-user experience.** Users have one `extension radius` directive in their Bicep files for everything Radius ships. No need to teach users which extension owns which namespace. +- **No migration churn.** Existing users on the `radius` extension automatically pick up the new namespaces on upgrade. No update required to start using contrib types. + +The disadvantages noted under Option 2 (pipeline coupling, ordering dependency, build complexity) are accepted as engineering tradeoffs to optimize for the end-user experience. They are mitigated by: +- Keeping the contrib generation step idempotent and isolated so a failure surfaces clearly without partially corrupting the unified output. +- Failing the build fast if the contrib step errors, rather than publishing a partial extension. +- Documenting the ordering requirement in the Makefile and CI workflow. + +The remainder of this document focuses on Option 2 details. Option 1 is retained above as a documented alternative. + +### Automating extension publishing + +Extension publishing should be automated in two places: + +1. **Radius release pipeline (versioned).** During release, generate and publish the extension with the version tag (e.g., `br:biceptypes.azurecr.io/radiusresources:v0.58.0` for Option 1, or `br:biceptypes.azurecr.io/radius:v0.58.0` for Option 2). This ensures the published extension matches the schemas embedded in the binary. + +2. **`resource-types-contrib` CI (latest).** On merge to `main`, publish the latest extension so developers authoring Bicep against edge always have the latest type definitions. + +**Option 1 automation:** `resource-types-contrib` CI reads `defaults.yaml`, runs the merged type generation, and publishes to `br:biceptypes.azurecr.io/radiusresources:latest`. Since the `radiusResources` extension is Go-only and independent of the TypeScript autorest pipeline, this runs entirely within `resource-types-contrib` CI without any cross-pipeline coordination. + +**Option 2 automation:** Publishing the unified `radius` extension requires running both the TypeScript autorest pipeline (for `Applications.*` and `Radius.Core`) and the Go converter (for contrib types) before merging into a single `index.json`. Because both pipelines live in the Radius repo and the contrib manifest copies are also committed there (see the [default registration design](2026-04-automated-default-resource-type-registration.md) decision to use Alternative approach 1), publishing is fully self-contained in the Radius repo: + +- **Versioned tag (`v0.58.0`, etc.):** Published by the Radius release pipeline. Uses the manifest copies committed to `deploy/manifest/built-in-providers/`, which were last refreshed via `make update-resource-types` against the `resource-types-contrib` version pinned in `go.mod`. +- **`latest` tag:** Published by a Radius CI workflow on every merge to `main`. The same workflow is also triggered via `repository_dispatch` from `resource-types-contrib` CI on merges to its `main`. Important nuance: the dispatch alone does not refresh the manifest copies in Radius (those only update via the explicit `make update-resource-types` PR), so dispatching `latest` after a contrib merge would publish stale types. Instead, contrib's dispatch should open (or update) a Radius PR that runs `make update-resource-types`; the `latest` republish happens when that PR merges. This requires a cross-repo PAT scoped to triggering the dispatch event and opening PRs. + +A staging-location approach (where `resource-types-contrib` CI publishes intermediate artifacts that the Radius pipeline later consumes) was considered but rejected: it adds an extra artifact and storage location without solving the core requirement that the unified extension must be assembled in one place by the pipeline that owns both generators. + +## Error Handling + +| Scenario | Behavior | +|---|---| +| Contrib manifest fails type conversion | `Convert()` returns error. Extension is not published. Build fails with the specific manifest identified. | +| Merged index.json is invalid | `bicep publish-extension` fails. Extension is not published. | +| OCI registry is unreachable | Publish step fails. Retried by CI. | + +## Compatibility + +- **Breaking change for existing users:** Users currently using `extension radiusCompute` / `extension radiusData` / `extension radiusSecurity` in their Bicep files will need to switch to `extension radius`. Since these types were recently made available as defaults, the user base is likely small. This will be communicated in release notes. +- **`bicepconfig.json` update:** Users will need to remove the `radiusCompute`, `radiusData`, and `radiusSecurity` entries from `bicepconfig.json`. The `radius` entry stays unchanged. A migration guide and an updated `rad init` template will handle this for new and existing projects. +- **No change for users already on the `radius` extension** for core types: contrib types become available automatically on upgrade. + +## Development plan + +1. **PR 1 (radius)**: Add `--output-dir` flag to `bicep-tools/cmd/manifest-to-bicep` for generating `types.json` without publishing. +2. **PR 2 (radius)**: Update `generate-bicep-types` Makefile target to include contrib manifest processing (driven by `deploy/manifest/defaults.yaml` and the manifest copies in `deploy/manifest/built-in-providers/`, both maintained by `make update-resource-types`; see the default registration design). Update or add `rebuild-index` step so `index.json` covers both TypeScript- and Go-generated types. Remove `radiusCompute`, `radiusData`, `radiusSecurity` from `bicepconfig.json` and from the install scripts (`deploy/install.sh`, `deploy/install.ps1`). +3. **PR 3 (radius)**: Update the release pipeline to publish the unified `radius` extension (versioned tag) and add a CI workflow that publishes the `latest` tag on merges to `main`. Wire up `repository_dispatch` from `resource-types-contrib` so contrib merges also trigger a `latest` republish. +4. **PR 4 (resource-types-contrib)**: Add a workflow that fires `repository_dispatch` to the Radius repo on merges to `main`. The dispatch opens or updates a Radius PR running `make update-resource-types`; merging that PR refreshes the committed manifest copies and triggers the `latest` republish via PR 3's workflow. \ No newline at end of file