diff --git a/.github/instructions/ado-pipeline.instructions.md b/.github/instructions/ado-pipeline.instructions.md new file mode 100644 index 00000000000..32b7448db54 --- /dev/null +++ b/.github/instructions/ado-pipeline.instructions.md @@ -0,0 +1,302 @@ +--- +applyTo: ".github/workflows/ado/*.yml,.github/workflows/ado/templates/*.yml,.github/workflows/scripts/**" +description: "Authoring and maintenance rules for Azure DevOps YAML pipelines under .github/workflows/ado/ (wrappers + raw stage templates under templates/) and their helper scripts under .github/workflows/scripts/ that run as GitHub PR checks or in the merge queue. Apply when creating or modifying any pipeline in that folder, or any script invoked by one — covers the wrapper/raw split, required OneBranch templates (Official vs NonOfficial), Workload Identity Federation service connections, Control Tower audience URIs, internal-only dependency sources (Go / Python / container images), Python-over-shell scripting, and security hardening." +--- + +# Azure DevOps Pipelines (PR check & merge queue) + +These instructions cover ADO YAML pipelines under `.github/workflows/ado/` that run as GitHub PR checks or in the merge queue, plus their helper scripts under `.github/workflows/scripts/`. Follow every MUST below — they encode internal Microsoft policy plus security hardening for this repo. + +Helper scripts invoked from these pipelines MUST live under `.github/workflows/scripts//` (one subdirectory per logical area / pipeline). Keep them self-contained and follow the same security hardening rules as the pipeline YAML itself. + +> If anything below conflicts with what the user is asking for, **stop and ask the user** rather than guessing — especially for the Official vs NonOfficial template choice. + +## Pipeline structure: wrapper + raw stages (MANDATORY) + +Every ADO pipeline in this repo is split into **two YAML files**: + +1. **Wrapper** — `.github/workflows/ado/.yml`. This is the file passed to ADO as the pipeline definition. It is the *only* place that knows about OneBranch: + - declares `resources.repositories` for `OneBranch.Pipelines/GovernedTemplates`, + - picks the OneBranch variant via `extends:` (`Official` vs `NonOfficial` — see next section), + - configures the OneBranch `parameters.featureFlags`, + - injects the raw stages template via `parameters.stages: - template: …@self` and supplies concrete values for its `parameters:`. + + The wrapper MUST NOT contain `stages:`, `jobs:`, or `steps:` directly. If you find yourself adding a `script:` block to a wrapper, stop — it belongs in the raw stages template. + +2. **Raw stages template** — `.github/workflows/ado/templates/-stages.yml`. This file declares `parameters:` + `stages:` and contains the actual jobs/steps. It MUST be OneBranch-agnostic: + - no `resources:`, no `extends:`, no `featureFlags:`, + - no string mentions of `OneBranch` or `GovernedTemplates`, + - any value coupled to the OneBranch contract (output dir, artifact base name, container image, pool type, service connection, variable group, timeout) is exposed as a **parameter with a neutral name** and bound by the wrapper. + + `ob_*` job-scope variables and `LinuxContainerImage` still appear here (ADO requires them at job scope), but they are set from `${{ parameters.* }}`, so the raw author never has to know which OneBranch convention they satisfy. + +File-pairing convention: a wrapper at `.github/workflows/ado/.yml` pairs with a raw stages template at `.github/workflows/ado/templates/-stages.yml`. **Multiple wrappers MAY share a single raw stages template** — that is in fact a primary motivation for the split: define several ADO pipelines (e.g. a DEV NonOfficial wrapper and a PROD Official wrapper, or per-environment variants) that all run the same stages/jobs/steps but differ in OneBranch variant, `featureFlags`, service connection, variable group, container image, etc. When wrappers share a raw template, name them so the relationship is obvious (e.g. `sources-upload-dev.yml` + `sources-upload-prod.yml` both pointing at `templates/sources-upload-stages.yml`). The variant choice cannot be hoisted into a shared sub-template because ADO requires `extends:` at the root of the entry pipeline — that is exactly why each wrapper exists. + +See [.github/workflows/ado/sources-upload.yml](.github/workflows/ado/sources-upload.yml) and [.github/workflows/ado/templates/sources-upload-stages.yml](.github/workflows/ado/templates/sources-upload-stages.yml) for the canonical example. + +## OneBranch templates (MANDATORY — wrapper only) + +The rules in this section apply to the **wrapper** file. The raw stages template MUST NOT reference OneBranch at all. + +To comply with internal policy, every wrapper MUST extend a OneBranch governed template: + +```yaml +resources: + repositories: + - repository: templates + type: git + name: OneBranch.Pipelines/GovernedTemplates + ref: refs/heads/main # exception to the "no floating refs" rule (see Security) + +extends: + template: v2/OneBranch..CrossPlat.yml@templates +``` + +Pick the variant by **what the pipeline talks to at runtime AND how it is classified in ADO**: + +| Variant | When to use | +|---------|-------------| +| `v2/OneBranch.Official.CrossPlat.yml@templates` | Pipeline interacts with **production** resources, endpoints, registries, or feeds — **OR** the ADO pipeline itself is marked as a production pipeline, regardless of what its YAML logic appears to do. | +| `v2/OneBranch.NonOfficial.CrossPlat.yml@templates` | Everything else (PR validation hitting dev/staging only, lint/test, dry-runs) **and** the ADO pipeline is not classified as production. | + +Production-classified pipelines MUST use `Official` even if the current YAML only does dev-looking work — the classification is the source of truth, not the inline logic. If you cannot determine the classification or whether the pipeline touches production, **ask the user** before choosing — do not default. + +### Required / recommended OneBranch variables + +OneBranch templates require these per job. In the wrapper/raw split, the raw stages template declares them in its `variables:` block but binds them from `${{ parameters.* }}`; the **wrapper** supplies the concrete values when invoking the raw template. + +| Variable | Required? | Purpose | Suggested raw parameter name | +|----------|-----------|---------|------------------------------| +| `ob_outputDirectory` | **Required** | Where build outputs are staged (typically `$(Build.ArtifactStagingDirectory)/output`). | `outputDirectory` | +| `ob_artifactBaseName` | Recommended | Base name for the published artifact; helps when multiple jobs publish artifacts. | `artifactBaseName` | +| `LinuxContainerImage` | Required when `pool.type: linux` with a container | Must come from `mcr.microsoft.com` (see Container images below). | `containerImage` | + +> Consult the **1ES MCP** at `https://eschat.microsoft.com/mcp` for the most current OneBranch guidance and feature flags. If it is not configured in this workspace, follow the setup notes at https://eng.ms/docs/coreai/devdiv/one-engineering-system-1es/1es-docs/es-chat/askeschat-mcp-vscode and surface that link to the user. Do not block on its absence. + +## Triggers + +Always set both triggers off in the **wrapper** YAML — pipeline wiring (PR check / merge queue) is configured in the ADO pipeline settings, outside the YAML, and is out of scope here. Raw stage templates do not declare triggers (they are not entry points): + +```yaml +trigger: none +pr: none +``` + +## Authentication & Azure access (MANDATORY) + +**Service Connections backed by Workload Identity Federation (OIDC) are the only accepted way** to access Azure resources or Control Tower endpoints from these pipelines. + +- Use `AzureCLI@2` with `azureSubscription: ` to obtain credentials/tokens. The task auto-issues a federated token; no secret material is embedded in the pipeline. +- **Do not** use PATs, account passwords, client secrets stored in pipeline variables, or `az login -u/-p` flows. +- Use a **separate, least-privilege Service Connection per environment** (e.g. one for dev, one for prod), each scoped to the minimum subscription / resource group / role required. +- Federated credentials should be scoped to the specific ADO service connection (`subject: sc:////`). + +## Control Tower endpoints + +When the pipeline calls a Control Tower API: + +- **Audience URI MUST be correct for the target environment.** The token is issued for `api://` and that client ID differs per environment (DEV vs PROD). Pull both the audience URI and the base URL from a **variable group** — never hardcode them in YAML or scripts. +- Use the per-environment Service Connection that is paired with the matching Control Tower app registration. Mixing dev SC with a prod audience (or vice versa) will fail token validation. +- To look up Control Tower endpoint shapes and request/response contracts, consult the upstream repo at `https://dev.azure.com/mariner-org/azl/_git/azl-ControlTower`. Default to the `main` branch, but **confirm with the user** whether they want to target a different commit/branch (e.g. an in-flight feature branch). +- This is a private ADO repo. Two ways to read it, in order of preference: + 1. Use the **Azure DevOps MCP** (default name `ado`) to browse it directly. If it is not configured, point the user at the setup guide: https://learn.microsoft.com/en-us/azure/devops/mcp-server/mcp-server-overview?view=azure-devops. + 2. Ask the user to provide a **path to a local clone** of `azl-ControlTower` and read endpoint definitions from there. Confirm which branch/commit the local clone is on before relying on it. + + If neither is available, fall back to best-effort guidance from what is in the current workspace and clearly flag the limitation to the user. + +## Internal-only dependency sources (MANDATORY) + +Pipelines MUST NOT pull from public/upstream registries. Use the internal mirrors / proxies for every dependency type. + +### Go modules + +Enable the OneBranch internal Go module proxy via feature flag on the `extends` block: + +```yaml +extends: + template: v2/OneBranch.NonOfficial.CrossPlat.yml@templates + parameters: + featureFlags: + golang: + internalModuleProxy: + enabled: true +``` + +### Python (pip) packages + +Authenticate pip against the internal feed before any `pip install`: + +```yaml +- task: PipAuthenticate@1 + displayName: "Authenticate pip" + inputs: + artifactFeeds: "azl/ControlTowerFeed" +``` + +Use the appropriate internal feed name for the workload (this example matches the Control Tower feed). Do not invoke `pip install` from public PyPI without authenticating to an internal feed first. + +### Container images + +Third-party (non-Microsoft) images MUST come from `mcr.microsoft.com`. This applies to: + +- `LinuxContainerImage` (and any `WindowsContainerImage`) on the OneBranch pool. +- Any image the pipeline pulls or builds from in a step (base images in Dockerfiles, sidecar containers, tools, etc.). + +Example: + +```yaml +variables: + - name: LinuxContainerImage + value: mcr.microsoft.com/onebranch/azurelinux/build:3.0 +``` + +The authoritative list and rules live at https://eng.ms/docs/more/containers-secure-supply-chain/approved-images. The agent SHOULD attempt to fetch that page for the latest guidance; if the request fails (the page is auth-walled), try the **1ES MCP** at `https://eschat.microsoft.com/mcp` — it can surface the current approved-images guidance directly. If neither source is reachable, give the user both URLs and fall back to the `mcr.microsoft.com` rule above. + +## Scripting + +Avoid shell scripts beyond the smallest possible wiring (env exports, `##vso[...]` log commands, single-command tool invocations). For anything more complex — argument parsing, control flow, JSON/YAML handling, HTTP, branch-name/SHA parsing — write a **Python script** under `.github/workflows/scripts//` and invoke it from the pipeline: + +```yaml +- task: AzureCLI@2 + displayName: "Call Control Tower endpoint" + inputs: + azureSubscription: + scriptType: bash + scriptLocation: inlineScript + inlineScript: | + set -euo pipefail + python3 .github/workflows/scripts//