feat: add sources upload pipeline#16656
Conversation
PawelWMS
force-pushed
the
asalinas/tomls/pr-check
branch
from
33bdc7a to
d664085
Compare
This reverts commit e8b5ac4.
|
Azure Pipelines: 3 pipeline(s) were filtered out due to trigger conditions. |
|
Azure Pipelines: 3 pipeline(s) were filtered out due to trigger conditions. |
|
Azure Pipelines: 3 pipeline(s) were filtered out due to trigger conditions. |
|
Azure Pipelines: 3 pipeline(s) were filtered out due to trigger conditions. |
There was a problem hiding this comment.
Pull request overview
Adds an Azure DevOps (OneBranch) pipeline entrypoint and raw stages template to call Control Tower’s prcheck API for merge-queue/CI scenarios, along with supporting helper scripts and repo authoring guidance for future ADO pipelines under .github/workflows/ado/.
Changes:
- Added ADO wrapper pipeline + OneBranch-agnostic stages template to render specs, detect affected components, and call Control Tower
prcheck. - Added Python helper scripts for Control Tower job submission/polling and for rendered-spec drift detection.
- Refactored spec-review helper scripts into a
spec-review/subdirectory and updated references/docs accordingly; added ADO pipeline authoring instructions.
Reviewed changes
Copilot reviewed 9 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| .vscode/settings.json | Adds VS Code YAML association so ADO pipeline YAMLs are treated as Azure Pipelines. |
| .github/workflows/spec-review.disabled | Updates paths to spec-review scripts/requirements after refactor. |
| .github/workflows/scripts/spec-review/spec_review_schema.py | New Pydantic-based validator + report comparison CLI. |
| .github/workflows/scripts/spec-review/requirements.txt | Python deps for spec-review validator tooling. |
| .github/workflows/scripts/spec-review/format_pr_comment.py | Formats spec-review JSON into a PR comment with links. |
| .github/workflows/scripts/spec-review/create_check_annotations.py | Generates GitHub Actions annotations from spec-review JSON. |
| .github/workflows/scripts/spec-review/_common.py | Shared path utilities for spec-review helper scripts. |
| .github/workflows/scripts/control-tower-prcheck/run_control_tower_prcheck.py | Calls CT prcheck and polls job status using Azure credential + requests. |
| .github/workflows/scripts/control-tower-prcheck/requirements.txt | Python deps for the CT prcheck caller script. |
| .github/workflows/scripts/check_rendered_specs.py | Detects rendered-spec drift (timestamp-noise filtered) and optionally comments/patches. |
| .github/workflows/scripts/README.md | Updates local dev instructions to new spec-review script paths. |
| .github/workflows/ado/templates/sources-upload-stages.yml | Raw stages template: installs tools, renders specs, computes diffs, calls CT API. |
| .github/workflows/ado/sources-upload.yml | ADO wrapper: OneBranch governed template wiring + parameters for raw stages template. |
| .github/instructions/ado-pipeline.instructions.md | Adds repo-specific rules for authoring ADO pipelines + helper scripts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| git config --unset extensions.worktreeConfig || true | ||
|
|
||
| # Full history is needed for spec rendering to work. | ||
| git fetch --unshallow |
There was a problem hiding this comment.
git fetch --unshallow exits non-zero when the checkout is already a full clone (“--unshallow on a complete repository…”), which would fail the job. Consider making this robust by detecting a shallow repo first (e.g., git rev-parse --is-shallow-repository) or falling back to a normal git fetch when --unshallow fails.
| git fetch --unshallow | |
| if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then | |
| git fetch --unshallow | |
| else | |
| git fetch | |
| fi |
| # Branch format: refs/heads/gh-readonly-queue/{base_branch}/pr-{pr_number}-{head_sha} | ||
| # Using 'test/' branches until ready with the merge queue, but the parsing logic is the same. | ||
| if ! base_branch=$(grep -oP '(?<=test/gh-readonly-queue/).+(?=/pr-[^/]+$)' <<< "$SOURCE_BRANCH"); then | ||
| echo "##[error]Unsupported SOURCE_BRANCH '$SOURCE_BRANCH' for non-PullRequest build. Expected 'refs/heads/test/gh-readonly-queue/<base>/pr-<n>-<sha>'." |
There was a problem hiding this comment.
The inline comment says the merge-queue branch format is refs/heads/gh-readonly-queue/{base_branch}/pr-..., but the parsing regex only matches test/gh-readonly-queue/.... If the pipeline runs on the actual merge queue branch (without the test/ prefix), base_branch extraction will fail and block the pipeline. Consider supporting both prefixes (or matching on gh-readonly-queue/ regardless of an optional test/ prefix) and align the error message/comment with the accepted formats.
| # Branch format: refs/heads/gh-readonly-queue/{base_branch}/pr-{pr_number}-{head_sha} | |
| # Using 'test/' branches until ready with the merge queue, but the parsing logic is the same. | |
| if ! base_branch=$(grep -oP '(?<=test/gh-readonly-queue/).+(?=/pr-[^/]+$)' <<< "$SOURCE_BRANCH"); then | |
| echo "##[error]Unsupported SOURCE_BRANCH '$SOURCE_BRANCH' for non-PullRequest build. Expected 'refs/heads/test/gh-readonly-queue/<base>/pr-<n>-<sha>'." | |
| # Supported branch formats: | |
| # - refs/heads/gh-readonly-queue/{base_branch}/pr-{pr_number}-{head_sha} | |
| # - refs/heads/test/gh-readonly-queue/{base_branch}/pr-{pr_number}-{head_sha} | |
| if ! base_branch=$(grep -oP '(?<=/(?:test/)?gh-readonly-queue/).+(?=/pr-[^/]+$)' <<< "$SOURCE_BRANCH"); then | |
| echo "##[error]Unsupported SOURCE_BRANCH '$SOURCE_BRANCH' for non-PullRequest build. Expected 'refs/heads/gh-readonly-queue/<base>/pr-<n>-<sha>' or 'refs/heads/test/gh-readonly-queue/<base>/pr-<n>-<sha>'." |
| azure-identity>=1.17.0 | ||
| requests>=2.31.0 |
There was a problem hiding this comment.
These dependencies are specified with open-ended version ranges (>=), which makes the pipeline non-reproducible over time (the resolved wheel versions can change between runs). Given the new ADO pipeline guidance in this PR calls out “Pin all tool versions”, consider pinning to exact versions (or using a constraints/lock file) so reruns are deterministic.
| azure-identity>=1.17.0 | |
| requests>=2.31.0 | |
| azure-identity==1.17.0 | |
| requests==2.31.0 |
| ], | ||
| // All YAMLs under .github/workflows/ado should be interpreted as Azure Pipelines, not GitHub Actions. | ||
| "files.associations": { | ||
| ".github/workflows/ado/*.yml": "azure-pipelines" |
There was a problem hiding this comment.
The comment says “All YAMLs under .github/workflows/ado”, but the association only matches .github/workflows/ado/*.yml (not ado/templates/*.yml, and not *.yaml). This will leave the raw stage templates still highlighted as GitHub Actions in VS Code. Consider expanding the glob(s) to include ado/templates/* (and optionally **/*.yml / **/*.yaml) and removing trailing whitespace on the value line.
| ".github/workflows/ado/*.yml": "azure-pipelines" | |
| ".github/workflows/ado/**/*.yml": "azure-pipelines", | |
| ".github/workflows/ado/**/*.yaml": "azure-pipelines" |
| if: steps.changed-specs.outputs.skip != 'true' | ||
| run: pip install -r .github/workflows/scripts/requirements.txt | ||
| run: pip install -r .github/workflows/scripts/spec-review/requirements.txt | ||
|
|
There was a problem hiding this comment.
This workflow now installs only .github/workflows/scripts/spec-review/requirements.txt and calls the validator via the new scripts/spec-review/spec_review_schema.py path, but the underlying driver scripts (.github/workflows/scripts/spec_review.sh and spec_review_multi.sh) still hardcode the old validator location ${SCRIPT_DIR}/spec_review_schema.py (i.e., .github/workflows/scripts/spec_review_schema.py). With the validator moved into scripts/spec-review/, the multi-model step will break unless those shell scripts are updated to the new path or a compatibility wrapper is kept at the old path.
| - name: Create compatibility wrapper for spec_review_schema.py | |
| if: steps.changed-specs.outputs.skip != 'true' | |
| run: | | |
| mkdir -p .github/workflows/scripts | |
| cat > .github/workflows/scripts/spec_review_schema.py <<'PY' | |
| #!/usr/bin/env python3 | |
| import runpy | |
| from pathlib import Path | |
| runpy.run_path( | |
| str(Path(__file__).resolve().parent / "spec-review" / "spec_review_schema.py"), | |
| run_name="__main__", | |
| ) | |
| PY | |
| chmod +x .github/workflows/scripts/spec_review_schema.py |
Adding a pipeline to trigger CT's workflow for uploading sources for new or updated components. It's meant to be triggered by each merge queue item.
Why an ADO pipeline instead of GitHub Actions: calling CT endpoints would require authentication secrets kept in GitHub and we want don't want that.
NOTE:
.github/workflows/scripts/check_rendered_specs.pyis temporary until #16656 is merged and I can switch to the version of that script from that PR.Other work:
requiremenets.txtfiles.