feat(deploy): add LLM agentic release notes skill suite for Data Commons Platform

deploy/generate_release_notes/README.md

@@ -0,0 +1,69 @@
+# Data Commons Platform (DCP) Release Notes Generator
+
+An agentic, skill-driven tool suite for generating publication-ready, partner-facing release notes for Data Commons Platform (DCP) releases.
+
+---
+
+## Quick Start (How to Use)
+
+Simply point your LLM coding assistant (e.g. Antigravity / Gemini) at [`SKILL.md`](SKILL.md):
+
+> **Prompt Example**:
+> *"Please read `deploy/generate_release_notes/SKILL.md` and generate release notes for version v1.1.0 to v1.1.1."*
+
+---
+
+## How It Works (4 Automated Steps)
+
+1. **PR Extraction**: The agent spawns dedicated subagents to query merged Pull Requests across all Data Commons repositories for each component layer within the release window (`[prev_version .. new_version]`).
+2. **Developer Verification Checkpoint**: The agent generates human-readable text files per component under `output/prs_*.txt`. You can open and inspect these files to verify extracted PRs, change summaries, DCP impact, and excluded noise.
+3. **Release Delta Synthesis**: An agent analyzes the verified PR lists to distinguish true platform bug fixes present in prior releases from intermediate intra-release fixes, generating `output/IMAGE_DELTAS_<version>.txt`.
+4. **Final Release Notes Authoring**: The agent applies domain context and release-writing guidelines to generate the final publication-ready release notes: `output/RELEASE_NOTES_<version>.md`.
+
+---
+
+## Output Artifacts & Verification
+
+All output artifacts are generated into `deploy/generate_release_notes/output/`:
+
+- `output/prs_<component>.txt`: Extracted PRs per component with **Change Summary**, **DCP Impact**, and an **Excluded PRs Audit Log** (with explicit reasons for every ignored PR).
+- `output/IMAGE_DELTAS_<version>.txt`: Intermediate summary of salient features, configuration updates, and true platform bug fixes per container image.
+- `output/RELEASE_NOTES_<version>.md`: Final release notes formatted for external developers and instance operators.
+
+---
+
+## Architecture & Skill Reference
+
+The pipeline is organized into modular skill instruction sets under `deploy/generate_release_notes/`:
+
+```
+deploy/generate_release_notes/
+├── SKILL.md                          <-- 1. Master Orchestrator Skill (Entrypoint)
+├── skills/
+│   ├── pr-extraction/
+│   │   └── SKILL.md                  <-- 2. PR Extraction Skill (Subagent Extraction)
+│   ├── release-delta-synthesis/
+│   │   └── SKILL.md                  <-- 3. Release Delta Synthesis Skill (Image Delta Analysis)
+│   ├── dcp-context/
+│   │   └── SKILL.md                  <-- 4. DCP Domain Context & Architectural Map (Single Source of Truth)
+│   └── release-writer/
+│       └── SKILL.md                  <-- 5. Partner-Facing Release Notes Writer
+└── output/                            <-- Verification & Output Directory
+```
+
+### Skill Breakdown:
+- **Orchestrator (`SKILL.md`)**: Coordinates subagents across component layers and manages the step-by-step workflow.
+- **PR Extraction (`skills/pr-extraction/SKILL.md`)**: Instructions for date-range `gh pr list` queries, Artifact Registry tag resolution (with prompt on missing tags), and noise filtering.
+- **Release Delta Synthesis (`skills/release-delta-synthesis/SKILL.md`)**: Rules for image delta synthesis, separating Mixer, MCP Agent Toolkit, and Website UI into dedicated sections.
+- **DCP Domain Context (`skills/dcp-context/SKILL.md`)**: **Single Source of Truth** for component keys, repository mappings, subdirectory path filters, container image URIs, and persona guidelines.
+- **Release Writer (`skills/release-writer/SKILL.md`)**: Guidelines for authoring publication-ready release notes with dynamic Executive Summary scaling and two-tier feature formatting (**What's New** + **Specific Capabilities** with `[repo#PR](URL)` links).
+
+---
+
+## Maintenance & Updating for Stack Changes
+
+All component mappings, repository definitions, and image URIs are centralized in **[`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md)** (Single Source of Truth).
+
+- **Adding/Modifying a Component or Image**: Add or update the row in the Component Registry table in `skills/dcp-context/SKILL.md`. The orchestrator will dynamically spawn subagents for it.
+- **Updating Path Filters**: Update the **Source Repositories & Subdirectory Filters** column in `skills/dcp-context/SKILL.md`.
+- **Updating Writing Rules or Persona**: Update `skills/release-writer/SKILL.md` for formatting and tone, or `skills/dcp-context/SKILL.md` for user touchpoints.

deploy/generate_release_notes/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: dcp-release-notes
+description: Master orchestrator skill for generating publication-ready, partner-facing Data Commons Platform (DCP) release notes across core repositories and platform components using agentic subagents.
+---
+
+# DCP Release Notes Generator (Orchestrator Skill)
+
+**PRIME DIRECTIVE**: You are an expert Data Commons Release Engineer. Your objective is to orchestrate the end-to-end generation of publication-ready, partner-facing Data Commons Platform (DCP) release notes by coordinating specialized subagents across core repositories and platform components.
+
+---
+
+## Input & Output Contracts
+
+### Inputs
+- **`prev_version`**: Previous release tag (e.g., `v1.1.0`).
+- **`new_version`**: Target release tag (e.g., `v1.1.1`).
+
+### Target Output Artifacts
+- **Raw PR Verification Files**: `deploy/generate_release_notes/output/prs_*.txt`
+- **Unified Image Delta Summary**: `deploy/generate_release_notes/output/IMAGE_DELTAS_<new_version>.txt`
+- **Publication-Ready Release Notes**: `deploy/generate_release_notes/output/RELEASE_NOTES_<new_version>.md`
+
+---
+
+## Component & Container Image Registry Reference
+
+The authoritative mapping of component keys, container image URIs, source repositories, subdirectory path filters, and output verification files is defined strictly in **[`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md)** (Single Source of Truth).
+
+The orchestrator MUST read `skills/dcp-context/SKILL.md` dynamically to inspect the active component registry without hardcoding component lists in this file.
+
+---
+
+## Workflow Execution SOP
+
+### Step 0: Mandated Orchestrator Thinking Phase
+Before executing steps, open a `<thinking>` block to record:
+1. Format validation for `<prev_version>` and `<new_version>` (verify both follow semver `vX.Y.Z` format).
+2. Verification of `deploy/generate_release_notes/output/` directory creation.
+3. Verification of `skills/dcp-context/SKILL.md` accessibility.
+4. Orchestration plan to spawn extraction subagents concurrently across all registry rows.
+
+### Step 1: Version Resolution & Output Directory Setup
+1. Validate the previous release tag (`<prev_version>`, e.g., `v1.1.0`) and target release tag (`<new_version>`, e.g., `v1.1.1`).
+2. Create the `deploy/generate_release_notes/output/` directory if it does not already exist.
+
+> [!IMPORTANT]
+> **DO NOT resolve image tags or run `gcloud` commands in the orchestrator.**
+> The orchestrator MUST NOT query Artifact Registry or inspect image creation timestamps up front. Simply pass the raw version strings (`<prev_version>` and `<new_version>`) to each subagent and let them resolve their assigned image tags concurrently.
+
+### Step 2: Dynamically Spawn PR Extraction Subagents
+1. Read the **Component & Repository Registry** table in [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md).
+2. For **each row** in the Component Registry table, call `invoke_subagent` to spawn a dedicated extraction subagent concurrently.
+
+Provide each subagent with:
+- The **PR Extraction Skill**: [`skills/pr-extraction/SKILL.md`](skills/pr-extraction/SKILL.md).
+- The **DCP Context Skill**: [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md).
+- Assigned `component_key`, `component_name`, `image_uri`, `source_repos`, `<prev_version>`, `<new_version>`, and `output_file` from that row.
+
+### Step 3: Verification Checkpoint & Release Delta Synthesis
+1. Verify that all expected `deploy/generate_release_notes/output/prs_*.txt` files have been written successfully by the subagents.
+2. Notify the developer that raw PR verification files under `deploy/generate_release_notes/output/prs_*.txt` are ready for review.
+3. Call `invoke_subagent` to spawn a specialized **Release Delta Synthesis Subagent** (`delta-synthesizer`).
+4. Provide the subagent with the **Release Delta Synthesis Skill**: [`skills/release-delta-synthesis/SKILL.md`](skills/release-delta-synthesis/SKILL.md).
+5. Verify that the subagent outputs the unified image delta summary to `deploy/generate_release_notes/output/IMAGE_DELTAS_<new_version>.txt`.
+
+> [!WARNING]
+> If any extraction subagent fails or fails to write its output verification file, DO NOT proceed to Step 4 silently. Log an explicit warning to the developer detailing which component failed and ask how to proceed.
+
+### Step 4: Author Publication-Ready Release Notes
+1. Read the **DCP Domain Context Skill**: [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md).
+2. Read the **Release Writer Skill**: [`skills/release-writer/SKILL.md`](skills/release-writer/SKILL.md).
+3. Read `deploy/generate_release_notes/output/IMAGE_DELTAS_<new_version>.txt`.
+4. Author the final release notes from the verified image delta summary into `deploy/generate_release_notes/output/RELEASE_NOTES_<new_version>.md`.
+5. Display a summary of generated artifacts to the developer.

deploy/generate_release_notes/output/.gitignore

@@ -0,0 +1,4 @@
+# Ignore generated release notes outputs and verification logs
+*.txt
+*.md
+!.gitkeep

deploy/generate_release_notes/output/.gitkeep

@@ -0,0 +1 @@
+# Preserve output directory structure for generated release notes artifacts

deploy/generate_release_notes/skills/dcp-context/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: dcp-context
+description: Architectural reference, domain context, and component map for Data Commons Platform (DCP) release notes generation.
+---
+
+# Data Commons Platform (DCP) Domain Context & Architectural Map
+
+**PRIME DIRECTIVE**: You are an expert Data Commons Architectural & Domain Analyst. Your objective is to provide the authoritative architectural context, user touchpoint principles, and component registry for evaluating PR relevance and framing partner-facing release notes across all Data Commons Platform components.
+
+---
+
+## Input & Output Contracts
+
+### Inputs
+- **PR Metadata & Code Footprints**: PR title, body, changed files, diffs, and labels extracted from source repositories.
+
+### Target Output Context
+- **Relevance Classification**: `RELEVANT_USER_CAPABILITY`, `RELEVANT_OPERATOR_TOOL`, `RELEVANT_BUG_FIX`, or `EXCLUDED_INTERNAL_MECHANIC`.
+
+---
+
+## 1. What is Data Commons Platform (DCP)?
+
+- **Data Commons**: An open-knowledge graph that unifies public datasets across demographics, economics, climate, health, and geography into a standardized, interconnected graph structure.
+- **Data Commons Platform (DCP)**: The self-hosted, enterprise-grade deployment of Data Commons. It allows organizations and partners to deploy an isolated Data Commons instance backed by Cloud Spanner, load custom proprietary datasets alongside public Data Commons data, expose standardized SDMX 3.0 REST and FastMCP AI agent interfaces, and manage infrastructure via Terraform and CLI automation.
+
+---
+
+## 2. User & Operator Touchpoint Principles for PR Relevance
+
+When analyzing Pull Requests and synthesizing release notes, agents MUST categorize changes based on **where and how the user or operator interacts with the platform**:
+
+### A. Data Input & Ingestion Pipeline (What Data Engineers & Operators Care About)
+- **Data Input Configurations & Schemas**: Anything that changes **what types of input are accepted** by the preprocessor (custom CSV/MCF formats, column mapping definitions, schema validation rules, subject node integrity).
+- **CLI & Operational Control**: `datacommons admin` CLI parameters, flags (`--instance_name`), and deployment automation.
+- **Ingestion Speed, Performance, & Accuracy**: While import is running, operators care deeply about **throughput, execution speed, multi-threaded parsing, streaming JSON-LD batching, failure resilience, and data accuracy**.
+
+### B. Serving & Data Access (How Users & AI Interact With Their Data)
+- **Mixer Serving APIs (Primary Data Touchpoint)**: Users care deeply about the **shape and speed** of Mixer APIs (SDMX 3.0 REST Data & Availability endpoints, all `/v2/` Mixer REST & gRPC endpoints, place containment expansion `containedInPlace+`, query latency). This is their primary avenue for interacting with their data!
+- **MCP Agent Tools & Capabilities (AI Touchpoint)**: FastMCP tools (`get_multi_entity_observations`, `search_indicators`, `get_variable_metadata`) and target scopes (`custom_only`, `base_only`) — because this is a primary avenue for how AI agents and researchers query and analyze their data!
+- **Web Applications & Exploration UI**: Explore UI, Download Tool, Place Browser, Croissant JSON-LD dataset metadata — how end-users visualize, query, and export datasets.
+- **Infrastructure & Scaling Controls**: Terraform modules (`infra/dcp/`), variables (`ingestion_dataflow_max_workers`, `spanner_processing_units`), and vector search profile configurations (`--spanner_search_config_path`).
+
+### C. Internal Implementation Mechanics (Non-User Facing Noise — DO NOT EXPOSE)
+These are internal engine mechanics that partners do NOT interact with directly. They should be framed around **high-level user impact** (e.g., *"98% lower query latency"*) without exposing internal table names or DDLs:
+- Internal Cloud Spanner DDL graph table schemas and KeyValueStore tables.
+- Dataflow TFRecord chunking and intermediate GCS staging paths.
+- Cloud Workflows internal execution IDs and status tracking tables (`IngestionHistory`).
+- Internal SQL parameter unrolling and join ordering optimizations.
+
+### D. Sequential Decision SOP for Evaluating PR Relevance
+
+When evaluating any PR against domain context, follow this exact step-by-step sequence:
+
+1. **Step 1: Identify Target Component Layer**: Match modified paths against the Component Registry table (Section 3).
+2. **Step 2: Evaluate Touchpoint Category**:
+   - Check if the PR alters Data Input / Ingestion (Section 2.A) $\rightarrow$ Classify as **`RELEVANT_DATA_INPUT_OR_INGESTION`**.
+   - Check if the PR alters Serving APIs, MCP tools, or UI (Section 2.B) $\rightarrow$ Classify as **`RELEVANT_SERVING_OR_UI`**.
+   - Check if the PR is an internal DB/engine refactor (Section 2.C) $\rightarrow$ Classify as **`INTERNAL_MECHANIC`** (Reframe to high-level impact or exclude).
+3. **Step 3: Mandated Evaluation Thinking Phase**:
+   Open a `<thinking>` block to record:
+   - What changed in the code.
+   - Which touchpoint (Section 2.A, 2.B, or 2.C) is affected.
+   - The exact 1-2 sentence user capability or operator benefit statement.
+
+> [!IMPORTANT]
+> **Cross-Component PR Guardrail**: If a single PR touches multiple repository components (e.g., both Mixer proto and Website UI), assign its release note entry to the primary user-facing layer (Website UI / MCP) while referencing the underlying API change.
+
+---
+
+## 3. Component & Repository Registry (SINGLE SOURCE OF TRUTH)
+
+All skills and subagents MUST use this table as the single authoritative source of truth for component keys, source repositories, subdirectory path filters, target container images, and output verification files:
+
+| Component Key | Component Name | Source Repositories & Subdirectory Filters | Container Image URI / Release Artifact | Output Verification File |
+| :--- | :--- | :--- | :--- | :--- |
+| `services` | Core Services (Website, Mixer, MCP Agent) | `datacommonsorg/website` (`server/`, `static/`, `build/cdc_services/`)<br>`datacommonsorg/mixer` (`internal/server/`, `proto/`, `deploy/`)<br>`datacommonsorg/agent-toolkit` (`src/datacommons_mcp/`) | `gcr.io/datcom-ci/datacommons-services` | `output/prs_services.txt` |
+| `preprocessing` | Data Preprocessor | `datacommonsorg/import` (`simple/`) | `gcr.io/datcom-ci/datacommons-data` | `output/prs_preprocessing.txt` |
+| `dataflow_worker` | Dataflow Ingestion Worker | `datacommonsorg/import` (`pipeline/ingestion/`) | `us-docker.pkg.dev/datcom-ci/gcr.io/dataflow-templates/ingestion` | `output/prs_dataflow_worker.txt` |
+| `ingestion_helper` | Ingestion Helper Service | `datacommonsorg/import` (`pipeline/workflow/ingestion-helper/`) | `gcr.io/datcom-ci/datacommons-ingestion-helper` | `output/prs_ingestion_helper.txt` |
+| `postprocessing` | Postprocessing Helper Service | `datacommonsorg/import` (`pipeline/workflow/aggregation-helper/`) | `gcr.io/datcom-ci/datacommons-aggregation-helper` | `output/prs_postprocessing.txt` |
+| `dcp_monorepo` | DCP Monorepo & Terraform Infra | `datacommonsorg/datacommons` (`infra/dcp/`, `packages/`) | DCP Monorepo & Terraform Modules | `output/prs_dcp_monorepo.txt` |