feat(deploy): add LLM agentic release notes skill suite for Data Commons Platform

deploy/generate_release_notes/README.md

@@ -0,0 +1,99 @@
+# Data Commons Platform (DCP) Release Notes Generator
+
+An agentic, skill-driven tool suite for generating publication-ready, partner-facing release notes for the Data Commons Platform (DCP).
+
+The tool automatically extracts merged Pull Requests across all 6 core Data Commons repositories, classifies them according to standard SOP categories, filters out internal test noise and regressions, writes human-verifiable PR lists per container image (`output/prs_<component>.txt`), and formats concise release notes tailored for developers and platform operators building on top of DCP.
+
+---
+
+## Agentic Skill Suite Architecture
+
+The release notes generation pipeline is structured into 4 modular `SKILL.md` instruction sets. Point your LLM agent at these skills to execute the generation process:
+
+```
+deploy/generate_release_notes/
+├── SKILL.md                          <-- 1. Master Orchestrator Skill (Entrypoint)
+├── skills/
+│   ├── pr-extraction/
+│   │   └── SKILL.md                  <-- 2. PR Extraction Skill (Subagent Extraction)
+│   ├── release-delta-synthesis/
+│   │   └── SKILL.md                  <-- 3. Release Delta Synthesis Skill (Image Delta Analysis)
+│   ├── dcp-context/
+│   │   └── SKILL.md                  <-- 4. DCP Domain Context & Architectural Map
+│   └── release-writer/
+│       └── SKILL.md                  <-- 5. Partner-Facing Release Notes Writer
+└── output/                            <-- Verification & Output Directory
+    ├── prs_services.txt              <-- Verified PRs for Core Services (Website, Mixer, MCP)
+    ├── prs_preprocessing.txt         <-- Verified PRs for Data Preprocessor (datacommons-data)
+    ├── prs_dataflow_worker.txt       <-- Verified PRs for Dataflow Worker
+    ├── prs_ingestion_helper.txt      <-- Verified PRs for Ingestion Helper
+    ├── prs_postprocessing.txt        <-- Verified PRs for Postprocessing Helper
+    ├── prs_dcp_monorepo.txt          <-- Verified PRs for DCP Monorepo & Infra
+    ├── IMAGE_DELTAS_v1.1.1.txt       <-- Intermediate Image Delta Summary (Delta vs. Previous Release)
+    └── RELEASE_NOTES_v1.1.1.md       <-- Final Publication-Ready Release Notes
+```
+
+---
+
+## Developer Usage Instructions (Prompting Your LLM Agent)
+
+### Step 1: Point Your LLM Agent at the Orchestrator Skill
+To generate release notes for a release range, point your LLM agent at [`SKILL.md`](SKILL.md):
+
+> **Prompt Example**:
+> *"Please read `deploy/generate_release_notes/SKILL.md` and generate release notes for version v1.1.0 to v1.1.1."*
+
+### Step 2: PR Extraction & Intermediate Verification Files
+Your LLM agent will spawn concurrent subagents using `skills/pr-extraction/SKILL.md` and `skills/dcp-context/SKILL.md` to:
+1. Resolve container image tags across Artifact Registry via `gcloud`.
+2. Query merged Pull Requests across all 6 Data Commons repositories via `gh pr list`.
+3. Analyze PR content, change summary, and DCP impact.
+4. Output human-verifiable text files per container image into `deploy/generate_release_notes/output/prs_*.txt`.
+
+Developers can open and inspect these `.txt` files to verify that all relevant PRs for each image are correctly captured, and inspect the **Irrelevant / Excluded PRs** audit log at the bottom.
+
+### Step 3: Release Delta Synthesis (Delta vs. Previously Published Image)
+Your LLM agent will spawn a **Release Delta Synthesis Subagent** using `skills/release-delta-synthesis/SKILL.md` to:
+1. Analyze all `output/prs_*.txt` files.
+2. Distinguish true platform bug fixes present in `<prev_version>` vs. intermediate bug fixes introduced and resolved within `<new_version>` (omitting intra-release fixes).
+3. Summarize salient features and operator capabilities added to each container image relative to `<prev_version>`.
+4. Output the intermediate delta summary file: `deploy/generate_release_notes/output/IMAGE_DELTAS_<new_version>.txt`.
+
+### Step 4: Final Release Notes Authoring
+Your LLM agent will load `skills/release-writer/SKILL.md` and author the publication-ready release notes:
+`deploy/generate_release_notes/output/RELEASE_NOTES_<new_version>.md`
+
+---
+
+## Modular Skill Breakdown
+
+### 1. Orchestrator Skill (`SKILL.md`)
+The master entrypoint skill that coordinates subagent execution, manages the step-by-step pipeline, and ensures intermediate verification files are generated before writing the final release notes.
+
+### 2. PR Extraction Skill (`skills/pr-extraction/SKILL.md`)
+Contains exact commands and rules for `gcloud container images list-tags` resolution, date-range `gh pr list` queries, non-production test filtering, and intermediate regression exclusion.
+
+### 3. DCP Domain Context Skill (`skills/dcp-context/SKILL.md`)
+Defines the architectural map across all 6 core repositories (`datacommons`, `website`, `mixer`, `agent-toolkit`, `import`) and enforces strict **External Contracts & Operator Capabilities** vs. **Zero Internal Implementation Mechanics** (no internal database table names or DDLs).
+
+### 4. Release Writer Skill (`skills/release-writer/SKILL.md`)
+Defines the non-verbose, partner-facing GFM format:
+- **Executive Summary**: 1 single sentence (max 25 words).
+- **Key Feature Updates**: **What's New** (1 paragraph combining description + benefit) followed by **Specific Capabilities** (bullet points with `[repo#PR](URL)` links).
+- **Improvements & Configuration Updates**: Bullet points extracting concrete enums (`custom_only`, `base_only`) and scaling limits (`max_workers`).
+- **Bug Fixes**: 3–5 high-level functional categories (Deployment, Ingestion, Serving APIs, UI).
+
+---
+
+## Repository Mapping
+
+| Repository | Scope / Path Filter | Target Component & Image |
+| :--- | :--- | :--- |
+| `datacommonsorg/datacommons` | All PRs (`infra/dcp/`, `packages/`) | DCP Monorepo & Infra (`dcp`) |
+| `datacommonsorg/website` | All PRs (excluding `cdc_data/`) | Core Services (`services`) $\rightarrow$ `gcr.io/datcom-ci/datacommons-services` |
+| `datacommonsorg/mixer` | All PRs (`internal/server/`, `proto/`, `deploy/`) | Core Services (`services`) $\rightarrow$ `gcr.io/datcom-ci/datacommons-services` |
+| `datacommonsorg/agent-toolkit` | All PRs (`src/datacommons_mcp/`) | Core Services (`services`) $\rightarrow$ `gcr.io/datcom-ci/datacommons-services` |
+| `datacommonsorg/import` | `simple/` | Data Preprocessor (`preprocessing`) $\rightarrow$ `gcr.io/datcom-ci/datacommons-data` |
+| `datacommonsorg/import` | `pipeline/ingestion/` | Dataflow Worker (`dataflow_worker`) $\rightarrow$ Dataflow Templates |
+| `datacommonsorg/import` | `pipeline/workflow/ingestion-helper/` | Ingestion Helper (`ingestion_helper`) $\rightarrow$ `datacommons-ingestion-helper` |
+| `datacommonsorg/import` | `pipeline/workflow/aggregation-helper/` | Postprocessing Helper (`postprocessing`) $\rightarrow$ `datacommons-aggregation-helper` |

deploy/generate_release_notes/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: dcp-release-notes
+description: Master orchestrator skill for generating publication-ready, partner-facing Data Commons Platform (DCP) release notes across all 6 core repositories using agentic subagents.
+---
+
+# DCP Release Notes Generator (Orchestrator Skill)
+
+This skill orchestrates the end-to-end generation of publication-ready, partner-facing release notes for the Data Commons Platform (DCP). It coordinates specialized subagents to extract PRs per container image, write human-readable verification `.txt` files for developer review, apply domain context, and author concise release notes.
+
+---
+
+## Component & Container Image Registry
+
+| Component Key | Component Name | Container Image URI / Artifact | Source Repos & Content Focus | Output Verification File |
+| :--- | :--- | :--- | :--- | :--- |
+| `services` | Core Services (Website, Mixer, MCP Agent) | `gcr.io/datcom-ci/datacommons-services` | `datacommonsorg/website`<br>`datacommonsorg/mixer`<br>`datacommonsorg/agent-toolkit`<br>*(Serving APIs, SDMX 3.0, FastMCP, UI)* | `output/prs_services.txt` |
+| `preprocessing` | Data Preprocessor | `gcr.io/datcom-ci/datacommons-data` | `datacommonsorg/import`<br>*(CSV/MCF validation, JSON-LD streaming batching)* | `output/prs_preprocessing.txt` |
+| `dataflow_worker` | Dataflow Ingestion Worker | `us-docker.pkg.dev/datcom-ci/gcr.io/dataflow-templates/ingestion` | `datacommonsorg/import`<br>*(Dataflow pipelines, TFRecord loading, Spanner graph transforms)* | `output/prs_dataflow_worker.txt` |
+| `ingestion_helper` | Ingestion Helper Service | `gcr.io/datcom-ci/datacommons-ingestion-helper` | `datacommonsorg/import`<br>*(Cloud Workflows status tracking, run history tables)* | `output/prs_ingestion_helper.txt` |
+| `postprocessing` | Postprocessing Helper Service | `gcr.io/datcom-ci/datacommons-aggregation-helper` | `datacommonsorg/import`<br>*(Graph postprocessing rollups, StatVar/Place aggregations, summary store)* | `output/prs_postprocessing.txt` |
+| `dcp_monorepo` | DCP Monorepo & Terraform Infra | DCP Monorepo | `datacommonsorg/datacommons`<br>*(Terraform modules, Admin CLI, deployment infra)* | `output/prs_dcp_monorepo.txt` |
+
+---
+
+## Workflow Instructions
+
+When requested to generate release notes (e.g., *"Generate release notes for v1.1.0 to v1.1.1"*):
+
+### Step 1: Version Resolution & Output Directory Setup
+1. Identify the previous release tag (`<prev_version>`, e.g., `v1.1.0`) and target release tag (`<new_version>`, e.g., `v1.1.1`).
+2. Ensure `deploy/generate_release_notes/output/` directory exists.
+
+> [!IMPORTANT]
+> **DO NOT resolve image tags or run `gcloud` commands in the orchestrator.**
+> The orchestrator MUST NOT query Artifact Registry or inspect image creation timestamps up front. Simply pass the raw version strings (`<prev_version>` and `<new_version>`) to each subagent and let them resolve their assigned image tags concurrently.
+
+### Step 2: Spawn PR Extraction Subagents
+Call `invoke_subagent` to spawn subagents concurrently across the components above. 
+
+Provide each subagent with:
+1. The **PR Extraction Skill**: [`skills/pr-extraction/SKILL.md`](skills/pr-extraction/SKILL.md).
+2. The **DCP Context Skill**: [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md).
+3. Its assigned **Component Name**, **Image URI**, **Source Repos**, `<prev_version>`, `<new_version>`, and target **Output Verification File**.
+
+#### Dedicated Subagent Tasks (1-to-1 with Component Output Files):
+- **Subagent 1 (`services-extractor`)**: Extract PRs for `gcr.io/datcom-ci/datacommons-services` from `website`, `mixer`, `agent-toolkit` $\rightarrow$ write `output/prs_services.txt`.
+- **Subagent 2 (`preprocessing-extractor`)**: Extract PRs for `gcr.io/datcom-ci/datacommons-data` (preprocessor) from `import` repo $\rightarrow$ write `output/prs_preprocessing.txt`.
+- **Subagent 3 (`dataflow-worker-extractor`)**: Extract PRs for Dataflow Ingestion Worker from `import` repo $\rightarrow$ write `output/prs_dataflow_worker.txt`.
+- **Subagent 4 (`ingestion-helper-extractor`)**: Extract PRs for Ingestion Helper Service from `import` repo $\rightarrow$ write `output/prs_ingestion_helper.txt`.
+- **Subagent 5 (`postprocessing-extractor`)**: Extract PRs for Postprocessing Helper Service from `import` repo $\rightarrow$ write `output/prs_postprocessing.txt`.
+- **Subagent 6 (`monorepo-extractor`)**: Extract PRs for `datacommonsorg/datacommons` monorepo & Terraform infra $\rightarrow$ write `output/prs_dcp_monorepo.txt`.
+
+### Step 3: Verification Checkpoint & Release Delta Synthesis Subagent
+1. Notify the developer that raw PR verification files have been generated under `deploy/generate_release_notes/output/prs_*.txt` for review.
+2. Call `invoke_subagent` to spawn a specialized **Release Delta Synthesis Subagent** (`delta-synthesizer`).
+3. Provide the subagent with the **Release Delta Synthesis Skill**: [`skills/release-delta-synthesis/SKILL.md`](skills/release-delta-synthesis/SKILL.md).
+4. The subagent will:
+   - Read all `output/prs_*.txt` files.
+   - Investigate and distinguish true bug fixes present in `<prev_version>` vs. intermediate bug fixes introduced and fixed within `<new_version>` (omitting intra-release fixes).
+   - Summarize salient features and configuration updates per container image relative to `<prev_version>`.
+   - Output the unified image delta summary to: `deploy/generate_release_notes/output/IMAGE_DELTAS_<new_version>.txt`.
+
+### Step 4: Author Publication-Ready Release Notes
+1. Read the **DCP Domain Context Skill**: [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md).
+2. Read the **Release Writer Skill**: [`skills/release-writer/SKILL.md`](skills/release-writer/SKILL.md).
+3. Read `deploy/generate_release_notes/output/IMAGE_DELTAS_<new_version>.txt`.
+4. Author the final release notes from the verified image delta summary into: `deploy/generate_release_notes/output/RELEASE_NOTES_<new_version>.md`.

deploy/generate_release_notes/output/.gitignore

@@ -0,0 +1,4 @@
+# Ignore generated release notes outputs and verification logs
+*.txt
+*.md
+!.gitkeep

deploy/generate_release_notes/output/.gitkeep

@@ -0,0 +1 @@
+# Preserve output directory structure for generated release notes artifacts

deploy/generate_release_notes/skills/dcp-context/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: dcp-context
+description: Architectural reference and domain context for Data Commons Platform (DCP) release notes generation.
+---
+
+# Data Commons Platform (DCP) Domain Context & Architectural Map
+
+This skill provides the domain context, repository mapping, and architectural principles for writing publication-ready, partner-facing DCP release notes.
+
+---
+
+## 1. Core Architectural Overview
+
+Data Commons Platform (DCP) is a self-hosted, Cloud Spanner-backed deployment of Data Commons. It replaces legacy Bigtable with Cloud Spanner graph tables and vector embeddings. It features custom data ingestion pipelines, specialized serving APIs, and deployment automation across 6 core repositories:
+
+1. `datacommonsorg/datacommons` (Monorepo & Infra):
+   - **Terraform Modules** (`infra/dcp/`, `infra/modules/`): Infrastructure provisioning for Spanner, Cloud Run, BigQuery, and Dataflow.
+   - **CLI Tools** (`packages/datacommons-cli/`): `datacommons admin init`, `datacommons admin deploy`.
+   - **Admin Portal** (`packages/datacommons-admin/`): Web management interface.
+
+2. `datacommonsorg/website` (Web Application & Frontend):
+   - Serves UI pages (Explore, Visualization Tools, Place Browser) and REST API routing (`server/`, `static/`, `build/cdc_services/`).
+
+3. `datacommonsorg/mixer` (Core Serving Engine):
+   - High-performance gRPC graph and StatVar serving engine (`internal/server/`, `proto/`, `deploy/helm_charts/`, ESPv2 gateway).
+   - Serves SDMX 3.0 REST Data & Availability endpoints, `/v2/observation`, and vector search embeddings.
+
+4. `datacommonsorg/agent-toolkit` (Model Context Protocol / MCP):
+   - Model Context Protocol (MCP) server & FastMCP tools for AI agent integrations (`src/datacommons_mcp/`).
+   - Enables agentic research playbooks, multi-entity observation retrieval, and indicator search across custom or base instances.
+
+5. `datacommonsorg/import` (Ingestion Stack & Cloud Workflows):
+   - **Data Preprocessor** (`simple/`): CSV/MCF validation and streaming JSON-LD batching (built into `datacommons-data` container image).
+   - **Dataflow Ingestion Worker** (`pipeline/ingestion/`): Parallelized BigQuery/Spanner graph loading.
+   - **Cloud Workflow Helpers** (`pipeline/workflow/ingestion-helper/`, `pipeline/workflow/aggregation-helper/`): Ingestion status tracking and postprocessing aggregations (StatVar, Place, Entity rollups).
+
+---
+
+## 2. Architectural Boundary & Persona Principles
+
+### Focus on External Contracts & Operator Capabilities
+- **Partner & Operator Focus**: Write specifically for external developers, data engineers, and instance operators building ON TOP OF DCP.
+- **User Capabilities**: Frame every feature and improvement around *what the user can now do*, *which input formats are supported*, or *how compute resources scale*.
+- **Extract Concrete Enums & Configuration Values**: Always extract valid enums (`custom_only`, `base_only`, `base_and_custom`), CLI flags (`--instance_name`), and scaling bounds (`max_workers`, BigQuery slots).
+
+### Zero Internal Implementation Mechanics (STRICT)
+- **NO Internal Database Terms**: NEVER output feature titles or section names containing internal database table names, schema DDLs, or storage migration mechanics (e.g. no "KeyValueStore", "Spanner Graph DDL", "Bigtable Cutover", "Database Schema Modification").
+- **Frame Performance Speedups Around User Impact**: If an internal storage or cache layer change improves serving speed, title it around user impact: **"API Serving Latency & Query Throughput"** or **"Faster API Response Speed"** without naming internal database tables.
+
+---
+
+## 3. Section Mapping & Release Content Principles
+
+Rather than using arbitrary internal categories, map changes directly into the three standard release notes sections based on technical impact:
+
+1. **Key Feature Updates**:
+   - Major, high-impact capabilities introduced in this release (e.g., SDMX 3.0 REST Data & Availability APIs, FastMCP AI agent tools, streaming JSON-LD preprocessors, vector search embeddings).
+   - Must follow the non-verbose format: **What's New** (1 paragraph combining description + benefit) followed by **Specific Capabilities** (bullet points with `[repo#PR](URL)` links).
+
+2. **Improvements & Configuration Updates**:
+   - Incremental enhancements, operator tools, Terraform variables (`max_workers`, BigQuery slots), CLI flags (`--instance_name`), and scaling optimizations.
+   - Must extract concrete enums (e.g. `custom_only`, `base_only`) and explicit configuration parameters.
+
+3. **Bug Fixes**:
+   - Synthesized into 3 to 5 functional categories (*Deployment & Infrastructure*, *Ingestion Pipeline Reliability*, *Serving API & Query Robustness*, *Web UI & Visualization*).
+   - Must ONLY include true platform bug fixes present in prior releases (excluding intra-release intermediate fixes).