diff --git a/deploy/generate_release_notes/README.md b/deploy/generate_release_notes/README.md new file mode 100644 index 00000000..c1dac9ee --- /dev/null +++ b/deploy/generate_release_notes/README.md @@ -0,0 +1,69 @@ +# Data Commons Platform (DCP) Release Notes Generator + +An agentic, skill-driven tool suite for generating publication-ready, partner-facing release notes for Data Commons Platform (DCP) releases. + +--- + +## Quick Start (How to Use) + +Simply point your LLM coding assistant (e.g. Antigravity / Gemini) at [`SKILL.md`](SKILL.md): + +> **Prompt Example**: +> *"Please read `deploy/generate_release_notes/SKILL.md` and generate release notes for version v1.1.0 to v1.1.1."* + +--- + +## How It Works (4 Automated Steps) + +1. **PR Extraction**: The agent spawns dedicated subagents to query merged Pull Requests across all Data Commons repositories for each component layer within the release window (`[prev_version .. new_version]`). +2. **Developer Verification Checkpoint**: The agent generates human-readable text files per component under `output/prs_*.txt`. You can open and inspect these files to verify extracted PRs, change summaries, DCP impact, and excluded noise. +3. **Release Delta Synthesis**: An agent analyzes the verified PR lists to distinguish true platform bug fixes present in prior releases from intermediate intra-release fixes, generating `output/IMAGE_DELTAS_.txt`. +4. **Final Release Notes Authoring**: The agent applies domain context and release-writing guidelines to generate the final publication-ready release notes: `output/RELEASE_NOTES_.md`. + +--- + +## Output Artifacts & Verification + +All output artifacts are generated into `deploy/generate_release_notes/output/`: + +- `output/prs_.txt`: Extracted PRs per component with **Change Summary**, **DCP Impact**, and an **Excluded PRs Audit Log** (with explicit reasons for every ignored PR). +- `output/IMAGE_DELTAS_.txt`: Intermediate summary of salient features, configuration updates, and true platform bug fixes per container image. +- `output/RELEASE_NOTES_.md`: Final release notes formatted for external developers and instance operators. + +--- + +## Architecture & Skill Reference + +The pipeline is organized into modular skill instruction sets under `deploy/generate_release_notes/`: + +``` +deploy/generate_release_notes/ +├── SKILL.md <-- 1. Master Orchestrator Skill (Entrypoint) +├── skills/ +│ ├── pr-extraction/ +│ │ └── SKILL.md <-- 2. PR Extraction Skill (Subagent Extraction) +│ ├── release-delta-synthesis/ +│ │ └── SKILL.md <-- 3. Release Delta Synthesis Skill (Image Delta Analysis) +│ ├── dcp-context/ +│ │ └── SKILL.md <-- 4. DCP Domain Context & Architectural Map (Single Source of Truth) +│ └── release-writer/ +│ └── SKILL.md <-- 5. Partner-Facing Release Notes Writer +└── output/ <-- Verification & Output Directory +``` + +### Skill Breakdown: +- **Orchestrator (`SKILL.md`)**: Coordinates subagents across component layers and manages the step-by-step workflow. +- **PR Extraction (`skills/pr-extraction/SKILL.md`)**: Instructions for date-range `gh pr list` queries, Artifact Registry tag resolution (with prompt on missing tags), and noise filtering. +- **Release Delta Synthesis (`skills/release-delta-synthesis/SKILL.md`)**: Rules for image delta synthesis, separating Mixer, MCP Agent Toolkit, and Website UI into dedicated sections. +- **DCP Domain Context (`skills/dcp-context/SKILL.md`)**: **Single Source of Truth** for component keys, repository mappings, subdirectory path filters, container image URIs, and persona guidelines. +- **Release Writer (`skills/release-writer/SKILL.md`)**: Guidelines for authoring publication-ready release notes with dynamic Executive Summary scaling and two-tier feature formatting (**What's New** + **Specific Capabilities** with `[repo#PR](URL)` links). + +--- + +## Maintenance & Updating for Stack Changes + +All component mappings, repository definitions, and image URIs are centralized in **[`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md)** (Single Source of Truth). + +- **Adding/Modifying a Component or Image**: Add or update the row in the Component Registry table in `skills/dcp-context/SKILL.md`. The orchestrator will dynamically spawn subagents for it. +- **Updating Path Filters**: Update the **Source Repositories & Subdirectory Filters** column in `skills/dcp-context/SKILL.md`. +- **Updating Writing Rules or Persona**: Update `skills/release-writer/SKILL.md` for formatting and tone, or `skills/dcp-context/SKILL.md` for user touchpoints. diff --git a/deploy/generate_release_notes/SKILL.md b/deploy/generate_release_notes/SKILL.md new file mode 100644 index 00000000..3f12c8ed --- /dev/null +++ b/deploy/generate_release_notes/SKILL.md @@ -0,0 +1,74 @@ +--- +name: dcp-release-notes +description: Master orchestrator skill for generating publication-ready, partner-facing Data Commons Platform (DCP) release notes across core repositories and platform components using agentic subagents. +--- + +# DCP Release Notes Generator (Orchestrator Skill) + +**PRIME DIRECTIVE**: You are an expert Data Commons Release Engineer. Your objective is to orchestrate the end-to-end generation of publication-ready, partner-facing Data Commons Platform (DCP) release notes by coordinating specialized subagents across core repositories and platform components. + +--- + +## Input & Output Contracts + +### Inputs +- **`prev_version`**: Previous release tag (e.g., `v1.1.0`). +- **`new_version`**: Target release tag (e.g., `v1.1.1`). + +### Target Output Artifacts +- **Raw PR Verification Files**: `deploy/generate_release_notes/output/prs_*.txt` +- **Unified Image Delta Summary**: `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt` +- **Publication-Ready Release Notes**: `deploy/generate_release_notes/output/RELEASE_NOTES_.md` + +--- + +## Component & Container Image Registry Reference + +The authoritative mapping of component keys, container image URIs, source repositories, subdirectory path filters, and output verification files is defined strictly in **[`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md)** (Single Source of Truth). + +The orchestrator MUST read `skills/dcp-context/SKILL.md` dynamically to inspect the active component registry without hardcoding component lists in this file. + +--- + +## Workflow Execution SOP + +### Step 0: Mandated Orchestrator Thinking Phase +Before executing steps, open a `` block to record: +1. Format validation for `` and `` (verify both follow semver `vX.Y.Z` format). +2. Verification of `deploy/generate_release_notes/output/` directory creation. +3. Verification of `skills/dcp-context/SKILL.md` accessibility. +4. Orchestration plan to spawn extraction subagents concurrently across all registry rows. + +### Step 1: Version Resolution & Output Directory Setup +1. Validate the previous release tag (``, e.g., `v1.1.0`) and target release tag (``, e.g., `v1.1.1`). +2. Create the `deploy/generate_release_notes/output/` directory if it does not already exist. + +> [!IMPORTANT] +> **DO NOT resolve image tags or run `gcloud` commands in the orchestrator.** +> The orchestrator MUST NOT query Artifact Registry or inspect image creation timestamps up front. Simply pass the raw version strings (`` and ``) to each subagent and let them resolve their assigned image tags concurrently. + +### Step 2: Dynamically Spawn PR Extraction Subagents +1. Read the **Component & Repository Registry** table in [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md). +2. For **each row** in the Component Registry table, call `invoke_subagent` to spawn a dedicated extraction subagent concurrently. + +Provide each subagent with: +- The **PR Extraction Skill**: [`skills/pr-extraction/SKILL.md`](skills/pr-extraction/SKILL.md). +- The **DCP Context Skill**: [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md). +- Assigned `component_key`, `component_name`, `image_uri`, `source_repos`, ``, ``, and `output_file` from that row. + +### Step 3: Verification Checkpoint & Release Delta Synthesis +1. Verify that all expected `deploy/generate_release_notes/output/prs_*.txt` files have been written successfully by the subagents. +2. Notify the developer that raw PR verification files under `deploy/generate_release_notes/output/prs_*.txt` are ready for review. +3. Call `invoke_subagent` to spawn a specialized **Release Delta Synthesis Subagent** (`delta-synthesizer`). +4. Provide the subagent with the **Release Delta Synthesis Skill**: [`skills/release-delta-synthesis/SKILL.md`](skills/release-delta-synthesis/SKILL.md). +5. Verify that the subagent outputs the unified image delta summary to `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt`. + +> [!WARNING] +> If any extraction subagent fails or fails to write its output verification file, DO NOT proceed to Step 4 silently. Log an explicit warning to the developer detailing which component failed and ask how to proceed. + +### Step 4: Author Publication-Ready Release Notes +1. Read the **DCP Domain Context Skill**: [`skills/dcp-context/SKILL.md`](skills/dcp-context/SKILL.md). +2. Read the **Release Writer Skill**: [`skills/release-writer/SKILL.md`](skills/release-writer/SKILL.md). +3. Read `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt`. +4. Author the final release notes from the verified image delta summary into `deploy/generate_release_notes/output/RELEASE_NOTES_.md`. +5. Display a summary of generated artifacts to the developer. diff --git a/deploy/generate_release_notes/output/.gitignore b/deploy/generate_release_notes/output/.gitignore new file mode 100644 index 00000000..6e9fd814 --- /dev/null +++ b/deploy/generate_release_notes/output/.gitignore @@ -0,0 +1,4 @@ +# Ignore generated release notes outputs and verification logs +*.txt +*.md +!.gitkeep diff --git a/deploy/generate_release_notes/output/.gitkeep b/deploy/generate_release_notes/output/.gitkeep new file mode 100644 index 00000000..8ab527fd --- /dev/null +++ b/deploy/generate_release_notes/output/.gitkeep @@ -0,0 +1 @@ +# Preserve output directory structure for generated release notes artifacts diff --git a/deploy/generate_release_notes/skills/dcp-context/SKILL.md b/deploy/generate_release_notes/skills/dcp-context/SKILL.md new file mode 100644 index 00000000..51f2225a --- /dev/null +++ b/deploy/generate_release_notes/skills/dcp-context/SKILL.md @@ -0,0 +1,82 @@ +--- +name: dcp-context +description: Architectural reference, domain context, and component map for Data Commons Platform (DCP) release notes generation. +--- + +# Data Commons Platform (DCP) Domain Context & Architectural Map + +**PRIME DIRECTIVE**: You are an expert Data Commons Architectural & Domain Analyst. Your objective is to provide the authoritative architectural context, user touchpoint principles, and component registry for evaluating PR relevance and framing partner-facing release notes across all Data Commons Platform components. + +--- + +## Input & Output Contracts + +### Inputs +- **PR Metadata & Code Footprints**: PR title, body, changed files, diffs, and labels extracted from source repositories. + +### Target Output Context +- **Relevance Classification**: `RELEVANT_USER_CAPABILITY`, `RELEVANT_OPERATOR_TOOL`, `RELEVANT_BUG_FIX`, or `EXCLUDED_INTERNAL_MECHANIC`. + +--- + +## 1. What is Data Commons Platform (DCP)? + +- **Data Commons**: An open-knowledge graph that unifies public datasets across demographics, economics, climate, health, and geography into a standardized, interconnected graph structure. +- **Data Commons Platform (DCP)**: The self-hosted, enterprise-grade deployment of Data Commons. It allows organizations and partners to deploy an isolated Data Commons instance backed by Cloud Spanner, load custom proprietary datasets alongside public Data Commons data, expose standardized SDMX 3.0 REST and FastMCP AI agent interfaces, and manage infrastructure via Terraform and CLI automation. + +--- + +## 2. User & Operator Touchpoint Principles for PR Relevance + +When analyzing Pull Requests and synthesizing release notes, agents MUST categorize changes based on **where and how the user or operator interacts with the platform**: + +### A. Data Input & Ingestion Pipeline (What Data Engineers & Operators Care About) +- **Data Input Configurations & Schemas**: Anything that changes **what types of input are accepted** by the preprocessor (custom CSV/MCF formats, column mapping definitions, schema validation rules, subject node integrity). +- **CLI & Operational Control**: `datacommons admin` CLI parameters, flags (`--instance_name`), and deployment automation. +- **Ingestion Speed, Performance, & Accuracy**: While import is running, operators care deeply about **throughput, execution speed, multi-threaded parsing, streaming JSON-LD batching, failure resilience, and data accuracy**. + +### B. Serving & Data Access (How Users & AI Interact With Their Data) +- **Mixer Serving APIs (Primary Data Touchpoint)**: Users care deeply about the **shape and speed** of Mixer APIs (SDMX 3.0 REST Data & Availability endpoints, all `/v2/` Mixer REST & gRPC endpoints, place containment expansion `containedInPlace+`, query latency). This is their primary avenue for interacting with their data! +- **MCP Agent Tools & Capabilities (AI Touchpoint)**: FastMCP tools (`get_multi_entity_observations`, `search_indicators`, `get_variable_metadata`) and target scopes (`custom_only`, `base_only`) — because this is a primary avenue for how AI agents and researchers query and analyze their data! +- **Web Applications & Exploration UI**: Explore UI, Download Tool, Place Browser, Croissant JSON-LD dataset metadata — how end-users visualize, query, and export datasets. +- **Infrastructure & Scaling Controls**: Terraform modules (`infra/dcp/`), variables (`ingestion_dataflow_max_workers`, `spanner_processing_units`), and vector search profile configurations (`--spanner_search_config_path`). + +### C. Internal Implementation Mechanics (Non-User Facing Noise — DO NOT EXPOSE) +These are internal engine mechanics that partners do NOT interact with directly. They should be framed around **high-level user impact** (e.g., *"98% lower query latency"*) without exposing internal table names or DDLs: +- Internal Cloud Spanner DDL graph table schemas and KeyValueStore tables. +- Dataflow TFRecord chunking and intermediate GCS staging paths. +- Cloud Workflows internal execution IDs and status tracking tables (`IngestionHistory`). +- Internal SQL parameter unrolling and join ordering optimizations. + +### D. Sequential Decision SOP for Evaluating PR Relevance + +When evaluating any PR against domain context, follow this exact step-by-step sequence: + +1. **Step 1: Identify Target Component Layer**: Match modified paths against the Component Registry table (Section 3). +2. **Step 2: Evaluate Touchpoint Category**: + - Check if the PR alters Data Input / Ingestion (Section 2.A) $\rightarrow$ Classify as **`RELEVANT_DATA_INPUT_OR_INGESTION`**. + - Check if the PR alters Serving APIs, MCP tools, or UI (Section 2.B) $\rightarrow$ Classify as **`RELEVANT_SERVING_OR_UI`**. + - Check if the PR is an internal DB/engine refactor (Section 2.C) $\rightarrow$ Classify as **`INTERNAL_MECHANIC`** (Reframe to high-level impact or exclude). +3. **Step 3: Mandated Evaluation Thinking Phase**: + Open a `` block to record: + - What changed in the code. + - Which touchpoint (Section 2.A, 2.B, or 2.C) is affected. + - The exact 1-2 sentence user capability or operator benefit statement. + +> [!IMPORTANT] +> **Cross-Component PR Guardrail**: If a single PR touches multiple repository components (e.g., both Mixer proto and Website UI), assign its release note entry to the primary user-facing layer (Website UI / MCP) while referencing the underlying API change. + +--- + +## 3. Component & Repository Registry (SINGLE SOURCE OF TRUTH) + +All skills and subagents MUST use this table as the single authoritative source of truth for component keys, source repositories, subdirectory path filters, target container images, and output verification files: + +| Component Key | Component Name | Source Repositories & Subdirectory Filters | Container Image URI / Release Artifact | Output Verification File | +| :--- | :--- | :--- | :--- | :--- | +| `services` | Core Services (Website, Mixer, MCP Agent) | `datacommonsorg/website` (`server/`, `static/`, `build/cdc_services/`)
`datacommonsorg/mixer` (`internal/server/`, `proto/`, `deploy/`)
`datacommonsorg/agent-toolkit` (`src/datacommons_mcp/`) | `gcr.io/datcom-ci/datacommons-services` | `output/prs_services.txt` | +| `preprocessing` | Data Preprocessor | `datacommonsorg/import` (`simple/`) | `gcr.io/datcom-ci/datacommons-data` | `output/prs_preprocessing.txt` | +| `dataflow_worker` | Dataflow Ingestion Worker | `datacommonsorg/import` (`pipeline/ingestion/`) | `us-docker.pkg.dev/datcom-ci/gcr.io/dataflow-templates/ingestion` | `output/prs_dataflow_worker.txt` | +| `ingestion_helper` | Ingestion Helper Service | `datacommonsorg/import` (`pipeline/workflow/ingestion-helper/`) | `gcr.io/datcom-ci/datacommons-ingestion-helper` | `output/prs_ingestion_helper.txt` | +| `postprocessing` | Postprocessing Helper Service | `datacommonsorg/import` (`pipeline/workflow/aggregation-helper/`) | `gcr.io/datcom-ci/datacommons-aggregation-helper` | `output/prs_postprocessing.txt` | +| `dcp_monorepo` | DCP Monorepo & Terraform Infra | `datacommonsorg/datacommons` (`infra/dcp/`, `packages/`) | DCP Monorepo & Terraform Modules | `output/prs_dcp_monorepo.txt` | diff --git a/deploy/generate_release_notes/skills/pr-extraction/SKILL.md b/deploy/generate_release_notes/skills/pr-extraction/SKILL.md new file mode 100644 index 00000000..cd4eb0cb --- /dev/null +++ b/deploy/generate_release_notes/skills/pr-extraction/SKILL.md @@ -0,0 +1,124 @@ +--- +name: dcp-pr-extraction +description: Subagent instruction skill for extracting, filtering, and verifying merged Pull Requests for assigned container images and repositories. +--- + +# DCP PR Extraction & Image Verification Skill (Subagent Skill) + +**PRIME DIRECTIVE**: You are an expert Data Commons Release Subagent. Your objective is to extract, filter, verify, and document merged Pull Requests for your assigned container image and repository path filter within exact release boundaries into a human-readable verification file. + +--- + +## Input & Output Contracts + +### Inputs Provided by Orchestrator +- **`component_key`**: Internal component identifier (e.g. `services`, `preprocessing`). +- **`component_name`**: Human-readable component name (e.g. `Core Services (Website, Mixer, MCP Agent)`). +- **`image_uri`**: Container Image URI in Artifact Registry (e.g. `gcr.io/datcom-ci/datacommons-services`). +- **`source_repos`**: List of source repositories and path filters to extract PRs from. +- **`prev_version`**: Previous release tag (e.g. `v1.1.0`). +- **`new_version`**: Target release tag (e.g. `v1.1.1`). +- **`output_file`**: Output file path (e.g. `deploy/generate_release_notes/output/prs_services.txt`). + +### Target Output Artifact +- **Verification File**: `deploy/generate_release_notes/output/prs_.txt` containing relevant production PRs and complete audit logs of excluded PRs with explicit 1-sentence reasons. + +--- + +## Execution SOP Sequence + +### Step 1: Container Image Tag & Timestamp Resolution (NO AUTOMATIC FALLBACK) +1. **Artifact Registry Tag Resolution**: + Resolve the creation timestamp for `` and `` for your assigned `image_uri`: + ```bash + gcloud container images list-tags --filter="tags:" --format="value(timestamp.datetime)" + ``` +2. **STRICT MANDATE — ASK USER ON MISSING TAGS**: + If an image tag does NOT exist in Artifact Registry for `` or ``, **DO NOT automatically guess, synthesize, or fall back to git tags**. + Stop immediately and ask the user how to proceed (e.g., provide an alternative tag, specify custom date boundaries, or pass `--allow-missing-images` to use `NOW()`). + +### Step 2: Single Date-Range PR Search per Repository +1. For each assigned source repository, execute a single `gh pr list` query spanning `[t_prev .. t_new]`: + ```bash + gh pr list --repo --state merged --search "merged:.." --json number,title,body,author,url,labels,files,mergedAt --limit 200 + ``` +2. *(IMPORTANT: Do NOT pass `base:main` inside `--search`; use `--search "merged:.."` directly to prevent GitHub Search API parse errors!)* + +> [!NOTE] +> **Zero PR Range Guardrail**: If `gh pr list` returns 0 PRs within the date range, verify image tag timestamps. If verified, write the `prs_.txt` file with `Total Relevant PRs: 0` and explicitly state: *"No merged PRs found in release window."* + +### Step 3: DCP Context & Semantic Content Analysis +1. Read [`skills/dcp-context/SKILL.md`](../dcp-context/SKILL.md) to understand how your assigned component fits into platform architecture and user touchpoints (Section 2). +2. Analyze the actual content of each PR (title, description body, labels, and changed code context) against the DCP context touchpoints to evaluate relevance: + - **Data Preprocessor (`preprocessing` / `datacommons-data`)**: Include PRs affecting CSV/MCF parsing, streaming JSON-LD batching, schema validation, column mapping, or preprocessor execution. + - **Dataflow Ingestion Worker (`dataflow_worker`)**: Include PRs affecting Dataflow pipelines, TFRecord loading, BigQuery/Spanner graph transformations, or batch import scaling (`max_workers`). + - **Ingestion Helper Service (`ingestion_helper`)**: Include PRs affecting Cloud Workflows orchestration, ingestion status tracking, execution IDs, status polling, or run history tables. + - **Postprocessing Helper Service (`postprocessing`)**: Include PRs affecting graph postprocessing rollups, StatVar/Place/Entity aggregations, Data-Point Vectors (DPVs), or pre-computed summary stores. + - **Core Services (`services` / `datacommons-services`)**: Include PRs affecting serving APIs (Mixer gRPC, SDMX 3.0 REST, MCP agent tools, `/v2/` endpoints), vector embeddings, or Website Explore UI tools. + - **DCP Monorepo & Infra (`dcp_monorepo`)**: Include PRs affecting Terraform modules (`infra/dcp/`), Admin CLI tools (`datacommons admin`), or deployment infrastructure. + +### Step 4: Noise, Revert PRs, and Regression Categorization +Categorize every PR into either **Relevant PRs** or **Excluded PRs**: +1. **Relevant PRs**: Direct partner/operator features, configuration capabilities, or true platform bug fixes. +2. **Excluded PRs**: + - **Revert / Superseded PR Pairs**: If a PR reverts or supersedes another PR merged *within the same release window* (`[t_prev..t_new]`), exclude BOTH PRs. + - **Base DC Only / Flag Flips**: PRs that only affect internal Google-hosted Base DC or internal flag flips without platform impact. + - **Intermediate Regressions**: Bug fix PRs that address features/code introduced within the same release window (`[t_prev..t_new]`). + - **Bot & Non-Production Chores**: Dependabot bumps, automated version bumps, unit/integration test harness refactors, or test sample data removals. + +### Step 5: Mandated Classification Thinking Phase +Before writing `output_file`, open a `` block to record: +1. Resolved date range boundaries (`t_prev` and `t_new`). +2. Total raw merged PRs retrieved across all assigned repositories. +3. List of Revert PR pairs identified and excluded. +4. List of Intermediate Regression Fixes identified and excluded. +5. List of Relevant Production PRs with 1-2 sentence Change Summary and DCP Impact for each. +6. List of Excluded PRs with explicit 1-sentence Exclusion Reasons. + +### Step 6: Write Verification File (`prs_.txt`) +Format and write the extracted PRs into your assigned `output_file` using the exact template below. + +*MANDATE*: Every single PR that is NOT included in Relevant Production PRs MUST be listed under Excluded PRs with an explicit, 1-sentence `Reason:` explaining why it was ignored (e.g., Base DC flag flip, revert pair, intermediate regression fix, bot bump, or test harness refactor). + +``` +================================================================================ +Component: {component_name} +Image URI: {image_uri} +Release Range: {prev_version} ({t_prev}) -> {new_version} ({t_new}) +Total Relevant PRs: {relevant_count} | Total Excluded PRs: {excluded_count} +================================================================================ + +--- RELEVANT PRODUCTION PRS --- + +[{repo_short}#{number}] {title} (Author: {author} | Merged: {merged_at}) +URL: {url} +- Change Summary: {1-2 sentence summary of what changed in this PR} +- DCP Impact: {1-2 sentence explanation of user capability, API contract, or operator benefit} + +[{repo_short}#{number}] {title} (Author: {author} | Merged: {merged_at}) +URL: {url} +- Change Summary: {1-2 sentence summary of what changed in this PR} +- DCP Impact: {1-2 sentence explanation of user capability, API contract, or operator benefit} + +================================================================================ +--- IRRELEVANT / EXCLUDED PRS (AUDIT LOG) --- +================================================================================ + +[{repo_short}#{number}] {title} +Reason: Excluded - Base DC-only flag flip / internal feature toggle without platform impact +URL: {url} + +[{repo_short}#{number}] {title} +Reason: Excluded - Revert PR pair (reverted by PR {revert_pr_id} merged in current release window) +URL: {url} + +[{repo_short}#{number}] {title} +Reason: Excluded - Intermediate regression fix for PR {parent_pr_id} merged in current release window +URL: {url} + +[{repo_short}#{number}] {title} +Reason: Excluded - Unit test harness refactor / test sample data update +URL: {url} +``` + +Confirm when your assigned verification file has been written cleanly to `output_file`. diff --git a/deploy/generate_release_notes/skills/release-delta-synthesis/SKILL.md b/deploy/generate_release_notes/skills/release-delta-synthesis/SKILL.md new file mode 100644 index 00000000..d160f392 --- /dev/null +++ b/deploy/generate_release_notes/skills/release-delta-synthesis/SKILL.md @@ -0,0 +1,133 @@ +--- +name: dcp-release-delta-synthesis +description: Subagent skill for analyzing raw PR verification files (prs_*.txt) and synthesizing image-level release deltas relative to the previously published release image. +--- + +# DCP Release Delta Synthesis Skill (Image Impact & Delta Analysis) + +**PRIME DIRECTIVE**: You are an expert Data Commons Release Delta Synthesizer. Your objective is to analyze all raw PR verification files (`deploy/generate_release_notes/output/prs_*.txt`), evaluate component-level changes relative to previously published container images, filter out intra-release intermediate bug fixes, and synthesize a unified `IMAGE_DELTAS_.txt` document. + +--- + +## Input & Output Contracts + +### Input Files +- **Raw Verification Files**: `deploy/generate_release_notes/output/prs_*.txt` (`prs_services.txt`, `prs_preprocessing.txt`, `prs_dataflow_worker.txt`, `prs_ingestion_helper.txt`, `prs_postprocessing.txt`, `prs_dcp_monorepo.txt`). +- **Context Reference**: [`skills/dcp-context/SKILL.md`](../dcp-context/SKILL.md) (User Touchpoint Principles Section 2). + +### Target Output Artifact +- **Unified Image Delta Summary**: `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt`. + +--- + +## Execution SOP Sequence + +### Step 1: Read & Consolidate PR Verification Files +1. Check that all expected `prs_*.txt` files exist in `deploy/generate_release_notes/output/`. +2. Read all `prs_*.txt` files. +3. Group PRs by component key (`services`, `preprocessing`, `dataflow_worker`, `ingestion_helper`, `postprocessing`, `dcp_monorepo`). + +> [!IMPORTANT] +> **Input Verification Guardrail**: If any `prs_*.txt` file is missing or unreadable, halt execution immediately and report the missing component file to the orchestrator agent before proceeding. + +### Step 2: Mandated Bug Fix Lineage Tracing (Thinking Phase) +For EVERY bug fix PR found across all `prs_*.txt` files, open a `` block to record your step-by-step investigation: + +1. **PR Title & Description Reference Check**: + - Check if the PR description references another PR merged in the current range (e.g., *"Fixes #2015"*, *"Follow up to #2000"*, *"Regression introduced by #1967"*). + - If it references an intermediate PR merged within `[t_prev..t_new]`, classify it as **`INTRA_RELEASE_INTERMEDIATE_FIX` (EXCLUDE FROM PUBLIC NOTES)**. +2. **Footprint & Feature Matching**: + - Compare the PR's `Change Summary` and modified files against major features in `prs_*.txt` merged earlier in the window. + - If the bug is for a feature first introduced in this release window (e.g. SDMX 3.0 REST endpoints, FastMCP tools), classify it as **`INTRA_RELEASE_INTERMEDIATE_FIX` (EXCLUDE FROM PUBLIC NOTES)** because users running `` were never exposed to this bug. +3. **Git History Verification (If Ambiguous)**: + - If a bug fix is ambiguous, run `git log -S "" ` or inspect `git diff ..` via shell to check if the code existed in ``. + - If the code existed in `` and was broken, classify it as **`PRIOR_RELEASE_TRUE_FIX` (INCLUDE IN PUBLIC NOTES)**. + +### Step 3: Synthesize Salient Image Deltas & Preserve URLs +For each container image / component, summarize the salient changes from the perspective of an operator upgrading from `` to ``. + +*CRITICAL MANDATE*: Preserve the full `URL` string for EVERY PR referenced in the delta summary so the final Release Writer can format clean GFM links `[repo#PR](URL)` without guessing! + +1. **Major Feature Capabilities Added**: + - What new capabilities exist in this image that were not present in ``? + - What new API endpoints, protocols (e.g. SDMX 3.0, FastMCP), or UI tools are now available? +2. **Configuration & Infra Updates**: + - What new Terraform variables, CLI parameters (`--instance_name`), or environment variables were added? + - What scaling bounds (`max_workers`, BigQuery slots) or memory optimizations were introduced? +3. **True Platform Bug Fixes**: + - What issues present in `` were resolved in this container image? Include full PR URLs for each fix! + +--- + +## Component & Service Separation Rules + +Do NOT club Website, Mixer, and MCP Agent Toolkit together in the output synthesis. Separate them into distinct, dedicated component sections so developers and operators can clearly see changes per layer: + +1. **`Mixer Serving Engine & SDMX APIs`** (`datacommonsorg/mixer`): Core gRPC serving engine, SDMX 3.0 REST Data/Availability endpoints, all `/v2/` Mixer REST & gRPC endpoints, vector search indexing, SQL query planner optimizations. +2. **`MCP Agent Toolkit & FastMCP Tools`** (`datacommonsorg/agent-toolkit`): FastMCP tools, `get_multi_entity_observations`, indicator search tools, target scope resolution (`custom_only`, `base_only`). +3. **`Website UI & Exploration Tools`** (`datacommonsorg/website`): Explore UI, Download Tool, Place Browser, Croissant dataset metadata, web server routing and caching. +4. **`Data Preprocessor`** (`datacommonsorg/import` - `datacommons-data`): CSV/MCF validation, 10k-node streaming JSON-LD batching, namespace mapping. +5. **`Dataflow Ingestion Worker`** (`datacommonsorg/import` - Dataflow Templates): TFRecord loading, Spanner graph transformations, `max_workers` auto-scaling. +6. **`Ingestion Helper Service`** (`datacommonsorg/import` - `datacommons-ingestion-helper`): Cloud Workflows status tracking, execution IDs, history tables. +7. **`Postprocessing Aggregation Helper`** (`datacommonsorg/import` - `datacommons-aggregation-helper`): StatVar/Place/Entity rollups, summary store, DPV aggregations. +8. **`DCP Monorepo & Infrastructure`** (`datacommonsorg/datacommons`): Terraform modules (`infra/dcp/`), Admin CLI (`datacommons admin`), Cloud Run job orchestration. + +--- + +## Output Document Structure (`IMAGE_DELTAS_.txt`) + +Write `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt` using the exact structure below: + +``` +================================================================================ +DCP RELEASE DELTA SUMMARY: {prev_version} -> {new_version} +Generated Date: {date} +================================================================================ + +-------------------------------------------------------------------------------- +1. COMPONENT: Mixer Serving Engine & SDMX APIs (datacommonsorg/mixer) + Container Image: gcr.io/datcom-ci/datacommons-services (Mixer binary) +-------------------------------------------------------------------------------- + +SALIENT FEATURES & CAPABILITIES (vs. {prev_version}): +- SDMX 3.0 REST Data & Availability Endpoints: Serves multi-entity observations in SDMX-CSV 2.0 format with facetId filtering and containedInPlace+ expansion ([mixer#1976](https://github.com/datacommonsorg/mixer/pull/1976), [mixer#1988](https://github.com/datacommonsorg/mixer/pull/1988), [mixer#2000](https://github.com/datacommonsorg/mixer/pull/2000)). + +CONFIGURATIONS & OPERATOR UPDATES: +- Vector Search Profiles: Support custom embedding profiles via --spanner_search_config_path ([mixer#2039](https://github.com/datacommonsorg/mixer/pull/2039)). + +TRUE PLATFORM BUG FIXES (Fixes issues present in {prev_version}): +- Serving SQL Optimization: Unroll SQL array parameters for size <= 10 to resolve latency spikes ([mixer#1993](https://github.com/datacommonsorg/mixer/pull/1993)). + +[INTRA-RELEASE FIXES EXCLUDED FROM PUBLIC NOTES: mixer#2025, mixer#2070] + +-------------------------------------------------------------------------------- +2. COMPONENT: MCP Agent Toolkit & FastMCP Tools (datacommonsorg/agent-toolkit) + Container Image: gcr.io/datcom-ci/datacommons-services (MCP binary) +-------------------------------------------------------------------------------- + +SALIENT FEATURES & CAPABILITIES (vs. {prev_version}): +- FastMCP Agent Integration: Exposes get_multi_entity_observations tool and indicator search with custom_only/base_only target scopes ([agent-toolkit#211](https://github.com/datacommonsorg/agent-toolkit/pull/211), [agent-toolkit#212](https://github.com/datacommonsorg/agent-toolkit/pull/212)). + +TRUE PLATFORM BUG FIXES (Fixes issues present in {prev_version}): +- Agent API Protocol: Updated V2AgentGetObservations to HTTP POST for large payload handling ([agent-toolkit#213](https://github.com/datacommonsorg/agent-toolkit/pull/213)). + +-------------------------------------------------------------------------------- +3. COMPONENT: Website UI & Exploration Tools (datacommonsorg/website) + Container Image: gcr.io/datcom-ci/datacommons-services (Website binary) +-------------------------------------------------------------------------------- + +SALIENT FEATURES & CAPABILITIES (vs. {prev_version}): +- Download Tool Redesign: Enhanced export interface for custom variable datasets ([website#6411](https://github.com/datacommonsorg/website/pull/6411)). +- Croissant Dataset Metadata: Inject Croissant JSON-LD for dataset indexing ([website#6443](https://github.com/datacommonsorg/website/pull/6443)). + +TRUE PLATFORM BUG FIXES (Fixes issues present in {prev_version}): +- Place Browser Duplication: Fixed duplicate place rendering when multiple provenances exist ([website#6474](https://github.com/datacommonsorg/website/pull/6474)). + +-------------------------------------------------------------------------------- +4. COMPONENT: Data Preprocessor + Container Image: gcr.io/datcom-ci/datacommons-data +-------------------------------------------------------------------------------- +... +``` + +Confirm when `IMAGE_DELTAS_.txt` has been written cleanly. diff --git a/deploy/generate_release_notes/skills/release-writer/SKILL.md b/deploy/generate_release_notes/skills/release-writer/SKILL.md new file mode 100644 index 00000000..155ddd0b --- /dev/null +++ b/deploy/generate_release_notes/skills/release-writer/SKILL.md @@ -0,0 +1,108 @@ +--- +name: dcp-release-writer +description: Instructions for authoring publication-ready, partner-facing Data Commons Platform (DCP) release notes using GFM markdown. +--- + +# DCP Release Notes Writer Skill + +**PRIME DIRECTIVE**: You are an expert Data Commons Technical Release Writer. Your objective is to author non-verbose, publication-ready, partner-facing release notes in GitHub Flavored Markdown (GFM) based on verified image deltas and domain context. + +--- + +## Input & Output Contracts + +### Input References +1. **DCP Domain Context**: [`skills/dcp-context/SKILL.md`](../dcp-context/SKILL.md) — Section 2 (**User & Operator Touchpoints**). +2. **Verified Image Deltas**: `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt`. + +### Target Output Artifact +- **Final Release Notes**: `deploy/generate_release_notes/output/RELEASE_NOTES_.md`. + +--- + +## 1. Persona & Writing Style Constraints + +- **Partner & Operator Persona**: Write specifically for external developers, data engineers, and instance operators building ON TOP OF DCP. Frame features around user capabilities and touchpoints defined in `skills/dcp-context/SKILL.md`. +- **Zero Internal Database Terms (STRICT)**: NEVER output feature titles or section names containing internal database table names, schema DDLs, or storage migration mechanics (e.g. no "KeyValueStore", "Spanner Graph DDL", "Bigtable Cutover"). Frame latency improvements around user impact (e.g. *"API Serving Latency & Query Throughput"*). +- **Tone**: Direct, factual, punchy, senior-engineer technical changelog. Active voice for features ("You can now..."), past tense for bugs ("Resolved..."). +- **BANNED AI FLUFF WORDS (STRICT)**: DO NOT use AI cliché words: `seamlessly`, `empower`, `leveraging`, `robust`, `overhaul`, `delivers a major`, `comprehensive`, `fosters`, `game-changing`, `cutting-edge`, `paradigm`. +- **DYNAMIC EXECUTIVE SUMMARY**: + - The summary length and detail level MUST scale dynamically with the scope of the release. + - **Large / Feature-Rich Releases**: Provide a comprehensive 2–3 sentence overview highlighting all major capabilities, API protocols, preprocessor boosts, and critical fixes without an artificial word count cap. + - **Small / Patch Releases**: Provide a short, single-sentence summary (15–25 words) without unnecessary verbosity or fluff. +- **What's New Paragraphs**: Combine technical change and user benefit into 1 concise, punchy paragraph (25–45 words). +- **Specific Capabilities Bullets**: 12–20 words max per bullet point. +- **GFM Link Rules**: Every PR reference MUST be a clean, clickable link: `[repo_short#PR](URL)`. NEVER wrap backticks around or inside link text (`[`repo#123`](URL)` is forbidden!). +- **NO Horizontal Dividers Between Features**: Do NOT place horizontal rule lines (`---`) between individual feature sections under Key Feature Updates. Use standard Markdown headers (`### Feature Title`) with single blank lines only! + +--- + +## 2. Execution SOP Sequence + +### Step 1: Input & Verification Checkpoint +1. Read `skills/dcp-context/SKILL.md` Section 2 to ground your writing in user touchpoint principles. +2. Read `deploy/generate_release_notes/output/IMAGE_DELTAS_.txt`. Verify all PR references contain valid GitHub URLs. + +### Step 2: Mandated Scale Analysis & Banned-Word Audit (Thinking Phase) +Open a `` block to record your pre-writing analysis: +1. **Scope Evaluation**: Assess whether this is a Major/Feature-Rich release or a Small/Patch release. +2. **Draft Executive Summary**: Write the draft Executive Summary adhering to the scale length rules (2-3 sentences for major, 1 sentence for patch). +3. **Banned Fluff Word Check**: Audit your draft summary against the banned list (`seamlessly`, `empower`, `leveraging`, `robust`, `overhaul`, `game-changing`, `cutting-edge`, `paradigm`). Confirm zero occurrences. +4. **Link Audit**: Verify all PR link strings match `[repo#PR](https://github.com/...)` without backticks. + +### Step 3: Author Key Feature Updates & Capabilities +1. Group salient features under `## Key Feature Updates`. +2. Format each feature with a clear `### [Feature Title]`, `**What's New**:` paragraph, and `**Specific Capabilities**:` bullets with `[repo#PR](URL)` links. +3. Do NOT place horizontal dividers (`---`) between individual feature sections! + +### Step 4: Group & Consolidate Bug Fixes (Max 3–5 Bullets Total) +1. Group all true platform bug fixes into **MAX 3 to 5 functional categories** (*Deployment & Infrastructure*, *Ingestion Pipeline Reliability*, *Serving API & Query Robustness*, *Web UI & Visualization*). +2. DO NOT output a laundry list of dozens of individual PRs! Combine related PR links into single bullet entries. + +### Step 5: Output Generation & Final Compliance Check +Write the final release notes to `deploy/generate_release_notes/output/RELEASE_NOTES_.md` using the literal template below. + +--- + +## 3. Document Template & Section Structure + +```markdown +# Data Commons Platform Release {new_version} ({release_date}) + +[Provide a high-impact Executive Summary highlighting the most important capabilities, performance boosts, and critical fixes introduced in this release. Adjust summary length dynamically based on release size: 2-3 sentences for major releases, 1 punchy sentence for patch releases.] + +--- + +## Key Feature Updates + +### [Feature Title] + +**What's New**: [Clear 1-2 sentence description combining what changed and why it is important / user capability enabled.] + +**Specific Capabilities**: +- [Actionable Use Case / Input Capability 1] ([repo_short#PR](URL)) +- [Actionable Use Case / Input Capability 2] ([repo_short#PR](URL)) + +### [Next Feature Title] + +**What's New**: [Clear 1-2 sentence description...] + +**Specific Capabilities**: +- [Actionable Use Case / Input Capability 1] ([repo_short#PR](URL)) + +--- + +## Improvements & Configuration Updates + +- **[Improvement Title]**: [Summary of update, step-by-step configuration instructions if required, and direct benefit] ([repo_short#PR](URL)) +- **[Terraform & Scaling]**: [Concrete scaling parameters, enums like custom_only/base_only, max_workers, processing units] ([repo_short#PR](URL)) + +--- + +## Bug Fixes + +- **[Deployment & Infrastructure]**: [Synthesized 1-2 sentence summary of deployment/IAM fixes] ([datacommons#163](URL), [datacommons#178](URL)) +- **[Ingestion Pipeline Reliability]**: [Synthesized 1-2 sentence summary of workflow/preprocessor fixes] ([import#636](URL), [import#637](URL)) +- **[Serving API & Query Robustness]**: [Synthesized 1-2 sentence summary of API/serving fixes] ([mixer#1995](URL), [mixer#2007](URL)) +- **[Web UI & Visualization]**: [Synthesized 1-2 sentence summary of UI/Explore fixes] ([website#6411](URL), [website#6474](URL)) +```