Repository navigation
Add agent-experience-eval npm package and GitHub Action - #47785
manzoorwanijk wants to merge 29 commits into
Conversation
|
Thank you for your PR! When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:
This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖 Follow this PR Review Process:
If you have questions about anything, reach out in #jetpack-developers for guidance! |
Code Review2 blockers, 6 important, 6 suggestions Blockers
Important
Suggestions
Simplification
|
There was a problem hiding this comment.
Pull request overview
Adds a new composite GitHub Action under projects/github-actions/agent-experience-eval to score a repository’s “AI agent experience” (instruction files quality) by running the Claude Code CLI with a fixed rubric/prompt, plus a workflow and local helper script to run it and collect a JSON report.
Changes:
- Introduce the
agent-experience-evalcomposite action (prompt, runner script, docs, changelog scaffolding). - Add a scheduled/PR/push GitHub workflow to execute the action and upload the resulting JSON report as an artifact.
- Add a local
tools/score-agent-experience.shhelper and register the new project in the maintenance-commit lister allowlist.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
tools/score-agent-experience.sh |
Local CLI wrapper that runs the action’s shared run.sh and prints a score breakdown + progress bars. |
tools/list-repo-maintenance-commits.sh |
Adds github-actions/agent-experience-eval to the “maybe maintenance” projects allowlist. |
projects/github-actions/agent-experience-eval/run.sh |
Core logic: runs Claude CLI, extracts JSON, performs basic validation, and optionally writes GitHub Action outputs. |
projects/github-actions/agent-experience-eval/prompt.md |
Defines the rubric and the required JSON schema for the evaluator. |
projects/github-actions/agent-experience-eval/composer.json |
Adds standard project metadata and changelogger dev dependency for release tooling. |
projects/github-actions/agent-experience-eval/changelog/initial-release |
Changelog fragment for the initial release. |
projects/github-actions/agent-experience-eval/changelog/.gitkeep |
Keeps the changelog directory present. |
projects/github-actions/agent-experience-eval/action.yml |
Composite action definition wiring inputs, env, and step outputs. |
projects/github-actions/agent-experience-eval/README.md |
Usage/docs for the action, schema example, and prerequisites. |
projects/github-actions/agent-experience-eval/CHANGELOG.md |
Initializes changelog file. |
projects/github-actions/agent-experience-eval/.gitignore |
Ignores vendor and composer.lock for this action project. |
projects/github-actions/agent-experience-eval/.gitattributes |
Excludes changelog fragments from production builds. |
.github/workflows/agent-experience-eval.yml |
Workflow to run the evaluation on schedule/push (and temporarily PR), then upload artifact. |
| jq -r '.criteria | to_entries[] | "\(.key)\t\(.value.score)\t\(.value.max)"' "$OUTPUT" | | ||
| while IFS=$'\t' read -r key score max; do | ||
| label=$(echo "$key" | tr '_' ' ' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) substr($i,2)}1') | ||
| if (( max > 0 )); then | ||
| filled=$(( score * 20 / max )) | ||
| else | ||
| filled=0 | ||
| fi |
There was a problem hiding this comment.
The summary rendering assumes .criteria.*.score and .criteria.*.max are numeric; if Claude returns strings (or missing fields), the arithmetic (( max > 0 )) / $(( score * 20 / max )) will error under set -euo pipefail and the script will exit. Either hard-validate the criteria schema before this loop (preferred, possibly in run.sh) or add defensive parsing/defaulting here.
| pull_request: # TODO: Remove after initial testing | ||
| paths: | ||
| - 'projects/github-actions/agent-experience-eval/**' | ||
| - '.github/workflows/agent-experience-eval.yml' | ||
| - 'tools/score-agent-experience.sh' | ||
| push: |
There was a problem hiding this comment.
This workflow runs on pull_request and uses secrets.ANTHROPIC_API_KEY. For PRs from forks, secrets are not provided and the job will fail, creating noisy red checks for external contributors. Consider gating the job/steps with a head.repo.full_name == base.repo.full_name condition (pattern used in other workflows) or removing the pull_request trigger once initial testing is done.
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Install Claude Code | ||
| run: npm install -g @anthropic-ai/claude-code | ||
|
|
||
| - name: Evaluate Agent Experience | ||
| uses: ./projects/github-actions/agent-experience-eval | ||
| with: | ||
| anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }} | ||
|
|
||
| - name: Upload evaluation artifact | ||
| uses: actions/upload-artifact@v4 | ||
| with: |
There was a problem hiding this comment.
This workflow uses older action versions (actions/checkout@v4 and actions/upload-artifact@v4) while the rest of the repo appears to have standardized on newer major versions (e.g., checkout@v6 and upload-artifact@v7). Aligning versions reduces maintenance overhead and keeps behavior consistent across workflows.
| with: | ||
| name: agent-experience-eval | ||
| path: agent-experience-eval.json | ||
| retention-days: 30 |
There was a problem hiding this comment.
In addition to updating the version of the action as the AI suggested, you might consider setting archive: false since you're uploading a single file. That avoids GitHub wrapping the .json file in a .zip.
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Install Claude Code |
There was a problem hiding this comment.
Is there a reason not to put the "Install Claude" and "Upload artifact" steps into projects/github-actions/agent-experience-eval/action.yml too, to save more boilerplate?
| ANTHROPIC_API_KEY: ${{ inputs.anthropic_api_key }} | ||
| MAX_TURNS: ${{ inputs.max_turns }} | ||
| OUTPUT_PATH: ${{ inputs.output_path }} | ||
| ACTION_PATH: ${{ github.action_path }} |
There was a problem hiding this comment.
GitHub already provides GITHUB_ACTION_PATH, you could use that.
| OUTPUT_PATH: ${{ inputs.output_path }} | ||
| ACTION_PATH: ${{ github.action_path }} | ||
| run: | | ||
| PROMPT=$(cat "$ACTION_PATH/prompt.md") |
There was a problem hiding this comment.
Could simplify this:
| PROMPT=$(cat "$ACTION_PATH/prompt.md") | |
| PROMPT=$(< "$ACTION_PATH/prompt.md") |
| - The "score" field must equal the sum of all criteria scores. | ||
| - The "grade" must match the score: A (90-100), B (70-89), C (50-69), D (30-49), F (0-29). | ||
| - All file paths in "files_found" must be relative to the repo root (no absolute paths). | ||
| - Output ONLY the JSON object. No text before or after. |
There was a problem hiding this comment.
Could you make use of https://platform.claude.com/docs/en/agent-sdk/structured-outputs to better ensure it really does output only JSON using the correct schema? That might be more reliable than including repeated instructions in the prompt.
|
|
||
| if ! RAW=$(claude --print --output-format text --max-turns "$MAX_TURNS" "$PROMPT" < /dev/null 2>"$CLAUDE_STDERR"); then | ||
| echo "ERROR: Claude CLI failed:" >&2 | ||
| head -c 1000 "$CLAUDE_STDERR" >&2 |
There was a problem hiding this comment.
Why capture and print only the first 1000 bytes of the error output? Why not just let it all go into GitHub's log?
| fi | ||
|
|
||
| # Extract JSON: direct parse, code fences, or brace extraction | ||
| if echo "$RAW" | jq . > "$OUTPUT_PATH" 2>/dev/null; then |
There was a problem hiding this comment.
Since this uses bash, you could simplify this.
| if echo "$RAW" | jq . > "$OUTPUT_PATH" 2>/dev/null; then | |
| if jq . <<<"$RAW" > "$OUTPUT_PATH" 2>/dev/null; then |
| elif FENCED=$(echo "$RAW" | sed -En '/^```(json)?$/,/^```$/{ /^```/d; p; }') && \ | ||
| [[ -n "$FENCED" ]] && echo "$FENCED" | jq . > "$OUTPUT_PATH" 2>/dev/null; then | ||
| : | ||
| elif BRACED=$(echo "$RAW" | sed -n '/^{/,/^}/p') && \ | ||
| [[ -n "$BRACED" ]] && echo "$BRACED" | jq . > "$OUTPUT_PATH" 2>/dev/null; then | ||
| : |
There was a problem hiding this comment.
The --json-schema mode should let you drop this stuff too.
| if [[ "$SCORE" == "null" || -z "$SCORE" ]]; then | ||
| echo "ERROR: JSON missing 'score' field." >&2; exit 1 | ||
| fi | ||
| if [[ "$GRADE" == "null" || -z "$GRADE" ]]; then | ||
| echo "ERROR: JSON missing 'grade' field." >&2; exit 1 | ||
| fi | ||
| if ! [[ "$SCORE" =~ ^[0-9]+$ ]]; then | ||
| echo "ERROR: Invalid score: $SCORE" >&2; exit 1 | ||
| fi | ||
| if (( SCORE > 100 )); then | ||
| echo "ERROR: Score $SCORE exceeds maximum 100." >&2; exit 1 | ||
| fi | ||
| if ! [[ "$GRADE" =~ ^[ABCDF]$ ]]; then | ||
| echo "ERROR: Invalid grade: $GRADE" >&2; exit 1 | ||
| fi |
There was a problem hiding this comment.
If you can write the schema strictly enough, you could probably omit all of this too.
| GRADE=$(jq -r '.grade' "$OUTPUT") | ||
| echo "" >&2 | ||
| echo "Score: $SCORE/100 (Grade: $GRADE)" >&2 | ||
| jq -r '.criteria | to_entries[] | "\(.key)\t\(.value.score)\t\(.value.max)"' "$OUTPUT" | |
There was a problem hiding this comment.
You might use jq's @tsv here.
98b7d33 to
a9b998b
Compare
|
Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.
Interested in more tips and information?
|
Code Coverage SummaryThis PR did not change code coverage! That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷 |
1bcfb1d to
f0b8462
Compare
|
Nice work on this, a few things worth addressing: Tests: The test file is just placeholder comments right now. Worth adding coverage for the pure functions at minimum ( Convention nits:
Defensive coding:
Minor: the "Initial version." changelog entries are valid but could say what the packages actually do. |
anomiex
left a comment
There was a problem hiding this comment.
Just took a quick look at the deps, didn't look at the gigantic pile of code added here now yet.
Wow @actions/artifact brings in a ton of extra deps.
| }, | ||
| "devDependencies": { | ||
| "@actions/artifact": "^6.2.1", | ||
| "@actions/core": "^1.11.1", |
There was a problem hiding this comment.
Why such an outdated version?
There was a problem hiding this comment.
It was probably Claude who changed it for some reason while trying to resolve the build failure. Fixed.
| "@types/jest": "30.0.0", | ||
| "@types/node": "^24.12.0", | ||
| "@typescript/native-preview": "7.0.0-dev.20260225.1", | ||
| "jest": "30.2.0", |
There was a problem hiding this comment.
The rest of the monorepo is on 30.3.0. Why not match that?
| typescript@3.9.10: | ||
| resolution: {integrity: sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==} | ||
| engines: {node: '>=4.2.0'} | ||
| hasBin: true | ||
|
|
||
| typescript@5.4.5: | ||
| resolution: {integrity: sha512-vcI4UpRgg81oIRUFwR0WSIHKt11nJ7SAVlYNIu+QpqeyXP+gpQJy/Z4+F0aGxSE4MqwjyXvW/TzgkLAx2AGHwQ==} | ||
| engines: {node: '>=14.17'} | ||
| hasBin: true |
There was a problem hiding this comment.
🙄 Really, @actions/artifact? You need two different old versions of typescript in your dependency tree?
4c9ffe4 to
7c814d0
Compare
Composite action that evaluates a repo's AI agent experience quality by running the claude-md-improver skill and producing a structured JSON report. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address all 15 code review findings: Blockers: - Add composer.json, changelog/, .gitattributes, .gitignore for monorepo integration - Rewrite CHANGELOG.md to standard Keep a Changelog format Important: - Extract shared evaluation logic into run.sh to eliminate duplication - Capture Claude CLI stderr instead of discarding with 2>/dev/null - Validate score/grade format before writing to GITHUB_OUTPUT - Check for null/missing JSON fields from jq - Scope push trigger to trunk branch only, remove pull_request trigger - Guard division by zero in progress bar rendering - Remove phantom inputs/outputs from README Suggestions: - Add permissions: contents: read to workflow - Fix -- handler to break out of arg parsing loop - Add missing-argument guards for -o and -m flags - Replace skill invocation with inline evaluation instructions - Show first 500 chars on parse failure instead of raw output - Remove contradictory "no code fences" prompt instruction
- Fix grade regex to reject invalid "E" grade (^[ABCDF]$ not ^[A-F]$) - Add score 0-100 range check - Fix GNU sed extensions that silently fail on macOS BSD sed - Replace seq-based progress bar with bash for-loop to fix boundary rendering - Add agent-experience-eval to maintenance projects list
New TypeScript package that evaluates a repository's AI agent experience quality. Uses the Anthropic SDK directly with Zod structured output. - Programmatic API via evaluate() and lower-level helpers - CLI via agent-experience-eval command - Local file discovery with guardrails (size/count limits, dedup, excludes) - Currency validation (checks referenced paths and commands exist) - Prompt builder with 180K char truncation
Convert from composite bash action to a proper TypeScript Node action that imports @automattic/jetpack-agent-experience-eval programmatically. - node24 runner, ncc-bundled dist/index.js (built during release) - Calls evaluate() API directly — no Claude CLI required - Handles artifact upload internally via @actions/artifact - Removes old run.sh and prompt.md (logic moved to npm package)
- Workflow reduced to checkout + one action step (no Claude CLI install) - Add fork PR guard and expanded trigger paths - Remove tools/score-agent-experience.sh (use the CLI directly)
The SDK's zodOutputFormat() imports from zod/v4 internally. Our schema must also use zod/v4 to produce compatible schema objects.
- Rewrite action README with usage, inputs, outputs, and rubric docs - Rewrite JS package README with CLI and programmatic API examples - Remove unused PHP config from action composer.json
- Add build steps to workflow (JS package + action must be built before the node24 action can run, since dist/ is gitignored) - Make command validation monorepo-aware: walk from instruction file dir up to repo root when searching for commands in package.json/Makefile - Add jp/jetpack to recognized command patterns - Add post-parse cross-field validation: fix score sum and grade band if Claude returns inconsistent values - Align workflow push paths with discovery patterns (use **/ prefix for dot-directories like .claude, .cursor, .codex)
- Fix npm command regex to skip flags (--filter etc.) before script name - Separate jp/jetpack CLI commands from manifest-validated commands - Reject path traversal (../) and absolute paths in reference extraction
The package is meant to be generic and open-sourceable. Repo-specific CLI tools like jp/jetpack should not be hardcoded.
New --format flag (json|human|auto, default: auto): - auto: human report on TTY, JSON when piped - With --output: JSON always written to file, human report to stdout on TTY The human report shows score/grade, per-criteria breakdown with progress bars and notes, discovered files with sizes, currency validation results (missing paths/commands), issues, recommendations, and token usage.
- Validate --format flag values (reject invalid input) - Render unknown criteria keys (future-proofing) - Scale formatBytes to MB for large files
--output now writes whatever --format resolves to: -o eval.json → JSON to file, human summary to TTY --format human -o report.txt → human report to file --format json -o eval.json → JSON to file (no --output) → output to stdout in resolved format
- Fix ReDoS: replace polynomial regex with two-pass token extraction - Fix path validation: resolve paths relative to instruction file dir first, fall back to repo root, with escape-check - Fix --output auto: writes JSON to file by default, shows human summary on TTY. Explicit --format overrides file format. - Status messages go to stderr (not stdout) - Update package README with --format docs and correct --output desc - Fix LICENSE.txt links in both READMEs (file doesn't exist)
pnpm was not available because the workflow used actions/setup-node directly instead of the repo's .github/actions/tool-setup which handles pnpm installation. Also simplified pnpm install to full workspace install (filtered installs can miss transitive deps).
- Replace ncc with tsdown (rolldown-powered) which handles ESM deps - Upgrade @actions/artifact to v6 (latest, ESM-only) - Add pnpmfile fix for missing @protobuf-ts/runtime-rpc dep in @actions/artifact (actions/toolkit#1898) - Use .github/actions/tool-setup for pnpm/node setup in workflow
- Action tsconfig: extend jetpack-js-tools/tsconfig.base.json instead of setting module/moduleResolution directly - JS package: set repository.url to mirror repo in npm canonical format - JS package composer.json: remove phpunit-select-config (not a PHP project), remove unused autoload/classmap and test-coverage script
- Add schema.test.ts with 5 tests: valid parsing, missing criteria, score bounds, invalid grade, version check - Enable --experimental-vm-modules for Jest ESM support - Remove empty scaffold test file
- tsconfig.json: includes src + tests, noEmit, with jest types (for IDE) - tsconfig.build.json: extends base, rootDir=src, outDir=build (for tsgo) - Build script now uses tsconfig.build.json
The project structure linter forbids setting noEmit directly. tsconfig.base.json already has noEmit: true which is what the root tsconfig needs (IDE type checking only). tsconfig.build.json extends tsc.json for actual compilation.
- Update @actions/core to 3.0.0 (matches rest of monorepo) - Update jest to 30.3.0 (matches rest of monorepo) - Move bin entry to bin/agent-experience-eval.js wrapper (committed file) so pnpm install doesn't warn about missing build/cli.js on clean checkout
7c814d0 to
b0b3144
Compare
|
This PR has been marked as stale. This happened because:
If this PR is still useful, please do a [trunk merge or rebase](https://github.com/Automattic/jetpack/blob/trunk/docs/git-workflow.md#keeping-your-branch-up-to-date) and otherwise make sure it's up to date and has clear testing instructions. You may also want to ping possible reviewers in case they've forgotten about it. Please close this PR if you think it's not valid anymore — if you do, please add a brief explanation. If the PR is not updated (or at least commented on) in another month, it will be automatically closed. |
|
This PR has been automatically closed as it has not been updated in some time. If you want to resume work on the PR, feel free to restore the branch and reopen the PR. |
Proposed changes
Restructures agent-experience-eval from a composite bash GitHub Action (shelling out to Claude CLI) into two proper packages:
@automattic/jetpack-agent-experience-eval— npm package with programmatic API + CLI. Uses the Anthropic SDK directly with Zod structured output. No Claude CLI dependency.projects/github-actions/agent-experience-eval— Node-based GitHub Action (node24, ncc-bundled) that imports the package, callsevaluate(), and handles artifact upload via@actions/artifact.Key improvements over the original implementation
messages.parse()+zodOutputFormat()for structured outputevaluate(),discoverFiles(),validateCurrency(),buildPrompt()are all importable for custom workflows--format humanmode with progress bars, criteria notes, currency validation results, issues, and recommendationsOther information
Related product discussion/links
Does this pull request change what data or activity we track or use?
No.
Testing instructions
Prerequisites
ANTHROPIC_API_KEYenv var)Test the CLI locally
Verify the output
Test the GitHub Action (CI)
The workflow runs on:
For this PR, verify:
agent-experience-eval