Skip to content

Add agent-experience-eval npm package and GitHub Action - #47785

Closed
manzoorwanijk wants to merge 29 commits into
trunkfrom
add/agent-experience-eval
Closed

manzoorwanijk wants to merge 29 commits into
trunkfrom
add/agent-experience-eval

Conversation

@manzoorwanijk

@manzoorwanijk manzoorwanijk commented Mar 25, 2026 •

Copy link
Copy Markdown
Member

Proposed changes

Restructures agent-experience-eval from a composite bash GitHub Action (shelling out to Claude CLI) into two proper packages:

  1. @automattic/jetpack-agent-experience-eval — npm package with programmatic API + CLI. Uses the Anthropic SDK directly with Zod structured output. No Claude CLI dependency.
  2. projects/github-actions/agent-experience-eval — Node-based GitHub Action (node24, ncc-bundled) that imports the package, calls evaluate(), and handles artifact upload via @actions/artifact.

Key improvements over the original implementation

  • No Claude CLI dependency — Uses the Anthropic TypeScript SDK directly with messages.parse() + zodOutputFormat() for structured output
  • Programmatic API — evaluate(), discoverFiles(), validateCurrency(), buildPrompt() are all importable for custom workflows
  • Smart file discovery — Finds AI instruction files locally with guardrails (100KB max size, 200 file cap, excludes node_modules/.git, dedup, stable sort)
  • Currency validation — Cross-references paths and commands in instruction files against the actual codebase (monorepo-aware, walks up to repo root)
  • Zod schema — Single source of truth for types, validation, and API schema (using zod/v4 for SDK compatibility)
  • Human-readable output — --format human mode with progress bars, criteria notes, currency validation results, issues, and recommendations
  • Post-parse validation — Corrects score sum and grade band if Claude returns inconsistent values
  • Self-contained action — One step: checkout + action. Handles install, evaluation, and artifact upload internally
  • Security — ReDoS-safe regex, path traversal prevention, fork PR guard

Other information

  • Generate changelog entries for this PR (using AI).

Related product discussion/links

  • P2 post: peGwbA-4S1-p2
  • AI Enablement Cohort 2 project

Does this pull request change what data or activity we track or use?

No.

Testing instructions

Prerequisites

  • Node.js >= 22
  • An Anthropic API key (ANTHROPIC_API_KEY env var)

Test the CLI locally

# Build the package
pnpm --filter @automattic/jetpack-agent-experience-eval build

# Run with human-readable output (default on terminal)
ANTHROPIC_API_KEY=your-key node projects/js-packages/agent-experience-eval/build/cli.js

# Run with JSON output to file + human summary to terminal
ANTHROPIC_API_KEY=your-key node projects/js-packages/agent-experience-eval/build/cli.js -o eval.json

# Run with explicit JSON to stdout (for piping)
ANTHROPIC_API_KEY=your-key node projects/js-packages/agent-experience-eval/build/cli.js --format json | jq .result.score

# Run against a different repo
ANTHROPIC_API_KEY=your-key node projects/js-packages/agent-experience-eval/build/cli.js --repo /path/to/other/repo

Verify the output

  • Human output shows score, grade, per-criteria breakdown with progress bars and notes
  • Human output shows discovered files with sizes
  • Human output shows currency validation (missing paths/commands)
  • Human output shows issues and recommendations
  • JSON output has valid structure matching the Zod schema (version, score, grade, files_found, criteria, issues, recommendations)
  • Score equals sum of all criteria scores
  • Grade matches score band (A: 90-100, B: 70-89, C: 50-69, D: 30-49, F: 0-29)

Test the GitHub Action (CI)

The workflow runs on:

  • Push to trunk (when AI instruction files change)
  • Pull request (when action/package files change)
  • Weekly schedule (Monday 6am UTC)
  • Manual dispatch

For this PR, verify:

  • The workflow builds both packages before running the action
  • The action produces an artifact named agent-experience-eval
  • Fork PRs are skipped (no secret available)

@github-actions github-actions Bot added [Status] In Progress Actions GitHub actions used to automate some of the work around releases and repository management Docs [Action] Agent Experience Eval labels Mar 25, 2026
@github-actions

github-actions Bot commented Mar 25, 2026 •

Copy link
Copy Markdown
Contributor

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

  • ✅ Include a description of your PR changes.
  • ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  • ✅ Add testing instructions.
  • ✅ Specify whether this PR includes any changes to data or privacy.
  • ✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖


Follow this PR Review Process:

  1. Ensure all required checks appearing at the bottom of this PR are passing.
  2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
  3. You can use GitHub's Reviewers functionality to request a review.
  4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

@github-actions github-actions Bot added the [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. label Mar 25, 2026
@LiamSarsfield

Copy link
Copy Markdown
Contributor

Code Review

2 blockers, 6 important, 6 suggestions

Blockers

  1. Missing composer.json and changelog/ directory. Every other action in projects/github-actions/ has both. Required for changelogger, autotagger, and mirror-repo. Scaffold with jp generate github-action or copy from push-to-mirrors/.

  2. Wrong CHANGELOG.md format. Hand written format will break the release pipeline. Delete it and let changelogger manage it after adding composer.json.

Important

  1. Silent CLI failures (action.yml:43, score-agent-experience.sh:41). 2>/dev/null combined with set -e means claude errors crash with zero diagnostics. Capture stderr instead: RAW=$(claude ... 2>&1) || { echo "::error::Claude CLI failed: ${RAW:0:500}"; exit 1; }.

  2. GH Actions output injection (action.yml:66-67). LLM controlled SCORE/GRADE written unsanitized to $GITHUB_OUTPUT. Validate before writing: [[ "$SCORE" =~ ^[0-9]+$ ]] and [[ "$GRADE" =~ ^[A-F]$ ]].

  3. No jq output validation (action.yml:62-63). Missing JSON fields produce "null" strings silently. Combine with Merge the changes in 2.7.1 upstream to master. #4 validation.

  4. Unscoped push trigger (workflow L3-10). No branches filter so this fires on every branch. Add branches: ['trunk'].

  5. Division by zero (score-agent-experience.sh:66). max: 0 crashes the progress bar. Guard it: filled=$(( max > 0 ? score * 20 / max : 0 )).

  6. README documents phantom inputs/outputs. Three inputs (upload_artifact, artifact_name, artifact_retention_days) and one output (has_ai_files) don't exist in action.yml. Remove until implemented.

Suggestions

  1. Add permissions: contents: read to the workflow.
  2. -- handler needs a break (score-agent-experience.sh:20).
  3. -o/-m need missing argument guards before accessing $2.
  4. Prompt depends on /claude-md-management:claude-md-improver skill which may not be available in CI.
  5. Raw LLM output gets echoed to logs on parse failure. Mask or redirect to artifact.
  6. Prompt says "no code fences" but code handles fences anyway. Remove the contradictory instruction.

Simplification

  1. Duplicated JSON extraction. Same 13 line parsing block in both action.yml and score-agent-experience.sh. Extract into a shared script (see the pr-is-up-to-date pattern).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new composite GitHub Action under projects/github-actions/agent-experience-eval to score a repository’s “AI agent experience” (instruction files quality) by running the Claude Code CLI with a fixed rubric/prompt, plus a workflow and local helper script to run it and collect a JSON report.

Changes:

  • Introduce the agent-experience-eval composite action (prompt, runner script, docs, changelog scaffolding).
  • Add a scheduled/PR/push GitHub workflow to execute the action and upload the resulting JSON report as an artifact.
  • Add a local tools/score-agent-experience.sh helper and register the new project in the maintenance-commit lister allowlist.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tools/score-agent-experience.sh Local CLI wrapper that runs the action’s shared run.sh and prints a score breakdown + progress bars.
tools/list-repo-maintenance-commits.sh Adds github-actions/agent-experience-eval to the “maybe maintenance” projects allowlist.
projects/github-actions/agent-experience-eval/run.sh Core logic: runs Claude CLI, extracts JSON, performs basic validation, and optionally writes GitHub Action outputs.
projects/github-actions/agent-experience-eval/prompt.md Defines the rubric and the required JSON schema for the evaluator.
projects/github-actions/agent-experience-eval/composer.json Adds standard project metadata and changelogger dev dependency for release tooling.
projects/github-actions/agent-experience-eval/changelog/initial-release Changelog fragment for the initial release.
projects/github-actions/agent-experience-eval/changelog/.gitkeep Keeps the changelog directory present.
projects/github-actions/agent-experience-eval/action.yml Composite action definition wiring inputs, env, and step outputs.
projects/github-actions/agent-experience-eval/README.md Usage/docs for the action, schema example, and prerequisites.
projects/github-actions/agent-experience-eval/CHANGELOG.md Initializes changelog file.
projects/github-actions/agent-experience-eval/.gitignore Ignores vendor and composer.lock for this action project.
projects/github-actions/agent-experience-eval/.gitattributes Excludes changelog fragments from production builds.
.github/workflows/agent-experience-eval.yml Workflow to run the evaluation on schedule/push (and temporarily PR), then upload artifact.

Comment thread tools/score-agent-experience.sh Outdated
Comment on lines +56 to +63
jq -r '.criteria | to_entries[] | "\(.key)\t\(.value.score)\t\(.value.max)"' "$OUTPUT" |
while IFS=$'\t' read -r key score max; do
label=$(echo "$key" | tr '_' ' ' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) substr($i,2)}1')
if (( max > 0 )); then
filled=$(( score * 20 / max ))
else
filled=0
fi

Copilot AI Mar 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The summary rendering assumes .criteria.*.score and .criteria.*.max are numeric; if Claude returns strings (or missing fields), the arithmetic (( max > 0 )) / $(( score * 20 / max )) will error under set -euo pipefail and the script will exit. Either hard-validate the criteria schema before this loop (preferred, possibly in run.sh) or add defensive parsing/defaulting here.

Copilot uses AI. Check for mistakes.
Comment on lines +6 to +11
pull_request: # TODO: Remove after initial testing
paths:
- 'projects/github-actions/agent-experience-eval/**'
- '.github/workflows/agent-experience-eval.yml'
- 'tools/score-agent-experience.sh'
push:

Copilot AI Mar 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow runs on pull_request and uses secrets.ANTHROPIC_API_KEY. For PRs from forks, secrets are not provided and the job will fail, creating noisy red checks for external contributors. Consider gating the job/steps with a head.repo.full_name == base.repo.full_name condition (pattern used in other workflows) or removing the pull_request trigger once initial testing is done.

Copilot uses AI. Check for mistakes.
Comment on lines +40 to +52
- uses: actions/checkout@v4

- name: Install Claude Code
run: npm install -g @anthropic-ai/claude-code

- name: Evaluate Agent Experience
uses: ./projects/github-actions/agent-experience-eval
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}

- name: Upload evaluation artifact
uses: actions/upload-artifact@v4
with:

Copilot AI Mar 26, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow uses older action versions (actions/checkout@v4 and actions/upload-artifact@v4) while the rest of the repo appears to have standardized on newer major versions (e.g., checkout@v6 and upload-artifact@v7). Aligning versions reduces maintenance overhead and keeps behavior consistent across workflows.

Copilot uses AI. Check for mistakes.
with:
name: agent-experience-eval
path: agent-experience-eval.json
retention-days: 30

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to updating the version of the action as the AI suggested, you might consider setting archive: false since you're uploading a single file. That avoids GitHub wrapping the .json file in a .zip.

https://github.blog/changelog/2026-02-26-github-actions-now-supports-uploading-and-downloading-non-zipped-artifacts/

steps:
- uses: actions/checkout@v4

- name: Install Claude Code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason not to put the "Install Claude" and "Upload artifact" steps into projects/github-actions/agent-experience-eval/action.yml too, to save more boilerplate?

ANTHROPIC_API_KEY: ${{ inputs.anthropic_api_key }}
MAX_TURNS: ${{ inputs.max_turns }}
OUTPUT_PATH: ${{ inputs.output_path }}
ACTION_PATH: ${{ github.action_path }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OUTPUT_PATH: ${{ inputs.output_path }}
ACTION_PATH: ${{ github.action_path }}
run: |
PROMPT=$(cat "$ACTION_PATH/prompt.md")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could simplify this:

Suggested change
PROMPT=$(cat "$ACTION_PATH/prompt.md")
PROMPT=$(< "$ACTION_PATH/prompt.md")

- The "score" field must equal the sum of all criteria scores.
- The "grade" must match the score: A (90-100), B (70-89), C (50-69), D (30-49), F (0-29).
- All file paths in "files_found" must be relative to the repo root (no absolute paths).
- Output ONLY the JSON object. No text before or after.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you make use of https://platform.claude.com/docs/en/agent-sdk/structured-outputs to better ensure it really does output only JSON using the correct schema? That might be more reliable than including repeated instructions in the prompt.


if ! RAW=$(claude --print --output-format text --max-turns "$MAX_TURNS" "$PROMPT" < /dev/null 2>"$CLAUDE_STDERR"); then
echo "ERROR: Claude CLI failed:" >&2
head -c 1000 "$CLAUDE_STDERR" >&2

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why capture and print only the first 1000 bytes of the error output? Why not just let it all go into GitHub's log?

fi

# Extract JSON: direct parse, code fences, or brace extraction
if echo "$RAW" | jq . > "$OUTPUT_PATH" 2>/dev/null; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this uses bash, you could simplify this.

Suggested change
if echo "$RAW" | jq . > "$OUTPUT_PATH" 2>/dev/null; then
if jq . <<<"$RAW" > "$OUTPUT_PATH" 2>/dev/null; then

Comment on lines +30 to +35
elif FENCED=$(echo "$RAW" | sed -En '/^```(json)?$/,/^```$/{ /^```/d; p; }') && \
[[ -n "$FENCED" ]] && echo "$FENCED" | jq . > "$OUTPUT_PATH" 2>/dev/null; then
:
elif BRACED=$(echo "$RAW" | sed -n '/^{/,/^}/p') && \
[[ -n "$BRACED" ]] && echo "$BRACED" | jq . > "$OUTPUT_PATH" 2>/dev/null; then
:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --json-schema mode should let you drop this stuff too.

Comment on lines +46 to +60
if [[ "$SCORE" == "null" || -z "$SCORE" ]]; then
echo "ERROR: JSON missing 'score' field." >&2; exit 1
fi
if [[ "$GRADE" == "null" || -z "$GRADE" ]]; then
echo "ERROR: JSON missing 'grade' field." >&2; exit 1
fi
if ! [[ "$SCORE" =~ ^[0-9]+$ ]]; then
echo "ERROR: Invalid score: $SCORE" >&2; exit 1
fi
if (( SCORE > 100 )); then
echo "ERROR: Score $SCORE exceeds maximum 100." >&2; exit 1
fi
if ! [[ "$GRADE" =~ ^[ABCDF]$ ]]; then
echo "ERROR: Invalid grade: $GRADE" >&2; exit 1
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you can write the schema strictly enough, you could probably omit all of this too.

Comment thread tools/score-agent-experience.sh Outdated
GRADE=$(jq -r '.grade' "$OUTPUT")
echo "" >&2
echo "Score: $SCORE/100 (Grade: $GRADE)" >&2
jq -r '.criteria | to_entries[] | "\(.key)\t\(.value.score)\t\(.value.max)"' "$OUTPUT" |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might use jq's @tsv here.

@manzoorwanijk manzoorwanijk changed the title WIP: Add agent-experience-eval GitHub Action Add agent-experience-eval npm package and restructure GitHub Action Mar 31, 2026
@manzoorwanijk
manzoorwanijk force-pushed the add/agent-experience-eval branch from 98b7d33 to a9b998b Compare March 31, 2026 19:42
@manzoorwanijk manzoorwanijk changed the title Add agent-experience-eval npm package and restructure GitHub Action Add agent-experience-eval npm package and GitHub Action Mar 31, 2026
@github-actions

github-actions Bot commented Mar 31, 2026 •

Copy link
Copy Markdown
Contributor

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

  • To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the add/agent-experience-eval branch.
  • To test on Simple, run the following command on your sandbox:
bin/jetpack-downloader test jetpack add/agent-experience-eval
bin/jetpack-downloader test jetpack-mu-wpcom-plugin add/agent-experience-eval

Interested in more tips and information?

  • In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
  • Read more about our development workflow here: PCYsg-eg0-p2
  • Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

@jp-launch-control

jp-launch-control Bot commented Mar 31, 2026 •

Copy link
Copy Markdown

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

@manzoorwanijk
manzoorwanijk force-pushed the add/agent-experience-eval branch from 1bcfb1d to f0b8462 Compare March 31, 2026 20:06
@LiamSarsfield

Copy link
Copy Markdown
Contributor

Nice work on this, a few things worth addressing:

Tests: The test file is just placeholder comments right now. Worth adding coverage for the pure functions at minimum (extractReferencedPaths, extractReferencedCommands, progressBar, buildPrompt) since there's a decent amount of regex parsing and output logic.

Convention nits:

  • timeout-minutes: 10 needs the date stamped comment that all other workflows have (e.g. # 2026-03-31: Evaluation typically takes 2-3 minutes)
  • Changelog entry in restructure-as-node-action uses past tense ("Restructured") — should be imperative ("Restructure")
  • The JS package composer.json has autoload.classmap pointing at TypeScript files and phpunit-select-config in require dev. Neither makes sense for a pure TS package, peer packages don't have these

Defensive coding:

  • validator.ts:137 interpolates a command name into new RegExp() without escaping. Currently safe because the extraction regex constrains the input, but that's a fragile coupling. Quick fix with a regex escape helper
  • progressBar in human-output.ts will throw a RangeError if an LLM returns a criterion score exceeding its max (negative arg to String.repeat). A Math.min clamp would prevent this
  • Related: criterionSchema has z.number().min(0) but no upper bound, which is the root cause. Adding a .refine() for score <= max would fix both the progress bar issue and the score exceeding 100 in evaluator.ts

Minor: the "Initial version." changelog entries are valid but could say what the packages actually do.

@anomiex anomiex left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just took a quick look at the deps, didn't look at the gigantic pile of code added here now yet.

Wow @actions/artifact brings in a ton of extra deps.

},
"devDependencies": {
"@actions/artifact": "^6.2.1",
"@actions/core": "^1.11.1",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why such an outdated version?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was probably Claude who changed it for some reason while trying to resolve the build failure. Fixed.

"@types/jest": "30.0.0",
"@types/node": "^24.12.0",
"@typescript/native-preview": "7.0.0-dev.20260225.1",
"jest": "30.2.0",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest of the monorepo is on 30.3.0. Why not match that?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks

Comment thread pnpm-lock.yaml
Comment on lines +18146 to +18154
typescript@3.9.10:
resolution: {integrity: sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==}
engines: {node: '>=4.2.0'}
hasBin: true

typescript@5.4.5:
resolution: {integrity: sha512-vcI4UpRgg81oIRUFwR0WSIHKt11nJ7SAVlYNIu+QpqeyXP+gpQJy/Z4+F0aGxSE4MqwjyXvW/TzgkLAx2AGHwQ==}
engines: {node: '>=14.17'}
hasBin: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙄 Really, @actions/artifact? You need two different old versions of typescript in your dependency tree?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then it misses the obvious ones 😄

@manzoorwanijk
manzoorwanijk force-pushed the add/agent-experience-eval branch from 4c9ffe4 to 7c814d0 Compare April 1, 2026 19:09
manzoorwanijk and others added 4 commits April 3, 2026 09:51
Composite action that evaluates a repo's AI agent experience quality
by running the claude-md-improver skill and producing a structured
JSON report.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address all 15 code review findings:

Blockers:
- Add composer.json, changelog/, .gitattributes, .gitignore for monorepo integration
- Rewrite CHANGELOG.md to standard Keep a Changelog format

Important:
- Extract shared evaluation logic into run.sh to eliminate duplication
- Capture Claude CLI stderr instead of discarding with 2>/dev/null
- Validate score/grade format before writing to GITHUB_OUTPUT
- Check for null/missing JSON fields from jq
- Scope push trigger to trunk branch only, remove pull_request trigger
- Guard division by zero in progress bar rendering
- Remove phantom inputs/outputs from README

Suggestions:
- Add permissions: contents: read to workflow
- Fix -- handler to break out of arg parsing loop
- Add missing-argument guards for -o and -m flags
- Replace skill invocation with inline evaluation instructions
- Show first 500 chars on parse failure instead of raw output
- Remove contradictory "no code fences" prompt instruction
- Fix grade regex to reject invalid "E" grade (^[ABCDF]$ not ^[A-F]$)
- Add score 0-100 range check
- Fix GNU sed extensions that silently fail on macOS BSD sed
- Replace seq-based progress bar with bash for-loop to fix boundary rendering
- Add agent-experience-eval to maintenance projects list
New TypeScript package that evaluates a repository's AI agent experience
quality. Uses the Anthropic SDK directly with Zod structured output.

- Programmatic API via evaluate() and lower-level helpers
- CLI via agent-experience-eval command
- Local file discovery with guardrails (size/count limits, dedup, excludes)
- Currency validation (checks referenced paths and commands exist)
- Prompt builder with 180K char truncation
Convert from composite bash action to a proper TypeScript Node action
that imports @automattic/jetpack-agent-experience-eval programmatically.

- node24 runner, ncc-bundled dist/index.js (built during release)
- Calls evaluate() API directly — no Claude CLI required
- Handles artifact upload internally via @actions/artifact
- Removes old run.sh and prompt.md (logic moved to npm package)
- Workflow reduced to checkout + one action step (no Claude CLI install)
- Add fork PR guard and expanded trigger paths
- Remove tools/score-agent-experience.sh (use the CLI directly)
The SDK's zodOutputFormat() imports from zod/v4 internally. Our schema
must also use zod/v4 to produce compatible schema objects.
- Rewrite action README with usage, inputs, outputs, and rubric docs
- Rewrite JS package README with CLI and programmatic API examples
- Remove unused PHP config from action composer.json
- Add build steps to workflow (JS package + action must be built before
  the node24 action can run, since dist/ is gitignored)
- Make command validation monorepo-aware: walk from instruction file dir
  up to repo root when searching for commands in package.json/Makefile
- Add jp/jetpack to recognized command patterns
- Add post-parse cross-field validation: fix score sum and grade band
  if Claude returns inconsistent values
- Align workflow push paths with discovery patterns (use **/ prefix for
  dot-directories like .claude, .cursor, .codex)
- Fix npm command regex to skip flags (--filter etc.) before script name
- Separate jp/jetpack CLI commands from manifest-validated commands
- Reject path traversal (../) and absolute paths in reference extraction
The package is meant to be generic and open-sourceable. Repo-specific
CLI tools like jp/jetpack should not be hardcoded.
New --format flag (json|human|auto, default: auto):
- auto: human report on TTY, JSON when piped
- With --output: JSON always written to file, human report to stdout on TTY

The human report shows score/grade, per-criteria breakdown with progress
bars and notes, discovered files with sizes, currency validation results
(missing paths/commands), issues, recommendations, and token usage.
- Validate --format flag values (reject invalid input)
- Render unknown criteria keys (future-proofing)
- Scale formatBytes to MB for large files
--output now writes whatever --format resolves to:
  -o eval.json                    → JSON to file, human summary to TTY
  --format human -o report.txt    → human report to file
  --format json -o eval.json      → JSON to file
  (no --output)                   → output to stdout in resolved format
- Fix ReDoS: replace polynomial regex with two-pass token extraction
- Fix path validation: resolve paths relative to instruction file dir
  first, fall back to repo root, with escape-check
- Fix --output auto: writes JSON to file by default, shows human
  summary on TTY. Explicit --format overrides file format.
- Status messages go to stderr (not stdout)
- Update package README with --format docs and correct --output desc
- Fix LICENSE.txt links in both READMEs (file doesn't exist)
pnpm was not available because the workflow used actions/setup-node
directly instead of the repo's .github/actions/tool-setup which handles
pnpm installation. Also simplified pnpm install to full workspace
install (filtered installs can miss transitive deps).
- Replace ncc with tsdown (rolldown-powered) which handles ESM deps
- Upgrade @actions/artifact to v6 (latest, ESM-only)
- Add pnpmfile fix for missing @protobuf-ts/runtime-rpc dep in
  @actions/artifact (actions/toolkit#1898)
- Use .github/actions/tool-setup for pnpm/node setup in workflow
- Action tsconfig: extend jetpack-js-tools/tsconfig.base.json instead
  of setting module/moduleResolution directly
- JS package: set repository.url to mirror repo in npm canonical format
- JS package composer.json: remove phpunit-select-config (not a PHP
  project), remove unused autoload/classmap and test-coverage script
- Add schema.test.ts with 5 tests: valid parsing, missing criteria,
  score bounds, invalid grade, version check
- Enable --experimental-vm-modules for Jest ESM support
- Remove empty scaffold test file
- tsconfig.json: includes src + tests, noEmit, with jest types (for IDE)
- tsconfig.build.json: extends base, rootDir=src, outDir=build (for tsgo)
- Build script now uses tsconfig.build.json
The project structure linter forbids setting noEmit directly.
tsconfig.base.json already has noEmit: true which is what the root
tsconfig needs (IDE type checking only). tsconfig.build.json extends
tsc.json for actual compilation.
- Update @actions/core to 3.0.0 (matches rest of monorepo)
- Update jest to 30.3.0 (matches rest of monorepo)
- Move bin entry to bin/agent-experience-eval.js wrapper (committed
  file) so pnpm install doesn't warn about missing build/cli.js on
  clean checkout
@manzoorwanijk
manzoorwanijk force-pushed the add/agent-experience-eval branch from 7c814d0 to b0b3144 Compare April 3, 2026 13:52
@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

This PR has been marked as stale. This happened because:

  • It has been inactive for the past 3 months.
  • It hasn't been labeled `[Pri] BLOCKER`, `[Pri] High`, `[Status] Keep Open`, etc.

If this PR is still useful, please do a [trunk merge or rebase](https://github.com/Automattic/jetpack/blob/trunk/docs/git-workflow.md#keeping-your-branch-up-to-date) and otherwise make sure it's up to date and has clear testing instructions. You may also want to ping possible reviewers in case they've forgotten about it. Please close this PR if you think it's not valid anymore — if you do, please add a brief explanation.

If the PR is not updated (or at least commented on) in another month, it will be automatically closed.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically closed as it has not been updated in some time. If you want to resume work on the PR, feel free to restore the branch and reopen the PR.

@github-actions github-actions Bot closed this Jul 4, 2026
@github-actions
github-actions Bot deleted the add/agent-experience-eval branch July 4, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[Action] Agent Experience Eval Actions GitHub actions used to automate some of the work around releases and repository management Docs [JS Package] Agent Experience Eval RNA [Status] In Progress [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. [Status] Stale [Tests] Includes Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants