Skip to content

Support AGENTS.md and CLAUDE.md evaluation targets - #11

Open
aaronfc wants to merge 2 commits into
mainfrom
codex/issue-10-instruction-targets
Open

Support AGENTS.md and CLAUDE.md evaluation targets#11
aaronfc wants to merge 2 commits into
mainfrom
codex/issue-10-instruction-targets

Conversation

@aaronfc

@aaronfc aaronfc commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Fixes #10

Why

unvibe could evaluate only a directory containing SKILL.md, even though its plan and assertion engine operates on plain instruction text. Repositories that keep AGENTS.md or CLAUDE.md could not reuse the same lightweight evaluation workflow, and silently approximating native loading semantics would overstate what the runner measures.

Current main also standardized the suite filename as EVALUATIONS.yaml and added selectable Claude, Codex, and OpenCode harnesses. This branch needs to compose with those changes instead of restoring the old singular filename or Claude-only execution.

What

  • Resolve explicit SKILL.md, AGENTS.md, and CLAUDE.md files while preserving the existing single-file directory workflow.
  • Create EVALUATIONS.yaml beside the selected document by default; creation reads but never creates or rewrites the instruction document.
  • Retain the legacy EVALUATION.yaml fallback for normal runs and support custom suites through --evaluation.
  • Reject missing, unsupported, and ambiguous targets with actionable errors.
  • Use format-neutral narration, generation prompts, reports, help text, and package metadata across all supported harnesses.
  • Keep evaluation literal, warn when CLAUDE.md imports are not expanded, and document the native effective-context boundary.
  • Merge current main, including multi-harness execution, typed scenario results, SIGINT handling, and the plural suite migration.

Testing Steps

Prerequisite: install and authenticate a supported harness. The following uses the suggested Codex models from the README.

proof_dir="$(mktemp -d)"
echo "Always inspect repository status before changing files." > "$proof_dir/AGENTS.md"

uv run unvibe --create "$proof_dir/AGENTS.md" \
  --harness codex \
  --evaluation-model gpt-5.6-sol \
  --rubric-model gpt-5.6-luna

ls -1 "$proof_dir"

Expected: the command reports created .../EVALUATIONS.yaml; the directory contains the original AGENTS.md and new EVALUATIONS.yaml.

Cheap ambiguity edge case:

echo "Follow @AGENTS.md." > "$proof_dir/CLAUDE.md"
uv run unvibe "$proof_dir" \
  --harness codex \
  --evaluation-model gpt-5.6-sol \
  --rubric-model gpt-5.6-luna

Expected: exit 1, list AGENTS.md and CLAUDE.md, and request an explicit file target.

Cleanup:

rm -r "$proof_dir"

Proofs

Direct deterministic CLI proof with a stub Codex harness:

$ uv run unvibe --create /private/tmp/unvibe-pr11-proof/AGENTS.md ...
created /private/tmp/unvibe-pr11-proof/EVALUATIONS.yaml

$ ls -1 /private/tmp/unvibe-pr11-proof
AGENTS.md
EVALUATIONS.yaml
codex-proof

$ uv run unvibe /private/tmp/unvibe-pr11-proof ...
error: target directory /private/tmp/unvibe-pr11-proof contains multiple supported instruction documents: AGENTS.md, CLAUDE.md. Pass one of those files explicitly.

Automated checks:

  • uv run pytest -q — 88 passed in 0.35s.
  • tests/smoke.shSMOKE OK; packaged flows passed for Claude, Codex, and OpenCode.

@aaronfc

aaronfc commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

Needs proper dogfooding before merging. Still not sure if this is a good idea.

@aaronfc aaronfc self-assigned this Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support AGENTS.md and CLAUDE.md as evaluation targets

1 participant