fix(metrics): count opencode skill invocations - #939
Conversation
`n_skill_invocations` was 0 for every opencode rollout that used a skill,
because opencode's ACP shape matches none of the recognized signals:
{"type": "tool_call", "kind": "other", "title": "skill",
"content": [{"type": "content", "content": {"type": "text",
"text": "<skill_content name=\"...\">..."}}]}
`kind` is "other", the event carries no tool-name key, the title check used
an inline `{"invoke_skill", "activate_skill"}` literal that dropped the
"skill" spelling `_SKILL_TOOL_NAMES` already had, and the content sniffer
only knew the OpenHands `Tool: invoke_skill` header. So the one metric that
answers "did the agent actually use the skill" silently read as "no" for a
whole harness -- and absent measurement is indistinguishable from a real
zero, which makes with-skill/no-skill comparisons unfalsifiable.
Recognize both opencode signals, keeping the module's existing discipline
that a tool declaring a real ACP kind is never reclassified from its title
or output text:
- `<skill_content name="...">` is matched only when it *opens* a structured
tool-result text block (anchored `\A`, same as the OpenHands header), so a
grep that quotes the tag is not counted.
- bare `title == "skill"` is honored only for unclassified kinds, unlike the
explicit `invoke_skill` spellings which stay trusted outright.
Verified against 206 recorded opencode trajectories: the counter now agrees
exactly with an independent `<skill_content name=...>` scan on every one,
56 with-skill rollouts move from 0 to a true count, and no no-skill rollout
is counted -- the no-skill invariant that backstops this path still holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be6a1a03dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| """opencode reports a skill call as kind="other" / title="skill" with the | ||
| skill body in a <skill_content> envelope. Neither the canonical kind nor the | ||
| OpenHands header is present, so before this shape was recognized every | ||
| opencode with-skill rollout reported n_skill_invocations=0.""" |
There was a problem hiding this comment.
Name the guarded change in regression docstrings
The three newly added regression tests—this OpenCode-envelope case and the two negative cases below it—describe the behavior they protect but do not identify the PR or commit being guarded. Add the relevant PR/commit reference to each docstring so they comply with the repository's explicit regression-test convention.
AGENTS.md reference: AGENTS.md:L17-L17
Useful? React with 👍 / 👎.
Problem
n_skill_invocationsis 0 for every opencode rollout, including ones thatdemonstrably loaded a skill. opencode's ACP skill call looks like this:
{"type": "tool_call", "kind": "other", "title": "skill", "status": "completed", "content": [{"type": "content", "content": {"type": "text", "text": "<skill_content name=\"polar-electrostatics-mentor\">\n# Skill: ..."}}]}is_skill_invocation_eventmisses it four ways over:kindis"other", not the canonical"skill".tool_name/name/functionkey, so_event_tool_namereturns"".{"invoke_skill", "activate_skill"}which dropped the
"skill"spelling that_SKILL_TOOL_NAMES— used by thekindcheck three lines above — already contains.Tool: invoke_skillheader.This matters more than a cosmetic counter:
n_skill_invocationsis the fieldthat answers "did the agent actually use the skill?", and it is the control
variable for every with-skill vs no-skill comparison. Reading a hard
0for anentire harness makes those comparisons unfalsifiable — you cannot tell a skill
that didn't help from a skill that was never loaded.
Fix
Recognize both opencode signals, preserving the module's existing rule that a
tool declaring a real ACP kind is never reclassified from its title or output:
<skill_content name="...">counts only when it opens a structuredtool-result text block (anchored
\A, exactly as the OpenHands header is), soa
grepwhose output quotes the tag is not counted.title == "skill"is honored only for unclassified kinds(
""/other/tool) — the same gate content sniffing sits behind. Theexplicit
invoke_skill/activate_skillspellings stay trusted outright, soexisting behavior is unchanged. A
readorexecutetool titledskill(a filename, say) is still not a skill invocation.
Verification
Replayed against 206 recorded opencode trajectories from a 29-task,
6-arm evaluation:
<skill_content name=...>scanThe no-skill invariant that backstops the text-derived path still holds.
New tests cover the opencode envelope, a quoted-tag negative, and a
real-kind tool titled
skill.tests/test_skill_invocation_artifacts.pypasses 10/10 (7 pre-existing unchanged); 697 tests pass across the
skill/metrics/trajectory suites.
ruff checkandruff format --checkclean.🤖 Generated with Claude Code