fix(report): expose per-case execution metrics - #191
Conversation
zpzjzj
left a comment
There was a problem hiding this comment.
Code review summary
Review scope: codex/fix-report-case-metrics → main at 9a18eb6f
Findings
- Blocker: 0
- Major: 1
- Info: 1
Compatibility assessment
The machine-readable result.json contract remains backward-compatible for normal JSON consumers:
- Existing fields such as
total_tokens,duration_ms,input_tokens, andoutput_tokensare retained. total_tokenspreserves its previous meaning: tested-agent tokens only.- Judge and overall metrics are additive fields.
- Regenerating reports from the existing legacy fixture succeeds and preserves
total_tokens=12500, while derivingoverall_tokens=12500.
There is still a compatibility consideration for consumers that parse derived presentation formats. The HTML embedded DATA object renames duration / total_tokens and per-case duration_ms / duration, while the Markdown table schema also changes. No repository or publicly searchable downstream consumer was found, so this is informational rather than blocking. If these representations are consumed by internal automation, consider retaining the old HTML aliases for one release or explicitly documenting that only result.json is a stable machine-readable contract. Strict JSON schemas using additionalProperties: false should also be checked before rollout because the new additive fields would require a schema update.
Overall assessment
Validation reviewed: all GitHub checks are green; make test and targeted race-enabled report, runner, and CLI tests pass.
Summary
total_tokensJSON field for compatibilityRoot cause
The top-level report duration comes from
EndTime - StartTime, while each case duration comes from the tested-agent session. The HTML report labeled both scopes ambiguously and omitted the per-case token fields that were already present inCaseResult.Agent-judge sessions also exposed duration and token usage internally, but those metrics were not copied into the report model before
JudgeSessionwas excluded from JSON serialization.This change is engine-independent. Agent-specific token acquisition, including QoderCLI session parsing, is intentionally out of scope.
User impact
Reports now make Skill execution cost visible without allowing metrics to dominate the response and grading content:
Validation
make fmtmake verifygo test -race ./...expression-calculatorusing QoderCLI:Closes #190