Background
PR #330 introduced two CI exit codes for the eval gate:
76 = transient Gemini 5xx (502/503/504) → CI retries.
75 = Gemini 429 RESOURCE_EXHAUSTED (project spend cap) → CI fails fast.
These numbers are inverted vs the Unix sysexits.h convention:
| Code |
sysexits.h name |
Meaning |
| 75 |
EX_TEMPFAIL |
"temporary failure; user is invited to retry" |
| 76 |
EX_PROTOCOL |
"remote protocol error" |
| 77 |
EX_NOPERM |
"permission denied" |
| 78 |
EX_CONFIG |
"configuration error" |
The standard "please retry me" code is 75, but PR #330 used it for "don't retry."
Proposed fix
Swap the constants:
_EXIT_GEMINI_TRANSIENT = 75 (matches EX_TEMPFAIL)
_EXIT_GEMINI_429 = 77 (closer to EX_NOPERM — cap-exhausted is "you're not permitted to spend more")
Update retry_on_exit_code: 75 in .github/workflows/ci.yml and the contract test in tests/test_run_evals_exit_codes.py.
Acceptance
- Constants match sysexits.h semantics.
- Workflow's
retry_on_exit_code lines up.
- Contract test pins the new numbers.
- No behaviour change to users — purely a legibility improvement for the next person who reads the workflow.
Out of scope
Considered + rejected:
Surfaced during PR #330 review thread.
Background
PR #330 introduced two CI exit codes for the eval gate:
76= transient Gemini 5xx (502/503/504) → CI retries.75= Gemini 429 RESOURCE_EXHAUSTED (project spend cap) → CI fails fast.These numbers are inverted vs the Unix
sysexits.hconvention:EX_TEMPFAILEX_PROTOCOLEX_NOPERMEX_CONFIGThe standard "please retry me" code is 75, but PR #330 used it for "don't retry."
Proposed fix
Swap the constants:
_EXIT_GEMINI_TRANSIENT = 75(matchesEX_TEMPFAIL)_EXIT_GEMINI_429 = 77(closer toEX_NOPERM— cap-exhausted is "you're not permitted to spend more")Update
retry_on_exit_code: 75in.github/workflows/ci.ymland the contract test intests/test_run_evals_exit_codes.py.Acceptance
retry_on_exit_codelines up.Out of scope
Considered + rejected:
nick-fields/retry. Has merit but is a bigger surface change.Surfaced during PR #330 review thread.