Skip to content

Align eval-gate exit codes 75/76 with sysexits.h convention #331

Description

@augustineuzokwe

Background

PR #330 introduced two CI exit codes for the eval gate:

  • 76 = transient Gemini 5xx (502/503/504) → CI retries.
  • 75 = Gemini 429 RESOURCE_EXHAUSTED (project spend cap) → CI fails fast.

These numbers are inverted vs the Unix sysexits.h convention:

Code sysexits.h name Meaning
75 EX_TEMPFAIL "temporary failure; user is invited to retry"
76 EX_PROTOCOL "remote protocol error"
77 EX_NOPERM "permission denied"
78 EX_CONFIG "configuration error"

The standard "please retry me" code is 75, but PR #330 used it for "don't retry."

Proposed fix

Swap the constants:

  • _EXIT_GEMINI_TRANSIENT = 75 (matches EX_TEMPFAIL)
  • _EXIT_GEMINI_429 = 77 (closer to EX_NOPERM — cap-exhausted is "you're not permitted to spend more")

Update retry_on_exit_code: 75 in .github/workflows/ci.yml and the contract test in tests/test_run_evals_exit_codes.py.

Acceptance

  • Constants match sysexits.h semantics.
  • Workflow's retry_on_exit_code lines up.
  • Contract test pins the new numbers.
  • No behaviour change to users — purely a legibility improvement for the next person who reads the workflow.

Out of scope

Considered + rejected:

Surfaced during PR #330 review thread.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions