Skip to content

ci(test): make the Codecov coverage upload a required signal - #85

Closed
bketelsen wants to merge 1 commit into
mainfrom
ci/codecov-fail-on-error
Closed

ci(test): make the Codecov coverage upload a required signal#85
bketelsen wants to merge 1 commit into
mainfrom
ci/codecov-fail-on-error

Conversation

@bketelsen

@bketelsen bketelsen commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Blocked on an operator action — do not merge yet. Making this signal
required immediately exposed that Firn's Codecov upload has never actually
worked. See "What this uncovered" below.

Summary

  • codecov.yml gates pull requests on project and patch coverage statuses, but those statuses only come into existence once the Unit Tests job's coverage profile actually reaches Codecov.
  • The upload step disabled both failure paths at once: fail_ci_if_error: false told the action to swallow upload errors, and continue-on-error: true suppressed the step's own failure. A failed or misconfigured upload therefore left Unit Tests green with no coverage statuses reported — the coverage gate could vanish silently while CI stayed all-checks-passing.
  • Set fail_ci_if_error: true and remove the step-level continue-on-error so an upload or configuration failure fails the Unit Tests job.
  • The tokenless OIDC upload is preserved unchanged (use_oidc: true plus the job-scoped id-token: write permission), as is the action's pinned SHA. codecov.yml is not touched: project (target: auto, threshold: 1%) and patch (target: 70%, threshold: 5%) thresholds are byte-for-byte identical.
  • The diff is one file, .github/workflows/test.yml, and touches only the Codecov upload step and its explanatory comment.

What this uncovered

The change did its job on its very first run, and the answer is worse than a hypothetical: the upload is not merely fragile, it is already failing on every run, including main.

Codecov rejects the upload outright:

Upload queued for processing result --- {"result": "RequestResult(
  error=RequestError(code='HTTP Error 404', params={},
  description='{"message":"Repository not found"}\n'),
  status_code=404, text='{"message":"Repository not found"}\n')"}
error -- Upload queued for processing failed: {"message":"Repository not found"}
==> Failed to run upload-coverage

This is not a regression introduced here. The identical 404 appears in the
Unit Tests job of the most recent Tests run on main
(run 33028261901) —
and that run is recorded as success, because the two suppressions this
pull request removes hid it. Consequences of the status quo:

  • No coverage report has ever reached Codecov for this repository.
  • The codecov/project and codecov/patch statuses in codecov.yml have therefore never been posted, so every threshold in that file is currently inert. The coverage gate exists only on paper.
  • CI has been reporting green on a signal that does not exist.

Repository not found is a Codecov-side account/activation error, not a
workflow defect — nothing in this repository can fix it. An operator must
activate frostyard/firn on Codecov (and confirm the org's tokenless OIDC
upload is enabled for it) before this pull request can go green.
Once that is
done, re-running this job is the verification: the upload succeeds and the two
coverage statuses appear on this pull request for the first time.

Merging this branch before that operator action would turn a silently-missing
signal into a hard CI failure on every subsequent pull request. That trade is
the maintainer's call, which is why this stays a draft.

Test plan

  • actionlint .github/workflows/test.yml — exit 0, no findings (go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 .github/workflows/test.yml)
  • make verify — passed: go mod tidy -diff, gofmt, golangci-lint 2.13.1 (0 issues.), go vet ./..., go test ./... all ok
  • make ci==> ci passed (adds coverage, go test -race -short ./internal/..., and the linux/amd64 + linux/arm64 cross-arch builds)
  • git diff origin/main -- codecov.yml is empty — the thresholds are unchanged
  • CI on this head (07406bf): Repository gate pass, Security Scan pass, E2E Harness Syntax pass, Unit Tests fail — failing exactly at Upload coverage to Codecov with the 404 above, which is the intended new behavior firing against a genuinely broken upload.
  • Successful Codecov upload with codecov/project and codecov/patch reported — blocked: requires the Codecov-side repository activation described above. Not achievable from this repository.

Risk classification

High. .github/workflows/** is a protected boundary in policies/agent-governance.json (workflow-and-permissions, decision: review-required, minimum_risk_tier: high), and the repository classifies by highest-applicable tier. The change only tightens a check — never_relax lists required-checks and coverage-checks, and this moves in the permitted direction — but it converts a currently-and-permanently failing upload into a hard CI failure, so it must not merge before the Codecov activation lands.

codecov.yml gates pull requests on project and patch coverage statuses,
but those statuses only exist once the Unit Tests job's coverage profile
actually reaches Codecov. The upload step both asked the action not to
fail on error (fail_ci_if_error: false) and suppressed the step's own
failure (continue-on-error: true), so an upload or configuration failure
left the job green with no coverage statuses to report — the gate could
silently disappear.

Set fail_ci_if_error: true and drop the step-level continue-on-error so
an upload failure fails the Unit Tests job. The tokenless OIDC upload
(use_oidc, id-token: write) and every codecov.yml threshold are
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UigLsmd17TVDuaaSN4yfdU
@bketelsen bketelsen closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant