Skip to content

ci: verifiable releases — signed images, provenance, SBOMs, admission policy and release gates - #126

Merged
devopssquaddev merged 8 commits into
mainfrom
feat/ci-verifiable-release
Nov 16, 2026
Merged

devopssquaddev merged 8 commits into
mainfrom
feat/ci-verifiable-release

Conversation

@devopssquaddev

Copy link
Copy Markdown
Owner

Summary

Makes every Community Edition release verifiable: images built only in CI, signed, with provenance and SBOMs, and checkable by users.

  • Pinned actions: every action in .github/workflows is pinned by SHA, and a new pin-audit workflow fails on any unpinned action or tool download. scripts/build-images.sh refuses to push a semver tag outside CI and gains --list / --list-json, so CI derives the image matrix from it.
  • Release workflow (release.yml, triggered by a v*-ce tag):
    1. checks that the tag matches the chart versions, image tags and Zelkor base images;
    2. builds every first-party image in CI and never overwrites an existing semver tag;
    3. signs each image digest keyless with cosign, bound to this workflow's identity;
    4. attaches an SPDX SBOM and SLSA v1 provenance to each image;
    5. pushes both charts to oci://ghcr.io/devopssquaddev/charts and signs them;
    6. round-trips every signature with verify_artifacts.sh.
  • Upstream images: upstream images we ship (Langfuse and others) are resolved to digests and listed in a signed release manifest attached to the chart. They are never mirrored.
  • Admission:
    • security.imageVerification.enabled (off by default) renders a Kyverno ClusterPolicy that rejects unsigned or mismatched zelkor-* images. It needs Kyverno 1.12 or later.
    • The new zelkor verify <release> command checks the images running in a release against the release identity.
  • Fail-closed matrix: scripts/release/fail_closed_matrix.py turns the security-gate JUnit results into a per-release table of security checks. A category with no evidence shows as GAP, never as passed.
  • Security docs: SECURITY.md (private reporting, 72-hour target for Critical CVEs in first-party images), a public CVE log in docs/security/cve-log.md, and a guide in docs/security/verify-release.md.
  • Release gates: upgrade-gates.yml on kind:
    • an upgrade from the previous release, then helm rollback, with zero failed requests;
    • a 30-minute storm of always-failing requests to a provider, with gateway memory kept under the ceiling in gate-thresholds.json.

Verification

  • 382 offline tests pass (release scripts, Kyverno policy render, zelkor verify with cosign mocked, schema, config matrix, snapshots, docs). shellcheck and helm lint are clean, and pin-audit passes on all workflows.
  • Not yet run: any GitHub Actions workflow, signing or GHCR push, Kyverno admission on a cluster, or the kind gates. The memory ceiling (384 MiB) is a first guess, to calibrate after the first gate run.
  • actionlint was not available locally; the workflows were checked by YAML parsing and structural tests only.

Before the first release

  • Allow Actions to request id-token, packages, attestations and contents write.
  • Make the GHCR packages public, including the new charts/zelkor-platform and charts/zelkor-agent, and link them to this repo.
  • Enable private vulnerability reporting. Tag protection for v*-ce is recommended.
  • Commit the security-gate JUnit to release-evidence/<tag>/s-gate.junit.xml before tagging; without it the matrix is all GAP.

Notes

  • Provenance uses predicate type slsaprovenance1 (SLSA v1); the CLI and the Kyverno policy expect the same.
  • It overlaps with the HTTP APIs PR in values.yaml, values.schema.json and docs/README.md. Whichever merges second needs a small rebase and a schema regeneration.

🤖 Generated with Claude Code

devopssquaddev and others added 7 commits November 16, 2026 17:35
- scripts/release/pin_audit.py fails on any `uses:` that is not a 40-hex
  commit SHA with a `# vX` comment, a `latest` version input, or an
  unpinned tool download (pip without ==, npm -g without @Version,
  go install @latest, /latest/ URLs, curl | sh). Workflows and composite
  actions are both scanned; pin-audit.yml runs it on PRs.
- docs, security-scan, renovate workflows: actions resolved to commit SHAs
  (git ls-remote, annotated tags dereferenced); pip-audit pinned to 2.10.1.
- scripts/build-images.sh: --list / --list-json expose the first-party
  image list (with first-party base and build tier) as the single source
  for the release matrix; --push of a semver tag outside GitHub Actions is
  refused so release tags only come from CI. :dev pushes stay allowed.
  Base images can be pinned by digest via BASE_IMAGE_REF; OCI version and
  revision labels are added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (CE-19d)

security.imageVerification.enabled (default false) renders a Kyverno
ClusterPolicy with verifyImages for ghcr.io/devopssquaddev/zelkor-*:
keyless attestor bound to the release workflow identity (issuer
token.actions.githubusercontent.com, subject regexp on
release.yml@refs/tags/vX.Y.Z-ce), mutateDigest + verifyDigest, and an
SLSA v1 provenance attestation whose workflow repository and path must
match. Unsigned, re-tagged or mismatched images are rejected at admission.

The issuer and identity regexp live once, as the chart defaults; the CLI
and release tooling are tested against them. failureAction is a schema
enum (Enforce|Audit); an empty identity fails the render. The toggle is a
PICT model parameter (matrix regenerated).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…release identity (CE-19d)

Lists the containers (init included) of the release's pods by
app.kubernetes.io/instance, resolves each Zelkor image to the digest the
node pulled, and runs `cosign verify` plus `cosign verify-attestation
--type slsaprovenance1` once per digest with the fixed keyless identity
(issuer token.actions.githubusercontent.com, release.yml on a vX.Y.Z-ce
tag). Any failure, an unknown release, or no Zelkor image at all exits 1;
a missing cosign binary is a clear error. Upstream images are listed as
covered by the release attestation, not verified here.

The identity constants are tested equal to the chart's
security.imageVerification defaults.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (CE-19e)

scripts/release/fail_closed_matrix.py turns the S-gate pytest JUnit XML
into a markdown matrix over the published categories: auth, guardrails,
egress, egress via DNS, admin plane unreachable from the data plane,
provider keys unreadable by tenants, component-down behaviour. Tests map
to categories by S-id (from test_sNN_* names) or file stem in the
checked-in fail_closed_categories.yaml; unmapped S-ids are listed.

A category is FAIL on any failure or error, GAP when nothing passed
(SKIP never counts as PASS), PASS otherwise. Missing evidence renders an
all-GAP matrix with a banner and a workflow warning instead of being
skipped. Only test names and counts are published, never failure text.
DNS egress has no S-id yet and shows as GAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gests (CE-19a/b/c)

.github/workflows/release.yml runs only on a vX.Y.Z-ce tag:
- plan: check_version.py fails the run unless the tag's X.Y.Z equals every
  Chart.yaml version/appVersion, every first-party image tag in chart and
  example values, and the Zelkor FROM/ARG bases of released images; then
  pin audit; the image matrix comes from build-images.sh --list-json,
  split by build tier so derived images build FROM the signed base digest.
- images-tier0..2: the release-image composite action builds and pushes
  each image in CI (an existing semver tag is never overwritten), signs the
  digest keyless, attests an SPDX SBOM (syft) and an SLSA v1 provenance
  naming repository, source tag, commit and workflow (cosign attest, plus
  GitHub attest-build-provenance pushed to the registry). Composite, not a
  reusable workflow, so the certificate identity stays release.yml.
- chart: helm package/push both charts to oci://ghcr.io/devopssquaddev/charts,
  cosign sign + provenance per chart digest; upstream_digests.py resolves
  the upstream images the chart ships (Langfuse web/worker, datastores,
  init images) to digests with crane, never mirroring them; the release
  manifest (first-party, chart and upstream digests) is attested on the
  platform chart; verify_artifacts.sh round-trips every signature and
  attestation with the identity read from the chart defaults.
- evidence + publish: fail-closed matrix from
  release-evidence/<tag>/s-gate.junit.xml (all-GAP when missing) and all
  evidence attached to the GitHub release, created as a draft if absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
upgrade-gates.yml runs after a successful release (or on dispatch) and
publishes both gate results to the release; logic is in scripts/release:
- upgrade_gate.sh: install the previous vX.Y.Z-ce (ce_tags.py) with
  ./install.sh --profile=perimeter, upgrade to N with the same installer,
  then `helm rollback` to the N-1 revision, while loadgen.py drives a
  fixed-rate /v1 load through the gateway; any non-2xx or connection
  error fails the gate (max_failed_requests = 0).
- retry_storm_gate.sh: 30 minutes of client retries against mock_llm.py
  answering 500 for every request, sampling Envoy proxy container working
  set from the kubelet summary API (gateway_memory.py, no metrics-server);
  fails over the published ceiling, on growth past max_growth_pct between
  the first and last window, on too few samples, or if any storm request
  succeeded.
- gate-thresholds.json is the one published file for rates, durations,
  the memory ceiling and growth limit. Every kubectl/helm call goes through
  wrappers that name the kind context. The mock provider is CI-only and
  never part of the chart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…(CE-19f)

- SECURITY.md: report through GitHub private vulnerability reporting,
  scope (first-party images, charts, CLI vs upstream images and customer
  code), fix targets with 72 hours for a Critical in a first-party image;
  contracted SLAs are not part of CE.
- docs/security/cve-log.md: CVE, image, severity, disclosed, fixed
  version, time to fix. Seeded with CVE-2026-32871 (fastmcp in zelkor-mcp,
  confirmed and fixed 2026-11-16 in v2.4.6-ce, #124).
- docs/security/verify-release.md: cosign verify / verify-attestation for
  images and charts with the release identity, the attested release
  manifest with upstream digests, `zelkor verify`, the opt-in Kyverno
  policy, and the published fail-closed matrix and gate results.
- CLI docs gain `zelkor verify`; Helm values reference gains
  security.imageVerification; docs nav gains a Security section.
- Shorter descriptions on a few pages keep llms.txt under its 2,000-token
  budget with the two new pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅

View logs ↗
No Preview URL. Enable ↗ f0e270f 2026-11-16T18:13:29.750Z Visit the dashboard ↗

@devopssquaddev
devopssquaddev merged commit 38abebb into main Nov 16, 2026
11 checks passed
devopssquaddev added a commit that referenced this pull request Nov 17, 2026
…ases) (#128)

* chore(platform): release 2.5.0 (secretless HTTP APIs, verifiable releases)

MINOR: new Helm contracts workspace.tools.httpApis (#125) and
security.imageVerification (#126).

- charts, first-party image tags, Zelkor Dockerfile bases and example
  charts at 2.5.0; example chart dependencies rebuilt
- profiles/values-production.yaml at tag 2.5.0 with empty digests: the
  release workflow builds and signs the images from the tag, so digests
  are pinned in a follow-up PR from the signed release
- test_production_overlay_uses_semver_not_dev accepts the release tag
  or a digest, never :dev
- THIRD_PARTY_NOTICES: 2.5.0 row (no third-party pin changes)

scripts/release/check_version.py v2.5.0-ce passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test,release: S-gate evidence for v2.5.0-ce; trace test uses the tenant mapping

- release-evidence/v2.5.0-ce/s-gate.junit.xml: S-gate run against the
  release candidate (101 tests, 0 failures, 11 environment skips);
  re-runs and the tool-policy overlay cases merged in, file paths removed.
  release.yml builds the published fail-closed matrix from it
  (PASS 6, GAP 1: DNS egress has no S-test yet).
- test_agent_trace compared the trace user with the test label; with real
  IdP tokens the platform tenant differs, so it now uses tenant_of().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant