Skip to content

ci: push the published images with zstd -19 layers - #7159

Merged
hectorvent merged 1 commit into
mainfrom
ci/release-zstd-layers
Nov 1, 2026
Merged

hectorvent merged 1 commit into
mainfrom
ci/release-zstd-layers

Conversation

@pgermosen

@pgermosen pgermosen commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Summary

Publishes the release and nightly images with every layer compressed by zstd at level 19 instead of gzip. The binaries are the same, so runtime speed doesn't change, and zstd layers also unpack about three times faster than gzip ones.

Measured by running the new steps locally on the 2.4.0 binaries:

Image 2.4.0 (gzip) zstd -19
Native, amd64 94.6 MB 70.1 MB (-26%)
Native, arm64 92.6 MB 67.1 MB (-28%)
Compat, amd64 223.2 MB 167.7 MB (-25%)

BuildKit's own compression=zstd tops out at about level 11 (73.8 MB for the amd64 binary layer, against 64.9 MB at 19), so the pushes go through a composite action, .github/actions/push-zstd-image:

  1. It builds the image exactly as before, but into an OCI layout with uncompressed layers.
  2. recompress.py recompresses every layer with the zstd CLI at level 19 and rewrites the manifests and index that point at them. Each layer's uncompressed digest is checked against the diff_ids in the image config, so the content can't change on the way.
  3. A second build whose Dockerfile is only FROM image, with the layout passed in as that named context, pushes it. BuildKit keeps the recompressed layers as they are and attaches provenance and the SBOM to the digests it actually pushes, so sign-images keeps working from the same digest output.

Release (native, baseline, compat) and nightly (native, compat) both use it, so the nightly runs this path every night before a release tag depends on it. Dependabot now also scans .github/actions/*, so the build-push-action pin inside the action stays current.

Since only release and nightly call the action, a new workflow, push-zstd-image.yml, tests it on the pull requests that change it. It runs recompress.py against fixture layouts: a single-platform image, a nested index, a layer that doesn't match its diff_id, and a second run over the same layout. It then pushes a two-platform image through the whole action to a registry on the runner, and checks that every layer is zstd, that each platform image has its attestations, and that the runner's Docker can pull and run it.

Two things change for the people pulling these images:

  • Pulling zstd layers needs Docker Engine 23.0 or later. zstd support landed in 23.0, and the backport to 20.10 was closed unmerged because 20.10 had reached end of life. On 20.10 the pull fails, so the installation docs now say 23.0+.
  • The published provenance now describes the re-export build. Its only input is the recompressed OCI image, so the ubi9-micro base digest and the Dockerfile steps are no longer listed in it. The image history still shows every Dockerfile step, the SBOM is unchanged, and the cosign signatures cover the same set of manifests as before. Keeping the full Dockerfile provenance at level 19 would mean rewriting BuildKit's attestation subjects after the fact. If the provenance matters more than the size, compression=zstd,compression-level=19 on the existing build-push steps gets BuildKit's best level instead, about 9 MB larger on the amd64 binary layer, with the provenance untouched.

Tested locally with a BuildKit v0.33.1 container builder and a local registry:

  • Native amd64 from the 2.4.0 binary. Every layer comes out zstd, the layer digests in the registry match the ones in the recompressed layout (so BuildKit didn't touch them), and the image config matches 2.4.0's apart from the per-build labels. The provenance and SBOM subjects match the pushed manifest. Pulled with Docker 29.9 and run, it goes healthy and answers SQS calls.
  • An amd64 plus arm64 re-export of floci/floci:2.4.0, which covers gzip input layers, the nested index and layers shared between the two platforms.
  • Compat amd64, built on top of the zstd native image. It goes healthy, and aws sqs create-queue and boto3 both work inside it.
  • Running recompress.py twice on the same layout gives the same image digest.
  • The new workflow's job, simulated with the same registry image and builder settings: it passes, and its check fails on a plain gzip image.

Not covered locally: arm64 builds of the two Dockerfiles (this machine has no QEMU), and pushing to Docker Hub and ECR Public. The first nightly covers both.

Recompressing costs CPU time in the push job. From 8-thread timings, I'd expect about 6 extra minutes on a 4-vCPU runner across the three release images, well inside the job's 30-minute timeout.

Type of change

  • Bug fix (fix:)
  • New feature (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

AWS Compatibility

No wire change. The same binaries are packaged with a different layer compression.

Checklist

  • ./mvnw test passes locally
  • New or updated integration test added
  • Commit messages follow Conventional Commits
  • I have no more than 2 open, non-draft pull requests and no more than 4 open pull requests in total (drafts included) in this repository (maintainers and dependency bots are exempt).

@pgermosen
pgermosen requested a review from hectorvent as a code owner November 1, 2026 00:58
@pgermosen pgermosen added chore docker Container image and docker/ build tooling (Dockerfiles, entrypoints, compose) github-actions labels Nov 1, 2026
@greptile-apps

greptile-apps Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge, with the previous test-coverage gap addressed and no new actionable issues found.

Summary

Publishes release and nightly images through a shared action that recompresses OCI layers with zstd at level 19.

  • Release and nightly images now use smaller zstd layers.
  • Dependabot now checks actions inside .github/actions/*.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Build original Dockerfile] --> B[Export OCI layout]
  B --> C[Recompress layers with zstd]
  C --> D[Rewrite manifest and index digests]
  D --> E[Re-export and push image]
  E --> F[Attach provenance and SBOM]
  E --> G[Return pushed digest]
  G --> H[Sign release images]
Loading

Reviews (2) · Last reviewed commit: "ci: push the published images with zstd ..." · Reviewed by Greptile

BuildKit compresses zstd at about level 11 at most, so the release and
nightly images now go through a composite action: the image is built
into an OCI layout, every layer is recompressed with the zstd CLI at
level 19, and a second build that only names the layout pushes it.
BuildKit keeps the recompressed layers and still attaches provenance
and the SBOM to the pushed digests.

On 2.4.0 this takes the native image from 94.6 to 70.1 MB on amd64 and
from 92.6 to 67.1 MB on arm64, and compat amd64 from 223.2 to 167.7 MB.
Pulling zstd layers needs Docker Engine 23.0 or later, so the
installation requirements move up from 20.10.

A new workflow tests the action on the pull requests that change it:
recompress.py against fixture layouts, then the whole action pushing a
two-platform image to a registry on the runner.
@pgermosen
pgermosen force-pushed the ci/release-zstd-layers branch from 69694a4 to 3e0c25d Compare November 1, 2026 01:07
@hectorvent

Copy link
Copy Markdown
Member

@pgermosen thanks, checking every recompressed layer against its diff_id before anything is written means a bad rewrite fails the job instead of publishing, and the end to end push workflow covers the path that release and nightly never run on a PR.

On the two trade-offs you laid out: I'm taking the zstd -19 path with the thinner provenance. The SBOM, the signatures and the image history stay intact, and about 25% off every pull is worth more here. Since a chore: title adds nothing to the release notes, I'll add the Docker 23.0+ requirement to the next release notes by hand, so people on 20.10 aren't surprised.

No blockers from my side, merging. I'll watch the first nightly for the arm64 build and the Docker Hub and ECR pushes.

@hectorvent
hectorvent merged commit e77823f into main Nov 1, 2026
5 checks passed
@hectorvent
hectorvent deleted the ci/release-zstd-layers branch November 1, 2026 02:38
@hectorvent

Copy link
Copy Markdown
Member

🎉 This PR is included in version 2.5.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore docker Container image and docker/ build tooling (Dockerfiles, entrypoints, compose) github-actions released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants