Repository navigation
ci: push the published images with zstd -19 layers - #7159
Conversation
[Medium impact] The PR appears safe to merge, with the previous test-coverage gap addressed and no new actionable issues found.
|
BuildKit compresses zstd at about level 11 at most, so the release and nightly images now go through a composite action: the image is built into an OCI layout, every layer is recompressed with the zstd CLI at level 19, and a second build that only names the layout pushes it. BuildKit keeps the recompressed layers and still attaches provenance and the SBOM to the pushed digests. On 2.4.0 this takes the native image from 94.6 to 70.1 MB on amd64 and from 92.6 to 67.1 MB on arm64, and compat amd64 from 223.2 to 167.7 MB. Pulling zstd layers needs Docker Engine 23.0 or later, so the installation requirements move up from 20.10. A new workflow tests the action on the pull requests that change it: recompress.py against fixture layouts, then the whole action pushing a two-platform image to a registry on the runner.
69694a4 to
3e0c25d
Compare
|
@pgermosen thanks, checking every recompressed layer against its On the two trade-offs you laid out: I'm taking the zstd -19 path with the thinner provenance. The SBOM, the signatures and the image history stay intact, and about 25% off every pull is worth more here. Since a No blockers from my side, merging. I'll watch the first nightly for the arm64 build and the Docker Hub and ECR pushes. |
|
🎉 This PR is included in version 2.5.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Summary
Publishes the release and nightly images with every layer compressed by zstd at level 19 instead of gzip. The binaries are the same, so runtime speed doesn't change, and zstd layers also unpack about three times faster than gzip ones.
Measured by running the new steps locally on the 2.4.0 binaries:
BuildKit's own
compression=zstdtops out at about level 11 (73.8 MB for the amd64 binary layer, against 64.9 MB at 19), so the pushes go through a composite action,.github/actions/push-zstd-image:recompress.pyrecompresses every layer with the zstd CLI at level 19 and rewrites the manifests and index that point at them. Each layer's uncompressed digest is checked against thediff_idsin the image config, so the content can't change on the way.FROM image, with the layout passed in as that named context, pushes it. BuildKit keeps the recompressed layers as they are and attaches provenance and the SBOM to the digests it actually pushes, sosign-imageskeeps working from the samedigestoutput.Release (native, baseline, compat) and nightly (native, compat) both use it, so the nightly runs this path every night before a release tag depends on it. Dependabot now also scans
.github/actions/*, so thebuild-push-actionpin inside the action stays current.Since only release and nightly call the action, a new workflow,
push-zstd-image.yml, tests it on the pull requests that change it. It runsrecompress.pyagainst fixture layouts: a single-platform image, a nested index, a layer that doesn't match itsdiff_id, and a second run over the same layout. It then pushes a two-platform image through the whole action to a registry on the runner, and checks that every layer is zstd, that each platform image has its attestations, and that the runner's Docker can pull and run it.Two things change for the people pulling these images:
compression=zstd,compression-level=19on the existing build-push steps gets BuildKit's best level instead, about 9 MB larger on the amd64 binary layer, with the provenance untouched.Tested locally with a BuildKit v0.33.1 container builder and a local registry:
floci/floci:2.4.0, which covers gzip input layers, the nested index and layers shared between the two platforms.aws sqs create-queueand boto3 both work inside it.recompress.pytwice on the same layout gives the same image digest.Not covered locally: arm64 builds of the two Dockerfiles (this machine has no QEMU), and pushing to Docker Hub and ECR Public. The first nightly covers both.
Recompressing costs CPU time in the push job. From 8-thread timings, I'd expect about 6 extra minutes on a 4-vCPU runner across the three release images, well inside the job's 30-minute timeout.
Type of change
fix:)feat:)feat!:orfix!:)AWS Compatibility
No wire change. The same binaries are packaged with a different layer compression.
Checklist
./mvnw testpasses locally