Skip to content

mz334: eliminate and squash fully cached stages#871

Draft
mzihlmann wants to merge 1 commit into
mz873-rolling-cache-keyfrom
mz334-stage-elimination
Draft

mz334: eliminate and squash fully cached stages#871
mzihlmann wants to merge 1 commit into
mz873-rolling-cache-keyfrom
mz334-stage-elimination

Conversation

@mzihlmann

@mzihlmann mzihlmann commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Fixes #334

With the cache lookahead and inferred cross-stage keys in place, a multistage build still built every stage even when all its consumers were served from cache. This PR adds the payoff: after the precompute pass the stage graph optimization from MakeKanikoStages runs again, this time over the precompute stage builders. A COPY --from that resolved via its inferred key is a CachingCopyCommand and no longer counts as a reference, so stages that only fed cached copies get eliminated and newly linear FROM chains get squashed. In the fully warm case a build that forked through COPY --from collapses into a single stage where every layer comes from cache and nothing is unpacked.

Two design points make this safe. First, the composite cache is now recursive: every key is folded into a fixed-size state and the state is the canonical cache key, like OCI chainIDs or git commits. A stage boundary is a resumable digest, so squashed and unsquashed chains produce identical keys by construction and the cache-state-dependent squash cannot change any key. This also fixes the latent separator collision of the joined-text scheme and shrinks the redirect pointer payload to a digest. It busts every existing cache once, which needs a loud release-notes entry. Second, one memoized layer cache is shared between the precompute and the build pass, so every pointer resolution and layer retrieval the precompute made is answered from memory during the build. Nothing is retrieved twice and the build cannot fail on a pointer or layer that vanished in between, which matters because an eliminated stage cannot be rebuilt. The key-match assert stays for resolutions the build pass makes fresh, those copies kept their source stage alive so the files are there to verify against.

The plan renders after the pass, so it shows what will actually be built. dockerfile.go is deliberately untouched apart from exporting Squash, the pass mirrors its loops inline so the two can be merged in a follow-up. This lands as one PR for verification and will be split up for review.

@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch 2 times, most recently from 67af00c to 6417e73 Compare July 5, 2026 10:43
@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.34328% with 25 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/executor/build.go 81.06% 18 Missing and 7 partials ⚠️

📢 Thoughts on this report? Let us know!

@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch 3 times, most recently from b85dc65 to 83e3fe7 Compare July 5, 2026 12:08
@mzihlmann mzihlmann mentioned this pull request Jul 5, 2026
4 tasks
@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch 3 times, most recently from 3c74703 to d5ad42a Compare July 5, 2026 18:57
@mzihlmann mzihlmann changed the base branch from main to mz873-rolling-cache-key July 5, 2026 18:57
@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch from d5ad42a to f00fca5 Compare July 5, 2026 20:05
@mzihlmann mzihlmann force-pushed the mz873-rolling-cache-key branch from 0bc1ff7 to 87f7892 Compare July 5, 2026 20:23
@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch from f00fca5 to 242e297 Compare July 5, 2026 20:25
@mzihlmann mzihlmann force-pushed the mz873-rolling-cache-key branch from 87f7892 to fb01a77 Compare July 5, 2026 20:41
@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch from 242e297 to 3116248 Compare July 5, 2026 20:49
@mzihlmann mzihlmann force-pushed the mz873-rolling-cache-key branch from fb01a77 to 31db7b2 Compare July 5, 2026 21:31
@mzihlmann mzihlmann force-pushed the mz334-stage-elimination branch from 3116248 to ca9e9be Compare July 5, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

multistage cache lookahead

1 participant