mz334: eliminate and squash fully cached stages#871
Draft
mzihlmann wants to merge 1 commit into
Draft
Conversation
67af00c to
6417e73
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
b85dc65 to
83e3fe7
Compare
4 tasks
3c74703 to
d5ad42a
Compare
d5ad42a to
f00fca5
Compare
0bc1ff7 to
87f7892
Compare
f00fca5 to
242e297
Compare
87f7892 to
fb01a77
Compare
242e297 to
3116248
Compare
fb01a77 to
31db7b2
Compare
3116248 to
ca9e9be
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #334
With the cache lookahead and inferred cross-stage keys in place, a multistage build still built every stage even when all its consumers were served from cache. This PR adds the payoff: after the precompute pass the stage graph optimization from MakeKanikoStages runs again, this time over the precompute stage builders. A COPY --from that resolved via its inferred key is a CachingCopyCommand and no longer counts as a reference, so stages that only fed cached copies get eliminated and newly linear FROM chains get squashed. In the fully warm case a build that forked through COPY --from collapses into a single stage where every layer comes from cache and nothing is unpacked.
Two design points make this safe. First, the composite cache is now recursive: every key is folded into a fixed-size state and the state is the canonical cache key, like OCI chainIDs or git commits. A stage boundary is a resumable digest, so squashed and unsquashed chains produce identical keys by construction and the cache-state-dependent squash cannot change any key. This also fixes the latent separator collision of the joined-text scheme and shrinks the redirect pointer payload to a digest. It busts every existing cache once, which needs a loud release-notes entry. Second, one memoized layer cache is shared between the precompute and the build pass, so every pointer resolution and layer retrieval the precompute made is answered from memory during the build. Nothing is retrieved twice and the build cannot fail on a pointer or layer that vanished in between, which matters because an eliminated stage cannot be rebuilt. The key-match assert stays for resolutions the build pass makes fresh, those copies kept their source stage alive so the files are there to verify against.
The plan renders after the pass, so it shows what will actually be built. dockerfile.go is deliberately untouched apart from exporting Squash, the pass mirrors its loops inline so the two can be merged in a follow-up. This lands as one PR for verification and will be split up for review.