feat(transaction): clean expired metadata in ExpireSnapshotsAction - #3064
Draft
alessandro-nori wants to merge 4 commits into
Draft
feat(transaction): clean expired metadata in ExpireSnapshotsAction#3064alessandro-nori wants to merge 4 commits into
alessandro-nori wants to merge 4 commits into
Conversation
alessandro-nori
commented
Aug 25, 2026
| } | ||
|
|
||
| // Unlike specs, a snapshot's schema is recorded in metadata, so this needs no I/O. A | ||
| // snapshot that predates `schema-id` leaves its schema unknown; treating it as referencing |
Author
There was a problem hiding this comment.
schema-id is an optional field on a snapshot and snapshots written by older writers don't have it
alessandro-nori
marked this pull request as ready for review
August 25, 2026 09:28
alessandro-nori
marked this pull request as draft
August 25, 2026 09:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Follows up on #2591, which landed
ExpireSnapshotsActionand left this as remaining work: "cleanExpiredMetadata(removing unreferenced partition specs and schemas)."What changes are included in this PR?
Adds
clean_expired_metadata(bool)toExpireSnapshotsAction, off by default. When enabled, the action also emitsRemoveSchemas/RemovePartitionSpecsfor specs and schemas that no surviving snapshot references.Schemas need no I/O, since each snapshot records its
schema_id. Specs require reading each surviving snapshot's manifest list, so those reads are spawned on the table's IO runtime with bounded concurrency, skipped entirely for single-spec tables, and stopped early once every spec has been seen. The current schema and default spec are always seeded as reachable.commit()no longer returns early when nothing expires, since cleanup can still have work to do.Because the removals are computed against the base metadata, enabling the flag also asserts
CurrentSchemaIdMatch,DefaultSpecIdMatchandLastAssignedPartitionIdMatch. Without them a concurrent commit that makes a targeted spec or schema current/default turns the removal into a non-retryableDataInvalidfrom the metadata builder; asserting them yields a retryable conflict so the action re-plans against the refreshed table. These are not added for plain expiries, which cannot be invalidated that way.