-
-
Notifications
You must be signed in to change notification settings - Fork 399
feat: add experimental support for rectilinear (variable-sized) chunks #3802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
maxrjones
merged 153 commits into
zarr-developers:main
from
maxrjones:poc/unified-chunk-grid
Apr 7, 2026
Merged
Changes from 101 commits
Commits
Show all changes
153 commits
Select commit
Hold shift + click to select a range
4f63867
Add prospectus
maxrjones fb96207
Initial prospectus POC
maxrjones b3e72ec
V2 prospectus
maxrjones d6d551a
V3 prospectus
maxrjones 8b8af74
Fastforward POC to V3
maxrjones 784f4e7
Remove prospectus
maxrjones f1a1bc3
Fix sharding
maxrjones 30fa867
Fix bugs
maxrjones 1282c7b
Support sequence in array functions
maxrjones ea89f33
Add end-to-end tests
maxrjones 2eba460
Collapse indexing paths
maxrjones fa07396
Add DimensionGrid protocol
maxrjones a0acd95
Remove the try/except escape hatch from ChunkGrid.chunk_shape
maxrjones ce0527d
Cache is_regular
maxrjones f433668
Produce RLE directly
maxrjones 02fd7c5
Fix bugs
maxrjones 42ef639
Separate chunk grid serialization
maxrjones 55e720b
Retain comments
maxrjones 1b7871d
Update block and coordinate indexing
maxrjones 9c0f582
POC: TiledDimension
maxrjones 1e2fa97
Support rectilinear shards
maxrjones 1f424b0
Revert "POC: TiledDimension"
maxrjones 0967e53
Fix __getitem__ for 1d chunk grids
maxrjones 67a684d
Implement resize
maxrjones 61c48a4
Fix spec compliance
maxrjones 7d5ebb8
Fix .info
maxrjones e74586a
Fix typing
maxrjones 8ab3ca8
Adopt joe's property testing strategy
maxrjones 80d8280
Remove RegularChunkGrid
maxrjones ffc7805
Use none rather than sentinel value
maxrjones 9beaee6
Remove regular chunk grid
maxrjones da2c08b
Fix boundary handling in VaryingDimension
maxrjones 6d9de38
Add chunk_sizes property
maxrjones cc2999a
Add docs
maxrjones b47ddba
Improve polymorphism
maxrjones 44d845f
Merge branch 'main' into poc/unified-chunk-grid
maxrjones 2caa927
always return based on inner chunks
maxrjones 6af91a6
Fix from_array
maxrjones e04d864
Add V3 of the prospectus
maxrjones 8dcea81
Fastforward design docs
maxrjones 4eb01c5
Require array extent
maxrjones d893d6f
Add overflow chunk tests
maxrjones 308bb24
Design doc for chunk grid metadata separation
maxrjones 0f52822
minor simplifications
maxrjones 5823fbb
Gatekeep rectilinear chunks behind feature flag
maxrjones 27f28e7
Fix off-by-one bug
maxrjones a35cf56
Fix chunk indexing boundary checks
maxrjones cbb28fe
Standardize docstrings
maxrjones 280eb68
fix spec compliance
maxrjones e88c06b
Handle integer floats
maxrjones 58bd336
More spec compliance
maxrjones e0fbab4
Fix block indexing error
maxrjones 5277739
Add V2 regression tests
maxrjones c9858c0
Add comments
maxrjones 9e4fa30
Consistent bounds checking between dimension types
maxrjones a21d587
use pre-computed extent
maxrjones e062580
Improve sharding validation logic
maxrjones 3591734
Improve sharding validation logic
maxrjones 4be96b0
Remove deferred design
maxrjones 087382b
Update design doc
maxrjones 38fd5aa
Remove unnecessary casts
maxrjones bbc0703
Improve typing
maxrjones 460d683
Add another deferred item
maxrjones 73164b6
Add to design doc
maxrjones aec0abd
Add design principles
maxrjones abb9d9d
Polish design doc
maxrjones aa002c8
Update migration sequence
maxrjones fffe4da
Remove stale sections
maxrjones 6777ec5
Use TypeGuard
maxrjones adec422
Cache nchunks
maxrjones 4903b09
Add cubed example
maxrjones 67e540c
move chunk grid off metadata (#6)
d-v-b 14370e6
Fixup after refactor
maxrjones bfc5d6b
Fixup
maxrjones 0f78339
Remove duplicated code
maxrjones fa6980d
Add to experimental
maxrjones 2360392
Avoid divide by zero
maxrjones 21aa18b
Improve RLE validation
maxrjones 90476b8
Improve RLE validation
maxrjones 7e171f5
Raise error on unknown chunk grid
maxrjones b6b271f
Add utility function
maxrjones c19e9db
Minor improvements
maxrjones 764eeaf
Update shorthand
maxrjones 54b399d
Fix zero chunks
maxrjones becd392
Remove extraneous validation
maxrjones 6764ba1
Improve tests
maxrjones 9b36448
Improve docstrings
maxrjones 11a47ff
Update design doc
maxrjones 4d7c724
Update docs
maxrjones 826e030
DRY
maxrjones 6f51e1c
Add test
maxrjones 879f20f
Simplify
maxrjones edbdb5d
Consistent .chunks and .shards
maxrjones 4a940b1
Remove separators
maxrjones 475de21
Polish
maxrjones 5f24ce6
Merge branch 'main' into poc/unified-chunk-grid
d-v-b a5715b9
Improve layout of work in progress page (#3841)
dstansby 3a9d042
perf: oindex optimization (#3830)
maxrjones 1c2efa6
chore: spec0 compat (python 3.14 compat, python 3.12 min) (#3564)
ilan-gold 1e66846
fix: remove numcodecs off-spec warning (#3833)
slevang f2ed1c4
Improve design doc
maxrjones a857bac
Merge branch 'main' into poc/unified-chunk-grid
maxrjones 7318255
Add demo notebook
maxrjones 662ceef
Add release notes
maxrjones 994e329
Fix indexing empty slices
maxrjones f1c5182
Add open support
maxrjones c4f7cf4
Improve release note
maxrjones 6436db6
Create and use resolve_chunks functions
maxrjones 26b4760
Make chunk_grid property private
maxrjones 5a7280b
Improve docstrings
maxrjones 60ad5cb
Update notebook
maxrjones f2ec718
Remove dead code
maxrjones 2c06fb2
chore: simplify sharding codec validation against varying chunk grid …
d-v-b 8965d09
refactor: allow regular-style chunk grid declaration for rectilinear …
d-v-b b3b5933
Fix typo
maxrjones f80f798
Normalize
maxrjones 3327cc1
Remove shim
maxrjones 5642e03
Consistent typing
maxrjones 6a4c01a
Move design doc outside public docs
maxrjones e3ba71f
Update config.md
maxrjones 6667fda
Remove get_chunk_shape
maxrjones 0f0576e
Make dimensions private
maxrjones e1d57d6
Merge branch 'main' into poc/unified-chunk-grid
maxrjones 4c6d805
Merge branch 'main' into poc/unified-chunk-grid
maxrjones c63f786
Update tests/test_properties.py
maxrjones b25ee98
Rename metadata classes
maxrjones 2a6186f
Backwards compatibility shim
maxrjones 1e06946
Sync design doc with implementation
maxrjones 0374429
Remove confusing behavioral adjective
maxrjones ca442f8
Update glossary
maxrjones 6f30fdb
Expand hypothesis strategies
maxrjones 8b15c98
expand draw for test_oindex
maxrjones 7b522f9
Document unique_edge_lengths purpose
maxrjones 6792801
Use
maxrjones c4df1ce
Add complex_rectilinear_arrays strategy
maxrjones beb0226
Add complex_rectilinear_arrays strategy
maxrjones 9940a5f
Remove dead code
maxrjones 7d9c6a2
explain the underflow motivation
maxrjones 2d0499a
Add fastpath todo
maxrjones 5cef539
Add another todo
maxrjones e9d06c2
Merge branch 'main' into poc/unified-chunk-grid
maxrjones 4d467a7
Test more edge cases
maxrjones 7a9931b
Make _unique_edge_lengths private
maxrjones 3e322ef
Fix typing for JSON objects
maxrjones 278f44b
Clarify backwards compatibility design
maxrjones af0f896
Revert "Backwards compatibility shim"
maxrjones 86956a8
Merge branch 'main' into poc/unified-chunk-grid
maxrjones dbd2cf8
Merge branch 'main' into poc/unified-chunk-grid
d-v-b ce8caca
Merge branch 'main' into poc/unified-chunk-grid
maxrjones a0faed8
Update design doc
maxrjones 1f46392
Update notebook
maxrjones aa7cea5
Update changelog
maxrjones 88b8a1c
Merge branch 'main' into poc/unified-chunk-grid
maxrjones File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Upgrade to spec0 compat (python 3.14 max, python 3.12 min). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Optimize the performance of indexing operations when using an array-like indexer on a single dimension. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Remove the warning that is emitted when any Numcodecs codec is instantiated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.