Skip to content

fix chunk/shard iteration #3299

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
cc3698b
factor array element iteration routines into stand-alone functions, a…
d-v-b Jul 25, 2025
39b5c07
add shard_grid_shape, chunk_grid_shape
d-v-b Jul 25, 2025
70a4876
docstrings
d-v-b Jul 25, 2025
35b4891
handle null shards
d-v-b Jul 25, 2025
8846c96
use shard_grid_shape instead of cdata_shape in nchunks
d-v-b Jul 25, 2025
31698d4
add improved set of low-level iteration routines for arrays
d-v-b Jul 25, 2025
a258780
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Jul 25, 2025
5c34e4a
correct deprecation message
d-v-b Jul 25, 2025
943af28
Merge branch 'fix/_iter_chunk_keys' of github.com:d-v-b/zarr-python i…
d-v-b Jul 25, 2025
54d7fd7
check for deprecation warnings around iter_chunk_keys
d-v-b Jul 25, 2025
82d9bdc
plug tiny coverage holes
d-v-b Jul 25, 2025
ec5e0f1
lint
d-v-b Jul 25, 2025
83a3ea5
rename chunks_initialized to shards_initialized
d-v-b Jul 28, 2025
29c1cdd
add nshards, nshards_initialized
d-v-b Jul 28, 2025
2d7daca
fix doctests
d-v-b Jul 28, 2025
1dad7b3
remove iter_chunk_keys
d-v-b Jul 29, 2025
0b99164
make nchunks_initialized report the product of the number of shards a…
d-v-b Jul 29, 2025
7e73301
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b Jul 31, 2025
96e5564
changelog
d-v-b Jul 31, 2025
66fb795
correct name of changelog entry
d-v-b Jul 31, 2025
b9d6ea3
add test for unsharded data
d-v-b Aug 1, 2025
96f0e95
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 1, 2025
8b33dc3
add test for cdata_shape, shard_grid_shape, chunk_grid_shape
d-v-b Aug 1, 2025
80f6ac4
add nshards
d-v-b Aug 1, 2025
8db5fa7
expand nchunks_initialized test conditions
d-v-b Aug 1, 2025
8805c35
restore memorystore
d-v-b Aug 1, 2025
ac27ea7
expand test parametrization
d-v-b Aug 4, 2025
ecf961f
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 4, 2025
f773ace
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 4, 2025
9323400
unbreak tests
d-v-b Aug 4, 2025
61edfa8
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 4, 2025
fb53121
include asyncarray cdata_shape in test
d-v-b Aug 4, 2025
877a96a
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 4, 2025
97e4b3a
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 5, 2025
fe9c2ff
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 5, 2025
fcb2b3a
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b Aug 6, 2025
1c2c6cd
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 11, 2025
70f9f4f
Merge branch 'main' into fix/_iter_chunk_keys
d-v-b Aug 13, 2025
aa9d0b7
make new API private
d-v-b Aug 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions changes/3299.bugfix.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Fix a bug in ``create_array`` caused by iterating over chunk-aligned regions instead of
shard-aligned regions when writing data. To make the distinction between chunks and shards more
obvious in the ``Array`` API, new properties ``chunk_grid_shape``,
``shard_grid_shape``, ``nshards``, ``nshards_initialized`` were added to the ``Array`` class.
Additionally, the behavior of ``nchunks_initialized`` has been adjusted. This function consistently
reports the number of chunks present in stored objects, even when the array uses the sharding codec.
2 changes: 1 addition & 1 deletion docs/user-guide/performance.rst
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ the time required to write an array with different values.::
... start = time.time()
... arr[:] = value
... elapsed = time.time() - start
... result.append((elapsed, arr.nchunks_initialized))
... result.append((elapsed, arr.nshards_initialized))
... return result
... # log results
>>> for write_empty_chunks in (True, False):
Expand Down
Loading
Loading