Improve memory load for map_complete_blocks #6730
Conversation
|
|
for more information, see https://pre-commit.ci
…mplete_memory # Conflicts: # lib/iris/_lazy_data.py
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6730 +/- ##
==========================================
- Coverage 90.29% 90.25% -0.05%
==========================================
Files 91 91
Lines 24475 24631 +156
Branches 4571 4609 +38
==========================================
+ Hits 22100 22230 +130
- Misses 1607 1624 +17
- Partials 768 777 +9 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
for more information, see https://pre-commit.ci
| df = [False] * len(max_outchunks) | ||
| for dim in dims: | ||
| df[dim] = True | ||
| df = tuple(df) |
There was a problem hiding this comment.
I think this can be done more tidily
| df = [False] * len(max_outchunks) | |
| for dim in dims: | |
| df[dim] = True | |
| df = tuple(df) | |
| df = tuple(i in dims for i in range(len(shape))) |
| ----- | ||
| .. note: | ||
| If the output chunks would larger than the maximum chunksize set |
There was a problem hiding this comment.
Tiny typo
| If the output chunks would larger than the maximum chunksize set | |
| If the output chunks would be larger than the maximum chunksize set |
| # Note that one chunk is irregularly rechunked and the other isn't. | ||
| expected_chunk = (2, 2, 1, 2, 2) | ||
| assert result.chunks[1] == expected_chunk |
There was a problem hiding this comment.
I think this would be clearer if you checked chunks in both the rechunked dims
| # Note that one chunk is irregularly rechunked and the other isn't. | |
| expected_chunk = (2, 2, 1, 2, 2) | |
| assert result.chunks[1] == expected_chunk | |
| # Note that one chunk is irregularly rechunked and the other isn't. | |
| assert result.chunks[0] == (1, 1, 1, 1, 1) | |
| assert result.chunks[1] == (2, 2, 1, 2, 2) # split from the original chunks of (5, 4) |
|
|
||
| result = map_complete_blocks( | ||
| cube, self.func, dims=(2, 3), out_sizes=(30, 40), dtype=lazy_array.dtype | ||
| ) | ||
| assert is_lazy_data(result) |
There was a problem hiding this comment.
I don't think this bit is contributing anything much, as we don't check any properties of the result, except that it is lazy.
I think we can remove this, and only check the main result (i.e. the rechunked one).
| result = map_complete_blocks( | |
| cube, self.func, dims=(2, 3), out_sizes=(30, 40), dtype=lazy_array.dtype | |
| ) | |
| assert is_lazy_data(result) | |
| #(nothing) |
| # Reduce the optimum dask chunksize. | ||
| with dask.config.set({"array.chunk-size": "32KiB"}): | ||
| result = map_complete_blocks( | ||
| cube, self.func, dims=(2, 3), out_sizes=(30, 40), dtype=lazy_array.dtype |
There was a problem hiding this comment.
I think it would be a more convincing example if the "fixed" dims weren't at the end of the shape.
I think you can easily transpose the example so the initial content of
da.ones((5, 9, 10, 10), chunks=(2, 5, 10, 5))
==> da.ones((5, 10, 9, 10), chunks=(2, 10, 5, 5))
And here in the map call we use dims=(1, 3).
(N.B. I did try this, and it does actually seem to work!)
3 participants
🚀 Pull Request
Closes #3808.