NetCDF IO cleanup, including in-memory reads/writes with netCDF4 #10624
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
topic-backends
topic-zarr
I am thinking I might try to split this into a separate PR, because it's unrelated to the netCDF4 in-memory changes. |
This PR includes a handful of significant changes: 1. It refactors the internal structure of `DataTree.to_netcdf()` and `DataTree.to_zarr()` to use lower level interfaces, rather than calling `Dataset` methods. This allows for properly supporting `compute=False` (and likely various other improvements). 2. Reading and writing in-memory data with netCDF4-python is now supported, including DataTree. 3. The `engine` argument in `DataTree.to_netcdf()` is now set consistently with `Dataset.to_netcdf()`, preferring `netcdf4` to `h5netcdf`. 3. Calling `Dataset.to_netcdf()` without a target now always returns a `memoryview` object, *including* in the case where `engine='scipy'` is used (which currently returns `bytes`). This is a breaking change, rather than merely issuing a warning as is done in pydata#10571. I believe it probably makes sense to do as a this breaking change because (1) it offers significant performance benefits, (2) the default behavior without specifying an engine will already change (because `netcdf4` is preferred to the `scipy` backend) and (3) restoring previous behavior is easy (by wrapping the memoryview with `bytes()`). mypy
2 tasks
3 tasks
shoyer
changed the title
shoyer
changed the title
shoyer
changed the title
|
This PR is ready for review now. |
5 tasks
shoyer
changed the title
2 tasks
shoyer
force-pushed
the
netcdf4-memory
branch
from
8c524c5 to
2100673
Compare
|
I added a regression test for #10712 |
shoyer
removed
topic-zarr
github-actions
bot
added
the
topic-DataTree
kmuehlbauer
approved these changes
Sep 9, 2025
Co-authored-by: Kai Mühlbauer <[email protected]>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes several related improvements to netCDF reading/writing:
engine='h5netcdf'can now be pickled, as long as the underlying file-like object also support pickle.engine='scipy'no longer closes the underlying file, consistent the h5netcdf backend.xarray.backends.file_managerandxarray.backends.locks.PickleableFileManagerclass, which is used for wrapping file-like objects that do not natively support pickling (e.g., netCDF4.Dataset and h5netcdf.File) in cases where a global cache is not desirable (e.g., for netCDF files opened from bytes in memory, or from existing file objects).whats-new.rst