Copy DataStub datasets through the backend writer - #868
Open
ehennestad wants to merge 2 commits into
Open
Conversation
This was referenced Aug 25, 2026
ehennestad
force-pushed
the
zarr-support/7-writer-dataset-copy
branch
from
August 25, 2026 17:45
f3016a8 to
07e906b
Compare
ehennestad
marked this pull request as ready for review
August 25, 2026 17:54
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## zarr-support/6-writer-reference-probe #868 +/- ##
========================================================================
Coverage ? 95.26%
========================================================================
Files ? 234
Lines ? 8315
Branches ? 0
========================================================================
Hits ? 7921
Misses ? 394
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ehennestad
force-pushed
the
zarr-support/7-writer-dataset-copy
branch
from
August 26, 2026 17:43
07e906b to
906309c
Compare
DataStub.export reached through the abstract writer to a raw HDF5 file id and drove the copy itself with H5F/H5D/H5T/H5O calls: skip when the source and destination are the same file, rewrite element-wise when the dataset is compound with reference members (HDF5 library bug, see HDFGroup/hdf5#3429), otherwise fast-copy via H5O.copy when the destination does not exist yet. Give the base Writer a copyDatasetFromFile method stating those semantics in backend-neutral terms and move the HDF5 mechanics into HDF5Writer unchanged, including the same-file check via canonical H5F.get_name names. DataStub.export shrinks to a single delegation and no longer touches writer.FileId or any H5* function. The moved code also closes the source file id on the same-file early return, which the original leaked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ehennestad
force-pushed
the
zarr-support/7-writer-dataset-copy
branch
from
August 31, 2026 10:42
906309c to
1de59d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Background — This PR is part of a series that makes the write path storage-agnostic (stacked on #867), so that a Zarr v3 writer can be added later.
Problem — When a file is read and then exported to a new file, datasets whose data still lives in the source file (
types.untyped.DataStub) are copied over during export. This copy was done with raw HDF5 calls inside@DataStub/export.m: a check whether source and destination are the same file, a fast object copy, and a workaround that rewrites compound datasets containing references because an HDF5 library bug prevents copying them directly. All of this only works with the HDF5 backend, and a writer for another storage format cannot reuse any of it.Solution — Move the copy into the writer.
io.backend.base.WritergainscopyDatasetFromFile(sourceFilename, sourcePath, destinationPath), and the HDF5 code moves unchanged intoio.backend.hdf5.HDF5Writer.DataStub.exportnow only calls this method. Re-exporting a read file behaves the same as before.What changed
copyDatasetFromFile(sourceFilename, sourcePath, destinationPath)onio.backend.base.Writer, implemented inio.backend.hdf5.HDF5Writer.types.untyped.DataStub.exportcontains no HDF5 calls and nowriter.FileIdaccess; it calls the writer method.Implementation notes
Ordinary datasets are copied with
H5O.copy. Compound datasets containing references cannot be copied this way (HDFGroup/hdf5#3429), so they are read withio.parseCompoundand written again throughwriteValue, as before. The base-class help states that an implementation may replace the raw copy with a rewrite when its storage layer cannot copy a dataset directly.How to test
Read a file and export it to a new file; the data, which is still on disk in the source file, is carried over:
(
ref_demo.nwbis the file created in the test snippet of #867.)Checklist
fix #XXwhereXXis the issue number?🤖 Generated with Claude Code