Skip to content

Export DataPipes through the backend writer and remove FileId - #869

Draft
ehennestad wants to merge 2 commits into
zarr-support/7-writer-dataset-copyfrom
zarr-support/8-writer-datapipe
Draft

Export DataPipes through the backend writer and remove FileId#869
ehennestad wants to merge 2 commits into
zarr-support/7-writer-dataset-copyfrom
zarr-support/8-writer-datapipe

Conversation

@ehennestad

@ehennestad ehennestad commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Background — Last PR in the series that makes the write path storage-agnostic (stacked on #868), so that a Zarr v3 writer can be added later.

ProblemDataPipe.export passed a raw HDF5 file id to its internal pipe object (obj.internal.write(writer.FileId, fullpath)). This was the last place that used Writer.FileId. As long as the base writer exposes an HDF5 file id, export code can depend on HDF5 directly, and a writer for another storage format cannot implement the interface.

Solution — Add exportDataPipe(dataPipe, destinationPath) to the writer interface. It writes the pipe's dataset with its chunking and compression settings, and returns the pipe state bound to the written dataset. With its last user gone, the FileId property is removed from io.backend.base.Writer. After this PR, no code outside the HDF5 backend can reach an HDF5 file id. Writing Zarr v3 files is not part of this PR; that comes in a separate PR.

What changed

  • New method exportDataPipe(dataPipe, destinationPath) on io.backend.base.Writer, implemented in io.backend.hdf5.HDF5Writer.
  • types.untyped.DataPipe.export calls the writer method and no longer uses a file id.
  • Removed the FileId property from io.backend.base.Writer. HDF5Writer keeps its H5FileId, and its unit test reads that directly.
  • Added the new method to the expected-methods list in the base writer test.
Implementation notes

exportDataPipe returns the written pipe instead of assigning it, because DataPipe.internal has SetAccess = private — only DataPipe.export can assign it. The pipe classes themselves (BoundPipe, BlueprintPipe, chunking, compression, dynamic filters) stay HDF5-specific and are now only reachable from the HDF5 writer. Exporting a pipe that is bound to a dataset in a different file still raises NWB:BoundPipe:CannotExportToNewFile.

How to test

Export a chunked DataPipe and read it back, then check that the base writer no longer has a FileId property:

nwb = NwbFile( ...
    'identifier', 'pipe-demo', ...
    'session_description', 'DataPipe export demo', ...
    'session_start_time', datetime(2026,8,25, 'TimeZone', 'local'));
pipe = types.untyped.DataPipe('data', rand(10, 100), 'chunkSize', [10 10]);
nwb.acquisition.set('pipedTs', types.core.TimeSeries( ...
    'data', pipe, 'data_unit', 'n/a', 'starting_time', 0, 'starting_time_rate', 1));
nwbExport(nwb, 'pipe_demo.nwb');
in = nwbRead('pipe_demo.nwb', 'ignorecache');
disp(class(in.acquisition.get('pipedTs').data))
disp(isprop(io.backend.hdf5.HDF5Writer('x.nwb', 'overwrite'), 'FileId'))
types.untyped.DataPipe
   0

Checklist

  • Have you ensured the PR description clearly describes the problem and solutions?
  • Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
  • If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code

@ehennestad

Copy link
Copy Markdown
Collaborator Author

Internal datapipe classes are still HDF5 bound. Need decision on this. Generalise, or rename to HD5FBoundPipe and HD5FBluePrintPipe and add equivalent ZarrPipes?

ehennestad and others added 2 commits August 31, 2026 12:42
DataPipe.export handed writer.FileId to its internal pipe, the last
call site reaching through the abstract writer to a raw HDF5 file id.
The pipe internals (BoundPipe/BlueprintPipe) drive H5P/H5D directly for
chunking, compression, dynamic filters and extendable datasets; that is
legitimately HDF5-specific and stays put, but the raw id should not
cross the backend interface to get there.

Give the base Writer an exportDataPipe method: it writes the pipe's
dataset at the destination path and returns the pipe state bound to the
written dataset, which DataPipe.export stores as its new internal
state. HDF5Writer implements it by passing its own H5FileId to the
pipe's write method.

With no consumers left, delete the FileId dependent property and the
getFileId plumbing from the base Writer and HDF5Writer. HDF5WriterTest
now inspects the file through HDF5Writer's own H5FileId property.
Non-export DataPipe internals (the append/bind path) are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ehennestad
ehennestad force-pushed the zarr-support/8-writer-datapipe branch from d97b407 to 74c27fa Compare August 31, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant