Ragged Arrays with Ragged Dim>2 don't Save

[CSSFrancis]

This isn't exactly an issue that we can solve in hyperspy without some kind of hacky workarounds but something that I wanted to bring up here.

I ran into this issue while working on creating a diffraction vectors class where both the zarr and hdf5 saving methods fail when trying to save a 2 dimensional array. There was movement to solve this issue in zarr: zarr-developers/numcodecs#200 but this largely seems to have stalled.

Example of the Failing Code

import numpy as np
import hyperspy.api as hs

x = np.empty(shape=(3), dtype=object)

for i in np.ndindex(x.shape):
    x[i] = np.random.random((4, 2))

s = hs.signals.BaseSignal(x).T

s.save("test.zspy")
s.save("test.hspy")

For a solution to this I might try to see what it takes to bring the zarr PR across the finish line if no one is working on it. I think that this is kind of an edge case, but supporting it in zarr might be significantly easier than trying to get it to work with hdf5. From my understanding there, the work around is to cast the data into a new datatype which seems complicated to get to work in our case.

[ericpre]

Yes, indeed, the alternative approach of flattening the array when saving and un-flattening it when reading will be required to support the h5py format - see h5py/h5py#876. Not the most elegant, but since it should be possible to change only the HierarchicalReader and HierarchicalWriter classes, it wouldn't be that bad neither! ;)

[CSSFrancis]

Interesting work around, but not the worst thing as you said. That is looking more promising to just make the changes to the HierarchicalReader and HierarchicalWriter classes because it would solve the problem for both cases.

I will start writing something up for fixing the issue.

[CSSFrancis]

This is effectively closed with #2906.