Support zipped zarr

[aulemahal]

Is your feature request related to a problem? Please describe.
We, at Ouranos, have an internal database of many datasets in netCDF and in Zarr. The database is (partially) duplicated between our internal server (HPC-like but not quite) and an externally shared HPC. This other machine has a strict quota on inodes (10M over 2PB, which means an average file size of 215 Mo).

Thus, conventional Zarr folders are a no-go because they are composed of numerous really small files. A while ago, we decided to zip all of them. The impact on reading speed is insignificant compared to other productivity gains (not needing netCDF4 for example).

However, Zarr 3 has dropped the transparent support of zipped directory. One can't do xr.open_zarr('path/to/dataset.zarr.zip') anymore. One has to create a zarr.ZipStore and pass that to xarray.

Zarr's upcoming "url pipeline" (ZEP 8) fixes this in a way, but it has been "upcoming" for a while now. And the addition of kerchunk to intake-esm's dependencies has implicitly pinned zarr to >3.

Describe the solution you'd like
Somewhere in source.py::_open_dataset I think we could detect that the path is a zipped zarr and act accordingly. For example is the url endswith .zip and format == 'zarr'.

Describe alternatives you've considered
Not updating intake-esm nor zarr. But that's not long-term.

Additional context
As a side questions, if any other data managers are reading this : have you had this inode issue before ? How did you solve it ?

I'd rather fix this in intake-esm or zarr before converting our full database back to netCDFs or an other format.

[charles-turner-1]

As a side questions, if any other data managers are reading this : have you had this inode issue before ? How did you solve it ?

I didn't realise this could be done - looks like a really interesting solution. We've been mulling over how we might deal with this without much progress. cc @dougiesquire @joshuatorrance.

I recently discovered that saving zarr stores to an icechunk repo seems to (N.B I really do mean seems to here, I haven't checked thoroughly) massively reduce the inode consumption. I don't understand how or why yet, working that out is on the todo list. With that said, we'd need to update intake-esm to work with icechunk if we tried to implement that as a solution.

[dougiesquire]

As a side questions, if any other data managers are reading this : have you had this inode issue before ? How did you solve it ?

I didn't realise this could be done - looks like a really interesting solution. We've been mulling over how we might deal with this without much progress. cc @dougiesquire @joshuatorrance.

In my previous life at CSIRO I made heavy use of zarr.ZipStore to deal with this issue. But I was under the impression that this less problematic with Zarr v3, which supports sharding.

[aulemahal]

The new sharding feature looks interesting indeed! However, ignoring the fact that it would mean converting the full database, the inode decrease is not as great as a zipped store. AFAIU, xarray/dask/zarr currently require that the in-memory chunk equals the shard at the moment of writing, which limits either the shard size or the writing performance (no parallel writing to a shard possible).

As for icechunk, I haven't looked into that much yet, but I feel that its use case doesn't overlap completely with the problem at hand ? In any case, it looks like integrating it with intake-esm would look a lot like the Zip integration I'm suggesting. And AFAIU, it could also be magically fixed by ZEP8's "url-pipeline".