Use Zarr 3

[forman]

Is your feature request related to a problem? Please describe.

This issue comprises 2 important sub-issues

1. xcube still uses and relies on Zarr 2, this concerns the Python package and the format. This has manyfold implications.
2. when xcube is installed with no version requirement but an explicit zarr requirement, version xcube 1.7.1 will be installed as this is the last one that doesn't enforce zarr >=2.18,<3. Many xcube data store plugins are broken in this constellation (e.g., xcube-cmems, xcube-sh).

Describe the solution you'd like

Immediate actions:

• In the current xcube version, check the zarr version eagerly and raise an error if it is not zarr >=2.18,<3. Error message: give advice on how to proceed, say that the issue will be fixed asap. Release this asap.
• Optionally patch older xcube versions on conda-forge to enforce zarr >=2.18,<3.

More effort, but should be addressed immediately:

• Migrate code to allow for using Zarr 3 wrt to API and format compatibility. Stay with Zarr 2 format by default when writing for time being. Adjust unit tests, verify against all example notebooks.
• Update documentation, where applicable.
• Do the same as above for applicable plugins.

After that:

• Exploit new Zarr 3 features such as shards.
• Adjust for new and existing EO-, geo- (e.g., GeoZarr), and multi-resolution specs.

Additional context

• See also https://github.com/xcube-dev/xcube-roadmap/issues/11
• The ESA EOPF Zarr Format will also be based on Zarr 3

[dpabon]

Hi Forman,

I hit this issue today. Glad to see that there is already a plan. I'm a bit worried because I was expecting to use the xcube visualizer functionality to deploy a web server and show some Zarr v3 products we're working on, but xcube doesn't support Zarr v3 yet. Is there any progress we can expect in the coming weeks or a branch we can already test?

Thanks in advance for your help.

Best,
Daniel