Deprecate rechunker?

[rabernat]

We created this package because Dask was unable to rechunk arrays in a scalable way. The motivation is described in this five-year-old blog post: https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster

Since then, Dask has overhauled how it handles this problem. We're now able to reliably rechunk Arrays of any size using Dask. (See https://discourse.pangeo.io/t/rechunking-large-data-at-constant-memory-in-dask-experimental/3266.)

Given this, is it worth continuing to maintain Rechunker?

There are of course features that Rechunker provides which Dask does not, especially the other executors. However, it's worth discussing whether it's a worthwhile tradeoff to maintain this code base going forward. I personally don't have a lot of time to spend on it, so unless other maintainers are eager to keep going, my inclination is to wind it down.

[TomAugspurger]

cc @aulemahal and @shoyer (beam).

@tomwhite IIUC, cubed doesn't depend on rechunker, right? It's vendored here?

I'm also not able to help maintain rechunker these days so would be open to deprecating this in favor of alternatives.

[shoyer]

Xarray-Beam has a dependency on rechunker only for rechunker.algorithm. This would be straightforward to fork into Xarray-Beam if needed.

[tomwhite]

@tomwhite IIUC, cubed doesn't depend on rechunker, right? It's vendored here?

That's right. Cubed also only needs the rechunker algorithm.

[aulemahal]

On our side, the dependency on rechunker was there because we weren't aware of all the advancements in dask. I'm ok with deprecating it.