Reduce list of base requirements

[frazane]

Currently, the setup.py specifies the following:

base_requires = [
    "apache_beam[gcp]>=2.31.0",
    "cftime>=1.6.2",
    "numpy>=2.1.3",
    "pandas>=2.2.3",
    "scipy",
    "scikit-learn",
    "xarray>=2024.11.0",
    "zarr",
    "fsspec",
    "gcsfs",
    "absl-py",
    "pyarrow",
    "opt_einsum",
]

however, several of these libraries are not needed if one doesn't intend to use Beam pipelines or Parquet. Would it be possible to make those optional dependencies?

[raspstephan]

Hi, thanks for the issue. Beam is pretty central to WB-X. Can you describe the use case you have that doesn't require beam?

Regarding other dependencies, do you have an example of how this is handled in other libraries. E.g. parquet is used for a particular data loader. I could add specific requires to the setup script but I worry that this will get out of hand. Just thinking out loud here, not really opposed if people find it helpful.

[frazane]

Hi Stephan, thanks for your reply. In my case I have several functions that were specifically optimized to work with dask, which I can scale pretty well on a HPC platform. I am wrapping them to create new statistics and metrics for WB-X. However, I observe an important performance degradation when I use them in a Beam pipeline.
Instead, currently I am writing the loop over chunks explicitly, using a combination of load_chunk, compute_unique_statistics_for_all_metrics and the aggregators. This gives me more control over the computation using dask's chunks, persist, compute (these are also present inside the functions for the metrics). The performance improves substantially and it wasn't particularly difficult to implement.
I definitely see the advantages of the Beam pipeline API and many users will find it extremely useful and simple to use. But from my perspective the strength of WB-X lies in its core functionalities and classes: dataloaders, statistics, metrics and aggregators, and how they operate together. I personally think those deserve to be the main focus - it could be then up to the users to decide how they want to scale computations since it's a separate concern.

Regarding the other dependencies for data loaders: data is usually very much specific to a user's environment. Xarray itself doesn't specify any of the I/O libraries as core dependencies, even though they are supported. I'd follow the same principle here (maybe with the exception of Zarr).