Add xarray/dask configuration to config.yml

[bschroeter]

So that we don't have to manually edit the source to change xarray/dask resources, we should add a section to the config.yml that maps to the keyword arguments of the cluster and open_mfdataset stages of the code.

This will allow for faster development, as well as user-specified resources as needed.

For example:

cluster:
  n_workers: 1
  processes: true
open_mfdataset:
  parallel: true
  chunks:
    time: 24
    latitude: -1
    longitude: -1

Then:

# ...assume config is loaded as per usual

cluster = LocalCluster(**config.get("cluster"))
dataset = xr.open_mfdataset(file_list, **config.get("open_mfdataset"))

See:

• https://docs.dask.org/en/stable/deploying-python.html#distributed.deploy.local.LocalCluster
• https://docs.xarray.dev/en/stable/generated/xarray.open_mfdataset.html

[abhaasgoyal]

Sounds like a great idea. Would be working on this after #17 gets merged.

[bschroeter]

This is partially completed in #17 whereby we have these fields available in the config.yaml but they are not used in the software (which is still hardcoded).

Still something to do.