Limit inode consumption of output data

[ankitpatnala]

Is your feature request related to a problem? Please describe.

For IMERG data, the grid used is of finer resolution reduced Gaussian grid with much more grid points compared to ERA5. Right now, the target_times which repeats for all grid points is concatenated.
It gets difficult to operate on such a large array, i.e. when we generate zarr file containing inference samples for 1 month with 6 hour timestamps.

Describe the solution you'd like

currently chunk size can be changed using the module level variable utils.io.CHUNK_N_SAMPLES. To cope better with streams as described above, some logic should be implemented keep the number of chunks reasonable. Two possible solutions could be:

A) Define the number of chunks as a constant instead of the size of a chunk => scale chunksize per stream accordingly
B) Add optional parameter chunk_size to stream config files: if it is present it is passed to the relevant method (utils.io.ZarrIO._write_arrays) otherwise utils.io.CHUNK_N_SAMPLES is used

Describe alternatives you've considered

Using zarr3 is currently not possible due to incompatibility with anemoi-dataset. However once this is fixed zarr 3 should be used additionally to the ability to specify stream chunk size.

Additional context

Option A) would add less complexity to the code/configuration and is easier to implement. option B) would provide more control to the user. As this ultimately is more of a temporary fix to limit the inode consumption until the capabilities of zarr3 can be used, I would argue option A) should be the preferred solution.

Organisation

JSC

[kacpnowak]

Hi Ankit, it can be handled by xarray. As our model output is now compatible with xarray you can run indexing, computing statistical function, all in lazy fashion. You can see more in xarray docs https://docs.xarray.dev/en/latest/internals/internal-design.html

[csjfwang]

Yes, this happened when we evaluate the model on IMERG Dataset.

For 1 month evaluation, the generated .zarr file will directly burn out all the storage budget (3000K/4000K Inodes) under the project, and this occurred while the evaluation was still in progress.
And then, a quota error will be raised: OSError: [Errno 122] Disk quota exceeded.

[ankitpatnala]

Hi Ankit, it can be handled by xarray. As our model output is now compatible with xarray you can run indexing, computing statistical function, all in lazy fashion. You can see more in xarray docs https://docs.xarray.dev/en/latest/internals/internal-design.html

@kacpnowak Thanks! I will give it a try.
Any hack to sort out the inodes issue that @csjfwang mentioned?

[clessig]

There's the proposal to use zarr3 (where chunking and sharding is separated) to avoid the issue.

What is the size of the zarr file?

[csjfwang]

There's the proposal to use zarr3 (where chunking and sharding is separated) to avoid the issue.

What is the size of the zarr file?

For example, under the zarr file, for IMERG dataset, the targets/prediction folder might have 600K-1000K files. I cannot remember the size of the .zarr file coz I've deleted them to avoid causing quota error to others.

[csjfwang]

@clessig @ankitpatnala @kacpnowak

Update:

For a two-day evaluation of IMERG dataset, start_date=202201010000, end_date=202201030000, the created .zarr file has:

File numbers: 172788 (~173K)
Total size: ~0.5GB

[clessig]

@clessig @ankitpatnala @kacpnowak

Update:

For a two-day evaluation of IMERG dataset, start_date=202201010000, end_date=202201030000, the created .zarr file has:

File numbers: 172788 (~173K) Total size: ~0.5GB

As a work around, you can adjust the chunk size here and in the lines below:

WeatherGenerator/src/weathergen/utils/validation_io.py

Line 113 in 2ec1b0f

"sources", data=source_k, chunks=(1024, *source_k.shape[1:])

@grassesi : Please keep in mind in the revised version that there might be a need to overwrite the chunk size on a case-by-case basis.

[grassesi]

@grassesi : Please keep in mind in the revised version that there might be a need to overwrite the chunk size on a case-by-case basis.

There is a module level constant that can be changed if needed: utils.io.CHUNK_N_SAMPLES

[clessig]

@grassesi : Please keep in mind in the revised version that there might be a need to overwrite the chunk size on a case-by-case basis.

There is a module level constant that can be changed if needed: utils.io.CHUNK_N_SAMPLES

Given the situation we see above, we should be able to overwrite it for a specific stream. The stream config doesn't seem like a great place for it since it's mainly scientific params but it would still be the best place at the moment.

[grassesi]

I hijacked this issue since we need a solution for this until zarr3 comes around