IFS HRES Zarr dataset contains NaNs in geopotential

[jacobbieker]

Hi,

There seems to be an issue with the HRES Zarr 0.25degree forecast dataset, where there is NaNs in the geopotential, specifically in the 850hPa level. The exact amount varies on the random times, but is always at least 24ish NaN values.

    import numpy as np
    import xarray as xr

    ds = xr.open_zarr('gs://weatherbench2/datasets/hres/2016-2022-0012-1440x721.zarr')
    print(ds)
    ds = ds.isel(prediction_timedelta=2)
    print(ds)
    times = np.random.choice(
        ds.time.values, 1000)
    ds = ds.sel(time=times)["geopotential"].load()
    for level in ds.level.values:
        level_ds = ds.sel(level=level)
        print(f"Level {level} Has NaNs: {np.isnan(level_ds).sum().values} Is Finite: {np.isfinite(level_ds).sum().values}")

The output generally looks like:

Level 50 Has NaNs: 0 Is Finite: 519120000
Level 100 Has NaNs: 0 Is Finite: 519120000
Level 150 Has NaNs: 0 Is Finite: 519120000
Level 200 Has NaNs: 0 Is Finite: 519120000
Level 250 Has NaNs: 0 Is Finite: 519120000
Level 300 Has NaNs: 0 Is Finite: 519120000
Level 400 Has NaNs: 0 Is Finite: 519120000
Level 500 Has NaNs: 0 Is Finite: 519120000
Level 600 Has NaNs: 0 Is Finite: 519120000
Level 700 Has NaNs: 0 Is Finite: 519120000
Level 850 Has NaNs: 54 Is Finite: 519119946
Level 925 Has NaNs: 0 Is Finite: 519120000
Level 1000 Has NaNs: 0 Is Finite: 519120000

[raspstephan]

Yes, this is a know issue. I am not exactly sure what the reason is but you just need to make sure to handle it downstream.