Issues about "unknown dimensionality" related to the variable sea_surface_height

[Angie-P]

Dear Opendrift community,
I've tried to run a simulation with oceandrift using as readers a .nc model output from shyfem and a shapefile for the landmask.
The run didn't work because an error occurred, and the output in the console was the following:
14:18:55 INFO opendrift.models.basemodel:1844: Backwards simulation, starting from last seeded element
14:18:55 INFO opendrift.models.basemodel:1927: Backwards simulation, starting at time of last seeded element
14:18:56 INFO opendrift.models.basemodel:935: Using existing reader for land_binary_mask
14:18:56 INFO opendrift.models.basemodel:946: All points are in ocean
14:18:56 WARNING opendrift.models.basemodel:729: Seafloor check not being run because environment is missing. This will happen the first time the function is run but if it happens subsequently there is probably a problem.
14:18:56 INFO opendrift.models.basemodel:2035: 2018-12-31 03:00:00 - step 1 of 72 - 10000 active elements (0 deactivated)
14:18:56 ERROR opendrift.readers.unstructured.shyfem:213: unknown dimensionality
14:18:56 WARNING opendrift.models.basemodel:2133: The simulation stopped before requested end time was reached.
14:18:56 INFO opendrift.models.basemodel:2135: ========================
14:18:56 INFO opendrift.models.basemodel:2136: End of simulation:
14:18:56 INFO opendrift.models.basemodel:2137: 'sea_surface_height'
14:18:56 INFO opendrift.models.basemodel:2138: Traceback (most recent call last):
File "C:\Users\angel\opendrift\opendrift\models\basemodel_init_.py", line 2068, in run
self.env.get_environment(list(self.required_variables),
File "C:\Users\angel\opendrift\opendrift\models\basemodel\environment.py", line 699, in get_environment
env_tmp[var][0:len(missing_indices)]).astype('float32')
~~~~~~~^^^^^
KeyError: 'sea_surface_height'

14:18:56 INFO opendrift.models.basemodel:2139: 'The simulation stopped before requested end time was reached.'

An exception has occurred, use %tb to see the full traceback.

SystemExit: Simulation aborted. 'The simulation stopped before requested end time was reached.'

From what I understand, there seems to be an issue that deals with the function get_environment. In particular, the error seems to concern the dimensionality of the variable sea_surface_height. I looked in the file shyfem.py at the point where the error occurred and there are only two cases considered: the one for variables which have exactly 1 dimension and the one for variables which have more than 2 dimensions. An option considering the case of 2 dimensions seems not to be contemplated, and this is exactly the case of sea_surface_height, which dimensions are "nodes" (since the model has an unstructured grid) and "time", and that by definition can't have "levels" as a third dimension. So, how could I import this variable and use it in the simulation without getting this error?

I don't even understand the warning that tells that "environment is missing", since when I write o.list_config I get:
Environment variables:

sea_floor_depth_below_sea_level
sea_surface_height
x_sea_water_velocity
y_sea_water_velocity
1) model_dec_2018.nc

land_binary_mask
1) shape

So it seems that the variables from the readers have been correctly added to the oceandrift simulation instance.

The simulation that I'm trying to run takes place in the Venice lagoon, an environment in which taking into account the role of the tide and therefore the dynamic changes on the sea level is fundamental, especially because there are regions that are periodically immersed and emerged, that's why I really need to use the variable sea_surface_height in the simulation.

Any suggestion would be really appreciated.
Thanks a lot in advance,
Angelica

[knutfrode]

Hi,

Yes, it seems the error is that your variables have exactly 2 dimensions. The following error about environment is a bit misleading, as it is a downstream consequence.

Could you try to change the line
if len(dvar.shape) > 2:
to
if len(dvar.shape) >= 2:

If that does not work, I am not sure how to proceed, as I have never used this reader or worked with SYFEM data myself.
It could be that @gauteh or @manuelaghito would have some ideas here.

[Angie-P]

Thank you for your quick reply,
I've changed the line as you suggested, and the run didn't stop this time, but the reader has been discarded at some point and I got warning and errors on the console. The exact output was:
15:38:59 INFO opendrift.models.basemodel:1844: Backwards simulation, starting from last seeded element
15:38:59 INFO opendrift.models.basemodel:1927: Backwards simulation, starting at time of last seeded element
15:39:00 INFO opendrift.models.basemodel:935: Using existing reader for land_binary_mask
15:39:00 INFO opendrift.models.basemodel:946: All points are in ocean
15:39:00 WARNING opendrift.models.basemodel:729: Seafloor check not being run because environment is missing. This will happen the first time the function is run but if it happens subsequently there is probably a problem.
15:39:00 INFO opendrift.models.basemodel:2035: 2018-12-31 03:00:00 - step 1 of 72 - 10000 active elements (0 deactivated)
15:39:00 INFO opendrift.models.basemodel.environment:665: ========================
15:39:00 ERROR opendrift.models.basemodel.environment:666: slicing expression exceeds the number of dimensions of the variable
Traceback (most recent call last):
File "C:\Users\angel\opendrift\opendrift\models\basemodel\environment.py", line 641, in get_environment
reader.get_variables_interpolated(
File "C:\Users\angel\opendrift\opendrift\readers\basereader\variables.py", line 904, in get_variables_interpolated
env, env_profiles = self.get_variables_interpolated_xy(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\angel\opendrift\opendrift\readers\basereader\variables.py", line 770, in get_variables_interpolated_xy
env, env_profiles = self.get_variables_interpolated(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\angel\opendrift\opendrift\readers\basereader\unstructured.py", line 61, in get_variables_interpolated
env = self.get_variables(variables, time, reader_x, reader_y, z)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\angel\opendrift\opendrift\readers\unstructured\shyfem.py", line 192, in get_variables
block = dvar[indx_nearest,
^^^^^^^^^^^^^^^^^^
File "src\netCDF4_netCDF4.pyx", line 4943, in netCDF4._netCDF4.Variable.getitem
File "C:\Users\angel\anaconda3\envs\opendrift\Lib\site-packages\netCDF4\utils.py", line 338, in _StartCountStride
raise ValueError("slicing expression exceeds the number of dimensions of the variable")
ValueError: slicing expression exceeds the number of dimensions of the variable
15:39:00 INFO opendrift.models.basemodel.environment:668: ========================
15:39:00 INFO opendrift.models.basemodel:2035: 2018-12-31 02:00:00 - step 2 of 72 - 10000 active elements (0 deactivated)
15:39:00 INFO opendrift.models.basemodel.environment:665: ========================
15:39:00 ERROR opendrift.models.basemodel.environment:666: slicing expression exceeds the number of dimensions of the variable
Traceback (most recent call last):
File "C:\Users\angel\opendrift\opendrift\models\basemodel\environment.py", line 641, in get_environment
reader.get_variables_interpolated(
File "C:\Users\angel\opendrift\opendrift\readers\basereader\variables.py", line 904, in get_variables_interpolated
env, env_profiles = self.get_variables_interpolated_xy(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\angel\opendrift\opendrift\readers\basereader\variables.py", line 770, in get_variables_interpolated_xy
env, env_profiles = self.get_variables_interpolated(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\angel\opendrift\opendrift\readers\basereader\unstructured.py", line 61, in get_variables_interpolated
env = self.get_variables(variables, time, reader_x, reader_y, z)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\angel\opendrift\opendrift\readers\unstructured\shyfem.py", line 192, in get_variables
block = dvar[indx_nearest,
^^^^^^^^^^^^^^^^^^
File "src\netCDF4_netCDF4.pyx", line 4943, in netCDF4._netCDF4.Variable.getitem
File "C:\Users\angel\anaconda3\envs\opendrift\Lib\site-packages\netCDF4\utils.py", line 338, in _StartCountStride
raise ValueError("slicing expression exceeds the number of dimensions of the variable")
ValueError: slicing expression exceeds the number of dimensions of the variable
15:39:00 INFO opendrift.models.basemodel.environment:668: ========================
15:39:00 WARNING opendrift.models.basemodel.environment:673: Reader model_dec_2018.nc is discarded after failing more times than allowed (1)

And then:
15:39:00 INFO opendrift.models.basemodel:2035: 2018-12-31 01:00:00 - step 3 of 72 - 10000 active elements (0 deactivated)
and so on until
15:39:09 INFO opendrift.models.basemodel:2035: 2018-12-28 04:00:00 - step 72 of 72 - 10000 active elements (0 deactivated)

[knutfrode]

Looking at the reader code, changing >2 to >=2 might perhaps not be correct. Instead, probably seems we need to deal with two dimensions (time and node, but no level) as a separate case.
As untested pseudo-code, I would suggest inserting something like this:

elif len(dvar.shape) == 2:
                level_ind = self.__nearest_level__(z)

                # Reading the smallest block covering the actual data
                block = dvar[indx_nearest,
                             slice(nodes.min(),
                                   nodes.max() + 1),]

                # Picking the nearest value
                variables[var] = block[
                        nodes - nodes.min(),
                        ]

Thus you could try to insert this part in the reader.
If it does not work, perhaps @loriscalgaro would have a suggestion, as I believe he is using SHYFEM output.

[loriscalgaro]

Hello,
I have been using SHYFEM data, however only for variables with 3 dimensions ("nodes", "time", level") such as temperature, salinity, and water currents. My simulations assumed a constant bathymetry, using an interpolated version of "sea_floor_depth_below_sea_level" to a 0.0025° regular grid and imported using "reader_netCDF_CF_generic.Reader".
Best regards

Loris

[Angie-P]

Thank you for your replies,
I've tried your solution @knutfrode and it seems to work because the run doesn't stop and there aren't errors anymore on the console. The only warning that is still present is "Seafloor check not being run because environment is missing. This will happen the first time the function is run but if it happens subsequently there is probably a problem", I don't know if it is is something to care about

[knutfrode]

Ok, this sounds promising.
And that warning is also not important, and could probably be made clearer.

If you seem to get meaningful results, we can update the reader permanently.
A pull request is then welcome, otherwise I can add it as I wrote it above, but then without further testing by me.

[Angie-P]

I've just created a pull request, sorry for the late I was not used to these operations on GitHub and I was learning how to do it.
Thank you again for your feedback

[knutfrode]

Great, thank you for the contribution.
Then closing this issue.