Add converted CSV files

[sjnoone]

Hi,
I have converted all the tables .dat to more user friendly .csv files and saved them in tables_csv directory on this branch. Note that the table_definitions are already in .csv format. Please can you check over, review then merge into main.

best
Simon

[EddyCMWF]

Hi @sjnoone ,
Just to let you know we will discuss this PR in our next meeting, we need to double check that operational things will not break before we merge.
Thanks,
Eddy

[garciampred]

Hi,

There seem to be non UTF characters in the CSV files:

(cadsobs) garciam@bender:~/.cdsobs/cdm-obs$ grep -axv -r '.*' tables_csv/
tables_csv/observed_variable.csv:67,radiation,atmospheric,,global radiation,W m-2,"The part of the global radiation in-cident on a surface orthogonal (or �normal�) to the sun�s beam, which originates from the solid angle subtended by the sun�s disk. It is also called direct normal irradiance."
tables_csv/observed_variable.csv:191,methane,atmospheric,upper-air,methane total column,mol/m^2,retrieved total vertical CH4 column between boundaries of the altitude grid. The multiplication factor to convert mol/m2 to molec/cm2 is 6.02214�10^19.
tables_csv/spatial_representativeness.csv:6,"Drainage area - An area (also known as �catchment�) having a common outlet for its surface runoff, in km2"
tables_csv/observation_code_table.csv:0,BUFR,0 20 003,Present weather,230,Duststorm or sandstorm with temperature below 0 �?C
tables_csv/units.csv:301,parts per thousand,\�,NULL,,,,,,,,,,,,,,,,,,,

Can you fix this please? Also, I think it would be better to keep the files in the same folder (tables) and delete the old .dat files.

Note that the table_definitions are already in .csv format. Please can you check over, review then merge into main.

I'm afraid they are not, the file names have the .csv extension, but the files are tab separated files. Could you transform them too please?

[sjnoone]

Hi,

I have fixed the non UTF characters and converted table_definitions to actual comma sep files .csv . I have also removed all the .dat files etc.

[garciampred]

Thanks. The catalogue manager tests are all passing now, and I checked all the tables in a separate script just in case and they seem OK. I am using on_bad_lines="error" to be sure that all the lines have the same number of columns. I just had to fix one missing comma in one file.

The CI is not going to pass as it would need the PR ecmwf-projects/cads-obs-catalogue-manager#95 to be merged first, so don't worry about it. I think this is ready to be merged.