using lfric2lfric for C896 to UKV

[cjohnson-pi]

Are there any linked Issues or Pull Requests?

• Update App calls to partition type api #324
• Lfric2lfric read order #275
• Lfric2lfric: interpolate W2 fields #238
• Lfric2lfric multi cpu #201

Brief description

A number of tickets related to lfric2lfric are in progress. A branch that includes the above PRs has been created to enable testing: main...ukmo-juan-castillo:lfric_apps:lfric2lfric_horizontal_merged

The purpose of this issue is to document the changes that are required so that the branch can regrid from C896 (global) to UKV LAM (regional).

Further details of the issue.

Local changes to this branch are at /home/users/christine.johnson/git/ukmo-juan-castillo/lfric_apps

Changes required are:

• The gen_weights tasks doesn't work correctly with a multigrid mesh file, so single mesh files are specified in
  rose-stem/templates/runtime/generate_runtime_gen_weights.cylc. A new C896 mesh file config is created at rose-stem/app/mesh/opt/rose-app-C896.conf.

• In applications/lfric2lfric/source/initialisation/lfric2lfric_init_mesh.f90 the mesh reading tasks are reordered and clear global mesh calls are included, to prevent out of memory problems. The old code reads the target mesh and then the source mesh and then creates the local meshes. The new code reads the source mesh, creates the local mesh, clears the global mesh. Then reads the target mesh, creates the local mesh, clears the global mesh.

• In rose-stem/app/generate_weights/opt/rose-app-lfric2lfric.conf this now has INT_METHOD=conserve2nd rather than conserve to enable a higher order horizontal interpolation method.

• In rose-stem/templates/default_directives.cylc the gen_weights_tasks is increased from 8 to 16.

• To prevent the source file being copied to the working directory (which is painfully slow for a large file!), mode=symlink is added to [file:...] in rose-stem/app/lfric2lfric/opt/rose-app-clim_gal9_ral_C896_ukv.conf. This creates a soft link from the working directory to the source file directory.

• In rose-stem/site/common/lfric2lfric/tasks_lfric2lfric.cylc task oasis_C896_ukv_lam uses mpi_parts 768, and memory 250.

• In rose-stem/app/lfric2lfric/opt/rose-app-multi-cpu_C896_ukv.conf the cubedsphere partitioner uses panel_xproc=8 and
  panel_yproc=16.

• rose-stem/app/lfric2lfric/file/iodef.xml has been adjusted to include fields defined with grid_ref=full_level_face_grid and half_level_face_grid in the checkpoint_read, rather than domain_ref=checkpoint_W3 and checkpoint_Wtheta. This is because the C896 source file is an LFRic diagnostics file, rather than a checkpoint file.

• The required meshes are only created by the suite if they are listed under resolution in rose-stem/site/common/lfric2lfric/tasks_lfric2lfric.cylc rather than dst_mesh and src_mesh.

In summary:

Source	Target	mpi	memory (GB)	panel_xproc	panel_yproc	number fields regridded	time taken
C896 diagnostics file	UKV (1024x960)	768	250	8	16	9 (theta, rho, p, u_in_w3, v_in_w3, mr, mcl, mv, mcf)	10 mins
C896 diagnostics file	UKV (1024x960)	384	250	8	8	1 (theta)	3 mins
C224 checkpoint file	648x648	36	250	2	3	3 (theta, rho, zh)
C224 checkpoint file	672x672	36	250	2	3	3 (theta, rho, zh)
C224 checkpoint file	UKV (1024x960)	48	250	2	4	3 (theta, rho, zh)
C224 checkpoint file	UKV (1024x960)	384	250	8	8	33 theta, rho, m_v, m_cl, m_r, m_ci, area_fraction, bulk_fraction, liquid_fraction, frozen_fraction, tile_snow_mass, n_snow_layers, snow_depth, tile_snow_r_grain, snowpack_density, snow_under_canopy, canopy_water, tile_temperature, snow_layer_thickness, snow_layer_ice_mass, snow_layer_liq_mass, snow_layer_temp, snow_layer_rgrain, soil_temperature, soil_moisture, zh, z0msea, dd_ff_cb, snow_soot, soil_sat_frac, water_table, wetness_under_soil, soil_thermal_cond	12 mins

Remaining problems:

• The code is successful in writing data to a UKV checkpoint file. However, there seems to be a problem with the coordinates when trying to plot the data. I'm using iris.load to load in the file, and this works fine for the global file. But for the lam file I can't get it to read the coordinates. It says "Assuming 'topology_dimension'=0, from the attached connectivities of the mesh variable checkpoint_Wtheta. However, checkpoint_Wtheta:topology_dimension = 2 -- ignoring this as it is inconsistent."
  So I'm using netcdf4 to read the coordinates which seems to be fine for plotting.