Add capability to convert on the fly GSI netCDF diagnostics to IODA format

[CoryMartin-NOAA]

Description

This PR will add the capability to convert GSI netCDF diagnostic files to IODA-compliant netCDF files, which will also facilitate computing summary stats from GSI leveraging new, IODA-based tools.

Resolves #4145

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO

• Does this change require a documentation update? NO

• Does this change require an update to any of the following submodules? YES

  • EMC verif-global
  • GDAS Changes needed to include GSI ncdiag to IODA GDASApp#1934
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

Example:

• Clone and build on Ursa
• Cycled test on Ursa

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[CoryMartin-NOAA]

This will produce, for example in:
COMROOT/C96_atm3DVar/gdas.20211221/00/products/atmos/anlmon

gdas.t00z.atmos_gsi_ioda_diags.tgz  gdas.t00z.atmos_gsi_iodastat.tgz  sfc_ps_2021122100_output_atmos_gsi.nc

Where
gdas.t00z.atmos_gsi_ioda_diags.tgz contains numerous IODA files converted from GSI ncdiags:

scatwind_obs_2021122100.nc4.gz
abi_g17_obs_2021122100.nc4.gz
seviri_m11_obs_2021122100.nc4.gz
seviri_m08_obs_2021122100.nc4.gz
sfc_ps_obs_2021122100.nc4.gz
sfc_uv_obs_2021122100.nc4.gz
sfcship_uv_obs_2021122100.nc4.gz
sondes_q_obs_2021122100.nc4.gz
avhrr_n18_obs_2021122100.nc4.gz
sfc_q_obs_2021122100.nc4.gz
ompstc8_npp_obs_2021122100.nc4.gz
aircraft_tsen_obs_2021122100.nc4.gz
aircraft_q_obs_2021122100.nc4.gz
sfcship_tv_obs_2021122100.nc4.gz
sondes_ps_obs_2021122100.nc4.gz
atms_npp_obs_2021122100.nc4.gz

gdas.t00z.atmos_gsi_iodastat.tgz currently only contains sfc_ps_2021122100_output_atmos_gsi.nc

[CoryMartin-NOAA]

@RussTreadon-NOAA @ADCollard one thing here is that the GSI diags are separate files for _ges and _anl, and I don't know which one ends up getting saved (depends on what is processed last). Do we need to find a way to keep both GsiHofXBc from ges and anl, can we just use ges? Thoughts?

[CoryMartin-NOAA]

Note to self for next week: I'm thinking I should move _ges to one directory and _anl to another directory, run the proc_gsi_ncdiag code twice, once on each directory, and save two tarballs of the raw IODA files, and perhaps only run the stats code on one of them (perhaps ges only?)

[RussTreadon-NOAA]

@RussTreadon-NOAA @ADCollard one thing here is that the GSI diags are separate files for _ges and _anl, and I don't know which one ends up getting saved (depends on what is processed last). Do we need to find a way to keep both GsiHofXBc from ges and anl, can we just use ges? Thoughts?

@CoryMartin-NOAA. I checked a JEDI atmstat tarball from the prjedi. The observation specific netcdf files contain hofx0 and hofx1.

The prjedi atmvar.yaml has

variational:
  minimizer:
    algorithm: DRPCG
  iterations:
  - ninner: 50
    gradient norm reduction: 1e-10
    test: true
    geometry:
      fms initialization:
        namelist filename: ./fv3jedi/fmsmpp.nml
        field table filename: ./fv3jedi/field_table
      akbk: ./fv3jedi/akbk.nc4
      layout:
      - 8
      - 8
      npx: 193
      npy: 193
      npz: 127
      vert coordinate: logp
    diagnostics:
      departures: bkgmob
final:
  diagnostics:
    departures: anlmob

It's my impression that bkgmob creates hofx0. This is the ges H(x) since we only have a single outer loop. The anlmob creates hofx1. This is the anl H(x). Seems we have both the ges and anl H(x) is the JEDI diagnostic file. Not sure I'm answering your question.