Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 #1355

jshaw35 · 2025-08-05T22:39:39Z

brian-eaton · 2025-08-12T00:25:34Z

src/physics/cam/cospsimulator_intr.F90

    cospstateIN%phalf           = state%pint(:ncol,ktop:pverp)
    cospstateIN%hgt_matrix      = zmid
-    cospstateIN%hgt_matrix_half = zint
+    cospstateIN%hgt_matrix_half = zint(1:ncol,2:nlayp) ! COSP wants half levels without model top


This is very confusing. It appears that cosp_change_vertical_grid does want the half-level heights at the bottom of each layer. But there is a lot of code still that assumes hgt_matrix_half is dimensioned (npoints,nlevels+1). For example in function COSP_SIMULATOR we find

rttovIN%h_surf => cospgridIN%hgt_matrix_half(:,cospIN%Nlevels+1)

And in subroutine lidar_column we find

real(wp),intent(in),dimension(npoints,nlevels+1) :: & zlev_half ! Model half levels

I think having hgt_matrix_half only contain the bottom half levels is confusing because you don't know without searching code whether the array contains the top or the bottom levels. It's also confusing because it's unexpected. I expect the heights in hgt_matrix_half to correspond to the pressures in phalf.

My suggestion is that hgt_matrix_half be left as it is, and that the calls to cosp_change_vertical_grid be modified to pass the correct section containing the bottom half levels, e.g.,

call cosp_change_vertical_grid(cloudsatIN%Npoints,1,cloudsatIN%Nlevels, & cospgridIN%hgt_matrix(:,cloudsatIN%Nlevels:1:-1), & cospgridIN%hgt_matrix_half(:,cloudsatIN%Nlevels+1:2:-1),betamol_in, &

@brian-eaton thanks for digging into this! I had been comparing with the current COSP version (https://github.com/CFMIP/COSPv2.0) rather than the tag that is currently used in CESM2. hgt_matrix_half is consistently dimensioned (npoints,nlevels+1) there and the calls to lidar_column and cosp_change_vertical_grid do not use the height at the top of the model.

So it seems that the changes I requested require a more recent version of COSP. @cacraigucar does it then make the most sense to wait to merge this PR until we have an official COSP tag with pending updates that do not involve RTTOV? Then this PR can include a change in the git-fleximod settings to point to an appropriate COSP tag that works with these changes. So next steps would be:

Merge pending changes to COSP that do not involve RTTOV. Produce a tag for CESM.

Point to that tag in this PR, merge.

Produce regression test values.

Test COSP-RTTOV PR and evaluate against regression tests.

Yes, that sounds like a plan. We will need to proceed cautiously before we move forward with step 2-3 where we commit the new COSP (with the bug fix). It probably will need to have some testing by @cecilehannay before we bring it in. COSP is used at times for the CESM3 science runs, and we need to make sure it is working properly before we commit it.

Hi @jshaw35. I've taken a look at the current head of the master branch of COSPv2.0 and see that the dimension of hgt_matrix_half has been changed to (npoints,nlevels) and that it now contains just the bottom interfaces. I agree that the path forward is to bring in an updated COSP tag which will require the changes in this PR along with appropriate changes to how hgt_matrix_half is initialized from the local variable zint.

Note also that cospsimulator_intr.F90 has an option to output the variables that are COSP inputs to the cosp_histfile_aux stream. The addfld call for ZLEV_HALF_COSP needs the vertical dimension updated.

Note for self. Relevant changes between the CESM tag and the master branch of COSP are here:
CFMIP/COSPv2.0@master...CESM_v2.1.4
CFMIP/COSPv2.0@CESM_v2.1.4...CFMIP:COSPv2.0:master
I will need to update the cospsimulator_intr.F90 to allocate and assign both hgt_matrix_half and phalf without the model top level when we use a more current tag of COSP.

brianpm

I agree that bringing in the updated version of COSP is the right move. I think it will also allow us to add some additional diagnostics (which I will try to look into separately).

Thanks @jshaw35 and @brian-eaton for moving this forward.

cacraigucar · 2025-08-26T18:02:56Z

@jshaw35 - Just a reminder that as soon as you add the COSP external update (and you have tested it with your mods), we can have the reviewers test and approve this PR and we can move it forward.

jshaw35 · 2025-08-27T18:31:09Z

@cacraigucar I have just updated .gitmodules to reference the new COSP tag (most current COSP with no RTTOV). We will need to update the COSP driver following the discussion above.

cacraigucar · 2025-08-27T19:31:41Z

@cacraigucar I have just updated .gitmodules to reference the new COSP tag (most current COSP with no RTTOV). We will need to update the COSP driver following the discussion above.

@jshaw35 I assume that you will be making the change required?

jshaw35 · 2025-08-27T23:43:32Z

@cacraigucar I have just updated .gitmodules to reference the new COSP tag (most current COSP with no RTTOV). We will need to update the COSP driver following the discussion above.

@jshaw35 I assume that you will be making the change required?

@cacraigucar yes! I don't expect it to take too long but I am sure yet when I will have the time. Hopefully sooner than later but almost certainly by September 15.

@brian-eaton the new tag for COSP should now align with the changes in this PR, resolving your points. I will review any additional changes needed to the driver since the cosp2.1.4 tag and then request an additional review.

…s are now being produced!

brianpm · 2025-09-04T14:49:35Z

@jshaw35 -- If you have a chance, could you also see if the MODIS cloud phase diagnostics are included in the updated version of COSP? I think they should be, but we might need to add them to the CAM outputs. See, e.g., Duran et al. 2025.

jshaw35 · 2025-09-04T15:20:17Z

@jshaw35 -- If you have a chance, could you also see if the MODIS cloud phase diagnostics are included in the updated version of COSP? I think they should be, but we might need to add them to the CAM outputs. See, e.g., Duran et al. 2025.

@brianpm , yes I don't think this should be a problem. I've just put in the new warm rain diagnostics and can add the joint histograms you're requesting. To be clear, you want the "modis_LWP_vs_ReffLIQ" and "modis_IWP_vs_ReffICE" variables, anything else?

jshaw35 · 2025-09-04T18:00:50Z

@brianpm I've just added the MODIS LWP-REFF histograms in addition to the warm rain diagnostics from Michibata (2019). The fields are not modified by the CESM coupler (besides separating variables, naming, etc) and I've tried to follow existing naming conventions and supply units/bounds appropriately. Would you mind reviewing the output?

Average fields from a month of CAM can be found here: /glade/work/jonahshaw/COSP_RTTOV_files/COSP2.1.8_tests/cosp2.1.8_newvars.nc

brianpm · 2025-09-04T18:14:00Z

Thanks @jshaw35 !

Looking at the MODIS simulator, it looks like they have histograms of:

tau-CTP ✅ (already in CAM)
tau-CTP for liquid clouds ❓(not sure of status)
tau-CTP for ice clouds ❓
LWP-ReffLiq ☑︎ (you have added)
IWP-ReffIce ☑︎
tau-ReffIce ❓
tau-ReffLiq ❓

I think we should add all of them if we can. What do you think?

jshaw35 · 2025-09-04T18:24:03Z

Thanks @jshaw35 !

Looking at the MODIS simulator, it looks like they have histograms of:
* tau-CTP ✅ (already in CAM)

* tau-CTP for liquid clouds ❓(not sure of status)

* tau-CTP for ice clouds ❓

* LWP-ReffLiq ☑︎ (you have added)

* IWP-ReffIce ☑︎

* tau-ReffIce ❓

* tau-ReffLiq ❓
I think we should add all of them if we can. What do you think?

@brianpm easy to do it now! I believe those additional fields are internally named in COSP as:
modis_Optical_Thickness_vs_ReffICE (already CLRIMODIS in CESM)
modis_Optical_Thickness_vs_ReffLIQ (already CLRLMODIS in CESM)
modis_Optical_Thickness_vs_Cloud_Top_Pressure_Liq
modis_Optical_Thickness_vs_Cloud_Top_Pressure_Ice

If this looks right to you I will go ahead and add the last two histograms. Any preferences on how to name the last two fields?

brianpm · 2025-09-04T18:31:29Z

Yeah, that's exactly what I was looking at! Thanks!!

jshaw35 · 2025-09-05T00:23:23Z

@brianpm all MODIS cloud histograms should now be available as output

tau-CTP (CLMODIS)
tau-CTP for liquid clouds (CLMODIS_LIQ)
tau-CTP for ice clouds (CLMODIS_ICE)
LWP-ReffLiq (IWP_REFFCLI_MODIS)
IWP-ReffIce (LWP_REFFCLW_MODIS)
tau-ReffIce (CLRIMODIS)
tau-ReffLiq (CLRLMODIS)

Would you mind reviewing the outputs? Average fields from a month of CAM can be found here: /glade/work/jonahshaw/COSP_RTTOV_files/COSP2.1.8_tests/cosp2.1.8_newvars2.nc

brianpm · 2025-09-05T19:21:16Z

@jshaw35 -- Looking through your file, two small things.

First I think the units for the coordinate variables might be wrong:

cosp_prs units = hPa, but values are in Pa
cosp_refliq and cosp_refice units say microns, but it looks like the values are probably meters (?)
cosp_iwp_modis units = "kg m-2" but values go up to 10.5, so I'm guessing this is "g m-2"?
cosp_lwp_modis units = "kg m-2" ... also goes up to 10.125... maybe that's correct, but seems really high so maybe it is also "g m-2"

Second, I don't see CLRIMODIS and CLRLMODIS in the file.

jshaw35 · 2025-09-05T22:32:32Z

@brianpm thanks for reviewing! Comments in-line below.

First I think the units for the coordinate variables might be wrong:
* `cosp_prs` units = hPa, but values are in Pa

Yep, you're totally right here. Now that I think about it, this has been in the CESM code for years.

* `cosp_refliq` and `cosp_refice` units say microns, but it looks like the values are probably meters (?)

Yep again!

* `cosp_iwp_modis` units = "kg m-2" but values go up to 10.5, so I'm guessing this is "g m-2"?

* `cosp_lwp_modis` units = "kg m-2" ... also goes up to 10.125... maybe that's correct, but seems really high so maybe it is also "g m-2"

These are actually correct. The upper bound on the LWP and IWP is 20 kg m-2 for both liquid and ice, so the bin center appears to be abnormally high. I think this is just meant to include all large values. The bounds arrays from cosp_config.F90 make things a bit clearer:

    real(wp),parameter,dimension(nLWP+1) :: &
         LWP_binBounds = (/0., 0.01, 0.03, 0.06, 0.10, 0.15, 0.25, 20.0/) ! kg/m2
    real(wp),parameter,dimension(nIWP+1) :: &
         IWP_binBounds = (/0., 0.02, 0.05, 0.10, 0.20, 0.40, 1.00, 20.0/) ! kg/m2

Second, I don't see CLRIMODIS and CLRLMODIS in the file.

I've updated the file with them now from the same run (so units will still be off).

jshaw35 · 2025-09-15T20:21:15Z

@cacraigucar @brianpm @brian-eaton I think this PR is ready for review unless their are additional outstanding issues.

brian-eaton · 2025-09-17T13:07:32Z

src/physics/cam/cospsimulator_intr.F90

+       call addfld('CS_WR_CFODD_REFF_SMALL',(/'cosp_cfodd_dbze','cosp_cfodd_icod'/), 'A', 'fraction', &
+            '# of CFODD (05 < Reff < 12 micron)', flag_xyfill=.true., fill_value=R_UNDEF)
+       call addfld('CS_WR_CFODD_REFF_MEDIUM',(/'cosp_cfodd_dbze','cosp_cfodd_icod'/), 'A', 'fraction', &
+            '# of CFODD (12 < Reff < 18 micron)', flag_xyfill=.true., fill_value=R_UNDEF)
+       call addfld('CS_WR_CFODD_REFF_LARGE',(/'cosp_cfodd_dbze','cosp_cfodd_icod'/), 'A', 'fraction', &
+            '# of CFODD (18 < Reff < 35 micron)', flag_xyfill=.true., fill_value=R_UNDEF)


'fraction' is not a valid UDUNIT. Should be '1'.

src/physics/cam/cospsimulator_intr.F90

Merge pull request ESCOMP#1361 from jimmielin/hplin/hb_diff

Fixes for input interpolation routines (aircraft runs only)

Fix frontogenesis function bug and change CISM2 to SGLC in FCHIST_GC compset

… to avoid dangling pointer error in NAG.

jshaw35 · 2025-10-17T15:47:23Z

@cacraigucar I think I may have identified and corrected the NAG error (due to pulling in the arrays defining the bin centers for the new dimensions and passing them into add_hist_coord without explicitly defining them as targets), would you mind testing on izumi?

cacraigucar · 2025-10-20T16:29:43Z

.gitmodules

+	url = https://github.com/CFMIP/COSPv2.0/releases/tag/v2.1.9
        fxrequired = AlwaysRequired
        fxsparse = ../.cosp_sparse_checkout
-        fxtag = v2.1.4cesm
+        fxtag = v2.1.8


When doing a fresh clone using this .gitmodules file, git-fleximod indicates an error occurred with the cosp checkout. Two items to note:
1: Typically we have the url just point to the github repo and the tag is set in fxtag. (Note I tried setting the url back to https://github.com/CFMIP/COSPv2.0 and using either tag v2.1.9 or releases/tag/v2.1.9 and neither worked)
2: The old tag was a cesm specific tag. Do we need a new cesm specific tag?

I have made changes to git-fleximod that allow me to run ./bin/git-fleximod update successfully when I clone the PR into Derecho (CAM only). The changes to git-fleximod are AI-produced and should be reviewed.

No, this PR resolves the inconsistencies between the hgt_matrix_half fields that led to a CESM specific COSP2 tag.

jedwards4b · 2025-10-21T20:17:20Z

You would need to redirect this to the correct repository, but I think that
the latest tag 1.0.3, may already have the proposed change.

jshaw35 · 2025-10-22T14:41:00Z

@jedwards4b if I just want to update the git-fleximod version in CAM how should I proceed?

I see two versions of git-fleximod in CAM. I believe that the first (and older) version is used when I call "./bin/git-fleximod update" because modifying that source code allowed me to bring in the new COSP2 tag without errors. Does this mean that the older version is needed for some operations and the newer version is used elsewhere? Some guidance on this code structure would be helpful, I don't want to break other functionality by updating the first git-fleximod path.

.lib/git-fleximod/git_fleximod/ (0.9.4)
.lib/git_fleximod/ (1.0.2)

jedwards4b · 2025-10-22T16:07:36Z

You are correct, there is a problem in the way that git-fleximod was updated and cam is still using an older version. I have opened issue #1421 to address this. Thanks

jedwards4b · 2025-10-22T19:15:44Z

I have confirmed that with the PR #1422
the cosp2 tag v2.1.9 can be brought in without issue.

jedwards4b

Changes to git-fleximod should be removed and this PR should be merged after #1422

…ted CAM development branch

correct allocation and assignment error for 'hgt_matrix_half' field

f85f6b0

cacraigucar requested review from brian-eaton and brianpm August 11, 2025 17:37

cacraigucar self-assigned this Aug 11, 2025

brian-eaton requested changes Aug 12, 2025

View reviewed changes

brianpm reviewed Aug 18, 2025

View reviewed changes

cacraigucar added this to CAM Development Aug 18, 2025

update COSP tag to most current version 20250827

52352c5

jshaw35 requested a review from brian-eaton August 27, 2025 23:41

jshaw35 added 2 commits September 3, 2025 17:28

Update the CAM COSP driver for COSP2.1.8

0c70f30

Correct syntax and variable naming errors. COSP2 warm rain diagnostic…

f8d87da

…s are now being produced!

Add COSP's MODIS LWP-REFF joint histograms to CESM

8ded365

Add MODIS tau-pressure histograms for liquid and ice clouds.

5daff8d

Correct units of cosp_prs, cosp_reffliq, and cosp_reffice coordinates.

5bdbeb5

brian-eaton reviewed Sep 17, 2025

View reviewed changes

src/physics/cam/cospsimulator_intr.F90 Show resolved Hide resolved

cacraigucar changed the title ~~cam6_4_119?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ cam6_4_120?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Sep 23, 2025

cacraigucar changed the title ~~cam6_4_120?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Sep 26, 2025

cacraigucar added 2 commits September 29, 2025 15:49

Merge tag 'cam6_4_120' into hgt_matrix_half_error

5d9b69e

Merge pull request ESCOMP#1361 from jimmielin/hplin/hb_diff

Update cosp submodule

f290fcf

cacraigucar changed the title ~~Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ cam6_4_122?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Sep 29, 2025

cacraigucar changed the title ~~cam6_4_122?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ cam6_4_122: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Sep 30, 2025

cacraigucar added 2 commits September 30, 2025 16:46

Merge tag 'cam6_4_121' into hgt_matrix_half_error

e7fcdf3

Fixes for input interpolation routines (aircraft runs only)

fix NAG compiler error

dc1fde3

cacraigucar changed the title ~~cam6_4_122: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ cam6_4_123?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Oct 1, 2025

cacraigucar changed the title ~~cam6_4_123?: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Oct 2, 2025

Update COSP external

d09761a

cacraigucar changed the title ~~Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ cam6_4_126: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Oct 13, 2025

cacraigucar changed the title ~~cam6_4_126: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Oct 14, 2025

cacraigucar and others added 3 commits October 16, 2025 15:47

put fields in correct COSP simulator

6041faf

Merge tag 'cam6_4_126' into hgt_matrix_half_error

50a18f6

Fix frontogenesis function bug and change CISM2 to SGLC in FCHIST_GC compset

Add explicit assignment of CFODD DBZE and ICOD bin centers as targets…

431cd39

… to avoid dangling pointer error in NAG.

cacraigucar changed the title ~~Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ cam6_4_127: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Oct 17, 2025

cacraigucar reviewed Oct 20, 2025

View reviewed changes

Update gitmodules to COSP2.1.9 and handling of COSP sparse checkout.

c1cdb38

cacraigucar changed the title ~~cam6_4_127: Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90~~ Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 Oct 20, 2025

cacraigucar mentioned this pull request Oct 20, 2025

git-fleximod crashes when bringing in new COSP tag in ESCOMP/CAM branch ESMCI/git-fleximod#83

Open

jedwards4b requested changes Oct 23, 2025

View reviewed changes

Revert changes to git-fleximod submodule.py for later merge with upda…

a25c0d7

…ted CAM development branch

jshaw35 requested a review from jedwards4b October 23, 2025 14:16

jedwards4b approved these changes Oct 23, 2025

View reviewed changes

Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 #1355

Are you sure you want to change the base?

Correct allocation and assignment error for 'hgt_matrix_half' field in cospsimulator_intr.F90 #1355

Uh oh!

Conversation

jshaw35 commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brian-eaton Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

jshaw35 Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

cacraigucar Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

brian-eaton Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

jshaw35 Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brianpm left a comment

Choose a reason for hiding this comment

Uh oh!

cacraigucar commented Aug 26, 2025

Uh oh!

jshaw35 commented Aug 27, 2025

Uh oh!

cacraigucar commented Aug 27, 2025

Uh oh!

jshaw35 commented Aug 27, 2025

Uh oh!

brianpm commented Sep 4, 2025

Uh oh!

jshaw35 commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jshaw35 commented Sep 4, 2025

Uh oh!

brianpm commented Sep 4, 2025

Uh oh!

jshaw35 commented Sep 4, 2025

Uh oh!

brianpm commented Sep 4, 2025

Uh oh!

jshaw35 commented Sep 5, 2025

Uh oh!

brianpm commented Sep 5, 2025

Uh oh!

jshaw35 commented Sep 5, 2025

Uh oh!

jshaw35 commented Sep 15, 2025

Uh oh!

brian-eaton Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

jshaw35 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jshaw35 commented Oct 17, 2025

Uh oh!

cacraigucar Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

jshaw35 Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jedwards4b commented Oct 21, 2025

Uh oh!

jshaw35 commented Oct 22, 2025

Uh oh!

jedwards4b commented Oct 22, 2025

Uh oh!

jedwards4b commented Oct 22, 2025

Uh oh!

jshaw35 commented Aug 5, 2025 •

edited

Loading

jshaw35 Aug 13, 2025 •

edited

Loading

jshaw35 commented Sep 4, 2025 •

edited

Loading

jshaw35 Oct 20, 2025 •

edited

Loading