Defuse advection kernels

[imreddyTeja]

Purpose

I took a look at a prognostic EDMF simulation with 1M microphysics and noticed that there are some easy performance improvements in the advection code (at the cost of readability)

To-do

• Unrelated to this PR, but I need to document guidelines on code patterns for atmos developers to avoid for performance reasons

Content

• fix inaccurate comment in update_jacobian
• cache expensive ᶠwinterp_matrix(ᶜJ * ᶜρ) ⋅ DiagonalMatrixRow(g³ʰ(ᶜgⁱʲ)) in update_jacobian
• change updraft_sedimentation to an in-place eager computation
• defuses advection code

With the following config options, this PR improves SYPD from 0.11 to 0.124

aerosol_radiation: true
approximate_linear_solve_iters: 2
cloud_model: "quadrature"
dt_cloud_fraction: "1hours"
dt_save_state_to_disk: "30days"
dz_bottom: 30.0
edmfx_detr_model: "Generalized"
edmfx_entr_model: "Generalized"
edmfx_filter: true
edmfx_nh_pressure: true
edmfx_sgs_diffusive_flux: true
edmfx_sgs_mass_flux: true
edmfx_upwinding: "first_order"
edmfx_vertical_diffusion: true
h_elem: 16
implicit_diffusion: true
implicit_sgs_advection: true
implicit_sgs_entr_detr: true
implicit_sgs_mass_flux: true
implicit_sgs_nh_pressure: true
implicit_sgs_vertdiff: true
insolation: "timevarying"
max_newton_iters_ode: 3
moist: "nonequil"
netcdf_output_at_levels: true
precip_model: "1M"
prescribed_aerosols: ["CB1", "CB2", "DST01", "DST02", "DST03", "DST04", "DST05", "OC1", "OC2", "SO4", "SSLT01", "SSLT02", "SSLT03", "SSLT04", "SSLT05"]
prognostic_tke: true
rad: "allskywithclear"
rayleigh_sponge: true
t_end: "120days"
time_varying_trace_gases: ["CO2", "O3"]
toml: [toml/longrun_aquaplanet_diagedmf.toml]
turbconv: "prognostic_edmfx"
viscous_sponge: true
z_elem: 63
z_max: 60000.0

FLOAT_TYPE: "Float32"
albedo_model: "CouplerAlbedo"
atmos_config_file: "config/atmos_configs/climaatmos_progedmf_1m.yml"
checkpoint_dt: "366days"
coupler_toml: ["toml/amip_progedmf_1m.toml"]
dt: "30secs"
dt_cpl: "30secs"
dt_rad: "600secs"
energy_check: false
land_model: "integrated"
mode_name: "amip"
radiation_reset_rng_seed: true
start_date: "20100101"
surface_setup: "PrescribedSurface"
topo_smoothing: true
topography: "Earth"

---

• I have read and checked the items on the review checklist.

[szy21]

You can probably use ᶜtemp_scalar_4 here and ᶜtemp_scalar_5 for vtt?

[szy21]

Use temp_scalar_3?

[szy21]

Maybe more readable if you do something like:

ᶠρ = p.scratch.ᶠtemp_scalar
@. p.scratch.ᶠtemp_scalar = ᶠinterp(Y.c.ρ * ᶜJ) / ᶠJ
ᶜq = p.scratch.ᶜtemp_scalar_2
@. ᶜq = specific(ᶜρq, Y.c.ρ)

and use those variables?

[szy21]

It doesn't look right here

[imreddyTeja]

Good catch, that is a typo

[szy21]

Thanks, this looks fine to me. Maybe it would be good if @sajjadazimi takes a look as well.

[sajjadazimi]

Thank you. I just found a minor issue in water advection.

[sajjadazimi]

This is not equal to temp_scalar_2. Here instead of T^j we should use T^0

[szy21]

Ah, right, good catch!

[szy21]

our regression test on 1M is useless

[imreddyTeja]

😳 good catch, I'll fix this and double check the other changes