Improve GW performance efficiency with different tasks per node and layout #4141

weihuang-jedi · 2025-10-09T15:50:08Z

Description

Improve GW performance by adjust URSA, and GAEAC6 tasks per node, and px-py layout to gain short wall-clock time, or using less resource (nodes) with similar elapse time, or both.

Resolves #4140

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)
[ x] Performance improvement

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Forecast-only on Ursa, GaeaC6

Checklist

[x ] Any dependent changes have been merged and published
[x ] My code follows the style guidelines of this project
[ x] I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
[x ] My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
Any new scripts have been added to the .github/CODEOWNERS file with owners
I have made corresponding changes to the system documentation if necessary

…into feature/adjust_tasks_per_node_layout * 'develop' of github.com:NOAA-EPIC/global-workflow-cloud: Adding workflow infrastructure for wave_stat and wave_stat_pnt jobs (NOAA-EMC#3846) Add ush and parm of SPOC (NOAA-EMC#3944) Bugfixes from IMS C++ code additions (NOAA-EMC#4105) Increase C96 gcdas fcst wall time to 1.5 hours (NOAA-EMC#4108) Use default stochastic physics (NOAA-EMC#4073) Fix new feature template (NOAA-EMC#4104)

…into feature/adjust_tasks_per_node_layout * 'develop' of github.com:NOAA-EPIC/global-workflow-cloud: Update CODEOWNERS for new oceanice_products assignments (NOAA-EMC#4134) Improve GW forecast timing of high-resolution runs with findings from containerized GW (NOAA-EMC#4123) Update CODEOWNERS to change job assignments (NOAA-EMC#4128) Update j-jobs to be bash linter compliant (NOAA-EMC#4124) Update the marine analysis archiving (NOAA-EMC#4082) Port tracker/genesis tasks to cloud; load wgrib2; add noaacloud to load_gw_gsi_modules (NOAA-EMC#4111) Only run wave bnd pnt for gfs (NOAA-EMC#4112)

…into feature/adjust_tasks_per_node_layout * 'develop' of github.com:NOAA-EPIC/global-workflow-cloud: Correct parametric and ensemble background error statistics filenames in marine DA (NOAA-EMC#4120)

…into feature/adjust_tasks_per_node_layout * 'develop' of github.com:NOAA-EPIC/global-workflow-cloud: Ctest case updates (NOAA-EMC#4118) Consolidate load_*_modules scripts into a generic load_modules.sh script (NOAA-EMC#4126) Updates for Gaea C6 following OS upgrade (NOAA-EMC#4110)

aerorahul

LGTM, but I note that these changes are not exercised by running the global-workflow automated tests, and the resource changes should be reviewed by those routinely running at these resolutions.

DavidHuber-NOAA · 2025-10-09T16:40:58Z

dev/parm/config/gfs/config.resources.GAEAC6

    case "${CASE}" in
      "C768")
-        export tasks_per_node=144
+        export tasks_per_node=180


@GeorgeVandenberghe-NOAA had noted that the UFS did not run properly on C6 when allowed to use more than 144 tasks at high resolution. Is this issue fixed?

DavidHuber-NOAA · 2025-10-09T16:43:04Z

dev/parm/config/gfs/config.ufs

          elif [[ "${RUN}" = "gfs" || "${RUN}" = "gcafs" ]]; then
            export layout_x=12
-            export layout_y=16
+            export layout_y=15


Is this in combination with the config.resources.<host> changes so that the tiles land on the same nodes? Might want inputs from @dpsarmie on this change as this will affect all hosts. This would also need to be tested on WCOSS2.

I agree that some justification would be needed to have this change (on top of thorough testing since this seems to have only been tested on Ursa and Gaea).

768/15 = 51.2 so I also think this means we can't be guaranteed reproducibility.

@DavidHuber-NOAA Yes, that is the thought, so we hope that more communications can be within node.

@JessicaMeixner-NOAA From my understanding 768/15 = 51.2, which is not an integer, which should impact load balance, but not reproducibility. But with C768 is a pretty high-resolution, there are should be plenty of tasks to do, load-balance is probably not as important as at low-res. I guess reduce cross-node communication is more important.
From my understanding task per node impacts memory per task can use, and layout will mostly impact load balance (and probably cross cube-spheric tiles communication).

@dpsarmie can you weigh in here. It’s my understanding the layout chosen here will not reproduce with other layouts since it’s not divisible. Maybe things have changed?

JessicaMeixner-NOAA · 2025-10-09T16:49:08Z

dev/parm/config/gfs/config.resources.GAEAC6

        ;;
      "C1152")
-        export tasks_per_node=144
+        export tasks_per_node=160


Can someone share test logs for this change? I know after the OS update we still have to make manual changes to the number of write grid tasks for gaea specifically, so we'd need to confirm this is working before updating as this will impact GFS retro tests.

I can not run C1152 on Ursa, which exceed the maximum numbers of nodes I can require.

This is changing the layout on gaea. We shouldn’t change the layout without testing and showing that this will work.

GeorgeVandenberghe-NOAA · 2025-10-09T16:50:11Z

no. It persists. FIrst issue is UPP memory use but there are other memory exhausting components behind UPP and this is going to require some work.

…

On Thu, Oct 9, 2025 at 12:41 PM David Huber ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In dev/parm/config/gfs/config.resources.GAEAC6 <#4141 (comment)> : > @@ -10,10 +10,10 @@ case ${step} in "fcst" | "efcs") case "${CASE}" in "C768") - export tasks_per_node=144 + export tasks_per_node=180 @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> had noted that the UFS did not run properly on C6 when allowed to use more than 144 tasks at high resolution. Is this issue fixed? — Reply to this email directly, view it on GitHub <#4141 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FXMCJMCN65VQSVWUA33W2F27AVCNFSM6AAAAACIX3UFUWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTGMJZG44TAOJYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

…yout

weihuang-jedi and others added 9 commits September 27, 2025 13:46

testing ursa speed up with distribution

f613832

gaeac6 module load reorder, and and explicit distribution

d1bf01e

create adjust tasks layout branch

787df18

Merge branch 'develop' of github.com:NOAA-EPIC/global-workflow-cloud …

2a1f9de

…into feature/adjust_tasks_per_node_layout * 'develop' of github.com:NOAA-EPIC/global-workflow-cloud: Correct parametric and ensemble background error statistics filenames in marine DA (NOAA-EMC#4120)

increase tasks per node for c1152

95ad6f2

add ursa change as well

a8fb88c

weihuang-jedi force-pushed the feature/adjust_tasks_per_node_layout branch from c767c57 to a8fb88c Compare October 9, 2025 15:53

weihuang-jedi requested review from DavidHuber-NOAA, JessicaMeixner-NOAA and WalterKolczynski-NOAA October 9, 2025 16:05

weihuang-jedi marked this pull request as ready for review October 9, 2025 16:05

weihuang-jedi requested a review from aerorahul as a code owner October 9, 2025 16:05

aerorahul approved these changes Oct 9, 2025

View reviewed changes

DavidHuber-NOAA reviewed Oct 9, 2025

View reviewed changes

JessicaMeixner-NOAA reviewed Oct 9, 2025

View reviewed changes

Merge branch 'NOAA-EMC:develop' into feature/adjust_tasks_per_node_la…

ef967a3

…yout

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve GW performance efficiency with different tasks per node and layout #4141

Improve GW performance efficiency with different tasks per node and layout #4141

weihuang-jedi commented Oct 9, 2025

Uh oh!

aerorahul left a comment

Uh oh!

DavidHuber-NOAA Oct 9, 2025

Uh oh!

DavidHuber-NOAA Oct 9, 2025

Uh oh!

dpsarmie Oct 9, 2025

Uh oh!

JessicaMeixner-NOAA Oct 9, 2025

Uh oh!

weihuang-jedi Oct 10, 2025

Uh oh!

weihuang-jedi Oct 10, 2025

Uh oh!

JessicaMeixner-NOAA Oct 10, 2025

Uh oh!

JessicaMeixner-NOAA Oct 9, 2025

Uh oh!

weihuang-jedi Oct 10, 2025

Uh oh!

JessicaMeixner-NOAA Oct 10, 2025

Uh oh!

GeorgeVandenberghe-NOAA commented Oct 9, 2025 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Improve GW performance efficiency with different tasks per node and layout #4141

Are you sure you want to change the base?

Improve GW performance efficiency with different tasks per node and layout #4141

Conversation

weihuang-jedi commented Oct 9, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GeorgeVandenberghe-NOAA commented Oct 9, 2025 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants