Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 6, 2025

Description

This PR addresses issue #3051 by replacing standard cp commands with atomic copy utilities (cpfs and cpreq from prod_util) for all copies to COM directories. This is required per NCO Implementation Standards v11 to prevent downstream programs from accessing incomplete files.

Problem

Using standard cp for large files can result in downstream programs accessing incomplete copies while the copy operation is still in progress. This is particularly problematic in operational weather forecasting where timing and data integrity are critical.

Solution

Replaced all instances of cp to COM directories with:

  • cpfs: For output files being copied TO COM directories (11 instances)
  • cpreq: For required input files being copied FROM COM directories (1 instance)

The cpfs utility provides atomic copies by:

  1. Copying to a temporary file
  2. Renaming the temp file only when the copy is complete

This ensures downstream processes never see partially written files.

Changes

Modified 3 files with 12 total changes:

ush/tropcy_relocate.sh (5 changes)

  • Pathname files copied within COMOUT_OBS
  • Required tcvitals input from COMOUT_OBS
  • Empty file creation for error handling
  • Relocation output files (inform, tcvitals)

ush/syndat_qctropcy.sh (6 changes)

  • Empty file creation for null tcvitals scenarios
  • Processed tcvitals output to COMOUT_OBS
  • JTWC/FNOC tcvitals output to COMOUT_OBS

ush/wave_prnc_ice.sh (1 change)

  • Ice data output to COMOUT_WAVE_PREP

Testing

  • ✅ All modified scripts pass bash syntax validation
  • ✅ No new shellcheck warnings introduced
  • ✅ Verified no remaining plain cp to COM directories
  • ✅ Verified no manual atomic copy patterns (copy to temp + mv)
  • ✅ Changes maintain functional equivalence (identical output)

Impact

  • Functional: No change to output files or behavior
  • Operational: Improved reliability by preventing access to incomplete files during copy operations
  • Standards: Now complies with NCO Implementation Standards v11

Fixes #3051

Co-authored-by: @DavidHuber-NOAA

Original prompt

This section details on the original issue you should resolve

<issue_title>Use cpfs for copies that should be atomic</issue_title>
<issue_description>### What new functionality do you need?

Using the standard cp can result in downstream programs using incomplete copies, particularly for large files. prod_util provides a utility, cpfs, that does an atomic copy by copying to a temp file and then renaming the temp file when the copy is complete. This is an NCO requirement (see Implementation Standards v11).

There are also places where we are doing this type of copy manually. These should be replaced with using cpfs to simplify scripts.

What are the requirements for the new functionality?

All copies that should be atomic (e.g. most or all copies to COM) should use cpfs or cpreq instead of cp. Should use cpreq when files are required to be able to run, and cpfs when output is being copied to COMOUT.

cpfs does not allow globbing, so some modification may be needed.

Acceptance Criteria

  • No usage of cp when the copy could be used before completion (replaced with cpfs)
  • Replace instances where we are manually doing an atmoic copy with cpfs
  • Output identical to previous

Suggest a solution (optional)

Should we leverage NCP? There are still a lot of places we are directly using cp instead of NCP.</issue_description>

Comments on the Issue (you are @copilot in this section)

@WalterKolczynski-NOAA Related: #712 @DavidHuber-NOAA A thorough search of the code shows only a handful of calls to `cp` rather than `cpfs` or `cpreq`. Of these, only the following lines copy into COM (where `cpfs` should be used):

cp "${COMOUT_OBS}/${RUN}.${cycle}.sgesprep_pre-relocate_pathname.${tmmark}" \
"${COMOUT_OBS}/${RUN}.${cycle}.sgesprep_pathname.${tmmark}"

cp "${COMOUT_OBS}/${RUN}.${cycle}.syndata.tcvitals.${tmmark}" "tcvitals.now"

cp "rel_inform1" "${COMOUT_OBS}/${RUN}.${cycle}.inform.relocate.${tmmark}"
cp "tcvitals" "${COMOUT_OBS}/${RUN}.${cycle}.tcvitals.relocate.${tmmark}"

cp current "${COMOUT_OBS}/${RUN}.${cycle}.syndata.tcvitals.${tmmark}"
# Create the DBNet alert
if [[ "${SENDDBN}" == "YES" ]]; then
"${DBNROOT}/bin/dbn_alert" "MODEL" "GDAS_TCVITALS" "${job}" "${COMOUT_OBS}/${RUN}.${cycle}.syndata.tcvitals.${tmmark}"
fi
# Write JTWC/FNOC Tcvitals to /com path since not saved anywhere else
cp fnoc "${COMOUT_OBS}/${RUN}.${cycle}.jtwc-fnoc.tcvitals.${tmmark}"

cp ice.ww3 "${COMOUT_WAVE_PREP}/${icefile}"
</comment_new>

Fixes #2349

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

@Copilot Copilot AI changed the title [WIP] Use cpfs for copies that should be atomic Replace cp with cpfs/cpreq for atomic copies to COM directories Oct 6, 2025
@Copilot Copilot AI requested a review from DavidHuber-NOAA October 6, 2025 18:08
Copilot finished work on behalf of DavidHuber-NOAA October 6, 2025 18:08
@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review October 6, 2025 18:26
Copy link
Contributor

@JessicaMeixner-NOAA JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change in ush/wave_prnc_ice.sh is fine.

Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks ok.
@copilot Are these the only place where cp is being used now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use cpfs for copies that should be atomic

4 participants