Skip to content

Use cpfs for copies that should be atomic #2349

@WalterKolczynski-NOAA

Description

@WalterKolczynski-NOAA

What new functionality do you need?

Using the standard cp can result in downstream programs using incomplete copies, particularly for large files. prod_util provides a utility, cpfs, that does an atomic copy by copying to a temp file and then renaming the temp file when the copy is complete. This is an NCO requirement (see Implementation Standards v11).

There are also places where we are doing this type of copy manually. These should be replaced with using cpfs to simplify scripts.

What are the requirements for the new functionality?

All copies that should be atomic (e.g. most or all copies to COM) should use cpfs or cpreq instead of cp. Should use cpreq when files are required to be able to run, and cpfs when output is being copied to COMOUT.

cpfs does not allow globbing, so some modification may be needed.

Acceptance Criteria

  • No usage of cp when the copy could be used before completion (replaced with cpfs)
  • Replace instances where we are manually doing an atmoic copy with cpfs
  • Output identical to previous

Suggest a solution (optional)

Should we leverage NCP? There are still a lot of places we are directly using cp instead of NCP.

Sub-issues

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions