Remove duplication between gen_mksurfdata_jobscript_multi and gen_mksurfdata_jobscript_single

[ekluzek]

This was brought up in #2290. gen_mksurfdata_jobscript_single has parts that are duplicated with gen_mksurfdata_jobscript_multi and hence changes have to go in two places. This was for example an issue in #2427 which means that extra testing needs to be done because of that duplication.

An improvement to the design would be to have gen_mksurfdata_jobscript_single as the base level script that is expanded on by gen_mksurfdata_jobscript_multi which then doesn't duplicate code, but only adds the additional functionality needed there.

This is something that we will do as a post-CTSM5.2 development.

[ekluzek]

Note, this also helps with #2341 and will help getting mksurfdata_esmf work on any CESM supported machine.

[ekluzek]

Digging into the issues with external updates for ctsm5.2 I now see that some development here is important to do. I've done some work here, I've tried to limit it to what's important in order to have things working and to easily identify problems with externals as quick as possible.

[ekluzek]

One idea to do later would be to separate single and multi into a class that has a base class used by both, and then extends it for each specific script.

This was brought up here:

https://github.com/ESCOMP/CTSM/pull/2427/files#r1543707838

Athough #2427 does some work on this. Some other duplication that would be useful to remove would be:

• Remove duplication in running the .env_mach_specific.sh script
• The write of the part of the script that does the actual
• The write of the bash error checks for previous commands
• The write of the ending parts for success...
• Move of some utilities at the top to a shared utility level

[ekluzek]

A really good point that @samsrabin gave here for doing this is that the current structure does NOT make it obvious that changes in gen_mksurfdata_jobscript_single MAY also the multi script. That link is unlikely to be obvious to the Developer working on single, and as such they may do something that breaks the multi script. This is a really good point.