Create compute build option

[DavidHuber-NOAA]

Description

This creates scripts to run compute-node builds and also refactors the build_all.sh script to make it easier to build all executables.

In place of various options to control what components are built when using build_all.sh, instead it takes in a list of one or more systems to build:

• gfs builds everything needed for forecast-only gfs (UFS model with unstructured wave grid, gfs_utils, ufs_utils, upp, ww3 pre/post for unstructured wave grid)
• gefs builds everything needed for GEFS (UFS model with structured wave grid, gfs_utils, ufs_utils, upp, ww3 pre/post for structured wave grid)
• sfs builds everything needed SFS (UFS model in hydrostatic mode with unstructured wave grid, gfs_utils, ufs_utils, upp, ww3 pre/post for structured wave grid)
• gsi builds GSI-based DA components (gsi_enkf, gsi_monitor, gsi_utils)
• gdas builds JEDI-based DA components (gdas app, gsi_monitor, gsi_utils)

all will build all of the above (mostly for testing)

Examples:
Build for forecast-only GFS:
./build_all.sh gfs
Build cycled GFS including coupled DA:
./build_all.sh gfs gsi gdas
Build GEFS:
./build_all.sh gefs
Build everything (for testing purposes):
./build_all.sh all
Other options, such as -d to build in debug mode, remain unchanged.

The full script signature is now:

./build_all.sh [-a UFS_app][-c build_config][-d][-f][-h][-v] [gfs] [gefs] [sfs] [gsi] [gdas] [all]

Additionally, there is a new script to build components on the compute nodes using the job scheduler instead of the login node. This method takes the load off of the login nodes and may be faster in some cases. Compute build is invoked using the build_compute.sh script, which behaves similarly to the new build_all.sh:

./build_compute.sh [-h][-v][-A <hpc-account>] [gfs] [gefs] [sfs] [gsi] [gdas] [all]

Compute build will generate a rocoto workflow and then call rocotorun itself repeatedly until either a build fails or all builds succeed, at which point the script will exit. Since the script is calling rocotorun itself, you don't need to set up your own cron to do it, but advanced users can also use all the regular rocoto tools on build.xml and build.db if you wish.

Some things to note with the compute build:

• When a build fails, other build jobs are not cancelled and will continue to run.
• Since the script stops running rocotorun once one build fails, the rocoto database will no longer update with the status of the remaining jobs after that point.
• Similarly, if the terminal running build_compute.sh gets disconnected, the rocoto database will no longer update.
• In either of the above cases, you could run rocotorun yourself manually to update the database as long as the job information hasn't aged off the scheduler yet.

Resolves #3131

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES (build jobs are no longer specifiable)
• Does this change require a documentation update? YES
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• Built and linked all executables with both build_all.sh and compute_build.sh
• Run a C96C48_hybatmDA case on Hera
• Run a C48_S2SWA_gefs case on Hera

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[aerorahul]

looks good. thanks for taking the suggestions and updating to accommodate all systems.
A few more suggestions based on the code.

[DavidHuber-NOAA]

@aerorahul Thanks for the quick review. I'm rerunning build tests now and will re-request a review when complete.

[aerorahul]

@aerorahul What do you think of the workaround I came up with to add cmake to the default modules. This will cause cmake to load when the UPP compile_upp.sh script runs module reset.

It works. Though, will this not be applied for all machines? It might have unintended consequences.

[DavidHuber-NOAA]

It will be applied to all machines that do not have cmake in the search path. Having tested compute nodes on Hera, Gaea-C5, Gaea-C6, Hercules, and S4, I can verify that these all have cmake in their search paths.

I could go one step further and 1) add a detect_machine.sh call to build_upp.sh and 2) only run this block if we are on WCOSS2. Thoughts?

[DavidHuber-NOAA]

@DavidHuber-NOAA Creating a git diff patch and applying it via build_compute.sh on WCOSS2 might be sufficient until we get the permanent fix in the upp module file. Just a thought.

This could also work, though the patch can only be applied once. Thus, if build_compute.sh needs to be called again, it will need to be smart enough to know that the patch was already applied or just ignore the resulting error.

[aerorahul]

Since all of this is temporary, just do what's necessary and easy. Hopefully, upp will make a fix soon.

[WenMeng-NOAA]

@WenMeng-NOAA We would like to build upp on the compute nodes on all platforms. Can we include cmake in wcoss2.lua in the upp modulefiles?

@aerorahul Sure, please submit a PR to the UPP repos.

[DavidHuber-NOAA]

Test compute build on WCOSS2 was successful.

[aerorahul]

Lgtm

[RussTreadon-NOAA]

FYI: updating $HOMEgfs/sorc/gdas.cd/modulefiles/GDAS/wcoss2.intel.lua to NCO's test installation of spack-stack/1.6.0 along with adding cmake to the UPP wcoss2 modulefile allows ./build_compute.sh -v gdas to successfully run to completion on Dogwood.

[WalterKolczynski-NOAA]

UPP now builds successfully on compute nodes.