Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create compute build option #3186

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
01e537f
Simplify build_all.sh
DavidHuber-NOAA Dec 19, 2024
609d017
Create shell compute build script
DavidHuber-NOAA Dec 19, 2024
c4521a6
add a script to parse a yaml containing build specs and create a Roco…
aerorahul Dec 20, 2024
7e6e797
fix pynorms
aerorahul Dec 20, 2024
12dc75c
add line at the end of yaml
aerorahul Dec 20, 2024
0c924a5
remove stray nodes
aerorahul Dec 20, 2024
643a62b
some jobs were not specifying the number of cores to build with
aerorahul Dec 20, 2024
c3ac55b
add a shell script for convinence next to build_all.sh
aerorahul Dec 20, 2024
5eabc7b
Merge remote-tracking branch 'rahul/feature/compute' into feature/com…
DavidHuber-NOAA Dec 20, 2024
ba0c92b
Remove compute-build option from build_all.sh
DavidHuber-NOAA Dec 20, 2024
5227b7b
Added build options for compute build
DavidHuber-NOAA Dec 20, 2024
c93b1f6
Remove remnants of compute build from build_all.sh
DavidHuber-NOAA Dec 20, 2024
0a29f2a
Merge remote-tracking branch 'origin/develop' into feature/compute_bu…
DavidHuber-NOAA Dec 20, 2024
ba44f1b
Create log directory if it doesn't exist
DavidHuber-NOAA Dec 20, 2024
f95ea79
Debug/cleanup build scripts
DavidHuber-NOAA Dec 20, 2024
e22d00c
Template rocoto commands, delete database on completion
DavidHuber-NOAA Dec 20, 2024
c86fc00
Add new script to CODEOWNERS
DavidHuber-NOAA Dec 20, 2024
81ee6cd
Removed unused compute_build script
DavidHuber-NOAA Dec 20, 2024
9334f12
Add build options
DavidHuber-NOAA Dec 20, 2024
ef4eb87
Correct hydrostatic flag
DavidHuber-NOAA Dec 20, 2024
58cbefb
Simplify build selections when 'all' is specified
DavidHuber-NOAA Dec 20, 2024
ee942ad
Keep the database file
DavidHuber-NOAA Dec 20, 2024
f16325c
Update build_all.sh call in the Jenkinsfile
DavidHuber-NOAA Dec 20, 2024
02e1a1b
Remove unused j option
DavidHuber-NOAA Dec 20, 2024
a81a802
Remove -B option and update build_all call in generate_workflows.sh
DavidHuber-NOAA Dec 20, 2024
e574ec2
Send select_all_yamls a yaml_list for gfs also
DavidHuber-NOAA Dec 20, 2024
88359b7
Reverse verbose flag location
DavidHuber-NOAA Dec 21, 2024
f76af3b
Remove unneeded mkdir
DavidHuber-NOAA Dec 23, 2024
1b75fc3
Update sorc/build_all.sh
DavidHuber-NOAA Dec 23, 2024
a7a3e1d
Make default builds easier to modify
DavidHuber-NOAA Dec 23, 2024
9de64ba
Merge branch 'feature/compute_builds' of github.com:davidhuber-noaa/g…
DavidHuber-NOAA Dec 23, 2024
4217006
Concatenate inputs
DavidHuber-NOAA Dec 23, 2024
99c05f3
Reenable quick-kill build feature
DavidHuber-NOAA Dec 23, 2024
5b633df
Enable compute builds in Jenkins
DavidHuber-NOAA Dec 23, 2024
8b61fad
Update build documentation
DavidHuber-NOAA Dec 23, 2024
85e6647
Update build_compute.sh to match build_all.sh API
DavidHuber-NOAA Dec 23, 2024
150c37e
Update Jenkins build_compute.sh call
DavidHuber-NOAA Dec 23, 2024
09212d7
Merge remote-tracking branch 'origin/develop' into feature/compute_bu…
DavidHuber-NOAA Dec 24, 2024
5620696
Remove unused 'system'
DavidHuber-NOAA Dec 24, 2024
8a77c5b
Load a cmake module if command not found for upp build
DavidHuber-NOAA Dec 24, 2024
72c8d7a
Also delete the lock file
DavidHuber-NOAA Dec 24, 2024
a8a6d6f
Use associative build_systems array
DavidHuber-NOAA Dec 24, 2024
d2453ed
Use array index
DavidHuber-NOAA Dec 24, 2024
7470a74
Check if on WCOSS2 before changing default modules
DavidHuber-NOAA Dec 24, 2024
04a1d24
Set HOMEgfs if it wasn't already for the upp build
DavidHuber-NOAA Dec 24, 2024
0f0be65
Ignore sc2155
DavidHuber-NOAA Dec 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -211,3 +211,4 @@ ush/python/pygfs/utils/marine_da_utils.py @guillaumevernieres @AndrewEichmann-NO

# Specific workflow scripts
workflow/generate_workflows.sh @DavidHuber-NOAA
workflow/build_compute.py @DavidHuber-NOAA @aerorahul
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ parm/wafs

# Ignore sorc and logs folders from externals
#--------------------------------------------
sorc/build.xml
sorc/build.db
sorc/build_lock.db
sorc/*log
sorc/logs
sorc/calc_analysis.fd
Expand Down
4 changes: 1 addition & 3 deletions ci/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -120,9 +120,7 @@ pipeline {
def error_logs_message = ""
dir("${HOMEgfs}/sorc") {
try {
sh(script: './build_all.sh -kgu') // build the global-workflow executables for GFS variant (UFS-wx-model, WW3 pre/post executables)
sh(script: './build_ww3prepost.sh -w > ./logs/build_ww3prepost_gefs.log 2>&1') // build the WW3 pre/post processing executables for GEFS variant
sh(script: './build_ufs.sh -w -e gefs_model.x > ./logs/build_ufs_gefs.log 2>&1') // build the UFS-wx-model executable for GEFS variant
sh(script: './build_compute.sh all') // build the global-workflow executables
} catch (Exception error_build) {
echo "Failed to build global-workflow: ${error_build.getMessage()}"
if ( fileExists("logs/error.logs") ) {
Expand Down
56 changes: 19 additions & 37 deletions docs/source/clone.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,35 +18,39 @@ Clone the `global-workflow` and `cd` into the `sorc` directory:
git clone --recursive https://github.com/NOAA-EMC/global-workflow
cd global-workflow/sorc

For forecast-only (coupled or uncoupled) build of the components:
.. _build_examples:

The build_all.sh script can be used to build all required components of the global workflow. The accepted arguments is a list of systems to be built. This includes builds for GFS and GEFS forecast-only experiments, GSI and GDASApp-based DA for cycled GFS experiments. See `feature availability <hpc.html#feature-availability-by-hpc>`__ to see which system(s) are available on each supported system.

::

./build_all.sh
./build_all.sh [gfs] [gefs] [gs] [gdas] [all]

For cycled (w/ data assimilation) use the `-g` option during build:
For example, to run GFS experiments with GSI DA, execute:

::

./build_all.sh -g
./build_all.sh gfs gsi

For coupled cycling (include new UFSDA) use the `-gu` options during build:
This builds the GFS, UFS-utils, GFS-utils, WW3 with PDLIB (structured wave grids), UPP, GSI, GSI-monitor, and GSI-utils executables.

[Currently only available on Hera, Orion, and Hercules]
For coupled cycling (include new UFSDA) execute:

::

./build_all.sh -gu
./build_all.sh gfs gdas

This builds all of the same executables, except it builds the GDASApp instead of the GSI.

For building without PDLIB (unstructured grid) for the wave model, use the `-w` options during build:
To run GEFS (forecast-only) execute:

::

./build_all.sh -w
./build_all.sh gefs

This builds the GEFS, UFS-utils, GFS-utils, WW3 *without* PDLIB (unstructure wave grids), and UPP executables.

Build workflow components and link workflow artifacts such as executables, etc.
Once the building is complete, link workflow artifacts such as executables, configuration files, and scripts via

::

Expand Down Expand Up @@ -107,40 +111,19 @@ Under the ``/sorc`` folder is a script to build all components called ``build_al

::

./build_all.sh [-a UFS_app][-g][-h][-u][-v]
./build_all.sh [-a UFS_app][-k][-h][-v] [list of system(s) to build]
-a UFS_app:
Build a specific UFS app instead of the default
-g:
Build GSI
-k:
Kill all builds immediately if one fails
-h:
Print this help message and exit
-j:
Specify maximum number of build jobs (n)
-u:
Build UFS-DA
-v:
Execute all build scripts with -v option to turn on verbose where supported

For forecast-only (coupled or uncoupled) build of the components:

::

./build_all.sh

For cycled (w/ data assimilation) use the `-g` option during build:

::

./build_all.sh -g

For coupled cycling (include new UFSDA) use the `-gu` options during build:

[Currently only available on Hera, Orion, and Hercules]

::

./build_all.sh -gu
Lastly, pass to build_all.sh a list of systems to build. This includes `gfs`, `gefs`, `sfs` (not fully supported), `gsi`, `gdas`, and `all`.

For examples of how to use this script, see :ref:`build examples <build_examples>`.

^^^^^^^^^^^^^^^
Link components
Expand All @@ -156,4 +139,3 @@ After running the checkout and build scripts run the link script:

Where:
``-o``: Run in operations (NCO) mode. This creates copies instead of using symlinks and is generally only used by NCO during installation into production.

Loading
Loading