Running regression test using rt.sh

NOTE: THIS PAGE IS UNDER CONSTRUCTION

Update 05/31/2023: On May 31st, 2023, Major updates to this wiki were made. The most recent RT changes bring in: the argument -a was added for users to provide their HPC account to access the queues. Intel/GNU tests are now in the same rt.conf. opnReqTests script will run only supported cases if -c is not provided. Logs now provide hash information.

Update 01/06/2021: On January 6, 2021, the argument -f was removed. Adding it will force rt.sh to exit immediately. The default for rt.sh is to run the full regression tests in rt.conf unless -l xyz.conf is provided.

Shell script-based Regression Test: rt.sh

• Set of simple shell script files and input files
• Build + Run
• Run only (workaround)
• Supports Rocoto and ecFlow workflow managers
• ./rt.sh to run full regression tests
• ./rt.sh -c to create new baselines (in your own directory)
• ./rt.sh -m to verify your own baselines that you created beforehand with -c
• Regression test root directory: ufs-weather-model/tests

rt.sh related Files

rt.sh calls:

• detect_machine.sh
• compile.sh
• run_test.sh which calls rt_utils.sh
• rt.sh uses following input files:
  • rt.conf
  • default_vars.sh
  • <test-name>
  • <run-setup-name>

rt.sh related Files

• detect_machine.sh: detect and assign machine name, set account (nems is default)
• compile.sh: calls build.sh which in turn invokes CMake build
• run_test.sh: sets env variables, run directory, etc., prepares a canned case in the run directory, and calls rt_utils functions
• bl_date.conf: Set date for where to place RT baselines (YYYYMMDD)
• rt_utils.sh: contains utility functions, e.g.,
  • submit_and_wait
  • check_results
  • rocoto_create_compile_task, rocoto_create_run_task, rocoto_run, rocoto_kill
  • ecflow_create_compile_task, ecflow_create_run_task, ecflow_run, ecflow_kill

Input Files: 1) rt.conf

• COMPILE Line ( Items separated by a | )
  • Item 1: COMPILE - This tells rt.conf the following information is to be used in setting up a compile job
  • Item 2: Compile number - must be sequential in rt.conf - use as a reference for compile failures
  • Item 3: Compiler to use in build (intel or gnu)
  • Item 4: CMAKE Options - Provide all CMAKE options for the build
  • Item 5: Machines to run on (- is used to ignore specified machines, + is used to only run on specified machines)
    • -> EX: + hera orion gaea = compile will only run on hera orion and gaea machines
    • -> EX: - wcoss2 acorn = compile will NOT be run on wcoss2 or acorn
  • Item 6: [set as fv3]. Used to control the compile job only if FV3 was present, previously used to run a test w/o compiling code
• RUN Line ( Items separated by a | ) NOTE: The build resulting from the COMPILE line above the RUN line will be used to run the test
  • Item 1: RUN - This tells rt.conf the following information is to be used in setting up a model run
  • Item 2: Test name. (Which test in the tests/tests directory should be sourced)
  • Item 3: Machines to run on (- is used to ignore specified machines, + is used to only run on specified machines).
    • reference example above
  • Item 4: Controls whether the run creates its own baseline or it uses the baseline from a different (control) test.
  • Item 5: Test name to compare baselines with if not itself.

Input Files: 2) tests/<test-name>

• Two levels to set simulation parameters
  • default_vars.sh sets default values
  • <test-name> overrides default values, adds test-specific parameters, e.g.,
    • SYEAR=2013, FHMAX=24, FDIAG=6, WLCLK=30
• Set environment variables that are passed onto various template files in parm/
  • input.*.nml.IN
  • nems.configure.*.IN
  • model_configure.IN
• Specify configuration templates to use, e.g.,
  • INPUT_NML=”input.mom6_ccpp.nml.IN”
  • NEMS_CONFIGURE=”nems.configure.med_atm_ocn_ice_wav.IN”
  • FV3_RUN=”cpld_fv3_mom6_cice_atm_flux_run.IN”

Input Files: 3) fv3_conf/<run-setup-name>

• Set up input data, grid data, etc. by copying files from baseline directory to run directory
• Baseline directory contains
  • Subdirectories for input data (e.g., CICE_IC, MOM6_IC, FV3_input_data, MEDIATOR_ccpp)
  • Subdirectories for previous run results (e.g., cpld_control_p8_intel or cpld_control_p8_gnu)
• Make sure directories and files exist in RTPWD

Default Directories specified in rt.sh

• Baseline directory (RTPWD)
  • Hera: /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/develop-YYYYMMDD
  • Orion: /work/noaa/nems/emc.nemspara/RT/NEMSfv3gfs/develop-YYYYMMDD
• Run directory root (RUNDIR_ROOT)
  • Hera: /scratch1/NCEPDEV/stmp2/${USER}/FV3_RT/rt_$$
  • Orion: /work/noaa/stmp/${USER}/stmp/${USER}/FV3_RT/rt_$$
  • RUNDIR=${RUNDIR_ROOT}/${TEST_NAME}
• New baseline directory (NEW_BASELINE)
  • Hera: /scratch1/NCEPDEV/stmp4/${USER}/FV3_RT/REGRESSION_TEST
  • Orion: /work/noaa/stmp/${USER}/stmp/${USER}/FV3_RT/REGRESSION_TEST

Build

• Triggered by COMPILE row in rt.conf with specified <MAKE_OPT>
• Build is done using CMakeLists.txt in ..
• compile.sh is a simple wrapper to interface with rt.sh
• If you build exe file separately (i.e., w/o rt.sh), make a copy as ufs-weather-model/tests/fv3_0.exe. Also,
  • $ cp ../NEMS/src/conf/modules.nems modules.fcst_0
• If you want to reuse your exe, keep a copy with a different name

rt.sh Usage

• ./rt.sh: display usage information
• ./rt.sh -a | -c | -e | -h | -k | -w | -d | -l | -m | -n | -r
  • -a to use on for HPC queue"
  • -c create new baseline results"
  • -e use ecFlow workflow manager"
  • -h display this help"
  • -k keep run directory after rt.sh is completed"
  • -l runs test specified in "
  • -m compare against new baseline results"
  • -n run single test "
  • -r use Rocoto workflow manager"
  • -w for weekly_test, skip comparing baseline results"
  • -d delete run directories that are not used by other tests"

Run Full Regression Tests

• If you make code changes that are not expected to change simulation results, you can run full regression tests afterward to demonstrate your changes do not break anything
• Currently, there are over 150 standard regression tests built and run on supported TIER-1 platforms.
• In ufs-weather-model/tests/ directory, use any one of the following:
  • $ ./rt.sh -a >output 2>&1 & (run rt.sh using HPC account )
  • $ ./rt.sh -e >output 2>&1 &(use ecFlow)
  • $ ./rt.sh -r >output 2>&1 &(use Rocoto)
  • $ ./rt.sh -ek >output 2>&1 &(use ecFlow, keep run directory for post-run diagnosis)

Run a Single Regression Test

• Create a file, say my_test.conf, with a single COMPILE and a single RUN
  • $ cp rt.conf my_test.conf
  • $ vi my_test.conf
  • $ ./rt.sh -l my_test.conf
• Or use -n option
  • $ ./rt.sh -n cpld_control_p8 >output 2>&1 &

Create a New Baseline of Existing Test

• Your code changes are expected to change simulation results (e.g., physics change), and thus cannot be compared against existing baseline results
• You still need RTPWD as it contains the simulation input data
• ./rt.sh -c -f OR ./rt.sh -c -l my_test.conf
  • rt.sh will copy input data from RTPWD to NEW_BASELINE
  • Simulation results will be copied from RUNDIR to NEW_BASELINE
• Manually move your NEW_BASELINE to emc.nemspara

Run Regression Test against New Baseline

• You have generated new baseline
• You want to compare all your subsequent tests against the new baseline
• ./rt.sh -m -f OR ./rt.sh -m -l my_test.conf
• Internally, -m flag sets RTPWD=${NEW_BASELINE}

Add a New Test

• Configuration files (select, or copy and modify):
  • rt.conf
  • tests/<test-name>
  • tests/fv3_conf/
  • parm/input.*.nml.IN
  • parm/nems.configure.*.IN
  • parm/model_configure.IN
  • parm/ice_in_template
  • parm/MOM_input_template
• ./rt.sh -c -l my_test.conf
  • Will not compare with existing baselines
• If your case requires new input data not in RTPWD, set RTPWD to your local directory

Already have an Executable File (needs update)

• Remove COMPILE row in rt.conf
• $ cp ../NEMS/exe/NEMS.x fcst_0.exe
• $ cp ../NEMS/src/conf/modules.nems modules.fcst_0
  • This module file needs to be identical to the one you used for build
• $ ./rt.sh
• This approach does not work with workflow managers because RUN depends on COMPILE

Output Files and Log Files for Diagnosis

• Summary files
  • Hera: RegressionTests_hera.log
  • Orion: RegressionTests_orion.log
  • MISSING file, MISSING baseline, OK, NOT OK...
• ./rt.sh >output 2>&1 &: output of rt.sh
• Log files in log_hera/ and log_orion/
  • compile_*.log: output of compile.sh
  • run_*.log: output of run_test.sh
• Run directory RUNDIR_ROOT/
  • subdir: contains all files necessary for simulation, e.g., sbatch job_card
  • QUEUE is set in rt.sh