Merge Pull Request #3044 from E3SM-Project/scream/jgfouca/update_eamx…

…x_testing_docs

Automatically Merged using E3SM Pull Request AutoTester
PR Title: Update EAMxx docs and improve help formatters
PR Author: jgfouca
PR LABELS: AT: AUTOMERGE, AT: Integrate Without Testing

components/eamxx/docs/developer/ci_nightly.md

@@ -1,4 +1,17 @@
 # Continuous Integration and Nightly Testing
 
-* Autotester quick overview
-* Nightly overview, CDash
+## Autotester ##
+
+EAMxx using github actions and a Sandia product called Autotester 2
+to run CI testing on a CPU and GPU machine for every github pull
+request. By default, we run the e3sm_scream_v1_at suite and the
+standalone eamxx tests (test-all-scream).
+
+## Nightly overview, CDash ##
+
+Our nightly testing is much more extensive than the CI testing. You
+can see our dashboard here under the section "E3SM_SCREAM":
+https://my.cdash.org/index.php?project=E3SM
+
+We run a variety of CIME test suites and standalone testing on a number
+of platforms. We even do some performance testing on frontier.

components/eamxx/docs/developer/cime_testing.md

@@ -1,7 +1,59 @@
 # Full Model Testing
 
-Quickly review CIME test infrastructure and how EAMxx uses it
+Full model system testing of eamxx is done through CIME test cases
+(much like the rest of E3SM).
 
-* test types, specifiers (`_LnX`,`_D`,`_PMxN`,..), grids, compsets, test-mods
-* available grids/compsets for EAMxx, and where to find them
-* how to add atmchange in `shell_commands` test mods
+We offer a number of test suites, including:
+* e3sm_scream_v0: Test the full set of V0 (pre-C++) tests
+* e3sm_scream_v1: Test the full set of V1 (C++) tests
+* e3sm_scream_v1_at: A smaller and quicker set of tests for autotesting
+* e3sm_scream_hires: A small number of bigger, longer-running tests to measure performance
+
+Example for running a suite:
+```
+% cd $repo/cime/scripts
+% ./create_test e3sm_scream_v1_at --wait
+```
+
+Example for running a single test case:
+```
+% cd $repo/cime/scripts
+% ./create_test SMS.ne4_ne4.F2010-SCREAMv1 --wait
+```
+
+There are many behavioral tweaks you can make to a test case, like
+changing the run length, test type, etc. Most of this is not specific
+to eamxx and works for any CIME case. This generic stuff
+is well-documentated here:
+http://esmci.github.io/cime/versions/master/html/users_guide/testing.html
+
+When it comes to things specific to eamxx, you have grids, compsets, and
+testmods.
+
+Common EAMxx grids are:
+* ne4_ne4 (low resolution)
+* ne4pg2_ne4pg2 (low resolution with phys grid)
+* ne30_ne30 (med resolution)
+* ne30pg2_ne30pg2 (med resolution with phys grid)
+* ne1024pg2_ne1024pg2 (ultra high with phys grid)
+
+More grid info can be found here:
+https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/933986549/ATM+Grid+Resolution+Summary
+
+Common EAMxx compsets are:
+* F2010-SCREAM-LR: V0 low res compset with eamxx V0 atmosphere
+* F2010-SCREAMv1: V1 standard compset with eamxx V1 atmosphere
+* FIOP-SCREAMv1-DP: V1 with dpxx (doubly-periodic lateral boundary condition in C++)
+* F2010-SCREAMv1-noAero: V1 without aerosol forcing
+
+Full info on supported compsets can be found by looking at this file:
+`$scream_repo/components/eamxx/cime_config/config_compsets.xml`
+
+Common EAMxx testmods are:
+* small_kernels: Enable smaller-granularity kernels, can improve performance on some systems
+* scream-output-preset-[1-6]: Our 6 output presets. These turn some combination of our three output streams (phys_dyn, phys, and diags),
+  various remaps, etc.
+* bfbhash: Turns on bit-for-bit hash output: https://acme-climate.atlassian.net/wiki/spaces/NGDNA/pages/3831923056/EAMxx+BFB+hashing
+
+More info on running EAMxx can be found here:
+https://acme-climate.atlassian.net/wiki/spaces/DOC/pages/3386015745/How+To+Run+EAMxx+SCREAMv1

components/eamxx/docs/developer/standalone_testing.md

@@ -21,7 +21,43 @@ configurations. We use several types of tests
   the [ctest](https://cmake.org/cmake/help/latest/manual/ctest.1.html) command.
 
 We also support a `test-all-scream` configuration that runs all of the
-standalone tests for an EAMxx configuration.
+standalone tests for an EAMxx configuration. Note, your current machine
+must be known to EAMxx before `test-all-scream` will work. A machine can
+be made known to EAMxx by editing the eamxx/scripts/machines_specs.py files.
+There are some instructions on what to do at the top of this file.
+
+`test-all-scream` has a good help dump
+```
+% cd $scream_repo/components/eamxx
+% ./scripts/test-all-scream -h
+```
+
+If you are unsure of the cmake configuration for you development cycle, one
+trick you can use is to run `test-all-scream` for the `dbg` test and just
+copy the cmake command it prints (then ctrl-C the process).
+```
+% cd $scream_repo/components/eamxx
+% ./scripts/test-all-scream -t dbg -m $machine
+* wait for a few seconds*
+* Ctrl-C *
+* Copy the contents of DCMAKE_COMMAND that was passed to ctest *
+* Add "cmake" to beginning of contents and path to eamxx at the end. *
+```
+
+Considerations for using `test-all-scream`:
+* Your machine must be known to our scripts, see above.
+* If you try to run commands by-hand (outside of test-all-scream; cmake, make, ctest, etc), you'll need to remember to
+  load the scream-env into your shell, which can be done like this:
+  `cd eamxx/scripts; eval $(./scripts/scream-env-cmd $machine)`
+* test-all-scream expects to be run from a compute node if you
+  are on a batch machine.
+* You'll need to think about your baseline situation, as many of our
+  tests rely on pre-existing baselines. The -b option controls the baseline
+  location and can have the following values:
+  * AUTO: A common public baseline area shared by all developers
+  * LOCAL: A private baseline area for the current developer in the current repo
+  * $path: A specific arbitrary path
+  * None: If there is no -b at all, no baseline testing will be done
 
 ## Running EAMxx's Tests with CTest
 

components/eamxx/scripts/atmchange

@@ -15,7 +15,7 @@ sys.path.append(os.path.dirname(os.path.realpath(__file__)))
 from eamxx_buildnml_impl import check_value, is_array_type
 from eamxx_buildnml import create_raw_xml_file
 from atm_manip import atm_config_chg_impl, buffer_changes, reset_buffer, get_xml_nodes, parse_change
-from utils import run_cmd_no_fail, expect
+from utils import run_cmd_no_fail, expect, GoodFormatter
 
 # Add path to cime
 _CIMEROOT = os.path.join(os.path.dirname(os.path.realpath(__file__)), "..","..","..","cime")
@@ -110,7 +110,7 @@ OR
     > {0} foo=hi bar+=there
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument(

components/eamxx/scripts/atmquery

@@ -14,7 +14,7 @@ sys.path.append(os.path.dirname(os.path.realpath(__file__)))
 
 from eamxx_buildnml_impl import check_value
 from atm_manip import atm_query_impl
-from utils import expect
+from utils import expect, GoodFormatter
 
 ###############################################################################
 def atm_query(variables,listall=False,full=False,value=False, \
@@ -55,7 +55,7 @@ OR
 
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument(

components/eamxx/scripts/check-hashes-ers

@@ -10,7 +10,7 @@ hash output after a test has run.
 
 import sys, re, glob, pathlib, argparse
 
-from utils import run_cmd_no_fail, expect
+from utils import run_cmd_no_fail, expect, GoodFormatter
 
 ###############################################################################
 def parse_command_line(args, description):
@@ -25,7 +25,7 @@ OR
     > {0} /my/case/dir
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument(

components/eamxx/scripts/check-input

@@ -4,7 +4,7 @@
 Download SCREAM input files using CIME
 """
 
-from utils import check_minimum_python_version
+from utils import check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 4)
 
 import argparse, sys, pathlib
@@ -24,7 +24,7 @@ OR
     > {0} /my/input/root my/file
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument("input_root", help="The root input area (called DIN_LOC_ROOT in CIME)")

components/eamxx/scripts/check-tendencies

@@ -10,7 +10,7 @@
  (other dims omitted for brevity).
 """
 
-from utils import check_minimum_python_version
+from utils import check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 4)
 
 import argparse, sys, pathlib
@@ -24,7 +24,7 @@ def parse_command_line(args, description):
         usage=" {0} -f <file> -v <variables_names> -t <tendencies_names>"
               .format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.RawDescriptionHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     # The name of the nc files where to grab data from

components/eamxx/scripts/cime-nml-tests

@@ -6,7 +6,7 @@ namelist-related infrastructure.
 """
 
 from utils import check_minimum_python_version, expect, ensure_pylint, get_timestamp, \
-                  run_cmd_assert_result, run_cmd_no_fail, run_cmd
+                  run_cmd_assert_result, run_cmd_no_fail, run_cmd, GoodFormatter
 
 check_minimum_python_version(3, 6)
 
@@ -410,7 +410,7 @@ OR
     parser = argparse.ArgumentParser(
         usage=help_str,
         description=desc,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+        formatter_class=GoodFormatter)
 
     parser.add_argument("-m", "--machine",
                         help="Provide machine name. This is required for full (not dry) runs")

components/eamxx/scripts/compare-nc-files

@@ -5,7 +5,7 @@ Populates a netcdf file adding requested variables, either importing them
 from another file, or by computing them as function of other existing ones.
 """
 
-from utils import check_minimum_python_version
+from utils import check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 4)
 
 import argparse, sys, pathlib
@@ -36,7 +36,7 @@ OR
 
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     # The name of the nc files where to grab data from

components/eamxx/scripts/eamxx-params-docs-autogen

@@ -10,7 +10,7 @@ a doc string and its type, as well as, if present, constraints and valid values.
 
 import argparse, sys, os, pathlib
 
-from utils import _ensure_pylib_impl
+from utils import _ensure_pylib_impl, GoodFormatter
 
 _ensure_pylib_impl("mdutils")
 
@@ -29,7 +29,7 @@ def parse_command_line(args, description):
         usage="""{0}
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     return parser.parse_args(args[1:])

components/eamxx/scripts/gather-all-data

@@ -20,7 +20,7 @@ A number of options support automatic magic strings. See the help for individual
 options to see which magic strings are supported.
 """
 
-from utils import expect, check_minimum_python_version
+from utils import expect, check_minimum_python_version, GoodFormatter
 from git_utils import get_current_commit
 
 check_minimum_python_version(3, 5)
@@ -52,7 +52,7 @@ OR
     > {0} '$scream/scripts/perf-analysis ncol:8000 km1:128 km2:256 minthresh:0.001 repeat:10 --kokkos=$kokkos --test=lin-interp/li_ref --test=lin-interp/li_kokkos --test=lin-interp/li_vect -s ncol:2:128000' -m blake -o
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument("run", help="What to run on the machine, cwd for the command will be root-dir. "

components/eamxx/scripts/gen-boiler

@@ -11,7 +11,7 @@ To test this tool, run:
 % python3 -m doctest gen_boiler.py
 """
 
-from utils import expect, check_minimum_python_version
+from utils import expect, check_minimum_python_version, GoodFormatter
 from git_utils import get_git_toplevel_dir
 
 check_minimum_python_version(3, 5)
@@ -42,7 +42,7 @@ OR
     > {0} foo_bar --piece=f90_f2c_bind --piece=f90_c2f_bind -o --physics=p3
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.RawTextHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     default_repo = get_git_toplevel_dir()

components/eamxx/scripts/git-merge-ref

@@ -4,7 +4,7 @@
 A command-line tool foro calling merge_git_ref
 """
 
-from utils import check_minimum_python_version
+from utils import check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 4)
 
 import argparse, sys, pathlib
@@ -24,7 +24,7 @@ OR
     > {0} origin/master
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument("git_ref", help="The git ref to merge")

components/eamxx/scripts/machines_specs.py

@@ -8,7 +8,7 @@
 
 # MACHINE -> (env_setup,                      # list of shell commands to set up scream-approved env
 #             compilers,                      # list of compilers [CXX, F90, C]
-#             batch submit prefix,            # string shell commmand prefix
+#             batch submit prefix,            # string shell commmand prefix, leave blank for non-batch machines
 #             pre-existing baselines root dir # string path)
 MACHINE_METADATA = {
     # NOTE: blake must use a different minor version of python than weaver

components/eamxx/scripts/nc-file-init

@@ -5,7 +5,7 @@ Create a netcdf file for a cubed sphere geometry, with given vertical and
 horizontal resolution, adding requested variable (to be set to 0).
 """
 
-from utils import check_minimum_python_version
+from utils import check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 4)
 
 import argparse, sys, pathlib
@@ -32,7 +32,7 @@ OR
 
 """.format(pathlib.Path(args[0]).name),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     # The file name

components/eamxx/scripts/perf-analysis

@@ -5,7 +5,7 @@ Run different versions of an app and compare performance. It is
 expected that this will be run from micro-apps directory.
 """
 
-from utils import expect, check_minimum_python_version
+from utils import expect, check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 4)
 
 import argparse, sys, os, socket
@@ -38,7 +38,7 @@ OR
     > {0} ni:64 --test=ref:"src/physics/shoc/tests/shoc_run_and_cmp_cxx -r 10 -s 30 -i NI foo" -s ni:2:8192 --cd
 """.format(os.path.basename(args[0])),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument("argmap", nargs="+", help="Argument map, NAME:STARTING_VAL. First arg is assumed to be the arg/s used in the core metric")

components/eamxx/scripts/plot

@@ -4,7 +4,7 @@
 Plot results from perf_analysis
 """
 
-from utils import expect, check_minimum_python_version
+from utils import expect, check_minimum_python_version, GoodFormatter
 check_minimum_python_version(3, 5)
 
 import matplotlib.pyplot as plt
@@ -34,7 +34,7 @@ OR
     > {0} <sha1>/weaver <sha2>/weaver -v commit:version
 """.format(os.path.basename(args[0])),
         description=description,
-        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+        formatter_class=GoodFormatter
     )
 
     parser.add_argument("datafiles", nargs="+", help="Data file to plot")