Feature/ufs sr wx app Version of regional workflow to use with ufs-sr…

…weather-app (NOAA-EMC#231)

* Modify paths in setup.sh to work in ufs-srweather-app directory structure.

* Remove Externals.cfg, build scripts and manage_externals, since these
are now part of the ufs-srweather-app.

* Add path for SFC_CLIMO_INPUT_DIR for jet.

* Update exregional_make_grid.sh with community_develop.

* Feature/ufs sr wx app Merge code changes from develop branch (NOAA-EMC#229)

* Add Python plotting script for FV3SAR grib2 output (NOAA-EMC#215)

Co-authored-by: Benjamin.Blake EMC <[email protected]>

* Adding pull request template file, moving special files to .github directory (NOAA-EMC#217)

* Move CODOWNERS file to .github directory; make draft pull request template

* Change format a bit, I like this better since it looks nicer on github

* Edit README.md. (NOAA-EMC#220)

Print newlines before step 4 and step 5 so that they show up as separate entries in the list.

* Improve the way fixed files are handled. (NOAA-EMC#212)

* Improve the way fixed files are handled.

* Improve the way fixed files are handled so that what is happening with the fixed files is clearer.
* Reduce the number of symlinks needed in each cycle directory.
* Fix the EMC_CONUS_3km grid settings so that it has a valid BLOCKSIZE.
* Add new WE2E test for EMC_CONUS_3km grid.

List of modifications common to more than one file (these are referenced below for each file as needed):
-------------------------------------------------------------------------------------------------------

(A) Instead of passing in NPROCS as an environment variable via the rocoto XML, calculate nprocs locally using workflow variables.
(B) Use new function check_for_preexist_dir_file() that replaces old function check_for_preexist_dir().  The new one works for both directories and files.
(C) Use the new function set_FV3_sfc_climo_filenames() to set those variables in the FV3 namelist file that specify the paths to the surface climatology files on the native grid.
(D) Improve and/or clean up comments and/or informational and/or error messages.
(E) Put in a check to make sure that the executable chgres_cube.exe exists.
(F) Do not export variables when not necessary.
(G) Remove commented-out (unused) code.
(H) Remove blank lines.

File-by-file modifications:
--------------------------

jobs/JREGIONAL_GET_EXTRN_FILES:
* For clarity, rename this to JREGIONAL_GET_EXTRN_MDL_FILES.

jobs/JREGIONAL_MAKE_ICS: (A)

jobs/JREGIONAL_MAKE_LBCS: (A)

jobs/JREGIONAL_MAKE_SFC_CLIMO: (B)

jobs/JREGIONAL_RUN_POST: (B)
* Add check to make sure forecast hour contains exactly two digits.  That's all EMC_post seems to be able to handle at the moment.

scripts/exregional_make_grid.sh: (C), (D), (F)
* Remove topo_dir as a variable since it is not needed in this script.

scripts/exregional_make_ics.sh: (D), (E)

scripts/exregional_make_lbcs.sh: (D), (E)

scripts/exregional_make_orog.sh: (B), (F)
* Remove definition of topo_dir in CHEYENNE stanza since this variable gets redefined later on anyway (for all machines).

scripts/exregional_make_sfc_climo.sh: (A)

scripts/exregional_run_fcst.sh: (A), (D)
* Do not use the (now removed) workflow arrays FIXgsm_FILENAMES and FIXam_FILENAMES to create links in the cycle directory to fixed files in the FIXam directory.  Instead, use the (new) workflow array CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING that specifies the mapping between the symlinks that need to be created in the cycle directory and the corresponding files in the FIXam directory.
* In community mode, make the symlinks that are created in the cycle directory to the certain files relative.  These files consist of the data_table, field_table, the FV3 namelist, nems_configure, CCPP physics suite, and (for some physics suites) the CCN_ACTIVATE.BIN files.
* Do not copy the forecast executable to the cycle directory.  Instead, just call it using its full path in its original location (in EXECDIR).  This saves disk space and reduces clutter.

scripts/exregional_run_post.sh: (A), (G), (H)

tests/baseline_configs/config.EMC_CONUS_3km.sh:
* Add this new WE2E test configuration.  This tests the workflow on EMC's 3km CONUS grid in NCO mode (i.e. RUN_ENVIR set to "nco").

tests/baseline_configs/config.regional_005.sh:
* Specify non-default computational resources for this WE2E test (the 50km Alaska domain) such that the make_ics, make_lbcs, and run_post tasks do not crash (the number of MPI processes cannot be too high; otherwise, these tasks will fail).

tests/baseline_configs/config.regional_009.sh:
* Uncomment lines specific for Hera and comment out lines specific for Jet.  This has to be changed to include an if-statement that sets parameters based on the machine.

tests/baseline_configs/config.regional_015.sh:
* Add this new WE2E test configuration.  This tests the workflow on a coarse GFDLgrid type of grid that has a stretch factor of almost 1 (it cannot be exactly 1 because then the ufs_weather_model code assumes that it is a global grid and thus expects 6 tiles; this is a bug in the ufs_weather_model code that needs to be fixed).  A more refined version of this grid (refine_ratio changed from 2 as it is here to 36) is being tested by GSL until issues with the JPgrid verification are worked out.  Thus, we make sure capability of the workflow to run on this GFDLgrid doesn't break.

tests/baselines_list.txt:
* Update the list to include the new WE2E tests that are included in this commit (regional_014, regional_015, and EMC_CONUS_3km).  This file isn't really necessary and should be deleted at some point (e.g. in a different commit that fixes up various deficiencies with the WE2E tests).

ush/bash_utils/check_for_preexist_dir.sh:
* Modify the function this file defines such that it now works for files as well as directories.
* Rename the file and the function that it defines from check_for_preexist_dir(.sh) to check_for_preexist_dir_file(.sh).

ush/compare_config_scripts.sh:
* Change variable DEFAULT_EXPT_CONFIG_FN to EXPT_DEFAULT_CONFIG_FN (this variable is renamed in ush/setup.sh).

ush/config_defaults.sh:
* Put the variable SCHED specifying the job scheduler to use in this file so that it becomes user-specifiable (and avaiable as a workflow variable).  This may be needed when adding the capability to run outside of the scheduler (in a future commit/PR).
* Include detailed descriptions of the NCO mode variables (COMINgfs, STMP, NET, envir, RUN, PTMP) are used.
* Add new variable (EXPT_CONFIG_FN) that specifies the name of the experiment/workflow configuration file containing user-specified values.
* Add new variable (FV3_EXEC_FN) that specifies the name to use for the forecast model executable if/when it is copied from the directory in which the build step creates it to the directory in which the workflow scripts look for it (EXECDIR).  This copy step should really be done in during the build.
* For clarity and consistency, rename variables FV3_NML_FN and FV3_NML_CONFIG to FV3_NML_BASE_FN and FV3_NML_YAML_CONFIG_FN, respectively.
* Add new array SFC_CLIMO_FIELDS containing the names of the surface climatology fields used by the sfc_climo_gen code.
* Remove some variables (FNALBC, FNALBC2, etc) whose names are now specified in the new array FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING.
* Add new array FIXgsm_FILES_TO_COPY_TO_FIXam that specifies the fixed files that must be copied from the system directory FIXgsm to the local directory FIXam (which is under the experiment directory).
* Add new array FV3_NML_VARNAME_TO_FIXam_FILES_MAPPING that specifies the FV3 namelist variable names and the corresponding files in the FIXam directory that they are associated with (the variables need to be set to the relative or full paths to these files).
* Add new array FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING that specifies the FV3 namelist variable names and the corresponding surface climatology fields in the sfc_climo_gen code (the variables need to be set to the relative or full paths to the files containing the corresponding fields).
* Add new array CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING specifying the symlinks that need to be created in each cycle directory and their corresponding target files in the FIXam directory.
* Add new variables specifying the workflow task names and the corresponding values of the number of nodes, logical processes per node, and maximum wallclock time for each task.

ush/generate_FV3SAR_wflow.sh: (D)
* Remove variables PROC_RUN_FCST and NPROCS_RUN_FCST since they are now superceded by the variables NNODES_RUN_FCST and PPN_RUN_FCST.
* Remove the variable CYCLE_DIR since it is being replaced by CYCLE_BASEDIR (which is set in ush/setup.sh; see description of changes to that file).
* For clarity, reorder (and include comments between) the lines in the variable "settings" that specifies how to set the variables (entities, etc) in the template rocoto XML.
* Replace code that copies files from the FIXgsm directory with code that does the same thing using (looping over) the new workflow array variable FIXgsm_FILES_TO_COPY_TO_FIXam.
* Put in new code that copies the forecast model executable from the directoy in which the forecast model repository was cloned into the executable directory (EXECDIR) but ONLY IF such a copy does not
already exist in EXECDIR or, if it does exist, it is older than the source executable.  This copying should really be done in the build step, but it currently is not.
* For clarity, reorder some of the lines in the variable "settings" that is used to specify values in the FV3 fortran namelist file (input.nml) such that they are grouped by type (e.g. grid-related, computational, etc).  Also, include some comments between the lines.
* Remove the hard-coded settings of the "FN..." variables from the "settings" variable.  Instead:
  * For the "FN..." variables that specify paths to fixed files in the FIXam directory, loop through the new workflow array variable FV3_NML_VARNAME_TO_FIXam_FILES_MAPPING and, for each element of this array, create an appropriate entry in "settings".
  * For the "FN..." variables that specify the paths to surface climatology files:  (1) If not running the make_grid task, call the new function set_FV3nml_sfc_climo_filenames that sets those FV3 namelist variables that specify the paths to surface climatology files to the names of the surface climatology files on the FV3SAR grid (these files are either pregenerated or are created by the make_sfc_climo task).  (2) If running the make_grid task, call this same funtion in the make_grid task.  This is because the names of the surface climatology files depend on the CRES parameter (the C-resolution of the grid), and usually, that is not known until the grid is created (by the make_grid task).

ush/link_fix.sh:
* Replace local variable sfc_climo_fields with new workflow array variable SFC_CLIMO_FIELDS.
* Add code that creates symlinks in the FIXsar directory to surface climatology files with names that end in "tile7.nc" that have names that end in "tile1.nc".  This is apparently necessary for FV3 to be able to use the surface climatology files on the native FV3SAR grid (which are either pregenerated or created by the make_sfc_climo task).

ush/load_modules_run_task.sh: (D)
* Fix indentation.

ush/set_FV3nml_sfc_climo_filenames.sh:
* This is a new file that defines a function (set_FV3nml_sfc_climo_filenames) that sets the values of the variables in the forecast model's namelist file that specify the paths to the surface climatology files on the FV3SAR native grid (which are either regenerated or created by the MAKE_SFC_CLIMO_TN task).  We make this a function because it is needed in a couple of places in the scripts.

ush/set_fix_filenames.sh:
* Remove this file since the fixed files are being handled differently now using the new workflow array variables defined in ush/config_defaults.sh.

ush/set_ozone_param.sh:
* This is a new file that defines a function (set_ozone_param) that sets various workflow parameters depending on which ozone parameterization is being used by the experiment.  The ozone parameterization is determined by inspecting the CCPP suite defintion file.

ush/set_predef_grid_params.sh: (G)
* Set BLOCKSIZE to an allowed value for the EMC_CONUS_3km grid.

ush/setup.sh: (D), (G)
* For clarity, rename variable DEFAULT_EXPT_CONFIG_FN to EXPT_DEFAULT_CONFIG_FN.
* Remove variable EXPT_CONFIG_FN since that is now defined in config_defaults.sh.
* Set SCHED to a machine-dependent value only if it is not already specified by the user in config.sh.
* Remove stanza for machine THEIA (macine no longer exists).
* Change SCHED to lowercase (since that is how rocoto accepts it) and Test to make sure that SCHED is set to a valid value.
* Remove setting of variables that specify the task names since those are now in config_defaults.sh.
* Set the new workflow variable CYCLE_BASEDIR that contains the base directory under which the cycle directories are created.  This replaces CYCLE_DIR.  Note that CYCLE_BASEDIR is the old CYCLE_DIR but without the last directory (which is the CDATE of the cycle).  Now, CYCLE_DIR is formed within the XML by combining CYCLE_BASEDIR and the current cycle's date information from rocoto.  Also, add CYCLE_BASEDIR to the variable definitions file (var_defns.sh).
* For clarity, rename variable FV3_NML_CONFIG_FP to FV3_NML_YAML_CONFIG_FP.
* For convenience, create variable FV3_NML_FN (that is set to the name of the namelist file that FV3 expects to read in).  Then use this to set FV3_NML_FP, which is the full path to that namelist file.
* Create new workflow variable FV3_EXEC_FP that conains the full path to the copy of the FV3SAR executable in the executables directory EXECDIR.  Add this new variable to the variable definitions file (in a "cat" statement towards the end of the file).
* Set the new variable LOAD_MODULES_RUN_TASK_FP that specifies the full path to the script that loads the modules for each task and then runs the task.  This used to be hard-coded inside of the rocoto XML but is now needed as a workflow variable (because it is needed in at least a couple of places).  Add this variable to the variable definitions file (var_defns.sh).
* Fix bug (the way the local variable glob_pattern is set) to allow for the use of a dot as the fixed file name separator when in NCO mode (i.e. when RUN_ENVIR is set to "nco").
* Rename the variable NUM_NODES to NNODES_RUN_FCST to be consistent with the naming convention for number of nodes used for other tasks.
* Remove call to function set_fix_filenames since it is no longer needed.
* Remove recording of the array variables FIXgsm_FILENAMES and FIXam_FILENAMES in the variable defintions file since these arrays are no longer needed (they were previously set by the function set_fix_filenames).
* Call the new function set_ozone_param that sets various workflow parameters depending on which ozone parameterization is being used (which is specified in the CCPP suite defintion file).
* Clean up comments.

ush/source_util_funcs.sh: (D)
* Change the file name check_for_preexist_dir.sh to check_for_preexist_dir_file.sh.

ush/templates/FV3SAR_wflow.xml:
* For clarity, reorder the entities that contain jinja parameters by grouping similar items together.
* Make LOAD_MODULES_RUN_TASK_FP an ENTITY that is set externally by the workflow generation scripts (as opposed to hard-coded in the XML) because this is now a variable that is needed in another file (the script that is used to run a task outside of rocoto).
* Replace CYCLE_DIR as an externally specified ENTITY with CYCLE_BASEDIR.
* Create new externally-specified ENTITYs for the number of nodes, the number of MPI processes per node, and the wallclock time for each task.  These will get set by the workflow generation scripts.
* Make LOAD_MODULES_RUN_TASK_FP an ENTITY that is set externally by the workflow generation scripts (as opposed to hard-coded in the XML) because this is now a variable that is needed in another file (the script that is used to run a task outside of rocoto).
* Use the ENTITYs for the number of nodes, the number of MPI processes per node, and the wallclock time for each task to set the PROC_... and RSRC_... ENTITYs for each task (as opposed to hard-coding the latter).
* For brevity, rename ENTITYs containing the strings "MAKE_ICS_SURF_LBC0" and "MAKE_LBC1_TO_LBCN" to "MAKE_ICS" and "MAKE_LBCS", respectively.
* For consistency with other tasks, rename PROC_POST and RSRC_POST to PROC_RUN_POST and RSRC_RUN_POST.
* Change name of ENTITY RSRV_RUN_FCST to RSRV_FCST to be consistent with name of ENTITY QUEUE_FCST.  Names of ENTITYs related to the job scheduler need to be revisited (e.g. maybe change QUEUE_FCST to QUEPART_FCST since in slurm, there are partitions, not queues).
* Remove variable PDY from pre-processing tasks (which are cycle-independent).
* Fix bug where the task name "make_grid" is hard-coded as opposed to using the ENTITY MAKE_GRID_TN.
* For clarity, rename file JREGIONAL_GET_EXTRN_FILES to JREGIONAL_GET_EXTRN_MDL_FILES.
* Reorder environment variable settings in each task to go from least to most general.
* Within each cycle-dependent task, set CYCLE_DIR based on CYCLE_BASEDIR.

ush/valid_param_vals.sh:
* Add valid values for the new user-specifiable workflow variable SCHED.
* Remove the unused predefined grid GSD_HRRR_AK_3km since it has been renamed GSD_RRFSAK_3km.

* Manual merge of Mike's changes for file ush/load_modules_run_task.sh.

* Improve/streamline generation of rocoto workflow XML.

Modifications made to more than one file:
----------------------------------------
(A) For uniformity among workflow tasks, split the worfklow variables that specify the number of nodes, logical processes per node, and wall clock time for the generic GET_EXTRN_MDL_FILES task into two variables each -- one for the GET_EXTRN_ICS_TN and another for the GET_EXTRN_LBCS_TN task.
(B) Remove USHDIR as an ENTITY in the XML since it is not used after being set.

ush/config_defaults.sh: (A)

ush/generate_FV3SAR_wflow.xml: (B)

ush/templates/FV3SAR_wflow.xml: (A), (B)
* Clean up comments.
* To reduce the number of entities rocoto has to deal with, remove the ENTITY definitions for various entities that only appear in one location in the file and replace them with the corresponding jinja2 placeholders or combinations of such placeholders.  Do this for the following entities:
  * The per-task entities NNODES_..., PPN_..., WTIME_..., PROC_..., and RSRC_....
  * EXTRN_MDL_NAME_ICS and EXTRN_MDL_NAME_LBCS

* Get EMC's changes into community workflow (Ratko's PR plus other necessary changes).

File-by-file description of modifications:
-----------------------------------------

fix/fix_upp/nam_nests.hiresf_nn.txt:
* Deleted as requested by EMC (from Ratko's original PR).

fix/fix_upp/postxconfig-NT-fv3sar.txt
* Modified as requested by EMC (from Ratko's original PR).

scripts/exregional_get_extrn_files.sh:
* If extracting the external model files from one or more HPSS archive files, put in a check to ensure that each external model file was extracted exactly once from one of the archive files.

scripts/exregional_make_ics.sh:
* Add parameter settings for the FV3_GFS_2017_gfdlmp_regional physics suite (from Ratko's original PR).
* The FV3_GFS_2017_gfdlmp_regional physics suite requires a variable called tref (reference temperature) in the initial conditions.  That variable is available only for external files coming from the FV3GFS model.  tref can be extracted from the surface analysis file if the namelist variable convert_nst is set to .true. in the chgres_cube namelist file.  Thus, create a new local variable named convert_nst that gets set to true for the FV3GFS model but is false otherwise and pass it to the chgres_cube namelist.

scripts/exregional_make_lbcs.sh:
* Add parameter settings for the FV3_GFS_2017_gfdlmp_regional physics suite (from Ratko's original PR).

tests/baseline_configs/config.EMC_CONUS_3km.sh:
* Rename to tests/baseline_configs/config.nco_conus.sh (see that file).

tests/baseline_configs/config.nco_conus.sh:
* Renamed from tests/baseline_configs/config.EMC_CONUS_3km.sh.
* Change queue from debug to batch since we always want batch as the default (so that if running many tests, they are not limited by the max number of jobs a user can have in the debug queue, which on hera is just 2).
* Set the physics suite for this WE2E test to FV3_GFS_2017_gfdlmp_regional because that's the one EMC runs with.
* Reset the RUN variable (which is used in creating output directory if in NCO mode) to a value that is unique to this WE2E test so that it doesn't interfere with other tests that are running at the same time (and vice versa).

tests/baseline_configs/config.nco_conus_c96.sh:
* New WE2E test for the coarse EMC CONUS grid.  For rapid testing, this can be run instead of config.nco_conus.sh.

tests/baseline_configs/config.new_JPgrid.sh
* New WE2E test for specifying a new (i.e. not predefined) JPgrid.

tests/baseline_configs/config.regional_009.sh:
* Reset the RUN variable (which is used in creating output directory if in NCO mode) to a value that is unique to this WE2E test so that it doesn't interfere with other tests that are running at the same time (and vice versa).

tests/baselines_list.txt:
* Update the full list of WE2E test names to the latest set.

ush/config_defaults.sh:
* Edit comments.

ush/generate_FV3SAR_wflow.sh:
* Remove FNZORC, FNTSFA, FNACNA, and FNSNOA from the "settings" variable used in generating the FV3 input namelist file.  These variables are now instead specified in the base namelist file (ush/templates/input.nml.FV3) because they are the same for all physics suites considered thus far.

ush/get_extrn_mdl_file_dir_info.sh:
* Remove unused code.

ush/link_fix.sh:
* Bug fix:  Change the targets of the "...tile1..." symlinks so that they point to the tile7.halo0.nc files, not the tile7.nc symlinks (which in turn point to the tile7.halo4.nc files, which is not what we want).

ush/set_namelist.py:
* Add option "sort=True" to nml.write method to sort the resulting FV3 namelist file in alphabetical order.  This makes it easier to compare (diff) namelists.

ush/setup.sh:
* Add a temporary section of code that makes fixes to the suite defintion file for the FV3_GFS_2017_gfdlmp_regional physics suite.  This is necessary because the version of the file in the dtc/develop branch of the NCAR fork of the fv3atm repo has not yet been updated to have these corrections.  This code should be removed once that branch/repo have been updated.

ush/templates/FV3.input.yml:
* Add section for the new FV3_GFS_2017_gfdlmp_regional physics suite.  Note that this references the preexisting FV3_GFS_2017_gfdlmp physics suite.

ush/templates/input.nml.FV3:
* Reorder lines so that both namelists and the variables within each namelist are lower case and in alphabetical order.  This makes it easier to compare this base FV3 namelist file to other FV3 namelist files generated using the set_namelist.py script.

ush/templates/diag_table.FV3_GFS_2017_gfdlmp_regional:
* New diag_table file for FV3_GFS_2017_gfdlmp_regional suite.

ush/templates/field_table.FV3_GFS_2017_gfdlmp_regional:
* New field_table file for FV3_GFS_2017_gfdlmp_regional suite.

ush/templates/model_configure.FV3_GFS_2017_gfdlmp_regional:
* New model_configure file for FV3_GFS_2017_gfdlmp_regional suite.

ush/valid_param_vals.sh:
* Add FV3_GFS_2017_gfdlmp_regional to the list of valid physic suites (as requested by EMC).

* Change compile options for EMC as requested by Ratko.

* Allow names of external model files to be different on disk vs. from HPSS (NOAA-EMC#221)

Allow names of external model files to be different on disk vs. from HPSS.

Motivation:
----------
At least on jet, the names of the external model files can be different depending on whether they are being copied from disk or are being extracted from an archive file in HPSS (for the same external model).  Currently, the workflow assumes that these file names are the same regardless of where/how they are being obtained.  Thus, the workflow breaks on jet.  This PR generalizes the workflow to allow for the external model file names to be different (if necessary) depending on where/how they're being obtained.

Summary of Modifications:
------------------------
* Allow the names of the external model files copied from or linked to on disk to be different than the ones extracted from an archive file in HPSS.  This is the case on jet.
* In each of the get_extrn_ics and get_extrn_lbcs workflow tasks, after obtaining the external model files (either from disk or from HPSS), create a new file and save in it certain parameters that depend on this process of obtaining the external model files (e.g. the names of the external model files that were obtained, since now these names depend on whether they are obtained from disk or HPSS).  Note that one file is created for the get_extrn_ics task and another one for the get_extrn_lbcs task.  These two files are actually shell scripts containing variable definitions that get sourced later by the make_ics and make_lbcs tasks, respectively.  This change removes the need to call the get_extrn_mdl_file_dir_info again as part of the make_ics and make_lbcs tasks.
* Modify the way the temporary fix for the suite_FV3_GFS_2017_gfdlmp_regional physics suite works such that the suite definition file (SDF) is not changed in the ufs_weather_model directory where the clone of the ufs-weather-model repo is located (so that no changes to tracked files are reported in the clone of that repo) and instead, the SDF is first copied to the experiment directory and changed there.


Modifications common to more than one file (used below in listing of file-by-file modifications):
------------------------------------------------------------------------------------------------
(A) Rename local variables to be lower case.
(B) Fix/add/delete comments and/or informational and/or error messages.
(C) Remove theia stanza in machine-dependent case-statement.
(D) Remove calls to save_shell_opts and restore_shell_opts in the machine-dependent case-statement because those calls were to suppress output from the "module load" commands that were previously located in this case-statement, but the "module load"s have been moved out of this case-statement and into the ush/load_modules_run_task.sh script.
(E) Remove the call to the get_extrn_mdl_file_dir_info funtion because the information that this call returned is now obtained by sourcing new variable definition files created by the get_extrn_ics and get_extrn_lbcs workflow tasks.  The full paths to these new files (actually shell scripts that just contain the variable definitions) are, respectively:
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_ICS}/ICS/extrn_mdl_ics_var_defns.sh
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_LBCS}/LBCS/extrn_mdl_ics_var_defns.sh
(F) Remove some of the arguments to the call to scripts/exregional_make_ics.sh since those arguments are now obtained by sourcing the new variable definitions file extrn_mdl_ics_var_defns.sh created by the get_extrn_ics task.
(G) Remove some of the arguments to the call to scripts/exregional_make_lbcs.sh since those arguments are now obtained by sourcing the new variable definitions file extrn_mdl_lbcs_var_defns.sh created by the get_extrn_lbcs task.
(H) For clarity, change name of workflow variable from EXTRN_MDL_LBC_UPDATE_FHRS to EXTRN_MDL_LBC_SPEC_FHRS (where "SPEC_FHRS" stands for specification forecast hours, because the LBCs aren't being updated at these forecast hours, they're being specified).  Also, do same for the local (i.e. lowercase) version of this variable.
(I) For clarity, change name of workflow variable from LBC_UPDATE_INTVL_HRS to LBC_SPEC_INTVL_HRS for the same reason as explained above in (I).
(J) Change LBC_SPEC_INTVL_HRS from 6 to 3 to test having more than one LBC specification time after hour 0 (the forecast goes to hour 6).  For now, do this only for those WE2E tests having both EXTRN_MDL_NAME_ICS and EXTRN_MDL_NAME_LBCS set to "FV3GFS".  Don't do it for the external model being set to "GSMGFS" because that model does not provide output with 3-hour frequency, only 6-hour frequency (I think).

File-by-file modifications:
--------------------------

jobs/JREGIONAL_GET_EXTRN_MDL_FILES: (A), (B), (H)
* Use the function check_var_valid_vals to ensure that the value specified for ICS_OR_LBCS is valid.
* In the call to the function get_extrn_mdl_file_dir_info, instead of returning only one set of external model file names (varname_extrn_mdl_fns), return two sets -- one if the files are on disk (varname_extrn_mdl_fns_on_disk) and another if the files are in archive file on HPSS (varname_extrn_mdl_fns_in_arcv).

jobs/JREGIONAL_MAKE_ICS: (A), (B), (C), (D), (E), (F)

jobs/JREGIONAL_MAKE_LBCS: (A), (B), (C), (D), (E), (G)

jobs/JREGIONAL_RUN_POST: (B)

scripts/exregional_get_extrn_files.sh:
* Rename to scripts/exregional_get_extrn_mdl_files.sh.  See below for other changes.

scripts/exregional_get_extrn_mdl_files.sh: (A), (B), (H)
* Add new input argument ics_or_lbcs instead of assuming that an environment variable named ICS_OR_LBCS exists.
* Add new input argument extrn_mdl_cdate that will be written to the new variable definition file (extrn_mdl_ics_var_defns.sh or extrn_mdl_lbcs_var_defns.sh) that this script now creates (see bullet point below).
* Replace input argument EXTRN_MDL_FNS with the two new input arguments extrn_mdl_fns_on_disk and extrn_mdl_fns_in_arcv that are the external model file names if copying them from disk and if extracting them from an archive file(s), respectively.  Also, replace EXTRN_MDL_FNS throughout the script with either extrn_mdl_fns_on_disk or extrn_mdl_fns_in_arcv as appropriate.  This is part of the modifications to allow the names of the external model files copied from or linked to on disk to be different than the ones extracted from an archive file in HPSS (which is the case on jet).
* Remove the fix for resetting external model file names on jet because this is now done by the use of two different input arguments for the file names (extrn_mdl_fns_on_disk and extrn_mdl_fns_in_arcv).
* Add code that, after the external model files are obtained (copied, linked to, or extracted from archive), creates a new file (a shell script) that contains the definitions of several external-model related variables that will be needed by downstream tasks.  For the ICs (i.e. the get_extrn_ics task), the full path to this file is at
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_ICS}/ICS/extrn_mdl_ics_var_defns.sh
For the LBCs (i.e. the get_extrn_ics task), the full path to this file is at
  ${EXPT_DIR}/${CYCLE_DIR}/${EXTRN_MDL_NAME_LBCS}/LBCS/extrn_mdl_ics_var_defns.sh
These files are sourced by the make_ics and make_lbcs tasks, respectively, to obtain the necessary variable definitions.

scripts/exregional_make_ics.sh: (A), (B), (F)

scripts/exregional_make_lbcs.sh: (A), (B), (G), (H)

tests/baseline_configs/config.nco_conus.sh: (I), (J)

tests/baseline_configs/config.nco_conus_c96.sh: (I), (J)

tests/baseline_configs/config.new_JPgrid.sh: (I), (J)

tests/baseline_configs/config.regional_001.sh: (I)

tests/baseline_configs/config.regional_002.sh: (I), (J)

tests/baseline_configs/config.regional_003.sh: (I)

tests/baseline_configs/config.regional_004.sh: (I)

tests/baseline_configs/config.regional_005.sh: (I)

tests/baseline_configs/config.regional_006.sh: (I), (J)

tests/baseline_configs/config.regional_007.sh: (I), (J)

tests/baseline_configs/config.regional_008.sh: (I), (J)

tests/baseline_configs/config.regional_009.sh: (I)

tests/baseline_configs/config.regional_010.sh: (I)

tests/baseline_configs/config.regional_011.sh: (I)

tests/baseline_configs/config.regional_012.sh: (I)

tests/baseline_configs/config.regional_013.sh: (I)

tests/baseline_configs/config.regional_014.sh: (I)

tests/baseline_configs/config.regional_015.sh: (I), (J)

ush/config_defaults.sh: (B), (I)
* Add the new workflow variables EXTRN_MDL_ICS_VAR_DEFNS_FN and EXTRN_MDL_LBCS_VAR_DEFNS_FN that specify the names of the two new external-model-associated variable definitions files that the get_extrn_ics and get_extrn_lbcs now create.
* Remove extra spaces at ends of elements of array CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING.

ush/generate_FV3SAR_wflow.sh: (I)
* Put in an if-statement that renames the temporary SDF (suite_FV3_GFS_2017_gfdlmp_regional.xml.tmp) created in the experiment directory by the temporay fix in ush/setup.sh (see below) for the FV3_GFS_2017_gfdlmp_regional physics suite to the actual suite file (suite_FV3_GFS_2017_gfdlmp_regional.xml).  This code needs to be removed once the SDF for this physics suite is fixed in the dtc/develop branch of the NCAR fork of the ufs-weather-model repo.

ush/get_extrn_mdl_file_dir_info.sh: (A), (C), (H)
* In the arguments list, instead of returning only one set of external model file names (varname_extrn_mdl_fns), return two sets -- one if the files are on disk (varname_extrn_mdl_fns_on_disk) and another if the files are in archive file on HPSS (varname_extrn_mdl_fns_in_arcv).
* Remove some unused code.
* Add new local variables needed to create sets of file names for both the on-disk case and the in-archive-file case.
* Allow the input argument anl_or_fcst to contain a value that is either upper case or lower case (instead of just upper case as before).
* Change code to specify both the local array variables fns_on_disk and fns_in_arcv instead of the single array variable fns.  These two new variables are (so far) the same on all machines except Jet, in which case they may be different depending on the external model considered.
* Change the way output variables are set so that each is set only if the corresponding input argument that specifies its name is not empty.  This way, if certain output variables are not needed, the user can simply not specify them in the call to this function (in which case the process_args function will simply set them to empty strings) and not obtain any errors.

ush/set_ozone_param.sh: (B)

ush/setup.sh: (A), (I)
* Change the code that is a temporary fix for the fact that the SDF for the FV3_GFS_2017_gfdlmp_regional physics suite in the dtc/develop branch of the NCAR fork of the ufs-weather-model repo is currently incorrect so that it first copies the suite defintion file to the experiment directory and makes fixes to it there (instead of fixing it directly in its original location at sorc/ufs_weather_model/...).  As part of this change:
  a) Move the creation of the experiment directory (EXPTDIR) to before the temporary code.
  b) Move the original code that calls the function set_ozone_param in the else" portion of the "if" statement that runs the temporary code.
  These two changes can be undone later after the temporary code is removed.

* Fix for contrib module handling. (NOAA-EMC#226)

## DESCRIPTION OF CHANGES:.
All the module files contained in this repository have been updated for the change on Jet/Hera to remove the contrib module, and instead use the `module use -a` method for locating necessary contrib package modulefiles. This will be the ONLY way to use these modules after June 11, 2020, and is already in place to work now. As such, this is a **time sensitive** PR.

## TESTS CONDUCTED:.
Loaded a few different modules on front end node. This included modules that need to be loaded, and those "modules" that are actually shell scripts and must be sourced. Both methods worked as expected.

End-to-end tests completed successfully on Hera: 
new_JPgrid
regional_002
regional_003
regional_004
regional_005
regional_006
regional_007
regional_008
regional_015

* Fix directory overwrite bug in NCO mode. (NOAA-EMC#222)

Bug fix to get the WE2E tests to work properly in NCO mode.

Summary of modifications:
------------------------
* Modify scripts and WE2E configuration files that run in NCO mode to prevent simultaneously running experiments in NCO mode from writing to the same set of directories.
* Add a WE2E test on the GSD_RAP13km grid.

Modifications common to more than one file (used below in listing of file-by-file modifications):
------------------------------------------------------------------------------------------------
(A) Fix/add/delete comments and/or informational and/or error messages.
(B) Remove unused code and/or fix code indentation.
(C) Set the variables RUN and envir to EXPT_SUBDIR, which when the WE2E test is run will contain the name of the test.  Note that:
  * We can't get the name of the WE2E test from the file name because before running the experiment generation script for this WE2E test, the script that runs the WE2E tests copies this configuration file into ush and renames it config.sh (because that's the name the experiment generation script expects the configuration file to have).  Thus, at the time the experiment generation script is called, the file name no longer contains the name of the test, so it cannot be used to obtain the test name.
  * Even though EXPT_SUBDIR is set to a null string near the beginning of this file, the script that runs the WE2E tests inserts the name of the test into the definition of EXPT_SUBDIR (i.e. it changes the line 'EXPT_SUBDIR=""' into 'EXPT_SUBDIR="test_name"') before calling the experiment generation script.  Thus, EXPT_SUBDIR will be properly set to the name of the test when the experiment generation script is called.
(D) Bug fix - change variable name from LBC_UPDATE_INTVL_HRS to LBC_SPEC_INTVL_HRS.

File-by-file description of modifications:
-----------------------------------------

jobs/JREGIONAL_RUN_POST:
* In the definition of COMOUT, use the new workflow variable COMOUT_BASEDIR that specifies the base directory under which each cycle's UPP output directories will be placed.

scripts/exregional_get_extrn_files.sh:
* Remove this file.  This has been replaced with scripts/exregional_get_extrn_mdl_files.sh.

tests/baseline_configs/config.GSD_RAP13km.sh:
* This is a new WE2E test on the GSD_RAP13km grid using the FV3_GSD_v0 physics suite.

tests/baseline_configs/config.nco_conus.sh: (A), (C)

tests/baseline_configs/config.nco_conus_c96.sh: (A), (C)

tests/baseline_configs/config.regional_001.sh: (A)

tests/baseline_configs/config.regional_002.sh: (A)

tests/baseline_configs/config.regional_003.sh: (A)

tests/baseline_configs/config.regional_004.sh: (A)

tests/baseline_configs/config.regional_005.sh: (A)

tests/baseline_configs/config.regional_006.sh: (A)

tests/baseline_configs/config.regional_007.sh: (A)

tests/baseline_configs/config.regional_008.sh: (A)

tests/baseline_configs/config.regional_009.sh: (A), (B), (C)
* Change the time interval for specifying external model LBCs from 6 hrs to 3 hrs (to match what is done in all the other WE2E tests that use FV3GFS as the external model for LBCs).

tests/baseline_configs/config.regional_010.sh: (A)

tests/baseline_configs/config.regional_011.sh: (A)

tests/baseline_configs/config.regional_012.sh: (A)

tests/baseline_configs/config.regional_013.sh: (A)

tests/baseline_configs/config.regional_014.sh: (A)

tests/baseline_configs/config.regional_015.sh: (A)

tests/baselines_list.txt:
* Add the new GSD_RAP13km test to the full list of WE2E tests.

tests/run_experiments.sh: (A), (B)

ush/config.community.sh: (D)

ush/config.nco.sh: (D)

ush/config_defaults.sh: (A)
* Increase wallclock times for certain tasks since on hera some of the WE2E tests aren't able to complete within the allowed time.

ush/set_predef_grid_params.sh:
* Change the time step on the GSD_RAP13km grid from 90 sec to 50 sec to get a stable forecast.

ush/setup.sh: (A)
* For simplicity, combine two blocks of code that create different directories into a single block (that creates both sets of directories).
* In NCO mode, change defintion of CYCLE_BASEDIR from "$STMP/tmpnwprd/${EMC_GRID_NAME}" to "$STMP/tmpnwprd/$RUN".  This is to make it easier to associate this directory with an experiment (wich can be done by setting RUN to the experiment name in the experiment's configuration file).  That way, two experiments running simultaneously (but having different names) will not end up writing to the same CYCLE_BASEDIR (which currently can happen if the two experiments are using the same grid, i.e. the same value of EMC_GRID_NAME).
* Add COMOUT_BASEDIR as a new secondary workflow variable.  In NCO mode, set this to the base directory under which the output from the RUN_POST_TN task will be placed (i.e. the cycle-independent portion of the RUN_POST_TN task's output directory) and then check whether this directory exists (and deal with it as specified by the variable PREEXISTING_DIR_METHOD).  The goal here is to be able to uniquely associate this directory to an experiment so that simultaneously running experiments do not write their UPP outputs into the same directory (and possibly overwrite each other's results).  This directory is given by COMOUT_BASEDIR="$COMROOT/$NET/$envir", so we can uniquely associate it with an experiment by setting envir to the name of the experiment (which can be done in the experiment's configuration file).  This is why in the WE2E configuration files listed above (config.nco_conus.sh, config.nco_conus_c96.sh, and config.regional_009.sh), we set envir to the name of the experiment.  In community mode, the new variable COMOUT_BASEDIR is not used, so in that case we set it to an empty string.

* WCOSS port changes (NOAA-EMC#228)

Co-authored-by: Benjamin Blake <[email protected]>

Not tested on hera yet.  Will do soon.

Co-authored-by: BenjaminBlake-NOAA <[email protected]>
Co-authored-by: Benjamin.Blake EMC <[email protected]>
Co-authored-by: Michael Kavulich <[email protected]>
Co-authored-by: gsketefian <[email protected]>
Co-authored-by: Christina Holt <[email protected]>

* Modify link to module file for forecast run.  modules.nems is no longer
created by the cmake, static build for the ufs-community/ufs-weather-model.
Use modulefiles/$machine.intel/fv3.

Co-authored-by: BenjaminBlake-NOAA <[email protected]>
Co-authored-by: Benjamin.Blake EMC <[email protected]>
Co-authored-by: Michael Kavulich <[email protected]>
Co-authored-by: gsketefian <[email protected]>
Co-authored-by: Christina Holt <[email protected]>