Bugfix: Pass domain name to python plotting scripts (#783)

## DESCRIPTION OF CHANGES: 
PR #[763](ufs-community/regional_workflow#763) introduced the experiment variable `POST_OUTPUT_DOMAIN_NAME` and used it to modify the names of the UPP output files.  This PR adds `POST_OUTPUT_DOMAIN_NAME` as an argument to the python plotting scripts to enable those scripts to form the proper UPP output file names to read in.  It also modifies the batch scripts (`qsub_job.sh`, `qsub_job_diff.sh`, `sq_job.sh`, and `sq_job_diff.sh`) to add the new argument to the calls to the python plotting scripts that are called.

## TESTS CONDUCTED: 
Both python plotting scripts `plot_allvars.py` and `plot_allvars_diff.py` were run on Hera on preexisting UPP output files and completed successfully.  The batch scripts `sq_job.sh` and `sq_job_diff.sh` were also run successfully on Hera (`qsub_job.sh` and `qsub_job_diff.sh` were not run on e.g. Cheyenne).

## DEPENDENCIES:
PR # in `ufs-srweather-app`.

## DOCUMENTATION:
Modifications to the documentation are in PR #.

## ISSUE: 
Partially resolves issue #781 (still need to introduce tests in `regional_workflow` that run the plotting scripts).

## CONTRIBUTORS: 
@mark-a-potts pointed out the problem and first made the necessary fixes in the `NOAA-EPIC` fork of `regional_workflow`; @gsketefian introduced those changes to this PR and made edits to the comments and documentation.

ush/Python/plot_allvars.py

@@ -26,12 +26,16 @@
 #                            CARTOPY_DIR/shapefiles/natural_earth/cultural/*.shp
 #                          -More information regarding files needed to setup
 #                            display maps in Cartopy, see SRW App Users' Guide
+#                       7. POST_OUTPUT_DOMAIN_NAME:  Name of native domain
+#                          used in forecast and in constructing the names 
+#                          of the post output files.
 #
-#           		To create plots for forecast hours 20-24 from 5/7 00Z
-#                        cycle with hourly output:
+#           		To create plots for a forecast on the RRFS_CONUS_25km
+#                       grid for hours 20-24 from the 5/7 00Z cycle with 
+#                       hourly output:
 #                          python plot_allvars.py 2020050700 20 24 1 \
 #                          /path/to/expt_dirs/experiment/name \
-#                          /path/to/base/cartopy/maps
+#                          /path/to/base/cartopy/maps RRFS_CONUS_25km
 #
 #                       The variable domains in this script can be set to either
 #                         'conus' for a CONUS map or 'regional' where the map
@@ -227,6 +231,7 @@ def rotate_wind(true_lat,lov_lon,earth_lons,uin,vin,proj,inverse=False):
 parser.add_argument("Forecast hour increment")
 parser.add_argument("Path to experiment directory")
 parser.add_argument("Path to base directory of cartopy shapefiles")
+parser.add_argument("Name of native domain used in forecast (and in constructing post file names)")
 args = parser.parse_args()
 
 # Read date/time, forecast hour, and directory paths from command line
@@ -252,6 +257,7 @@ def rotate_wind(true_lat,lov_lon,earth_lons,uin,vin,proj,inverse=False):
 
 EXPT_DIR = str(sys.argv[5])
 CARTOPY_DIR = str(sys.argv[6])
+POST_OUTPUT_DOMAIN_NAME = str(sys.argv[7]).lower()
 
 # Loop over forecast hours
 for fhr in fhours:
@@ -261,7 +267,7 @@ def rotate_wind(true_lat,lov_lon,earth_lons,uin,vin,proj,inverse=False):
   vtime = ndate(itime,int(fhr))
 
 # Define the location of the input file
-  data1 = pygrib.open(EXPT_DIR+'/'+ymdh+'/postprd/rrfs.t'+cyc+'z.prslevf'+fhour+'.tm00.grib2')
+  data1 = pygrib.open(EXPT_DIR+'/'+ymdh+'/postprd/rrfs.t'+cyc+'z.prslev.f'+fhour+'.'+POST_OUTPUT_DOMAIN_NAME+'.grib2')
 
 # Get the lats and lons
   grids = [data1]

ush/Python/plot_allvars_diff.py

@@ -29,12 +29,17 @@
 #                            CARTOPY_DIR/shapefiles/natural_earth/cultural/*.shp
 #                          -More information regarding files needed to setup
 #                            display maps in Cartopy, see SRW App Users' Guide
+#                       8. POST_OUTPUT_DOMAIN_NAME:  Name of native domain
+#                          used in forecasts and in constructing the names 
+#                          of the post output files.  This must be the same 
+#                          for both forecasts.
 #
-#           		To create plots for forecast hours 20-24 from 5/7 00Z
-#                        cycle with hourly output:
+#           		To create difference plots for two forecasts on the
+#                       RRFS_CONUS_25km grid for hours 20-24 from the 5/7
+#                       00Z cycle with hourly output:
 #                          python plot_allvars_diff.py 2020050700  20 24 1 \
 #                          /path/to/expt_dir_1 /path/to/expt_dir_2 \
-#                          /path/to/base/cartopy/maps
+#                          /path/to/base/cartopy/maps RRFS_CONUS_25km
 #
 #                       The variable domains in this script can be set to either
 #                         'conus' for a CONUS map or 'regional' where the map
@@ -231,6 +236,7 @@ def rotate_wind(true_lat,lov_lon,earth_lons,uin,vin,proj,inverse=False):
 parser.add_argument("Path to experiment 1 directory")
 parser.add_argument("Path to experiment 2 directory")
 parser.add_argument("Path to base directory of cartopy shapefiles")
+parser.add_argument("Name of native domain used in forecasts (and in constructing post file names)")
 args = parser.parse_args()
 
 # Read date/time, forecast hour, and directory paths from command line
@@ -257,6 +263,7 @@ def rotate_wind(true_lat,lov_lon,earth_lons,uin,vin,proj,inverse=False):
 EXPT_DIR_1 = str(sys.argv[5])
 EXPT_DIR_2 = str(sys.argv[6])
 CARTOPY_DIR = str(sys.argv[7])
+POST_OUTPUT_DOMAIN_NAME = str(sys.argv[8]).lower()
 
 # Loop over forecast hours
 for fhr in fhours:
@@ -267,8 +274,8 @@ def rotate_wind(true_lat,lov_lon,earth_lons,uin,vin,proj,inverse=False):
 
 
 # Define the location of the input files
-  data1 = pygrib.open(EXPT_DIR_1+'/'+ymdh+'/postprd/rrfs.t'+cyc+'z.prslevf'+fhour+'.tm00.grib2')
-  data2 = pygrib.open(EXPT_DIR_2+'/'+ymdh+'/postprd/rrfs.t'+cyc+'z.prslevf'+fhour+'.tm00.grib2')
+  data1 = pygrib.open(EXPT_DIR_1+'/'+ymdh+'/postprd/rrfs.t'+cyc+'z.prslev.f'+fhour+'.'+POST_OUTPUT_DOMAIN_NAME+'.grib2')
+  data2 = pygrib.open(EXPT_DIR_2+'/'+ymdh+'/postprd/rrfs.t'+cyc+'z.prslev.f'+fhour+'.'+POST_OUTPUT_DOMAIN_NAME+'.grib2')
 
 # Get the lats and lons
   grids = [data1, data2]

ush/Python/qsub_job.sh

@@ -35,7 +35,7 @@ export FCST_END=${FCST_LEN_HRS}
 export FCST_INC=3
 
 # Usage statement:      Make sure all the necessary modules can be imported.
-#                       Five command line arguments are needed:
+#                       The following command line arguments are needed:
 #                       1. Cycle date/time in YYYYMMDDHH format
 #                       2. Starting forecast hour in HHH format
 #                       3. Ending forecast hour in HHH format
@@ -46,5 +46,10 @@ export FCST_INC=3
 #                       6. CARTOPY_DIR:  Base directory of cartopy shapefiles
 #                          -File structure should be:
 #                            CARTOPY_DIR/shapefiles/natural_earth/cultural/*.shp
+#                       7. POST_OUTPUT_DOMAIN_NAME:  Name of native domain
+#                          used in forecast and in constructing the names
+#                          of the post output files.
 
-python plot_allvars.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} ${EXPTDIR} ${SHAPE_FILES}
+
+python plot_allvars.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} \
+                       ${EXPTDIR} ${SHAPE_FILES} ${POST_OUTPUT_DOMAIN_NAME}

ush/Python/qsub_job_diff.sh

@@ -28,15 +28,15 @@ ncar_pylib /glade/p/ral/jntp/UFS_SRW_app/ncar_pylib/python_graphics
 #Cheyenne:
 SHAPE_FILES=/glade/p/ral/jntp/UFS_SRW_app/tools/NaturalEarth
 
-export GLOBAL_VAR_DEFNS_FP="${EXPTDIR}/var_defns.sh"
+export GLOBAL_VAR_DEFNS_FP="${EXPTDIR1}/var_defns.sh"
 source ${GLOBAL_VAR_DEFNS_FP}
 export CDATE=${DATE_FIRST_CYCL}${CYCL_HRS}
 export FCST_START=3
 export FCST_END=${FCST_LEN_HRS}
 export FCST_INC=3
 
 # Usage statement:      Make sure all the necessary modules can be imported.
-#                       Seven command line arguments are needed:
+#                       The following command line arguments are needed:
 #                       1. Cycle date/time in YYYYMMDDHH format
 #                       2. Starting forecast hour in HHH format
 #                       3. Ending forecast hour in HHH format
@@ -50,5 +50,12 @@ export FCST_INC=3
 #                       7. CARTOPY_DIR:  Base directory of cartopy shapefiles
 #                          -File structure should be:
 #                            CARTOPY_DIR/shapefiles/natural_earth/cultural/*.shp
-
-python plot_allvars_diff.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} ${EXPTDIR1} ${EXPTDIR2} ${SHAPE_FILES}
+#                       8. POST_OUTPUT_DOMAIN_NAME:  Name of native domain
+#                          used in forecasts and in constructing the names 
+#                          of the post output files.  This must be the same 
+#                          for both forecasts.
+
+python plot_allvars_diff.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} \
+                            ${EXPTDIR1} ${EXPTDIR2} \
+                            ${SHAPE_FILES} \
+                            ${POST_OUTPUT_DOMAIN_NAME}

ush/Python/sq_job.sh

@@ -57,8 +57,8 @@ export FCST_START=6
 export FCST_END=${FCST_LEN_HRS}
 export FCST_INC=6
 
-# Usage statement:	Make sure all the necessary modules can be imported.
-#                       Six command line arguments are needed:
+# Usage statement:      Make sure all the necessary modules can be imported.
+#                       The following command line arguments are needed:
 #                       1. Cycle date/time in YYYYMMDDHH format
 #                       2. Starting forecast hour in HHH format
 #                       3. Ending forecast hour in HHH format
@@ -69,5 +69,10 @@ export FCST_INC=6
 #                       6. CARTOPY_DIR:  Base directory of cartopy shapefiles
 #                          -File structure should be:
 #                            CARTOPY_DIR/shapefiles/natural_earth/cultural/*.shp
+#                       7. POST_OUTPUT_DOMAIN_NAME:  Name of native domain
+#                          used in forecast and in constructing the names
+#                          of the post output files.
 
-python plot_allvars.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} ${EXPTDIR} ${SHAPE_FILES}
+
+python plot_allvars.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} \
+                       ${EXPTDIR} ${SHAPE_FILES} ${POST_OUTPUT_DOMAIN_NAME}

ush/Python/sq_job_diff.sh

@@ -52,15 +52,15 @@ SHAPE_FILES=/scratch2/BMC/det/UFS_SRW_app/v1p0/fix_files/NaturalEarth
 #Gaea: 
 #SHAPE_FILES=/lustre/f2/pdata/esrl/gsd/ufs/NaturalEarth
 
-export GLOBAL_VAR_DEFNS_FP="${EXPTDIR}/var_defns.sh"
+export GLOBAL_VAR_DEFNS_FP="${EXPTDIR1}/var_defns.sh"
 source ${GLOBAL_VAR_DEFNS_FP}
 export CDATE=${DATE_FIRST_CYCL}${CYCL_HRS}
 export FCST_START=6
 export FCST_END=${FCST_LEN_HRS}
 export FCST_INC=3
 
-# Usage statement:	Make sure all the necessary modules can be imported.
-#                       Seven command line arguments are needed:
+# Usage statement:      Make sure all the necessary modules can be imported.
+#                       The following command line arguments are needed:
 #                       1. Cycle date/time in YYYYMMDDHH format
 #                       2. Starting forecast hour
 #                       3. Ending forecast hour
@@ -74,5 +74,12 @@ export FCST_INC=3
 #                       7. CARTOPY_DIR:  Base directory of cartopy shapefiles
 #                          -File structure should be:
 #                            CARTOPY_DIR/shapefiles/natural_earth/cultural/*.shp
+#                       8. POST_OUTPUT_DOMAIN_NAME:  Name of native domain
+#                          used in forecasts and in constructing the names 
+#                          of the post output files.  This must be the same 
+#                          for both forecasts.
 
-python plot_allvars_diff.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} ${EXPTDIR1} ${EXPTDIR2} ${SHAPE_FILES}
+python plot_allvars_diff.py ${CDATE} ${FCST_START} ${FCST_END} ${FCST_INC} \
+                            ${EXPTDIR1} ${EXPTDIR2} \
+                            ${SHAPE_FILES} \
+                            ${POST_OUTPUT_DOMAIN_NAME}

ush/config_defaults.sh

@@ -419,18 +419,6 @@ WFLOW_LAUNCH_LOG_FN="log.launch_FV3LAM_wflow"
 #
 #-----------------------------------------------------------------------
 #
-# Set output file name. Definitions:
-#
-# POST_OUTPUT_DOMAIN_NAME:
-# Domain name used in naming the output files of run_post by UPP or inline post.
-# Output file name: $NET.tHHz.[var_name].f###.$POST_OUTPUT_DOMAIN_NAME.grib2
-#
-#-----------------------------------------------------------------------
-#
-POST_OUTPUT_DOMAIN_NAME=""
-#
-#-----------------------------------------------------------------------
-#
 # Set forecast parameters.  Definitions:
 #
 # DATE_FIRST_CYCL:
@@ -484,8 +472,11 @@ FCST_LEN_HRS="24"
 # with the prefix "YYYYMMDD.HHmmSS." in the RESTART directory
 #
 # WRITE_DOPOST:
-# Flag that determines whether or not to use the INLINE POST option
-# When TRUE, force to turn off run_post (RUN_TASK_RUN_POST=FALSE) in setup.sh
+# Flag that determines whether or not to use the inline post feature 
+# [i.e. calling the Unified Post Processor (UPP) from within the weather 
+# model].  If this is set to "TRUE", the RUN_POST_TN task is deactivated 
+# (i.e. RUN_TASK_RUN_POST is set to "FALSE") to avoid unnecessary 
+# computations.
 #
 #-----------------------------------------------------------------------
 #
@@ -1757,8 +1748,7 @@ DT_SUBHOURLY_POST_MNTS="00"
 #
 #-----------------------------------------------------------------------
 #
-# Set parameters associated with defining a customized post configuration 
-# file.
+# Set parameters for customizing the post-processor (UPP).  Definitions:
 #
 # USE_CUSTOM_POST_CONFIG_FILE:
 # Flag that determines whether a user-provided custom configuration file
@@ -1773,10 +1763,24 @@ DT_SUBHOURLY_POST_MNTS="00"
 # used for post-processing. This is only used if CUSTOM_POST_CONFIG_FILE
 # is set to "TRUE".
 #
+# POST_OUTPUT_DOMAIN_NAME:
+# Domain name (in lowercase) used in constructing the names of the output 
+# files generated by UPP [which is called either by running the RUN_POST_TN 
+# task or by activating the inline post feature (WRITE_DOPOST set to "TRUE")].  
+# The post output files are named as follows:
+#
+#   $NET.tHHz.[var_name].f###.${POST_OUTPUT_DOMAIN_NAME}.grib2
+#
+# If using a custom grid, POST_OUTPUT_DOMAIN_NAME must be specified by 
+# the user.  If using a predefined grid, POST_OUTPUT_DOMAIN_NAME defaults
+# to PREDEF_GRID_NAME.  Note that this variable is first changed to lower
+# case before being used to construct the file names.
+#
 #-----------------------------------------------------------------------
 #
 USE_CUSTOM_POST_CONFIG_FILE="FALSE"
 CUSTOM_POST_CONFIG_FP=""
+POST_OUTPUT_DOMAIN_NAME=""
 #
 #-----------------------------------------------------------------------
 #

ush/setup.sh

@@ -1240,9 +1240,6 @@ check_for_preexist_dir_file "$EXPTDIR" "${PREEXISTING_DIR_METHOD}"
 #
 # COMOUT_BASEDIR is not used by the workflow in community mode.
 #
-# POST_OUTPUT_DOMAIN_NAME:
-# The PREDEF_GRID_NAME is set by default.
-#
 #-----------------------------------------------------------------------
 #
 LOGDIR="${EXPTDIR}/log"
@@ -1262,8 +1259,20 @@ else
   COMROOT=""
   COMOUT_BASEDIR=""
 fi
-
+#
+#-----------------------------------------------------------------------
+#
+#
+# If POST_OUTPUT_DOMAIN_NAME has not been specified by the user, set it
+# to PREDEF_GRID_NAME (which won't be empty if using a predefined grid).
+# Then change it to lowercase.  Finally, ensure that it does not end up 
+# getting set to an empty string.
+#
+#-----------------------------------------------------------------------
+#
 POST_OUTPUT_DOMAIN_NAME="${POST_OUTPUT_DOMAIN_NAME:-${PREDEF_GRID_NAME}}"
+POST_OUTPUT_DOMAIN_NAME=$(echo_lowercase ${POST_OUTPUT_DOMAIN_NAME})
+
 if [ -z "${POST_OUTPUT_DOMAIN_NAME}" ]; then
   print_err_msg_exit "\
 The domain name used in naming the run_post output files (POST_OUTPUT_DOMAIN_NAME)
@@ -1273,7 +1282,6 @@ If this experiment is not using a predefined grid (i.e. if PREDEF_GRID_NAME
 is set to a null string), POST_OUTPUT_DOMAIN_NAME must be set in the SRW 
 App's configuration file (\"${EXPT_CONFIG_FN}\")."
 fi
-POST_OUTPUT_DOMAIN_NAME=$(echo_lowercase ${POST_OUTPUT_DOMAIN_NAME})
 #
 #-----------------------------------------------------------------------
 #