[develop]: Add plotting and template variable documentation

docs/UsersGuide/requirements.txt

@@ -1,2 +1,2 @@
 sphinxcontrib-bibtex
-sphinx_rtd_theme
+sphinx_rtd_theme

docs/UsersGuide/source/ConfigWorkflow.rst

@@ -490,6 +490,12 @@ Verification Tasks
 
 .. COMMENT: COMMENT: Define "ensemble-stat verification for gridded data," "ensemble point verification," "ensemble-stat point verification," and "point verification of ensemble-stat output"?
 
+Plotting Task
+----------------
+
+``RUN_TASK_PLOT_ALLVARS:`` (Default: false)
+   Flag that determines whether to run python plotting scripts.
+
 .. _make-grid:
 
 MAKE_GRID Configuration Parameters
@@ -1134,6 +1140,9 @@ These parameters are associated with the fixed (i.e., static) files. On `Level 1
 ``FIXlut``: (Default: "")
    System directory where the lookup tables for optics properties are located.
 
+``FIXshp``: (Default: "")
+   System directory where the graphics shapefiles are located. On Level 1 systems, these are set within the machine files. Users on other systems will need to provide the path to the directory that contains the *Natural Earth* shapfiles.
+
 ``TOPO_DIR``: (Default: "")
    The location on disk of the static input files used by the ``make_orog`` task (i.e., ``orog.x`` and ``shave.x``). Can be the same as ``FIXgsm``.
 
@@ -1726,6 +1735,49 @@ Non-default parameters for the ``run_enspointvx_prob`` task are set in the ``tas
 ``MAXTRIES_VX_ENSPOINT_PROB``: (Default: 1)
    Maximum number of times to attempt the task.
 
+.. _PlotVars:
+
+PLOT_ALLVARS Configuration Parameters
+========================================
+
+Non-default parameters for the ``plot_allvars`` task are set in the ``task_plot_allvars:`` section of the ``config.yaml`` file. 
+
+Basic Task Parameters
+--------------------------
+
+For each workflow task, certain parameter values must be passed to the job scheduler (e.g., Slurm), which submits a job for the task. Typically, users do not need to adjust the default values. 
+
+``PLOT_ALLVARS_TN``: (Default: "plot_allvars")
+   Set the name of this Rocoto workflow task. Users typically do not need to change this value.
+
+``NNODES_PLOT_ALLVARS``: (Default: 1)
+   Number of nodes to use for the job.
+
+``PPN_PLOT_ALLVARS``: (Default: 24)
+   Number of :term:`MPI` processes per node.
+
+``WTIME_PLOT_ALLVARS``: (Default: 01:00:00)
+   Maximum time for the task to complete.
+
+``MAXTRIES_PLOT_ALLVARS``: (Default: 1)
+   Maximum number of times to attempt the task.
+
+Additional Parameters
+------------------------
+
+Typially, the following parameters must be set explicitly by the user in the configuration file (``config.yaml``) when executing the plotting tasks. 
+
+``COMOUT_REF``: (Default: "")
+   The directory where the GRIB2 files from post-processing are located. In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will correspond to the location in the experiment directory where the post-processed output can be found (e.g., ``$EXPTDIR/$DATE_FIRST_CYCL/postprd``). In *nco* mode, this directory should be set to the location of the COMOUT directory and end with ``$PDY/$cyc``.
+
+``PLOT_FCST_START``: (Default: 0)
+   The starting forecast hour for the plotting task. For example, if a forecast starts at 18h/18z, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. If a forecast starts at 18h/18z, but the user only wants plots from the 6th forecast hour on, "starting forecast hour" should be 6.
+
+``PLOT_FCST_INC``: (Default: 3)
+   Forecast hour increment for the plotting task. This may be the same as ``INCR_CYCL_FREQ``, or it may be a multiple of ``INCR_CYCL_FREQ``. For example, if ``INCR_CYCL_FREQ`` is set to 3, there will be forecast output every three hours for the duration of the forecast. If the user wants plots of all of this output, they should set ``PLOT_FCST_INC: 3``. If the user only wants plots for some of the output (e.g., every 6 hours), they should set ``PLOT_FCST_INC: 6``. However, there must be forecast output available at the designated increments to produce the plots. In this example, setting ``PLOT_FCST_INC: 7`` would produce an error because there is only forecast output available for hours 3, 6, 9, ..., etc. 
+
+``PLOT_FCST_END``: (Default: "")
+   The last forecast hour for the plotting task. For example, if a forecast run for 24 hours, and the user wants plots for each available hour of forecast output, they should set ``PLOT_FCST_END: 24``. If the user only wants plots from the first 12 hours of the forecast, the "last forecast hour" should be 12.
 
 Global Configuration Parameters
 ===================================

docs/UsersGuide/source/RunSRW.rst

@@ -534,6 +534,80 @@ Valid values for configuration variables should be consistent with those in the
 
 To configure an experiment and python environment for a general Linux or Mac system, see the :ref:`next section <LinuxMacEnvConfig>`. To configure an experiment to run METplus verification tasks, see :numref:`Section %s <VXConfig>`. Otherwise, skip to :numref:`Section %s <GenerateWorkflow>` to generate the workflow.
 
+.. _PlotOutput:
+
+Plot the Output
+-----------------
+
+An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2`
+output over the :term:`CONUS`. It generates graphics plots for a number of variables, including:
+
+* 2-m temperature
+* 2-m dew point temperature
+* 10-m winds
+* 250 hPa winds
+* Accumulated precipitation
+* Composite reflectivity
+* Surface-based :term:`CAPE`/:term:`CIN`
+* Max/Min 2-5 km updraft helicity
+* Sea level pressure (SLP)
+
+.. * 500 hPa heights, winds, and vorticity --> seems to be omitted?
+
+This workflow task can produce both plots from a single experiment and difference plots that compare the same cycle from two different experiments. When plotting the difference, the two experiments must be on the same domain and available for 
+the same cycle starting date/time and forecast hours. Other parameters may differ (e.g., the experiments may use different physics suites).
+
+.. _Cartopy:
+
+Cartopy Shapefiles
+^^^^^^^^^^^^^^^^^^^^^
+
+The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 <https://github.com/ufs-community/ufs-srweather-app/wiki/Supported-Platforms-and-Compilers>`__ systems, this path is already set in the machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$HOME/ush/machine/macos.yaml`` for a generic MacOS system, where ``$HOME`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket <https://noaa-ufs-srw-pds.s3.amazonaws.com/NaturalEarth/NaturalEarth.tgz>`__. The full set of Cartopy shapefiles can be downloaded `here <https://www.naturalearthdata.com/downloads/>`__. 
+
+Task Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true:
+
+.. code-block:: console
+
+   workflow_switches:
+      RUN_TASK_PLOT_ALLVARS: true
+
+Plotting Output From a Single Experiment
+````````````````````````````````````````````
+
+Users may also wish to adjust the start, end, and increment value for the plotting task. For example:  
+
+.. code-block:: console
+
+   task_plot_allvars:
+      PLOT_FCST_START: 0
+      PLOT_FCST_INC: 6
+      PLOT_FCST_END: 12
+
+If the user chooses not to set these values, the default values will be used (see :numref:`Section %s <PlotVars>`).
+
+.. note::
+   If a forecast starts at 18h, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. 
+
+The output files (in ``.png`` format) will be located in the experiment directory under the ``$CDATE/postprd`` subdirectory where ``$CDATE`` 
+corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``).
+
+Plotting the Difference Between Two Experiments
+```````````````````````````````````````````````````
+
+When plotting the difference between two experiments, users must set the baseline experiment directory using the ``COMOUT_REF`` template variable. For example, in *community* mode, users can set:
+
+.. code-block:: console
+
+   task_plot_allvars:
+      COMOUT_REF: '${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd'
+
+In *community* mode, using default directory names and settings, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables are provided in :numref:`Section %s <PlotVars>`. 
+
+The output files (in ``.png`` format) will be located in the ``postprd`` directory for the experiment.
+
 .. _LinuxMacEnvConfig:
 
 User-Specific Configuration on a Generic Linux/MacOS System
@@ -1260,11 +1334,3 @@ Users can access log files for specific tasks in the ``$EXPTDIR/log`` directory.
 
 .. note::
    On most HPC systems, users will need to submit a batch job to run multi-processor jobs. On some HPC systems, users may be able to run the first two jobs (serial) on a login node/command-line. Example scripts for Slurm (Hera) and PBS (Cheyenne) resource managers are provided (``sq_job.sh`` and ``qsub_job.sh``, respectively). These examples will need to be adapted to each user's system. Alternatively, some batch systems allow users to specify most of the settings on the command line (with the ``sbatch`` or ``qsub`` command, for example). 
-
-
-
-.. _PlotOutput:
-
-Plot the Output
-===============
-Two python scripts are provided to generate plots from the :term:`FV3`-LAM post-processed :term:`GRIB2` output. Information on how to generate the graphics can be found in :numref:`Chapter %s <Graphics>`.

docs/UsersGuide/source/TemplateVars.rst

@@ -0,0 +1,72 @@
+.. _TemplateVars:
+
+===============================================================
+Using Template Variables in the Experiment Configuration Files
+===============================================================
+
+The SRW App's experiment configuration system supports the use of template variables
+in ``config_defaults.yaml`` and ``config.yaml``. A template variable --- or "template" --- is an experiment configuration variable that contains references to values of other variables. 
+These references are **not** set to the values of the referenced variables (or "expanded") when the experiment's variable definitions file (``var_defns.sh``) is generated or sourced.
+Instead, they are expanded and evaluated **at run time** when bash's
+``eval`` command is used on the template. 
+
+Generic Example
+==================
+
+As an example, consider a hypothetical template variable named ``MY_CMD`` that is defined in ``config_defaults.yaml``
+(or redefined by the user in ``config.yaml``) as follows:
+
+   .. code-block:: console
+
+      MY_CMD: 'cd ${some_dir}'
+
+Here, ``some_dir`` may be another experiment variable defined in ``var_defns.sh`` or a
+local variable defined in a script or function that will evaluate the template. 
+It is important to use single quotes on the right-hand side of the definition above;
+otherwise, bash will try to evaluate ``${some_dir}`` when constructing ``var_defns.sh``,
+which may result in an error and/or unexpected behavior (e.g., if ``${some_dir}`` 
+is not yet defined). The experiment generation system will define ``MY_CMD`` in 
+``var_defns.sh`` in exactly the same way as in ``config_defaults.yaml`` and/or 
+``config.yaml``, e.g., ``MY_CMD: 'cd ${some_dir}'``. Then the following code snippet 
+in a script or function will evaluate the contents of ``MY_CMD`` using a locally-set 
+value of ``some_dir``:
+
+   .. code-block:: none
+
+      ...
+      . var_defns.sh       # Source the experiment's variable definition file (assuming
+                           # it is in the current directory). This defines the MY_CMD
+                           # template variable (in addition to other variables).
+      ...
+      some_dir="20200715"  # Set the local variable some_dir.
+      ...
+      eval ${MY_CMD}       # Use eval to evaluate the contents of MY_CMD. The value of
+                           # some_dir specified in this file a few lines above is substituted
+                           # for ${some_dir} in MY_CMD before MY_CMD is evaluated.
+
+Graphics Plotting Example
+============================
+
+When attempting to generate graphics plots from a forecast, users have the option to 
+produce difference plots from two experiments that are on the same domain and 
+available for the same cycle starting date/time and forecast hours. 
+To generate difference plots, users must use the template variable ``COMOUT_REF`` 
+to indicate where the :term:`GRIB2` files from post-processing are located. 
+
+In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will 
+take the form ``/path/to/exptdir/$PDY$cyc/postprd``, where ``$PDY`` refers to the 
+cycle date in YYYYMMDD format, and ``$cyc`` refers to the starting hour of the cycle. 
+(These variables are set in previous tasks based on the value of ``DATE_FIRST_CYCL``.)
+Concretely, users can set ``COMOUT_REF`` as follows:
+
+.. code-block:: console
+
+   COMOUT_REF: '${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd'
+
+In *nco* mode, this directory should be set to the location of the ``COMOUT`` directory 
+(``${COMOUT}`` in the example below) and end with ``${PDY}/${cyc}``. For example:
+
+.. code-block:: console
+
+   COMOUT_REF: '${COMOUT}/${PDY}/${cyc}/'
+

docs/UsersGuide/source/index.rst

@@ -22,7 +22,7 @@ UFS Short-Range Weather App Users Guide
    ConfigWorkflow
    RocotoInfo
    WE2Etests
+   TemplateVars
    VXCases
-   Graphics
    FAQ
    Glossary