Merge branch 'develop' of https://github.com/dtcenter/METplus into de…

…velop

.github/actions/run_tests/entrypoint.sh

@@ -56,7 +56,7 @@ if [[ "$INPUT_CATEGORIES" == pytests* ]]; then
 	 .
 
   echo Running Pytests
-  command="export METPLUS_PYTEST_HOST=docker; cd internal_tests/pytests;"
+  command="export METPLUS_PYTEST_HOST=docker; cd internal/tests/pytests;"
   command+="status=0;"
   for x in `cat $PYTESTS_GROUPS_FILEPATH`; do
     marker="${x//_or_/ or }"

.github/jobs/get_requirements.py

@@ -14,7 +14,7 @@
                                                os.pardir))
 sys.path.insert(0, METPLUS_TOP_DIR)
 
-from internal_tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
+from internal.tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
 
 def main():
     all_requirements = set()

.github/jobs/get_use_case_commands.py

@@ -13,7 +13,7 @@
                                                os.pardir))
 sys.path.insert(0, METPLUS_TOP_DIR)
 
-from internal_tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
+from internal.tests.use_cases.metplus_use_case_suite import METplusUseCaseSuite
 from metplus.util.met_util import expand_int_string_to_list
 from docker_utils import VERSION_EXT
 

docs/Contributors_Guide/add_use_case.rst

@@ -680,7 +680,7 @@ will be used in the final pull request.
 Add use case to the test suite
 ------------------------------
 
-The *internal_tests/use_cases/all_use_cases.txt* file in the METplus
+The *internal/tests/use_cases/all_use_cases.txt* file in the METplus
 repository contains the list of all use cases.
 Add the new use case to this file so it will be available in
 the tests. See the :ref:`cg-ci-all-use-cases` section for details.
@@ -848,7 +848,7 @@ steps were unsuccessful in lowering memory usage, please take the following step
   Utilize a Python memory profiler to identify as specifically as possible 
   where the script exceeds the memory limit.
 - Add the use case to the :ref:`memory-intense-use-cases` list.
-- In the *internal_tests/use_cases/all_use_cases.txt* file, ensure that the 
+- In the *internal/tests/use_cases/all_use_cases.txt* file, ensure that the 
   use case is listed as the lowest-listed use case in its respective category. 
   Change the number in front of the new use case to an 'X', preceded 
   by the ‘#’ character::

docs/Contributors_Guide/continuous_integration.rst

@@ -601,7 +601,7 @@ All Use Cases
 ^^^^^^^^^^^^^
 
 All of the existing use cases are listed in **all_use_cases.txt**,
-found in *internal_tests/use_cases*.
+found in *internal/tests/use_cases*.
 
 The file is organized by use case category. Each category starts
 a line that following the format::

docs/Contributors_Guide/testing.rst

@@ -1,7 +1,7 @@
 Testing
 =======
 
-Test scripts are found in the GitHub repository in the internal_tests
+Test scripts are found in the GitHub repository in the internal/tests
 directory.
 
 .. _cg-unit-tests:
@@ -13,15 +13,15 @@ Unit tests are run with pytest. They are found in the *pytests* directory.
 Each tool has its own subdirectory containing its test files.
 
 Unit tests can be run by running the 'pytest' command from the
-internal_tests/pytests directory of the repository.
+internal/tests/pytests directory of the repository.
 The 'pytest' Python package must be available.
 A report will be output showing which pytest categories failed.
 When running on a new computer, a **minimum_pytest.<HOST>.sh**
 file must be created to be able to run the script. This file contains
 information about the local environment so that the tests can run.
 
 All unit tests must include one of the custom markers listed in the
-internal_tests/pytests/pytest.ini file. Some examples include:
+internal/tests/pytests/pytest.ini file. Some examples include:
 
     * util
     * wrapper_a

docs/Release_Guide/release_steps/metplus/update_web_server_data.rst

@@ -7,7 +7,7 @@ Create Directory for Next Release
 On the DTC web server where the sample input data for use cases is hosted,
 run the setup_next_release_data.py script for the next upcoming release
 to set up the data directory for the next major/minor version development.
-The script can be found in the METplus repository in internal_tests/use_cases.
+The script can be found in the METplus repository in internal/tests/use_cases.
 The file should be found in the home directory of the met_test user on
 the DTC web server host. It is linked to the file in the METplus repository.
 Pull the latest changes from the develop branch before running the script::

docs/Users_Guide/glossary.rst

@@ -8672,8 +8672,8 @@ METplus Configuration Glossary
 
      | *Used by:*  GenEnsProd
 
-   GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDF
-     Specify the value for 'ensemble_flag.climo_cdf' in the MET configuration file for GenEnsProd.
+   GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDP
+     Specify the value for 'ensemble_flag.climo_cdp' in the MET configuration file for GenEnsProd.
 
      | *Used by:*  GenEnsProd
 

docs/Users_Guide/installation.rst

@@ -227,7 +227,7 @@ The METplus Wrappers source code contains the following directory structure::
     build_components/
     docs/
     environment.yml
-    internal_tests/
+    internal/
     manage_exernals/
     metplus/
     parm/
@@ -253,7 +253,7 @@ developer has Doxygen installed on the host.
 The Doxygen documentation is useful to
 contributors and is not necessary for METplus end-users.
 
-The **internal_tests/** directory contains test scripts that are only
+The **internal/** directory contains scripts that are only
 relevant to METplus developers and contributors.
 
 The **manage_externals/** directory contains scripts used to

docs/Users_Guide/wrappers.rst

@@ -1107,7 +1107,7 @@ METplus Configuration
 | :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_NEP`
 | :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_NMEP`
 | :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO`
-| :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDF`
+| :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDP`
 | :term:`GEN_ENS_PROD_ENS_MEMBER_IDS`
 | :term:`GEN_ENS_PROD_CONTROL_ID`
 | :term:`GEN_ENS_PROD_MET_CONFIG_OVERRIDES`
@@ -1408,8 +1408,8 @@ see :ref:`How METplus controls MET config file settings<metplus-control-met>`.
      - ensemble_flag.nmep
    * - :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO`
      - ensemble_flag.climo
-   * - :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDF`
-     - ensemble_flag.climo_cdf
+   * - :term:`GEN_ENS_PROD_ENSEMBLE_FLAG_CLIMO_CDP`
+     - ensemble_flag.climo_cdp
 
 **${METPLUS_ENS_MEMBER_IDS}**
 

docs/use_cases/model_applications/precipitation/EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.py → docs/use_cases/model_applications/precipitation/GenEnsProd_fcstHRRRE_FcstOnly_NetCDF.py

@@ -1,8 +1,8 @@
 """
-Ensemble-Stat: Basic Post-Processing only  
-================================================================
+Gen-Ens-Prod: Basic Post-Processing only
+========================================
 
-model_application/precipitation/EnsembleStat_fcstHRRRE
+model_application/precipitation/GenEnsProd_fcstHRRRE
 _FcstOnly_NetCDF.conf
 
 """
@@ -25,15 +25,15 @@
 # METplus Components
 # ------------------
 #
-# This use case runs Ensemble-Stat on HRRRE data from 3 members after
+# This use case runs gen_ens_prod on HRRRE data from 3 members after
 # running it through pcp_combine to create a 3 hour precipitation accumulation
 
 ###############################################################################
 # METplus Workflow
 # ----------------
 #
 # The following tools are used for each run time:
-# EnsembleStat
+# GenEnsProd
 #
 # This example loops by initialization time. For each initialization time
 # it will process forecast leads 3, 6, 9 and 12
@@ -70,12 +70,11 @@
 # METplus Configuration
 # ---------------------
 #
-# METplus first loads all of the configuration files found in parm/metplus_config,
+# METplus first loads all of the configurations found in parm/metplus_config,
 # then it loads any configuration files passed to METplus via the command line
-# with the -c option, i.e. -c parm/use_cases/model_applications/precipitation/EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.conf
 #
 # .. highlight:: bash
-# .. literalinclude:: ../../../../parm/use_cases/model_applications/precipitation/EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.conf
+# .. literalinclude:: ../../../../parm/use_cases/model_applications/precipitation/GenEnsProd_fcstHRRRE_FcstOnly_NetCDF.conf
 
 ##############################################################################
 # MET Configuration
@@ -89,39 +88,19 @@
 # If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to:
 # :ref:`Overriding Unsupported MET config file settings<met-config-overrides>`
 #
-# .. note:: See the :ref:`EnsembleStat MET Configuration<ens-stat-met-conf>` section of the User's Guide for more information on the environment variables used in the file below:
+# .. note:: See the :ref:`GenEnsProd MET Configuration<gen-ens-prod-met-conf>` section of the User's Guide for more information on the environment variables used in the file below:
 #
 # .. highlight:: bash
-# .. literalinclude:: ../../../../parm/met_config/EnsembleStatConfig_wrapped
+# .. literalinclude:: ../../../../parm/met_config/GenEnsProdConfig_wrapped
 
 ##############################################################################
 # Running METplus
 # ---------------
 #
-# This use case can be run two ways:
+# The command to run this use case is::
 #
-# 1) Passing in EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.conf then a user-specific system configuration file::
+#    run_metplus.py /path/to/METplus/parm/use_cases/model_applications/precipitation/GenEnsProd_fcstHRRRE_FcstOnly_NetCDF.conf
 #
-#        run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/precipitation/EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.conf -c /path/to/user_system.conf
-#
-# 2) Modifying the configurations in parm/metplus_config, then passing in EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.conf::
-#
-#        run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/precipitation/EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.conf
-#
-# The former method is recommended. Whether you add them to a user-specific configuration file or modify the metplus_config files, the following variables must be set correctly:
-#
-# * **INPUT_BASE** - Path to directory where sample data tarballs are unpacked (See Datasets section to obtain tarballs). This is not required to run METplus, but it is required to run the examples in parm/use_cases
-# * **OUTPUT_BASE** - Path where METplus output will be written. This must be in a location where you have write permissions
-# * **MET_INSTALL_DIR** - Path to location where MET is installed locally
-#
-# Example User Configuration File::
-#
-#   [dir]
-#   INPUT_BASE = /path/to/sample/input/data
-#   OUTPUT_BASE = /path/to/output/dir
-#   MET_INSTALL_DIR = /path/to/met-X.Y 
-#
-# **NOTE:** All of these items must be found under the [dir] section.
 #
 
 ##############################################################################
@@ -133,22 +112,22 @@
 #   INFO: METplus has successfully finished running.
 #
 # Refer to the value set for **OUTPUT_BASE** to find where the output data was generated.
-# Output for this use case will be found in model_applications/precipitation/EnsembleStat_fcstHRRRE_FcstOnly_NetCDF/EnsembleStat (relative to **OUTPUT_BASE**)
+# Output for this use case will be found in model_applications/precipitation/GenEnsProd_fcstHRRRE_FcstOnly_NetCDF/GenEnsProd (relative to **OUTPUT_BASE**)
 # The following folder/file combination will be created:
 #
 # -201905191200
 #
-# * ensemble_stat_APCP_03_20190519_150000V_ens.nc
-# * ensemble_stat_APCP_03_20190519_180000V_ens.nc
-# * ensemble_stat_APCP_03_20190519_210000V_ens.nc
-# * ensemble_stat_APCP_03_20190520_000000V_ens.nc
+# * gen_ens_prod_APCP_03_20190519_150000V_ens.nc
+# * gen_ens_prod_APCP_03_20190519_180000V_ens.nc
+# * gen_ens_prod_APCP_03_20190519_210000V_ens.nc
+# * gen_ens_prod_APCP_03_20190520_000000V_ens.nc
 #
 # -201905200000
 #
-# * ensemble_stat_APCP_03_20190520_030000V_ens.nc
-# * ensemble_stat_APCP_03_20190520_060000V_ens.nc
-# * ensemble_stat_APCP_03_20190520_090000V_ens.nc
-# * ensemble_stat_APCP_03_20190520_120000V_ens.nc
+# * gen_ens_prod_APCP_03_20190520_030000V_ens.nc
+# * gen_ens_prod_APCP_03_20190520_060000V_ens.nc
+# * gen_ens_prod_APCP_03_20190520_090000V_ens.nc
+# * gen_ens_prod_APCP_03_20190520_120000V_ens.nc
 
 
 ##############################################################################
@@ -157,7 +136,7 @@
 #
 # .. note::
 #
-#   * EnsembleStatToolUseCase
+#   * GenEnsProdToolUseCase
 #   * NOAAHWTOrgUseCase
 #   * PrecipitationAppUseCase
 #   * NetCDFFileUseCase
@@ -170,5 +149,5 @@
 #
 #
 #
-# sphinx_gallery_thumbnail_path = '_static/precipitation-EnsembleStat_fcstHRRRE_FcstOnly_NetCDF.png'
+# sphinx_gallery_thumbnail_path = '_static/precipitation-GenEnsProd_fcstHRRRE_FcstOnly_NetCDF.png'
 #

docs/use_cases/model_applications/short_range/EnsembleStat_fcstHRRR_fcstOnly_SurrogateSevere.py → docs/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.py

@@ -1,18 +1,18 @@
 """
-Surrogate Severe Calculation: PCPCombine, EnsembleStat, and RegridDataPlane 
-===================================================================================================
+Surrogate Severe Calculation: PCPCombine, GenEnsProd, and RegridDataPlane
+=========================================================================
 
 model_applications/
 short_range/
-EnsembleStat_fcstHRRR_fcstOnly
+GenEnsProd_fcstHRRR_fcstOnly
 _SurrogateSevere.conf
 
 """
 ###################################################################################################
 # Scientific Objective
 # --------------------
 #
-# Run PCPCombine, EnsembleStat, and RegridDataPlane tools to create surrogate severe probability 
+# Run PCPCombine, GenEnsProd, and RegridDataPlane tools to create surrogate severe probability
 # forecasts (SSPFs) for a given date. SSPFs are a severe weather forecasting tool and is a techniqu
 # used by the Storm Prediction Center (SPC) as well as others. SSPFs are based on updraft helicity 
 # (UH; UH = ∫z0 to zt (ω * ζ) dz) since certain thresholds of UH have been shown as good proxies for# severe weather. SSPFs can be thought of as the perfect model forecast. They are derived as follows:
@@ -44,25 +44,25 @@
 # METplus Components
 # ------------------
 #
-# This use case runs the PCPCombine, EnsembleStat, and RegridDataPlane MET tools.
+# This use case runs the PCPCombine, GenEnsProd, and RegridDataPlane MET tools.
 
 ###################################################################################################
 # METplus Workflow
 # ----------------
 #
 # This workflow loops over the data by process, meaning that each MET tool will run over all times
-# before moving onto the tool. PCPCombine is called first, followed by EnsembleStat,
+# before moving onto the tool. PCPCombine is called first, followed by GenEnsProd,
 # and then, finally, RegridDataPlane.
 
 ###################################################################################################
 # METplus Configuration
 # ---------------------
 #
-# METplus first loads all of the configuration files found in parm/metplus_config. Then, it loads
-# any configuration files passed to METplus by the command line with the -c option.
+# METplus first loads all of the configurations found in parm/metplus_config.
+# Then it loads any configuration files passed to METplus by the command line.
 #
 # .. highlight:: bash
-# .. literalinclude:: ../../../../parm/use_cases/model_applications/short_range/EnsembleStat_fcstHRRR_fcstOnly_SurrogateSevere.conf
+# .. literalinclude:: ../../../../parm/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.conf
 
 ###################################################################################################
 # MET Configuration
@@ -76,18 +76,18 @@
 # If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to:
 # :ref:`Overriding Unsupported MET config file settings<met-config-overrides>`
 #
-# .. note:: See the :ref:`EnsembleStat MET Configuration<ens-stat-met-conf>` section of the User's Guide for more information on the environment variables used in the file below:
+# .. note:: See the :ref:`GenEnsProd MET Configuration<gen-ens-prod-met-conf>` section of the User's Guide for more information on the environment variables used in the file below:
 #
 # .. highlight:: bash
-# .. literalinclude:: ../../../../parm/met_config/EnsembleStatConfig_wrapped
+# .. literalinclude:: ../../../../parm/met_config/GenEnsProdConfig_wrapped
 
 ###################################################################################################
 # Running METplus
 # ---------------
 #
 # The command to run this use case is::
 #
-#    run_metplus.py -c /path/to/METplus/parm/use_cases/model_applications/short_range/EnsembleStat_fcstHRRR_fcstOnly_SurrogateSevere.conf
+#    run_metplus.py /path/to/METplus/parm/use_cases/model_applications/short_range/GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.conf
 
 ###################################################################################################
 # Expected Output
@@ -105,9 +105,9 @@
 #
 # * 20190518/hrrr_ncep_2019051800f036.nc
 #
-# **EnsembleStat output**:
+# **GenEnsProd output**:
 #
-# * ensemble_stat_20190519_120000V_ens.nc
+# * gen_ens_prod_20190519_120000V_ens.nc
 #
 # **RegridDataPlane output**:
 # 
@@ -121,12 +121,12 @@
 # .. note::
 #
 #   * PCPCombineUseCase
-#   * EnsembleStatUseCase
+#   * GenEnsProdUseCase
 #   * RegridDataPlaneUseCase
 #
 #   Navigate to the :ref:`quick-search` page to discover other similar use cases.
 #
 #
 #
-# sphinx_gallery_thumbnail_path = '_static/short_range-EnsembleStat_fcstHRRR_fcstOnly_SurrogateSevere.png'
+# sphinx_gallery_thumbnail_path = '_static/short_range-GenEnsProd_fcstHRRR_fcstOnly_SurrogateSevere.png'
 #

internal_tests/scanning/sonarqube/run_nightly.sh → internal/scripts/sonarqube/run_nightly.sh

@@ -9,7 +9,7 @@
 # the following commands:
 #
 #    git clone https://github.com/dtcenter/METplus
-#    METplus/internal_tests/scanning/sonarqube/run_nightly.sh name
+#    METplus/internal/scripts/sonarqube/run_nightly.sh name
 #
 # Usage: run_nightly.sh name
 #    where "name" specifies a branch, tag, or hash
@@ -39,7 +39,7 @@ SCRIPT_DIR=`dirname $0`
 if [[ ${0:0:1} != "/" ]]; then SCRIPT_DIR=$(pwd)/${SCRIPT_DIR}; fi 
 
 # Define the development environment
-ENV_FILE=${SCRIPT_DIR}/../environment/development.`hostname`
+ENV_FILE=${SCRIPT_DIR}/environment/development.`hostname`
 if [[ ! -e ${ENV_FILE} ]]; then
   echo "$0: ERROR -> Development environment file missing: ${ENV_FILE}"
   exit 1