Update develop-ref after #2457 and #2453 (#2458)

Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Mallory Row <[email protected]> Closes #1986 fix develop Fix broken documentation links (#2004) fix #2026 develop StatAnalysis looping (#2028) fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) fix #2070 var list numeric order (#2072) fix #2087 develop docs_pdf (#2091) fix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) Fix for Dockerfile smell DL4000 (#2112) fix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) fix #2137 develop PointStat -obs_valid_beg/end (#2141) fix failured introduced by urllib3 (see urllib3/urllib3#2168) fix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) fix #2168 develop - StatAnalysis time shift (#2169) fix releases. (#2183) fix #2189 develop - spaces in complex thresholds (#2191) fix #2179 develop TCPairs fix -diag argument (#2187) fixes (#2200) fix diff tests (#2217) fix automated tests (#2237) fix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) fix #2241 Create directory containing -out_stat file (#2242) fix #2245 use unique run ID to name logger instance (#2247) fix #2244 develop fix diff tests (#2254) fixture to set pytest tmpdir (#2261) fix #1853 develop - PointStat don't require mask variables to be set (#2262) fix #2279 develop - buoy station file from 2022 (#2280) fix (#2313) fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary fix bug described in pangeo-data/xESMF#246 fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls
dtcenter · Dec 19, 2023 · 972610f · 972610f
1 parent bf5d9d0
commit 972610f
Show file tree

Hide file tree

Showing 28 changed files with 415 additions and 202 deletions.
diff --git a/.github/actions/run_tests/Dockerfile b/.github/actions/run_tests/Dockerfile
@@ -3,8 +3,6 @@ FROM alpine:latest
 COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /entrypoint.sh
 
-RUN apk add --update --no-cache docker python3 py3-pip bash
-
-RUN pip3 install python-dateutil requests
+RUN apk add --update --no-cache docker python3 py3-pip bash py3-dateutil py3-requests
 
 ENTRYPOINT ["/entrypoint.sh"]
diff --git a/.github/parm/use_case_groups.json b/.github/parm/use_case_groups.json
@@ -1,7 +1,7 @@
 [
    {
     "category": "met_tool_wrapper",
-    "index_list": "0-29,59-61,63",
+    "index_list": "0-29,59-63",
     "run": false
   },
   {
@@ -101,7 +101,7 @@
   },
   {
     "category": "medium_range",
-    "index_list": "3-5",
+      "index_list": "3-5,10",
     "run": false
   },
   {

diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst
@@ -10302,20 +10302,19 @@ METplus Configuration Glossary
 
      | *Used by:* TCDiag
 
-   TC_DIAG_CENSOR_THRESH
-     Specify the value for 'censor_thresh' in the MET configuration file for TCDiag.
+   TC_DIAG_DOMAIN_INFO<n>_OVERRIDE_DIAGS
+     Specify the value for the nth 'domain_info.override_diags' in the MET configuration file for TCDiag.
 
      | *Used by:* TCDiag
 
-   TC_DIAG_CENSOR_VAL
-     Specify the value for 'censor_val' in the MET configuration file for TCDiag.
+   TC_DIAG_CENSOR_THRESH
+     .. warning:: **DEPRECATED:** Please use :term:`BOTH_VAR<n>_OPTIONS` instead.
 
-     | *Used by:* TCDiag
+   TC_DIAG_CENSOR_VAL
+     .. warning:: **DEPRECATED:** Please use :term:`BOTH_VAR<n>_OPTIONS` instead.
 
    TC_DIAG_CONVERT
-     Specify the value for 'convert' in the MET configuration file for TCDiag.
-
-     | *Used by:* TCDiag
+     .. warning:: **DEPRECATED:** Please use :term:`BOTH_VAR<n>_OPTIONS` instead.
 
    TC_DIAG_DATA_DOMAIN
      Specify the value for 'data.domain' in the MET configuration file for TCDiag.
@@ -10407,8 +10406,8 @@ METplus Configuration Glossary
 
      | *Used by:* TCDiag
 
-   TC_DIAG_NC_RNG_AZI_FLAG
-     Specify the value for 'nc_rng_azi_flag' in the MET configuration file for TCDiag.
+   TC_DIAG_NC_CYL_GRID_FLAG
+     Specify the value for 'nc_cyl_grid_flag' in the MET configuration file for TCDiag.
 
      | *Used by:* TCDiag
 
@@ -10423,7 +10422,10 @@ METplus Configuration Glossary
      | *Used by:* TCDiag
 
    TC_DIAG_OUTPUT_PREFIX
-     Specify the value for 'output_prefix' in the MET configuration file for TCDiag.
+     .. warning:: **DEPRECATED:** Please use :term:`TC_DIAG_OUTPUT_BASE_FORMAT` instead.
+
+   TC_DIAG_OUTPUT_BASE_FORMAT
+     Specify the value for 'output_base_format' in the MET configuration file for TCDiag.
 
      | *Used by:* TCDiag
 

diff --git a/docs/Users_Guide/release-notes.rst b/docs/Users_Guide/release-notes.rst
@@ -197,6 +197,13 @@ the use case will not run until the METplus configuration file has been updated.
 How to upgrade
 ^^^^^^^^^^^^^^
 
+This video provides a demonstration of the process to upgrade a use case.
+
+.. raw:: html
+
+  <iframe width="560" height="315" src="https://www.youtube.com/embed/QCBlCsxmBDo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+
+
 Removing **<WRAPPER_NAME>_CONFIG_FILE**, e.g. :term:`GRID_STAT_CONFIG_FILE`,
 from the METplus config file will prevent the errors and
 allow the use case to run.

diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst
@@ -8225,9 +8225,7 @@ METplus Configuration
 | :term:`TC_DIAG_DOMAIN_INFO<n>_N_AZIMUTH`
 | :term:`TC_DIAG_DOMAIN_INFO<n>_DELTA_RANGE_KM`
 | :term:`TC_DIAG_DOMAIN_INFO<n>_DIAG_SCRIPT`
-| :term:`TC_DIAG_CENSOR_THRESH`
-| :term:`TC_DIAG_CENSOR_VAL`
-| :term:`TC_DIAG_CONVERT`
+| :term:`TC_DIAG_DOMAIN_INFO<n>_OVERRIDE_DIAGS`
 | :term:`TC_DIAG_INPUT_DATATYPE`
 | :term:`TC_DIAG_DATA_DOMAIN`
 | :term:`TC_DIAG_DATA_LEVEL`
@@ -8247,10 +8245,10 @@ METplus Configuration
 | :term:`TC_DIAG_RADIAL_VELOCITY_LONG_FIELD_NAME`
 | :term:`TC_DIAG_VORTEX_REMOVAL`
 | :term:`TC_DIAG_ONE_TIME_PER_FILE_FLAG`
-| :term:`TC_DIAG_NC_RNG_AZI_FLAG`
+| :term:`TC_DIAG_NC_CYL_GRID_FLAG`
 | :term:`TC_DIAG_NC_DIAG_FLAG`
 | :term:`TC_DIAG_CIRA_DIAG_FLAG`
-| :term:`TC_DIAG_OUTPUT_PREFIX`
+| :term:`TC_DIAG_OUTPUT_BASE_FORMAT`
 | :term:`TC_DIAG_SKIP_IF_OUTPUT_EXISTS`
 | :term:`TC_DIAG_MET_CONFIG_OVERRIDES`
 | :term:`TC_DIAG_SKIP_VALID_TIMES`
@@ -8447,42 +8445,8 @@ ${METPLUS_DOMAIN_INFO_LIST}
      - domain_info.delta_range_km
    * - :term:`TC_DIAG_DOMAIN_INFO<n>_DIAG_SCRIPT`
      - domain_info.diag_script
-
-${METPLUS_CENSOR_THRESH}
-""""""""""""""""""""""""
-
-.. list-table::
-   :widths: 5 5
-   :header-rows: 1
-
-   * - METplus Config(s)
-     - MET Config File
-   * - :term:`TC_DIAG_CENSOR_THRESH`
-     - censor_thresh
-
-${METPLUS_CENSOR_VAL}
-"""""""""""""""""""""
-
-.. list-table::
-   :widths: 5 5
-   :header-rows: 1
-
-   * - METplus Config(s)
-     - MET Config File
-   * - :term:`TC_DIAG_CENSOR_VAL`
-     - censor_val
-
-${METPLUS_CONVERT}
-""""""""""""""""""
-
-.. list-table::
-   :widths: 5 5
-   :header-rows: 1
-
-   * - METplus Config(s)
-     - MET Config File
-   * - :term:`TC_DIAG_CONVERT`
-     - convert
+   * - :term:`TC_DIAG_DOMAIN_INFO<n>_OVERRIDE_DIAGS`
+     - domain_info.override_diags
 
 ${METPLUS_DATA_FILE_TYPE}
 """""""""""""""""""""""""
@@ -8670,17 +8634,17 @@ ${METPLUS_ONE_TIME_PER_FILE_FLAG}
    * - :term:`TC_DIAG_ONE_TIME_PER_FILE_FLAG`
      - one_time_per_file_flag
 
-${METPLUS_NC_RNG_AZI_FLAG}
-""""""""""""""""""""""""""
+${METPLUS_NC_CYL_GRID_FLAG}
+"""""""""""""""""""""""""""
 
 .. list-table::
    :widths: 5 5
    :header-rows: 1
 
    * - METplus Config(s)
      - MET Config File
-   * - :term:`TC_DIAG_NC_RNG_AZI_FLAG`
-     - nc_rng_azi_flag
+   * - :term:`TC_DIAG_NC_CYL_GRID_FLAG`
+     - nc_cyl_grid_flag
 
 ${METPLUS_NC_DIAG_FLAG}
 """""""""""""""""""""""
@@ -8715,8 +8679,8 @@ ${METPLUS_OUTPUT_PREFIX}
 
    * - METplus Config(s)
      - MET Config File
-   * - :term:`TC_DIAG_OUTPUT_PREFIX`
-     - output_prefix
+   * - :term:`TC_DIAG_OUTPUT_BASE_FORMAT`
+     - output_base_format
 
 ${METPLUS_MET_CONFIG_OVERRIDES}
 """""""""""""""""""""""""""""""

diff --git a/docs/_static/medium_range-GridStat_fcstGFS_obsOMI_TotalColumnOzone.png b/docs/_static/medium_range-GridStat_fcstGFS_obsOMI_TotalColumnOzone.png
diff --git a/docs/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.py b/docs/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.py
@@ -0,0 +1,136 @@
+"""
+Grid-Stat: Using Python Embedding for Total Column Ozone
+================================================================================
+
+model_applications/medium_range/GridStat_fcstGFS_obsOMI
+_TotalColumnOzone.conf
+
+"""
+##############################################################################
+# Scientific Objective
+# --------------------
+# To provide useful statistical information on the relationship between observation
+# data in gridded format to a gridded forecast. The Ozone Monitoring Instrument (OMI)
+# data covers a 36 hour period and is compared to the average of the gridded forecast
+# files (all from the same initialization time). 
+
+##############################################################################
+# Datasets
+# --------
+#
+# | **Forecast:** GFS (1 degree Lat/Lon grid)
+# | **Observation:** NASA's Level-3 Aura/OMI Global Total Ozone Mapping Spectrometer-Like (TOMS-Like) Total Column Ozone gridded product OMTO3e (0.25deg Lat/Lon grid)
+# | **Location:** Click here for the METplus releases page and download sample data for the appropriate release: https://github.com/dtcenter/METplus/releases
+# | This tarball should be unpacked into the directory that you will set the value of INPUT_BASE. See `Running METplus`_ section for more information.
+
+##############################################################################
+# METplus Components
+# ------------------
+#
+# This use case first runs PCPCombine on the forecast data to build a
+# 36-hour Total Colum Ozone mean/average file. Then the forecast data
+# are compared to the observation data using GridStat. This use case utilizes 
+# the METplus GridStat wrapper to read in the observation file using Python Embedding.
+
+##############################################################################
+# METplus Workflow
+# ----------------
+#
+# The following tools are used for each run time:
+#
+# PCPCombine (forecast) > GridStat
+#
+# It processes the following run times:
+#
+# | **Valid:** 2023-12-05 06Z (36 hour period covering 2023-12-03 18Z - 2023-12-05 06Z)
+# | **Init:** 2023-12-03 06Z
+# | **Forecast lead:** 48
+
+##############################################################################
+# METplus Configuration
+# ---------------------
+#
+# METplus first loads all of the configuration files found in parm/metplus_config,
+# then it loads any configuration files passed to METplus via the command line
+# parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.conf
+#
+# .. highlight:: bash
+# .. literalinclude:: ../../../../parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.conf
+
+##############################################################################
+# MET Configuration
+# -----------------
+#
+# METplus sets environment variables based on user settings in the METplus configuration file. 
+# See :ref:`How METplus controls MET config file settings<metplus-control-met>` for more details. 
+#
+# **YOU SHOULD NOT SET ANY OF THESE ENVIRONMENT VARIABLES YOURSELF! THEY WILL BE OVERWRITTEN BY METPLUS WHEN IT CALLS THE MET TOOLS!**
+#
+# If there is a setting in the MET configuration file that is currently not supported by METplus you'd like to control, please refer to:
+# :ref:`Overriding Unsupported MET config file settings<met-config-overrides>`
+#
+# .. note:: See the :ref:`GridStat MET Configuration<grid-stat-met-conf>` section of the User's Guide for more information on the environment variables used in the file below:
+#
+# .. highlight:: bash
+# .. literalinclude:: ../../../../parm/met_config/GridStatConfig_wrapped
+
+##############################################################################
+# Python Embedding
+# ----------------
+#
+# This use case uses one Python embedding script with GridStat to read the input observation data
+#
+# parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone/read_omi-aura_l3-omto3e.py
+#
+# .. highlight:: python
+# .. literalinclude:: ../../../../parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone/read_omi-aura_l3-omto3e.py
+
+
+##############################################################################
+# Running METplus
+# ---------------
+#
+# Pass the use case configuration file to the run_metplus.py script
+# along with any user-specific system configuration files if desired::
+#
+#    run_metplus.py /path/to/METplus/parm/use_cases/model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone.conf /path/to/user_system.conf
+#
+# See :ref:`running-metplus` for more information.
+#
+
+##############################################################################
+# Expected Output
+# ---------------
+#
+# A successful run will output the following both to the screen and to the logfile::
+#
+#   INFO: METplus has successfully finished running.
+#
+# Refer to the value set for **OUTPUT_BASE** to find where the output data was generated.
+# Output for this use case will be found in /model_applications/medium_range/GridStat_fcstGFS_obsOMI_TotalColumnOzone
+# (relative to **OUTPUT_BASE**)
+# and will contain the following files:
+#
+# * PCPCombine/pcp_combine.tozone_l0.mean.f048.init2023120306.nc
+# * GridStat/grid_stat_GFS_vs_OMI_480000L_20231205_060000V.stat 
+
+##############################################################################
+# Keywords
+# --------
+#
+# .. note::
+#
+#   * PCPCombineToolUseCase
+#   * GridStatToolUseCase
+#   * MediumRangeAppUseCase
+#   * GRIBFileUseCase
+#   * PythonEmbeddingFileUseCase
+#   * RegriddingInToolUseCase
+#   * NOAAEMCOrgUseCase
+#
+#   Navigate to the :ref:`quick-search` page to discover other similar use cases.
+#
+#
+#
+# sphinx_gallery_thumbnail_path = '_static/medium_range-GridStat_fcstGFS_obsOMI_TotalColumnOzone.png'
+#
diff --git a/internal/scripts/docker_env/README.md b/internal/scripts/docker_env/README.md
@@ -416,9 +416,6 @@ docker push dtcenter/metplus-envs:geovista.${METPLUS_ENV_VERSION}
 ```
 export METPLUS_ENV_VERSION=v6.0
 ./scripts/geovista_env.sh ${METPLUS_ENV_VERSION}
-
-cd /home/met_test/geovista
-/home/met_test/.conda/envs/geovista.${METPLUS_ENV_VERSION}/bin/pip3 install --no-deps --editable .
 ```
 
 ## pandac.v6.0 (from metplotpy.v6.0)

diff --git a/internal/scripts/docker_env/scripts/cfgrib_env.sh b/internal/scripts/docker_env/scripts/cfgrib_env.sh
@@ -21,7 +21,4 @@ METPLUS_VERSION=$1
 ENV_NAME=cfgrib.${METPLUS_VERSION}
 
 mamba create -y --name ${ENV_NAME} -c conda-forge python=3.10.4
-mamba install -y --name ${ENV_NAME} -c conda-forge metpy==1.4.0
-mamba install -y --name ${ENV_NAME} -c conda-forge netcdf4==1.6.2
-mamba install -y --name ${ENV_NAME} -c conda-forge cfgrib==0.9.10.1
-mamba install -y --name ${ENV_NAME} -c conda-forge pygrib==2.1.4
+mamba install -y --name ${ENV_NAME} -c conda-forge metpy==1.4.0 netcdf4==1.6.2 cfgrib==0.9.10.1 pygrib==2.1.4
diff --git a/internal/scripts/docker_env/scripts/cycloneplotter_env.sh b/internal/scripts/docker_env/scripts/cycloneplotter_env.sh
@@ -23,6 +23,4 @@ ENV_NAME=cycloneplotter.${METPLUS_VERSION}
 BASE_ENV=metplus_base.${METPLUS_VERSION}
 
 mamba create -y --clone ${BASE_ENV} --name ${ENV_NAME}
-mamba install -y --name ${ENV_NAME} -c conda-forge cartopy==0.20.3
-mamba install -y --name ${ENV_NAME} -c conda-forge matplotlib==3.5.2
-mamba install -y --name ${ENV_NAME} -c conda-forge pandas==1.4.3
+mamba install -y --name ${ENV_NAME} -c conda-forge cartopy==0.20.3 matplotlib==3.5.2 pandas==1.4.3
diff --git a/internal/scripts/docker_env/scripts/diff_env.sh b/internal/scripts/docker_env/scripts/diff_env.sh
@@ -24,8 +24,7 @@ ENV_NAME=diff.${METPLUS_VERSION}
 BASE_ENV=netcdf4.${METPLUS_VERSION}
 
 mamba create -y --clone ${BASE_ENV} --name ${ENV_NAME}
-mamba install -y --name ${ENV_NAME} -c conda-forge pandas==2.0.2
-mamba install -y --name ${ENV_NAME} -c conda-forge pillow==9.2.0
+mamba install -y --name ${ENV_NAME} -c conda-forge pandas==2.0.2 pillow==9.2.0
 
 apt-get update
 apt-get install -y poppler-utils

diff --git a/internal/scripts/docker_env/scripts/geovista_env.sh b/internal/scripts/docker_env/scripts/geovista_env.sh
@@ -25,6 +25,4 @@ apt install -y libgl1-mesa-glx
 apt install -y libegl1
 
 mamba create -y --name ${ENV_NAME} -c conda-forge python=3.10.4
-mamba install -y --name ${ENV_NAME} -c conda-forge geovista==0.3.0
-mamba install -y --name ${ENV_NAME} -c conda-forge xarray==2023.8.0
-mamba install -y --name ${ENV_NAME} -c conda-forge iris==3.7.0
+mamba install -y --name ${ENV_NAME} -c conda-forge geovista==0.3.0 xarray==2023.8.0 iris==3.7.0
diff --git a/internal/scripts/docker_env/scripts/icecover_env.sh b/internal/scripts/docker_env/scripts/icecover_env.sh
@@ -23,7 +23,4 @@ ENV_NAME=icecover.${METPLUS_VERSION}
 BASE_ENV=py_embed_base.${METPLUS_VERSION}
 
 mamba create -y --clone ${BASE_ENV} --name ${ENV_NAME}
-mamba install -y --name ${ENV_NAME} -c conda-forge xarray==2023.5.0
-mamba install -y --name ${ENV_NAME} -c conda-forge pyresample==1.27.1
-mamba install -y --name ${ENV_NAME} -c conda-forge scikit-learn==1.3.0
-mamba install -y --name ${ENV_NAME} -c conda-forge pyproj==3.6.0
+mamba install -y --name ${ENV_NAME} -c conda-forge xarray==2023.5.0 pyresample==1.27.1 scikit-learn==1.3.0 pyproj==3.6.0
diff --git a/internal/scripts/docker_env/scripts/metdataio_env.sh b/internal/scripts/docker_env/scripts/metdataio_env.sh
@@ -23,6 +23,4 @@ BASE_ENV=metplus_base.${METPLUS_VERSION}
 
 mamba create -y --clone ${BASE_ENV} --name ${ENV_NAME}
 
-mamba install -y --name ${ENV_NAME} -c conda-forge lxml==4.9.1
-mamba install -y --name ${ENV_NAME} -c conda-forge pymysql==1.0.2
-mamba install -y --name ${ENV_NAME} -c conda-forge pandas==1.5.1
+mamba install -y --name ${ENV_NAME} -c conda-forge lxml==4.9.1 pymysql==1.0.2 pandas==1.5.1