Update main_v11.1-ref after #2605 (#2606)

Co-authored-by: John Halley Gotway <[email protected]>
Co-authored-by: Seth Linden <[email protected]>
Co-authored-by: jprestop <[email protected]>
Co-authored-by: Daniel Adriaansen <[email protected]>
Co-authored-by: John and Cindy <[email protected]>
Co-authored-by: Howard Soh <[email protected]>
Co-authored-by: George McCabe <[email protected]>
Co-authored-by: hsoh-u <[email protected]>
Co-authored-by: MET Tools Test Account <[email protected]>
Co-authored-by: Seth Linden <[email protected]>
Co-authored-by: lisagoodrich <[email protected]>
Co-authored-by: davidalbo <[email protected]>
Co-authored-by: Lisa Goodrich <[email protected]>
Co-authored-by: metplus-bot <[email protected]>
Co-authored-by: j-opatz <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Vigh <[email protected]>
Co-authored-by: Tracy Hertneky <[email protected]>
Fix Python environment issue (#2407)
fix definitions of G172 and G220 based on comments in NOAA-EMC/NCEPLIBS-w3emc#157. (#2406)
fix #2380 develop override (#2382)
fix #2408 develop empty config (#2410)
fix #2390 develop compile zlib (#2404)
fix #2412 develop climo (#2422)
fix #2437 develop convert (#2439)
fix for develop, for #2437, forgot one reference to the search_parent for a dictionary lookup.
fix #2452 develop airnow (#2454)
fix #2449 develop pdf (#2464)
fix #2402 develop sonarqube (#2468)
fix #2426 develop buoy (#2475)
fix 2518 dtypes appf docs (#2519)
fix 2531 compilation errors (#2533)
fix #2531 compilation_errors_configure (#2535)

.github/jobs/free_disk_space.sh

@@ -0,0 +1,15 @@
+#! /bin/bash
+
+echo Checking disk usage before cleanup
+df -h
+
+printf "\nRemoving files as suggested by https://github.com/actions/virtual-environments/issues/2840"
+
+sudo rm -rf /usr/share/dotnet
+sudo rm -rf /opt/ghc
+sudo rm -rf "/usr/local/share/boost"
+sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+
+printf "\nChecking disk usage after cleanup"
+
+df -h

.github/workflows/testing.yml

@@ -1,7 +1,7 @@
 name: Testing
 
 # Compile MET for feature branches
-# Run unit tests for pull requests
+# Run unit tests for pull requests   
 
 on:
 
@@ -138,6 +138,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Run Unit Tests in Docker
         run: .github/jobs/run_unit_docker.sh
         env:
@@ -173,6 +176,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Run Unit Tests in Docker
         run: .github/jobs/run_unit_docker.sh
         env:
@@ -209,6 +215,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Run Unit Tests in Docker
         run: .github/jobs/run_unit_docker.sh
         env:
@@ -243,6 +252,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Download ref_config_leads output from artifact
         uses: actions/download-artifact@v3
         with:
@@ -286,6 +298,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Download 1a output from artifact
         uses: actions/download-artifact@v3
         with:
@@ -329,6 +344,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Download 1a output from artifact
         uses: actions/download-artifact@v3
         with:
@@ -403,6 +421,9 @@ jobs:
     steps:
       - uses: actions/checkout@v3
 
+      - name: Free disk space
+        run: .github/jobs/free_disk_space.sh
+
       - name: Download data from previous jobs
         uses: actions/download-artifact@v3
 

data/config/TCDiagConfig_default

@@ -79,7 +79,7 @@ domain_info = [
 
 //
 // Data censoring and conversion
-// May be set separately in each diag_data "field" entry
+// May be set separately in each data "field" array entry
 //
 // censor_thresh = [];
 // censor_val    = [];

docs/Users_Guide/config_options.rst

@@ -287,7 +287,7 @@ To run this utility:
 
 .. code-block:: none
 
-  build_ndbc_stations_from_web.py <-d> <-p> <-o OUTPUT_FILE>
+  build_ndbc_stations_from_web.py <-d> <-p> <-o OUTPUT_FILE> <-e EXISTING_FILE>
 
   Usage: build_ndbc_stations_from_web.py [options]
   Options:
@@ -296,6 +296,8 @@ To run this utility:
     -p, --prune           Prune files that are no longer online (optional, default: False)
     -o OUT_FILE, --out=OUT_FILE
                           Save the text into the named file (optional, default: merged.txt)
+    -e EXISTING_FILE, --existing=EXISTING_FILE
+                          Save the text into the named file (optional, default: ../../../data/table_files/ndbc_stations.xml)
 
 NOTE: The downloaded files are written to a subdirectory ndbc_temp_data which
 can be deleted once the final output file is created.

docs/Users_Guide/data_io.rst

@@ -307,7 +307,7 @@ The following is a summary of the input and output formats for each of the tools
 
     * **Output**: One ASCII file containing 2D object attributes, four ASCII files containing 3D object attributes, and one NetCDF file containing object indices for the gridded simple and cluster object fields.
 
-#. **TC-Dland Tool**
+#. **TC-DLand Tool**
 
     * **Input**: One or more files containing the longitude (Degrees East) and latitude (Degrees North) of all the coastlines and islands considered to be a significant landmass.
 

docs/Users_Guide/installation.rst

@@ -55,7 +55,7 @@ Three external libraries are required for compiling/building MET and should be d
 
 Two additional utilities are strongly recommended for use with MET:
 
-1. The Unified Post-Processor is recommended for post-processing the raw WRF model output prior to verifying the model forecasts with MET. The Unified Post-Processor is freely available for `download <https://dtcenter.org/community-code/unified-post-processor-upp>`_. MET can read data on a standard, de-staggered grid and on pressure or regular levels in the vertical. The Unified Post-Processor outputs model data in this format from both WRF cores, the NMM and the ARW. However, the Unified Post-Processor is not strictly required as long as the user can produce gridded model output on a standard de-staggered grid on pressure or regular levels in the vertical. Two-dimensional fields (e.g., precipitation amount) are also accepted for some modules.
+1. The Unified Post-Processor is recommended for post-processing the raw WRF model output prior to verifying the model forecasts with MET. The Unified Post-Processor is freely available for `download <https://epic.noaa.gov/unified-post-processor/>`_. MET can read data on a standard, de-staggered grid and on pressure or regular levels in the vertical. The Unified Post-Processor outputs model data in this format from both WRF cores, the NMM and the ARW. However, the Unified Post-Processor is not strictly required as long as the user can produce gridded model output on a standard de-staggered grid on pressure or regular levels in the vertical. Two-dimensional fields (e.g., precipitation amount) are also accepted for some modules.
 
 2. The copygb utility is recommended for re-gridding model and observation datasets in GRIB version 1 format to a common verification grid. The copygb utility is distributed as part of the Unified Post-Processor and is available from other sources as well. While earlier versions of MET required that all gridded data be placed on a common grid, MET version 5.1 added support for automated re-gridding on the fly. After version 5.1, users have the option of running copygb to regrid their GRIB1 data ahead of time or leveraging the automated regridding capability within MET. 
 
@@ -113,7 +113,7 @@ Installation of Optional Utilities
 
 As described in the introduction to this section, two additional utilities are strongly recommended for use with MET.
 
-1. The `Unified Post-Processor <https://dtcenter.org/community-code/unified-post-processor-upp>`_ is recommended for post-processing the raw WRF model output prior to verifying the data with MET. The Unified Post-Processor may be used on WRF output from both the ARW and NMM cores.
+1. The `Unified Post-Processor <https://epic.noaa.gov/unified-post-processor/>`_ is recommended for post-processing the raw WRF model output prior to verifying the data with MET. The Unified Post-Processor may be used on WRF output from both the ARW and NMM cores.
 
 2. The copygb utility is recommended for re-gridding model and observation datasets in GRIB format to a common verification grid. The copygb utility is distributed as part of the Unified Post-Processor and is available from other sources as well. Please refer to the "Unified Post-processor" utility mentioned above for information on availability and installation.
 

docs/Users_Guide/met-tc_overview.rst

@@ -9,14 +9,14 @@ Introduction
 
 The purpose of this User's Guide is to provide basic information to the users of the Model Evaluation Tools - Tropical Cyclone (MET-TC) to enable users to apply MET-TC to their tropical cyclone datasets and evaluation studies. MET-TC is intended for use with model forecasts run through a vortex tracking software or with operational model forecasts in Automated Tropical Cyclone Forecast (ATCF) file format.
 
-The following sections provide an overview of MET-TC and its components, as well as basic information on the software build. The required input, including file format and the MET-TC are discussed followed by a description of the TC-Dland tool, TC-Pairs, and TC-Stat tools. Each section covers the input, output and practical usage including a description of the configuration files. This is followed by a short overview of graphical utilities available within the MET-TC release.
+The following sections provide an overview of MET-TC and its components, as well as basic information on the software build. The required input, including file format and the MET-TC are discussed followed by a description of the TC-DLand, TC-Diag, TC-Pairs, TC-Stat, TC-RMW, and RMW-Analysis tools. Each section covers the input, output and practical usage including a description of the configuration files. This is followed by a short overview of graphical utilities available within the MET-TC release.
 
 MET-TC components
 =================
 
 The MET tools used in the verification of Tropical Cyclones are referred to as MET-TC. These tools are shown across the bottom of the flowchart in :numref:`overview-figure`. The MET-TC tools are described in more detail in later sections.
 
-The TC-Dland tool is used to generate a gridded file that determines the location of coastlines and islands, and is used as input to the TC-Pairs tool to determine the distance from land of a particular track point. The TC-Pairs tool matches pairs of input model data and BEST track (or any reference forecast) and calculates position errors. The TC-Stat tool uses the TC-Pairs output to perform filter and summary jobs over the matched pair dataset. The TC-Gen tool performs a categorical analysis for tropical cyclone genesis forecasts. The TC-RMW tool performs a coordinate transformation of gridded model data, centered on the storm's location. The RMW-Analysis tool aggregates TC-RMW output across multiple cases.
+Tropical cyclone forecasts and observations are quite different than numerical model forecasts, and thus they have their own set of tools. These consist of TC-DLand, TC-Diag, TC-Pairs, TC-Stat, TC-Gen, TC-RMW, and RMW-Analysis. The TC-DLand module calculates the distance to land from all locations on a specified grid. This information can be used in later modules to eliminate tropical cyclones that are over land from being included in the statistics. TC-Diag converts gridded model output into cylindrical coordinates for each storm location, calls Python scripts to compute storm-relative diagnostics, and writes ASCII output to be read by TC-Pairs. TC-Pairs matches up tropical cyclone forecasts and observations and writes all output to a file. In TC-Stat, these forecast / observation pairs are analyzed according to user preference to produce statistics. TC-Gen evaluates the performance of Tropical Cyclone genesis forecast using contingency table counts and statistics. TC-RMW performs a coordinate transformation for gridded model or analysis fields centered on the current storm location. RMW-Analysis filters and aggregates the output of TC-RMW across multiple cases.
 
 Input data format
 =================
@@ -112,4 +112,4 @@ If a user has gridded model output, the model data must be run through a vortex
 Output data format
 ==================
 
-The MET package produces output in four basic file formats: STAT files, ASCII files, NetCDF files, and Postscript plots. The MET-TC tool produces output in TCSTAT, which stands for Tropical Cyclone - STAT. This output format consists of tabular ASCII data that can be easily read by many analysis tools and software packages, making the output from MET-TC very versatile. Like STAT, TCSTAT is a specialized ASCII format containing one record on each line. Currently, the only line type available in MET-TC is TCMPR (Tropical Cyclone Matched Pairs). As more line types are included in future releases, all line types will be included in a single TCSTAT file. MET-TC also outputs a NetCDF format file in the TC-Dland tool, as input to the TC-Pairs tool.
+The MET package produces output in four basic file formats: STAT files, ASCII files, NetCDF files, and PostScript plots. The MET-TC tool produces output in TCSTAT, which stands for Tropical Cyclone - STAT. This output format consists of tabular ASCII data that can be easily read by many analysis tools and software packages, making the output from MET-TC very versatile. Like STAT, TCSTAT is a specialized ASCII format containing one record on each line. Currently, the only line type available in MET-TC is TCMPR (Tropical Cyclone Matched Pairs). As more line types are included in future releases, all line types will be included in a single TCSTAT file. The TC-DLand, TC-Diag, TC-RMW, and RMW-Analysis tools also write NetCDF files containing a variety of output data types.

docs/Users_Guide/mode.rst

@@ -112,10 +112,10 @@ Traditionally, MODE defines objects by smoothing and thresholding data from a si
 
 As described in :numref:`MODE-configuration-file`, the **field** entry in the forecast and observation dictionaries define the input data to be processed. If **field** is defined as a dictionary, the traditional method for running MODE is invoked, where objects are defined using a single input field. If **field** is defined as an array of dictionaries, each specifying a different input field, then the multi-variate MODE logic is invoked and requires the **multivar_logic** configuration entry to be set. Traditional MODE is run once for each input field to define objects for that field. Note that the object definition criteria can be defined separately for each field array entry. The objects from each input field are combined into forecast and observation data *super* objects 
 
-The **multivar_logic** and **multivar_intensity** configuration entries, described in :numref:`MODE-configuration-file`, define the boolean logic for combining objects from multiple fields into *super* objects, and then optionally computing intensities for individual input fields when the input is masked to non-missing only inside the *super* objects. If the **multivar_logic** configuration option iset, there must be the same number of fields defined in an array of dictionaries for fcst and for obs as indicated in the **multivar_logic**. Note that the multi-variate MODE forecast and observation input fields and combination logic do not need to
-match. If a particular  **multivar_intensity** value is TRUE the corresponding pair of fields (fcst and obs) are masked to non-missing inside the fcst and obs super objects, and traditional mode is run on that pair of masked inputs producing uniquely named outputs. If it is FALSE, mode is not run for that pair of inputs.
+The **multivar_logic** and **multivar_intensity_flag** configuration entries, described in :numref:`MODE-configuration-file`, define the boolean logic for combining objects from multiple fields into *super* objects, and then optionally computing intensities for individual input fields when the input is masked to non-missing only inside the *super* objects. If the **multivar_logic** configuration option iset, there must be the same number of fields defined in an array of dictionaries for fcst and for obs as indicated in the **multivar_logic**. Note that the multi-variate MODE forecast and observation input fields and combination logic do not need to
+match. If a particular  **multivar_intensity_flag** value is TRUE the corresponding pair of fields (fcst and obs) are masked to non-missing inside the fcst and obs super objects, and traditional mode is run on that pair of masked inputs producing uniquely named outputs. If it is FALSE, mode is not run for that pair of inputs.
 
-If all **multivar_intensity** values are FALSE, the forecast and observation *super* objects are written to NetCDF, text, and postscript output files in the standard mode output format, but with no intensity information. 
+If all **multivar_intensity_flag** values are FALSE, the forecast and observation *super* objects are written to NetCDF, text, and postscript output files in the standard mode output format, but with no intensity information. 
 
 
 Practical information
@@ -247,9 +247,9 @@ _____________________
 
 .. code-block:: none
 
-   multivar_intensity = [FALSE, TRUE, TRUE];
+   multivar_intensity_flag = [FALSE, TRUE, TRUE];
 
-The **multivar_intensity** entry appears only in the **MODEMultivarConfig_default** file. This option is paired with the **multivar_logic** entry, and can take on a value of TRUE or FALSE for each **field**.  In the multivar case, super objects are created using the **multivar_logic** settings. For each input for which **multivar_intensity** is TRUE, the input is masked to be non-missing only within the super objects, and traditional mode is run on that masked input.  For each input for which **multivar_intensity** is FALSE, the input is skipped over.   If all the multivar_intensity values are FALSE, traditional mode output is created for the super objects, but with no intensity information.
+The **multivar_intensity_flag** entry appears only in the **MODEMultivarConfig_default** file. This option is paired with the **multivar_logic** entry, and can take on a value of TRUE or FALSE for each **field**.  In the multivar case, super objects are created using the **multivar_logic** settings. For each input for which **multivar_intensity_flag** is TRUE, the input is masked to be non-missing only within the super objects, and traditional mode is run on that masked input.  For each input for which **multivar_intensity_flag** is FALSE, the input is skipped over.   If all the multivar_intensity_flag values are FALSE, traditional mode output is created for the super objects, but with no intensity information.
 
 
 _____________________
@@ -258,7 +258,7 @@ _____________________
 
    multivar_name = "Precip";
 
-The **multivar_name** entry appears only in the **MODEMultivarConfig_default** file. This option is used only when the multivar option is enabled, and only when all **multivar_intensity** flag values are FALSE. It can be thought of as an identifier for the multivariate super object.  It shows up in output files names and content.  If not set the default value is "Super".   It can be set separately for forecasts and observations, or as a common value for both.
+The **multivar_name** entry appears only in the **MODEMultivarConfig_default** file. This option is used only when the multivar option is enabled, and only when all **multivar_intensity_flag** values are FALSE. It can be thought of as an identifier for the multivariate super object.  It shows up in output files names and content.  If not set the default value is "Super".   It can be set separately for forecasts and observations, or as a common value for both.
 
 
 _____________________
@@ -267,7 +267,7 @@ _____________________
 
    multivar_level = "LO";
 
-The **multivar_level** entry appears only in the **MODEMultivarConfig_default** file. This option is used only when the multivar option is enabled, and only when all **multivar_intensity** flag values are FALSE. It is the identifier for the multivariate super object as regards level.  It shows up in output files names and content.  If not set the default value is "NA".   It can be set separately for forecasts and observations, or as a common value for both.
+The **multivar_level** entry appears only in the **MODEMultivarConfig_default** file. This option is used only when the multivar option is enabled, and only when all **multivar_intensity_flag** values are FALSE. It is the identifier for the multivariate super object as regards level.  It shows up in output files names and content.  If not set the default value is "NA".   It can be set separately for forecasts and observations, or as a common value for both.
 
 _____________________