Update develop-ref after #2247 (#2261)

Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Randy Bullock <[email protected]> Co-authored-by: davidfillmore <[email protected]> Co-authored-by: rgbullock <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: johnhg <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: Howard Soh <[email protected]> Co-authored-by: Seth Linden <[email protected]> Co-authored-by: hsoh-u <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: MET Tools Test Account <[email protected]> Co-authored-by: mo-mglover <[email protected]> Co-authored-by: davidalbo <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: Dave Albo <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: Molly Smith <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: bikegeek <[email protected]>
dtcenter · Sep 14, 2022 · 310c471 · 310c471
1 parent 9a56e08
commit 310c471
Show file tree

Hide file tree

Showing 50 changed files with 463 additions and 254 deletions.
diff --git a/.github/jobs/build_docker_image.sh b/.github/jobs/build_docker_image.sh
@@ -10,7 +10,8 @@ CMD_LOGFILE=${GITHUB_WORKSPACE}/docker_build.log
 
 time_command docker build -t ${DOCKERHUB_TAG} \
     --build-arg SOURCE_BRANCH \
-    --build-arg MET_BASE_IMAGE \
+    --build-arg MET_BASE_REPO \
+    --build-arg MET_BASE_TAG \
     -f $DOCKERFILE_PATH ${GITHUB_WORKSPACE}
 if [ $? != 0 ]; then
   cat ${GITHUB_WORKSPACE}/docker_build.log

diff --git a/.github/jobs/set_job_controls.sh b/.github/jobs/set_job_controls.sh
@@ -5,7 +5,8 @@ run_push=false
 run_unit_tests=false
 run_diff=false
 run_update_truth=false
-met_base_image=minimum
+met_base_repo=met-base
+met_base_tag=v1.0
 input_data_version=develop
 truth_data_version=develop
 
@@ -102,7 +103,7 @@ fi
 # if running unit tests, use unit_test MET base image and push image
 if [ "$run_unit_tests" == "true" ]; then
 
-  met_base_image=unit_test
+  met_base_repo=met-base-unit-test
   run_push=true
 
 fi
@@ -112,7 +113,8 @@ echo ::set-output name=run_push::$run_push
 echo ::set-output name=run_unit_tests::$run_unit_tests
 echo ::set-output name=run_diff::$run_diff
 echo ::set-output name=run_update_truth::$run_update_truth
-echo ::set-output name=met_base_image::$met_base_image
+echo ::set-output name=met_base_repo::$met_base_repo
+echo ::set-output name=met_base_tag::$met_base_tag
 echo ::set-output name=input_data_version::$input_data_version
 echo ::set-output name=truth_data_version::$truth_data_version
 
@@ -121,7 +123,8 @@ echo run_push: $run_push
 echo run_unit_tests: $run_unit_tests
 echo run_diff: $run_diff
 echo run_update_truth: $run_update_truth
-echo met_base_image: $met_base_image
+echo met_base_repo: $met_base_repo
+echo met_base_tag: $met_base_tag
 echo input_data_version: $input_data_version
 echo truth_data_version: $truth_data_version
 

diff --git a/.github/workflows/build_docker_and_trigger_metplus.yml b/.github/workflows/build_docker_and_trigger_metplus.yml
@@ -7,6 +7,8 @@ on:
     paths-ignore:
       - 'met/docs/**'
 
+  workflow_dispatch:
+
 env:
   DOCKERHUB_REPO: dtcenter/met-dev
 
@@ -26,7 +28,8 @@ jobs:
         run: .github/jobs/build_docker_image.sh
         env:
           SOURCE_BRANCH: ${{ steps.get_branch_name.outputs.branch_name }}-lite
-          MET_BASE_IMAGE: minimum
+          MET_BASE_REPO: met-base
+          MET_BASE_TAG: v1.0
 
       - name: Push Docker Image
         run: .github/jobs/push_docker_image.sh

diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml
@@ -54,7 +54,8 @@ jobs:
       run_unit_tests: ${{ steps.job_status.outputs.run_unit_tests }}
       run_diff: ${{ steps.job_status.outputs.run_diff }}
       run_update_truth: ${{ steps.job_status.outputs.run_update_truth }}
-      met_base_image: ${{ steps.job_status.outputs.met_base_image }}
+      met_base_repo: ${{ steps.job_status.outputs.met_base_repo }}
+      met_base_tag: ${{ steps.job_status.outputs.met_base_tag }}
       branch_name: ${{ steps.job_status.outputs.branch_name }}
       truth_data_version: ${{ steps.job_status.outputs.truth_data_version }}
       input_data_version: ${{ steps.job_status.outputs.input_data_version }}
@@ -74,7 +75,8 @@ jobs:
         run: .github/jobs/build_docker_image.sh
         env:
           SOURCE_BRANCH: ${{ needs.job_control.outputs.branch_name }}
-          MET_BASE_IMAGE: ${{ needs.job_control.outputs.met_base_image }}
+          MET_BASE_REPO: ${{ needs.job_control.outputs.met_base_repo }}
+          MET_BASE_TAG: ${{ needs.job_control.outputs.met_base_tag }}
 
       - name: Copy Docker build log into logs directory
         if: always()

diff --git a/.zenodo.json b/.zenodo.json
@@ -0,0 +1,128 @@
+{
+    "creators": [
+        {
+            "orcid": "0000-0002-7761-4962",
+            "affiliation": "",
+            "name": "Halley Gotway, John"
+        },
+        {
+            "orcid": "0000-0002-4066-0965",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Jensen, Tara"
+        },
+        {
+            "orcid": "0000-0002-9084-9766",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Adriaansen, Daniel"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Ahijevych, David"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Albo, David"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Biswas, Mrinal"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Brown, Barbara"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Bullock, Randy"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Burek, Tatiana"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Fillmore, David"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Gilleland, Eric"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Goodrich, Lisa"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Hertneky, Tracy"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Harrold, Michelle"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Kalb, Christina"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Linden, Seth"
+        },
+        {
+            "orcid": "0000-0001-6266-9090",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "McCabe, George"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Newman, Kathryn"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Opatz, John"
+        },
+        {
+            "orcid": "0000-0002-2379-5470",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Prestopnik, Julie"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Prestopnik, Paul"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Soh, Howard"
+        },
+        {
+            "orcid": "",
+            "affiliation": "Cooperative Institute for Research in the Atmosphere",
+            "name": "Strong, Bonny"
+        },
+        {
+            "orcid": "",
+            "affiliation": "National Center for Atmospheric Research",
+            "name": "Win-Gildenmeister, Minna"
+        }
+    ],
+
+    "license": "Apache-2.0",
+    "keywords": ["Model Evaluation Tools", "Verification", "Validation"]
+}
+
diff --git a/data/config/PlotPointObsConfig_default b/data/config/PlotPointObsConfig_default
@@ -78,3 +78,8 @@ point_data = [
 ];
 
 ////////////////////////////////////////////////////////////////////////////////
+
+tmp_dir = "/tmp";
+version = "V11.0.0";
+
+////////////////////////////////////////////////////////////////////////////////
diff --git a/data/table_files/met_header_columns_V11.0.txt b/data/table_files/met_header_columns_V11.0.txt
@@ -17,7 +17,7 @@ V11.0 : STAT : PJC      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID
 V11.0 : STAT : PRC      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL (N_THRESH) THRESH_[0-9]* PODY_[0-9]* POFD_[0-9]*
 V11.0 : STAT : PSTD     : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL (N_THRESH) BASER BASER_NCL BASER_NCU RELIABILITY RESOLUTION UNCERTAINTY ROC_AUC BRIER BRIER_NCL BRIER_NCU BRIERCL BRIERCL_NCL BRIERCL_NCU BSS BSS_SMPL THRESH_[0-9]*
 V11.0 : STAT : ECLV     : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL BASER VALUE_BASER (N_PTS) CL_[0-9]* VALUE_[0-9]*
-V11.0 : STAT : ECNT     : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_ENS CRPS CRPSS IGN ME RMSE SPREAD ME_OERR RMSE_OERR SPREAD_OERR SPREAD_PLUS_OERR CRPSCL CRPS_EMP CRPSCL_EMP CRPSS_EMP
+V11.0 : STAT : ECNT     : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_ENS CRPS CRPSS IGN ME RMSE SPREAD ME_OERR RMSE_OERR SPREAD_OERR SPREAD_PLUS_OERR CRPSCL CRPS_EMP CRPSCL_EMP CRPSS_EMP CRPS_EMP_FAIR
 V11.0 : STAT : RPS      : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL N_PROB RPS_REL RPS_RES RPS_UNC RPS RPSS RPSS_SMPL RPS_COMP
 V11.0 : STAT : RHIST    : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL (N_RANK) RANK_[0-9]*
 V11.0 : STAT : PHIST    : VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL BIN_SIZE (N_BIN) BIN_[0-9]*

diff --git a/docs/Contributors_Guide/coding_standards.rst b/docs/Contributors_Guide/coding_standards.rst
@@ -0,0 +1,5 @@
+****************
+Coding Standards
+****************
+
+Coming Soon!
diff --git a/docs/Contributors_Guide/continuous_integration.rst b/docs/Contributors_Guide/continuous_integration.rst
@@ -0,0 +1,17 @@
+**********************
+Continuous Integration
+**********************
+
+Coming Soon!
+
+Overview
+========
+
+
+Keywords
+========
+
+
+Upstream Testing
+================
+
diff --git a/docs/Contributors_Guide/dev_env.rst b/docs/Contributors_Guide/dev_env.rst
@@ -0,0 +1,14 @@
+***********************
+Development Environment
+***********************
+
+Coming Soon!
+
+Setting Up the Environment
+==========================
+
+
+
+Autoconf Version
+================
+
diff --git a/docs/Contributors_Guide/dockerhub.rst b/docs/Contributors_Guide/dockerhub.rst
@@ -0,0 +1,5 @@
+*********
+DockerHub
+*********
+
+Coming Soon!
diff --git a/docs/Contributors_Guide/documentation.rst b/docs/Contributors_Guide/documentation.rst
@@ -0,0 +1,6 @@
+*************
+Documentation
+*************
+
+Coming Soon!
+
diff --git a/docs/Contributors_Guide/github_workflow.rst b/docs/Contributors_Guide/github_workflow.rst
@@ -0,0 +1,5 @@
+***************
+GitHub Workflow
+***************
+
+Coming Soon!
diff --git a/docs/Contributors_Guide/index.rst b/docs/Contributors_Guide/index.rst
@@ -2,10 +2,22 @@
 Contributor's Guide
 ===================
 
+Welcome to the Model Evaluation Tools (MET) Contributor's Guide. 
+
 .. toctree::
    :titlesonly:
    :numbered:
 
+   coding_standards
+   dev_env
+   github_workflow
+   testing
+   continuous_integration
+   dockerhub
+   documentation
+   templates
+   user_support
+
 Indices and tables
 ==================
 

diff --git a/docs/Contributors_Guide/templates.rst b/docs/Contributors_Guide/templates.rst
@@ -0,0 +1,5 @@
+*********
+Templates
+*********
+
+Coming Soon!
diff --git a/docs/Contributors_Guide/testing.rst b/docs/Contributors_Guide/testing.rst
@@ -0,0 +1,15 @@
+*******
+Testing
+*******
+
+Coming Soon!
+
+make test
+=========
+
+
+Unit Tests
+==========
+
+Regression Tests
+================
diff --git a/docs/Contributors_Guide/user_support.rst b/docs/Contributors_Guide/user_support.rst
@@ -0,0 +1,5 @@
+************
+User Support
+************
+
+Coming Soon!
diff --git a/docs/Users_Guide/appendixC.rst b/docs/Users_Guide/appendixC.rst
@@ -972,9 +972,9 @@ where :math:`BS_m` is the Brier score for the m-th category (:ref:`Tödter and A
 CRPS
 ----
 
-Called "CRPS", "CRPSCL", "CRPS_EMP", and "CRPSCL_EMP" in ECNT output :numref:`table_ES_header_info_es_out_ECNT`
+Called "CRPS", "CRPSCL", "CRPS_EMP", "CRPS_EMP_FAIR" and "CRPSCL_EMP" in ECNT output :numref:`table_ES_header_info_es_out_ECNT`
 
-The continuous ranked probability score (CRPS) is the integral, over all possible thresholds, of the Brier scores (:ref:`Gneiting et al., 2004 <Gneiting-2004>`). In MET, the CRPS is calculated two ways: using a normal distribution fit to the ensemble forecasts (CRPS and CRPSCL), and using the empirical ensemble distribution (CRPS_EMP and CRPSCL_EMP). In some cases, use of other distributions would be better.
+The continuous ranked probability score (CRPS) is the integral, over all possible thresholds, of the Brier scores (:ref:`Gneiting et al., 2004 <Gneiting-2004>`). In MET, the CRPS is calculated two ways: using a normal distribution fit to the ensemble forecasts (CRPS and CRPSCL), and using the empirical ensemble distribution (CRPS_EMP and CRPSCL_EMP). The empirical ensemble CRPS is also adjusted by subtracting 1/2(m) times the mean absolute difference of the ensemble members (where m is the ensemble size), this is saved as CRPS_EMP_FAIR. In some cases, use of other distributions would be better.
 
 WARNING: The normal distribution is probably a good fit for temperature and pressure, and possibly a not horrible fit for winds. However, the normal approximation will not work on most precipitation forecasts and may fail for many other atmospheric variables.
 
@@ -988,6 +988,10 @@ The overall CRPS is calculated as the average of the individual measures. In equ
 
 The score can be interpreted as a continuous version of the mean absolute error (MAE). Thus, the score is negatively oriented, so smaller is better. Further, similar to MAE, bias will inflate the CRPS. Thus, bias should also be calculated and considered when judging forecast quality using CRPS.
 
+To calculate CRPS_EMP_FAIR (bias adjusted, empirical ensemble CRPS)
+
+.. math:: \text{CRPS_EMP_FAIR} = \text{CRPS_EMP} - \frac{1}{2*n} *  \frac{1}{n*(n-1)} \sum|f_{i} - f_{j}|
+
 CRPS Skill Score
 ----------------
 

diff --git a/docs/Users_Guide/ensemble-stat.rst b/docs/Users_Guide/ensemble-stat.rst
@@ -36,7 +36,7 @@ Often, the goal of ensemble forecasting is to reproduce the distribution of obse
 
 The relative position (RELP) is a count of the number of times each ensemble member is closest to the observation. For stochastic or randomly derived ensembles, this statistic is meaningless. For specified ensemble members, however, it can assist users in determining if any ensemble member is performing consistently better or worse than the others.
 
-The ranked probability score (RPS) is included in the Ranked Probability Score (RPS) line type. It is the mean of the Brier scores computed from ensemble probabilities derived for each probability category threshold (prob_cat_thresh) specified in the configuration file. The continuous ranked probability score (CRPS) is the average the distance between the forecast (ensemble) cumulative distribution function and the observation cumulative distribution function. It is an analog of the Brier score, but for continuous forecast and observation fields. The CRPS statistic is computed using two methods: assuming a normal distribution defined by the ensemble mean and spread (:ref:`Gneiting et al., 2004 <Gneiting-2004>`) and using the empirical ensemble distribution (:ref:`Hersbach, 2000 <Hersbach-2000>`). The CRPS statistic is included in the Ensemble Continuous Statistics (ECNT) line type, along with other statistics quantifying the ensemble spread and ensemble mean skill.
+The ranked probability score (RPS) is included in the Ranked Probability Score (RPS) line type. It is the mean of the Brier scores computed from ensemble probabilities derived for each probability category threshold (prob_cat_thresh) specified in the configuration file. The continuous ranked probability score (CRPS) is the average the distance between the forecast (ensemble) cumulative distribution function and the observation cumulative distribution function. It is an analog of the Brier score, but for continuous forecast and observation fields. The CRPS statistic is computed using two methods: assuming a normal distribution defined by the ensemble mean and spread (:ref:`Gneiting et al., 2004 <Gneiting-2004>`) and using the empirical ensemble distribution (:ref:`Hersbach, 2000 <Hersbach-2000>`). The CRPS statistic using the empirical ensemble distribution can be adjusted (bias corrected) by subtracting 1/2(m) times the mean absolute difference of the ensemble members (where m is the ensemble size), this is saved as a separate stastics called CRPS_EMP_FAIR. The CRPS (and CRPS FAIR) statistic is included in the Ensemble Continuous Statistics (ECNT) line type, along with other statistics quantifying the ensemble spread and ensemble mean skill.
 
 The Ensemble-Stat tool can derive ensemble relative frequencies and verify them as probability forecasts all in the same run. Note however that these simple ensemble relative frequencies are not actually calibrated probability forecasts. If probabilistic line types are requested (output_flag), this logic is applied to each pair of fields listed in the forecast (fcst) and observation (obs) dictionaries of the configuration file. Each probability category threshold (prob_cat_thresh) listed for the forecast field is applied to the input ensemble members to derive a relative frequency forecast. The probability category threshold (prob_cat_thresh) parsed from the corresponding observation entry is applied to the (gridded or point) observations to determine whether or not the event actually occurred. The paired ensemble relative freqencies and observation events are used to populate an Nx2 probabilistic contingency table. The dimension of that table is determined by the probability PCT threshold (prob_pct_thresh) configuration file option parsed from the forecast dictionary. All probabilistic output types requested are derived from the this Nx2 table and written to the ascii output files. Note that the FCST_VAR name header column is automatically reset as "PROB({FCST_VAR}{THRESH})" where {FCST_VAR} is the current field being evaluated and {THRESH} is the threshold that was applied.
 
@@ -718,6 +718,9 @@ The format of the STAT and ASCII output of the Ensemble-Stat tool are described
   * - 40 
     - CRPSS_EMP
     - The Continuous Ranked Probability Skill Score (empirical distribution)
+  * - 41 
+    - CRPS_EMP_FAIR
+    - The Continuous Ranked Probability Skill Score (empirical distribution) adjusted by subtracting 1/2(m) times the mean absolute difference of the ensemble members (m is the ensemble size)
 
 .. _table_ES_header_info_es_out_RPS: