Per #1450, update the user's guide with CRPS updates.

dtcenter · Feb 15, 2021 · e5a08ea · e5a08ea
1 parent 884171d
commit e5a08ea
Show file tree

Hide file tree

Showing 4 changed files with 27 additions and 7 deletions.
diff --git a/met/docs/Users_Guide/appendixC.rst b/met/docs/Users_Guide/appendixC.rst
@@ -887,9 +887,9 @@ ________________________________________________
 CRPS
 ~~~~
 
-Called "CRPS" in ECNT output :numref:`table_ES_header_info_es_out_ECNT`
+Called "CRPS", "CRPSCL", "CRPS_EMP", and "CRPSCL_EMP" in ECNT output :numref:`table_ES_header_info_es_out_ECNT`
 
-The continuous ranked probability score (CRPS) is the integral, over all possible thresholds, of the Brier scores (:ref:`Gneiting et al., 2004 <Gneiting-2004>`). In MET, the CRPS calculation uses a normal distribution fit to the ensemble forecasts. In many cases, use of other distributions would be better.
+The continuous ranked probability score (CRPS) is the integral, over all possible thresholds, of the Brier scores (:ref:`Gneiting et al., 2004 <Gneiting-2004>`). In MET, the CRPS is calculated two ways: using a normal distribution fit to the ensemble forecasts (CRPS and CRPSCL), and using the empirical ensemble distribution (CRPS_EMP and CRPSCL_EMP). In some cases, use of other distributions would be better.
 
 WARNING: The normal distribution is probably a good fit for temperature and pressure, and possibly a not horrible fit for winds. However, the normal approximation will not work on most precipitation forecasts and may fail for many other atmospheric variables.
 
@@ -906,12 +906,14 @@ The score can be interpreted as a continuous version of the mean absolute error
 CRPS Skill Score
 ~~~~~~~~~~~~~~~~
 
-Called "CRPSS" in ECNT output :numref:`table_ES_header_info_es_out_ECNT`
+Called "CRPSS" and "CRPSS_EMP" in ECNT output :numref:`table_ES_header_info_es_out_ECNT`
 
 The continuous ranked probability skill score (CRPSS) is similar to the MSESS and the BSS, in that it compares its namesake score to that of a reference forecast to produce a positively oriented score between 0 and 1.
 
 .. math:: \text{CRPSS} = 1 - \frac{\text{CRPS}_{fcst}}{ \text{CRPS}_{ref}}
 
+For the normal distribution fit (CRPSS), the reference CRPS is computed using the climatological mean and standard deviation. For the empirical distribution (CRPSS_EMP), the reference CRPS is computed by sampling from the assumed normal climatological distribution defined by the mean and standard deviation.  
+
 IGN
 ~~~
 

diff --git a/met/docs/Users_Guide/ensemble-stat.rst b/met/docs/Users_Guide/ensemble-stat.rst
@@ -31,7 +31,7 @@ Often, the goal of ensemble forecasting is to reproduce the distribution of obse
 
 The relative position (RELP) is a count of the number of times each ensemble member is closest to the observation. For stochastic or randomly derived ensembles, this statistic is meaningless. For specified ensemble members, however, it can assist users in determining if any ensemble member is performing consistently better or worse than the others.
 
-The ranked probability score (RPS) is included in the Ranked Probability Score (RPS) line type. It is the mean of the Brier scores computed from ensemble probabilities derived for each probability category threshold (prob_cat_thresh) specified in the configuration file. The continuous ranked probability score (CRPS) is the average the distance between the forecast (ensemble) cumulative distribution function and the observation cumulative distribution function. It is an analog of the Brier score, but for continuous forecast and observation fields. (:ref:`Gneiting et al., 2004 <Gneiting-2004>`). The CRPS statistic is included in the Ensemble Continuous Statistics (ECNT) line type, along with other statistics quantifying the ensemble spread and ensemble mean skill.
+The ranked probability score (RPS) is included in the Ranked Probability Score (RPS) line type. It is the mean of the Brier scores computed from ensemble probabilities derived for each probability category threshold (prob_cat_thresh) specified in the configuration file. The continuous ranked probability score (CRPS) is the average the distance between the forecast (ensemble) cumulative distribution function and the observation cumulative distribution function. It is an analog of the Brier score, but for continuous forecast and observation fields. The CRPS statistic is computed using two methods: assuming a normal distribution defined by the ensemble mean and spread (:ref:`Gneiting et al., 2004 <Gneiting-2004>`) and using the empirical ensemble distribution (:ref:`Hersbach, 2000 <Hersbach-2000>`). The CRPS statistic is included in the Ensemble Continuous Statistics (ECNT) line type, along with other statistics quantifying the ensemble spread and ensemble mean skill.
 
 Ensemble observation error
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -587,10 +587,10 @@ The format of the STAT and ASCII output of the Ensemble-Stat tool are described
     - Number of ensemble values
   * - 27
     - CRPS
-    - The Continuous Ranked Probability Score
+    - The Continuous Ranked Probability Score (normal distribution)
   * - 28
     - CRPSS
-    - The Continuous Ranked Probability Skill Score
+    - The Continuous Ranked Probability Skill Score (normal distribution)
   * - 29
     - IGN
     - The Ignorance Score
@@ -615,6 +615,18 @@ The format of the STAT and ASCII output of the Ensemble-Stat tool are described
   * - 36
     - SPREAD_PLUS_OERR
     - The square root of the sum of unperturbed ensemble variance and the observation error variance
+  * - 37 
+    - CRPSCL
+    - Climatological Continuous Ranked Probability Score (normal distribution)
+  * - 38
+    - CRPS_EMP 
+    - The Continuous Ranked Probability Score (empirical distribution)
+  * - 39
+    - CRPSCL_EMP 
+    - Climatological Continuous Ranked Probability Score (empirical distribution)
+  * - 40 
+    - CRPSS_EMP
+    - The Continuous Ranked Probability Skill Score (empirical distribution)
 
 .. _table_ES_header_info_es_out_RPS:
 

diff --git a/met/docs/Users_Guide/point-stat.rst b/met/docs/Users_Guide/point-stat.rst
@@ -170,7 +170,7 @@ When the "prob" entry is set as a dictionary to define the field of interest, se
 Measures for comparison against climatology
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-For each of the types of statistics mentioned above (categorical, continuous, and probabilistic), it is possible to calculate measures of skill relative to climatology. MET will accept a climatology file provided by the user, and will evaluate it as a reference forecast. Further, anomalies, i.e. departures from average conditions, can be calculated. As with all other statistics, the available measures will depend on the nature of the forecast. Common statistics that use a climatological reference include: the mean squared error skill score (MSESS), the Anomaly Correlation (ANOM_CORR and ANOM_CORR_UNCNTR), scalar and vector anomalies (SAL1L2 and VAL1L2), continuous ranked probability skill score (CRPSS), Brier Skill Score (BSS) (:ref:`Wilks, 2011 <Wilks-2011>`; :ref:`Mason, 2004 <Mason-2004>`).
+For each of the types of statistics mentioned above (categorical, continuous, and probabilistic), it is possible to calculate measures of skill relative to climatology. MET will accept a climatology file provided by the user, and will evaluate it as a reference forecast. Further, anomalies, i.e. departures from average conditions, can be calculated. As with all other statistics, the available measures will depend on the nature of the forecast. Common statistics that use a climatological reference include: the mean squared error skill score (MSESS), the Anomaly Correlation (ANOM_CORR and ANOM_CORR_UNCNTR), scalar and vector anomalies (SAL1L2 and VAL1L2), continuous ranked probability skill score (CRPSS and CRPSS_EMP), Brier Skill Score (BSS) (:ref:`Wilks, 2011 <Wilks-2011>`; :ref:`Mason, 2004 <Mason-2004>`).
 
 Often, the sample climatology is used as a reference by a skill score. The sample climatology is the average over all included observations and may be transparent to the user. This is the case in most categorical skill scores. The sample climatology will probably prove more difficult to improve upon than a long term climatology, since it will be from the same locations and time periods as the forecasts. This may mask legitimate forecast skill. However, a more general climatology, perhaps covering many years, is often easier to improve upon and is less likely to mask real forecast skill.
 

diff --git a/met/docs/Users_Guide/refs.rst b/met/docs/Users_Guide/refs.rst
@@ -132,6 +132,12 @@ References
 |   forecasts. *Monthly Weather Review*, 129, 550-560.
 | 
 
+.. _Hersbach-2000:
+
+| Hersbach, H., 2000: Decomposition of the Continuous Ranked Probability Score
+| for Ensemble Prediction Systems. *Weather and Forecasting*, 15, 559-570.
+| 
+
 .. _Jolliffe-2012:
 
 | Jolliffe, I.T., and D.B. Stephenson, 2012: *Forecast verification. A*