Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task 1455 doc #1585

Merged
merged 68 commits into from
Dec 1, 2020
Merged
Changes from 1 commit
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
3ba94d6
first stab at converting to sphinx
lisagoodrich Oct 27, 2020
283b518
removing all slashes
lisagoodrich Oct 27, 2020
040c19f
adding new link to README.rst file
lisagoodrich Oct 27, 2020
7d33ee1
working on lists
lisagoodrich Oct 28, 2020
4e09923
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Oct 29, 2020
deeb1cb
Made formatting changes
jprestop Oct 30, 2020
b92f4dc
Finished fcst section
jprestop Oct 30, 2020
8472f89
fixing spelling, bolding and italics issues
lisagoodrich Oct 30, 2020
57dea8b
Merge branch 'task_1455_doc' of https://github.com/dtcenter/MET into …
lisagoodrich Oct 30, 2020
1f2402a
updating web links
lisagoodrich Oct 30, 2020
aab6d90
working on formatting
lisagoodrich Nov 3, 2020
0172be2
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 3, 2020
48d5369
updating formatting
lisagoodrich Nov 3, 2020
1907294
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 4, 2020
8dd68ba
formatting
lisagoodrich Nov 4, 2020
83245c4
first attempt to clean up formatting completed.
lisagoodrich Nov 5, 2020
b17c51a
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 5, 2020
0afcabf
adding README to the index file
lisagoodrich Nov 5, 2020
2f6670c
fixing warning errors
lisagoodrich Nov 5, 2020
c6774a5
Bringing README_TC into sphinx. Updating section headers
lisagoodrich Nov 5, 2020
9fc2369
Adding README_TC
lisagoodrich Nov 5, 2020
ea20515
Made formatting updates to README.rst
jprestop Nov 6, 2020
f5b37a0
corrected section under wavelet
jprestop Nov 6, 2020
c35ec9c
small changes
lisagoodrich Nov 6, 2020
12b19a4
Merge branch 'task_1455_doc' of https://github.com/dtcenter/MET into …
lisagoodrich Nov 6, 2020
328ff52
removing met/data/config/README since it is now in met/docs/Users_Guide
lisagoodrich Nov 6, 2020
6d24e46
Added some formatting for headers
jprestop Nov 6, 2020
9a303e0
fixing chapters & sections
lisagoodrich Nov 6, 2020
0cb623f
Fixed warnings from building
jprestop Nov 6, 2020
2982be6
adding in code blocks
lisagoodrich Nov 6, 2020
fc5d39f
fixing git conflicts
lisagoodrich Nov 6, 2020
48c2c51
removing slashes
lisagoodrich Nov 6, 2020
b570fbe
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 9, 2020
9b682dd
changes
lisagoodrich Nov 9, 2020
aa09ec2
Made changes to formatting
jprestop Nov 9, 2020
2342339
removing For example code blocks
lisagoodrich Nov 10, 2020
b815a85
major updates
lisagoodrich Nov 11, 2020
2a51d3a
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 11, 2020
23cfc31
fist pass at document conversion complete.
lisagoodrich Nov 11, 2020
66e5c8f
cleaning up questions about dashes
lisagoodrich Nov 11, 2020
510dde3
Made some formatting modifications
jprestop Nov 11, 2020
2730ccf
Fixed conflicts
jprestop Nov 11, 2020
94019c9
Removing README_TC because it is being replaced by README_TC.rst in m…
jprestop Nov 11, 2020
d4dafad
Removing the reference to the README_TC file
jprestop Nov 12, 2020
772a06a
Making title capitalization consistent with README
jprestop Nov 12, 2020
ea269d8
Added a space in timestring
jprestop Nov 12, 2020
15a6ae0
changing to 'time string' with a space between the words.
lisagoodrich Nov 12, 2020
ba65cf3
adding a link to the new README_TC location in met/docs/Users_Guide
lisagoodrich Nov 12, 2020
ca530da
Merge branch 'task_1455_doc' of https://github.com/dtcenter/MET into …
lisagoodrich Nov 12, 2020
fcff100
Modified references to README and README_TC
jprestop Nov 13, 2020
352f32a
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 13, 2020
84b2f57
small formatting changes
lisagoodrich Nov 13, 2020
93e8813
small formatting changes
lisagoodrich Nov 13, 2020
16ff01c
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 19, 2020
1dbae19
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
jprestop Nov 20, 2020
4c89f7b
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 20, 2020
3031ec4
Merge branch 'task_1455_doc' of https://github.com/dtcenter/MET into …
lisagoodrich Nov 20, 2020
6bd1139
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 23, 2020
4f503ec
fixing tabs
lisagoodrich Nov 23, 2020
49de7f4
fixing spacing around number 11
lisagoodrich Nov 23, 2020
16239e2
removing parenthesis around reference dates.
lisagoodrich Nov 23, 2020
2eca472
adding parenthesis back in.
lisagoodrich Nov 23, 2020
8c998a9
fixing references
lisagoodrich Nov 24, 2020
09e8f11
updating references
lisagoodrich Nov 24, 2020
0eeca87
Merge branch 'develop' of https://github.com/dtcenter/MET into task_1…
lisagoodrich Nov 30, 2020
f985b5c
Update appendixC.rst
jprestop Dec 1, 2020
cec6ee4
Update plotting.rst
jprestop Dec 1, 2020
a0b60f3
Update point-stat.rst
jprestop Dec 1, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixing references
lisagoodrich committed Nov 24, 2020
commit 8c998a97a15d395980efc5cc4ea9915c27c21093
2 changes: 1 addition & 1 deletion met/docs/Users_Guide/appendixC.rst
Original file line number Diff line number Diff line change
@@ -975,7 +975,7 @@ The traditional contingency table statistics computed by the Grid-Stat neighborh

All of these measures are defined in :numref:`categorical variables`.

In addition to these standard statistics, the neighborhood analysis provides additional continuous measures, the Fractions Brier Score and the Fractions Skill Score. For reference, the Asymptotic Fractions Skill Score and Uniform Fractions Skill Score are also calculated. These measures are defined here, but are explained in much greater detail in :ref:`Ebert (2008) <Ebert-2008>` and :ref:`Roberts and Lean 2008 <Roberts-2008>`. Roberts and Lean (2008) also present an application of the methodology.
In addition to these standard statistics, the neighborhood analysis provides additional continuous measures, the Fractions Brier Score and the Fractions Skill Score. For reference, the Asymptotic Fractions Skill Score and Uniform Fractions Skill Score are also calculated. These measures are defined here, but are explained in much greater detail in :ref:`Ebert (2008) <Ebert-2008>` and :ref:`Roberts and Lean (2008) <Roberts-2008>`. Roberts and Lean (2008) also present an application of the methodology.

Fractions Brier Score
~~~~~~~~~~~~~~~~~~~~~
6 changes: 3 additions & 3 deletions met/docs/Users_Guide/ensemble-stat.rst
Original file line number Diff line number Diff line change
@@ -27,16 +27,16 @@ Ensemble statistics

Rank histograms and probability integral transform (PIT) histograms are used to determine if the distribution of ensemble values is the same as the distribution of observed values for any forecast field (:ref:`Hamill, 2001 <Hamill-2001>`). The rank histogram is a tally of the rank of the observed value when placed in order with each of the ensemble values from the same location. If the distributions are identical, then the rank of the observation will be uniformly distributed. In other words, it will fall among the ensemble members randomly in equal likelihood. The PIT histogram applies this same concept, but transforms the actual rank into a probability to facilitate ensembles of differing sizes or with missing members.

Often, the goal of ensemble forecasting is to reproduce the distribution of observations using a set of many forecasts. In other words, the ensemble members represent the set of all possible outcomes. When this is true, the spread of the ensemble is identical to the error in the mean forecast. Though this rarely occurs in practice, the spread / skill relationship is still typically assessed for ensemble forecasts (:ref:`Barker, 1991 <Barker-1991>`; :ref:`Buizza, 1997 <Buizza-1997>`). MET calculates the spread and skill in user defined categories according to :ref:`Eckel et al, 2012 <Eckel-2012>`.
Often, the goal of ensemble forecasting is to reproduce the distribution of observations using a set of many forecasts. In other words, the ensemble members represent the set of all possible outcomes. When this is true, the spread of the ensemble is identical to the error in the mean forecast. Though this rarely occurs in practice, the spread / skill relationship is still typically assessed for ensemble forecasts (:ref:`Barker, 1991 <Barker-1991>`; :ref:`Buizza,1997 <Buizza-1997>`). MET calculates the spread and skill in user defined categories according to :ref:`Eckel et al. (2012) <Eckel-2012>`.

The relative position (RELP) is a count of the number of times each ensemble member is closest to the observation. For stochastic or randomly derived ensembles, this statistic is meaningless. For specified ensemble members, however, it can assist users in determining if any ensemble member is performing consistently better or worse than the others.

The ranked probability score (RPS) is included in the Ranked Probability Score (RPS) line type. It is the mean of the Brier scores computed from ensemble probabilities derived for each probability category threshold (prob_cat_thresh) specified in the configuration file. The continuous ranked probability score (CRPS) is the average the distance between the forecast (ensemble) cumulative distribution function and the observation cumulative distribution function. It is an analog of the Brier score, but for continuous forecast and observation fields. (:ref:`Gneiting et al, 2004 <Gneiting-2004>`). The CRPS statistic is included in the Ensemble Continuous Statistics (ECNT) line type, along with other statistics quantifying the ensemble spread and ensemble mean skill.
The ranked probability score (RPS) is included in the Ranked Probability Score (RPS) line type. It is the mean of the Brier scores computed from ensemble probabilities derived for each probability category threshold (prob_cat_thresh) specified in the configuration file. The continuous ranked probability score (CRPS) is the average the distance between the forecast (ensemble) cumulative distribution function and the observation cumulative distribution function. It is an analog of the Brier score, but for continuous forecast and observation fields. (:ref:`Gneiting et al., 2004 <Gneiting-2004>`). The CRPS statistic is included in the Ensemble Continuous Statistics (ECNT) line type, along with other statistics quantifying the ensemble spread and ensemble mean skill.

Ensemble observation error
~~~~~~~~~~~~~~~~~~~~~~~~~~

In an attempt to ameliorate the effect of observation errors on the verification of forecasts, a random perturbation approach has been implemented. A great deal of user flexibility has been built in, but the methods detailed in :ref:`Candile and Talagrand (2008) <Candille-2008>`. can be replicated using the appropriate options. The user selects a distribution for the observation error, along with parameters for that distribution. Rescaling and bias correction can also be specified prior to the perturbation. Random draws from the distribution can then be added to either, or both, of the forecast and observed fields, including ensemble members. Details about the effects of the choices on verification statistics should be considered, with many details provided in the literature (*e.g.* :ref:`Candille and Talagrand, 2008 <Candille-2008>`; :ref:`Saetra et al., 2004 <Saetra-2004>`; :ref:`Santos and Ghelli, 2012 <Santos-2012>`). Generally, perturbation makes verification statistics better when applied to ensemble members, and worse when applied to the observations themselves.
In an attempt to ameliorate the effect of observation errors on the verification of forecasts, a random perturbation approach has been implemented. A great deal of user flexibility has been built in, but the methods detailed in :ref:`Candille and Talagrand (2008) <Candille-2008>`. can be replicated using the appropriate options. The user selects a distribution for the observation error, along with parameters for that distribution. Rescaling and bias correction can also be specified prior to the perturbation. Random draws from the distribution can then be added to either, or both, of the forecast and observed fields, including ensemble members. Details about the effects of the choices on verification statistics should be considered, with many details provided in the literature (*e.g.* :ref:`Candille and Talagrand, 2008 <Candille-2008>`; :ref:`Saetra et al., 2004 <Saetra-2004>`; :ref:`Santos and Ghelli, 2012 <Santos-2012>`). Generally, perturbation makes verification statistics better when applied to ensemble members, and worse when applied to the observations themselves.

Normal and uniform are common choices for the observation error distribution. The uniform distribution provides the benefit of being bounded on both sides, thus preventing the perturbation from taking on extreme values. Normal is the most common choice for observation error. However, the user should realize that with the very large samples typical in NWP, some large outliers will almost certainly be introduced with the perturbation. For variables that are bounded below by 0, and that may have inconsistent observation errors (e.g. larger errors with larger measurements), a lognormal distribution may be selected. Wind speeds and precipitation measurements are the most common of this type of NWP variable. The lognormal error perturbation prevents measurements of 0 from being perturbed, and applies larger perturbations when measurements are larger. This is often the desired behavior in these cases, but this distribution can also lead to some outliers being introduced in the perturbation step.

4 changes: 2 additions & 2 deletions met/docs/Users_Guide/grid-stat.rst
Original file line number Diff line number Diff line change
@@ -55,7 +55,7 @@ The Grid-Stat tool allows evaluation of model forecasts using model analysis fie
Statistical confidence intervals
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The confidence intervals for the Grid-Stat tool are the same as those provided for the Point-Stat tool except that the scores are based on pairing grid points with grid points so that there are likely more values for each field making any assumptions based on the central limit theorem more likely to be valid. However, it should be noted that spatial (and temporal) correlations are not presently taken into account in the confidence interval calculations. Therefore, confidence intervals reported may be somewhat too narrow (e.g., :ref:`Efron 2007 <Efron-2007>`). See :numref:`Appendix D, Section %s <appendixD>` for details regarding confidence intervals provided by MET.
The confidence intervals for the Grid-Stat tool are the same as those provided for the Point-Stat tool except that the scores are based on pairing grid points with grid points so that there are likely more values for each field making any assumptions based on the central limit theorem more likely to be valid. However, it should be noted that spatial (and temporal) correlations are not presently taken into account in the confidence interval calculations. Therefore, confidence intervals reported may be somewhat too narrow (e.g., :ref:`Efron, 2007 <Efron-2007>`). See :numref:`Appendix D, Section %s <appendixD>` for details regarding confidence intervals provided by MET.

Grid weighting
~~~~~~~~~~~~~~
@@ -78,7 +78,7 @@ The MET software will compute the full one-dimensional Fourier transform, then d

Decomposition via Fourier transform allows the user to evaluate the model separately at each spatial frequency. As an example, the Fourier analysis allows users to examine the "dieoff", or reduction, in anomaly correlation of geopotential height at various levels for bands of waves. A band of low wave numbers, say 0 - 3, represent larger frequency components, while a band of higher wave numbers, for example 70 - 72, represent smaller frequency components. Generally, anomaly correlation should be higher for frequencies with low wave numbers than for frequencies with high wave numbers, hence the "dieoff".

Wavelets, and in particular the MET wavelet tool, can also be used to define a band pass filter (:ref:`Casati et al, 2004 <Casati-2004>`; :ref:`Weniger et al 2016 <Weniger-2016>`). Both the Fourier and wavelet methods can be used to look at different spatial scales.
Wavelets, and in particular the MET wavelet tool, can also be used to define a band pass filter (:ref:`Casati et al., 2004 <Casati-2004>`; :ref:`Weniger et al., 2016 <Weniger-2016>`). Both the Fourier and wavelet methods can be used to look at different spatial scales.

Gradient Statistics
~~~~~~~~~~~~~~~~~~~
10 changes: 5 additions & 5 deletions met/docs/Users_Guide/wavelet-stat.rst
Original file line number Diff line number Diff line change
@@ -8,7 +8,7 @@ Wavelet-Stat Tool
Introduction
____________

The Wavelet-Stat tool decomposes two-dimensional forecasts and observations according to intensity and scale. This section describes the Wavelet-Stat tool, which enables users to apply the Intensity-Scale verification technique described by :ref:`Casati et al (2004) <Casati-2004>`.
The Wavelet-Stat tool decomposes two-dimensional forecasts and observations according to intensity and scale. This section describes the Wavelet-Stat tool, which enables users to apply the Intensity-Scale verification technique described by :ref:`Casati et al. (2004) <Casati-2004>`.

The Intensity-Scale technique is one of the recently developed verification approaches that focus on verification of forecasts defined over spatial domains. Spatial verification approaches, as opposed to point-by-point verification approaches, aim to account for the presence of features and for the coherent spatial structure characterizing meteorological fields. Since these approaches account for the intrinsic spatial correlation existing between nearby grid-points, they do not suffer from point-by-point comparison related verification issues, such as double penalties. Spatial verification approaches aim to account for the observation and forecast time-space uncertainties, and aim to provide feedback on the forecast error in physical terms.

@@ -26,7 +26,7 @@ __________________________________
The method
~~~~~~~~~~

:ref:`Casati et al (2004) <Casati-2004>` applied the Intensity-Scale verification to preprocessed and re-calibrated (unbiased) data. The preprocessing was aimed to mainly normalize the data, and defined categorical thresholds so that each categorical bin had a similar sample size. The recalibration was performed to eliminate the forecast bias. Preprocessing and recalibration are not strictly necessary for the application of the Intensity-Scale technique. The MET Intensity-Scale Tool does not perform either, and applies the Intensity-Scale approach to biased forecasts, for categorical thresholds defined by the user.
:ref:`Casati et al. (2004) <Casati-2004>` applied the Intensity-Scale verification to preprocessed and re-calibrated (unbiased) data. The preprocessing was aimed to mainly normalize the data, and defined categorical thresholds so that each categorical bin had a similar sample size. The recalibration was performed to eliminate the forecast bias. Preprocessing and recalibration are not strictly necessary for the application of the Intensity-Scale technique. The MET Intensity-Scale Tool does not perform either, and applies the Intensity-Scale approach to biased forecasts, for categorical thresholds defined by the user.

The Intensity Scale approach can be summarized in the following 5 steps:

@@ -42,9 +42,9 @@ The Intensity Scale approach can be summarized in the following 5 steps:

**Note** that the MSE of the original binary fields is equal to the proportion of the counts of misses (**c/n**) and false alarms (**b/n**) for the contingency table (:numref:`contingency_table_counts`) obtained from the original forecast and observation fields by thresholding with the same threshold used to obtain the binary forecast and observation fields: :math:`{MSE}(t)=(b+c)/n`. This relation is intuitive when comparing the forecast and observation binary field difference and their corresponding contingency table image (:numref:`contingency_table_counts`).

4. The MSE for the random binary forecast and observation fields is estimated by :math:`{MSE}(t) {random}= {FBI}*{Br}*(1-{Br}) + {Br}*(1- {FBI}*{Br})`, where :math:`{FBI}=(a+b)/(a+c)` is the frequency bias index and :math:`{Br}=(a+c)/n` is the sample climatology from the contingency table (:numref:`contingency_table_counts`) obtained from the original forecast and observation fields by thresholding with the same threshold used to obtain the binary forecast and observation fields. This formula follows by considering the :ref:`Murphy and Winkler (1987) <Murphy-1987>` framework, applying the Bayes' theorem to express the joint probabilities **b/n** and **c/n** as product of the marginal and conditional probability (e.g. :ref:`Jolliffe and Stephenson (2012) <Jolliffe-2012>`; :ref:`Wilks, (2010) <Wilks-2010>`), and then noticing that for a random forecast the conditional probability is equal to the unconditional one, so that **b/n** and **c/n** are equal to the product of the corresponding marginal probabilities solely.
4. The MSE for the random binary forecast and observation fields is estimated by :math:`{MSE}(t) {random}= {FBI}*{Br}*(1-{Br}) + {Br}*(1- {FBI}*{Br})`, where :math:`{FBI}=(a+b)/(a+c)` is the frequency bias index and :math:`{Br}=(a+c)/n` is the sample climatology from the contingency table (:numref:`contingency_table_counts`) obtained from the original forecast and observation fields by thresholding with the same threshold used to obtain the binary forecast and observation fields. This formula follows by considering the :ref:`Murphy and Winkler (1987) <Murphy-1987>` framework, applying the Bayes' theorem to express the joint probabilities **b/n** and **c/n** as product of the marginal and conditional probability (e.g. :ref:`Jolliffe and Stephenson, 2012 <Jolliffe-2012>`; :ref:`Wilks, 2010 <Wilks-2010>`), and then noticing that for a random forecast the conditional probability is equal to the unconditional one, so that **b/n** and **c/n** are equal to the product of the corresponding marginal probabilities solely.

5. For each threshold (**t**) and scale component (**j**), the skill score based on the MSE of binary forecast and observation scale components is evaluated (:numref:`wavelet-stat_Intensity_Scale_skill_score_NIMROD`). The standard skill score definition as in :ref:`Jolliffe and Stephenson (2012) <Jolliffe-2012>` or :ref:`Wilks, (2010) <Wilks-2010>` is used, and random chance is used as reference forecast. The MSE for the random binary forecast is equipartitioned on the **n+1** scales to evaluate the skill score: :math:`{SS} (t,j)=1- {MSE}(t,j)*(n+1)/ {MSE}(t) {random}`
5. For each threshold (**t**) and scale component (**j**), the skill score based on the MSE of binary forecast and observation scale components is evaluated (:numref:`wavelet-stat_Intensity_Scale_skill_score_NIMROD`). The standard skill score definition as in :ref:`Jolliffe and Stephenson (2012) <Jolliffe-2012>` or :ref:`Wilks (2010) <Wilks-2010>` is used, and random chance is used as reference forecast. The MSE for the random binary forecast is equipartitioned on the **n+1** scales to evaluate the skill score: :math:`{SS} (t,j)=1- {MSE}(t,j)*(n+1)/ {MSE}(t) {random}`

The Intensity-Scale (IS) skill score evaluates the forecast skill as a function of the precipitation intensity and of the spatial scale of the error. Positive values of the IS skill score are associated with a skillful forecast, whereas negative values are associated with no skill. Usually large scales exhibit positive skill (large scale events, such as fronts, are well predicted), whereas small scales exhibit negative skill (small scale events, such as convective showers, are less predictable), and the smallest scale and highest thresholds exhibit the worst skill. For the NIMROD case illustrated note the negative skill associated with the 160 km scale, for the thresholds to 4 mm/h, due to the 160 km storm displaced almost its entire length.

@@ -256,7 +256,7 @@ _______________________
member = 2;
}

The **wavelet_flag** and **wavelet_k** variables specify the type and shape of the wavelet to be used for the scale decomposition. The :ref:`Casati et al (2004) <Casati-2004>` method uses a Haar wavelet which is a good choice for discontinuous fields like precipitation. However, users may choose to apply any wavelet family/shape that is available in the GNU Scientific Library. Values for the **wavelet_flag** variable, and associated choices for k, are described below:
The **wavelet_flag** and **wavelet_k** variables specify the type and shape of the wavelet to be used for the scale decomposition. The :ref:`Casati et al. (2004) <Casati-2004>` method uses a Haar wavelet which is a good choice for discontinuous fields like precipitation. However, users may choose to apply any wavelet family/shape that is available in the GNU Scientific Library. Values for the **wavelet_flag** variable, and associated choices for k, are described below:

• **HAAR** for the Haar wavelet (member = 2).