Enhance Ensemble-Stat to compute probabilistic statistics for user-defined or climatology-based thresholds.

[JohnHalleyGotway]

Describe the Enhancement

Enhance Ensemble-Stat to compute probabilistic statistics on the fly for each threshold. This is needed by the NOAA/EMC global group for ensemble verification. Update the Ensemble-Stat configuration file to write output for PCT, PSTD, PRC, PRJ, and ECON line types. When climo_cdf thresholds are used, also write the mean of the stats across those bins (just like we do in Point-Stat and Grid-Stat).

Note that a related issue #1583 already adds probabilistic output to Ensemble-Stat for MET version 10.1.0. As of MET version 10.0.0, the ensemble relative frequency probabilities are computed for each categorical threshold (cat_thresh) entry listed in the ensemble (ens) dictionary. And those gridded relative frequencies (i.e. probabilities) are written to the NetCDF output file.

For this issue, we'll...

• Leave the ens logic in place as is for MET version 10.1.0. But note that the "ens" dictionary will be completely removed in MET version 11.0.0 by Remove ensemble post-processing from the Ensemble-Stat tool. #1908.
• Add support for "cat_thresh" being defined in the "fcst" dictionary. For each "cat_thresh", ensemble_stat will compute the relative frequencies on the fly, write them to the gridded NetCDF output file, and also verify them against the provided gridded or point observations. When writing the NetCDF output, make sure to NOT write the same variable name twice in case the user has requested relative frequencies for the same field in both the "ens" and "fcst" dictionaries.

This change makes Ensemble-Stat slightly more confusing in MET version 10.1.0, but it'll be simplified when we remove the "ens" dictionary entirely in MET version 11.0.0.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
No sub-issues needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2793541

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @JohnHalleyGotway
• Select scientist(s) or no scientist required: @j-opatz

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
• May need a METplus issue to make cat_thresh configurable in the fcst dictionary.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[ghost]

BSS and ROC are currently required by EMC global ensemble group. They are also interested in Reliability diagram.

[JohnHalleyGotway]

On 11/19/2020, we had a telecon with NOAA/EMC staff to discuss ensemble-related verification metrics. The key point here is that the ensemble probabilities need to be defined relative to climatology bins rather than absolute thresholds.
The following statistics need to be supported like this:

• RPSS
• CRPSS
• Brier Skill Score
• Reliability diagram
• ROC curve
• Economic value curve

[j-opatz]

Email chain of 1/29/21 resulted in confirmation of scores/output requested by EMC. These are given in priority order, highest first:

• CRPSS
• RPSS
• BSS
• ROC
• Reliability
• Brier Score and Resolution (potentially done if BSS is already created)
• Economic Value

[BinbinZhou-NOAA]

John, This new update means, in the new MET version if clim-mean and clim-stdev are used as reference and without explicitly defining the thresholds in ensemble_stat verification, probabilistic scores can be got from ECNT line type? Binbin

[JohnHalleyGotway]

@BinbinZhou-NOAA, yes, this should finally be doing what you were looking for a few years ago.

You can run Ensemble-Stat, providing a climo mean and standard deviation in the config file, and request probabilistic output. If you provide explicit thresholds using the prob_cat_thresh option, then those thresholds will be applied. If you leave prob_cat_thresh empty, but do request climo bins, then it'll do what you want.

Both of these examples are now tested in the unit tests, and I'll list some sample PSTD output files below to demonstrate:

1. For APCP/24 with prob_cat_thresh = [ >0, >0.1, >0.25, >0.5, >1.0, >1.5, >2.0 ]; we get this output file:
   ensemble_stat_20100101_120000V_pstd.txt
   Note in that output file, the FCST_VAR columns has strings like this: PROB(APCP_24>1.0)

2. For TMP/Z2 with prob_cat_thresh = []; and climo_cdf.cdf_bins = 10; we get this output file:
   ensemble_stat_NCEP_1.0DEG_20120410_120000V_pstd.txt
   Note in that output file, the VX_MASK column includes the "BIN#" or "BIN_MEAN" string. Also, note that FCST_VAR and FCST_LEV just state "TMP" and "Z2". So this is the result of thresholding relative to the climo distribution 9 times (>=CDP10, >=CDP20, ..., >=CDP90).

And its the "BIN_MEAN" output that you're really looking for. Once @jprestop has installed met-10.1.0-beta6 on WCOSS (released today), please test out this functionality to confirm that it makes sense.

I'm just realizing that for the BIN_MEAN line, the OBS_THRESH column is currently set to ">=CDP10.00000". I'm realizing that should probably be set to something else. Perhaps just set it to "NA"... or perhaps set it in some way to indicate the number of bins used? What do you think?

[BinbinZhou-NOAA]

John, Thanks for the detailed explanations. As for CDP setting, current ">=CDP10.00000". is fine for our global ensemble verification. I think this also should be fine for other global ensemble verifications. Binbin

[BinbinZhou-NOAA]

John, Per request by Julie to test the new released MET_10.1.0_beta6 for github #1259 <#1259> , Yes I did it by GEFS's 500mb Height. The running is successful and the stat file was generated. But I am not for sure the results are correct or not. Could you please double check my output files so I can reply to Julie? The testing directory on VENUS is /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/gens_met_10.1.0_beta6 My output_flag setting in EnsembleStat config file ( verf_g2g_ens_stat_config_bin11) is output_flag = { ecnt = BOTH; rps = BOTH; pct = BOTH; prc = BOTH; pstd = BOTH; prj = BOTH; eclv = BOTH; rhist = BOTH; phist = NONE; orank = NONE; ssvar = NONE; relp = BOTH; } The running script is run.sh and the output files are those ensemble_stat_BIN11* including stat file. Comparing my *pstd.txt file to the sample pstd file you attached, it looks good. But not for sure the stat file is also ok or not. Thanks! Binbin

[JohnHalleyGotway]

@BinbinZhou-NOAA I wanted to mention that @j-opatz and I discussed this more internally and decided that we'd prefer to report the OBS_THRESH for the PSTD BIN_MEAN line as "==CDP10" instead of ">=CDP10". I realize it's a very small difference, but I'd rather make it now while I remember all the details than have to remember how to explain those details later!

I committed this change today via #2082. And that small change will be included in the official release (not beta6). Figured you wouldn't have any objections.

As for testing, it seems to me you could do the following:

1. Presumably, you've already setup Ensemble-Stat previously to generate probability forecasts for >=CDP10, >=CDP20, >=CDP30, ..., >=CDP90.
2. Then you ran those probability forecasts through Point-Stat to generate PSTD output.
3. You loaded that PSTD output into METviewer and then plotted the mean across those thresholds.

You could run that same data directly through Ensemble-Stat now and look at the PSTD line with BIN_MEAN in the VX_MASK column.

Those BIN_MEAN PSTD stats should match the mean of the stats as computed by METviewer, at least for a single case. And this assumes you're using a consistent masking region and interpolation method in Point-Stat and Ensemble-Stat.

It seems to me that that'd be a good way to validate the new logic in Ensemble-Stat. If you have the time to run that comparison by next Monday or Tuesday, that'd be really helpful for us.

[BinbinZhou-NOAA]

John, No objection for that change. Sure I will test both ways and compare their results. Actually I have already done the second way, i.e. directly run Ensemble_stat to get the probabilistic type PSTD, etc. Checking BIN_MEAN PSTD data in the output stat file , yes, I can find BIN_MEAN for PSTD data line as shown below: V10.1.0 GEFS NA 2400000 20210306_000000 20210306_000000 000000 20210306_000000 20210306_000000 HGT gpm P500 HGT gpm P500 ANALYS NHM_BIN_MEAN NEAREST 1 ==0.25000

=CDP5.00000 NA NA PSTD 21960 5 0.58563 NA

NA 0.012816 0.023588 0.16915 0.69135 0.15838 NA NA 0.1796 NA NA 0.097027 0.03838 0 0.25 0.5 0.75 1 I have 2 questions for the first way: (1) In the Ensemble_stat config, within ensemble_flag, should I set frequency = TRUE; ? (2) In the Point_stat config, how to set thresholds for fcst field ( HGT/P500) ? since the second step is probabilistic grid-stat Binbin

[JohnHalleyGotway]

Binbin,

Yes, you'd configure Ensemble-Stat (or Gen-Ens-Prod) to run with:

ensemble_flag = {
...
   frequency = TRUE;
...
};

And in the ensemble dictionary, you'd reference the CDP thresholds, something like this:

ens = {
   cat_thresh = [ >=CDP10, >=CDP20, >=CDP30, >=CDP40, >=CDP50, >=CDP60, >=CD70, >=CDP80, >=CDP90 ]; 
   field = [ 
     { name = "TMP"; level = "P500"; } // or whatever field you're testing
   ];
}

In the Point-Stat config file, you'll verify the NetCDF output from Ensemble-Stat. It'll look something like this:

fcst = {
   level = "(*,*)"; // same level setting for all fields
   prob = TRUE; // same prob flag for all fields
   cat_thresh = ==0.10; // same PCT thresholds for all fields

   field = [
      { name = "TMP_P500_ENS_FREQ_geCDP10"; },
      { name = "TMP_P500_ENS_FREQ_geCDP20"; },
      { name = "TMP_P500_ENS_FREQ_geCDP30"; },
      { name = "TMP_P500_ENS_FREQ_geCDP40"; },
      { name = "TMP_P500_ENS_FREQ_geCDP50"; },
      { name = "TMP_P500_ENS_FREQ_geCDP60"; },
      { name = "TMP_P500_ENS_FREQ_geCDP70"; },
      { name = "TMP_P500_ENS_FREQ_geCDP80"; },
      { name = "TMP_P500_ENS_FREQ_geCDP90"; }
   ];
}

[BinbinZhou-NOAA]

John, For testing the second way (first run ensemble_stat and then Grid_stat), when running ensemble_stat, the error message says: ERROR : Simple_Node::check(double, double, double) const -> climatological distribution percentile threshold ">=CDP10" requested with invalid mean (-9999) or standard deviation (-9999). Binbin

[JohnHalleyGotway]

@BinbinZhou-NOAA yes, that error message indicates that you're using a climo-based threshold without sufficient climatology data (mean and standard deviation) provided in the config file.

This could happen for one of two reasons:

1. Your MET configuration file does not include the "climo_mean" and "climo_stdev" settings needed to derive this.
2. You did provide that data in the config file, but there is some bug in the MET code for how/when to set/check it.

Once you confirmed that the climo_mean and climo_stdev data has been provided, please point me to where/what command you're running so I can debug further.

[BinbinZhou-NOAA]

John, i can confirm that both clim_mean and clim_stdev are defined in the same testing directory The testing directory is /gpfs/dell2/emc/modeling/noscrub/Binbin.Zhou/met/gens_met_10.1.0_beta6 The running script is run_way2.sh All required files are stored in the testing directory The way1_output is the results of run_way1.sh that only runs ensemble_stat to get all required probabilistic scores. Binbin

[JohnHalleyGotway]

Binbin, my apologies for sending your bad directions. You should be able to use the gen_ens_prod tool instead of the ensemble_stat to compute those climo-based probabilities.

Here's why...
The Ensemble-Stat config file contains settings for the "ens", "fcst", "obs", "climo_mean", and "climo_stdev" dictionaries. Note that the "ens" dictionary will be removed in MET version 11.0.0 since it is replaced by Gen-Ens-Prod. The climatology data corresponds to the "fcst" and "obs" dictionaries. It does NOT correspond to the "ens" dictionary. That's why using ">=CDP10" did not work inside the "ens" dictionary... Ensemble-Stat isn't reading any climo data for those fields.

However, the Gen-Ens-Prod config file contains settings for the "ens", "climo_mean", and "climo_stdev" dictionaries. And the climatology data read there really does correspond to the "ens" dictionaries.

In run_way2.sh, please try swapping out your call to Ensemble-Stat with a call to Gen-Ens-Prod instead.

I did some work on setting this up more. Please take a look in /u/John.H.Gotway/test_way2.

[BinbinZhou-NOAA]

John, Thanks for your checking and suggestions. The testing of the first way was successful and all required probabilistic line types are generated. As for the second way, since ensemble product generation has been separated from Ensemble_stat after MET_10.1.0, using the new tool ens_prod_gen to generate probability at first, then use grid_stat to generate those prob line types. The testing run is successful and the final averaged probabilistic scores are generated. To compare the results of these 2 ways, I have following questions for line type PSTD: 1. For method 2, the scores like BS, BSS, or other metrics, have to use AnalysisStat to obtain them by summarizing over all thresholds? While the stat file of method 1 has summarized BIN_MEAN for PSTD line type. 2. Not for sure MetViewer can handle both methods so I can compare the results from these 2 methods Binbin

…

On Sat, Mar 5, 2022 at 7:27 PM johnhg ***@***.***> wrote: Binbin, my apologies for sending your bad directions. You should be able to use the gen_ens_prod tool instead of the ensemble_stat to compute those climo-based probabilities. Here's why... The Ensemble-Stat config file contains settings for the "ens", "fcst", "obs", "climo_mean", and "climo_stdev" dictionaries. Note that the "ens" dictionary will be removed in MET version 11.0.0 since it is replaced by Gen-Ens-Prod. The climatology data corresponds to the "fcst" and "obs" dictionaries. It does NOT correspond to the "ens" dictionary. That's why using ">=CDP10" did not work inside the "ens" dictionary... Ensemble-Stat isn't reading any climo data for those fields. However, the Gen-Ens-Prod config file contains settings for the "ens", "climo_mean", and "climo_stdev" dictionaries. And the climatology data read there really does correspond to the "ens" dictionaries. In run_way2.sh, please try swapping out your call to Ensemble-Stat with a call to Gen-Ens-Prod instead. I did some work on setting this up more. Please take a look in /u/John.H.Gotway/test_way2. — Reply to this email directly, view it on GitHub <#1259 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQBMPFJGEWF6RFHR3XCXDS3U6P3WLANCNFSM4KXL6DWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Binbin Zhou IMSG at NOAA/NWS/NCEP/EMC 5830 University Research Ct. College Park, MD 20740 ***@***.*** 301-683-3683

[BinbinZhou-NOAA]

The comparion between the 2ways shows that the results are verify closed.
Thanks!

Binbin

[JohnHalleyGotway]

Whew! That's excellent news and a relief. Thank you very much for testing this.