Feature 1908 remove ens

[JohnHalleyGotway]

As promised, these changes REMOVE ensemble post-processing logic from the Ensemble-Stat tool for MET version 11.0.0. That functionality has moved entirely to Gen-Ens-Prod.

Expected Differences

• Do these changes introduce new tools, command line arguments, or configuration file options? [Yes]
  
  If yes, please describe:
  

Listed below are some significant differences to note:

• The ens, nbrhd_prob, nmep_smooth dictionaries have been removed from the Ensemble-Stat config file.

• The ens_thresh and vld_thresh options have been moved from the ens dictionary to the fcst dictionary.

• The ensemble_flag dictionary has been renamed as nc_orank_flag since it controls the output of the "_orank.nc" file. Its contents has also been updated.

• Do these changes modify the structure of existing or add new output data types (e.g. statistic line types or NetCDF variables)? [Yes]
  
  If yes, please describe:
  

The Ensemble-Stat tool no longer generates "_ens.nc" NetCDF output files. Any processing that previously relied on that output should be modified to call Gen-Ens-Prod to create it instead.

Pull Request Testing

• Describe testing already performed for these changes:
  
  Confirmed that the code compiles.
  Confirmed that the unit tests run without runtime errors.
  Diffed the unit tests output against develop-ref and looked through the diffs. Please see this GHA run.

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

Please review the config file changes, documentation updates, and changes introduced in the regression test output.

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [Yes]

• Will this PR result in changes to the test suite? [Yes]
  
  If yes, describe the new output and/or changes to the existing output:
  

• 20 "_ens.nc" output file previously generated by Ensemble-Stat are no longer created, as expected.

• A couple "_orank.nc" output files are added based on the config settings.
  NOTE: @JohnHalleyGotway should check the GHA run and update the unit test xml files accordingly.

• Differences are introduced in 41 files:

  • 18 .stat files
  • 6 _ecnt.txt files, 6 _ssvar.txt files, 11 _orank.nc files

Most of the numeric differences are very small (e.g. 0.0001) are are likely caused by the previously writing the ensemble mean to a NetCDF file and then reading it back in. Now, its all handled in memory, which should like provide better precision.

The following differences should be checked more carefully:

Make sure the _orank.nc output variables are reasonable.

file1: /data/output/met_test_truth/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V.stat
ERROR: found     1 differences in row type ECNT         column CRPS          - max abs: 0.042 
ERROR: found     1 differences in row type ECNT         column IGN           - max abs: 0.00744 
ERROR: found     1 differences in row type ECNT         column ME            - max abs: 0.00001 
ERROR: found     1 differences in row type ECNT         column SPREAD        - max abs: 0.9927 
ERROR: failed tests for ECNT: num

and

file1: /data/output/met_test_truth/ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V.stat
ERROR: found     1 differences in row type ECNT         column CRPS          - max abs: 0.042 
ERROR: found     1 differences in row type ECNT         column IGN           - max abs: 0.00744 
ERROR: found     1 differences in row type ECNT         column ME            - max abs: 0.00001 
ERROR: found     1 differences in row type ECNT         column SPREAD        - max abs: 0.9927 
ERROR: failed tests for ECNT: num

and

file1: /data/output/met_test_truth/stat_analysis_es/AGG_SSVAR_out.stat
ERROR: found     2 differences in row type SSVAR        column FBAR_NCL      - max abs: 89837.42 
ERROR: found     2 differences in row type SSVAR        column FBAR_NCU      - max abs: 89837.42 
ERROR: found     2 differences in row type SSVAR        column FSTDEV        - max abs: 9999.003 
ERROR: found     2 differences in row type SSVAR        column FSTDEV_NCL    - max abs: 5206.059 
ERROR: found     3 differences in row type SSVAR        column FSTDEV_NCU    - max abs: 319069.9 
ERROR: found     1 differences in row type SSVAR        column PR_CORR       - max abs: 0.00001 
ERROR: found     1 differences in row type SSVAR        column PR_CORR_NCL   - max abs: 0.00001 
ERROR: found     1 differences in row type SSVAR        column ME_NCL        - max abs: 0.00522 
ERROR: found     1 differences in row type SSVAR        column ME_NCU        - max abs: 0.00522 
ERROR: found     2 differences in row type SSVAR        column ESTDEV        - max abs: 0.0020998 
ERROR: found     1 differences in row type SSVAR        column ESTDEV_NCL    - max abs: 0.0010932 
ERROR: found     1 differences in row type SSVAR        column ESTDEV_NCU    - max abs: 0.013197 
ERROR: found     1 differences in row type SSVAR        column MSE           - max abs: 0.00001 
ERROR: found     2 differences in row type SSVAR        column BCMSE         - max abs: 0.00001 
ERROR: found     1 differences in row type SSVAR        column RMSE          - max abs: 0.00002 

• Please complete this pull request review by [Tuesday 9/20/22].
  

Pull Request Checklist

See the METplus Workflow for details.

• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s)
  Select: Organization level software support Project or Repository level development cycle Project
  Select: Milestone as the version that will include these changes
• After submitting the PR, select Development with the original issue number.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[JohnHalleyGotway]

Met with @georgemccabe and @j-opatz on 9/19/22 about this.

Determine whether or not this PR should go into beta3 by doing the following:

1. Define tag for the feature_1908_remove_ens branch.
2. Create a METplus branch for testing and modify this file to define MET_FORCE_TAG = {feature branch tag name}
3. Git push using ci-run-all-cases in the commit message to trigger all the use cases to run.

If no runtime errors occur, this is a success and we should include it in beta3.
If runtime errors do occur, talk to @georgemccabe about those diffs to decided whether we should push to beta4 instead.

[georgemccabe]

These changes involve removing ensemble verification from ensemble_stat because it is now only handled in gen_ens_prod. I see that some config options have been removed from the EnsembleStatConfig files because they are no longer relevant to that app.

The documentation changes are clear and descriptive of the new state of the app.

We reviewed the differences in the regression test output and confirmed they are expected. Files ending with _ens.nc are no longer generated by ensemble_stat. In some cases an _orank.nc is now generated when it was not previously. The differences in the stat output are rounding differences due to the new approach to obtaining these data -- previously the data was written to a NetCDF file and reread to generate the stat output, but now the data is stored in memory. Precision was likely lost with the old method, so these changes are improvements. We noticed that the FFBAR differences were a bit high (~0.1), but @JohnHalleyGotway mentioned that this is likely because the values are the squared result of another value.

I approve.

[JohnHalleyGotway]

Did some additional work rectifying the diffs:

• Update the unit test xml's to list 4 additional orank.nc files that are now being created: DONE

ERROR: folder /data/output/met_test_truth missing 4 files
    grid_weight/ensemble_stat_AREA_WEIGHT_20120410_120000V_orank.nc 
    grid_weight/ensemble_stat_COS_LAT_WEIGHT_20120410_120000V_orank.nc 
    grid_weight/ensemble_stat_NO_WEIGHT_20120410_120000V_orank.nc 
    python/ensemble_stat_PYTHON_20050807_120000V_orank.nc 

• Inspect large diffs in ensemble_stat/ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V.stat: DONE

ERROR: found     1 differences in row type ECNT         column CRPS          - max abs: 0.042 
ERROR: found     1 differences in row type ECNT         column IGN           - max abs: 0.00744 
ERROR: found     1 differences in row type ECNT         column ME            - max abs: 0.00001 
ERROR: found     1 differences in row type ECNT         column SPREAD        - max abs: 0.9927 

There was actually a bug in the develop branch where doing gridded verification for all members in the same input file. The control member's value was included in the variance computation in place of the first ensemble member! And thus the SPREAD value is different. This bug is corrected in this feature branch. The other diffs are due to the relatively large values of pressure (101300+). Storing sums and sums of squares are large values decreases precision.

• Inspect large diffs in ensemble_stat/ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19820701_000000V.stat: DONE

ERROR: found     1 differences in row type ECNT         column CRPS          - max abs: 0.0007 
ERROR: found     1 differences in row type ECNT         column IGN           - max abs: 0.00772 
ERROR: found     1 differences in row type ECNT         column SPREAD        - max abs: 0.01075 

See above for bugfix in the handling of ensemble control values when all members are in the same input file.

• Inspect large diffs in stat_analysis_es/AGG_SSVAR_out.stat: DONE

ERROR: found     2 differences in row type SSVAR        column FBAR_NCL      - max abs: 89837.42 
ERROR: found     2 differences in row type SSVAR        column FBAR_NCU      - max abs: 89837.42 
ERROR: found     2 differences in row type SSVAR        column FSTDEV        - max abs: 9999.003 
ERROR: found     2 differences in row type SSVAR        column FSTDEV_NCL    - max abs: 5206.059 
ERROR: found     3 differences in row type SSVAR        column FSTDEV_NCU    - max abs: 319069.9 

Added more checks for bad data to CNTInfo::compute_ci(). Compute compute normal ci's when the forecast, observation, and error standard deviations are bad data.

• Confirm that the ensemble_flag and nc_orank_flag are consistent across versions: DONE
  By default, all nc_orank_flag settings are TRUE except for weight and that is consistent with the previous default settings for ensemble_flag.

[JohnHalleyGotway]

Reviewed the final GHA run for this PR to confirm that the changes from earlier today actually had the desired effect.

Note the following:

1. I see the same set of 4 new "_orank.nc" files as in the previous run.
2. I see the same set of 20 old Ensemble-Stat "_ens.nc" files that are removed as in the previous run.
3. I see the same set of 41 files containing differences as in the previous run (18 .stat, 6 _ecnt.txt, 6 _ssvar.txt, and 11 _orank.nc).
4. Reviewing the 41 diff files, the following are of note:
   • Large diffs in the SPREAD values in ensemble_stat_SINGLE_FILE_GRIB_WITH_CTRL_20160608_060000V.stat and ensemble_stat_SINGLE_FILE_NC_WITH_CTRL_19820701_000000V.stat, as explained above because this feature branch fixes a bug in the handling of the control member when all members are read from the same input file.
   • Very large diffs are noted in AGG_SSVAR_out.stat. However I manually inspected those diffs and see that the new output is preferred. Previously, we were using -9999 values of standard deviation in the computation of normal ci limits. Those unreasonably large values have changed to NA, as expected.
5. Manually ran unit_ascii2nc.xml and inspected the AIRNOW output files. The previous records containing -999 now contain -9999 instead, which is reported in the output of the pntnc2ascii.R Rscript as NA.