Add ORANK line type to the HiRA output from Point-Stat.

[JohnHalleyGotway]

Describe the Enhancement

This request arose via met-help issue from the US Army:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99476

The HiRA algorithm in Point-Stat applies probabilistic and ensemble verification techniques when verifying deterministic model output against point observations. The pseudo-ensemble is defined using forecast values from nearby grid points. So this is the analogue of neighborhood verification with gridded data but for point observations instead. The value at each forecast grid point is replaced by a number between 0 and 1 representing the frequency of a user-configurable event within the neighborhood.

As of met-9.1, the HiRA methodology can produce both probabilistic and ensemble output line types. This task is to also add the ORANK line type as an output for HiRA. During the initial HiRA development this seemed like data overload, but some scientists at the Army would find it useful.

When adding ORANK to the HiRA output from Point-Stat, also consider if any of the other probabilistic and ensemble output line types should be added at the same time.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
No sub-issues.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: Seth
• Select scientist(s) or no scientist required: no scientist needed

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
• METplus issue to make this output line type configurable: Make output_flag.orank configurable for Point-Stat. METplus#1103

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[jwraby]

ARL scientists are interested in this feature request which promises to provide ensemble matched pair data to support
our research into applying Design of Experiments to NWP models to better understand what aspects of the model are
contributing to the error. The matched pair data and the derivable bias and other statistics enables development of
statistical techniques which could isolate the model process or configuration setting which is contributing
significantly to the error. This information could assist in efforts to improve model performance. Smith and Penc (2016)
describe a promising approach which uses the statistical design of experiments (DoE) technique and present a method
for developing the design matrix for applying the technique to NWP forecasts. Smith et. al (2019) demonstrates an
application of this method to NWP. Although other methods are available to study these factor level effects,
for example Stein and Alpert (1993), Cleveland et al (2020) demonstrates that factor methods are a special case of DoE.
The DoE technique involves a controlled statistical analysis of numerous model runs which were configured and run as
prescribed by the design matrix.

Cleveland JL, Smith JA, Collins JP. Factor effects in numerical simulations.
J Atmos Sci. 2020;77(7):2439–2451. doi:10.1175/jas-d-19-0263.1.

Smith JA, Penc RS. A design of experiments approach to evaluating parameterization schemes for numerical weather
prediction: problem definition and proposed solution approach. Joint Statistical Meetings Proceedings,
Section on Statistics in Defense and National Security, Conference on Applied Statistics in Defense; 2015 Aug 8–13.
2017 Jan. p. 4183–4192.

Smith JA, Cleveland JL, Raby JW, Penc R. Applying design of experiments to numerical weather prediction.
Annual Joint Statistical Meeting, American Statistical Association; 2019.

Stein U, Alpert P. Factor separation in numerical simulations. J Atmos Sci. 1993;50(14):2107–2115.
doi:10.1175/1520-0469(1993)050<2107:FSINS>2.0.CO;2.

[sethlinden]

I just coordinated with John that all outstanding issues have been resolved. I'm closing the issue now.