Refactor verification tasks in the SRW App and allow for time-lagged ensemble members

[gsketefian]

Description

The SRW App currently has the capability to perform both deterministic and ensemble verification (vx) using the MET (Model Evaluation Tools) software and METplus configuration (conf) files. However, some aspects of the vx tasks have proven buggy. Thus, refactoring is needed to fix these issues.

Solution

The refactoring has already mostly been done in an older version of the App (before the regional_workflow repo was folded into ufs-srweather-app) and is currently under testing. Once testing is complete, it needs to be ported to the App's develop branch.

Requirements

• Include capability to use time-lagged ensemble members.
• Run MET's pcp_combine and pb2nc tools (which perform some vx pre-processing but no actual verification) in tasks that are separate from those that run the grid_stat, point_stat, gen_ens_prod, and ensemble_stat tools (which calculate vx statistic).
  • Both the deterministic and ensemble vx tasks depend on these two tasks being complete, so they should separated out and called before the grid_stat, point_stat, gen_ens_prod, and ensemble_stat tasks.
  • Launch the tasks that will run pcp_combine and pb2nc on obs only once per cycle, not for every ensemble member (as is done now). The current approach causes failures when two different ensemble members try to write to the same output obs file (and unnecessarily uses up core hours).
  • Launch the task that will run pcp_combine on forecasts for every ensemble member.
• Modify the rocoto template xml to allow generation of tasks in loops over fields (APCP, REFC, RETOP) or groups of fields (surface, upper air) to be verified and loops over APCP accumulation hours (01, 03, 06, 24; for APCP only).
  • Separate the tasks that run point_stat on surface and upper air fields into two separate ones (one for surface, another for upper air).
• As much as possible, set vx parameters all in one location and pass those to the METplus configuration files via environment variables (instead of hard-coding them in the conf files). This makes the various conf files more similar (and, in the case of APCP with 3, 6, and 24 hour accumulations, allows the conf files to be combined into one).
• Enable calling of deterministic and ensemble vx tasks independently, i.e. the user should be able to choose one or both sets of tasks.
• Enable running of vx tasks without running other tasks in the App, i.e. using staged observation and forecast data.
• For each MET tool called, pass in a set of forecast hours (for which the tool will run) that has been filtered to remove all forecast hours for which observations are not available (so that the MET tool will not return an error).
• In the ex-scripts that call the MET tools (via METplus), add error checking (i.e. cause the task to fail if METplus returns a non-zero error code). This is currently not the case.
• Rename vx tasks in the App to:
  • Distinguish between deterministic and ensemble workflow tasks.
  • Determine from the task names which MET tool(s) each task calls.
• Add WE2E tests to test the vx capabilities in various configurations.

Acceptance Criteria (Definition of Done)

The App should be able to run the vx tasks in various configurations and reproduce baselines from previous testing.

[mkavulich]

Resolved via #809