Enhancement: Add option to check for missing input and skip wrapper runs if threshold is not met

[georgemccabe]

#2460 was completed to add a check for missing inputs and configure a threshold to prevent errors if some inputs were not available. Shannon Shields from NOAA/EMC tested and noted that to match the behavior currently implemented for EVS, the check for missing inputs and comparison to configurable threshold should be done before running any wrappers and all calls to wrappers should not be executed if the threshold is not met.

Describe the Enhancement

Modify logic to loop through all times and gather report of available inputs to see if any wrappers should be called. Consider only doing this if METplus is configured to perform the missing input check.

More thought is needed to determine how to implement this change. For use cases with multiple items in the PROCESS_LIST, it may not be clear which inputs to a wrapper are actual inputs to the use case or output from another wrapper in the PROCESS_LIST. Perhaps a separate wrapper could be added where users can define the filename templates of the inputs to the use case to check for availability. This could allow different thresholds of missing/available input for each input dataset.

Time Estimate

~3 days

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

EVS 2.0 if possible, but scripting is in already place to handle this before the call to METplus wrappers.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[AliciaBentley-NOAA]

@JohnHalleyGotway @georgemccabe I'm following up with Shannon/others to see if this is critical for EVS v2.0 or not. I will let you know if it is. Thanks!

[georgemccabe]

We discussed the details of this issue in the METplus/NOAA Telecon on 6/10/2024. It was reiterated that these changes are not critical to be added for the next release to include in EVS v2.0. Shannon has scripting logic to handle this situation.

An InputCheck wrapper would satisfy this requirement. The wrapper would support the following configuration variable types:

• INPUT_CHECK_DIR = directory containing data (optional)
• INPUT_CHECK_TEMPLATE = path with templates to search
• INPUT_CHECK_THRESH = number between 0-1 (percentage of data that must be present)
• INPUT_CHECK_FAIL_LOG_LEVEL = DEBUG/INFO/WARN/ERROR
• INPUT_CHECK_FAIL_ACTION = EXIT/CONTINUE

Note that we need to consider how to handle multiple ensembles. Per Shannon, all members must be present for each init/lead time in order for the runtime to be counted as valid. Need to flesh out these details about ensemble member handling more.

George suggested using a comma-separated list in the _TEMPLATE variable to list multiple ensembles that correspond to the same time. We would like to be able to easily configure this to process multiple ensembles whose names may be differentiated by a string (rather than a number) that could be tedious to list out. Maybe the CUSTOM_LOOP_LIST behavior would help here?

Example:

[config]

PROCESS_LIST = InputCheck, InputCheck(abc), InputCheck(xyz), PB2NC, PointStat

INPUT_CHECK_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID

INPUT_CHECK_TEMPLATE = /my/dir/to/pb/pb_{valid?fmt=%Y%m%d_%H}.nc
INPUT_CHECK_THRESH = 0.9
INPUT_CHECK_FAIL_LOG_LEVEL = WARN
INPUT_CHECK_FAIL_ACTION = CONTINUE

[abc]

INPUT_CHECK_RUNTIME_FREQ = RUN_ONCE_FOR_EACH

INPUT_CHECK_DIR = /my/dir/to/abc
INPUT_CHECK_TEMPLATE = abc_{valid?fmt=%Y%m%d_%H}.abc
INPUT_CHECK_THRESH = 1.0
INPUT_CHECK_FAIL_LOG_LEVEL = ERROR
INPUT_CHECK_FAIL_ACTION = EXIT

[xyz]

INPUT_CHECK_RUNTIME_FREQ = RUN_ONCE_FOR_EACH
INPUT_CHECK_TEMPLATE = /my/dir/to/xyz/xyz_{valid?fmt=%Y%m%d_%H}.xyz
INPUT_CHECK_THRESH = 0.5
INPUT_CHECK_FAIL_LOG_LEVEL = INFO
INPUT_CHECK_FAIL_ACTION = EXIT

[config]

…

In this example, there are 3 input checks -- config (no instance ID), abc, and xyz.

1st check:
Checks for files matching /my/dir/to/pb/pb_{valid?fmt=%Y%m%d_%H}.nc for each INIT or VALID time (depending on LOOP_BY). If less than 90% of run times have all input files needed, output a WARNING message and continue running the subsequent items in the PROCESS_LIST.

2nd check:
Checks for files matching /my/dir/to/abc/abc_{valid?fmt=%Y%m%d_%H}.abc for each run time (init/valid and forecast leads). If any of run times do not have all input files needed (<100%), output an ERROR message and exit without running the subsequent items in the PROCESS_LIST.

3rd check:
Checks for files matching /my/dir/to/xyz/xyz_{valid?fmt=%Y%m%d_%H}.xyz for each run time (init/valid and forecast leads). If less than 50% of run times have all input files needed, output an INFO message and exit without running the subsequent items in the PROCESS_LIST.

[John-Sharples]

@georgemccabe As discussed the other day, the BoM would be very interested in seeing this feature in METplus.

In our case, the system running METplus is downstream of the forecast (and obs data) systems, and we need to wait for data to arrive before we can run METplus jobs. While this is mostly regular, there are delays for various reasons. This functionality would be useful as it allows jobs to bail out early and save CPU time.

As an alternative, we would also use a tool that generates a list of expected files and passes this back (or writes to a file) so the master scheduler can watch for these files then trigger the appropriate METplus jobs.

[John-Sharples]

One issue we have found with the above approach is that it cannot support wildcard character (*) matching. This shouldn't prevent the the implementation of this functionality, but users should be aware INPUT_CHECK_TEMPLATE will behave slightly differently that other file templates.

[jprestop]

Thanks for pointing this out @John-Sharples.

One issue we have found with the above approach is that it cannot support wildcard character (*) matching. This shouldn't prevent the the implementation of this functionality, but users should be aware INPUT_CHECK_TEMPLATE will behave slightly differently that other file templates.

We will want to ensure that this is well documented, perhaps with a note directive to draw attention to it.