Modify the interpretation of the message_type_group_map values to support the use of regular expressions.

[JohnHalleyGotway]

Describe the Enhancement

This issue arose via METplus Discussions dtcenter/METplus#1232. While the user was able to run madis2nc to compute time summaries, he was NOT able to get Point-Stat to read them to verify forecasts of daily temperature min/max.

I was able to replicate the problem using the sample data he provided in this comment. Close inspection reveals that madis2nc is writing the output level values as bad data. Next I inspected the output from the nightly build and found the same to be true there.

# on kiowa
ncdump -v obs_lvl NB20211116/MET-develop/test_output/madis2nc/metar_20120409_time_summary.nc
 obs_lvl = _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,

In general, Point/Ensemble-Stat have no way of processing observations with a bad level value.

However non-time-summary output from madis2nc does work in Point/Ensemble-Stat because of special handling for "surface" message types. The non-time-summary madis2nc output for METAR inputs has message_type = ADPSFC. However the time-summary output sets has message_type = ADPSFC_MIN_030000 (for example). Since that string is NOT included in the surface entry of the message_type_group_map, Point/Ensemble-Stat cannot process those observations.

message_type_group_map = [
   { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET,ADPSFC_MIN_030000,ADPSFC_MAX_030000"; },

This task is to modify the processing of each entry in the comma-separated "val" string. Interpret each entry as a regular expression instead of just doing string matching. Care must be give to differentiate between commas inside of RE's versus those that separate the list items.

Once that works, consider updating the message_type_group_map settings in default config file to match any message_type that begins with the specified string.

Time Estimate

1 day?

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2702691

Define the Metadata

Assignee

• Select engineer(s) or no engineer required: @hsoh-u
• Select scientist(s) or no scientist required: none required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Organization level Project for support of the current coordinated release
• Select Repository level Project for development toward the next official release or add alert: NEED PROJECT ASSIGNMENT label
• Select Milestone as the next bugfix version

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy
  No impacts.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Linked issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[hsoh-u]

The problem is that the pressure or height for the obs data are missing from the input data. Please try it without time summary. Most obs_lvl and obs_hgt are missing value.

[JohnHalleyGotway]

@hsoh-u thanks for pointing that out. Yes, I see in the METAR output from the madis2nc unit tests, that obs_lvl = obs_hgt = NA. I tested using these outputs as input to Point-Stat, using both the raw output (metar_2012040912_F000.nc) and the time summary output (metar_20120409_time_summary.nc).

The first run does produce matched pairs because when verifying against ADPSFC message types, the code DOES NOT actually check the observation level value.

The second run does NOT produce matched pairs because the message type is NOT "ADPSFC"... it's "ADPSFC_MIN_030000" or "ADPSFC_MAX_030000". So the question is, how should we handle this situation?

Instead of checking for message_type = ADPSFC, should we check that it BEGINS with "ADPSFC" instead?

[JohnHalleyGotway]

@hsoh-u please take a look at this comment. Perhaps I should rewrite this issue to support regular expressions when processing the message_type_group_map entries. What do you think?

[hsoh-u]

Yes, please rewrite issue. Is the regular expression (ADPSFC to ADPSFC*) applied to time summary only or any cases?

[JohnHalleyGotway]

@hsoh-u I was thinking of the "-pcprx" command line option for pcp_combine. The default regular expression to match everything is defined on this line. The "-pcprx" command line option can override that default. But on this line, we check whether or not each string matches that specified regular expression.

You could consider defining this function:

   StringArray::has_reg_exp(const std::string, bool forward=true) const;

It'd be very similar to the existing StringArray::has(...) function. However rather than just checking to see if the input string occurs in the StringArray elements, check to see if it matches any element when processed as a regular expression by calling the "check_reg_exp()" function.

However my concern is speed. We may find the check_reg_exp() function to be slow... I'm not sure. But hopefully this would provide a robust, general purpose solution.