Potential issue with obs input template

[michelleharrold]

Hello!

Gerard (@gsketefian) and I have both encountered a potential issue with specifying the obs input template in various METplus wrappers (e.g., grid-stat, and ensemble-stat) with METplus 4.1.1. Here is an example specifying the obs template for MRMS data within our ensemble-stat conf file:

OBS_ENSEMBLE_STAT_GRID_INPUT_TEMPLATE = {valid?fmt=%Y%m%d}/MergedReflectivityQCComposite_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2

We have a script that pulls the MRMS data from the NOAA HPSS, and the data is gzipped when it is pulled; our script also unzips the data, so it is in grib2 format for the verification. In our obs directory, we have both the .gz and .grib2 files. When METplus runs with both the .gz and .grib2 files in the same directory, it fails (TypeError: can only concatenate str (not "list") to str). When we remove the .gz file and only retain the .grib2 file it runs. It is easy enough to remove the .gz files (and we probably should for storage reasons!), but we would have assumed that since the .grib2 was specified in the template, it would not have been an issue (and therefore, wanted to report the behavior we are seeing).

Thanks!
Michelle

[JohnHalleyGotway]

@michelleharrold thanks for clearly defining the behavior. Yes, it certainly sounds like the presence of the *.gz file is affecting the behavior of the wrapper.

@georgemccabe is the best person to take a look at this behavior. And I'm responding to this one right away given that we have a release approaching quickly. George, please take a look when you get a chance to see if we can address this in the METplus 5.0 release.

@michelleharrold, obviously, please continue removing the *.gz files prior to running METplus for the time being to avoid this problem in version 4.1.1. Longer term, I wonder if encapsulating the data retrieval logic you're running as a custom script upstream of METplus could be added as a python wrapper within METplus. Pulling remote data certainly seems like a common task.

[michelleharrold]

@georgemccabe and @JohnHalleyGotway - thanks for the quick response!

Here is the log file on hera (from a recent run by @gsketefian):
/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/DTC_ensemble_task/expt_dirs/vx_set15/fcst_ens02_run.obs_get.vx_det_ens.timelag_00h_00h.vxfields_all.accums_all.2019070100_2019070112/log/metplus.log.ensemblestat_REFC_cmn_2019070112

If you'd like me to pull it down from hera, I can certainly do that. As a note @gsketefian is using METplus 4.1.1 but has it installed under a project space, so he could add some debug statements.

Thanks!
Michelle

[georgemccabe]

Thanks for sending that path. It looks like a file window is being used to find files that fall within a range of times around the valid time. It looks like the logic in EnsembleStat wrapper could be improved to properly handle when a list of obs files were found to prevent this failure. It shouldn't fail altogether if more than 1 file is found.

However, in this case, it looks like the logic to find files within a time window may have a bug where it is ignoring the end of the actual file path and considered a file with a .gz extension as a match. I'll do some testing and try to get this fixed for the next beta release.

[georgemccabe]

I created issue #1939 to document this work. Is this something that would warrant creating a new bugfix release for v4.1.X or can it be made available in the upcoming v5.0.0 release?

[michelleharrold]

@georgemccabe - thanks!

I will ultimately defer to @gsketefian, but I am good with it in the v5.0.0 release.

[georgemccabe]

OK, I fixed the logic for develop and main_v4.1. They are in PR review now (#1941 and #1942). Unless we hear back from you that you need a v4.1.5 bugfix release to obtain this fix, we will not be cutting the bugfix release at this time. The changes will be available in v5.0.0 regardless. Thanks for discovering this issue!