-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[develop] Use templates for METplus config files #683
Merged
MichaelLueken
merged 131 commits into
ufs-community:develop
from
gsketefian:feature/METplus_templates
May 12, 2023
Merged
Changes from 109 commits
Commits
Show all changes
131 commits
Select commit
Hold shift + click to select a range
7b2ae3b
Specify location of staged forecast output for vx in the machine file…
gsketefian 3b8c582
Merge branch 'develop' into feature/vx_stage_fcst_by_platform
gsketefian c6bcff0
Undo unnecessary changes in a previous commit.
gsketefian 1cf3266
Add experiment variable TEST_VX_FCST_INPUT_BASEDIR that specifies loc…
gsketefian ea0a460
Change condition for setting default value of VX_FCST_INPUT_BASEDIR s…
gsketefian 4653165
Merge branch 'develop' into feature/vx_stage_fcst_by_platform
gsketefian 771092d
Ensure that there is a vx task entry defined for each vx task and tha…
gsketefian dfd1122
Change vx task names to better match final ones we want.
gsketefian 1ba6e28
Modify the way the metplus log files are named to be in line with wha…
gsketefian 8202526
Merge branch 'develop' into feature/rename_vx_tasks
gsketefian da84e39
Spacing edits.
gsketefian 4aa513d
Merge branch 'develop' into feature/rename_vx_tasks
gsketefian f734eb1
Change names of METplus log files so that tool names are camel case.
gsketefian 5c00438
Rename ex-scripts for vx tasks so that they're all lower case (as req…
gsketefian 3306a03
Remove extra slashes before ${SLASH_ENSMEM_SUBDIR}.
gsketefian d9e425d
Bring in changes from another branch for splitting of SFC and UPA tas…
gsketefian 6522c5b
Rename METplus configuration files for surface and upper air.
gsketefian 01eeeda
Change names of METplus conf and log files to follow a more consisten…
gsketefian df84243
Merge branch 'develop' into feature/rename_vx_tasks
gsketefian 5fb1d72
Allow some of the EnsembleStat tasks to use staged ensemble forecast …
gsketefian b5f4cc5
In the ex-scripts for deterministic PointStat verification and Ensmeb…
gsketefian 55362d0
Fix for when the value in a bash script input key-value pair contains…
gsketefian 3e68fb5
Add a bash utility script for parsing a METplus time-string template.…
gsketefian 747eb8c
Add a bash utility script for parsing a METplus time-string template.…
gsketefian 561cfa8
Add METplus variable LOG_DIR to common.conf since it is used in all M…
gsketefian 5e89a0c
Merge commit '561cfa8b4a74' into feature/split_pcpcombine
gsketefian 93c48b0
Remove debugging code.
gsketefian 68de47e
Clean up METplus conf file for APCP01h to make it more similar to the…
gsketefian 5064213
In the output of the GridStat tool for APCP01h, add to the MODEL vari…
gsketefian 91e394a
Replace GET_OBS and TN_VX with the more descriptive variables GET_OBS…
gsketefian 28e50eb
Include changes inadvertantly left out of previous commit.
gsketefian 03307b4
Include more changes inadvertantly left out of previous commit.
gsketefian c6552f1
Split calls to METplus's PcpCombine tool out of the tasks that run de…
gsketefian 167d446
Add constant SECS_PER_HOUR needed by the script eval_METplus_timestr_…
gsketefian 4ebd888
Activate the deterministic tasks in the deterministic vx WE2E tasks. …
gsketefian a67de48
Add file inadvertantly left out of previous commit.
gsketefian f8b1e28
Formatting changes and typo fixes in EnsembleStat_APCP03h.conf.
gsketefian a945fe6
Clean up comments; remove LOG_DIR from conf file since it already app…
gsketefian 3dc7469
More fixes to spacing and comments.
gsketefian ec6215e
Add the new experiment ENS_TIME_LAG_HRS to config_defaults.sh that co…
gsketefian acce7ff
Modify the way the METplus configuration variable OBS_GRID_STAT_INPUT…
gsketefian 2a29f5c
Activate both deterministic (for each ensemble member) and ensemble v…
gsketefian 784221e
Strip out the PcpCombine tool out of the METplus config files Ensembl…
gsketefian 9d0833c
Modify the METplus conf files GridStat_ensmean_APCP[03|06|24].conf to…
gsketefian 8a8780f
Re-introduce the ENS_VAR1_THRESH and accompanying variables to Ensemb…
gsketefian f482905
Modify the METplus conf files GridStat_ensprob_APCP[03|06|24].conf to…
gsketefian e268c48
Fixes to comments in EnsembleStat_APCP01h.conf so that it is more sim…
gsketefian 385b8a6
Merge branch 'develop' into feature/split_pcpcombine
gsketefian ed0712d
Remove construction and use of the variable FCST_INPUT_FN_TEMPLATE in…
gsketefian 13e7287
For all deterministic vx tasks that don't already have it, add the me…
gsketefian f50a95b
Undo all cosmetic changes to GridStat_APCP01h.conf since that's bette…
gsketefian 6a08e87
Undo all cosmetic changes to EnsembleStat_APCP01h.conf since that's b…
gsketefian 923cc3d
Changes left out of previous commit.
gsketefian 3dd08e4
Changes left out of previous commit.
gsketefian 38c87e4
Move CDATE portion of input path to EnsembleStat tasks from METplus c…
gsketefian bcf0b8f
Changes left out of previous commit.
gsketefian c247990
Changes left out of previous commit.
gsketefian 50972bf
Add auxiliary functions needed in verification.
gsketefian 1bf887a
Edit comments.
gsketefian 6fe51f8
Rearrange variable location in config_defaults.sh.
gsketefian b34c40c
For consistency with names of other entities, drop the "_apcp" from t…
gsketefian 1e115e8
Clean up comments in PcpCombine METplus conf files.
gsketefian 9ccac48
Add j-job for PcpCombine tasks (inadvertantly left out of previous co…
gsketefian 10b1f1f
Clean up comments.
gsketefian 42116e9
Remove unused variable TN from PcpCombine tasks in the ROCOTO XML; us…
gsketefian 15d8103
Add j-job for running Pb2nc tasks.
gsketefian e5b77b3
Add a METplus configuration file for the Pb2nc tool.
gsketefian 4ff49fa
Add ex-script for running Pb2nc tasks.
gsketefian 8c75544
Split out calls to Pb2nc tool into separate tasks; a lot of other cle…
gsketefian 81260fc
Merge branch 'develop' into feature/split_pb2nc
gsketefian 2d0a43e
Merge branch 'develop' into feature/split_pb2nc
gsketefian 8116077
Address PR comments.
gsketefian 8f67a95
Jinja templates for GridStat running on APCP01h, REFC, and RETOP.
gsketefian bbfd59f
Changes to ROCOTO xml to get templated versions of GridStat tasks for…
gsketefian f0c035e
Remove trailing whitespace.
gsketefian 45d64aa
Add MRMS obs file name templates.
gsketefian 1b9b3c4
Bug fixes.
gsketefian e657394
Modify deterministic PointStat tasks in ROCOTO XML and the correspond…
gsketefian 5e7c277
Clean up commented out code.
gsketefian eb4c75e
In ROCOTO template XML, modify GridStat tasks for APCP > 01 hour so t…
gsketefian ffec5ac
Minor edits to comments and spacing.
gsketefian 8110865
Modify PointStat_ensmean tasks for SFC and UPA in ROCOTO XML and corr…
gsketefian 7e3f56e
Modify GridStat_ensmean tasks for APCP 01, 03, 06, and 24 hours such …
gsketefian 6a72353
Modify the EnsembleStat METplus config files for APCP so that the nam…
gsketefian 8b86207
Change the METplus config file for the GridStat_ensmean task for REFC…
gsketefian c7cf2da
Add J-jobs for ensemble mean and probabilistic GridStat and PointStat…
gsketefian e49845f
Modify ROCOTO XML and conf files to have GridStat_ensprob tasks for a…
gsketefian ba9fa76
Modify GridStat_enspoint tasks for REFC and RETOP so they can use Jin…
gsketefian 2b9ee61
Change METplus config files for GridStat_ensprob tasks so that they'r…
gsketefian d56d07c
Modificataions to allow PointStat_ensprob tasks for SFC and UPA to us…
gsketefian 300a88b
Minor edits to comments.
gsketefian 1c95a37
Remove files that are now replaced by GridStat_ensmean_APCPgt01h.conf.
gsketefian d2a1e1b
Change the METplus config file for the GridStat_ensmean task for RETO…
gsketefian 8744564
Minor edits to comments.
gsketefian 0627d6b
Add j-job and ex-script for running EnsembleStat for APCP, all accumu…
gsketefian 3a1d8b8
Minor bug fixes.
gsketefian 834f0a7
Bug fixes to get EnsembleStat to output all the fields required by do…
gsketefian 4591b61
Edits to comments.
gsketefian dd97a28
Change METplus conf files for EnsembleStat operating on REFC and RETO…
gsketefian a153a3a
Fix up comments and spacing.
gsketefian fcabf72
Change METplus conf files for EnsembleStat operating on SFC and UPA t…
gsketefian ffb7d25
Edits to comments.
gsketefian d457ede
Merge branch 'develop' into feature/METplus_templates
gsketefian 82f709e
Delete unneeded METplus config files (replaced by generic "_gt01h" ve…
gsketefian 8b23780
Minor edits for consistency between METplus config files.
gsketefian 3da6fd1
Remove unneeded ex-scripts.
gsketefian 7575132
Remove unneeded j-jobs.
gsketefian afa4d88
Add files left out of previous commits.
gsketefian 53c9c75
Remove LOG_SUFFIX from vx ex-scripts since it is no longer used in th…
gsketefian e60213c
Modifications to get NCO mode to work with staged files.
gsketefian 5035ebd
Further fixes to make vx tasks work in NCO mode; add WE2E test for en…
gsketefian 3db65cf
Bug fix.
gsketefian 527b501
Add MET_ensemble_verification_only_vx test to set of fundamental test…
gsketefian b5e520d
Fix type to address Mike L's PR comments.
gsketefian 450b1c9
Add missing file.
gsketefian a8d43e4
Introduce a new function (set_met_tool_name.sh) that sets the names o…
gsketefian 8001b51
Rename MET/METplus name-related variables for clarity.
gsketefian ec333ba
Merge branch 'develop' into feature/METplus_templates
JeffBeck-NOAA 2a07b16
Updates feature/METplus_templates to current develop (#1)
mkavulich d16b046
For clarity, rename VX_FCST_INPUT_DIR to VX_FCST_INPUT_BASEDIR.
gsketefian 188ff28
Remove "mode=serial" attribute of metatasks since that causes a task …
gsketefian 84f98cb
The PoinStat tasks need to depend on the Pb2nc task because METplus r…
gsketefian 67fd83a
For the WE2E test MET_verification_only_vx, in the "and" clause of th…
gsketefian 1ec8bd0
Reintroduce the variable FCST_SUBDIR_TEMPLATE because it is needed wh…
gsketefian 50a5b8b
Remove variable SLASH_ENSMEM_SUBDIR_OR_NULL from default vx task and …
gsketefian 5b46af8
For the MET_ensemble_verification_only_vx task, remove dependency of …
gsketefian 5114ab8
Bug fix.
gsketefian 46b6c1a
Define ENSMEM_INDX only in those vx tasks that need it (i.e. those th…
gsketefian 0bb6aad
For the Pb2nc obs task, remove the (undefined) ensemble member name/n…
gsketefian f3254cf
Merge branch 'develop' into feature/METplus_templates
gsketefian cf33f00
Bug fixes due to inadvertently leaving out some aspects of vx in the …
gsketefian File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All of this
_sc
vs_pc
logic seems very complicated for something that doesn't seem to influence the code in any way; unless I'm missing something the "snake case" variables are only ever printed to log files. Is there some other reason I am missing?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, it's left over from the time when some things were named "grid_stat_..." and others used "GridStat_...". Let me take a look and see if I can simplify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mkavulich I cleaned up the MET/METplus naming code, hopefully to your approval. I removed the code in the J-jobs, and now the ex-scripts don't need arguments and get the variable
MET_TOOL
directly from the environment. I added a function (get_met_metplus_tool_name.sh
) that sets the names of the MET and METplus tools. I want to keep both names still because they are both currently used; maybe we'll simplify the ex-scripts later and use only one of the names to refer to the tool, but let's leave that for another PR. Also, I know we want to do all python scripts, but I haven't gotten a chance to do that yet (will do it in a future PR if someone else doesn't get to it before). Let me know what you think.Also, I reran the WE2E tests
MET_verification_only_vx
andMET_ensemble_verification_only_vx
, and the both passed.