Code

Feature #2460 allow missing input#2493

Code

Merged

georgemccabe merged 106 commits intodevelopdtcenter/METplus:developfrom feature_2460_allow_missing_inputdtcenter/METplus:feature_2460_allow_missing_inputCopy head branch name to clipboardFeb 8, 2024

+1,838-851

Commits

Commits on Jan 23, 2024

• changed template to use datetime format that works on MacOS

  georgemccabe

  committed
  294b770

• update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not e…

  Show description for f1c4bf5
  georgemccabe

  committed
  f1c4bf5

• use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line

  georgemccabe

  committed
  bec330e

Commits on Jan 24, 2024

• Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables tha…

  Show description for 5fb7984
  georgemccabe

  committed
  5fb7984

• enhanced function to support different output variable types

  georgemccabe

  committed
  895702c

Commits on Jan 25, 2024

• removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list

  georgemccabe

  committed
  059e523

• clean up formatting

  georgemccabe

  committed
  8bc5895

• per #2460, start to implement logic to prevent errors when some input files are not found

  georgemccabe

  committed
  b4fad5a

• isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent

  georgemccabe

  committed
  45bd7fe

• Merge branch 'develop' into feature_2460_allow_missing_input

  georgemccabe

  committed
  13bf32a

• remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type

  georgemccabe

  committed
  de3b4b0

• turn on use cases to test error handling

  georgemccabe

  committed
  bb82cb0

• merge artifacts

  georgemccabe

  committed
  3348536

• run only failed cases

  georgemccabe

  committed
  6152f38

• always run merge step

  georgemccabe

  committed
  5668b17

• run on a case that will succeed to test error log merge step

  georgemccabe

  committed
  1f99ba9

• only run error log merge step if there were 'Save error logs' jobs that succeeded

  georgemccabe

  committed
  5f11aff

• run cases that will fail

  georgemccabe

  committed
  307a3cb

• fix condition to merge error logs

  georgemccabe

  committed
  449f79d

• run group that will succeed but have diffs - check error logs doesn't fail, ci-run-diff

  georgemccabe

  committed
  9219402

• testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case

  georgemccabe

  committed
  041f853

• run 3 jobs, 2 should error, to confirm that error_logs is created properly

  georgemccabe

  committed
  f99cf5e

• repeat diff no error test but with ci-run-diff

  georgemccabe

  committed
  b16b4f3

• per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully

  georgemccabe

  committed
  1e05c12

• run test to confirm diff does not cause merge error logs to fail, ci-run-diff

  georgemccabe

  committed
  ff2d1ca

• Revert "run test to confirm diff does not cause merge error logs to fail, ci-run-diff"

  Show description for 5ff86b1
  georgemccabe

  committed
  5ff86b1

• run test to confirm error logs are merged properly when 2 use case groups have errors

  georgemccabe

  committed
  8106666

• try checking output variable as string instead of boolean

  georgemccabe

  committed
  93e0415

• Revert "run test to confirm error logs are merged properly when 2 use case groups have errors"

  Show description for ec4904e
  georgemccabe

  committed
  ec4904e

• run test again

  georgemccabe

  committed
  7a0a99c

• test again ci-run-diff

  georgemccabe

  committed
  5e0971e

• move check for error logs for shell script and use github env vars, ci-run-diff to test again

  georgemccabe

  committed
  01a50de

• Revert "run test again"

  Show description for 2fa17ac
  georgemccabe

  committed
  2fa17ac

• break 2 use cases to test that error logs are still created properly

  georgemccabe

  committed
  cb6d0b4

• checkout repo to get script used to merge error logs

  georgemccabe

  committed
  950d06a

• Revert "break 2 use cases to test that error logs are still created properly"

  Show description for c6c865f
  georgemccabe

  committed
  c6c865f

• test merge error log again on no error diff run ci-run-diff

  georgemccabe

  committed
  e52321a

• fix script, ci-run-diff

  georgemccabe

  committed
  bdc0877

• move merge error logic back to workflow , ci-run-diff

  georgemccabe

  committed
  bb75588

• break 2 use cases to test that error logs are still created properly

  georgemccabe

  committed
  82aa0e1

Commits on Jan 26, 2024

• Revert "break 2 use cases to test that error logs are still created properly"

  Show description for 1897da0
  georgemccabe

  committed
  1897da0

• remove testing use case group

  georgemccabe

  committed
  501b6c9

• merge changes to fix artifacts in GHA

  georgemccabe

  committed
  520fce8

• Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type"

  Show description for 7c013cc
  georgemccabe

  committed
  7c013cc

• clean up lines

  georgemccabe

  committed
  6bdd0c0

• update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files vi…

  Show description for bd5f528
  georgemccabe

  committed
  bd5f528

• remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent, ci-run-all-cases

  georgemccabe

  committed
  88cfb4d

• merge develop and resolve conflicts

  georgemccabe

  committed
  2ca78eb

• return boolean from find_input_files function to be consistent with other functions

  georgemccabe

  committed
  ef95afc

• per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs

  georgemccabe

  committed
  91e4935

• per #2460, added check to report error if allowed missing input threshold is met, ci-run-all-cases

  georgemccabe

  committed
  daac6c9

• run clear before running plot_data_plane, ci-run-all-cases

  georgemccabe

  committed
  9518732

Commits on Jan 29, 2024

• removed test group

  georgemccabe

  committed
  378e7bd

• report warning instead of error if ALLOW_MISSING_INPUTS is True

  georgemccabe

  committed
  42485d5

• cleanup

  georgemccabe

  committed
  c42869b

Commits on Jan 30, 2024

• change function to pytest fixture so it can be used by other test scripts

  georgemccabe

  committed
  7e88b34

• update ascii2nc test to process more than 1 time to ensure commands are built properly for each run

  georgemccabe

  committed
  8f2cc13

• add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat

  georgemccabe

  committed
  98751d3

• set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic

  georgemccabe

  committed
  314b343

• remove unneccesary error checking

  georgemccabe

  committed
  6a0e710

• cleanup

  georgemccabe

  committed
  e0e5359

• call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only)

  georgemccabe

  committed
  7d48a62

• add time_info to ALL_FILES dictionaries to be consistent with other wrappers

  georgemccabe

  committed
  a861f8d

• clean up logging for reporting error when missing inputs exceeds threshold

  georgemccabe

  committed
  2c0360a

• added function to get files for a single run time to be consistent with other functions

  georgemccabe

  committed
  621d24e

• skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments

  georgemccabe

  committed
  5605fd5

• added empty test files

  georgemccabe

  committed
  d54c0e6

• remove redundant variables

  georgemccabe

  committed
  23a7c04

• view warnings on a failed test run

  georgemccabe

  committed
  aa5ac23

• add more empty test files

  georgemccabe

  committed
  ebfb559

• added unit tests for missing input logic

  georgemccabe

  committed
  7051710

• remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs

  georgemccabe

  committed
  50e27b0

• cleanup

  georgemccabe

  committed
  0c075db

• remove allow missing input logic from ExtractTiles wrapper

  georgemccabe

  committed
  6b1626c

• added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat

  georgemccabe

  committed
  2c639dd

• remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored

  georgemccabe

  committed
  e3896c9

• added required boolean for input templates, ci-run-all-diff

  georgemccabe

  committed
  e98f8f9

Commits on Jan 31, 2024

• per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files

  georgemccabe

  committed
  44c828e

• added tests for missing input logic for many wrappers

  georgemccabe

  committed
  8b1049e

• cleanup

  georgemccabe

  committed
  f50c344

• fix increment of number of runs

  georgemccabe

  committed
  6e90fd0

• skip missing input logic

  georgemccabe

  committed
  fe7aebd

• change how required is handled for input templates

  georgemccabe

  committed
  56d5196

• warn instead of error if missing input is allowed

  georgemccabe

  committed
  024d9d7

• remove increment of missing input counters because it is handled in RuntimeFreq

  georgemccabe

  committed
  3664f8a

• check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead

  georgemccabe

  committed
  89feada

• added unit tests for missing input logic

  georgemccabe

  committed
  bb0e1d7

• skip missing input logic

  georgemccabe

  committed
  0daac67

• cleanup, ci-run-all-diff

  georgemccabe

  committed
  5c0de3f

Commits on Feb 1, 2024

• cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it

  georgemccabe

  committed
  8f9c72b

Commits on Feb 5, 2024

• removed file that is not needed

  georgemccabe

  committed
  2345f60

Commits on Feb 6, 2024

• added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies

  georgemccabe

  committed
  caa46a4

• warn instead of error if allowing missing inputs

  georgemccabe

  committed
  7cad798

• cleanup

  georgemccabe

  committed
  3968d9b

• implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time …

  Show description for d893308
  georgemccabe

  committed
  d893308

• added unit tests that will fail until wrapper is updated

  georgemccabe

  committed
  414b3df

Commits on Feb 7, 2024

• replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests

  georgemccabe

  committed
  52d96cd

• ci-run-all-diff

  georgemccabe

  committed
  22d9d55

• Merge branch 'develop' into feature_2460_allow_missing_input

  georgemccabe

  committed
  3d5858f

• Merge branch 'develop' into feature_2460_allow_missing_input

  georgemccabe

  committed
  d1b5485

• cleanup

  georgemccabe

  committed
  c58e7de

• Merge branch 'develop' into feature_2460_allow_missing_input

  georgemccabe

  committed
  d3c193b

• removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior

  georgemccabe

  committed
  79461ba

• added unit tests for MTD missing input checks

  georgemccabe

  committed
  0df1a0e

• per #2491, add release notes for beta3

  georgemccabe

  committed
  40af11f

• Merge branch 'develop' into feature_2460_allow_missing_input

  georgemccabe

  committed
  0cd99cd