Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3
- Loading branch information
1 parent
c67050a
commit 0f5beca