Feature #2460 allow missing input #2493
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist
… template tag to override a config variable on the command line
…list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts
…nd added optional argument to skip clearing input file list
… files are not found
…ean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent
…ermine if a python script is used with always setting file_type
… fail, ci-run-diff
…ecaus there is no truth data - to confirm that the error log merge step behaves properly in this case
…or logs if any of the 'Save error logs' steps ran successfully
…ail, ci-run-diff" This reverts commit ff2d1ca.
… case groups have errors" This reverts commit 8106666.
22 tasks
JohnHalleyGotway
approved these changes
Feb 8, 2024
There was a problem hiding this comment.
I approve of this PR, as soon as we're able to get a clean GHA run of the testing workflow that does not flag any diffs.
Note that I proposed 2 minor edits in the release notes, updating the release date and fixing a typo. I did check all the issue links and confirmed that they do work.
As for the actual code changes, @georgemccabe walked me through them, and they make sense. But with the possibility of unintended consequences, we should get a clean run before merging.
| @@ -30,6 +30,49 @@ When applicable, release notes are followed by the | |||
| `GitHub issue <https://github.com/dtcenter/METplus/issues>`__ number which | |||
| describes the bugfix, enhancement, or new feature. | |||
|
|
|||
| METplus Version 6.0.0 Beta 3 Release Notes (2024-02-07) | |||
There was a problem hiding this comment.
Suggested change
| METplus Version 6.0.0 Beta 3 Release Notes (2024-02-07) | |
| METplus Version 6.0.0 Beta 3 Release Notes (2024-02-08) |
|
|
||
| .. dropdown:: Enhancements | ||
|
|
||
| * Add suport for MET land-mask settings in Point-Stat |
There was a problem hiding this comment.
Suggested change
| * Add suport for MET land-mask settings in Point-Stat | |
| * Add support for MET land-mask settings in Point-Stat |
|
@JohnHalleyGotway, I will fix the typos in the release notes in the PR for creating the release. |
georgemccabe
added a commit
that referenced
this pull request
Feb 15, 2024
* update versions to fix bugs for METplotpy/calcpy use cases * Bugfix #2026 develop StatAnalysis looping (#2028) * Feature #2022 update python version to 3.10.4 for repo files (#2047) * Feature #2022 Use Debian 10 / Python 3.10.4 in automated tests (#2050) * add use case group name to output path so that output and error logs will contain use case category to more easily see which case was run * remove -group0 from group names for output path * add manual workflow to update truth data * ci: set gh token on checkout to enable push, ci-skip-all * Set user name/email via git config to allow push * add token to checkout * Feature #2049 Multiple interp.type.method/width (#2051) * added missing argument to script if building envs locally * get main_vX.Y truth data if running on main_vX.Y branch * Updating README.md for label creation and adding bold to common_labels.txt * Updated color for bold label * Feature #2054 consensus.write_members in TCPairs (#2057) * Create LICENSE.md (#2061) * per #2006, fix priority of obs_window config variables so that wrapper-specific version is preferred over generic OBS_WINDOW_BEGIN/END (#2062) * Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <[email protected]> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: George McCabe <[email protected]> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <[email protected]> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <[email protected]> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <[email protected]> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update github_workflow.rst --------- Co-authored-by: George McCabe <[email protected]> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
georgemccabe
added a commit
that referenced
this pull request
Mar 11, 2024
* Feature 2065 v5.1.0 beta1 (#2066) * updated version number * added release notes * Update docs/Users_Guide/release-notes.rst Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * updated version with -dev * adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <[email protected]> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: George McCabe <[email protected]> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <[email protected]> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <[email protected]> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <[email protected]> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update github_workflow.rst --------- Co-authored-by: George McCabe <[email protected]> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y --------- Co-authored-by: j-opatz <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
georgemccabe
added a commit
that referenced
this pull request
Mar 20, 2024
* adding sphinx design to 3 files so dropdown menus will work in the release notes. * fix version numbers for python requirements for building documentation * fixed tab vs spaces * Bugfix #2070 var list numeric order (#2072) * Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <[email protected]> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: George McCabe <[email protected]> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <[email protected]> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <[email protected]> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <[email protected]> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update github_workflow.rst --------- Co-authored-by: George McCabe <[email protected]> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
jprestop
added a commit
that referenced
this pull request
Mar 28, 2024
* Adding files for jet and hera * added commented process list that was accidentally removed -- this is used if the pre-processing steps are needed * Bugfix #2087 develop docs_pdf (#2091) * Feature 2076 release dropdown menu (#2083) * test dropdowns * Update release-notes.rst Attempting to resolve WARNING * testing dropdowns * adding dropdown menus * testing dropdown remove from warning section * dropdown testing * dropdown menu tests * dropdown menus after the warnings section, the dropdown menu only has one space instead of the normal 2 spaces. If it has 2 spaces, it gets sucked into the warning section. * spaces with warning * no spaces for warning * Update release-notes.rst warning note has to be indented 1 space. Originally it was no spaces but then all of the dropdown menus end up in the warning message. If it's 2 spaces, as is normal, then the warning is indented, which is unnecessary. 1 space is slightly indented and keeps the dropdown menus separate. --------- Co-authored-by: jprestop <[email protected]> * Bugfix #2096/#2098 develop - fix skip if output exists and do not error if no commands were run (#2099) * Feature 1516 use case mvmode (#2094) Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: George McCabe <[email protected]> * Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <[email protected]> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <[email protected]> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <[email protected]> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update github_workflow.rst --------- Co-authored-by: George McCabe <[email protected]> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
georgemccabe
added a commit
that referenced
this pull request
Apr 12, 2024
* Fix for Dockerfile smell DL4000 (#2112) Co-authored-by: Giovanni Rosa <[email protected]> * Changed "PROJECT" to "CYCLE" ASSIGNMENT * Feature 2115 update use cases (#2133) * Bugfix #2082 develop regrid.convert/censor_thresh/censor_val (#2140) fix #2082 main_v5.0 regrid.convert/censor_thresh/censor_val (#2101) * Bugfix #2137 develop PointStat -obs_valid_beg/end (#2141) * Feature #2143 5.1.0-beta2 release (#2144) * update release guide steps with new info based on changes to GitHub web interface * update version for development towards rc1 release * Feature #1977 ARGO use case (#2148) Co-authored-by: j-opatz <[email protected]> * Feature develop cycle change (#2150) * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * change PROJECT ASSIGNMENT to CYCLE ASSIGNMENT * Added paths to ignore --------- Co-authored-by: jprestop <[email protected]> * Documentation: Update Truth Data instructions (#2154) * upgrade python to 3.10 for documentation builds to fix failured introduced by urllib3 (see urllib3/urllib3#2168) * update workflow to update input test data to use branch from pull down or enter branch by hand if branch does not exist in dtcenter/METplus repo, e.g. if updating data for a PR coming from a fork * Bugfix #2161 develop PCPCombine additional field arguments in -subtract mode (#2162) * changed order of workflow event triggering info to more easily see the repository and commit sha that triggered the workflow * fix docker compose install and call -- needed due to new Alpine Linux release 3.18.0 that does not include docker-compose in the apk repository * Bugfix #2168 develop - StatAnalysis time shift (#2169) * Feature 1978 update pyreq doc (#2177) * Updating for 5.1.0 * Adding greater than or equal to signs * Feature 2106 release support (#2182) * Added a paragraph for Release Support Policy * Added a sentence about contacting the team through the METplus GitHub Discussions Forum. * Added Release Support Policy info * Per #2159, modify instructions to reflect that we only want a coordinated release for the official releases and updates to the coordinated release section for the bugfix releases. (#2183) * Feature 2147 workflow doc (#2185) * Add new images for github workflow * Added PR section * Modified the Find the GitHub Issue to be Find or Create a GitHub issue * Trying out image keyword compared with figure keyword * Trying out various widths * Added new figures * Reverting to figures from images after modifying figures * Adding the old images which are better quality * Trying out new indention * Working on formatting again * Reducing image widths * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Contributors_Guide/github_workflow.rst Co-authored-by: George McCabe <[email protected]> * Update github_workflow.rst --------- Co-authored-by: George McCabe <[email protected]> * added scripts to create new conda environment for PANDA-C use cases * Bugfix #2189 develop - spaces in complex thresholds (#2191) * added release notes for completed issues for rc1 * Feature 2011 tci from cesm fluxnet2015 netcdf (#2192) Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2179 develop TCPairs fix -diag argument (#2187) * sort list of files to make command more readable and make logic more easily testable * fix unit test to expect alphabetical order of files * Feature #2197 MODE new multi-variate options (#2203) * added to release notes * added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 --------- Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: Giovanni Rosa <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: mrinalbiswas <[email protected]> Co-authored-by: Mrinal Biswas <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
DanielAdriaansen
added a commit
that referenced
this pull request
Apr 17, 2024
* added release notes for other items completed for rc1 release * Feature #1626 TCDiag Wrapper (#2201) * Feature #1610 first round of SonarQube fixes (#2200) * Feature #2195 create v5.1.0-rc1 release (#2204) * Update version for development towards official release * updated script to use metplotpy as base and install pygrib because conda was failing trying to install matplotlib in Docker for some strange reason * Feature 2188 use case panda c (#2202) * added first use case, need to finish docs and add to list * added documentation files, 3 new cases * added 5th use case, docs * final use case, corrected paths for output * Update GridStat_fcstGFS_obsERA5_lowAndTotalCloudFrac.py * removed chars, added space to last section * turn off 5/6 new air quality and comp use cases because they are causing the disk to fill up in the automated tests and only 1 is needed to test the new behavior. Reordered use case groups to be alphabetical * disabled other new air quality use case because the disk is still filling up from this case * Feature 2136 cross spectra (#2208) * per #2212, continue instead of return so other forecast leads will not be skipped * Feature #2210 MODE multi-variate variable rename (#2211) * Bugfix diff tests (#2217) * added change from NCEP produtil that was somehow not included in the changes pulled from their repo before starting SonarQube work * Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]>
georgemccabe
added a commit
that referenced
this pull request
May 17, 2024
* Feature #2228 v5.1.0-rc2 release (#2229) * update version after rc2 release to note development towards official 5.1.0 release * Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
georgemccabe
added a commit
that referenced
this pull request
May 22, 2024
* Feature #2188 move PANDA-C cases to clouds category and fix automated tests (#2237) Co-authored-by: j-opatz <[email protected]> * Feature 2198 use case amdar pbl (#2206) Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Bugfix #2235 rename multivar_itensity to multivar_intensity_flag (#2236) * Feature #2230 MvMODE usecase doc updates (#2239) This updates the use-case documentation to go along with the multivariate MODE enhancements. * combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jason-english <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
georgemccabe
added a commit
that referenced
this pull request
May 23, 2024
* combine release notes from development releases for 5.1.0 release * fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * added entry to update truth change log: develop dtcenter/MET#2892 and dtcenter/MET#2899 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
georgemccabe
added a commit
that referenced
this pull request
May 23, 2024
* fix indentation to prevent incorrect nesting of drop down menus * Bugfix #2241 Create directory containing -out_stat file (#2242) * Enhance diff logic to control decimal precision to prevent failures (#2243) * Bugfix #2245 use unique run ID to name logger instance (#2247) * per #2245, add METplusConfig class function that is called when object is deleted to close log handlers. This prevents OSError: [Errno 24] Too many open files from running all pytests * Feature #1626 TCDiag updates to wrapper and basic use case (#2248) Co-authored-by: Jonathan Vigh <[email protected]> * move new use case into another group * Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * added entry to update truth change log: develop No PR --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Jonathan Vigh <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
georgemccabe
added a commit
that referenced
this pull request
Jun 12, 2024
* Bugfix #2244 develop fix diff tests (#2254) * skip diff for specific png images that produce diffs occasionally but look the same visually * add 2 more keywords for use cases that occasionally flag small, unnoticeable diffs in png images * Feature #2253 update tests for diff_util (#2256) Co-authored-by: root <root@localhost> * Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * added entry to update truth change log: develop #2610 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: root <root@localhost> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
jprestop
added a commit
that referenced
this pull request
Jun 28, 2024
* Feature #1974 Documentation: update coding standards section (#2260) * Feature #2253 conftest fixture to set pytest tmpdir (#2261) * Feature #2151 Documentation: Add quick search keywords METplotpy/calcpy use cases (#2259) * Feature #2138 CI enhance MET Docker tag override (#2258) * Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * added entry to update truth change log: develop dtcenter/MET#2921 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
JohnHalleyGotway
added a commit
that referenced
this pull request
Aug 1, 2024
* Updating development instructions. * fixed typo * fixed another typo * Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * added entry to update truth change log: develop dtcenter/MET#2939 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
jprestop
added a commit
that referenced
this pull request
Aug 22, 2024
* Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * added entry to update truth change log: develop #2657 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
JohnHalleyGotway
added a commit
that referenced
this pull request
Aug 30, 2024
* Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * added entry to update truth change log: develop dtcenter/MET#2951 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
jprestop
added a commit
that referenced
this pull request
Sep 5, 2024
* Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * added entry to update truth change log: develop dtcenter/MET#2959 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
JohnHalleyGotway
added a commit
that referenced
this pull request
Sep 10, 2024
* Feature #2207 Documentation - Updates to Contributor's Guide (#2263) Co-authored-by: Dan Adriaansen <[email protected]> * Feature develop update install files (#2266) * Updating installation files * Removing run_commands.sh * Updated the file to add a MINICONDA_PATH * Adding files for jet * Adding orion file * Removing beta references * Corrected met version * Adding files for WCOSS2 machines * Removing rc1 from acorn file * Removing beta1 * Bugfix #1853 develop - PointStat don't require mask variables to be set (#2262) * update version of certifi to match auto-generated dependabot PR #2267 * feature #2253 more tests for diff_util.py (#2264) * update versions of METplus components for the next official release * updated release notes for 5.1.0 release * removed duplicate entries in release notes * Added EC2 instance recipee with S3 mounting ability (#2269) * Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in pangeo-data/xESMF#246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1ca. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b4. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e1. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b0. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809d. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f3) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe822. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- #2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * added entry to update truth change log: develop dtcenter/MET#2967 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: Dan Adriaansen <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
georgemccabe
added a commit
that referenced
this pull request
Sep 26, 2024
* Updating the Subsetting Tests by Marker section (#2272) * update release date for 5.1.0 release (#2276) * update version for development towards 6.0.0-beta1 release * added new use case that was missing * Bugfix #2279 develop - buoy station file from 2022 (#2280) * Feature 2253 series analysis tests (#2277) * Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in https://github.com/pangeo-data/xESMF/issues/246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1cac57c431a047ee250e9dae9b0a813a78ba. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666a73685e654e0146d4fed56f2382f1bfc7. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c6e7031c5dafb1177d4b4ca3f32a999dac. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b46db353b4b4709183be2fe7e5ce64ff5ff. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e11096aace3ccc2c79cd631533fc6426900. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b03a45bb871c71e770ff9e602739d6b63d5. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6b0b351ab1b4bca5b563c1f5beef7737a9. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4f0d7851bea720a50fac6011cd381017dc. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809dc3bd73ace969b046062967796035f4d86. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f33ab152a0b254041961a41895dde614ae0) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe8226c58e9a028a7979664bbf224296fdd6ea. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <[email protected]> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <[email protected]> * Updated language for tar file Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * added entry to update truth change log: develop #2701 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
georgemccabe
added a commit
that referenced
this pull request
Oct 14, 2024
* Fix typo in pull_request_template.md * Make code coverage report available (#2287) * Use updated dtcenter/metplus-action-data-update tag that fixes bug that causes GHA disk to exceed capacity when too many data volumes are created. Use specific commit of coveralls python action to ensure it will not change * removed climatology keyword from use cases that aren't relevant that were accidentally added with PR #1984 * update readme with current information * Feature #2282 coord_release_docs (#2288) Co-authored-by: George McCabe <[email protected]> * continue workflow if coveralls step fails -- this typically happens on a pull request from a fork because the authentication fails * Feature 2253 system util tests (#2297) * Feature #2294 LSR use case (#2295) * Feature 2253 run util tests (#2307) * Release Guide - remove beta/rc tags/releases (#2309) * Add 'component: repository maintenance' label. Already ran the script to push this change and the NOAA OAR reporting label to all the METplus repos. * Hotfix for labels, adding a new one for 'component: input data' and also fixing the get_lablels.sh to search for up to 200 existing labels. Also work on the log messages. * Fix typo in comment * Feature #2283 time looping consolidation (#2311) Co-authored-by: John Halley Gotway <[email protected]> * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for run_util, 'Usage' bugfix (#2313) * Feature #2338 Debian 12 Conda environments (#2341) Co-authored-by: John Halley Gotway <[email protected]> * Feature 1488 usecase satwinds (#2324) Co-authored-by: George McCabe <[email protected]> * Feature #2283 include times (#2345) Co-authored-by: John Halley Gotway <[email protected]> * feature #2253 tests for config_validate, mock logger in conftest (#2320) * Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter (#2347) Co-authored-by: John Halley Gotway <[email protected]> * remove MODEL and OBTYPE to use defaults from MET config * Feature #2348 v6.0.0-beta1 release (#2351) * update version for development towards 6.0.0-beta2 release * copy libGL and libEGL files into docker image to prevent errors with geovista python package * run use that uses geovista to test * Feature #2156 release_acceptance_testing (#2352) * feature #2253 print error logs from conftest (#2358) * feature #2253 met_db_load tests (#2355) * Rename 5.1.0.lua_wcoss2 to 6.0.0.lua_wcoss2 * Update and rename 5.1.0_acorn to 6.0.0_acorn * Update 6.0.0_acorn * feature #2253 add tests for gfdl_tracker (#2354) Co-authored-by: George McCabe <[email protected]> * Create 6.0.0_gaea * Update and rename 5.1.0_hera to 6.0.0_hera * Feature #2156 release_acceptance_testing take2 (#2361) Co-authored-by: lisagoodrich <[email protected]> * Feature #2329 Docker info in Installation Chapter (#2366) * feature_2253_extract_tiles_tests (#2368) * Feature 2253 tc pairs tests (#2369) * Feature 2253 tc csv writer (#2373) * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * Feature #2340 TCDiag one_time_per_file_flag (#2374) * Update and rename 5.1.0_jet to 6.0.0_jet * Added libssh * prune docker files before running use cases to prevent running out of disk space when running use cases, ci-run-all-diff * Feature 2253 command builder tests (#2378) * Feature 2253 series analysis test (#2380) * Feature 2253 py embed test (#2379) * ignore cyclone plotter from code coverage report because it will be replaced with METplotpy logic * Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in https://github.com/pangeo-data/xESMF/issues/246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1cac57c431a047ee250e9dae9b0a813a78ba. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666a73685e654e0146d4fed56f2382f1bfc7. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c6e7031c5dafb1177d4b4ca3f32a999dac. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b46db353b4b4709183be2fe7e5ce64ff5ff. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e11096aace3ccc2c79cd631533fc6426900. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b03a45bb871c71e770ff9e602739d6b63d5. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6b0b351ab1b4bca5b563c1f5beef7737a9. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4f0d7851bea720a50fac6011cd381017dc. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809dc3bd73ace969b046062967796035f4d86. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f33ab152a0b254041961a41895dde614ae0) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe8226c58e9a028a7979664bbf224296fdd6ea. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <[email protected]> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <[email protected]> * Updated language for tar file Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * Bugfix #2705 develop - SeriesAnalysis fix time substitution into field info (#2709) * fix SonarQube complaints * Per #2705, set correct time information in field info by using the time info used to find the input files (self.c_dict['ALL_FILES']) instead of relying on parsing the time info from the other input files (fcst vs. obs) * clean up unused code after bugfix for #2705 * Feature 2647 use case mv mode for rrfs (#2706) * establish documentation, need to fill out with provided info * updated documentation, needs to have img path and output path fixed * updated output strings, still need to update pic link * removing pound sign to render correctly * Revert "removing pound sign to render correctly" This reverts commit ffe607a96fd33dc5ba15a479f76778dcba7e9656. * targeted removal of pound sign with bullet points section. * Per #2647, attempt to fix syntax error * Per #2647, adding back contents with formatting changes to test if it works * Per #2647, adding back contents in a different location to try to resolve syntax error * Per #2647, testing pipe for horizontal scroll bar * Per #2647, removed the pipe symbol and are attempting to fix syntax error again * Per #2647, reverting to original failed attempt of a table of contents * Per #2647, trying new formatting * Per #2647, fixing new formatting * Per #2647, attempt formatting changes * Per #2647, reverting back to original state * added to automated testing, updated docs, config file * added docs back in * added image, new quick search term, disabled auto tests * testing use_case_groups.json to see if/how this fails * Updated to disable use_case_groups.json run * added use case to list of exceeding disk space --------- Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Bugfix 2675 move to pyproject.toml (#2699) * 2675: move to pyproject.toml * 2675: update toml, remove setup.py ref * bugfix 2675: reinstate VERSION file * bugfix 2675: hardcode version in pyproject.toml * use relative versions * slightly modify import to make it easier to move produtil into metplus * read version from version file * disable install of run_metplus script since it is broken * only install metplus and sub-packages, add version files to package data (parm cannot be added because it is non-package data) * add newline back * install produtil package so metplus can be imported in python * remove unused import * move produtil under metplus to differentiate it from the full NCEP produtil package and isolate it to metplus when installing metplus as a package -- previously produtil would be installed as its own package when metplus is installed * fixed more imports * Create links for parm directory and scripts/run_metplus.py under metplus so the parm files and run script will be installed as part of the metplus python package. Modified config_metplus to find the parm directory from under metplus to be consistent between running via python package and via scripts * Exclude parm from sonar sources because it will be found under metplus. Add metplus/parm/** to list of coverage exclusions * fixed more imports * add parm back to list of sonar sources --------- Co-authored-by: George McCabe <[email protected]> * Removing stated location of the version selector menu * Feature 2489 update docs (#2717) * adding new Literal Include section * under bold section adding "Environment" to Variables * Adding Substitution References section * removing blank space * fixing code typos * another typo * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2540 Point2Grid config file (#2580) * per #2540, add support for setting values in a config file for Point2Grid wrapper -- removed -qc flag argument in favor of setting obs_quality_inc in the config file * Added support for setting time_offset_warning now that the changes for other wrappers was merged into develop. Also added support for MET_CONFIG_OVERRIDES for Point2Grid * per #2540, set -goes_qc command line argument from POINT2GRID_GOES_QC_FLAGS and continue to support legacy POINT2GRID_QC_FLAGS. Other fixes to resolve incorrectly resolved conflicts from previous commit merging develop into branch * per #2540, remove quality_mark_thresh from config file because it has been removed from MET point2grid * Per discussion in PR #2580, remove time_offset_warning support from tools that don't actually use the value -- to add support for a new wrapper, update metplus/util/constants.py TIME_OFFSET_WARNING_WRAPPERS to include the wrapper's name * removed env vars for time_offset_warning for wrappers that no longer set it * added entry to update truth change log: develop dtcenter/MET#2988 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: metplus-bot <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: George McCabe <[email protected]> Co-authored-by: John Sharples <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: jprestop <[email protected]> Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: reza-armuei <[email protected]> Co-authored-by: Tracy Hertneky <[email protected]> Co-authored-by: Tracy <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: Mallory Row <[email protected]> Co-authored-by: j-opatz <[email protected]> Co-authored-by: bikegeek <[email protected]> Co-authored-by: Christina Kalb <[email protected]> Co-authored-by: Hank Fisher <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lisa Goodrich <[email protected]>
JohnHalleyGotway
added a commit
that referenced
this pull request
Dec 3, 2024
* Feature 898 headers (#2389) * changing _ to - for header consistency * changing _ to - for header consistency * updating all headers for consistency * updating all headers for consistency and adding spacing The spacing doesn't seem to be required but it is the convention that we follow for headers. * updating all headers for consistency * updating headers for consistency and adding capital letters to headers * Using the overline ### to keep index consistent with other indexes * updating all headers for consistency * update requirements for building docs to get around build failure where python packages required to build RTD have disappeared from the automatically generated commands from RTD * updating all headers and some spacing for consistency * updating headers for consistency * changing to ### for consistency * Per #898, fixed the headers for the Release Guide part, chapters, and sections. * Duplicating changes in develop branch for requirements.txt file * updating headers * Per #2669, updated header formatting to resolve errors * Per #2669, udpating header * Per #2669, udpating headers * Per #2669, udpating header * Per #2669, updated header formatting * Per #2669, update header formatting * updating headers * Per #898, removed space in front of title * Capitalizing Please * changing to just italics to match standard formatting * indenting for consistent formatting * fixing italics again * changing from note to warning for consistency * updating headers, adding some capitalizing to the headers * fixing typo Co-authored-by: George McCabe <[email protected]> * This file was committed but never updated or used Per Minna, ok to delete. * Restructuring table of contents to make it more clear which guides are for users and which are for contributors * fixing formatting for clairity Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2349 upgrade instructions for deprecated MET config files (#2386) * Feature 2123 di doc update (#2392) Co-authored-by: Tracy <[email protected]> * change log to list name of config variable , e.g. GRID_STAT_CONFIG_FILE, to easily see which variable should be removed * Major refactor including elimination of unnecessary imports, only computing the index for the season requested instead of all seasons all the time, which also fixes a bug selecting which season the user requested. Results are identical for all seasons for the test data for the use case. * Feature 1667 doc conventions (#2397) * New additions based on the old branch feature_1667_updating_overview * Moving the release-notes reference label * Added label for METplus_glossary for use in Documentation conentions section. * Adding images for the Documentation Conventions section * Modifying wording and testing formatting in Internal Links section * Second take on formatting * Third attempt at formatting * Fourth attempt at formatting * Modified wording, sections, and formatting * Minor modifications * Added period * Changed Pretty Table to PrettyTable * Modify informationg about converting an existing table and adding images * Resolving errors * Reformatting * Moving placement of reference label * Attempting to fix table title * Fixed incorrect alignment * Made changes based on Lisa's suggestions * Made changes based on Lisa's suggestions * Made corrections * Made corrections * Per #1667, fixing typos * Per #1667, corrected text --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2377 Log to terminal only (#2398) * Update conda envs to use conda-forge only (#2402) * rearrange MET config env var tables for GridStat so they are in the order that they appear in the wrapped MET config file * use mamba instead of conda and update version of netcdf python package used to 1.6.2 * skip s2s_mid_lat WeatherRegime use case that produces different results often * updated version of xesmf because <0.7.1 does not work with mamba * per #2412, fix bug in GenVxMask to put quotes around argument so a grid specification string with spaces will be an accepted value * downgrade version of esmf package to fix bug described in https://github.com/pangeo-data/xESMF/issues/246 * Feature #2219 SeriesAnalysis multiple input files (#2408) * Adding 3 new requestors to the list of common_labels.txt for NOAA/NHC, CSU/CIRA, and Australian BOM ci-skip-all * Feature 2405 match tables2wrapper (#2416) Co-authored-by: George McCabe <[email protected]> * per #2423, follow symbolic links when traversing directories to find files within a time window, ci-run-all-diff * Feature #2252 WaveletStat wrapper (#2427) Co-authored-by: j-opatz <[email protected]> * add WaveletStat use case to group and temporarily disable TCDiag use case because changes to the wrapper are needed to fix it after changes to the MET tool were merged * update version number for beta2 release (#2431) * update version for dev towards beta3 * Feature #2371 Embed use case upgrade demo video into upgrade instructions (#2444) * fix failing use case tests -- install python packages dateutil and requests via apk instead of pip because the pip commands began failing -- see PEP668 https://peps.python.org/pep-0668/ -- also changed scripts to create conda environments for use case tests to install all packages in a single call to mamba instead of individual calls * remove commands to install geovista from source because it is now installed from conda * Feature #1989: Add OMI use case (#2457) Co-authored-by: George McCabe <[email protected]> * Feature #2432 TCDiag new config changes (#2453) * move medium range 10 use case into group with 3-5 to speed up runs * Feature #2334 land_mask (and topo_mask) in PointStat (#2456) * added use cases with pygrib * Feature #2430 TCPairs consensus.diag_required and consensus.diag_min_req (#2439) * Quickfix cloud use case desc imgs (#2464) * added pics, updated desc * add last two imgs * Fixing spelling and capitalization * Feature 2454 doc overview conv (#2471) * adding documentation in different sections * adding grid table section * fixing links * grammar updates * Per #2454, updated sections and wording. * Per #2454, added a period to the end of a sentence. * Per #2454, fixing formatting * Per #2454, updating wording * adding a section for line breaks in a table * adding :code: information * trying to fix warning * take 2 * take 3 or 4 * maybe fixed * updating link * fixing web link again * web link saga continues * Changed "ReadTheDocs" to "Read the Docs" * Updated "main_v" references to be "main_v12.0" * Removed references to main_v*, replacing with raw RST It is not maintainable to have links to branches which will become old. Since we can avoid it by adding the raw RST in the documentation, I have removed all references to main_v* in favor of placing the raw RST in the documentation. * Modified the "Code in a Paragraph" section * Reworded for consistency within the document * Added back the link for Sphinx code blocks --------- Co-authored-by: Julie Prestopnik <[email protected]> * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 GHA Node20 deprecation warnings (#2473) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs * change arguments to workflow dispatch so they are no longer required -- these are not needed to be set when triggering by hand through the web interface * Feature dtcenter/MET#2796 develop - Fix error log artifact creation (#2475) * updated version of pillow to fix security vulnerability alerted by dependabot in PR #2477 * remove docker image after runtime image is created from metplus image and conda env image * turn on use case to test image removal * prune images if image tag doesn't exist -- it appears that if the image is built on the fly (when PR is coming from fork) then the tag exists, but if not, the image tag is set to <none> * support commands that must run in the shell to see if || will work in docker image pruning step * try to fix image removal * Feature 2383 use case sat alt (#2480) * new docs, files for use case * new files * updating to run use case * updated python libraries, changed test env * trying new point logic * added to script for nan removal * redid Python script to take adv of new MET ability for nans * Update run status * removed unused settings * run image prune commands separately * changed shell back to false * split up use case groups so the same envs are used by a group to see if that resolves the disk space issues * turn off use cases * feature 2253 fix empty pytest logs (#2485) * added more commands to free up disk space as suggested in https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh, ci-run-all-cases * Feature 2406 redo usecase rrfs (#2488) * issue #2406 RRFS use case files * issue #2406 added usecase to tests * Issue #2406 added metplotpy and metcalcpy as dependencies * Feature #2460 allow missing input (#2493) * changed template to use datetime format that works on MacOS * update logic to only write a file list file if there are more than 1 files, updated unit tests to match new behavior, added exception handling to series analysis to prevent crash if file does not exist * use getraw instead of getstr to prevent crash if providing a filename template tag to override a config variable on the command line * Add optional argument to subset file function to always write a file list text file even if there is only 1 file found. Use this argument in UserScript wrapper so that the environment variables that contain paths to file list files are consistent in format for use in user scripts * enhanced function to support different output variable types * removed the need for overriding clear function in specific wrappers and added optional argument to skip clearing input file list * clean up formatting * per #2460, start to implement logic to prevent errors when some input files are not found * isolate logic to find input files into find_input_files functions. clean up those functions to return boolean instead of sometimes returning None or a list of files to be consistent * remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type * turn on use cases to test error handling * merge artifacts * run only failed cases * always run merge step * run on a case that will succeed to test error log merge step * only run error log merge step if there were 'Save error logs' jobs that succeeded * run cases that will fail * fix condition to merge error logs * run group that will succeed but have diffs - check error logs doesn't fail * testing - add use case group that will succeed but will cause diffs becaus there is no truth data - to confirm that the error log merge step behaves properly in this case * run 3 jobs, 2 should error, to confirm that error_logs is created properly * repeat diff no error test but with * per dtcenter/MET#2796, fix error log artifact creation by merging error logs if any of the 'Save error logs' steps ran successfully * run test to confirm diff does not cause merge error logs to fail * Revert "run test to confirm diff does not cause merge error logs to fail" This reverts commit ff2d1cac57c431a047ee250e9dae9b0a813a78ba. * run test to confirm error logs are merged properly when 2 use case groups have errors * try checking output variable as string instead of boolean * Revert "run test to confirm error logs are merged properly when 2 use case groups have errors" This reverts commit 8106666a73685e654e0146d4fed56f2382f1bfc7. * run test again * test again * move check for error logs for shell script and use github env vars * Revert "run test again" This reverts commit 7a0a99c6e7031c5dafb1177d4b4ca3f32a999dac. * break 2 use cases to test that error logs are still created properly * checkout repo to get script used to merge error logs * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit cb6d0b46db353b4b4709183be2fe7e5ce64ff5ff. * test merge error log again on no error diff run * fix script * move merge error logic back to workflow * break 2 use cases to test that error logs are still created properly * Revert "break 2 use cases to test that error logs are still created properly" This reverts commit 82aa0e11096aace3ccc2c79cd631533fc6426900. * remove testing use case group * Revert "remove python embedding checks because MET is now smart enough to determine if a python script is used with always setting file_type" This reverts commit de3b4b03a45bb871c71e770ff9e602739d6b63d5. * clean up lines * update logic to check that python embedding is set up properly to only try to set file_type automatically if it is not already set and if the wrapper is a tool that supports multiple input files via python embedding (which require file_type to be set). also changed error if not set properly to warning and use PYTHON_NUMPY as a default * remove run_count increment before run_at_time_once - set closer to find_input_files so run count and missing input count are consistent * return boolean from find_input_files function to be consistent with other functions * per #2460, warn instead of error if missing inputs are allowed, track counters for number of runs and missing inputs * per #2460, added check to report error if allowed missing input threshold is met * run clear before running plot_data_plane * removed test group * report warning instead of error if ALLOW_MISSING_INPUTS is True * cleanup * change function to pytest fixture so it can be used by other test scripts * update ascii2nc test to process more than 1 time to ensure commands are built properly for each run * add unit tests to ensure missing input file logic works properly for ascii2nc and grid_stat * set variable to skip RuntimeFreq logic to find input files to prevent duplicate increment of run_count -- these will be removed when the wrapper has been updated to find files using RuntimeFreq logic * remove unneccesary error checking * cleanup * call function to handle input templates that need to be handled separately for each item in the comma-separated list (for UserScript and GridDiag only) * add time_info to ALL_FILES dictionaries to be consistent with other wrappers * clean up logging for reporting error when missing inputs exceeds threshold * added function to get files for a single run time to be consistent with other functions * skip increment of run_count when FIND_FILES=True and RuntimeFreq input file logic is skipped to prevent duplicate increments * added empty test files * remove redundant variables * view warnings on a failed test run * add more empty test files * added unit tests for missing input logic * remove MANDATORY setting for EnsembleStat and GenEnsProd and instead pass mandatory argument to call to find model files so warnings/errors are properly displayed for other inputs * cleanup * remove allow missing input logic from ExtractTiles wrapper * added functions to parse template/dir variables from config, removed explicit calls to read those variables from GridStat * remove error if more labels than inputs are provided (for UserScript and GridDiag only) -- extra labels will just be ignored * added required boolean for input templates * per #2460, change warning messages to debug when checking a list of DA offsets since it is common that a given offset will not always be found in the files * added tests for missing input logic for many wrappers * cleanup * fix increment of number of runs * skip missing input logic * change how required is handled for input templates * warn instead of error if missing input is allowed * remove increment of missing input counters because it is handled in RuntimeFreq * check status of input files and increment counters in overridden run_once_per_lead. remove increment of missing input counters because it is handled in run_once_per_lead * added unit tests for missing input logic * skip missing input logic * cleanup * cleanup, use fixture for tests, add unit tests for missing input, bypass missing input logic on wrappers that don't need it * removed file that is not needed * added unit tests for pb2nc to test -valid_beg/end arguments and changes to properly support any runtime frequencies * warn instead of error if allowing missing inputs * cleanup * implement changes to properly support all runtime frequencies for pb2nc. previously all files that match a wildcard will be used instead of selecting only files that fall within the specified time range. some functions moved into pb2nc wrapper will eventually be moved up so that they are used by all wrappers to be consistent * added unit tests that will fail until wrapper is updated * replace functions in RuntimeFreq wrapper used to find input files so they can be used by all wrappers, updated ioda2nc wrapper to find input files properly to fix tests * cleanup * removed mtd version of get_input_templates and added logic to RuntimeFreq's version to get the same behavior * added unit tests for MTD missing input checks * per #2491, add release notes for beta3 * Feature #2491 v6.0.0 beta3 (#2495) * update version for beta3 release * fixed typos in release notes * update version to note development towards beta4 release * Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update * added quotes to prevent error in echo caused by parenthesis * fix incorrect command * Revert "fix incorrect command" This reverts commit e7dffb6b0b351ab1b4bca5b563c1f5beef7737a9. * Revert "added quotes to prevent error in echo caused by parenthesis" This reverts commit c1cb3c4f0d7851bea720a50fac6011cd381017dc. * Revert "Per suggestion from @JohnHalleyGotway, create intermediate branch for updating truth data to avoid branch protection rules. I added a step to delete the intermediate branch locally if it exists to prevent conflicts with the update" This reverts commit 525809dc3bd73ace969b046062967796035f4d86. * Hotfix: Allow symbolic link to run_metplus.py to run (#2500) * Adding use case tests * Changing test environment * Testing environment changes * Documentation update * Updating Documentation * Updating documentation for disk space failure * Added new use case category * Fixing use case test * Fixing bug in use case file * Testing s2s after data removal * add back use cases that were accidentally removed * fix incorrect use case added * Setting tests to false for merge * Removes extraneous imports. * Switches to function call for the coupling index. * Correct number of args in comment. * Testing for old use cases * Setting tests to false for merge * update tests to update develop data -- modified commands to create new use case category directory if it does not already exist, move step to remove old data to be completed just after new data is copied to vX.Y * Summation has to have a dimension supplied for the gridded data, but for pandas the only dimension is time (but it is un-named). Therefore the numerator for the covariance term had to be split out between the fcst and obs case. * Feature 2463 modify table (#2508) * creating test dropdown menus * fixing warnings * fixing warnings * fixing warnings 3 * fixing warnings 4 * Attempt to fix documentation errors * adding 2 more test dropdowns please note. There is still a message about WARNING: Duplicate explicit target name: "gridstat: cloud fractions with neighborhood and probabilities (pygrib)". John O will fix this. I should not touch it. * fixing spacing * trying to fix link * take 2 * Removing double underscores added earlier * moving dropdown menus * Adding version to dropdown menu title * fixing spacing * dropdowns date util, eofs, h5py * fixing formatting * fixing formatting * Per #2463, adding template for future entries * adding imageio, lxml & matplotlib * dropdown up to nc-time-axis * fixing spacing problems * Fixing broken s2s links and other incorrect links * Fixing spelling and capitalization * Removing the dash in front of 1.4 for nc-time-axis * Modifying formatting * adding dropdowns thru pylab * fixing problems * dropdowns thru scikit-learn * fixing spacing * final dropdowns thru yaml * fixing spacing * fixing loose ends * Per #2463, moving information to an Appendix and adding text and links in an overview * Per #2463, adding to index.rst * Per #2463, reworded language and updated Python 3.8 reference * Per #2463, fixing errors * Per #2463, made updates based on feedback at the METplus Engineering meeting. * removing tables, changing most METplus wrappers, version numbers. * trying to fix met_version * Per #2463, adding necessary code for substitutions * Per #2463, fixing syntax error * adding period * removing section 1.5 * Per #2463, replace old label reference with new label reference and updated text to reflect the move to drop down menus * read python version from file to replace in docs * fixed typo in variable name * adding python_version to the overview. --------- Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: George McCabe <[email protected]> * per #2509, automate MET version used in documentation to be X+6.Y.0 of METplus version * Bugfix #2520 ASCII2NC file window issue and redundant wrapper initialization (#2522) * per PyCharm documentation, only ignore workspace.xml idea file and commit the rest of the .idea files to version control * per #2520, create function to get METplus wrapper class without initializing it and use that function to read list of deprecated env vars to prevent redundant initialization of wrappers that can cause unintended side effects * handle file_window variables consistently by using CommandBuilder function * comment out optional config variables that previously caused a failure when unset * Update update_truth.md Fix typo in the update_truth issue template in the develop branch. * Update update_truth.md Update wording in the update_truth issue template. * Feature #2530 dev timeline (#2532) * Per #2530, add a development timeline to the METplus Release Information section of the User's Guide. Also update the Release Guide instructions. * Per #2530, tweak the wording. * Update docs/Release_Guide/release_steps/update_release_notes_development.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Adds static station lookup file for use with Python embedding for FLUXNET observations. * Major overhaul to forecast Python embedding script for the TCI use case. * Major overhaul to observation Python embedding script for the TCI use case, to compute TCI from raw observations rather than read pre-computed TCI. * Updates documentation file for TCI use case. * Adds METcalcpy version number. * Refactors wording and fixes typo. * Fixes RST formatting. * Finally fixed RST error. * Adds support to remove leap days if requested. * Updates command line args for Python embedding scripts. * Feature #2537 develop sonarqube_gha (#2541) * Per #2537, add SonarQube workflow for METplus * Per #2537, update nightly build email list. * Per #2537, fix cut/paste error configure_sonarqube.sh * Per #2537, exclude test code from code coverage statistics. * Updated conf file for use case. * Removes new TCI function because it is in METcalcpy now. * Removes old code, somsome reorganization and clarification and setting of params, and also switches the fluxnet metadata file to a command line argument instead of an environment variable. * Update the 6.0.0 Coordinated Release development timeline in release-notes.rst * Support for environment variables or default options for filtering and filename patterns, DEBUG mode added and set to False by default, adjustment of print statements for logging, and refactoring filtering of stations to ensure we don't process a file that we shouldn't by better coupling of filenames and stations. * Makes DEBUG an env var for config via metplus wrappers. * Reorganization of config file, adds environment variables, and updates comments for use case changes. * Updates to documentation. * Fixes tables. * Adds table of contents to the top for users to click on. * Updates use case documentation file. * Updated config file with obs and fcst subirectories in the path. * Added optional key/value to use_case_groups.json to prevent a use case group from running to easily disable it temporarily. Disable short_range:14 use case until it can be fixed with #2551 * update pillow version based on recommendation from dependabot: https://github.com/dtcenter/METplus/security/dependabot/5 * Switches to using metplotpy_env to get metcalcpy dependency. * Adds filtering based on missing data values. * Finishing touches to debug statements for testing. * Fixing a few minor code smells from last week. * update link to METplus Components Python Requirements table in PR template * Update docs/use_cases/model_applications/land_surface/PointStat_fcstCESM_obsFLUXNET2015_TCI.py Co-authored-by: George McCabe <[email protected]> * Feature #2555 v6.0.0 beta4 (#2556) * update version for beta4 release * added release notes for beta4 release * update version for development towards beta5 release * update location of METviewer docker-compose.yml file that moved from PR dtcenter/METviewer#525 * Feature #2537 develop single_sq_project (#2558) * Update the beta4 release date wording * Feature #2433 Ugrid config variables in GridStat/PointStat (#2517) * update version for release * added new use case that was missing * Bugfix #2279 main_v5.1 - buoy station file from 2022 (#2281) * Fix typo in pull_request_template.md * added notes to make it clear that upgrade instructions are not needed if upgrading from 5.0 to 5.1 * New issue template: Update Truth (#2332) Co-authored-by: John Halley Gotway <[email protected]> (cherry picked from commit 44335f33ab152a0b254041961a41895dde614ae0) * add GitHub Actions workflow that is used to create Docker images that hold the conda environment used for the automated tests -- adding this to the default main_v5.1 branch so that it will become available to run for other branches as a workflow_dispatch workflow * Per #2433, added support for setting the ugrid MET config variables for GridStat and PointStat wrappers. Also moved the seeps variable up so that it matches the order of the default config files in the MET repo * add argument to workflow dispatch event so that MET DockerHub repo used for tests can be easily overridden to test changes in a MET PR before merging * Feature dtcenter/MET#2796 main_v5.1 GHA Node20 deprecation warnings (#2474) * per dtcenter/MET#2796, update versions of actions to prevent deprecated node warnings in GHA runs - main_v5.1 * fix ReadTheDocs requirements to include pillow which is a dependency of sphinx-gallery: see https://blog.readthedocs.com/defaulting-latest-build-tools/ for more info on why this was necessary * install python packages via apk instead of pip to prevent GHA failures that were fixed in develop but not in main_v5.1 * per dtcenter/MET#2796, fix error log merging for main_v5.1 same as develop * Bump pillow from 10.0.1 to 10.2.0 in /docs (#2477) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](https://github.com/python-pillow/Pillow/compare/10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Create 5.1.0_casper * per #2433, add support for setting the optional -config argument for a ugrid config file for PointStat and GridStat. Also moved the optional arguments to be added to the command after all of the required arguments so the command is easier to read * per #2433 and discussion on meeting 3/21/2024, change command line argument from -config to -ugrid_config * update unit tests to check for new command line argument name -ugrid_config * Updates information about GDAS surface winds having a QC value that is above the default settings in the PB2NC config file. --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: John Halley Gotway <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Julie Prestopnik <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> * SonarQube add python code coverage report (#2565) * run code coverage before SonarQube scan * generate xml report and configure SQ to read coverage.xml * exclude more files from code coverage report * exclude more files that should not be included in the code coverage report * more changes to code coverage exclude list * removed bad characters accidentally added * exclude cyclone plotter wrapper because it is excluded from code coverage report * ignore SonarQube lint files generated by PyCharm * Updating MTD conv radius/thresh description (#2566) * Updating MTD conv radius/thresh description * Update glossary.rst * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> * Update docs/Users_Guide/glossary.rst Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature update modulefiles (#2564) * Adding 6.0.0 file for derecho and removing 5.1.0 file for cheyenne * Updating modulefiles * Updating orion file * Adding file for hercules * Adding file for casper * Update internal/scripts/installation/modulefiles/6.0.0_casper Co-authored-by: George McCabe <[email protected]> --------- Co-authored-by: George McCabe <[email protected]> * Feature #1989: Add OMI to Verification Datasets Guide (#2585) Refs: #1989 * Feature #2479 time_offset_warning (#2587) * Per #2479, add support for setting time_offset_warning in all wrapped MET config files * Per #2479, add documentation and examples to basic use case param files for time_offset_warning * Feature 2346 develop update templates (#2589) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Bugfix #2567 PointStat multiple point observation files (#2582) * per #2567, properly pass multiple point observation files to point_stat using the -point_obs argument * per #2567, fix bug that caused file window logic to fail * Revert "per #2567, fix bug that caused file window logic to fail" This reverts commit 27fe8226c58e9a028a7979664bbf224296fdd6ea. * per #2567, fix bug that caused file window logic to fail * Feature #1514 MADIS2NC wrapper (#2591) * Per #1514, implement MADIS2NC wrapper and added example use case. Also updated the function to handle the time_summary dictionary in MET config files to support names that exactly match the name found in the dictionary, e.g. ASCII2NC_TIME_SUMMARY_OBS_VAR sets time_summary.obs_var (previously only ASCII2NC_TIME_SUMMARY_VAR_NAMES was supported and is still supported) * remove execute permissions from image files * Per #1514, add image for basic use case * removed large image files that are no longer being used in documentation * add support for time_offset_warning for MADIS2NC wrapper after the PR to add that support for other wrappers has been merged into develop * report error if output template is not defined * update contributor's guide with more up-to-date info on how to create a new wrapper and basic components of wrappers * fix warnings in documentation * fix formatting issues * Per #1514, add new basic use case to automated test suite * add step to comment out version number in wrapped MET config file * turn off use case to prepare for PR * added a pytest fixture to handle comparison of use case commands and environment variable values to remove a lot of redundant logic in each wrapper test. Added fake madis data * removed commented code * properly substitute template tags in all command line arguments * properly handle unset rec_beg and rec_end to prevent missing value from being added to command lien arg * added new madis2nc use case to existing met_tool_wrapper and temporarily disabled land_surface:0 until we can resolve the differences * Feature 2346 develop update templates (#2594) * Per #2346, modifying issue templates * Per #2346, modifying pull request template * Per #2346, modifying entries based on suggestions by @georgemccabe * Per 2346, making updates based on feedback at last MET Eng. Meeting * Enhance update truth data workflow to create a uniquely named branch to update *-ref branches and commit/append to a log file that tracks the reasons for updating the truth data. This is done to ensure that the *-ref branch testing workflow run that actually updates the truth data is always run even if there are no other changes to the METplus branch since the last update, e.g. when a change to another component like MET warrants the truth data update * git add change log file in case it doesn't already exist * changed logic to no longer push changes to develop/main_vX.Y, but instead merge changes from -ref into the update branch * retain update truth history file from *-ref * dtcenter/MET#2899 fixes a bug reading point observations using Python Embedding in PointStat and EnsembleStat, which should fix the PBL use case -- dtcenter/METplus#2246 -- so turned on diff test for PBL use case to ensure that results are consistent going forward * Feature #2429 MvMODE multivar intensity (#2603) * Feature #2547 ASCII2NC -valid_beg and -valid_end arguments (#2606) * prevent divide by zero if run_count is 0 * Per #2547, add support for setting -valid_beg and -valid_end command line arguments. Added changes to make ASCII2NC wrapper able to run for all runtime frequencies * Refactored system_util preprocess_file function to reduce cognitive complexity and add quotation marks around 'filenames' that contain spaces (typically python embedding commands) so explicit handling of these cases don't need to be done downstream. Added unit tests to test more python embedding cases * remove logic to add quotes around input file since it is handled already in preprocess_file * changed find_input_files function to return None or a time_info dictionary to be consistent across wrappers * added ReformatPointWrapper to use as parent class for ASCII2NC, MADIS2NC, PB2NC, and Point2Grid wrappers to consistently handle tools that reformat point observation data. Moved verbosity to the end of commands * clean up pb2nc wrapper to be more consistent with other ReformatPoint wrappers * per #2547, added glossary entries for new config variables to set -valid_beg/end and added commented example to basic use case config file * added glossary entries for *_RUNTIME_FREQ variables * Per #2513, remove TC_RMW_MAX_RANGE_KM * Feature 2494 update fv3 data (#2610) * Updated for new data * Updated to match new data * Updates due to new data/updates to data * Updates due to new data/data variables * Changes due to changes to data variables, date * Updates due to changes to data * Update the date to reflect the new data (with updates to variables) * fixed error with formatting * Remove redundant instructions. * For testing * Update use_case_groups.json * Revert to original location of input data to use the same data for all three FV3 Physics Tendency use cases. * Revert to original location of data from tarball * Remove typo in file directory name * Update use_case_groups.json Returned 10-12 to follow 11 * Update use_case_groups.json revert to false for testing the short range use cases for FV3 physics tendency * Update use_case_groups.json fix alignment of opening curly brace * Update use_case_groups.json revert to original formatting * removed config variable that should be set by the user because it is specific to the user's environment --------- Co-authored-by: George McCabe <[email protected]> * Feature #2578 PCPCombine -input_thresh for missing inputs (#2609) * refactoring to reduce duplicate/redundant code, reduce cognitive complexity to satisfy SonarQube, etc * change wording of error log to satisfy test * set -input_thresh argument if set for add, derive, and sum methods. refactor setting of method arguments, e.g. -add, -sum, etc., cleanup * refactor how level is handled in find_data function so that if the level has already been set, it will use that value, otherwise try to get it from {data_type}_level, e.g. fcst_level, otherwise set it to 0 to prevent errors * refactor duplicate code into function to satisfy SonarQube * removed unused variable * use find_data function to find input files to be consistent with other wrappers, only allow multiple input files to be found for a given call to find_data if using the -derive method * fix typo in key * suppress warnings when files aren't found because it is expected * formatting to be consistent in doc string * per #2578, add MISSING before file path that is not found if input_thresh is set and less than 1.0, added unit test to ensure correct behavior occurs * add documentation blocks for new functions, ci-run-all-diff * use pytest fixture instead of local function * add support for setting -vld_thresh argument * per #2578, added documentation and tests for setting -input_thresh and -vld_thresh * moved verbosity argument to end of command to more easily change it when debugging * refactor to reduce cognitive complexity to satisfy SonarQube * update usage statement to include METplus version number * Add copy button for code blocks to easily copy/paste commands (#2611) * add copy button for code blocks to easily copy/paste commands * turn off copy button always visible but leave code block so it can be turned on easily if desired * Update release-notes.rst Update METplus-6.0.0 development schedule. * Fix numbering. * fix broken link * Feature 2557 stratosphere qbo (#2621) * Initial commit for QBO driver * Adding initial documentation for Stratosphere * Updating documentation typo * Updates to Polar plot * Updates to Stratosphere use cases and documentation * Updates to polar use case * Updating documentation * Updates to Polar and QBO * Updated documentation for use cases that cannot be run in actions * Removed some unnecessary imports * Fixed staging dir settings * documentation update * More documentation edits * More updates * Fixed link in documentation * Updating documentation * Updated bullet list * More updates * Added some additional comments to the .conf file * Fixed typo in output file name in documentation * Updated output section of documentation * Updated documentation * Fixed list that was not working * Updated broken link * Fixed broken link again * Fixed another typo * Trying to fix documentation error * Trying to fix indent error * fixed capital typo * Added clarification * use v2 of metplus-action-release-checksum to prevent failure in cUrl command to upload asset * update version of certifi to fix dependabot alert -- see #2632 (#2633) * Feature #2626 v6.0.0-beta5 release (#2630) * update version and added release notes * Per #2626, updating dates --------- Co-authored-by: Julie Prestopnik <[email protected]> * Updated release dates * Updating for changes on wcoss2 * Feature #2656 update_truth (#2657) * Per #2656, update the read_iodav2_mpr.py scripts to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Per #2656, update the ugrid_lfric_mpr.py to write 3 additional values of NA in each MPR line corresponding to the OBS_CLIMO_CDF, FCST_CLIMO_MEAN, and FCST_CLIMO_STDEV columns. * Feature 2679 user support (#2681) * Per 2679, updating user support documentation * Per #2679, fixing formatting error * Update user_support.rst * Update user_support.rst --------- Co-authored-by: John Halley Gotway <[email protected]> * Feature 2452 update release guide (#2692) * Per #2452, updating MET official release instructions * Update to indicate the "DTC" web server machine Co-authored-by: John Halley Gotway <[email protected]> * Updated language for tar file Co-authored-by: John Halley Gotway <[email protected]> --------- Co-authored-by: John Halley Gotway <[email protected]> * Updating release schedule * Feature #2612 forecast lead groups (#2698) * Refactor to prevent calling sys.exit from functions that could cause the calling script to exit unintentionally. Instead return None or False and only sys.exit a non-zero value from a script that is called explicitly. Refactored validate config logic to reduce cognitive complexity to satisfy SonarQube * per #2596, ensure the same RUN_ID is set for a given METplus run even if instances are used in the PROCESS_LIST * fixed linter complaints * remove unused imports * cleanup for SonarQube and linter * remove unused imports * import parent wrappers directly so that PyCharm can follow class hierarchy * per #2612, add support for processing groups of forecast leads * remove call to prune_empty function and ci-run-all-diff * fixed check on DataFrame to properly check for an empty one * remove prune_empty function that is no longer used * update conda environment requirements for metdataio * set label to Group<n> for LEAD_SEQ_<n> if LEAD_SEQ_<n>_LABEL is not set (instead of erroring), use forecast lead label for lead if set, log lead group label and formatted list of leads for each run * increase timeout for request from 60 to 90 * handle exception and output an error message with a suggestion to rerun the failed GHA jobs when a hiccup with DockerHub prevents the list of available Docker data volumes from being obtained. FYI @jprestop * cleanup to address SonarQube complaints, move call to exit out of main function so a script calling the function does not exit when the function errors * cleanup to appease SonarQube * per #2612, add support for creating groups of forecast leads by specifying a list of forecast leads and a time interval to create groups of leads * per #2612, add unit tests for function that gets groups of forecast leads and added tests for new groups capabilities * various refactoring to satisfy SonarQube complaints * renamed config LEAD_SEQ_DIVISIONS to LEAD_SEQ_GROUP_SIZE and LEAD_SEQ_DIVISIONS_LABEL to LEAD_SEQ_GROUP_LABEL * per #2612, add documentation for new config variables including examples * fix empty check for DataFrame * fix rst typo * minor cleanup to satisfy linter * Per #2612, if runtime freq is RUN_ONCE_PER_LEAD and groups of forecast leads are specified, then the wrapper will be run once per lead GROUP instead of once for each unique lead in the lead groups. Update unit test to reflect new behavior * update versions packages to match METplotpy requirements and update call to cartopy feature download script -- see https://github.com/SciTools/cartopy/pull/2263 * Feature #2622 forecast climatology (#2696) * remove support for old environment variables CLIMO_MEAN_FILE and CLIMO_STDEV_FILE that were used in METplus versions earlier than v4.0.0 * fixed SonarQube bugs * add better error checking for DockerHub queries that occasionally fail in GHA * update pycharm files * per #2622, support setting climo_mean and climo_stdev dictionary values inside fcst and obs dictionaries for grid_stat, ensemble_stat, point_stat, and series_analysis. Quietly add support for setting climo dictionaries inside the ens dictionary for gen_ens_prod -- this support is not needed because there is only 1 dictionary that may contain the climo dicts, but adding support for completeness so it can be assumed that {APP_NAME}_{DICT}_{SUBDICT}_{VAR} is supported, e.g. GEN_ENS_PROD_ENS_CLIMO_MEAN_REGRID_METHOD * turn on use cases that started failing from MET changes * turn off all use cases from push events * reorder assert so that expected vs. actual in PyCharm diff are accurate * remove commented code * added unit tests for climo VAR<n> variables that were missing * add VAR<n> variables used to specify climo_mean/stdev field values * bugfix to allow empty list for diag_info_map.diag_name * another hotfix to allow empty list for diag_info_map.match_to_track * Feature #2651 SeriesAnalysis -aggr argument and new use case (#2701) * exclude build directory from PyCharm project * refactor to satisfy SonarQube complaints * Per #2651, add support for setting -aggr argument with file path. Refactor functions to resolve SonarQube complaints. Enhance unit tests to test multiple init times and -aggr argument * add new config variables to basic use case * change settings to match MET unit test * added new config variables to set -aggr argument in SeriesAnalysis wrapper to documentation * Per #2651, add new use case to demonstrate using the -aggr argument to SeriesAnalysis * turn on new use case to test it in GHA * remove blank line * turn off use case to prepare for PR * Bugfix #2705 develop - SeriesAnalysis fix time substitution into field info (#2709) * fix SonarQube complaints * Per #2705, set correct time information in field info by using the time info used to find the input files (self.c_dict['ALL_FILES']) instead of relying on parsing the time info from the other input files (fcst vs. obs) * clean up unused code after bugfix for #2705 * Feature 2647 use case mv mode for rrfs (#2706) * establish documentation, need to fill out with provided info * updated documentation, needs to have img path and output path fixed * updated output strings, still need to update pic link * removing pound sign to render correctly * Revert "removing pound sign to render correctly" This reverts commit ffe607a96fd33dc5ba15a479f76778dcba7e9656. * targeted removal of pound sign with bullet points section. * Per #2647, attempt to fix syntax error * Per #2647, adding back contents with formatting changes to test if it works * Per #2647, adding back contents in a different location to try to resolve syntax error * Per #2647, testing pipe for horizontal scroll bar * Per #2647, removed the pipe symbol and are attempting to fix syntax error again * Per #2647, reverting to original failed attempt of a table of contents * Per #2647, trying new formatting * Per #2647, fixing new formatting * Per #2647, attempt formatting changes * Per #2647, reverting back to original state * added to automated testing, updated docs, config file * added docs back in * added image, new quick search term, disabled auto tests * testing use_case_groups.json to see if/how this fails * Updated to disable use_case_groups.json run * added use case to list of exceeding disk space --------- Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Lisa Goodrich <[email protected]> Co-authored-by: Julie Prestopnik <[email protected]> * Bugfix 2675 move to pyproject.toml (#2699) * 2675: move to pyproject.toml * 2675: update toml, remove setup.py ref * bugfix 2675: reinstate VERSION file * bugfix 2675: hardcode version in pyproject.toml * use relative versions * slightly modify import to make it easier to move produtil into metplus * read version from version file * disable install of run_metplus script since it is broken * only install metplus and sub-packages, add version files to package data (parm cannot be added because it is non-package data) * add newline back * install produtil package so metplus can be imported in python * remove unused import * move produtil under metplus to differentiate it from the full NCEP produtil package and isolate it to metplus when installing metplus as a package -- previously produtil would be installed as its own package when metplus is installed * fixed more imports * Create links for parm directory and scripts/run_metplus.py under metplus so the parm files and run script will be installed as part of the metplus python package. Modified config_metplus to find the parm directory from under metplus to be consistent between running via python package and via scripts * Exclude parm from sonar sources because it will be found under metplus. Add metplus/parm/** to list of coverage exclusions * fixed more imports * add parm back to list of sonar sources --------- Co-authored-by: George McCabe <[email protected]> * Removing stated location of the version selector menu * Feature 2489 update docs (#2717) * adding new Literal Include section * under bold section adding "Environment" to Variables * Adding Substitution References section * removing blank space * fixing code typos * another typo * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Contributors_Guide/documentation.rst Co-authored-by: Julie Prestopnik <[email protected]> --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature #2540 Point2Grid config file (#2580) * per #2540, add support for setting values in a config file for Point2Grid wrapper -- removed -qc flag argument in favor of setting obs_quality_inc in the config file * Added support for setting time_offset_warning now that the changes for other wrappers was merged into develop. Also added support for MET_CONFIG_OVERRIDES for Point2Grid * per #2540, set -goes_qc command line argument from POINT2GRID_GOES_QC_FLAGS and continue to support legacy POINT2GRID_QC_FLAGS. Other fixes to resolve incorrectly resolved conflicts from previous commit merging develop into branch * per #2540, remove quality_mark_thresh from config file because it has been removed from MET point2grid * Per discussion in PR #2580, remove time_offset_warning support from tools that don't actually use the value -- to add support for a new wrapper, update metplus/util/constants.py TIME_OFFSET_WARNING_WRAPPERS to include the wrapper's name * removed env vars for time_offset_warning for wrappers that no longer set it * install ImageMagick convert because it appears to no longer be available in the image that runs the unit tests (ubuntu 24.04?) * Bugfix metdbload config (#2728) * Update METdbLoadConfig.xml A schema is now used to validate all XML specification files and elements must adhere to a particular order. The <group> element follows the headers and load elements and preceeds the folder tmpl element. * Update use_case_groups.json temporarily run tests for the short range use cases * Update use_case_groups.json Return all the short_range use case test "run" values back to false * install convert because some tests fail if it is not available * Feature #1882 docs: new use case that uses METplus Analysis (#2726) * Fixing mistake * Fixing reference * Fixing link * Updating sections * clean up new content --------- Co-authored-by: Christina Kalb <[email protected]> * Feature 918 add use case template (#2690) * updated template, 1st part * added more to template, updated internal references. Needs final two cats * finished updating template * updated syntax, length of lines * moved example to separate file, testing indents for discussion * added new collapse menu for MET configs, documentation on adding keywords, testing 2 separate options to stop paragraph jumping during bullet lists * updated spacing used for 2nd line indentation * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: lisagoodrich <[email protected]> * Minor changes, updates hyperlink to template to be directly to the file and not to the directoy in GitHub. * Update add_use_case.rst * Update docs/Contributors_Guide/add_use_case.rst Co-authored-by: George McCabe <[email protected]> * Update add_use_case.rst * Missing space in hyperlink. * Changes to use a keyword to assist users in checking off each section they should review and also PR reviewers in ensuring each section has been visited and reviewed by the contributor. * updated Python section to User Scripting --------- Co-authored-by: lisagoodrich <[email protected]> Co-authored-by: Daniel Adriaansen <[email protected]> Co-authored-by: George McCabe <[email protected]> * Feature #2727 point_weight_flag and obtype_as_group_val_flag (#2730) * Per #2727, add support for setting point_weight_flag * Per #2727, add support for obtype_as_group_val_flag * per #2714, deprecate master_metplus.py * Update VERSION * Feature #2710 - clean up use cases (#2711) * per #2710, remove line that has no effect because next line overwrites value * per #2710, fix bug in if/else block where both were doing the same thing. Removed commented code and create variable for constant to satisfy SonarQube * updated config file commands * updated level of winds to capture change in script --------- Co-authored-by: j-opatz <[email protected]> * Feature #2682 Component version lookup (#2733) * per #2714, deprecate master_metplus.py * exclude metplus/scripts from SonarQube scan to prevent incorrect duplicate code error since script lives in ush * added Dockerfile to build image with MET/METplus and METplus Analysis Python packages * start of script to get versions of METplus components * per #2682, created METplus component version lookup table and functions to get formatted version of requested components * add descriptions for tests * add update information and made script callable * made script executable * add workflow to trigger on release and create dtcenter/metplus and dtcenter/metplus-analysis Docker images * add scripts used by new workflow * removed build hook scripts since we will be building images via GHA * change logic to get MET version using component version script instead of build hook file that just adds 6 to major version * add missing shebang * make script callable directly and make output component a required input * pass LATEST_TAG to push script * ensure only 1st line of version file is read to avoid newline * turn on use case to test * turn off use case * Feature #2735 v6.0.0 beta6 (#2736) * change version for beta6 release * added script to help generate release notes for development releases * add default title for issues and added documentation issue template * sort issues by number, improve formatting for release notes title, add logging to alert users what is happening when timely github queries are running * added release notes for beta6 release * Add option to component version script to return 'develop' if the input version is a beta or rc version to preserve previous behavior of GHA scripts. Update GHA scripts to use new option * applied suggestions from feedback in PR #2736 * bold some release notes * update version for development towards rc1 release * fix logic to determine offical X.Y.Z release * Feature #2739 labels (#2740) * Per #2739, add NRL reporting label and add METplus-Internal to the list of repositories to be updated. * Per #2739, add info about labels to the Contributor's Guide. * Per #2739, update the list of METplus repos in process_labels.sh but remove them from the other shell scripts to avoid keeping the same list updated in multiple spots. * Per #2739, move label details from the README to the Contributor's Guide * Updated release notes adding this issue: Develop an RST template for use cases * Feature 2716 existing builds (#2748) * adding the first 4 paragraphs * fixing web addresses * Highest level of dropdown menus added * NCAR machines dropdown menus * CASPER dropdown menu * web addresses can't be bolded. Fixing dropdown problems * blank line problem * italics date adding Derechio parts * DERECHO dropdown * fixing problems * fixing mor problems * fixing bolding * trying to fix web address * adding NCAR RAL MACHINES and trying to fix website issue * NCAR/RAL Internal Development dropdown * adding remaining dropdown structure * WCOSS2 dropdown * HERA dropdown * fixing last updated to be consistent * HERCULES dropdown * fixing pip * orion dropdown * JET dropdown * GAEA dropdown * fixing typo * Community machines, Docker Hub & AWS * fixing typo * Changing to "Users should create a file like" * to set a personalized input... * adding last updated and Docker Hub web links * Updating Jet information after recent install and changing some formatting for consistency * Removing bolding for consistency * creating a separate file for existing builds to add to 2 areas make it section 2.1 where METplus Components Release Note Links and Release Notes * adding existing_builds as an include * Rename existing_builds_rst to existing_builds.rst I had an underline instead of a period * changing to a section, not a chapter * trying a different section * changing existing builds to an include file * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <[email protected]> * Update docs/Users_Guide/existing_builds.rst Co-authored-by: Julie Prestopnik <[email protected]> * removing existing builds completely from the Release Guide --------- Co-authored-by: Julie Prestopnik <[email protected]> * Update release-notes.rst * add custom string to time info dictionary for substituting values into the field info string * Feature #2742 develop fix StatAnalysis to set fcst_lev/obs_lev (#2760) * Feature 2763 existing builds (#2765) * Per #2763, updated build for Casper * Per #2763, updating Orion * Bugfix #2762 PCPCombine derive field info (#2764) * per #2762, change derive mode to add field info for each file instead of adding the first field info only * Per #2762, improve how the accumulation amounts are handling using relativedelta to ensure that the correct number of seconds are used for each search window when using inconsistent time intervals like months or years * Bugfix #2762 part 2 -- fix PCPCombine leap year issue (#2766) * remove check if script is master_metplus.py because it has been deprecated * per #2762, fix bug that caused failure depending on valid time relating to leap years * Feature #2754 rc1_main_vX.Y (#2755) * Per #2754, update instructions to create the main_vX.Y and main_vX.Y-ref branches (when applicable) for the first release candidate rather than waiting until the official release. Note that I deleted the push_release_branch.rst instructions since it's not needed. The preceeding instructions execute the push and these are redundant - even for METplus which has custom instructions. * Per #2754, add a note to make the detail stand out better * Per #2754, make the GitHub release instructions more explicit to match the names of the actual fields on GitHub. * Per #2754, not changing comment, just using more consisent formatting options. * Per #2754, change example from 'betaN' to 'rc1' since this step is only run for the rc1 release. * Per #2754, use consistent single quotes instead of italics for formatting. * Updates to METplus instructions to properly handle version numbers for rc1 release. Update METplus Official release instructions to use main branch and update version number for official release since main branch will contain -rcN version numbers * update instructions to checkout correct branch * clarify next deveopment version can be X+1.0 OR X.Y+1 * update version table for other components -- this could likely be refactored to make the version file path a variable and include the same update_version_on_develop.rst file for METplus, METcalcpy, METdataio, METplotpy, and METviewer --------- Co-authored-by: George McCabe <[email protected]> * Resolving use case template formatting problems and adding dropdowns * Resolving use case template formatting problem * Fixing formatting * Updating to add language if no Python embedding or User Scripting is used. * Feature #2562 version lookup documentation and automated tests (#2773) * set default input version to the latest official release, which is the 2nd highest coordinated release version number in the version lookup table. This assumes that the highest version is what is being developed towards * per #2562, update Release Guide to include instructions for updating the component versions lookup table in the METplus repository * Per #2562 and #2597, replace automated test logic that used manage_externals to use component version lookup to get versions of METplotpy/METcalcpy/METdataio * clean up SonarQube complaints * turn on some use cases to test changes * fix clone * do not get python dependencies for METplotpy/METcalcpy/METdataio installs * use full path to package to import * clone METplus Analysis repos on the same level as METplus to match previous location using manage_externals, remove manage_externals config files that are no longer used by the automated tests * clean up linter complaints * turn off use cases after confirming that they run successfully * clean up usage statement formatting * remove info about updating manage_externals config files used in automated tests that have been removed * added description of component versions script to Contributor's Guide and added stubs for other utility descriptions * change format of example results and add text to describe what is being demonstrated from each example * fix formatting * fixed bad variable replacement * Update docs/Contributors_Guide/utilities.rst --------- Co-authored-by: Julie Prestopnik <[email protected]> * Feature 2737 release instruct (#2775) * Per #2737 updating version instructions * Per #2737, update version instructions for METcalcpy * Per #2737, renaming development file * Per #2737, update version instructions for METplotpy * Per #2737, updating version instructions for METdataio * Per #2737, updating version instructions for METviewer * Per #2737, adding pyproject.toml to the list of files to update …
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Testing
Added unit tests for new capability to allow missing input files and still run without error
Ensured all existing tests and use cases pass
@JohnHalleyGotway and I met to discuss the details of this PR because there are a lot of changes. All tests should pass with no differences.
Review beta3 release notes: https://metplus--2493.org.readthedocs.build/en/2493/Users_Guide/release-notes.html#metplus-version-6-0-0-beta-3-release-notes-2024-02-07
Additional changes should be made after the beta3 release, including adding documentation for the new config variables. I created issue #2492 to complete remaining tasks.
Added and enhanced unit tests
Will this PR result in changes to the test suite? [No]
Please complete this pull request review by 2/7/2024.
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s)
Select: Organization level software support Project or Repository level development cycle Project
Select: Milestone as the version that will include these changes