Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature #2299 / #2310 Remove deprecated MET config env vars and TCMPRPlotter #2347

Merged
merged 105 commits into from
Sep 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
3c74be4
added helper function to clean up logic to add new dictionaries that …
georgemccabe Aug 8, 2023
60662e4
fix incorrect variable name
georgemccabe Aug 8, 2023
927d631
log error if wrapper was not loaded properly to help debug issue
georgemccabe Aug 8, 2023
ffe1f23
improve logging to print input dir/template once at beginning to avoi…
georgemccabe Aug 8, 2023
fee61a2
make wrappers LoopTimes (RuntimeFreq using run all times) to remove r…
georgemccabe Aug 8, 2023
89a34d8
fix function name in test, ci-run-all-diff
georgemccabe Aug 9, 2023
a3f6c70
set self.app_name if unset for all wrappers to prevent errors when cr…
georgemccabe Aug 9, 2023
5673a61
set self.app_name if unset for all wrappers to prevent errors when cr…
georgemccabe Aug 9, 2023
d82c5e8
change all CompareGridded (GridStat, PointStat, EnsembleStat, MODE, M…
georgemccabe Aug 9, 2023
39e8582
add test for usage wrapper, ci-run-all-diff
georgemccabe Aug 9, 2023
6dbc6aa
fix call to RegridDataPlane from ExtractTiles wrapper
georgemccabe Aug 9, 2023
82f04e4
changed more wrappers to use consistent time looping, ci-run-all-cases
georgemccabe Aug 9, 2023
469bbe5
formatting
georgemccabe Aug 9, 2023
83e0286
more formatting
georgemccabe Aug 9, 2023
9cb8b7e
remove unnessary check for skip times because it is called upstream, …
georgemccabe Aug 9, 2023
da19788
add SKIP_LEAD_SEQ check to RuntimeFreq run_at_time and change TCPairs…
georgemccabe Aug 9, 2023
f19aaf5
run each time input through ti_calculate before passing it to run_at_…
georgemccabe Aug 10, 2023
e0a68d2
move runtime log banner
georgemccabe Aug 11, 2023
21f8f27
rename function to avoid confusion with other find_input_files functions
georgemccabe Aug 11, 2023
ba81987
refactored logic in ti_calculate and added unit tests for common inpu…
georgemccabe Aug 14, 2023
3bc665a
formatting cleanup
georgemccabe Aug 14, 2023
92d5d26
moved logic to compute formatted init or valid so it is still run eve…
georgemccabe Aug 14, 2023
2c60cd4
pass data type to find_data and change template c_dict key if data ty…
georgemccabe Aug 14, 2023
77987c2
refactor MTD wrapper to use RuntimeFreq methods to find files to proc…
georgemccabe Aug 14, 2023
a1ba59b
use f-strings for command, ci-run-all-diff
georgemccabe Aug 14, 2023
c9610e0
added unit test for case that happens in gfdl tracker wrapper
georgemccabe Aug 14, 2023
95a245f
formatting
georgemccabe Aug 14, 2023
55dbcd8
fixed failures when trying to format times that are wildcards, fix ch…
georgemccabe Aug 14, 2023
c60b3eb
set default runtime freq for TCPairs wrapper
georgemccabe Aug 14, 2023
9bac908
fix use case to use init instead of valid, ci-run-all-diff
georgemccabe Aug 14, 2023
460851e
Merge branch 'develop' into feature_2283_include_times
georgemccabe Aug 15, 2023
9b7ee25
preserve old behavior by using init and first lead to compute valid t…
georgemccabe Aug 15, 2023
c133bb4
if [INIT/VALID]_BEG == [INIT/VALID]_END and runtime freq is RUN_ONCE,…
georgemccabe Aug 15, 2023
70fa3cb
fix bug introduced by using RuntimeFreq that causes duplicate regex g…
georgemccabe Aug 15, 2023
9a433e2
fix config to use same init beg/end for RUN_ONCE mode
georgemccabe Aug 15, 2023
4c903c6
use RuntimeFreq time looping, preserve old behavior by setting runtim…
georgemccabe Aug 15, 2023
95eac30
turn on use case groups that were failing to test , ci-run-diff
georgemccabe Aug 15, 2023
334f921
add missing import, ci-run-diff
georgemccabe Aug 15, 2023
9e1eeab
name file_list file with 'single' if MTD_SINGLE_RUN to prevent diffs …
georgemccabe Aug 15, 2023
0d53a0c
remove unused variable
georgemccabe Aug 15, 2023
101f5d0
call self.clear before running in RUN_ONCE mode to clear out command …
georgemccabe Aug 15, 2023
1e4a953
change TCGen wrapper to RuntimeFreq
georgemccabe Aug 15, 2023
df2d18a
set lead_hours/minutes/seconds even if lead is months or years so rel…
georgemccabe Aug 15, 2023
9a1a610
update use case config to use single run time (RUN_ONCE) and set init…
georgemccabe Aug 15, 2023
569c6cf
turn off use cases that now pass and test if other cases will pass, c…
georgemccabe Aug 15, 2023
50fbe5a
turn off all use cases
georgemccabe Aug 15, 2023
4383869
Create class variables for RuntimeFreq wrappers to define default val…
georgemccabe Aug 15, 2023
2bcda33
error and return if start and end times cannot be read, e.g. if end t…
georgemccabe Aug 15, 2023
8f308ff
update config files to use current config options, ci-run-all-diff
georgemccabe Aug 15, 2023
74e52bd
fixed bug when beg/end times are not set and aren't needed, ci-run-a…
georgemccabe Aug 15, 2023
b9eef8f
fix use case reference to init time
georgemccabe Aug 16, 2023
75da7ab
update contrib guide
georgemccabe Aug 16, 2023
407aba4
clean up contrib guide additions
georgemccabe Aug 16, 2023
cef425d
change supported runtime frequency values for wrappers that need enha…
georgemccabe Aug 16, 2023
4ed89d6
fix job args to remove hours from -init_beg and -init_end
georgemccabe Aug 16, 2023
fea1c43
per #2283, add support for INCLUDE_TIMES (opposite of SKIP_TIMES) and…
georgemccabe Aug 16, 2023
9739279
Merge branch 'feature_2283_time_looping' into feature_2283_include_times
georgemccabe Aug 16, 2023
c5a538e
move timing log output of group so it can be easily viewed without op…
georgemccabe Aug 16, 2023
6ef7f6d
per #2283, add documentation about INCLUDE_TIMES and using days of th…
georgemccabe Aug 16, 2023
58fcc2f
create URL to easily navigate to commit that triggered workflow run
georgemccabe Aug 29, 2023
a90e31b
turn on a use case that is failing due to this issue
georgemccabe Sep 5, 2023
5dc6222
updated version of Docker base image used for all conda env images to…
georgemccabe Sep 5, 2023
ab32aca
added instructions to rebuild conda.vX.Y images as needed
georgemccabe Sep 5, 2023
21c31c1
change version extension of new docker images for conda environments …
georgemccabe Sep 6, 2023
6a1ec46
try building geovista via conda to see if required functionality is a…
georgemccabe Sep 6, 2023
a62feab
per #2283, implement skip and include times for init and valid
georgemccabe Sep 6, 2023
66f69c5
merged develop and resolved conflicts
georgemccabe Sep 6, 2023
cb33318
per #2283, add unit tests to ensure skip/include init/valid times wor…
georgemccabe Sep 6, 2023
9b18228
add other packages that were missing
georgemccabe Sep 6, 2023
6491808
add files to create conda envs via GHA workflow
georgemccabe Sep 8, 2023
9e6637b
use job ID to set env var used to name conda environment -- note cann…
georgemccabe Sep 8, 2023
acbebce
whitespace
georgemccabe Sep 8, 2023
07d7278
undo whitespace change to, ci-run-all-diff
georgemccabe Sep 8, 2023
0a10dff
renamed function that was changed in geovista between v0.1.4 and v0.2.0
georgemccabe Sep 8, 2023
25dd1f1
remove specific versions from packages
georgemccabe Sep 11, 2023
95489f8
remove versions from packages
georgemccabe Sep 11, 2023
205fb88
removed unused environment
georgemccabe Sep 11, 2023
6c85d5f
update use case to replace use of deprecated pandas DataFrame.append …
georgemccabe Sep 11, 2023
273287f
update geovista env to install libGL and libEGL to prevent dynamic li…
georgemccabe Sep 12, 2023
7af0704
update steps to install Java OpenJDK for GempakToCF.jar
georgemccabe Sep 12, 2023
7d1cdf6
copy files from new java JDK location
georgemccabe Sep 12, 2023
f90c3e4
run use case that is failing due to gempak install issues
georgemccabe Sep 12, 2023
061317d
copy additional java files needed to run
georgemccabe Sep 12, 2023
2803c92
turn off use case now that it passes
georgemccabe Sep 12, 2023
2a64e07
added version numbers for python packages to ensure the env is reprod…
georgemccabe Sep 12, 2023
56262e2
Update internal/scripts/docker_env/Dockerfile.conda
georgemccabe Sep 12, 2023
5a09634
fixed incorrect info in input description
georgemccabe Sep 12, 2023
2fead00
Merge branch 'feature_2338_debian12_conda_envs' of github.com:dtcente…
georgemccabe Sep 12, 2023
787e037
merged feature branch that is in an open PR
georgemccabe Sep 13, 2023
63a3399
clarify version numbers
georgemccabe Sep 13, 2023
0da2c3f
add new variables to glossary/wrappers for each wrapper-specific vari…
georgemccabe Sep 13, 2023
4cba461
fix doc errors
georgemccabe Sep 13, 2023
b3569ed
fix function documentation
georgemccabe Sep 13, 2023
8ce5375
Merge branch 'develop' into feature_2283_include_times
georgemccabe Sep 13, 2023
f8ce239
per #2299, remove support for legacy environment variables that may b…
georgemccabe Sep 13, 2023
e2951ab
per #2310, remove TCMPRPlotter wrapper
georgemccabe Sep 14, 2023
962bb8a
add back call to get_output_prefix to properly substitute values into…
georgemccabe Sep 14, 2023
21bb0b5
Per #2299, update check for deprecated MET config env vars by using t…
georgemccabe Sep 14, 2023
cfc0f54
Merge branch 'develop' into feature_2299_rm_depr_met_config
georgemccabe Sep 14, 2023
117c775
fix bug in logging of column name when diffing stat files
georgemccabe Sep 14, 2023
e0cbf50
update tests for deprecated met config
georgemccabe Sep 14, 2023
a4d7929
update use case to preserve output values for MODEL and OBTYPE becaus…
georgemccabe Sep 14, 2023
e000454
Per #2299, updated list of deprecated env vars that were previously u…
georgemccabe Sep 14, 2023
2c1231c
cleanup
georgemccabe Sep 14, 2023
425d2af
updated OBTYPE to be consistent
georgemccabe Sep 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
292 changes: 1 addition & 291 deletions docs/Users_Guide/glossary.rst

Large diffs are not rendered by default.

4 changes: 0 additions & 4 deletions docs/Users_Guide/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,6 @@ The following software is required to run METplus Wrappers:

**Wrapper Specific Requirements**

- TCMPRPlotter wrapper

- R version 3.2.5

- SeriesAnalysis wrapper

- convert (ImageMagick) utility -
Expand Down
2 changes: 0 additions & 2 deletions docs/Users_Guide/quicksearch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ Use Cases by MET Tool:
| `SeriesAnalysis <../search.html?q=SeriesAnalysisUseCase&check_keywords=yes&area=default>`_
| `StatAnalysis <../search.html?q=StatAnalysisToolUseCase&check_keywords=yes&area=default>`_
| `TCDiag <../search.html?q=TCDiagToolUseCase&check_keywords=yes&area=default>`_
| `TCMPRPlotter <../search.html?q=TCMPRPlotterUseCase&check_keywords=yes&area=default>`_
| `TCGen <../search.html?q=TCGenToolUseCase&check_keywords=yes&area=default>`_
| `TCPairs <../search.html?q=TCPairsToolUseCase&check_keywords=yes&area=default>`_
| `TCRMW <../search.html?q=TCRMWToolUseCase&check_keywords=yes&area=default>`_
Expand Down Expand Up @@ -62,7 +61,6 @@ Use Cases by MET Tool:
| **SeriesAnalysis**: *SeriesAnalysisUseCase*
| **StatAnalysis**: *StatAnalysisToolUseCase*
| **TCDiag**: *TCDiagToolUseCase*
| **TCMPRPlotter**: *TCMPRPlotterUseCase*
| **TCGen**: *TCGenToolUseCase*
| **TCPairs**: *TCPairsToolUseCase*
| **TCRMW**: *TCRMWToolUseCase*
Expand Down
6 changes: 3 additions & 3 deletions docs/Users_Guide/systemconfiguration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1169,9 +1169,9 @@ configuration files to change input/output file paths, configuration file
paths, and more. The value of each list item can be referenced in the
METplus configuration variables by using {custom?fmt=%s}. The variable
CUSTOM_LOOP_LIST will apply the values to each wrapper in the PROCESS_LIST
unless the wrapper does not support this functionality. CyclonePlotter,
StatAnalysis, TCStat, and
TCMPRPlotter wrappers are not supported. If the variable is not set or set
unless the wrapper does not support this functionality.
CyclonePlotter, StatAnalysis, and TCStat wrappers are not supported.
If the variable is not set or set
to an empty string, the wrapper will execute as normal without additional
runs. The name of the wrapper-specific variables contain the name of the
wrapper, i.e. SERIES_ANALYSIS_CUSTOM_LOOP_LIST,
Expand Down
80 changes: 1 addition & 79 deletions docs/Users_Guide/wrappers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8853,85 +8853,7 @@ see :ref:`How METplus controls MET config file settings<metplus-control-met>`.
TCMPRPlotter
============

Description
-----------

The TCMPRPlotter wrapper is a Python script that wraps the R script
plot_tcmpr.R. This script is useful for plotting the calculated
statistics for the output from the MET-TC tools. This script, and other
R scripts are included in the MET installation. Please refer to
the MET User's Guide for usage information.

METplus Configuration
---------------------

| :term:`TCMPR_PLOTTER_TCMPR_DATA_DIR`
| :term:`TCMPR_PLOTTER_PLOT_OUTPUT_DIR`
| :term:`TCMPR_PLOTTER_CONFIG_FILE`
| :term:`TCMPR_PLOTTER_PREFIX`
| :term:`TCMPR_PLOTTER_TITLE`
| :term:`TCMPR_PLOTTER_SUBTITLE`
| :term:`TCMPR_PLOTTER_XLAB`
| :term:`TCMPR_PLOTTER_YLAB`
| :term:`TCMPR_PLOTTER_XLIM`
| :term:`TCMPR_PLOTTER_YLIM`
| :term:`TCMPR_PLOTTER_FILTER`
| :term:`TCMPR_PLOTTER_FILTERED_TCST_DATA_FILE`
| :term:`TCMPR_PLOTTER_DEP_VARS`
| :term:`TCMPR_PLOTTER_SCATTER_X`
| :term:`TCMPR_PLOTTER_SCATTER_Y`
| :term:`TCMPR_PLOTTER_SKILL_REF`
| :term:`TCMPR_PLOTTER_SERIES`
| :term:`TCMPR_PLOTTER_SERIES_CI`
| :term:`TCMPR_PLOTTER_LEGEND`
| :term:`TCMPR_PLOTTER_LEAD`
| :term:`TCMPR_PLOTTER_PLOT_TYPES`
| :term:`TCMPR_PLOTTER_RP_DIFF`
| :term:`TCMPR_PLOTTER_DEMO_YR`
| :term:`TCMPR_PLOTTER_HFIP_BASELINE`
| :term:`TCMPR_PLOTTER_FOOTNOTE_FLAG`
| :term:`TCMPR_PLOTTER_PLOT_CONFIG_OPTS`
| :term:`TCMPR_PLOTTER_SAVE_DATA`
| :term:`TCMPR_PLOTTER_DEP_LABELS`
| :term:`TCMPR_PLOTTER_PLOT_LABELS`
| :term:`TCMPR_PLOTTER_READ_ALL_FILES`
|

The following are TCMPR flags, if set to 'no', then don't set flag, if
set to 'yes', then set the flag

| :term:`TCMPR_PLOTTER_NO_EE`
| :term:`TCMPR_PLOTTER_NO_LOG`
| :term:`TCMPR_PLOTTER_SAVE`
|

.. warning:: **DEPRECATED:**

| :term:`TCMPR_PLOT_OUT_DIR`
| :term:`TITLE`
| :term:`SUBTITLE`
| :term:`XLAB`
| :term:`YLAB`
| :term:`XLIM`
| :term:`YLIM`
| :term:`FILTER`
| :term:`FILTERED_TCST_DATA_FILE`
| :term:`DEP_VARS`
| :term:`SCATTER_X`
| :term:`SCATTER_Y`
| :term:`SKILL_REF`
| :term:`SERIES`
| :term:`SERIES_CI`
| :term:`LEGEND`
| :term:`LEAD`
| :term:`PLOT_TYPES`
| :term:`RP_DIFF`
| :term:`DEMO_YR`
| :term:`HFIP_BASELINE`
| :term:`FOOTNOTE_FLAG`
| :term:`PLOT_CONFIG_OPTS`
| :term:`SAVE_DATA`
|
.. warning:: This wrapper was removed in v6.0.0

.. _tc_pairs_wrapper:

Expand Down
2 changes: 0 additions & 2 deletions docs/use_cases/met_tool_wrapper/TCMPRPlotter/README.rst

This file was deleted.

116 changes: 0 additions & 116 deletions docs/use_cases/met_tool_wrapper/TCMPRPlotter/TCMPRPlotter.py

This file was deleted.

Loading