diff --git a/docs/Contributors_Guide/add_use_case.rst b/docs/Contributors_Guide/add_use_case.rst index aa645eeaa6..c31a9e19c3 100644 --- a/docs/Contributors_Guide/add_use_case.rst +++ b/docs/Contributors_Guide/add_use_case.rst @@ -542,15 +542,27 @@ or develop directories. Add contents of existing tarfile to feature branch directory (if applicable) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -**If you have determined that there is an existing tarfile for the category -(from the previous step)**, then untar the sample data tarball into -the feature branch directory. If no tarfile exists yet, you can skip this -step:: +**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!** + +**CONDITION 1: IF you have determined that there is an existing tarfile +for the category (from the previous step)**, +then untar the sample data tarball into the feature branch directory:: tar zxf ${METPLUS_EXISTING_DATA_TARFILE} -C ${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH} -Create the new tarfile -^^^^^^^^^^^^^^^^^^^^^^ +**CONDITION 2: If no tarfile exists yet, you can skip this step** + +Rename or modify existing data or data structure (if applicable) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +**If the reason for your feature branch is to adjust an existing use case, such as renaming a use case +or changing the data file,** then adjust the directory structure and/or the data files which should now +be in your feature branch directory (from your last step). Changes to a use case name or input data for +a preexisting use case should be separately verified to run successfully, and noted in the Pull Request form +(described later). + +Add new data to feature branch directory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Untar the new data tarball into the feature branch directory:: @@ -559,13 +571,17 @@ Untar the new data tarball into the feature branch directory:: Verify that all of the old and new data exists in the directory that was created (i.e. model_applications/). +Create the new tarfile +^^^^^^^^^^^^^^^^^^^^^^ Create the new sample data tarball. -Model Application Use Case Example:: +**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!** + +**CONDITION 1:** Model Application Use Case Example:: tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}.tgz model_applications/${METPLUS_USE_CASE_CATEGORY} -MET Tool Wrapper Use Case Example:: +**CONDITION 2:** MET Tool Wrapper Use Case Example:: tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}.tgz met_test @@ -699,8 +715,8 @@ dependencies. Dependencies ^^^^^^^^^^^^ -Environments -"""""""""""" +Conda Environments +"""""""""""""""""" The keywords that end with "_env" are Python environments created in Docker images using Conda that can be used to run use cases. These images are stored @@ -789,7 +805,59 @@ environments, refer to the comments in the scripts found in developer, so please contact MET Help if none of these environments contain the package requirements needed to run a new use case. -**MORE INFO COMING SOON!** +A README file can be found in the ci/docker/docker_env directory that +provides commands that can be run to recreate a Docker image if the +conda environment needs to be updated. Please note that Docker must +be installed on the workstation used to create new Docker images and +a DockerHub account with access to the dtcenter repositories must +be used to push Docker images to DockerHub. + +The README file also contains commands to create a conda environment +that is used for the tests locally. Any base conda environments, +such as metplus_base and py_embed_base, must be created locally first +before creating an environment that builds upon these environments. +Please note that some commands in the scripts are specific to +the Docker environment and may need to be rerun to successfully +build the environment locally. + +**Installing METplus Components** + +These scripts +do not install any METplus components, +such as metplotpy/metcalcpy/metplus, in the Python environment that +may be needed for a use case. This is done because the automated tests +will install and use the latest version (develop) of the packages to +ensure that any changes to those components do not break any existing +use cases. These packages will need to be installed by the user +and need to be updated manually. To install these packages, +activate the Conda environment, obtain the source code from GitHub, +and run "pip3 install ." in the top level directory of the repository. + +Example:: + + conda activate weatherregime + git clone git@github.com:dtcenter/METplotpy + cd METplotpy + git checkout develop + git pull + pip3 install . + +**Cartopy Shapefiles** + +The cartopy python package automatically attempts to download +shapefiles as needed. +The URL that is used in cartopy version 0.18.0 and earlier no longer +exists, so use cases that needs these files will fail if they are +not found locally. If a conda environment uses cartopy, these +shapefiles may need to be downloaded by the user running the use case +even if the conda environment was created by another user. +Cartopy provides a script that can be used to obtain these shapefiles +from the updated URL:: + + wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py + python3 cartopy_feature_download.py cultural physical cultural-extra + + .. _add_new_category_to_test_runs: @@ -1034,19 +1102,23 @@ Compare the feature branch file to the develop directory file:: diff ${METPLUS_FEATURE_BRANCH}/volume_mount_directories develop/volume_mount_directories -**IF there is a new entry or change in the feature version**, +**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!** + +**CONDITION 1: IF there is a new entry or change in the feature version**, copy the feature file into the develop directory:: cp ${METPLUS_FEATURE_BRANCH}/volume_mount_directories develop/volume_mount_directories -Copy the data from the feature directory into the next version directory -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Copy data from the feature directory into the next version directory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Make sure the paths are correct before copying. +**Make sure the paths are correct before copying.** -Model Applications Use Cases:: +**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!** - from_directory=${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}/model_applications/${METPLUS_USE_CASE_CATEGORY} +**CONDITION 1:** Model Applications Use Cases:: + + from_directory=${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}/model_applications/${METPLUS_USE_CASE_CATEGORY}/${METPLUS_USE_CASE_NAME} echo $from_directory ls $from_directory @@ -1054,7 +1126,9 @@ Model Applications Use Cases:: echo $to_directory ls $to_directory -MET Tool Wrapper Use Cases:: +**OR** + +**CONDITION 2:** MET Tool Wrapper Use Cases:: from_directory=${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}/met_test echo $from_directory @@ -1066,14 +1140,16 @@ MET Tool Wrapper Use Cases:: Once you have verified the correct directories are set, copy the files:: - cp -r $from_directory/* $to_directory/ + cp -r $from_directory $to_directory/ List the tarfile for the use case category in the next release version directory:: cd ${METPLUS_DATA_TARFILE_DIR}/v${METPLUS_VERSION} ls -lh sample_data-${METPLUS_USE_CASE_CATEGORY}* -**IF the latest version of the tarfile is in this directory**, +**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!** + +**CONDITION 1: IF the latest version of the tarfile is in this directory**, then rename the existing sample data tarball for the use case category just in case something goes wrong:: @@ -1081,21 +1157,32 @@ the use case category just in case something goes wrong:: **OR** -**IF the sample data tarfile for the category is a link to another METplus +**CONDITION 2: IF the sample data tarfile for the category is a link to another METplus version**, then simply remove the tarfile link:: unlink sample_data-${METPLUS_USE_CASE_CATEGORY}.tgz Create the new sample data tarfile. -Model Applications Use Cases:: +**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!** + +**CONDITION 1:** Model Applications Use Cases:: tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}-${METPLUS_VERSION}.tgz model_applications/${METPLUS_USE_CASE_CATEGORY} -MET Tool Wrapper Use Cases:: +**OR** + +**CONDITION 2:** MET Tool Wrapper Use Cases:: tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}-${METPLUS_VERSION}.tgz met_test +Remove old data (if applicable) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the pull request notes mention an old directory path that should be removed, +please remove that directory. Be careful not to remove any files that are +still needed. + Update the link in the develop directory if needed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/Contributors_Guide/basic_components.rst b/docs/Contributors_Guide/basic_components.rst index cfced5f7a2..b63f94da91 100644 --- a/docs/Contributors_Guide/basic_components.rst +++ b/docs/Contributors_Guide/basic_components.rst @@ -1,8 +1,9 @@ +******************************************* Basic Components of METplus Python Wrappers -=========================================== +******************************************* CommandBuilder --------------- +============== CommandBuilder is the parent class of all METplus wrappers. Every wrapper is a subclass of CommandBuilder or another subclass of CommandBuilder. For example, GridStatWrapper, PointStatWrapper, EnsembleStatWrapper, and MODEWrapper are all a subclass of CompareGriddedWrapper. CompareGriddedWrapper is a subclass of CommandBuilder. CommandBuilder contains instance variables that are common to every wrapper, such as config (METplusConfig object), errors (a counter of the number of errors that have occurred in the wrapper), and c_dict (a dictionary containing common information). CommandBuilder also contains use class functions that can be called within each wrapper, such as create_c_dict, clear, and find_data. More information regarding these variables and functions can be found in the Doxygen documentation (link?). @@ -54,7 +55,8 @@ create_c_dict (CommandBuilder):: return c_dict isOK class variable -------------------- +=================== + isOK is defined in CommandBuilder (ush/command_builder.py). Its function is to note a failed process while not stopping a parent process. @@ -75,7 +77,8 @@ See MODEWrapper (ush/mode_wrapper.py) for other examples. run_at_time function --------------------- +==================== + run_at_time runs a process for one specific time. This is defined in CommandBuilder. @@ -97,33 +100,38 @@ This is defined in CommandBuilder. See ush/pb2nc_wrapper.py for an example. run_all_times function ----------------------- +====================== + run_all_times loops over a series of times calling run_at_time for one process for each time Defined in CommandBuilder but overridden in a wrappers that process all of the data from every run time at once. See SeriesByLeadWrapper (ush/series_by_lead_wrapper.py) for an example of overridding the function get_command function --------------------- +==================== + get_command assembles a MET command with arguments that can be run via the shell or the wrapper. It is defined in CommandBuilder but is overridden in most wrappers because the command line arguments differ for each MET tool. set_environment_variables function ----------------------------------- +================================== + Uses add_env_var function (CommandBuilder) to set any shell environment variables that MET or other METplus wrappers need to be set. This allows a wrapper to pass information into a MET configuration file. The MET config file refers to the environment variables. This is currently only set in wrappers that use MET config files, but the other wrappers will also need to set environment variables that are needed to be set in the environment when running, such as MET_TMP_DIR and MET_PYTHON_EXE. find_data/find_model/find_obs functions (in CommandBuilder) ------------------------------------------------------------ +=========================================================== + find_* uses the c_dict directory templates and then queries the file system to find the files you are looking for uses c_dict dictionary items [FCST/OBS]_FILE_WINDOW_[BEGIN/END], [FCST/OBS]_INPUT_[DIR/TEMPLATE], etc. If [FCST/OBS]_FILE_WINDOW_[BEGIN/END] are non-zero, these functions will list all files under [FCST/OBS]_INPUT_DIR and use [FCST/OBS]_INPUT_TEMPLATE to extract out time information from each file to determine which files within the file window range should be used. Some tools allow multiple files to be selected. If a tool does not allow multiple files, the file closest to the valid time is returned. If multiple files are the same distance from the valid time, the first file that was found is used. If a wrapper can read in multiple files, the c_dict item 'ALLOW_MULTIPLE_FILES' should be set to True. do_string_sub function ----------------------- +====================== + do_string_sub is found in ush/string_template_substitution.py and is the critical function for substituting the placeholder values in templates with the actual values needed for running a particular wrapper @@ -142,7 +150,8 @@ tc_pairs_wrapper has a good example time_info is a dictionary of current run time information that can be substituted into the template. See the 'Time Utilities' section for more information. Time Utilities --------------- +============== + time_util is a collection of functions to handle the idosyncracies of working with valid, initialization and observation times. METplus creates a dictionary containing the current time and either init or valid time:: @@ -166,3 +175,186 @@ Items that will be parsed from the input dictionary are: now, init, valid, lead, pcp_combine uses a variety of time_util functions like ti_calculate and ti_get_lead_string +Adding Support for MET Configuration Variables +============================================== + +The METplus wrappers utilize environment variables to override values in the +MET configuration files. There are functions in CommandBuilder that can be +used to easily add support for override MET configuration variables that did +were not previously supported in METplus configuration files. + +There is a utility that can be used to easily see what changes are needed to +add support for a new variable. The doc_util.py script can be run from the +command line to output a list of instructions to add new support. It can +be run from the top level of the METplus repository. The script can be called +to add a single MET configuration variable by supplying the MET tool name and +the variable name:: + + ./metplus/util/doc_util.py point_stat sid_exc + +This command will provide guidance for adding support for the sid_exc variable +found in the PointStatConfig file. + +The script can also be called with the name of a dictionary and the names of +each dictionary variable:: + + ./metplus/util/doc_util.py grid_stat distance_map baddeley_p baddeley_max_dist fom_alpha zhu_weight beta_value_n + +This command will provide guidance for adding support for the distance_map +dictionary found in the GridStatConfig file. The list of variables found inside +the distance_map variable follow the dictionary variable name. + +**PLEASE NOTE** that the information output from this script is intended to +assist a developer with adding support, but it cannot be assumed that every +suggestion is correct. Please review the guidance and determine if any +modifications are necessary to properly add support. + +Add Support for Single Item +--------------------------- + +The add_met_config function can be used to set a single MET config variable. +The function takes a few named arguments to determine how the variable +should be set. + +* name: Name of the variable to set, i.e. model +* data_type: Type of variable. Valid options are int, string, list, float, + bool, and thresh +* metplus_configs: List of METplus configuration variable names that should be + checked. Variable names are checked in order that they appear in the list. + If any of the variables are set in the config object, then the value will be + read and the environment variable will be set to override the value. +* env_var_name (optional): Name of environment to set if the MET config + variable should be overridden. Defaults to the name of the variable in all + caps with METPLUS\_ prepended, i.e. METPLUS_MODEL. +* extra_args (optional): Dictionary containing additional information about the + variable. Valid options are described below. + + * remove_quotes: If set to True, do not add quotation marks around value. + Used only if data_type is string or list. + * uppercase: If True, change all letters to capital letters. + Used only if data_type is string or list. + * allow_empty: If True and METplus configuration value is set to an empty + string, override the value to an empty list. This is used if the + value in the default MET config file is not an empty list. + +:: + + self.add_met_config(name='nc_pairs_var_name', + data_type='string', + metplus_configs=['GRID_STAT_NC_PAIRS_VAR_NAME']) + + +Add Support for MET Dictionary +------------------------------ + +The handle_met_config_dict function can be used to easily set a MET config +dictionary variable. The function takes 2 arguments: + +* dict_name: Name of the MET dictionary variable, i.e. distance_map. +* items: Dictionary containing information about the variables that are found + in the dictionary. The key is the name of the variable and the value is + either a string that contains the data type (see data_type above) or a tuple + that contains the data type and additional information about the variable. + +:: + + self.handle_met_config_dict('fcst_genesis', { + 'vmax_thresh': 'thresh', + 'mslp_thresh': 'thresh', + }) + +In the above example, the dictionary variable name is fcst_genesis and it +contains 2 variables inside it, vmax_thresh and mslp_thresh, which are both +threshold values. + +The additional information that can be supplied as a tuple to the value of +each item must be listed in the correct order: +data type, extra info, children, and nicknames. + +* data_type: Type of variable (see data_type above) +* extra: Additional info as a comma separated string (see extra_args above) +* children: Dictionary defining a nested dictionary where the key is the name + of the sub-directory and the value is the item info (see items above) +* nicknames: List of METplus variable names (with app name excluded) to also + search and use if it is set. For example, the GridStat variable mask.poly is + set by the METplus config variable GRID_STAT_MASK_POLY. However, in older + versions of the METplus wrappers, the variable used was + GRID_STAT_VERIFICATION_MASK_TEMPLATE. To preserve support for this name, the + nickname can be set to ['VERIFICATION_MASK_TEMPLATE'] and the old variable + will be checked if GRID_STAT_MASK_POLY is not set. + +Values must be set to None to preserve the order. +For example, if you need to define a nickname but no extra info or children, +then use: ('string', None, None, ['NICKNAME1]). + +If a complex MET configuration dictionary is used by multiple MET tools, then +a function is typically used to handle it. For example, this function is in +CompareGriddedWrapper and is used by GridStat, PointStat, and EnsembleStat:: + + def handle_climo_cdf_dict(self): + self.handle_met_config_dict('climo_cdf', { + 'cdf_bins': ('float', None, None, ['CLIMO_CDF_BINS']), + 'center_bins': 'bool', + 'write_bins': 'bool', + }) + +This function handles setting the climo_cdf dictionary. The METplus config +variable that fits the format {APP_NAME}_{DICTIONARY_NAME}_{VARIABLE_NAME}, +i.e. GRID_STAT_CLIMO_CDF_CDF_BINS for GridStat's climo_cdf.cdf_bins, is +quieried first. However, this default name is a little redundant, so adding +the nickname 'CLIMO_CDF_BINS' allows the user to set the variable +GRID_STAT_CLIMO_CDF_BINS instead. + +There are many MET config dictionaries that only contain beg and end to define +a window. A function in CommandBuilder called handle_met_config_window can be +used to easily set these variable by only supplying the name of the MET +dictionary variable. + +:: + + def handle_met_config_window(self, dict_name): + """! Handle a MET config window dictionary. It is assumed that + the dictionary only contains 'beg' and 'end' entries that are integers. + + @param dict_name name of MET dictionary + """ + self.handle_met_config_dict(dict_name, { + 'beg': 'int', + 'end': 'int', + }) + +This can be called from any wrapper, i.e. TCGen:: + + self.handle_met_config_window('fcst_hr_window') + +This will check if TC_GEN_FCST_HR_WINDOW_BEGIN (or TC_GEN_FCST_HR_WINDOW_BEG) +and TC_GEN_FCST_HR_WINDOW_END are set and override fcst_hr_window.beg and/or +fcst_hr_window.end if so. + +Other functions that are available to handle dictionaries that are common +to multiple MET tools are named starting with "handle\_" including +handle_climo_dict, handle_mask, and handle_interp_dict. + +:: + + def handle_interp_dict(self, uses_field=False): + """! Reads config variables for interp dictionary, i.e. + _INTERP_VLD_THRESH, _INTERP_SHAPE, _INTERP_METHOD, and + _INTERP_WIDTH. Also _INTERP_FIELD if specified + + @param uses_field if True, read field variable as well + (default is False) + """ + items = { + 'vld_thresh': 'float', + 'shape': ('string', 'remove_quotes'), + 'type': ('dict', None, { + 'method': ('string', 'remove_quotes'), + 'width': 'int', + }), + } + if uses_field: + items['field'] = ('string', 'remove_quotes') + + self.handle_met_config_dict('interp', items) + diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst index c2a874e28f..4443cecb70 100644 --- a/docs/Users_Guide/glossary.rst +++ b/docs/Users_Guide/glossary.rst @@ -7899,3 +7899,28 @@ METplus Configuration Glossary Specify the value for 'write_valid' in the MET configuration file for TCPairs. | *Used by:* TCPairs + + GRID_STAT_DISTANCE_MAP_BADDELEY_P + Specify the value for 'distance_map.baddeley_p' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_DISTANCE_MAP_BADDELEY_MAX_DIST + Specify the value for 'distance_map.baddeley_max_dist' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_DISTANCE_MAP_FOM_ALPHA + Specify the value for 'distance_map.fom_alpha' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_DISTANCE_MAP_ZHU_WEIGHT + Specify the value for 'distance_map.zhu_weight' in the MET configuration file for GridStat. + + | *Used by:* GridStat + + GRID_STAT_DISTANCE_MAP_BETA_VALUE_N + Specify the value for 'distance_map.beta_value(n)' in the MET configuration file for GridStat. + + | *Used by:* GridStat diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index ca7a066dc8..432dee32fb 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -2393,6 +2393,11 @@ METplus Configuration | :term:`GRID_STAT_CLIMO_STDEV_DAY_INTERVAL` | :term:`GRID_STAT_CLIMO_STDEV_HOUR_INTERVAL` | :term:`GRID_STAT_HSS_EC_VALUE` +| :term:`GRID_STAT_DISTANCE_MAP_BADDELEY_P` +| :term:`GRID_STAT_DISTANCE_MAP_BADDELEY_MAX_DIST` +| :term:`GRID_STAT_DISTANCE_MAP_FOM_ALPHA` +| :term:`GRID_STAT_DISTANCE_MAP_ZHU_WEIGHT` +| :term:`GRID_STAT_DISTANCE_MAP_BETA_VALUE_N` | :term:`GRID_STAT_MASK_GRID` (optional) | :term:`GRID_STAT_MASK_POLY` (optional) | :term:`GRID_STAT_MET_CONFIG_OVERRIDES` @@ -2840,6 +2845,24 @@ see :ref:`How METplus controls MET config file settings`. * - :term:`GRID_STAT_HSS_EC_VALUE` - hss_ec_value +**${METPLUS_DISTANCE_MAP_DICT}** + +.. list-table:: + :widths: 5 5 + :header-rows: 0 + + * - METplus Config(s) + - MET Config File + * - :term:`GRID_STAT_DISTANCE_MAP_BADDELEY_P` + - distance_map.baddeley_p + * - :term:`GRID_STAT_DISTANCE_MAP_BADDELEY_MAX_DIST` + - distance_map.baddeley_max_dist + * - :term:`GRID_STAT_DISTANCE_MAP_FOM_ALPHA` + - distance_map.fom_alpha + * - :term:`GRID_STAT_DISTANCE_MAP_ZHU_WEIGHT` + - distance_map.zhu_weight + * - :term:`GRID_STAT_DISTANCE_MAP_BETA_VALUE_N` + - distance_map.beta_value(n) .. _make_plots_wrapper: diff --git a/docs/_static/marine_and_coastal-GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.png b/docs/_static/marine_and_coastal-GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.png new file mode 100644 index 0000000000..6ae8af55e6 Binary files /dev/null and b/docs/_static/marine_and_coastal-GridStat_fcstRTOFS_obsGHRSST_climWOA_sst.png differ diff --git a/docs/_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_OPC.png b/docs/_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.png similarity index 100% rename from docs/_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_OPC.png rename to docs/_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.png diff --git a/docs/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.py b/docs/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.py similarity index 92% rename from docs/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.py rename to docs/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.py index de673d34fc..ea52376bb1 100644 --- a/docs/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.py +++ b/docs/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.py @@ -1,8 +1,8 @@ """ -CyclonePlotter: Use Case for OPC (EMC) cyclone data -=================================================== +CyclonePlotter: Extra-TC Tracker and Plotting Capabilities +========================================================== -model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf +model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf """ ########################################### @@ -72,7 +72,7 @@ # with the -c option, i.e. -c /path/to/TCPairs_extra_tropical.conf # # .. highlight:: bash -# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf +# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf ############################################################################## # MET Configuration @@ -101,10 +101,10 @@ # the user input. These storms are put into separate storm files, to better mimic how TC storms are # typically passed to TCPairs. # -# parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py +# parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py # # .. highlight:: python -# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py +# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py # @@ -116,7 +116,7 @@ # # Passing in TCPairs_extra_tropical.conf then a user-specific system configuration file:: # -# run_metplus.py -c /path/to/CyclonePlotter_fcstGFS_obsGFS_OPC.conf -c /path/to/user_system.conf +# run_metplus.py -c /path/to/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf -c /path/to/user_system.conf # # The following METplus configuration variables must be set correctly to run this example.: # @@ -167,7 +167,7 @@ # # # -# sphinx_gallery_thumbnail_path = '_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_OPC.png' +# sphinx_gallery_thumbnail_path = '_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.png' diff --git a/internal_tests/pytests/command_builder/test_command_builder.py b/internal_tests/pytests/command_builder/test_command_builder.py index 3f6c1f8aaf..29fede8943 100644 --- a/internal_tests/pytests/command_builder/test_command_builder.py +++ b/internal_tests/pytests/command_builder/test_command_builder.py @@ -70,7 +70,7 @@ def test_read_climo_file_name(metplus_config, config_overrides, actual_value = cbw.config.getraw('config', f'{prefix}FILE_NAME', '') - assert(actual_value == expected_value) + assert actual_value == expected_value # ------------------------ # test_find_data_no_dated @@ -100,7 +100,7 @@ def test_find_data_no_dated(metplus_config, data_type): pcw.c_dict[f'{data_type}INPUT_DIR'] = pcw.config.getdir('METPLUS_BASE')+"/internal_tests/data/obs" pcw.c_dict[f'{data_type}INPUT_TEMPLATE'] = "{valid?fmt=%Y%m%d}_{valid?fmt=%H%M}" obs_file = pcw.find_data(time_info, v, data_type) - assert(obs_file == pcw.c_dict[f'{data_type}INPUT_DIR']+'/20180201_0045') + assert obs_file == pcw.c_dict[f'{data_type}INPUT_DIR']+'/20180201_0045' # if the input dir/template combination is not a path, then find_data should just return that string @@ -127,7 +127,7 @@ def test_find_data_not_a_path(metplus_config, data_type): pcw.c_dict[f'{data_type}INPUT_DIR'] = '' pcw.c_dict[f'{data_type}INPUT_TEMPLATE'] = 'G003' obs_file = pcw.find_data(time_info, var_info=None, data_type=data_type) - assert(obs_file == 'G003') + assert obs_file == 'G003' def test_find_obs_no_dated(metplus_config): config = metplus_config() @@ -145,7 +145,7 @@ def test_find_obs_no_dated(metplus_config): pcw.c_dict['OBS_INPUT_DIR'] = pcw.config.getdir('METPLUS_BASE') + "/internal_tests/data/obs" pcw.c_dict['OBS_INPUT_TEMPLATE'] = "{valid?fmt=%Y%m%d}_{valid?fmt=%H%M}" obs_file = pcw.find_obs(time_info, v) - assert (obs_file == pcw.c_dict['OBS_INPUT_DIR'] + '/20180201_0045') + assert obs_file == pcw.c_dict['OBS_INPUT_DIR'] + '/20180201_0045' def test_find_obs_dated(metplus_config): config = metplus_config() @@ -163,7 +163,7 @@ def test_find_obs_dated(metplus_config): pcw.c_dict['OBS_INPUT_DIR'] = pcw.config.getdir('METPLUS_BASE')+"/internal_tests/data/obs" pcw.c_dict['OBS_INPUT_TEMPLATE'] = '{valid?fmt=%Y%m%d}/{valid?fmt=%Y%m%d}_{valid?fmt=%H%M}' obs_file = pcw.find_obs(time_info, v) - assert(obs_file == pcw.c_dict['OBS_INPUT_DIR']+'/20180201/20180201_0013') + assert obs_file == pcw.c_dict['OBS_INPUT_DIR']+'/20180201/20180201_0013' @pytest.mark.parametrize( 'offsets, expected_file, offset_seconds', [ @@ -195,9 +195,10 @@ def test_find_obs_offset(metplus_config, offsets, expected_file, offset_seconds) print(f"EXPECTED FILE: {expected_file}") if expected_file is None: - assert(not obs_file) + assert not obs_file else: - assert (os.path.basename(obs_file) == expected_file and time_info['offset'] == offset_seconds) + assert (os.path.basename(obs_file) == expected_file and + time_info['offset'] == offset_seconds) def test_find_obs_dated_previous_day(metplus_config): config = metplus_config() @@ -215,7 +216,7 @@ def test_find_obs_dated_previous_day(metplus_config): pcw.c_dict['OBS_FILE_WINDOW_BEGIN'] = -3600 pcw.c_dict['OBS_FILE_WINDOW_END'] = 0 obs_file = pcw.find_obs(time_info, v) - assert(obs_file == pcw.c_dict['OBS_INPUT_DIR']+'/20180131/20180131_2345') + assert obs_file == pcw.c_dict['OBS_INPUT_DIR']+'/20180131/20180131_2345' def test_find_obs_dated_next_day(metplus_config): config = metplus_config() @@ -233,7 +234,7 @@ def test_find_obs_dated_next_day(metplus_config): pcw.c_dict['OBS_FILE_WINDOW_BEGIN'] = 0 pcw.c_dict['OBS_FILE_WINDOW_END'] = 3600 obs_file = pcw.find_obs(time_info, v) - assert(obs_file == pcw.c_dict['OBS_INPUT_DIR']+'/20180202/20180202_0013') + assert obs_file == pcw.c_dict['OBS_INPUT_DIR']+'/20180202/20180202_0013' @pytest.mark.parametrize( 'overrides, c_dict', [ @@ -251,7 +252,7 @@ def test_override_config_in_c_dict(metplus_config, overrides, c_dict): pcw = CommandBuilder(config, config_overrides=overrides) for key, expected_value in c_dict.items(): - assert(pcw.c_dict.get(key) == expected_value) + assert pcw.c_dict.get(key) == expected_value @pytest.mark.parametrize( 'overrides', [ @@ -266,7 +267,7 @@ def test_override_config(metplus_config, overrides): pcw = CommandBuilder(config, config_overrides=overrides) for key, expected_value in overrides.items(): - assert(pcw.config.getraw('config', key) == expected_value) + assert pcw.config.getraw('config', key) == expected_value # dictionary items with values will be set in [test_section] # items with value None will not be set, so it should use @@ -303,7 +304,7 @@ def test_override_by_instance(metplus_config, section_items): pcw = CommandBuilder(config, instance='test_section') for key, value in section_items.items(): expected_value = 'default' if value is None else value - assert(pcw.config.getraw('config', key) == expected_value) + assert pcw.config.getraw('config', key) == expected_value @pytest.mark.parametrize( 'filename, file_list, output_dir', [ @@ -338,17 +339,17 @@ def test_write_list_file(metplus_config, filename, file_list, output_dir): cbw.write_list_file(filename, file_list, output_dir=output_dir) # ensure file was written - assert(os.path.exists(check_file)) + assert os.path.exists(check_file) with open(check_file, 'r') as file_handle: lines = file_handle.readlines() # ensure number of lines written is 1 greater than provided list # to account for first line that contains 'file_list' text - assert(len(lines) == len(file_list) + 1) + assert len(lines) == len(file_list) + 1 # ensure content of file is as expected for actual_line, expected_line in zip(lines[1:], file_list): - assert(actual_line.strip() == expected_line) + assert actual_line.strip() == expected_line @pytest.mark.parametrize( 'config_overrides, expected_value', [ @@ -377,7 +378,7 @@ def test_handle_description(metplus_config, config_overrides, expected_value): cbw.app_name = 'grid_stat' cbw.handle_description() - assert(cbw.env_var_dict.get('METPLUS_DESC', '') == expected_value) + assert cbw.env_var_dict.get('METPLUS_DESC', '') == expected_value @pytest.mark.parametrize( 'input, output', [ @@ -390,7 +391,7 @@ def test_handle_description(metplus_config, config_overrides, expected_value): ) def test_format_regrid_to_grid(metplus_config, input, output): cbw = CommandBuilder(metplus_config()) - assert(cbw.format_regrid_to_grid(input) == output) + assert cbw.format_regrid_to_grid(input) == output @pytest.mark.parametrize( 'config_overrides, set_to_grid, expected_dict', [ @@ -434,9 +435,9 @@ def test_handle_regrid_old(metplus_config, config_overrides, set_to_grid, cbw.handle_regrid(c_dict, set_to_grid=set_to_grid) - assert(len(c_dict) == len(expected_dict)) + assert len(c_dict) == len(expected_dict) for key, value in expected_dict.items(): - assert(c_dict.get(key, '') == value) + assert c_dict.get(key, '') == value @pytest.mark.parametrize( 'config_overrides, expected_output', [ @@ -477,7 +478,7 @@ def test_handle_regrid_new(metplus_config, config_overrides, expected_output): cbw.app_name = 'app' cbw.handle_regrid(cbw.c_dict) - assert(cbw.env_var_dict['METPLUS_REGRID_DICT'] == expected_output) + assert cbw.env_var_dict['METPLUS_REGRID_DICT'] == expected_output @pytest.mark.parametrize( 'mp_config_name,met_config_name,c_dict_key,remove_quotes,expected_output', [ @@ -516,7 +517,7 @@ def test_set_met_config_string(metplus_config, mp_config_name, met_config_name, if key is None: key = met_config_name.upper() - assert(c_dict.get(key, '') == expected_output) + assert c_dict.get(key, '') == expected_output @pytest.mark.parametrize( 'mp_config_name,met_config_name,c_dict_key,uppercase,expected_output, is_ok', [ @@ -566,8 +567,8 @@ def test_set_met_config_bool(metplus_config, mp_config_name, met_config_name, if key is None: key = met_config_name.upper() - assert(c_dict.get(key, '') == expected_output) - assert(cbw.isOK == is_ok) + assert c_dict.get(key, '') == expected_output + assert cbw.isOK == is_ok # int @pytest.mark.parametrize( @@ -608,8 +609,8 @@ def test_set_met_config_int(metplus_config, mp_config_name, met_config_name, if key is None: key = met_config_name.upper() - assert(c_dict.get(key, '') == expected_output) - assert(cbw.isOK == is_ok) + assert c_dict.get(key, '') == expected_output + assert cbw.isOK == is_ok @pytest.mark.parametrize( 'mp_config_name,met_config_name,c_dict_key,expected_output,is_ok', [ @@ -649,8 +650,8 @@ def test_set_met_config_float(metplus_config, mp_config_name, met_config_name, if key is None: key = met_config_name.upper() - assert(c_dict.get(key, '') == expected_output) - assert(cbw.isOK == is_ok) + assert c_dict.get(key, '') == expected_output + assert cbw.isOK == is_ok @pytest.mark.parametrize( 'mp_config_name,met_config_name,c_dict_key,expected_output,is_ok', [ @@ -698,8 +699,8 @@ def test_set_met_config_thresh(metplus_config, mp_config_name, met_config_name, if key is None: key = met_config_name.upper() - assert(c_dict.get(key, '') == expected_output) - assert(cbw.isOK == is_ok) + assert c_dict.get(key, '') == expected_output + assert cbw.isOK == is_ok @pytest.mark.parametrize( 'mp_config_name,met_config_name,c_dict_key,remove_quotes,expected_output', [ @@ -746,7 +747,7 @@ def test_set_met_config_list(metplus_config, mp_config_name, met_config_name, if key is None: key = met_config_name.upper() - assert(c_dict.get(key, '') == expected_output) + assert c_dict.get(key, '') == expected_output @pytest.mark.parametrize( 'mp_config_name,allow_empty,expected_output', [ @@ -776,7 +777,7 @@ def test_set_met_config_list_allow_empty(metplus_config, mp_config_name, met_config_name, allow_empty=allow_empty) - assert(c_dict.get(mp_config_name, '') == expected_output) + assert c_dict.get(mp_config_name, '') == expected_output @pytest.mark.parametrize( 'data_type, expected_function', [ @@ -805,20 +806,34 @@ def test_handle_met_config_dict(metplus_config): config.set('config', 'TC_GEN_FCST_HR_WINDOW_BEG', beg) config.set('config', 'TC_GEN_FCST_HR_WINDOW_END', end) cbw = CommandBuilder(config) + cbw.app_name = 'tc_gen' - dict_items = [] - item = met_config(name='beg', - data_type='int', - metplus_configs=['TC_GEN_FCST_HR_WINDOW_BEG']) - dict_items.append(item) - item = met_config(name='end', - data_type='int', - metplus_configs=['TC_GEN_FCST_HR_WINDOW_END']) - dict_items.append(item) - - cbw.handle_met_config_dict(dict_name, dict_items) + items = { + 'beg': 'int', + 'end': 'int', + } + + cbw.handle_met_config_dict(dict_name, items) print(f"env_var_dict: {cbw.env_var_dict}") - assert(cbw.env_var_dict.get('METPLUS_FCST_HR_WINDOW_DICT') == expected_value) + actual_value = cbw.env_var_dict.get('METPLUS_FCST_HR_WINDOW_DICT') + assert actual_value == expected_value + +def test_handle_met_config_window(metplus_config): + dict_name = 'fcst_hr_window' + beg = -3 + end = 5 + expected_value = f'{dict_name} = {{beg = -3;end = 5;}}' + + config = metplus_config() + config.set('config', 'TC_GEN_FCST_HR_WINDOW_BEG', beg) + config.set('config', 'TC_GEN_FCST_HR_WINDOW_END', end) + cbw = CommandBuilder(config) + cbw.app_name = 'tc_gen' + + cbw.handle_met_config_window(dict_name) + print(f"env_var_dict: {cbw.env_var_dict}") + actual_value = cbw.env_var_dict.get('METPLUS_FCST_HR_WINDOW_DICT') + assert actual_value == expected_value def test_add_met_config(metplus_config): config = metplus_config() @@ -831,7 +846,7 @@ def test_add_met_config(metplus_config): 'TC_GEN_VALID_FREQ',]) print(f"env_var_dict: {cbw.env_var_dict}") expected_value = f'valid_freq = {value};' - assert(cbw.env_var_dict['METPLUS_VALID_FREQ'] == expected_value) + assert cbw.env_var_dict['METPLUS_VALID_FREQ'] == expected_value def test_handle_met_config_dict_nested(metplus_config): dict_name = 'outer' @@ -851,37 +866,16 @@ def test_handle_met_config_dict_nested(metplus_config): config.set('config', 'APP_OUTER_INNER_VAR1', sub_dict_value1) config.set('config', 'APP_OUTER_INNER_VAR2', sub_dict_value2) cbw = CommandBuilder(config) + cbw.app_name = 'app' - dict_items = [] - item = met_config(name='beg', - data_type='int', - metplus_configs=['APP_OUTER_BEG']) - dict_items.append(item) - item = met_config(name='end', - data_type='int', - metplus_configs=['APP_OUTER_END']) - dict_items.append(item) - - sub_dict_items = [] - item = met_config(name='var1', - data_type='string', - metplus_configs=['APP_OUTER_INNER_VAR1'], - extra_args={'remove_quotes': True}) - sub_dict_items.append(item) - item = met_config(name='var2', - data_type='string', - metplus_configs=['APP_OUTER_INNER_VAR2'], - extra_args={'remove_quotes': True}) - sub_dict_items.append(item) - - dict_items.append( - cbw.get_met_config( - name=sub_dict_name, - data_type='dict', - children=sub_dict_items, - ) - ) + items = { + 'beg': 'int', + 'end': 'int', + 'inner': ('dict', None, {'var1': ('string', 'remove_quotes', None), + 'var2': ('string', 'remove_quotes', None), + }), + } - cbw.handle_met_config_dict(dict_name, dict_items) + cbw.handle_met_config_dict(dict_name, items) print(f"env_var_dict: {cbw.env_var_dict}") - assert(cbw.env_var_dict.get('METPLUS_OUTER_DICT') == expected_value) + assert cbw.env_var_dict.get('METPLUS_OUTER_DICT') == expected_value diff --git a/internal_tests/pytests/grid_stat/test_grid_stat_wrapper.py b/internal_tests/pytests/grid_stat/test_grid_stat_wrapper.py index 5ed1d7a1ce..3fd1e83f5e 100644 --- a/internal_tests/pytests/grid_stat/test_grid_stat_wrapper.py +++ b/internal_tests/pytests/grid_stat/test_grid_stat_wrapper.py @@ -540,6 +540,31 @@ def test_handle_climo_file_variables(metplus_config, config_overrides, {'METPLUS_OBS_FILE_TYPE': 'file_type = NETCDF_NCCF;'}), ({'GRID_STAT_HSS_EC_VALUE': '0.5', }, {'METPLUS_HSS_EC_VALUE': 'hss_ec_value = 0.5;'}), + ({'GRID_STAT_DISTANCE_MAP_BADDELEY_P': '1', }, + {'METPLUS_DISTANCE_MAP_DICT': 'distance_map = {baddeley_p = 1;}'}), + + ({'GRID_STAT_DISTANCE_MAP_BADDELEY_MAX_DIST': '2.3', }, + {'METPLUS_DISTANCE_MAP_DICT': 'distance_map = {baddeley_max_dist = 2.3;}'}), + + ({'GRID_STAT_DISTANCE_MAP_FOM_ALPHA': '4.5', }, + {'METPLUS_DISTANCE_MAP_DICT': 'distance_map = {fom_alpha = 4.5;}'}), + + ({'GRID_STAT_DISTANCE_MAP_ZHU_WEIGHT': '0.5', }, + {'METPLUS_DISTANCE_MAP_DICT': 'distance_map = {zhu_weight = 0.5;}'}), + + ({'GRID_STAT_DISTANCE_MAP_BETA_VALUE_N': 'n * n / 3.0', }, + {'METPLUS_DISTANCE_MAP_DICT': 'distance_map = {beta_value(n) = n * n / 3.0;}'}), + ({ + 'GRID_STAT_DISTANCE_MAP_BADDELEY_P': '1', + 'GRID_STAT_DISTANCE_MAP_BADDELEY_MAX_DIST': '2.3', + 'GRID_STAT_DISTANCE_MAP_FOM_ALPHA': '4.5', + 'GRID_STAT_DISTANCE_MAP_ZHU_WEIGHT': '0.5', + 'GRID_STAT_DISTANCE_MAP_BETA_VALUE_N': 'n * n / 3.0', + }, + {'METPLUS_DISTANCE_MAP_DICT': ('distance_map = {baddeley_p = 1;' + 'baddeley_max_dist = 2.3;' + 'fom_alpha = 4.5;zhu_weight = 0.5;' + 'beta_value(n) = n * n / 3.0;}')}), ] ) diff --git a/internal_tests/use_cases/all_use_cases.txt b/internal_tests/use_cases/all_use_cases.txt index a4b0e59892..d6fb40f187 100644 --- a/internal_tests/use_cases/all_use_cases.txt +++ b/internal_tests/use_cases/all_use_cases.txt @@ -137,6 +137,6 @@ Category: space_weather Category: tc_and_extra_tc 0::TCRMW_fcstGFS_fcstOnly_gonzalo:: model_applications/tc_and_extra_tc/TCRMW_fcstGFS_fcstOnly_gonzalo.conf -1::CyclonePlotter_fcstGFS_obsGFS_OPC:: model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf:: cycloneplotter_env,cartopy +1::CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC:: model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf:: cycloneplotter_env,cartopy 2::UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF:: model_applications/tc_and_extra_tc/UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF.conf,model_applications/tc_and_extra_tc/UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF/ci_overrides.conf 3::GridStat_fcstHAFS_obsTDR_NetCDF:: model_applications/tc_and_extra_tc/GridStat_fcstHAFS_obsTDR_NetCDF.conf:: py_embed diff --git a/metplus/wrappers/command_builder.py b/metplus/wrappers/command_builder.py index 5dcdb119d0..847af7f172 100755 --- a/metplus/wrappers/command_builder.py +++ b/metplus/wrappers/command_builder.py @@ -1757,13 +1757,17 @@ def get_output_prefix(self, time_info=None, set_env_vars=True): def _parse_extra_args(self, extra): """! Check string for extra option keywords and set them to True in - dictionary if they are found. Supports 'remove_quotes' and 'uppercase' + dictionary if they are found. Supports 'remove_quotes', 'uppercase' + and 'allow_empty' @param extra string to parse for keywords @returns dictionary with extra args set if found in string """ extra_args = {} - for extra_option in ['remove_quotes', 'uppercase']: + if not extra: + return extra_args + + for extra_option in ['remove_quotes', 'uppercase', 'allow_empty']: if extra_option in extra: extra_args[extra_option] = True return extra_args @@ -1774,78 +1778,27 @@ def handle_climo_dict(self): by wrapped MET configs pre 4.0 (CLIMO_MEAN_FILE and CLIMO_STDEV_FILE) """ - # define layout of climo_mean and climo_stdev dictionaries - climo_items = { - 'file_name': ('list', '', None), - 'field': ('list', 'remove_quotes', None), - 'regrid': ('dict', '', [ - ('method', 'string', - 'uppercase,remove_quotes'), - ('width', 'int', ''), - ('vld_thresh', 'float', ''), - ('shape', 'string', - 'uppercase,remove_quotes') - ]), - 'time_interp_method': ('string', 'remove_quotes,uppercase', - None), - 'match_month': ('bool', 'uppercase', None), - 'day_interval': ('int', '', None), - 'hour_interval': ('int', '', None), + items = { + 'file_name': 'list', + 'field': ('list', 'remove_quotes'), + 'regrid': ('dict', '', { + 'method': ('string', 'uppercase,remove_quotes'), + 'width': 'int', + 'vld_thresh': 'float', + 'shape': ('string', 'uppercase,remove_quotes'), + }), + 'time_interp_method': ('string', 'remove_quotes,uppercase'), + 'match_month': ('bool', 'uppercase'), + 'day_interval': 'int', + 'hour_interval': 'int', } for climo_type in self.climo_types: dict_name = f'climo_{climo_type.lower()}' - dict_items = [] # make sure _FILE_NAME is set from INPUT_TEMPLATE/DIR if used self.read_climo_file_name(climo_type) - # config prefix i.e GRID_STAT_CLIMO_MEAN_ - metplus_prefix = f'{self.app_name.upper()}_{dict_name.upper()}_' - for name, (data_type, extra, kids) in climo_items.items(): - # config name i.e. GRID_STAT_CLIMO_MEAN_FILE_NAME - metplus_name = f'{metplus_prefix}{name.upper()}' - metplus_configs = [] - - if data_type != 'dict': - children = None - metplus_configs.append(metplus_name) - # if dictionary, read get children from MET config - else: - children = [] - for kid, kid_type, kid_extra in kids: - metplus_configs.append(f'{metplus_name}_{kid.upper()}') - metplus_configs.append( - f'{metplus_prefix}{kid.upper()}') - - kid_args = self._parse_extra_args(kid_extra) - child_item = self.get_met_config( - name=kid, - data_type=kid_type, - metplus_configs=metplus_configs.copy(), - extra_args=kid_args, - ) - children.append(child_item) - - # reset metplus config list for next kid - metplus_configs.clear() - - # set metplus_configs - metplus_configs = None - - # parse extra options - extra_args = self._parse_extra_args(extra) - dict_item = ( - self.get_met_config( - name=name, - data_type=data_type, - metplus_configs=metplus_configs, - extra_args=extra_args, - children=children, - ) - ) - dict_items.append(dict_item) - - self.handle_met_config_dict(dict_name, dict_items) + self.handle_met_config_dict(dict_name, items) # handle deprecated env vars CLIMO_MEAN_FILE and CLIMO_STDEV_FILE # that are used by pre v4.0.0 wrapped MET config files @@ -2108,51 +2061,20 @@ def handle_mask(self, single_value=False, get_flags=False): are allowed. If False, they should be treated as as list @param get_flags if True, read grid_flag and poly_flag values """ - app = self.app_name.upper() - - dict_name = 'mask' - dict_items = [] - data_type = 'string' if single_value else 'list' - dict_items.append( - self.get_met_config( - name='grid', - data_type=data_type, - metplus_configs=[f'{app}_MASK_GRID', - f'{app}_GRID'], - extra_args={'allow_empty': True} - ) - ) + items = { + 'grid': (data_type, 'allow_empty', None, + ['GRID']), + 'poly': (data_type, 'allow_empty', None, + ['VERIFICATION_MASK_TEMPLATE', 'POLY']), + } - dict_items.append( - self.get_met_config( - name='poly', - data_type=data_type, - metplus_configs=[f'{app}_MASK_POLY', - f'{app}_VERIFICATION_MASK_TEMPLATE', - f'{app}_POLY'], - extra_args={'allow_empty': True} - ) - ) - # get grid_flag and poly_flag if requested if get_flags: - dict_items.append( - self.get_met_config(name='grid_flag', - data_type='string', - metplus_configs=[f'{app}_MASK_GRID_FLAG'], - extra_args={'remove_quotes': True, - 'uppercase': True}) - ) - dict_items.append( - self.get_met_config(name='poly_flag', - data_type='string', - metplus_configs=[f'{app}_MASK_POLY_FLAG'], - extra_args={'remove_quotes': True, - 'uppercase': True}) - ) + items['grid_flag'] = ('string', 'remove_quotes,uppercase') + items['poly_flag'] = ('string', 'remove_quotes,uppercase') - self.handle_met_config_dict(dict_name, dict_items) + self.handle_met_config_dict('mask', items) def set_met_config_function(self, item_type): """! Return function to use based on item type @@ -2220,14 +2142,78 @@ def handle_met_config_item(self, item, output_dict=None): **item.extra_args) return True - def handle_met_config_dict(self, dict_name, dict_items, - output_dict=None): + def handle_met_config_dict(self, dict_name, items): """! Read config variables for MET config dictionary and set env_var_dict with formatted values + @params dict_name name of MET dictionary variable + @params items dictionary where the key is name of variable inside MET + dictionary and the value is info about the item (see parse_item_info + function for more information) """ - if output_dict is None: - output_dict = self.env_var_dict + dict_items = [] + + # config prefix i.e GRID_STAT_CLIMO_MEAN_ + metplus_prefix = f'{self.app_name}_{dict_name}_'.upper() + for name, item_info in items.items(): + data_type, extra, kids, nicknames = self.parse_item_info(item_info) + + # config name i.e. GRID_STAT_CLIMO_MEAN_FILE_NAME + metplus_name = f'{metplus_prefix}{name.upper()}' + + # change (n) to _N i.e. distance_map.beta_value(n) + metplus_name = metplus_name.replace('(N)', '_N') + metplus_configs = [] + + if data_type != 'dict': + children = None + # if variable ends with _BEG, read _BEGIN first + if metplus_name.endswith('BEG'): + metplus_configs.append(f'{metplus_name}IN') + + metplus_configs.append(metplus_name) + if nicknames: + for nickname in nicknames: + metplus_configs.append( + f'{self.app_name}_{nickname}'.upper() + ) + + # if dictionary, read get children from MET config + else: + children = [] + for kid_name, kid_info in kids.items(): + kid_upper = kid_name.upper() + kid_type, kid_extra, _, _= self.parse_item_info(kid_info) + + metplus_configs.append(f'{metplus_name}_{kid_upper}') + metplus_configs.append(f'{metplus_prefix}{kid_upper}') + + kid_args = self._parse_extra_args(kid_extra) + child_item = self.get_met_config( + name=kid_name, + data_type=kid_type, + metplus_configs=metplus_configs.copy(), + extra_args=kid_args, + ) + children.append(child_item) + + # reset metplus config list for next kid + metplus_configs.clear() + + # set metplus_configs + metplus_configs = None + + extra_args = self._parse_extra_args(extra) + dict_item = ( + self.get_met_config( + name=name, + data_type=data_type, + metplus_configs=metplus_configs, + extra_args=extra_args, + children=children, + ) + ) + dict_items.append(dict_item) final_met_config = self.get_met_config( name=dict_name, @@ -2235,7 +2221,53 @@ def handle_met_config_dict(self, dict_name, dict_items, children=dict_items, ) - return self.handle_met_config_item(final_met_config, output_dict) + return self.handle_met_config_item(final_met_config, self.env_var_dict) + + @staticmethod + def parse_item_info(item_info): + """! Parses info about a MET config dictionary item. The input can + be a single string that is the data type of the item. It can also be + a tuple containing 2 to 4 values. The additional values must be + supplied in order: + * extra: string of extra information about item, i.e. + 'remove_quotes', 'uppercase', or 'allow_empty' + * kids: dictionary describing child values (used only for dict items) + where the key is the name of the variable and the value is item info + for the child variable in the same format as item_info that is + parsed in this function + * nicknames: list of other METplus config variable name that can be + used to set a value. The app name i.e. GRID_STAT_ is prepended to + each nickname in the list. Used for backwards compatibility for + METplus config variables whose name does not match the MET config + variable name + + @param item_info string or tuple containing information about a + dictionary item + @returns tuple of data type, extra info, children, and nicknames or + None for each tuple value that is not set + """ + if isinstance(item_info, tuple): + data_type, *rest = item_info + else: + data_type = item_info + rest = [] + + extra = rest.pop(0) if rest else None + kids = rest.pop(0) if rest else None + nicknames = rest.pop(0) if rest else None + + return data_type, extra, kids, nicknames + + def handle_met_config_window(self, dict_name): + """! Handle a MET config window dictionary. It is assumed that + the dictionary only contains 'beg' and 'end' entries that are integers. + + @param dict_name name of MET dictionary + """ + self.handle_met_config_dict(dict_name, { + 'beg': 'int', + 'end': 'int', + }) def add_met_config(self, **kwargs): """! Create METConfigInfo object from arguments and process diff --git a/metplus/wrappers/compare_gridded_wrapper.py b/metplus/wrappers/compare_gridded_wrapper.py index 286467c6ba..f065d942bd 100755 --- a/metplus/wrappers/compare_gridded_wrapper.py +++ b/metplus/wrappers/compare_gridded_wrapper.py @@ -400,29 +400,11 @@ def get_command(self): return cmd def handle_climo_cdf_dict(self): - app_name_upper = self.app_name.upper() - tmp_dict = {} - self.set_met_config_float(tmp_dict, - [f'{app_name_upper}_CLIMO_CDF_BINS', - f'{app_name_upper}_CLIMO_CDF_CDF_BINS'], - 'cdf_bins', - 'CLIMO_CDF_BINS') - self.set_met_config_bool(tmp_dict, - f'{app_name_upper}_CLIMO_CDF_CENTER_BINS', - 'center_bins', - 'CLIMO_CDF_CENTER_BINS') - self.set_met_config_bool(tmp_dict, - f'{app_name_upper}_CLIMO_CDF_WRITE_BINS', - 'write_bins', - 'CLIMO_CDF_WRITE_BINS') - climo_cdf = ( - self.format_met_config_dict(tmp_dict, - 'climo_cdf', - ['CLIMO_CDF_BINS', - 'CLIMO_CDF_CENTER_BINS', - 'CLIMO_CDF_WRITE_BINS']) - ) - self.env_var_dict['METPLUS_CLIMO_CDF_DICT'] = climo_cdf + self.handle_met_config_dict('climo_cdf', { + 'cdf_bins': ('float', None, None, ['CLIMO_CDF_BINS']), + 'center_bins': 'bool', + 'write_bins': 'bool', + }) def handle_interp_dict(self, uses_field=False): """! Reads config variables for interp dictionary, i.e. @@ -432,63 +414,15 @@ def handle_interp_dict(self, uses_field=False): @param uses_field if True, read field variable as well (default is False) """ - app = self.app_name.upper() - - dict_name = 'interp' - dict_items = [] - - metplus_prefix = f'{app}_{dict_name.upper()}_' - - # items to set for interp dictionary - # key is MET config name and used for METplus config and env var names - # value is a tuple of data type of item and names of any children items - interp_items = { - 'vld_thresh': ('float', None), - 'shape': ('string', None), - 'type': ('dict', [('method', 'string'), - ('width', 'int')]), + items = { + 'vld_thresh': 'float', + 'shape': ('string', 'remove_quotes'), + 'type': ('dict', None, { + 'method': ('string', 'remove_quotes'), + 'width': 'int', + }), } - if uses_field: - interp_items['field'] = ('string', None) - - for name, (data_type, kids) in interp_items.items(): - metplus_name = f'{metplus_prefix}{name.upper()}' - metplus_configs = [] - - # if dictionary, read get children from MET config - if data_type == 'dict': - children = [] - for kid, kid_type in kids: - # add APP_INTERP_TYPE_METHOD and APP_INTERP_METHOD - metplus_configs.append(f'{metplus_name}_{kid.upper()}') - metplus_configs.append(f'{metplus_prefix}{kid.upper()}') - - child_item = self.get_met_config( - name=kid, - data_type=kid_type, - metplus_configs=metplus_configs.copy(), - extra_args={'remove_quotes': True} - ) - children.append(child_item) - - # reset metplus config list for next kid - metplus_configs.clear() - - # set metplus_configs - metplus_configs = None - else: - children = None - metplus_configs.append(metplus_name) - - dict_items.append( - self.get_met_config( - name=name, - data_type=data_type, - metplus_configs=metplus_configs, - extra_args={'remove_quotes': True}, - children=children, - ) - ) - - self.handle_met_config_dict(dict_name, dict_items) + items['field'] = ('string', 'remove_quotes') + + self.handle_met_config_dict('interp', items) diff --git a/metplus/wrappers/grid_stat_wrapper.py b/metplus/wrappers/grid_stat_wrapper.py index 6bf5d5c8ec..3126fc012b 100755 --- a/metplus/wrappers/grid_stat_wrapper.py +++ b/metplus/wrappers/grid_stat_wrapper.py @@ -48,6 +48,7 @@ class GridStatWrapper(CompareGriddedWrapper): 'METPLUS_FCST_FILE_TYPE', 'METPLUS_OBS_FILE_TYPE', 'METPLUS_HSS_EC_VALUE', + 'METPLUS_DISTANCE_MAP_DICT', ] # handle deprecated env vars used pre v4.0.0 @@ -231,6 +232,13 @@ def create_c_dict(self): data_type='float', metplus_configs=['GRID_STAT_HSS_EC_VALUE']) + self.handle_met_config_dict('distance_map', { + 'baddeley_p': 'int', + 'baddeley_max_dist': 'float', + 'fom_alpha': 'float', + 'zhu_weight': 'float', + 'beta_value(n)': ('string', 'remove_quotes'), + }) return c_dict diff --git a/metplus/wrappers/mode_wrapper.py b/metplus/wrappers/mode_wrapper.py index 5b126d7455..94a608fe42 100755 --- a/metplus/wrappers/mode_wrapper.py +++ b/metplus/wrappers/mode_wrapper.py @@ -58,19 +58,19 @@ class MODEWrapper(CompareGriddedWrapper): 'METPLUS_INTEREST_FUNCTION_CONVEX_HULL_DIST', ] - WEIGHTS = [ - ('centroid_dist', 'float'), - ('boundary_dist', 'float'), - ('convex_hull_dist', 'float'), - ('angle_diff', 'float'), - ('aspect_diff', 'float'), - ('area_ratio', 'float'), - ('int_area_ratio', 'float'), - ('curvature_ratio', 'float'), - ('complexity_ratio', 'float'), - ('inten_perc_ratio', 'float'), - ('inten_perc_value', 'int'), - ] + WEIGHTS = { + 'centroid_dist': 'float', + 'boundary_dist': 'float', + 'convex_hull_dist': 'float', + 'angle_diff': 'float', + 'aspect_diff': 'float', + 'area_ratio': 'float', + 'int_area_ratio': 'float', + 'curvature_ratio': 'float', + 'complexity_ratio': 'float', + 'inten_perc_ratio': 'float', + 'inten_perc_value': 'int', + } NC_PAIRS_FLAGS = [ 'latlon', @@ -261,7 +261,7 @@ def create_c_dict(self): remove_quotes=True, uppercase=True) - self.handle_weight() + self.handle_met_config_dict('weight', self.WEIGHTS) self.handle_flags('nc_pairs') self.add_met_config(name='total_interest_thresh', @@ -324,26 +324,6 @@ def create_c_dict(self): return c_dict - def handle_weight(self): - """! Read weight dictionary values and set them into env_var_list - - """ - app = self.app_name.upper() - - dict_name = 'weight' - dict_items = [] - - for name, data_type in self.WEIGHTS: - dict_items.append( - self.get_met_config( - name=name, - data_type=data_type, - metplus_configs=[f'{app}_WEIGHT_{name.upper()}'], - ) - ) - - self.handle_met_config_dict(dict_name, dict_items) - def set_environment_variables(self, time_info): """!Set environment variables that will be read set when running this tool. diff --git a/metplus/wrappers/tc_gen_wrapper.py b/metplus/wrappers/tc_gen_wrapper.py index 6876731bfd..64eb22c497 100755 --- a/metplus/wrappers/tc_gen_wrapper.py +++ b/metplus/wrappers/tc_gen_wrapper.py @@ -148,12 +148,22 @@ def create_c_dict(self): data_type='int', metplus_configs=['TC_GEN_VALID_FREQUENCY', 'TC_GEN_VALID_FREQ']) - self.handle_fcst_hr_window() + self.handle_met_config_window('fcst_hr_window') self.add_met_config(name='min_duration', data_type='int', metplus_configs=['TC_GEN_MIN_DURATION']) - self.handle_fcst_genesis() - self.handle_best_genesis() + + self.handle_met_config_dict('fcst_genesis', { + 'vmax_thresh': 'thresh', + 'mslp_thresh': 'thresh', + }) + self.handle_met_config_dict('best_genesis', { + 'technique': 'string', + 'category': 'list', + 'vmax_thresh': 'thresh', + 'mslp_thresh': 'thresh', + }) + self.add_met_config(name='oper_technique', data_type='string', metplus_configs=['TC_GEN_OPER_TECHNIQUE']) @@ -210,8 +220,8 @@ def create_c_dict(self): self.add_met_config(name='dev_hit_radius', data_type='int', metplus_configs=['TC_GEN_DEV_HIT_RADIUS']) - self.handle_dev_hit_window() - self.handle_ops_hit_window() + self.handle_met_config_window('dev_hit_window') + self.handle_met_config_window('ops_hit_window') self.add_met_config(name='discard_init_post_genesis_flag', data_type='bool', metplus_configs=[ @@ -250,7 +260,7 @@ def create_c_dict(self): data_type='bool', metplus_configs=['TC_GEN_GENESIS_MATCH_POINT_TO_TRACK'] ) - self.handle_genesis_match_window() + self.handle_met_config_window('genesis_match_window') # get INPUT_TIME_DICT values since wrapper only runs # once (doesn't look over time) @@ -260,116 +270,6 @@ def create_c_dict(self): return c_dict - def handle_fcst_hr_window(self): - dict_name = 'fcst_hr_window' - dict_items = [] - dict_items.append( - self.get_met_config(name='beg', - data_type='int', - metplus_configs=['TC_GEN_FCST_HR_WINDOW_BEGIN', - 'TC_GEN_FCST_HR_WINDOW_BEG', - 'TC_GEN_LEAD_WINDOW_BEGIN', - 'TC_GEN_LEAD_WINDOW_BEG']) - ) - dict_items.append( - self.get_met_config(name='end', - data_type='int', - metplus_configs=['TC_GEN_FCST_HR_WINDOW_END', - 'TC_GEN_LEAD_WINDOW_END']) - ) - self.handle_met_config_dict(dict_name, dict_items) - - def handle_dev_hit_window(self): - dict_name = 'dev_hit_window' - dict_items = [] - dict_items.append( - self.get_met_config(name='beg', - data_type='int', - metplus_configs=['TC_GEN_DEV_HIT_WINDOW_BEGIN', - 'TC_GEN_DEV_HIT_WINDOW_BEG', - 'TC_GEN_GENESIS_WINDOW_BEGIN', - 'TC_GEN_GENESIS_WINDOW_BEG']) - ) - dict_items.append( - self.get_met_config(name='end', - data_type='int', - metplus_configs=['TC_GEN_DEV_HIT_WINDOW_END', - 'TC_GEN_GENESIS_WINDOW_END']) - ) - self.handle_met_config_dict(dict_name, dict_items) - - def handle_ops_hit_window(self): - dict_name = 'ops_hit_window' - dict_items = [] - dict_items.append( - self.get_met_config(name='beg', - data_type='int', - metplus_configs=['TC_GEN_OPS_HIT_WINDOW_BEGIN', - 'TC_GEN_OPS_HIT_WINDOW_BEG',]) - ) - dict_items.append( - self.get_met_config(name='end', - data_type='int', - metplus_configs=['TC_GEN_OPS_HIT_WINDOW_END',]) - ) - self.handle_met_config_dict(dict_name, dict_items) - - def handle_genesis_match_window(self): - dict_name = 'genesis_match_window' - dict_items = [] - dict_items.append( - self.get_met_config(name='beg', - data_type='int', - metplus_configs=['TC_GEN_GENESIS_MATCH_WINDOW_BEGIN', - 'TC_GEN_GENESIS_MATCH_WINDOW_BEG',]) - ) - dict_items.append( - self.get_met_config(name='end', - data_type='int', - metplus_configs=['TC_GEN_GENESIS_MATCH_WINDOW_END',]) - ) - self.handle_met_config_dict(dict_name, dict_items) - - def handle_fcst_genesis(self): - dict_name = 'fcst_genesis' - dict_items = [] - dict_items.append( - self.get_met_config(name='vmax_thresh', - data_type='thresh', - metplus_configs=['TC_GEN_FCST_GENESIS_VMAX_THRESH']) - ) - dict_items.append( - self.get_met_config(name='mslp_thresh', - data_type='thresh', - metplus_configs=['TC_GEN_FCST_GENESIS_MSLP_THRESH']) - ) - self.handle_met_config_dict(dict_name, dict_items) - - def handle_best_genesis(self): - dict_name = 'best_genesis' - dict_items = [] - dict_items.append( - self.get_met_config(name='technique', - data_type='string', - metplus_configs=['TC_GEN_BEST_GENESIS_TECHNIQUE']) - ) - dict_items.append( - self.get_met_config(name='category', - data_type='list', - metplus_configs=['TC_GEN_BEST_GENESIS_CATEGORY']) - ) - dict_items.append( - self.get_met_config(name='vmax_thresh', - data_type='thresh', - metplus_configs=['TC_GEN_BEST_GENESIS_VMAX_THRESH']) - ) - dict_items.append( - self.get_met_config(name='mslp_thresh', - data_type='thresh', - metplus_configs=['TC_GEN_BEST_GENESIS_MSLP_THRESH']) - ) - self.handle_met_config_dict(dict_name, dict_items) - def handle_filter(self): """! find all TC_GEN_FILTER_ values in the config files """ diff --git a/parm/met_config/GridStatConfig_wrapped b/parm/met_config/GridStatConfig_wrapped index 85e95abdd4..9152297a18 100644 --- a/parm/met_config/GridStatConfig_wrapped +++ b/parm/met_config/GridStatConfig_wrapped @@ -156,12 +156,8 @@ gradient = { // Distance Map statistics // May be set separately in each "obs.field" entry // -distance_map = { - baddeley_p = 2; - baddeley_max_dist = NA; - fom_alpha = 0.1; - zhu_weight = 0.5; -} +//distance_map = { +${METPLUS_DISTANCE_MAP_DICT} //////////////////////////////////////////////////////////////////////////////// diff --git a/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf b/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf index 852dc69786..d9d6d475ef 100644 --- a/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf +++ b/parm/use_cases/met_tool_wrapper/GridStat/GridStat.conf @@ -1,150 +1,104 @@ -# GridStat METplus Configuration - -# section heading for [config] variables - all items below this line and -# before the next section heading correspond to the [config] section [config] -# List of applications to run - only GridStat for this case PROCESS_LIST = GridStat -# time looping - options are INIT, VALID, RETRO, and REALTIME -# If set to INIT or RETRO: -# INIT_TIME_FMT, INIT_BEG, INIT_END, and INIT_INCREMENT must also be set -# If set to VALID or REALTIME: -# VALID_TIME_FMT, VALID_BEG, VALID_END, and VALID_INCREMENT must also be set -LOOP_BY = INIT +### +# Time Info +### -# Format of INIT_BEG and INT_END using % items -# %Y = 4 digit year, %m = 2 digit month, %d = 2 digit day, etc. -# see www.strftime.org for more information -# %Y%m%d%H expands to YYYYMMDDHH +LOOP_BY = INIT INIT_TIME_FMT = %Y%m%d%H - -# Start time for METplus run - must match INIT_TIME_FMT INIT_BEG=2005080700 - -# End time for METplus run - must match INIT_TIME_FMT INIT_END=2005080700 - -# Increment between METplus runs (in seconds if no units are specified) -# Must be >= 60 seconds INIT_INCREMENT = 12H -# List of forecast leads to process for each run time (init or valid) -# In hours if units are not specified -# If unset, defaults to 0 (don't loop through forecast leads) LEAD_SEQ = 12 -# Order of loops to process data - Options are times, processes -# Not relevant if only one item is in the PROCESS_LIST -# times = run all wrappers in the PROCESS_LIST for a single run time, then -# increment the run time and run all wrappers again until all times have -# been evaluated. -# processes = run the first wrapper in the PROCESS_LIST for all times -# specified, then repeat for the next item in the PROCESS_LIST until all -# wrappers have been run LOOP_ORDER = times -# Verbosity of MET output - overrides LOG_VERBOSITY for GridStat only -#LOG_GRID_STAT_VERBOSITY = 2 +### +# File I/O +### -# Location of MET config file to pass to GridStat -GRID_STAT_CONFIG_FILE = {PARM_BASE}/met_config/GridStatConfig_wrapped +FCST_GRID_STAT_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst +FCST_GRID_STAT_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/wrfprs_ruc13_{lead?fmt=%HH}.tm00_G212 -# grid to remap data. Value is set as the 'to_grid' variable in the 'regrid' dictionary -# See MET User's Guide for more information -GRID_STAT_REGRID_TO_GRID = NONE +OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}/met_test/new +OBS_GRID_STAT_INPUT_TEMPLATE = ST2ml{valid?fmt=%Y%m%d%H}_A03h.nc -#GRID_STAT_INTERP_FIELD = -#GRID_STAT_INTERP_VLD_THRESH = -#GRID_STAT_INTERP_SHAPE = -#GRID_STAT_INTERP_TYPE_METHOD = -#GRID_STAT_INTERP_TYPE_WIDTH = +GRID_STAT_CLIMO_MEAN_INPUT_DIR = +GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = -#GRID_STAT_NC_PAIRS_VAR_NAME = +GRID_STAT_CLIMO_STDEV_INPUT_DIR = +GRID_STAT_CLIMO_STDEV_INPUT_TEMPLATE = -#GRID_STAT_CLIMO_MEAN_TIME_INTERP_METHOD = -#GRID_STAT_CLIMO_STDEV_TIME_INTERP_METHOD = +GRID_STAT_OUTPUT_DIR = {OUTPUT_BASE}/met_tool_wrapper/GridStat/GridStat +GRID_STAT_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H} -#GRID_STAT_GRID_WEIGHT_FLAG = -# Name to identify model (forecast) data in output -MODEL = WRF +### +# Field Info +### -# Name to identify observation data in output +MODEL = WRF OBTYPE = MC_PCP -# set the desc value in the GridStat MET config file -GRID_STAT_DESC = NA +FCST_VAR1_NAME = APCP +FCST_VAR1_LEVELS = A03 +FCST_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2 -# List of variables to compare in GridStat - FCST_VAR1 variables correspond -# to OBS_VAR1 variables -# Note [FCST/OBS/BOTH]_GRID_STAT_VAR_NAME can be used instead if different evaluations -# are needed for different tools +OBS_VAR1_NAME = APCP_03 +OBS_VAR1_LEVELS = "(*,*)" +OBS_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2 -# Name of forecast variable 1 -FCST_VAR1_NAME = APCP -# List of levels to evaluate for forecast variable 1 -# A03 = 3 hour accumulation in GRIB file -FCST_VAR1_LEVELS = A03 -# List of thresholds to evaluate for each name/level combination for -# forecast variable 1 -FCST_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2 +### +# GridStat +### + +#LOG_GRID_STAT_VERBOSITY = 2 + +GRID_STAT_CONFIG_FILE = {PARM_BASE}/met_config/GridStatConfig_wrapped #FCST_GRID_STAT_FILE_TYPE = +#OBS_GRID_STAT_FILE_TYPE = -# Name of observation variable 1 -OBS_VAR1_NAME = APCP_03 +GRID_STAT_REGRID_TO_GRID = NONE -# List of levels to evaluate for observation variable 1 -# (*,*) is NetCDF notation - must include quotes around these values! -# must be the same length as FCST_VAR1_LEVELS -OBS_VAR1_LEVELS = "(*,*)" +#GRID_STAT_INTERP_FIELD = +#GRID_STAT_INTERP_VLD_THRESH = +#GRID_STAT_INTERP_SHAPE = +#GRID_STAT_INTERP_TYPE_METHOD = +#GRID_STAT_INTERP_TYPE_WIDTH = -# List of thresholds to evaluate for each name/level combination for -# observation variable 1 -OBS_VAR1_THRESH = gt12.7, gt25.4, gt50.8, gt76.2 +#GRID_STAT_NC_PAIRS_VAR_NAME = -#OBS_GRID_STAT_FILE_TYPE = +#GRID_STAT_CLIMO_MEAN_TIME_INTERP_METHOD = +#GRID_STAT_CLIMO_STDEV_TIME_INTERP_METHOD = + +#GRID_STAT_GRID_WEIGHT_FLAG = + +GRID_STAT_DESC = NA -# Time relative to valid time (in seconds) to allow files to be considered -# valid. Set both BEGIN and END to 0 to require the exact time in the filename -# Not used in this example. FCST_GRID_STAT_FILE_WINDOW_BEGIN = 0 FCST_GRID_STAT_FILE_WINDOW_END = 0 OBS_GRID_STAT_FILE_WINDOW_BEGIN = 0 OBS_GRID_STAT_FILE_WINDOW_END = 0 -# MET GridStat neighborhood values -# See the MET User's Guide GridStat section for more information - -# width value passed to nbrhd dictionary in the MET config file GRID_STAT_NEIGHBORHOOD_WIDTH = 1 - -# shape value passed to nbrhd dictionary in the MET config file GRID_STAT_NEIGHBORHOOD_SHAPE = SQUARE -# cov thresh list passed to nbrhd dictionary in the MET config file GRID_STAT_NEIGHBORHOOD_COV_THRESH = >=0.5 -# Set to true to run GridStat separately for each field specified -# Set to false to create one run of GridStat per run time that -# includes all fields specified. GRID_STAT_ONCE_PER_FIELD = False -# Set to true if forecast data is probabilistic FCST_IS_PROB = false -# Only used if FCST_IS_PROB is true - sets probabilistic threshold FCST_GRID_STAT_PROB_THRESH = ==0.1 -# Set to true if observation data is probabilistic -# Only used if configuring forecast data as the 'OBS' input OBS_IS_PROB = false -# Only used if OBS_IS_PROB is true - sets probabilistic threshold OBS_GRID_STAT_PROB_THRESH = ==0.1 GRID_STAT_OUTPUT_PREFIX = {MODEL}_{CURRENT_FCST_NAME}_vs_{OBTYPE}_{CURRENT_OBS_NAME} @@ -212,46 +166,11 @@ GRID_STAT_NC_PAIRS_FLAG_APPLY_MASK = FALSE #GRID_STAT_HSS_EC_VALUE = -# End of [config] section and start of [dir] section -[dir] - -# directory containing forecast input to GridStat -FCST_GRID_STAT_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst - -# directory containing observation input to GridStat -OBS_GRID_STAT_INPUT_DIR = {INPUT_BASE}/met_test/new - -# directory containing climatology mean input to GridStat -# Not used in this example -GRID_STAT_CLIMO_MEAN_INPUT_DIR = - -# directory containing climatology mean input to GridStat -# Not used in this example -GRID_STAT_CLIMO_STDEV_INPUT_DIR = - -# directory to write output from GridStat -GRID_STAT_OUTPUT_DIR = {OUTPUT_BASE}/met_tool_wrapper/GridStat/GridStat - -# End of [dir] section and start of [filename_templates] section -[filename_templates] - -# Template to look for forecast input to GridStat relative to FCST_GRID_STAT_INPUT_DIR -FCST_GRID_STAT_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/wrfprs_ruc13_{lead?fmt=%HH}.tm00_G212 - -# Template to look for observation input to GridStat relative to OBS_GRID_STAT_INPUT_DIR -OBS_GRID_STAT_INPUT_TEMPLATE = ST2ml{valid?fmt=%Y%m%d%H}_A03h.nc - -# Optional subdirectories relative to GRID_STAT_OUTPUT_DIR to write output from GridStat -GRID_STAT_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H} - -# Template to look for climatology input to GridStat relative to GRID_STAT_CLIMO_MEAN_INPUT_DIR -# Not used in this example -GRID_STAT_CLIMO_MEAN_INPUT_TEMPLATE = - -# Template to look for climatology input to GridStat relative to GRID_STAT_CLIMO_STDEV_INPUT_DIR -# Not used in this example -GRID_STAT_CLIMO_STDEV_INPUT_TEMPLATE = +#GRID_STAT_MASK_GRID = +#GRID_STAT_MASK_POLY = -# Used to specify one or more verification mask files for GridStat -# Not used for this example -GRID_STAT_VERIFICATION_MASK_TEMPLATE = +#GRID_STAT_DISTANCE_MAP_BADDELEY_P = +#GRID_STAT_DISTANCE_MAP_BADDELEY_MAX_DIST = +#GRID_STAT_DISTANCE_MAP_FOM_ALPHA = +#GRID_STAT_DISTANCE_MAP_ZHU_WEIGHT = +#GRID_STAT_DISTANCE_MAP_BETA_VALUE_N = diff --git a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_WeatherRegime/WeatherRegime_driver.py b/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_WeatherRegime/WeatherRegime_driver.py index 766fa4881c..790d0c1fca 100755 --- a/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_WeatherRegime/WeatherRegime_driver.py +++ b/parm/use_cases/model_applications/s2s/UserScript_obsERA_obsOnly_WeatherRegime/WeatherRegime_driver.py @@ -6,7 +6,6 @@ import warnings from WeatherRegime import WeatherRegimeCalculation -from metplus.util import getlist from metplotpy.contributed.weather_regime import plot_weather_regime as pwr from Blocking_WeatherRegime_util import parse_steps, read_nc_met, write_mpr_file @@ -112,7 +111,7 @@ def main(): if not ("EOF" in steps_list_obs): raise Exception('Must run observed EOFs before plotting observed EOFs.') print('Plotting Obs EOFs') - pltlvls_str = getlist(os.environ['EOF_PLOT_LEVELS']) + pltlvls_str = os.environ['EOF_PLOT_LEVELS'].split(',') pltlvls = [float(pp) for pp in pltlvls_str] eof_plot_outname = os.path.join(oplot_dir,os.environ.get('OBS_EOF_PLOT_OUTPUT_NAME','obs_eof')) pwr.plot_eof(eof_obs,wrnum_obs,variance_fractions_obs,lons_obs,lats_obs,eof_plot_outname,pltlvls) @@ -121,7 +120,7 @@ def main(): if not ("EOF" in steps_list_fcst): raise Exception('Must run forecast EOFs before plotting forecast EOFs.') print('Plotting Forecast EOFs') - pltlvls_str = getlist(os.environ['EOF_PLOT_LEVELS']) + pltlvls_str = os.environ['EOF_PLOT_LEVELS'].split(',') pltlvls = [float(pp) for pp in pltlvls_str] eof_plot_outname = os.path.join(oplot_dir,os.environ.get('FCST_EOF_PLOT_OUTPUT_NAME','fcst_eof')) pwr.plot_eof(eof_fcst,wrnum_fcst,variance_fractions_fcst,lons_fcst,lats_fcst,eof_plot_outname,pltlvls) @@ -153,7 +152,7 @@ def main(): if not ("KMEANS" in steps_list_obs): raise Exception('Must run observed Kmeans before plotting observed Kmeans.') print('Plotting Obs K Means') - pltlvls_str = getlist(os.environ['KMEANS_PLOT_LEVELS']) + pltlvls_str = os.environ['KMEANS_PLOT_LEVELS'].split(',') pltlvls = [float(pp) for pp in pltlvls_str] kmeans_plot_outname = os.path.join(oplot_dir,os.environ.get('OBS_KMEANS_PLOT_OUTPUT_NAME','obs_kmeans')) pwr.plot_K_means(kmeans_obs,wrnum_obs,lons_obs,lats_obs,perc_obs,kmeans_plot_outname,pltlvls) @@ -162,7 +161,7 @@ def main(): if not ("KMEANS" in steps_list_fcst): raise Exception('Must run forecast Kmeans before plotting forecast Kmeans.') print('Plotting Forecast K Means') - pltlvls_str = getlist(os.environ['KMEANS_PLOT_LEVELS']) + pltlvls_str = os.environ['KMEANS_PLOT_LEVELS'].split(',') pltlvls = [float(pp) for pp in pltlvls_str] kmeans_plot_outname = os.path.join(oplot_dir,os.environ.get('FCST_KMEANS_PLOT_OUTPUT_NAME','fcst_kmeans')) pwr.plot_K_means(kmeans_fcst,wrnum_fcst,lons_fcst,lats_fcst,perc_fcst,kmeans_plot_outname,pltlvls) diff --git a/parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf b/parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf similarity index 96% rename from parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf rename to parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf index 22020eb49a..69d33171dd 100644 --- a/parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf +++ b/parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf @@ -26,9 +26,9 @@ INIT_INCREMENT = 21600 USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID -USER_SCRIPT_PATH = {PARM_BASE}/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py +USER_SCRIPT_PATH = {PARM_BASE}/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py -USER_SCRIPT_INPUT_PATH = {INPUT_BASE}/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/trak.gfso.atcf_gen.glbl.{init?fmt=%Y} +USER_SCRIPT_INPUT_PATH = {INPUT_BASE}/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/trak.gfso.atcf_gen.glbl.{init?fmt=%Y} USER_SCRIPT_COMMAND = {USER_SCRIPT_PATH} {USER_SCRIPT_INPUT_PATH} {USER_SCRIPT_OUTPUT_DIR} {init?fmt=%Y%m%d%H} diff --git a/parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py b/parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py similarity index 100% rename from parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py rename to parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py