Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature 1187 name update #1193

Merged
merged 6 commits into from
Oct 6, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 110 additions & 23 deletions docs/Contributors_Guide/add_use_case.rst
Original file line number Diff line number Diff line change
Expand Up @@ -542,15 +542,27 @@ or develop directories.
Add contents of existing tarfile to feature branch directory (if applicable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**If you have determined that there is an existing tarfile for the category
(from the previous step)**, then untar the sample data tarball into
the feature branch directory. If no tarfile exists yet, you can skip this
step::
**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!**

**CONDITION 1: IF you have determined that there is an existing tarfile
for the category (from the previous step)**,
then untar the sample data tarball into the feature branch directory::

tar zxf ${METPLUS_EXISTING_DATA_TARFILE} -C ${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}

Create the new tarfile
^^^^^^^^^^^^^^^^^^^^^^
**CONDITION 2: If no tarfile exists yet, you can skip this step**

Rename or modify existing data or data structure (if applicable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**If the reason for your feature branch is to adjust an existing use case, such as renaming a use case
or changing the data file,** then adjust the directory structure and/or the data files which should now
be in your feature branch directory (from your last step). Changes to a use case name or input data for
a preexisting use case should be separately verified to run successfully, and noted in the Pull Request form
(described later).

Add new data to feature branch directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Untar the new data tarball into the feature branch directory::

Expand All @@ -559,13 +571,17 @@ Untar the new data tarball into the feature branch directory::
Verify that all of the old and new data exists in the directory that was
created (i.e. model_applications/<category>).

Create the new tarfile
^^^^^^^^^^^^^^^^^^^^^^
Create the new sample data tarball.

Model Application Use Case Example::
**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!**

**CONDITION 1:** Model Application Use Case Example::

tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}.tgz model_applications/${METPLUS_USE_CASE_CATEGORY}

MET Tool Wrapper Use Case Example::
**CONDITION 2:** MET Tool Wrapper Use Case Example::

tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}.tgz met_test

Expand Down Expand Up @@ -699,8 +715,8 @@ dependencies.
Dependencies
^^^^^^^^^^^^

Environments
""""""""""""
Conda Environments
""""""""""""""""""

The keywords that end with "_env" are Python environments created in Docker
images using Conda that can be used to run use cases. These images are stored
Expand Down Expand Up @@ -789,7 +805,59 @@ environments, refer to the comments in the scripts found in
developer, so please contact MET Help if none of these environments contain the
package requirements needed to run a new use case.

**MORE INFO COMING SOON!**
A README file can be found in the ci/docker/docker_env directory that
provides commands that can be run to recreate a Docker image if the
conda environment needs to be updated. Please note that Docker must
be installed on the workstation used to create new Docker images and
a DockerHub account with access to the dtcenter repositories must
be used to push Docker images to DockerHub.

The README file also contains commands to create a conda environment
that is used for the tests locally. Any base conda environments,
such as metplus_base and py_embed_base, must be created locally first
before creating an environment that builds upon these environments.
Please note that some commands in the scripts are specific to
the Docker environment and may need to be rerun to successfully
build the environment locally.

**Installing METplus Components**

These scripts
do not install any METplus components,
such as metplotpy/metcalcpy/metplus, in the Python environment that
may be needed for a use case. This is done because the automated tests
will install and use the latest version (develop) of the packages to
ensure that any changes to those components do not break any existing
use cases. These packages will need to be installed by the user
and need to be updated manually. To install these packages,
activate the Conda environment, obtain the source code from GitHub,
and run "pip3 install ." in the top level directory of the repository.

Example::

conda activate weatherregime
git clone [email protected]:dtcenter/METplotpy
cd METplotpy
git checkout develop
git pull
pip3 install .

**Cartopy Shapefiles**

The cartopy python package automatically attempts to download
shapefiles as needed.
The URL that is used in cartopy version 0.18.0 and earlier no longer
exists, so use cases that needs these files will fail if they are
not found locally. If a conda environment uses cartopy, these
shapefiles may need to be downloaded by the user running the use case
even if the conda environment was created by another user.
Cartopy provides a script that can be used to obtain these shapefiles
from the updated URL::

wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
python3 cartopy_feature_download.py cultural physical cultural-extra



.. _add_new_category_to_test_runs:

Expand Down Expand Up @@ -1034,27 +1102,33 @@ Compare the feature branch file to the develop directory file::

diff ${METPLUS_FEATURE_BRANCH}/volume_mount_directories develop/volume_mount_directories

**IF there is a new entry or change in the feature version**,
**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!**

**CONDITION 1: IF there is a new entry or change in the feature version**,
copy the feature file into the develop directory::

cp ${METPLUS_FEATURE_BRANCH}/volume_mount_directories develop/volume_mount_directories

Copy the data from the feature directory into the next version directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Copy data from the feature directory into the next version directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Make sure the paths are correct before copying.
**Make sure the paths are correct before copying.**

Model Applications Use Cases::
**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!**

from_directory=${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}/model_applications/${METPLUS_USE_CASE_CATEGORY}
**CONDITION 1:** Model Applications Use Cases::

from_directory=${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}/model_applications/${METPLUS_USE_CASE_CATEGORY}/${METPLUS_USE_CASE_NAME}
echo $from_directory
ls $from_directory

to_directory=${METPLUS_DATA_TARFILE_DIR}/v${METPLUS_VERSION}/model_applications/${METPLUS_USE_CASE_CATEGORY}
echo $to_directory
ls $to_directory

MET Tool Wrapper Use Cases::
**OR**

**CONDITION 2:** MET Tool Wrapper Use Cases::

from_directory=${METPLUS_DATA_TARFILE_DIR}/${METPLUS_FEATURE_BRANCH}/met_test
echo $from_directory
Expand All @@ -1066,36 +1140,49 @@ MET Tool Wrapper Use Cases::

Once you have verified the correct directories are set, copy the files::

cp -r $from_directory/* $to_directory/
cp -r $from_directory $to_directory/

List the tarfile for the use case category in the next release version directory::

cd ${METPLUS_DATA_TARFILE_DIR}/v${METPLUS_VERSION}
ls -lh sample_data-${METPLUS_USE_CASE_CATEGORY}*

**IF the latest version of the tarfile is in this directory**,
**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!**

**CONDITION 1: IF the latest version of the tarfile is in this directory**,
then rename the existing sample data tarball for
the use case category just in case something goes wrong::

mv sample_data-${METPLUS_USE_CASE_CATEGORY}-${METPLUS_VERSION}.tgz sample_data-${METPLUS_USE_CASE_CATEGORY}-${METPLUS_VERSION}.sav.`date +%Y%m%d%H%M`.tgz

**OR**

**IF the sample data tarfile for the category is a link to another METplus
**CONDITION 2: IF the sample data tarfile for the category is a link to another METplus
version**, then simply remove the tarfile link::

unlink sample_data-${METPLUS_USE_CASE_CATEGORY}.tgz

Create the new sample data tarfile.

Model Applications Use Cases::
**ONLY RUN THE COMMAND THAT IS APPROPRIATE TO YOUR USE CASE. READ CAREFULLY!**

**CONDITION 1:** Model Applications Use Cases::

tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}-${METPLUS_VERSION}.tgz model_applications/${METPLUS_USE_CASE_CATEGORY}

MET Tool Wrapper Use Cases::
**OR**

**CONDITION 2:** MET Tool Wrapper Use Cases::

tar czf sample_data-${METPLUS_USE_CASE_CATEGORY}-${METPLUS_VERSION}.tgz met_test

Remove old data (if applicable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If the pull request notes mention an old directory path that should be removed,
please remove that directory. Be careful not to remove any files that are
still needed.

Update the link in the develop directory if needed
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
"""
CyclonePlotter: Use Case for OPC (EMC) cyclone data
===================================================
CyclonePlotter: Extra-TC Tracker and Plotting Capabilities
==========================================================

model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf
model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf

"""
###########################################
Expand Down Expand Up @@ -72,7 +72,7 @@
# with the -c option, i.e. -c /path/to/TCPairs_extra_tropical.conf
#
# .. highlight:: bash
# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf
# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf

##############################################################################
# MET Configuration
Expand Down Expand Up @@ -101,10 +101,10 @@
# the user input. These storms are put into separate storm files, to better mimic how TC storms are
# typically passed to TCPairs.
#
# parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py
# parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py
#
# .. highlight:: python
# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py
# .. literalinclude:: ../../../../parm/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py
#


Expand All @@ -116,7 +116,7 @@
#
# Passing in TCPairs_extra_tropical.conf then a user-specific system configuration file::
#
# run_metplus.py -c /path/to/CyclonePlotter_fcstGFS_obsGFS_OPC.conf -c /path/to/user_system.conf
# run_metplus.py -c /path/to/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf -c /path/to/user_system.conf
#
# The following METplus configuration variables must be set correctly to run this example.:
#
Expand Down Expand Up @@ -167,7 +167,7 @@
#
#
#
# sphinx_gallery_thumbnail_path = '_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_OPC.png'
# sphinx_gallery_thumbnail_path = '_static/tc_and_extra_tc-CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.png'



Expand Down
2 changes: 1 addition & 1 deletion internal_tests/use_cases/all_use_cases.txt
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,6 @@ Category: space_weather

Category: tc_and_extra_tc
0::TCRMW_fcstGFS_fcstOnly_gonzalo:: model_applications/tc_and_extra_tc/TCRMW_fcstGFS_fcstOnly_gonzalo.conf
1::CyclonePlotter_fcstGFS_obsGFS_OPC:: model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf:: cycloneplotter_env,cartopy
1::CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC:: model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC.conf:: cycloneplotter_env,cartopy
2::UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF:: model_applications/tc_and_extra_tc/UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF.conf,model_applications/tc_and_extra_tc/UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF/ci_overrides.conf
3::GridStat_fcstHAFS_obsTDR_NetCDF:: model_applications/tc_and_extra_tc/GridStat_fcstHAFS_obsTDR_NetCDF.conf:: py_embed
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ INIT_INCREMENT = 21600

USER_SCRIPT_RUNTIME_FREQ = RUN_ONCE_PER_INIT_OR_VALID

USER_SCRIPT_PATH = {PARM_BASE}/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/extract_opc_decks.py
USER_SCRIPT_PATH = {PARM_BASE}/use_cases/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/extract_opc_decks.py

USER_SCRIPT_INPUT_PATH = {INPUT_BASE}/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC/trak.gfso.atcf_gen.glbl.{init?fmt=%Y}
USER_SCRIPT_INPUT_PATH = {INPUT_BASE}/model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_UserScript_ExtraTC/trak.gfso.atcf_gen.glbl.{init?fmt=%Y}

USER_SCRIPT_COMMAND = {USER_SCRIPT_PATH} {USER_SCRIPT_INPUT_PATH} {USER_SCRIPT_OUTPUT_DIR} {init?fmt=%Y%m%d%H}

Expand Down