feature 1147 final conf sorting (#1152)

* removed config variable that is no longer used/needed

* cleaned up functions that handle setting CURRENT_[FCST/OBS]_[NAME/LEVEL]

* clean up

* expand path of config argument to run_metplus.py for easier debugging

* removed redundant logic and added comments

* cleanup

* added logic to move config variables that are runtime specific to a [runtime] section in the final conf that is written after the run completes

* changed how config arguments are read so that order that arguments are found is preserved instead of setting explicit overrides at the end

* minor change to trigger build, ci-run-all-cases

* rearranged logic to save list of config inputs with full paths for easier debugging

* move all config vars that start with LOG or end with BASE to runtime section

* remove vars that start with CURRENT from final conf

* removed confusing logic allowing override of parm base and cleaned up logging to be easier to read/understand

* added METPLUS_PARM_BASE env var logic back

* added old vs. new default value mismatch info for pb_report_type in PB2NC config file

* renamed config var name

* use PARM_BASE instead of METPLUS_BASE/parm since it will still be set

* use develop version of METviewer docker image (may need to make this configurable in testing.yml in the future)

* run cyclone plotter use case to test that new conda environment in docker image now contains missing cartopy map files

* added dockerfile to copy cartopy shape files into run image if cartopy is listed as dependencies

* download script used to download cartopy shape files and run it

* added copy of conda environment

* added logic to download cartopy shape files for metplotpy environment

* added cartopy dependency to use cases so the shapefiles are copied into run environment

* trigger 1 use case that uses metplotpy_env to test cartopy shapefiles are found properly

* added instructions to create weatherregime environment that was missing

* cleaned up list of packages used

* removed extra use case group used for testing

* removed netcdf package from env because it is already installed and version number was causing a conflict

* added cartopy shapefile copy to use case that needs it

* download cartopy shape files for weatherregime python environment because files were not found from base env (metplotpy)

* removed debug message that fills up log files with info that is not very useful

* updated input directories to use _WeatherRegime directory instead of _Blocking

* updated location of input data that was not found in UserScript_fcstGFS_obsERA_WeatherRegime

* updated readme with info on how to install metplotpy/metcalcpy in conda environments

* added 3rd group of cartopy shapefiles to download because they are used in weatherregime use cases

* try using cartopy download script from forked repo that has missing shapefile zip file added

* updated input paths to match new data locations

.github/actions/run_tests/Dockerfile.run

@@ -7,5 +7,4 @@ ARG METPLUS_IMG_TAG=develop
 FROM dtcenter/metplus-dev:${METPLUS_IMG_TAG}
 
 COPY --from=env /usr/local/envs /usr/local/envs/
-
 COPY --from=env /usr/local/bin/conda /usr/local/bin/conda

.github/actions/run_tests/Dockerfile.run_cartopy

@@ -0,0 +1,12 @@
+ARG METPLUS_ENV_TAG=cycloneplotter
+ARG METPLUS_IMG_TAG=develop
+
+FROM dtcenter/metplus-envs:${METPLUS_ENV_TAG} as env
+
+ARG METPLUS_IMG_TAG=develop
+FROM dtcenter/metplus-dev:${METPLUS_IMG_TAG}
+
+COPY --from=env /usr/local/envs /usr/local/envs/
+COPY --from=env /usr/local/bin/conda /usr/local/bin/conda
+
+COPY --from=env /root/.local/share/cartopy /root/.local/share/cartopy/

.github/jobs/get_metviewer.sh

@@ -4,7 +4,7 @@
 export METVIEWER_DATA=$RUNNER_WORKSPACE
 export MYSQL_DIR=$RUNNER_WORKSPACE/mysql
 export METVIEWER_DIR=$RUNNER_WORKSPACE/output/metviewer
-export METVIEWER_DOCKER_IMAGE=dtcenter/metviewer
+export METVIEWER_DOCKER_IMAGE=dtcenter/metviewer:develop
 
 # install docker-compose
 apk add docker-compose

.github/jobs/setup_and_run_use_cases.py

@@ -76,6 +76,8 @@ def main():
             dockerfile_name = f'{dockerfile_name}_gempak'
         elif 'gfdl' in str(requirements).lower():
             dockerfile_name = f'{dockerfile_name}_gfdl'
+        elif 'cartopy' in str(requirements).lower():
+            dockerfile_name = f'{dockerfile_name}_cartopy'
 
         docker_build_cmd = (
             f"docker build -t {run_tag} "

ci/docker/docker_env/README.md

@@ -163,6 +163,61 @@ will fail through the script, but it should be installed using pip3 afterwards.
 ```
 ./scripts/metplotpy_env.sh metplus_base
 /home/met_test/.conda/envs/metplotpy/bin/pip3 install kaleido==0.2.1
+/home/met_test/.conda/envs/metplotpy/bin/python3 cartopy_feature_download.py cultural physical
+rm cartopy_feature_download.py
+```
+
+#### To install METplotpy and METcalcpy packages in environment
+
+```
+cd /home/met_test
+
+# git clone not necessary if repo is already available
+git clone https://github.com/dtcenter/METplotpy
+git clone https://github.com/dtcenter/METcalcpy
+
+cd /home/met_test/METplotpy
+git checkout develop
+/home/met_test/.conda/envs/metplotpy/bin/pip3 install .
+
+cd /home/met_test/METcalcpy
+git checkout develop
+/home/met_test/.conda/envs/metplotpy/bin/pip3 install .
+```
+
+## weatherregime (from metplotpy_env)
+
+### Docker
+
+```
+docker build -t dtcenter/metplus-envs:weatherregime --build-arg BASE_ENV=metplotpy --build-arg ENV_NAME=weatherregime .
+docker push dtcenter/metplus-envs:weatherregime
+```
+
+### Local
+
+```
+./scripts/weatherregime_env.sh metplotpy
+/home/met_test/.conda/envs/weatherregime/bin/python3 cartopy_feature_download.py cultural physical
+rm cartopy_feature_download.py
+```
+
+#### To install METplotpy and METcalcpy packages in environment
+
+```
+cd /home/met_test
+
+# git clone not necessary if repo is already available
+git clone https://github.com/dtcenter/METplotpy
+git clone https://github.com/dtcenter/METcalcpy
+
+cd /home/met_test/METplotpy
+git checkout develop
+/home/met_test/.conda/envs/weatherregime/bin/pip3 install .
+
+cd /home/met_test/METcalcpy
+git checkout develop
+/home/met_test/.conda/envs/weatherregime/bin/pip3 install .
 ```
 
 ## cycloneplotter (from metplus_base)

ci/docker/docker_env/scripts/cycloneplotter_env.sh

@@ -23,3 +23,7 @@ conda create -y --clone ${BASE_ENV} --name ${ENV_NAME}
 conda install -y --name ${ENV_NAME} -c conda-forge cartopy==0.17.0
 conda install -y --name ${ENV_NAME} -c conda-forge matplotlib==3.3.0
 conda install -y --name ${ENV_NAME} -c conda-forge pandas
+
+yum -y install wget
+wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
+/usr/local/envs/${ENV_NAME}/bin/python3 cartopy_feature_download.py cultural physical

ci/docker/docker_env/scripts/metplotpy_env.sh

@@ -42,3 +42,7 @@ conda install -y --name ${ENV_NAME} -c conda-forge xarray==0.17
 conda install -y --name ${ENV_NAME} -c conda-forge netcdf4==1.5.6
 conda install -y --name ${ENV_NAME} -c conda-forge pyyaml
 /usr/local/envs/${ENV_NAME}/bin/pip3 install kaleido==0.2.1
+
+yum -y install wget
+wget https://raw.githubusercontent.com/SciTools/cartopy/master/tools/cartopy_feature_download.py
+/usr/local/envs/${ENV_NAME}/bin/python3 cartopy_feature_download.py cultural physical

ci/docker/docker_env/scripts/weatherregime_env.sh

@@ -2,23 +2,13 @@
 
 ################################################################################
 # Environment: weatherregime
-# Last Updated: 2021-06-29 ([email protected])
+# Last Updated: 2021-09-16 ([email protected])
 # Notes: Adds Python packages needed to run weather regime use case
 #  METplotpy and METcalcpy
-#   Uses pip to install kaleido because
-#   could not install via Conda (glibc conflict)
 # Python Packages:
-#   matplotlib==3.3.0
-#   scipy==1.5.1
-#   plotly==4.9.0
-#   pingouin==0.3.8
-#   cartopy==0.18.0
-#   eofs==1.3.0
-#   cmocean==2.0
-#   xarray==0.17
-#   netcdf4==1.5.6
-#   pyyaml==?
-#   python-kaleido==0.2.1
+#   All packages from metplotpy_env
+#   scikit-learn==0.24.2
+#   eofs==1.4.0
 #
 # Other Content: None
 ################################################################################
@@ -32,7 +22,9 @@ BASE_ENV=$1
 
 conda create -y --clone ${BASE_ENV} --name ${ENV_NAME}
 conda install -y --name ${ENV_NAME} -c conda-forge scikit-learn==0.24.2
-#conda install -y --name ${ENV_NAME} -c conda-forge scipy==1.5.4
 conda install -y --name ${ENV_NAME} -c conda-forge eofs==1.4.0
-conda install -y --name ${ENV_NAME} -c conda-forge netcdf4==1.5.7
-#conda install -y --name ${ENV_NAME} -c conda-forge numpy==1.19.5
+
+rm cartopy_feature_download.py
+yum -y install wget
+wget https://raw.githubusercontent.com/georgemccabe/cartopy/master/tools/cartopy_feature_download.py
+/usr/local/envs/${ENV_NAME}/bin/python3 cartopy_feature_download.py cultural physical cultural-extra

docs/Users_Guide/systemconfiguration.rst

@@ -2282,6 +2282,17 @@ time_summary.step and time_summary.width
 |                  | | :term:`PB2NC_TIME_SUMMARY_WIDTH` = 3600 |
 +------------------+-------------------------------------------+
 
+pb_report_type
+^^^^^^^^^^^^^^
+
++------------------+----------------------------------------------------------------------------------------------------------------------------------------+
+| Old (Incorrect): | pb_report_type = [ 120, 220, 221, 122, 222, 223, 224, 133, 233, 188, 288, 180, 280, 181, 182, 281, 282, 183, 284, 187, 287 ];          |
++------------------+----------------------------------------------------------------------------------------------------------------------------------------+
+| New (Correct):   | pb_report_type    = [];                                                                                                                |
++------------------+----------------------------------------------------------------------------------------------------------------------------------------+
+| METplus Config:  | :term:`PB2NC_PB_REPORT_TYPE` = 120, 220, 221, 122, 222, 223, 224, 133, 233, 188, 288, 180, 280, 181, 182, 281, 282, 183, 284, 187, 287 |
++------------------+----------------------------------------------------------------------------------------------------------------------------------------+
+
 PointStatConfig
 ---------------
 

internal_tests/use_cases/all_use_cases.txt

@@ -1,5 +1,5 @@
 Category: met_tool_wrapper
-0::CyclonePlotter::met_tool_wrapper/CyclonePlotter/CyclonePlotter.conf,user_env_vars.MET_PYTHON_EXE=python3:: cycloneplotter_env
+0::CyclonePlotter::met_tool_wrapper/CyclonePlotter/CyclonePlotter.conf,user_env_vars.MET_PYTHON_EXE=python3:: cycloneplotter_env,cartopy
 1::PCPCombine_python_embedding:: met_tool_wrapper/PCPCombine/PCPCombine_python_embedding.conf:: h5py_env,py_embed
 2::ASCII2NC:: met_tool_wrapper/ASCII2NC/ASCII2NC.conf
 3::met_tool_wrapper/ASCII2NC/ASCII2NC_python_embedding.conf
@@ -117,16 +117,16 @@ Category: precipitation
 
 Category: s2s
 0::GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast:: model_applications/s2s/GridStat_SeriesAnalysis_fcstNMME_obsCPC_seasonal_forecast.conf:: netcdf4_env
-1::UserScript_fcstGFS_obsERA_Blocking:: model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking.conf:: metplotpy_env,metplus
-2::UserScript_obsERA_obsOnly_Blocking:: model_applications/s2s/UserScript_obsERA_obsOnly_Blocking.conf:: metplotpy_env,metplus
-3::UserScript_obsERA_obsOnly_WeatherRegime:: model_applications/s2s/UserScript_obsERA_obsOnly_WeatherRegime.conf:: weatherregime_env,metplus
-4::TCGen_fcstGFSO_obsBDECKS_GDF_TDF:: model_applications/s2s/TCGen_fcstGFSO_obsBDECKS_GDF_TDF.conf:: metplotpy_env,metplus
-5::UserScript_obsPrecip_obsOnly_Hovmoeller:: model_applications/s2s/UserScript_obsPrecip_obsOnly_Hovmoeller.conf:: metplotpy_env
+1::UserScript_fcstGFS_obsERA_Blocking:: model_applications/s2s/UserScript_fcstGFS_obsERA_Blocking.conf:: metplotpy_env,cartopy,metplus
+2::UserScript_obsERA_obsOnly_Blocking:: model_applications/s2s/UserScript_obsERA_obsOnly_Blocking.conf:: metplotpy_env,cartopy,metplus
+3::UserScript_obsERA_obsOnly_WeatherRegime:: model_applications/s2s/UserScript_obsERA_obsOnly_WeatherRegime.conf:: weatherregime_env,cartopy,metplus
+4::TCGen_fcstGFSO_obsBDECKS_GDF_TDF:: model_applications/s2s/TCGen_fcstGFSO_obsBDECKS_GDF_TDF.conf:: metplotpy_env,cartopy,metplus
+5::UserScript_obsPrecip_obsOnly_Hovmoeller:: model_applications/s2s/UserScript_obsPrecip_obsOnly_Hovmoeller.conf:: metplotpy_env,cartopy
 6:: UserScript_obsPrecip_obsOnly_CrossSpectraPlot:: model_applications/s2s/UserScript_obsPrecip_obsOnly_CrossSpectraPlot.conf:: spacetime_env
 7:: UserScript_fcstGFS_obsERA_PhaseDiagram:: model_applications/s2s/UserScript_fcstGFS_obsERA_PhaseDiagram.conf:: spacetime_env
 8:: UserScript_fcstGFS_obsERA_OMI:: model_applications/s2s/UserScript_fcstGFS_obsERA_OMI.conf:: spacetime_env, metdatadb
 9:: UserScript_fcstGFS_obsERA_RMM:: model_applications/s2s/UserScript_fcstGFS_obsERA_RMM.conf:: spacetime_env, metdatadb
-10::UserScript_fcstGFS_obsERA_WeatherRegime:: model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime.conf:: weatherregime_env,metplus
+10::UserScript_fcstGFS_obsERA_WeatherRegime:: model_applications/s2s/UserScript_fcstGFS_obsERA_WeatherRegime.conf:: weatherregime_env,cartopy,metplus
 
 Category: space_weather
 0::GridStat_fcstGloTEC_obsGloTEC_vx7:: model_applications/space_weather/GridStat_fcstGloTEC_obsGloTEC_vx7.conf
@@ -135,6 +135,6 @@ Category: space_weather
 
 Category: tc_and_extra_tc
 0::TCRMW_fcstGFS_fcstOnly_gonzalo:: model_applications/tc_and_extra_tc/TCRMW_fcstGFS_fcstOnly_gonzalo.conf
-1::CyclonePlotter_fcstGFS_obsGFS_OPC:: model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf:: cycloneplotter_env
+1::CyclonePlotter_fcstGFS_obsGFS_OPC:: model_applications/tc_and_extra_tc/CyclonePlotter_fcstGFS_obsGFS_OPC.conf:: cycloneplotter_env,cartopy
 2::UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF:: model_applications/tc_and_extra_tc/UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF.conf,model_applications/tc_and_extra_tc/UserScript_ASCII2NC_PointStat_fcstHAFS_obsFRD_NetCDF/ci_overrides.conf
 3::GridStat_fcstHAFS_obsTDR_NetCDF:: model_applications/tc_and_extra_tc/GridStat_fcstHAFS_obsTDR_NetCDF.conf:: py_embed