Merge pull request NCAR#130 from grantfirl/ufs-dev-PR173

UFS-dev PR#173

.github/pull_request_template.md

@@ -1,93 +1,144 @@
 <!-- INSTRUCTIONS: 
-- Please fill out all sections of this PR and complete the checklist below
-- Please be as descriptive as possible, this is really important.
-- Please "fill in" checkboxes. Use [X] for a filled in checkbox or leave it [ ] for an empty checkbox
-- Please use github markup as much as possible in linking
-i.e.:
-* Linking to UFSWM PR's and issues add - #<pr/issue number>
-* Linking to a subcomponent PR and issues add - <Group>/<Fork>/pull/<number> or - <Group>/<Fork>/issues/<number>
+- PLEASE READ/FOLLOW THE DIRECTIONS IN EACH SECTION
+- Complete the 'Commit Queue Requirements' below
+- Please use github markup as much as possible (https://docs.github.com/en/get-started/writing-on-github)
+- Please leave your PR in a draft state until all underlying work is completed.
 -->
 ## Commit Queue Requirements:
-<!-- Please note: PRs will be scheduled in the Commit Queue in the order received and only after all
-pre-requisite testing is complete and all PR requirements (e.g. Issues created and noted, Subcomponent
-PRs reviewed and accepted) are met. -->
+<!--
+- Please complete the items that follow this.
+- Please "check off" completed items. Use [X] for a filled in checkbox or leave it [ ] for an empty checkbox
+- Your PR will not be considered until all requirements are met.
+- THIS IS YOUR RESPONSIBILITY
+ -->
 - [ ] Fill out all sections of this template.
 - [ ] All sub component pull requests have been reviewed by their code managers.
-- [ ] Run the full RT suite (compared to current baselines) on either Hera/Derecho/Hercules AND have committed the log to my PR branch.
-- [ ] Add list of all failed regression tests in "Regression Tests" section.
-
-## PR Information
+- [ ] Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
+- [ ] Commit 'test_changes.list' from previous step
+---
+## Description:
+<!--
+Please provide a detailed verbose description of what this PR does
+-->
 
-### Description
-<!-- Provide a detailed description of what this PR does in the space provided below-->
 
-### Commit Message
+### Commit Message:
 <!--
-Please provide the following concise information:
-Description of all changes - 1 line
-Please list all individual issue titles addressed with github links at the end in parenthesis (using #<number> or <group>/<fork>/issues/<number>).
+Please provide concise information for The UFS-WM and/or each sub-component:
+Please delete what is not needed.
 -->
+```
+* UFSWM - 
+  * AQM - 
+  * CDEPS - 
+  * CICE - 
+  * CMEPS - 
+  * CMakeModules - 
+  * FV3 - 
+    * ccpp-physics - 
+    * atmos_cubed_sphere - 
+  * GOCART - 
+  * HYCOM - 
+  * MOM6 - 
+  * NOAHMP - 
+  * WW3 - 
+  * stochastic_physics - 
+```
 
+### Priority:
+<!--
+Please provide the priority you would prefer this pull request to have.
+* Critical Bugfix: Model is wrong.
+* High: Time-sensitive project.
+* Normal.
+Please delete the ones that are not applicable
+-->
+* Critical Bugfix: Reason
+* High: Reason
+* Normal
 
-### Priority
-- [ ] Critical Bugfix (This PR contains a critical bug fix and should be prioritized.)
-- [ ] High (This PR contains a feature or fix needed for a time-sensitive project (eg, retrospectives, implementations))
-- [ ] Normal
-
-### Blocking Dependencies
-<!-- If there are any PR's that are needed to be completed before this one, please add links
-     to them here -->
+## Git Tracking
+### UFSWM:
+<!--
+Please add the UFS-WM github issue here if there is one
+Please delete the one that is not applicable.
+-->
+* Closes #
+* None
 
-### Git Issues Fixed By This PR
-<!-- Example: - Closes #1698  or - Closes NOAA-EMC/fv3atm/issues/729 -->
+### Sub component Pull Requests:
+<!--
+Please provide a list of sub-components involved with this pull request.
+Please provide links to the sub-component pull requests as shown below.
+Please delete what is not needed.
+Example:
+* FV3: NOAA-EMC/fv3atm#734
+  * ccpp-physics: ufs-community/ccpp-physics#33
+* WW3: NOAA-EMC/WW3#321
+-->
+* AQM:
+* CDEPS:
+* CICE:
+* CMEPS:
+* CMakeModules:
+* FV3:
+  * ccpp-physics:
+  * atmos_cubed_sphere:
+* GOCART:
+* HYCOM:
+* MOM6:
+* NOAHMP:
+* WW3:
+* stochastic_physics:
+* None
 
+### UFSWM Blocking Dependencies:
+<!--
+If there are any UFSWM PR's that are needed to be completed before this one, please add links
+to them here
+Please delete what is not needed.
+-->
+* Blocked by #
+* None
 
+---
 ## Changes
-
-### Subcomponent (with links)
-<!-- (add links to subcomponent PR's here) -->
-<!-- Example:
-[X] FV3
-- NOAA-EMC/fv3atm/pull/734
-- NOAA-EMC/fv3atm/pull/735
+### Regression Test Changes (Please commit test_changes.list):
+<!--
+Please let us know if this PR creates new baselines, changes baselines or not.
+Please delete what is not needed.
+Please make sure you have properly submitted test_changes.list
 -->
-- [ ] AQM
-- [ ] CDEPS
-- [ ] CICE
-- [ ] CMEPS
-- [ ] CMakeModules
-- [ ] FV3
-- [ ] GOCART
-- [ ] HYCOM
-- [ ] MOM6
-- [ ] NOAHMP
-- [ ] WW3
-- [ ] stochastic_physics
-- [ ] none
+* PR Adds New Tests/Baselines.
+* PR Updates/Changes Baselines.
+* No Baseline Changes.
 
-### Input data
-- [ ] No changes are expected to input data.
-- [ ] Changes are expected to input data:
-  - [ ] New input data.
-  - [ ] Updated input data.
-
-### Regression Tests:
-- [ ] No changes are expected to any regression test.
-- [ ] Changes are expected to the following tests:
-<details><summary>FAILED REGRESSION TESTS</summary>
-<!-- List failed regression tests here or add "None" -->
-
-</details>
+### Input data Changes:
+<!--
+If there are any changes to input-data for a test, please provide information here.
+Please delete what is not needed.
+-->
+* None.
+* New input data.
+* Updated input data.
 
-### Libraries
-<!-- Library updates take time. If this PR needs updates to libraries, please make sure to accomplish the following tasks -->
-- [ ] Not Needed
-- [ ] Needed
-  - [ ] Create separate issue in [JCSDA/spack-stack](https://github.com/JCSDA/spack-stack) asking for update to library. Include library name, library version.
-  - [ ] Add issue link from JCSDA/spack-stack following this item <!-- for example: "- JCSDA/spack-stack/issue/1757" -->
+### Library Changes/Upgrades:
+<!-- Library updates take time. Please provide library and version information here.
+** SPECIAL INSTRUCTIONS **
+If this PR needs updates to libraries please make sure to accomplish the following tasks:
+- Create separate issue in (https://github.com/JCSDA/spack-stack) asking for update to library. Include library name, library version.
+- Add issue link from JCSDA/spack-stack following this item <!-- for example: "* JCSDA/spack-stack#1757"
 
+Please delete what is not needed.
+-->
+* Required
+  * Library names w/versions:
+  * Git Stack Issue (JCSDA/spack-stack#)
+* No Updates
+
+---
 <!-- STOP!!! THE FOLLOWING IS FOR CODE MANAGERS ONLY. PLEASE DO NOT FILL OUT -->
-### Testing Log:
+## Testing Log:
 - RDHPCS
   - [ ] Hera
   - [ ] Orion
@@ -98,8 +149,5 @@ Please list all individual issue titles addressed with github links at the end i
 - WCOSS2
   - [ ] Dogwood/Cactus
   - [ ] Acorn
-- CI
-  - [ ] Completed
-- opnReqTest
-  - [ ] N/A
-  - [ ] Log attached to comment
+- [ ] CI
+- [ ] opnReqTest (complete task if unnecessary)

CDEPS-interface/cdeps_files.cmake

@@ -12,8 +12,10 @@ list(APPEND cdeps_share_files
   CDEPS/share/shr_assert.h
   CDEPS/share/shr_cal_mod.F90
   CDEPS/share/shr_const_mod.F90
+  CDEPS/share/shr_file_mod.F90
   CDEPS/share/shr_kind_mod.F90
   CDEPS/share/shr_log_mod.F90
+  CDEPS/share/shr_nl_mod.F90
   CDEPS/share/shr_orb_mod.F90
   CDEPS/share/shr_precip_mod.F90
   CDEPS/share/shr_strconvert_mod.F90

CMEPS-interface/CMakeLists.txt

@@ -81,6 +81,11 @@ list(APPEND _mediator_files
   CMEPS/mediator/med_phases_post_rof_mod.F90
   CMEPS/mediator/med_phases_post_wav_mod.F90)
 
+if(CDEPS_INLINE)
+  list(APPEND _mediator_files CMEPS/mediator/med_phases_cdeps_mod.F90)
+  add_definitions(-DCDEPS_INLINE)
+endif()
+
 ###############################################################################
 ### Host model for CCPP
 ###############################################################################
@@ -184,6 +189,12 @@ if(OpenMP_Fortran_FOUND)
   target_link_libraries(cmeps PRIVATE OpenMP::OpenMP_Fortran)
 endif()
 
+### Set dependency for CDEPS if inline capability is requested
+if(CDEPS_INLINE)
+  add_dependencies(cmeps cdeps::cdeps)
+  target_link_libraries(cmeps PUBLIC cdeps::cdeps)
+endif()
+
 ###############################################################################
 ### Install
 ###############################################################################

CMakeLists.txt

@@ -16,7 +16,7 @@ list(APPEND CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/CMakeModules/Modules)
 ###############################################################################
 
 # Valid applications and choices
-list(APPEND VALID_APPS ATM ATMAERO ATMAQ ATMW ATMWM ATML LND S2S S2SA S2SW S2SWA S2SWAL HAFS HAFSW HAFS-ALL NG-GODAS)
+list(APPEND VALID_APPS ATM ATMAERO ATMAQ ATMW ATMWM ATML LND S2S S2SA S2SW S2SWA S2SWAL HAFS HAFSW HAFS-MOM6 HAFS-MOM6W HAFS-ALL NG-GODAS)
 set(APP NONE CACHE BOOL "Application Name")
 if(NOT (APP IN_LIST VALID_APPS))
   message(FATAL_ERROR "${APP} is not a valid application.\nValid Applications are: ${VALID_APPS}")
@@ -66,11 +66,13 @@ set(DISABLE_FMA     OFF CACHE BOOL "Disable Fused Multiply-Add instructions (wor
 set(INLINE_POST     ON  CACHE BOOL "Enable inline post")
 set(MULTI_GASES     OFF CACHE BOOL "Enable MULTI_GASES")
 set(MOVING_NEST     OFF CACHE BOOL "Enable moving nest code")
+set(REGIONAL_MOM6   OFF CACHE BOOL "Enable Regional MOM6")
 set(OPENMP          ON  CACHE BOOL "Enable OpenMP threading")
 set(PARALLEL_NETCDF OFF CACHE BOOL "Enable parallel NetCDF")
 set(JEDI_DRIVER     OFF CACHE BOOL "Enable JEDI as top level driver")
 set(CMEPS_AOFLUX    OFF CACHE BOOL "Enable atmosphere-ocean flux calculation in mediator")
 set(PDLIB           OFF CACHE BOOL "Enable Domain Decomposition in WW3 via PDLIB")
+set(CDEPS_INLINE    OFF CACHE BOOL "Enable CDEPS inline capability")
 
 set(CMAKE_Platform $ENV{CMAKE_Platform})
 if(CMAKE_Platform)
@@ -114,10 +116,12 @@ message("DEBUG ............ ${DEBUG}")
 message("INLINE_POST ...... ${INLINE_POST}")
 message("MULTI_GASES ...... ${MULTI_GASES}")
 message("MOVING_NEST ...... ${MOVING_NEST}")
+message("REGIONAL_MOM6..... ${REGIONAL_MOM6}")
 message("OPENMP ........... ${OPENMP}")
 message("PARALLEL_NETCDF .. ${PARALLEL_NETCDF}")
 message("JEDI_DRIVER ...... ${JEDI_DRIVER}")
 message("CMEPS_AOFLUX ..... ${CMEPS_AOFLUX}")
+message("CDEPS_INLINE ..... ${CDEPS_INLINE}")
 
 message("")
 
@@ -146,7 +150,7 @@ if(FMS)
   find_package(FMS 2022.04 REQUIRED COMPONENTS R4 R8)
   if(APP MATCHES "^(HAFSW)$")
     add_library(fms ALIAS FMS::fms_r4)
-  elseif (APP MATCHES "^(S2S|S2SA|S2SW|S2SWA|S2SWAL|NG-GODAS)$")
+  elseif (APP MATCHES "^(S2S|S2SA|S2SW|S2SWA|S2SWAL|NG-GODAS|HAFS-MOM6|HAFS-MOM6W)$")
     add_library(fms ALIAS FMS::fms_r8)
   endif()
   if(APP MATCHES "^(ATM|ATMAERO|ATMAQ|ATMWM|ATMW|ATML|HAFS|HAFS-ALL)$")

LICENSE.md

@@ -10,19 +10,32 @@ This project includes a mix of the following:
 
 This project utilizes the following unmodified works under various licenses, as shown below.
 
-- Atmos cube sphere dycore: [GNU Lesser General Public License 3.0](https://www.gnu.org/licenses/lgpl-3.0.en.html) (LGPL)
+- [atmos_cubed_sphere dycore](https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
 
-- FV3 Atm: [GNU Lesser General Public License 3.0](https://www.gnu.org/licenses/lgpl-3.0.en.html) (LGPL)
+- [fv3atm](https://github.com/NOAA-EMC/fv3atm): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
 
-- NEMS: [GNU Lesser General Public License 3.0](https://www.gnu.org/licenses/lgpl-3.0.en.html) (LGPL)
+- [Stochastic Physics](https://github.com/NOAA-PSL/stochastic_physics): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
 
-- FMS: [GNU Lesser General Public License 3.0](https://www.gnu.org/licenses/lgpl-3.0.en.html) (LGPL)
+- [CCPP Physics](https://github.com/ufs-community/ccpp-physics): [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
-- Stochastic Physics: [GNU Lesser General Public License 3.0](https://www.gnu.org/licenses/lgpl-3.0.en.html) (LGPL)
+- [CCPP Framework](https://github.com/NCAR/ccpp-framework): [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
-- CCPP Physics: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+- [AQM](https://github.com/NOAA-EMC/AQM): GNU General Public License 3.0 ([GPL](https://www.gnu.org/licenses/gpl-3.0.en.html))
+
+- [CICE](https://github.com/NOAA-EMC/CICE): CICE Consortium license ([LICENSE.pdf](https://github.com/NOAA-EMC/CICE/blob/emc/develop/LICENSE.pdf))
+
+- [CMakeModules](https://github.com/NOAA-EMC/CMakeModules): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
+
+- [GOCART](https://github.com/GEOS-ESM/GOCART): [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+- [HYCOM](https://github.com/NOAA-EMC/HYCOM-src): [MIT License](https://opensource.org/license/mit/)
+
+- [MOM6](https://github.com/NOAA-EMC/MOM6): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
+
+- [WW3](https://github.com/NOAA-EMC/WW3): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
+
+- [Unified Post Processor](https://github.com/NOAA-EMC/UPP): GNU Lesser General Public License 3.0 ([LGPL](https://www.gnu.org/licenses/lgpl-3.0.en.html))
 
-- CCPP Framework: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
 ## The rest of this project is in the worldwide public domain
 

MOM6-interface/CMakeLists.txt

@@ -28,8 +28,14 @@ include("mom6_files.cmake")
 add_library(mom6_obj OBJECT ${mom6_src_files})
 set_target_properties(mom6_obj PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/mod)
 target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
-target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
-                                            $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
+
+if(REGIONAL_MOM6)
+  target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_symmetric>
+                                              $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
+else()
+  target_include_directories(mom6_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
+                                              $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
+endif()
 target_link_libraries(mom6_obj PRIVATE fms
                                        esmf
                                        stochastic_physics
@@ -42,8 +48,13 @@ target_link_libraries(mom6_obj PRIVATE fms
 add_library(mom6_nuopc_obj OBJECT ${mom6_nuopc_src_files})
 set_target_properties(mom6_nuopc_obj PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/mod)
 target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_BINARY_DIR}/mod>)
-target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
-                                                  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
+if(REGIONAL_MOM6)
+  target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_symmetric>
+                                                    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
+else()
+  target_include_directories(mom6_nuopc_obj PRIVATE $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/config_src/memory/dynamic_nonsymmetric>
+                                                    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/MOM6/src/framework>)
+endif()
 target_link_libraries(mom6_nuopc_obj PRIVATE mom6_obj
                                              fms
                                              stochastic_physics