From 385ef54f3ec9c9a5bd06ab10a5b5ee3e13784eae Mon Sep 17 00:00:00 2001 From: Jim Edwards Date: Fri, 24 Jan 2025 08:35:37 -0700 Subject: [PATCH 1/5] update docs for cesm3 --- doc/source/build_cpl/adding-components.rst | 235 ++++++++++++++++++--- doc/source/build_cpl/introduction.rst | 3 +- doc/source/what_cime/index.rst | 12 +- 3 files changed, 209 insertions(+), 41 deletions(-) diff --git a/doc/source/build_cpl/adding-components.rst b/doc/source/build_cpl/adding-components.rst index c9fcf234dd1..0310d127530 100644 --- a/doc/source/build_cpl/adding-components.rst +++ b/doc/source/build_cpl/adding-components.rst @@ -1,36 +1,203 @@ .. _adding-components: -=================== -Adding components -=================== - -Here are the steps to add prognostic components to CIME models. - -There are a couple of aspects of a component interface to CIME, the -scripts interface which controls setting up component inputs and -building the component and the run interface which controls connecting -the component to the coupler and through the coupler, the other -components of the CIME based model. - -The component should have a subdirectory **cime_config** and this -subdirectory should have two files **buildnml** and **buildlib** The -**buildnml** script is used to build the components instructional, -runtime inputs. These have traditionally been in the form of fortran -namelists but may also follow other formats. The **buildnml** may -either be called from the command line or as a python subroutine. If -buildnml is called from the command line it will be passed the -caseroot directory on the command line. If it is called as a -subroutine, the subroutine name must be buildnml and it will take -three arguments, a Case object, a caseroot directory and a component -name. The **buildlib** script will always be called from the command -line, it is called in the case.build step and is expected to build the -the buildlib script will be called with three arguments in order they -are caseroot, libroot (the location of the installed library, -typically EXEROOT/lib) and bldroot, the location of the component -build directory. Look at the cime internal components such as datm -for an example. - -The coupler interface is dependent on which coupler is used, for the mct coupler in cime -the component model must provide NNN_INIT_MCT, NNN_RUN_MCT, NNN_FINAL_MCT where NNN is the -component type of the particular component (eg ATM for an atmosphere, LND for a land model) -these subroutines are expected to be in the component library. +==================================== +Adding a New Component Model to CIME +==================================== + +Overview +-------- + +CIME’s modular design allows for seamless integration of new component models into its infrastructure. To add a new component model, you must ensure compatibility with CIME’s interfaces and follow the defined setup for component registration, coupling, and testing. + +This guide covers the essential steps to integrate a new component model into CIME. + +--- + +Prerequisites +------------- + +- Familiarity with the CIME framework and component model architecture. +- A working implementation of the new component model, including its source code, build scripts, and runtime configuration files. +- Access to the CIME source code repository. + +--- + +Steps to Add a New Component Model +---------------------------------- + +1. Directory Structure +~~~~~~~~~~~~~~~~~~~~~ + +Each component model requires a dedicated directory under the ``components/`` directory in the CIME source tree. Follow these steps: + +1. **Create the Directory:** + + .. code-block:: bash + + mkdir components/ + +2. **Add Source Code:** + Place the source code, configuration files, and necessary scripts for the new model into the directory. + +3. **Follow Naming Conventions:** + Although not required, it is recommended to use consistent naming conventions for directory names and files. It is also conventional to define a subdirectory ``cime_config`` within the component directory. This subdirectory contains all files linking CIME to the component model. Three files are required: + + - **``config_component.xml``:** Describes the component and its input options. + - **``buildlib``:** A Python script providing the interface to build the model. + - **``buildnml``:** A Python script to stage the component's inputs. + + Additionally, other optional interface files may be provided in ``cime_config`` to support advanced functionality: + + - **``config_archive.xml``:** Defines archiving rules for model outputs. + - **``config_compsets.xml``:** Defines compsets specific to the component. + - **``config_tests.xml``:** Describes test configurations. + - **``config_pes.xml``:** Specifies processor layout and scaling. + + If the component includes system tests, it is conventional to add a subdirectory, typically named ``SystemTests``, to house these tests. This name is configurable via ``config_files.xml``. + + Example structure: + + .. code-block:: text + + components// + ├── src/ # Source code + ├── build/ # Build scripts + ├── cime_config/ # CIME interface files + │ ├── config_component.xml + │ ├── buildlib + │ ├── buildnml + │ ├── config_archive.xml (optional) + │ ├── config_compsets.xml (optional) + │ ├── config_tests.xml (optional) + │ └── config_pes.xml (optional) + ├── SystemTests/ # System tests (optional) + └── README.md # Documentation for the component + +--- + +2. Configure the Component Interface +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +CIME requires each component model to adhere to a standard interface for communication with the coupler. Implement the following: + +1. **Component-Specific Interface File:** + Each component is required to have a file named ``_comp_nuopc.F90``, where ```` specifies the component type (e.g., ``atm``, ``lnd``, ``ice``, ``rof``, ``glc``, ``wav``, ``cpl``, ``esp``). This file must define a public routine ``SetServices`` that sets pointers to the model's phase routines (``initialize``, ``run``, and ``finalize``). + + CESM uses the NUOPC (National Unified Operational Prediction Capability) system as the underlying framework for component interaction. For detailed guidance on implementing and configuring the NUOPC-based interfaces, refer to the `NUOPC Layer Documentation `_. + +2. **Initialize the Component:** + Include an initialization routine (``_init``) that defines initial conditions and grid mappings. + +3. **Run and Finalize Routines:** + Ensure the model includes ``run`` and ``finalize`` routines to handle time-stepping and cleanup. + +4. **Data Exchange:** + Define the variables exchanged with the coupler (e.g., fluxes, states) in the component's ``nuopc`` or ``drv`` directory. + +--- + +3. Register the Component +~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Define Component Source Paths:** + Each Earth system model must define its component model source paths in its ``config_files.xml`` file. For example, in CESM, these paths are specified in ``cime/CIME/data/config/cesm/config_files.xml``. Add an entry for your new component model here: + + .. code-block:: xml + + + char + components//src + + +2. **Update ``config_compsets.xml``:** + Add the new component to a compset definition: + + .. code-block:: xml + + + COMPSETNAME + ALIASNAME + + + + + +--- + +4. Modify the Coupler Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Edit ``cime_config/config_files.xml``:** + Add entries specifying how the new component interfaces with the coupler. + +2. **Define Flux Mappings:** + Ensure fluxes exchanged between the new model and other components (e.g., atmosphere, land, ocean) are well-defined. + +3. **Grid Compatibility:** + Verify the model supports the necessary grid resolutions and that mappings are registered. + +--- + +5. Test the Integration +~~~~~~~~~~~~~~~~~~~~~~ + +1. **Create Test Cases:** + Write test cases in the ``tests/`` directory to validate the integration of the new component. + +2. **Run Validation Tests:** + Use CIME’s testing framework to ensure the new component functions correctly: + + .. code-block:: bash + + ./create_test --xml-testlist-file + +3. **Debug Errors:** + Review log files in the ``CaseDocs`` and ``Logs`` directories for issues. + +--- + +6. Document the New Component +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. **Write Documentation:** + Provide a README file in the ``components//`` directory describing: + + - The component's purpose. + - Input and output data requirements. + - Build and runtime instructions. + +2. **Update Central Documentation:** + Add details about the new component to CIME’s central documentation files. + +--- + +7. Submit Changes +~~~~~~~~~~~~~~~~~ + +1. **Commit the Changes:** + + .. code-block:: bash + + git add components/ + git commit -m "Add new component model: " + +2. **Submit a Pull Request:** + Push the changes to the CIME repository and submit a pull request for review. + +--- + +Example +------- + +Adding a New Land Model (``MyLandModel``) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Create ``components/mylandmodel/`` with the required directory structure. +2. Implement the interface routines (``mylandmodel_init``, ``mylandmodel_run``, ``mylandmodel_finalize``). +3. Define the component's source paths in ``config_files.xml``. +4. Register ``MyLandModel`` in ``config_compsets.xml``. +5. Define flux mappings in the coupler configuration files. +6. Test the integration using predefined compsets and submit results for validation. + +--- + diff --git a/doc/source/build_cpl/introduction.rst b/doc/source/build_cpl/introduction.rst index 0352da61bf7..a5ab15b4d0b 100644 --- a/doc/source/build_cpl/introduction.rst +++ b/doc/source/build_cpl/introduction.rst @@ -4,9 +4,10 @@ Introduction Content to go here: How to add a new component model to cime. +The cime to a component model is expected to be found in directory components/model/cime_config. Each component will have xml file config_component.xml and will optionally have xml files config_compsets.xml, config_archive.xml, config_pes.xml, config_tests.xml. It will have python interface scripts buildlib which provides the component build instructions and buildnml which will create the input arguments to the component model - there is also a file user_nl_xxx (where xxx is the component name) which allows the user to override input defaults on a case by case basis. The cime_config directory may also have subdirectories SystemTests and testdefs which describe CIME based tests that are unique to the component model. How to replace an existing cime model with another one. - + How to integrate your model in to the cime build/configure system and coupler. How to work with the CIME-supplied models. diff --git a/doc/source/what_cime/index.rst b/doc/source/what_cime/index.rst index e2e3eade94a..a0d5be09841 100644 --- a/doc/source/what_cime/index.rst +++ b/doc/source/what_cime/index.rst @@ -17,20 +17,20 @@ Overview ********* -CIME (Common Infrastructure for Modeling the Earth, pronounced "SEAM") primarily consists of a Case Control System that supports the configuration, compilation, execution, system testing and unit testing of an Earth System Model. The three main components of the Case Control System are: +CIME (Common Infrastructure for Modeling the Earth, pronounced "SEAM") primarily consists of a Case Control System that supports the configuration, compilation, execution, system testing and unit testing of an Earth System Model. The four main components of the Case Control System are: 1. XML files that describe the available machines, models and model configurations. -2. Python scripts that take user commands and parse the XML files to configure a set of models with their +2. CMake files that describe the model build and compiler flags. +3. Python scripts that take user commands and parse the XML and CMake files to configure a set of models with their build and runtime options for a specified machine and the provide additional commands to build executables and submit jobs. -3. Testing utilities to run defined system tests and report results for different configurations of the coupled system. +4. Testing utilities to run defined system tests and report results for different configurations of the coupled system. CIME also contains additional stand-alone tools useful for Earth system modeling including: 1. Parallel regridding weight generation program -2. Scripts to automate off-line load-balancing. -3. Scripts to conduct ensemble-based statistical consistency tests. -4. Netcdf file comparison program (for bit-for-bit). +2. Scripts to conduct ensemble-based statistical consistency tests. +3. Netcdf file comparison program (for bit-for-bit): cprnc. CIME does **not** contain the source code for any Earth System Model drivers or components. It is typically included alongside the source code of a host model. However, CIME does include pointers to external repositories that contain drivers, data models and other test components. These external components can be easily assembled to facilitate end-to-end system tests of the CIME infrastructure, which are defined in the CIME repository. From e88ca7761632ac778fc0de5776e6238870822de1 Mon Sep 17 00:00:00 2001 From: Jim Edwards Date: Fri, 24 Jan 2025 08:46:49 -0700 Subject: [PATCH 2/5] revert change to introduction page --- doc/source/build_cpl/introduction.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/source/build_cpl/introduction.rst b/doc/source/build_cpl/introduction.rst index a5ab15b4d0b..0352da61bf7 100644 --- a/doc/source/build_cpl/introduction.rst +++ b/doc/source/build_cpl/introduction.rst @@ -4,10 +4,9 @@ Introduction Content to go here: How to add a new component model to cime. -The cime to a component model is expected to be found in directory components/model/cime_config. Each component will have xml file config_component.xml and will optionally have xml files config_compsets.xml, config_archive.xml, config_pes.xml, config_tests.xml. It will have python interface scripts buildlib which provides the component build instructions and buildnml which will create the input arguments to the component model - there is also a file user_nl_xxx (where xxx is the component name) which allows the user to override input defaults on a case by case basis. The cime_config directory may also have subdirectories SystemTests and testdefs which describe CIME based tests that are unique to the component model. How to replace an existing cime model with another one. - + How to integrate your model in to the cime build/configure system and coupler. How to work with the CIME-supplied models. From 5ea52efd0cb8da29e698e12c6075f887f2670b5a Mon Sep 17 00:00:00 2001 From: Jim Edwards Date: Fri, 24 Jan 2025 09:20:11 -0700 Subject: [PATCH 3/5] more doc updates --- doc/source/build_cpl/adding-components.rst | 43 +++++++++++----------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/doc/source/build_cpl/adding-components.rst b/doc/source/build_cpl/adding-components.rst index 0310d127530..bedbd740655 100644 --- a/doc/source/build_cpl/adding-components.rst +++ b/doc/source/build_cpl/adding-components.rst @@ -1,6 +1,5 @@ -.. _adding-components: +.. _adding-component-cime: -==================================== Adding a New Component Model to CIME ==================================== @@ -99,28 +98,32 @@ CIME requires each component model to adhere to a standard interface for communi 3. Register the Component ~~~~~~~~~~~~~~~~~~~~~~~~~ -1. **Define Component Source Paths:** - Each Earth system model must define its component model source paths in its ``config_files.xml`` file. For example, in CESM, these paths are specified in ``cime/CIME/data/config/cesm/config_files.xml``. Add an entry for your new component model here: +1. **Define Component Root Directory:** + Each component model must define its root directory using the variable ``COMP_ROOT_DIR_XXX`` in the ``config_files.xml`` file, where ``XXX`` represents the component class (e.g., ATM, LND, ICE). For example: .. code-block:: xml - - char - components//src - + + char + + $SRCROOT/components/cdeps/datm + $CIMEROOT/CIME/non_py/src/components/stub_comps_$COMP_INTERFACE/satm + $CIMEROOT/CIME/non_py/src/components/xcpl_comps_$COMP_INTERFACE/xatm + + $SRCROOT/components/cam/ + $SRCROOT/components/fv3/ + $SRCROOT/components/myatm/ + + case_comps + env_case.xml + Root directory of the case atmospheric component + $CIMEROOT/CIME/data/config/xml_schemas/config_compsets.xsd + 2. **Update ``config_compsets.xml``:** - Add the new component to a compset definition: - - .. code-block:: xml + The ``config_compsets.xml`` file can be used to define aliases for long compset names, easing the burden of specifying full names. However, aliases are optional. - - COMPSETNAME - ALIASNAME - - - - + The component name in the long compset name must match the name used in the ``description`` section of the ``config_component.xml`` file. Additionally, the ``config_component.xml`` file must include a variable ``COMP_XXX`` (where ``XXX`` is the component class, e.g., ATM, LND, ICE). This variable is essential for defining the component in the context of CIME. --- @@ -194,10 +197,8 @@ Adding a New Land Model (``MyLandModel``) 1. Create ``components/mylandmodel/`` with the required directory structure. 2. Implement the interface routines (``mylandmodel_init``, ``mylandmodel_run``, ``mylandmodel_finalize``). -3. Define the component's source paths in ``config_files.xml``. +3. Define the component's root directory in ``config_files.xml`` using the variable ``COMP_ROOT_DIR_LND``. 4. Register ``MyLandModel`` in ``config_compsets.xml``. 5. Define flux mappings in the coupler configuration files. 6. Test the integration using predefined compsets and submit results for validation. ---- - From 6c112de3a59de7505224f426621c4724a49fbabe Mon Sep 17 00:00:00 2001 From: Jim Edwards Date: Fri, 24 Jan 2025 09:43:00 -0700 Subject: [PATCH 4/5] more doc updates --- .github/workflows/docs.yml | 10 +++++----- doc/source/build_cpl/adding-components.rst | 7 ++++--- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index f819a5ccc96..13b64e7cd92 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -25,7 +25,7 @@ jobs: runs-on: ubuntu-latest if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: ref: 'gh-pages' fetch-depth: 0 @@ -64,17 +64,17 @@ jobs: name: Build and deploy documentation runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: fetch-depth: 0 lfs: true - name: Install python 3.x - uses: actions/setup-python@v2 + uses: actions/setup-python@v5 with: python-version: '3.x' # https://github.com/actions/cache/blob/main/examples.md#python---pip - name: Cache pip - uses: actions/cache@v3 + uses: actions/cache@v4 with: path: ~/.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('doc/requirements.txt') }} @@ -91,7 +91,7 @@ jobs: if: | github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository - uses: peaceiris/actions-gh-pages@v3 + uses: peaceiris/actions-gh-pages@v4 with: github_token: ${{secrets.GITHUB_TOKEN}} publish_dir: './_build/html' diff --git a/doc/source/build_cpl/adding-components.rst b/doc/source/build_cpl/adding-components.rst index bedbd740655..6acb8e6bf0b 100644 --- a/doc/source/build_cpl/adding-components.rst +++ b/doc/source/build_cpl/adding-components.rst @@ -79,10 +79,11 @@ Each component model requires a dedicated directory under the ``components/`` di CIME requires each component model to adhere to a standard interface for communication with the coupler. Implement the following: -1. **Component-Specific Interface File:** - Each component is required to have a file named ``_comp_nuopc.F90``, where ```` specifies the component type (e.g., ``atm``, ``lnd``, ``ice``, ``rof``, ``glc``, ``wav``, ``cpl``, ``esp``). This file must define a public routine ``SetServices`` that sets pointers to the model's phase routines (``initialize``, ``run``, and ``finalize``). +1. **Component-Specific Interface File:** Each component is required to have a file named _comp_.F90, where specifies the component type (e.g., atm, lnd, ice, rof, glc, wav, cpl, esp) and driver specifies the model driver (nuopc or mct). For nuopc this file must define a public routine SetServices that sets pointers to the model's phase routines, for mct the phase routine names (initialize, run, and finalize) are assumed and must be present. - CESM uses the NUOPC (National Unified Operational Prediction Capability) system as the underlying framework for component interaction. For detailed guidance on implementing and configuring the NUOPC-based interfaces, refer to the `NUOPC Layer Documentation `_. +CESM uses the NUOPC (National Unified Operational Prediction Capability) system as the underlying framework for component interaction. For detailed guidance on implementing and configuring the NUOPC-based interfaces, refer to the NUOPC Layer Documentation _. + +E3SM uses the MCT (Model Coupling Toolkit) system as the framework.   Detailed guidance on implementing and configuring the MCT-based interfaces can be found in _. 2. **Initialize the Component:** Include an initialization routine (``_init``) that defines initial conditions and grid mappings. From 265c668276d31edf6bdf392ad596f58846c229b8 Mon Sep 17 00:00:00 2001 From: Jim Edwards Date: Mon, 27 Jan 2025 07:44:53 -0700 Subject: [PATCH 5/5] improve doc --- doc/source/build_cpl/adding-components.rst | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/doc/source/build_cpl/adding-components.rst b/doc/source/build_cpl/adding-components.rst index 6acb8e6bf0b..0f3826ae53a 100644 --- a/doc/source/build_cpl/adding-components.rst +++ b/doc/source/build_cpl/adding-components.rst @@ -79,11 +79,12 @@ Each component model requires a dedicated directory under the ``components/`` di CIME requires each component model to adhere to a standard interface for communication with the coupler. Implement the following: -1. **Component-Specific Interface File:** Each component is required to have a file named _comp_.F90, where specifies the component type (e.g., atm, lnd, ice, rof, glc, wav, cpl, esp) and driver specifies the model driver (nuopc or mct). For nuopc this file must define a public routine SetServices that sets pointers to the model's phase routines, for mct the phase routine names (initialize, run, and finalize) are assumed and must be present. +**Component-Specific Interface File:** +For components, the interface file should be named _comp_.F90, where specifies the component type (e.g., atm, lnd, ice, rof, glc, wav, cpl, esp) and specifies the coupling framework (nuopc or mct). -CESM uses the NUOPC (National Unified Operational Prediction Capability) system as the underlying framework for component interaction. For detailed guidance on implementing and configuring the NUOPC-based interfaces, refer to the NUOPC Layer Documentation _. +**NUOPC:** For components using the NUOPC (National Unified Operational Prediction Capability) system, the file _comp_nuopc.F90 must define a public routine SetServices that sets pointers to the model's phase routines (initialize, run, and finalize). -E3SM uses the MCT (Model Coupling Toolkit) system as the framework.   Detailed guidance on implementing and configuring the MCT-based interfaces can be found in _. +**MCT:** For components using the MCT (Model Coupling Toolkit) system, phase routines must be explicitly named as initialize, run, and finalize. 2. **Initialize the Component:** Include an initialization routine (``_init``) that defines initial conditions and grid mappings. @@ -91,9 +92,6 @@ E3SM uses the MCT (Model Coupling Toolkit) system as the framework.   Detailed 3. **Run and Finalize Routines:** Ensure the model includes ``run`` and ``finalize`` routines to handle time-stepping and cleanup. -4. **Data Exchange:** - Define the variables exchanged with the coupler (e.g., fluxes, states) in the component's ``nuopc`` or ``drv`` directory. - --- 3. Register the Component