Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: C2SM/spack-c2sm
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.21.1.5
Choose a base ref
...
head repository: C2SM/spack-c2sm
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 9 commits
  • 43 files changed
  • 9 contributors

Commits on Nov 27, 2024

  1. use v7 on Balfrin (#1033)

    * use v7 on Balfrin
    
    * fix
    
    * set cuda version for pytorch-fortran
    jonasjucker authored Nov 27, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    615f043 View commit details
  2. cosmo-eccodes-definitions: Update maintainers (#1034)

    dominichofer authored Nov 27, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    a6dffef View commit details

Commits on Nov 28, 2024

  1. gt4py: Add 1.0.3.10 (#1018)

    * add new gt4py version for tag icon4py_20241113
    
    * add new version to system test parametrization
    
    * add gt4py version to icon4py system test
    
    * add new version to icon4py, fix system test
    
    * fix version number typo
    
    * remove main - test two last versions
    
    * fix gt4py version for icon4py 0.0.13
    
    * GitHub Action: Apply Pep8-formatting
    
    ---------
    
    Co-authored-by: github-actions <[email protected]>
    halungge and github-actions authored Nov 28, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    5887a0b View commit details
  2. Add "Jenkins test with uenv" part to Testing.rst (#1036)

    dominichofer authored Nov 28, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    d48365c View commit details
  3. Add version numbering info (#1037)

    dominichofer authored Nov 28, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    e5479e6 View commit details

Commits on Jan 28, 2025

  1. Dev 0.22 (#1011)

    * Set submodule spack to v0.22.2
    
    * fdb: Remove package
    
    * metkit: Remove package
    
    * py-frozendict: Remove package
    
    * xpmem: Remove package
    
    * py-hatchling: Update from upstream
    
    * py-tabulate: Update from upstream
    
    * icon-ham: Use spack/icon
    
    * icon: Split package
    
    * py-lark: Remove package
    
    * Remove unnecessary tests
    
    * Migrate unit tests to pytest
    
    * Rename icon to icon-c2sm
    
    * cosmo: Remove package
    
    * Jenkins: Use mch/prgenv-icon:rc1
    
    * GitHub Action: Apply Pep8-formatting
    
    * Add tests for 'spack find' and 'spack compilers'.
    
    * Start uenv in test stages
    
    * deactivate
    
    * Simplify run_with_spack
    
    * Keep venv
    
    * Bring back devirtualization
    
    * Update uenv
    
    * Use .venv
    
    * GitHub Action: Apply Pep8-formatting
    
    * Shuffle Jenkinsfile
    
    * revert
    
    * uenv
    
    * revert clean env
    
    * Remove fdb-fortran
    
    * flexpart-cosmo: Add workaround
    
    * Update icon-mch tests
    
    * Typo
    
    * icontools: Workaround
    
    * Fix some icon tests
    
    * Fix type
    
    * Remove icon-mch gcc test
    
    * GitHub Action: Apply Pep8-formatting
    
    * Revert "cosmo: Remove package"
    
    This reverts commit 2a7a337.
    
    * cosmo: remove package
    
    * cosmo: remove tests
    
    * Fix icon-mch test
    
    * fix icon test
    
    * eccodes: remove version 2.19.0
    
    * Clean up system tests
    
    * GitHub Action: Apply Pep8-formatting
    
    * Update uenv
    
    * cosmo-eccodes-definitions: Remove versions 2.19.0.x
    
    * icon-mch: Add workarounds
    
    * icon-c2sm: Inherit from icon
    
    * Improve tests
    
    * Disable failing package tests.
    
    * Move mpi workaround
    
    * GitHub Action: Apply Pep8-formatting
    
    * icon-mch: Remove gcc test and add version tests
    
    * icon-c2sm: Remove copy_runscript_related_input_files because there's no self.out_of_source_build
    
    * icon-mch: fix tags
    
    * icon: Set new maintainer
    
    * Rename icon-c2sm to icon
    
    * Use require
    
    * GitHub Action: Apply Pep8-formatting
    
    * pytorch-fortran-proxy: requires gcc
    
    * GitHub Action: Apply Pep8-formatting
    
    * dirty gt4py fix
    
    * GitHub Action: Apply Pep8-formatting
    
    * rm exclaim
    
    * GitHub Action: Apply Pep8-formatting
    
    * fix test and add icon-exlcaim
    
    * GitHub Action: Apply Pep8-formatting
    
    * make out-of-source work
    
    * GitHub Action: Apply Pep8-formatting
    
    * fix url
    
    * rm file
    
    * update test
    
    * cleanup
    
    * GitHub Action: Apply Pep8-formatting
    
    * fix
    
    * fix version
    
    * rm include
    
    * GitHub Action: Apply Pep8-formatting
    
    * fix icon-ham
    
    * add maintainer
    
    * Update repos/c2sm/packages/icon-exclaim/package.py
    
    Co-authored-by: Dominic Hofer <[email protected]>
    
    * GitHub Action: Apply Pep8-formatting
    
    * change
    
    * GitHub Action: Apply Pep8-formatting
    
    * disable version
    
    * GitHub Action: Apply Pep8-formatting
    
    * remove loop-exchange
    
    * fix icon
    
    * dirty fix icon4py
    
    * readd loop-exchange
    
    * Update package.py
    
    Set default loop-exchange to False
    
    * fix bug again of serialbox dep, cleanup
    
    * reintroduce skip configure
    
    * remove libtorch
    
    * upgrade docs (#1038)
    
    * upgrade docs
    
    * add yaml example
    
    * update docs
    
    * update docs
    
    * Update cosmo-eccodes-definitions versions to 2.36.0.3
    This is used with eccodes 2.36.4
    This is required to test v8 env on balfrin
    
    * use explicit zlib-ng dependency for icon (#1040)
    
    * use rc 4
    
    * remove obsolete packages
    
    * adjust integration tests
    
    * upgrade manual to build icon
    
    ---------
    
    Co-authored-by: github-actions <[email protected]>
    Co-authored-by: juckerj <[email protected]>
    Co-authored-by: Daniel Hupp <[email protected]>
    Co-authored-by: Jonas Jucker <[email protected]>
    Co-authored-by: Xavier Lapillonne <[email protected]>
    Co-authored-by: Ben Cumming <[email protected]>
    7 people authored Jan 28, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    6ed9b0f View commit details

Commits on Feb 5, 2025

  1. removing icon-exclaim package (#1044)

    * removing temporarily icon-exclaim package
    The package is causing an error with the icon package
    Deactivate icon-exclaim test
    lxavier authored Feb 5, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    bbc220d View commit details

Commits on Feb 13, 2025

  1. Update QuickStart.rst (#1045)

    AnnikaLau authored Feb 13, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    9942a8e View commit details
  2. [Icontools] use package from Sergey (#902)

    * add recipe from Sergey
    
    * add c2sm version
    
    * GitHub Action: Apply Pep8-formatting
    
    * Update system_test.py
    
    * GitHub Action: Apply Pep8-formatting
    
    * Update system_test.py
    
    * GitHub Action: Apply Pep8-formatting
    
    * Update system_test.py
    
    * GitHub Action: Apply Pep8-formatting
    
    * Update system_test.py
    
    ---------
    
    Co-authored-by: github-actions <[email protected]>
    jonasjucker and github-actions authored Feb 13, 2025

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    1d356a9 View commit details
Showing with 502 additions and 2,093 deletions.
  1. +23 −11 Jenkinsfile
  2. +2 −0 README.md
  3. +12 −36 docs/CodeDevelopment.rst
  4. +46 −37 docs/QuickStart.rst
  5. +26 −0 docs/Testing.rst
  6. +6 −34 repos/c2sm/packages/cosmo-eccodes-definitions/package.py
  7. +0 −329 repos/c2sm/packages/cosmo/package.py
  8. +0 −1 repos/c2sm/packages/cosmo/patches/5.09a.mch1.2.p2
  9. +0 −20 repos/c2sm/packages/cosmo/patches/apn-mch/spec_as_yaml/patch.serialize_cosmo
  10. +0 −21 repos/c2sm/packages/cosmo/patches/apn-mch/spec_as_yaml/patch.test_cosmo
  11. +0 −1 repos/c2sm/packages/cosmo/patches/c2sm-features/spec_as_yaml/patch.serialize_cosmo
  12. +0 −1 repos/c2sm/packages/cosmo/patches/c2sm-master
  13. +0 −1 repos/c2sm/packages/cosmo/patches/empa-ghg/spec_as_yaml/patch.serialize_cosmo
  14. +0 −19 repos/c2sm/packages/cosmo/patches/org-master/spec_as_yaml/patch.serialize_cosmo
  15. +0 −22 repos/c2sm/packages/cosmo/patches/org-master/spec_as_yaml/patch.test_cosmo
  16. +0 −10 repos/c2sm/packages/eccodes/package.py
  17. +0 −33 repos/c2sm/packages/fdb-fortran/package.py
  18. +0 −21 repos/c2sm/packages/fdb/package.py
  19. +3 −5 repos/c2sm/packages/flexpart-cosmo/package.py
  20. +3 −11 repos/c2sm/packages/flexpart-ifs/package.py
  21. +4 −2 repos/c2sm/packages/icon-ham/package.py
  22. +56 −529 repos/c2sm/packages/icon/package.py
  23. +187 −107 repos/c2sm/packages/icontools/package.py
  24. +0 −185 repos/c2sm/packages/int2lm/package.py
  25. +0 −10 repos/c2sm/packages/int2lm/patches/testsuite/patch.to_python3
  26. +0 −49 repos/c2sm/packages/libtorch/package.py
  27. +0 −10 repos/c2sm/packages/metkit/package.py
  28. +0 −154 repos/c2sm/packages/oasis/package.py
  29. +0 −13 repos/c2sm/packages/oasis/patches/patch.PSMILE_add_timer_as_dep_of_string
  30. +0 −22 repos/c2sm/packages/py-frozendict/package.py
  31. +10 −1 repos/c2sm/packages/py-gt4py/package.py
  32. +37 −25 repos/c2sm/packages/py-hatchling/package.py
  33. +8 −0 repos/c2sm/packages/py-icon4py/package.py
  34. +0 −48 repos/c2sm/packages/py-lark/package.py
  35. +4 −0 repos/c2sm/packages/py-tabulate/package.py
  36. +0 −39 repos/c2sm/packages/pytorch-fortran-proxy/package.py
  37. +0 −57 repos/c2sm/packages/pytorch-fortran/package.py
  38. +0 −120 repos/c2sm/packages/xpmem/package.py
  39. +0 −15 repos/c2sm/packages/xpmem/xpmem_v2.6.5-36.patch
  40. +1 −1 spack
  41. +11 −5 test/integration_test.py
  42. +59 −81 test/system_test.py
  43. +4 −7 test/unit_test.py
34 changes: 23 additions & 11 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -20,20 +20,30 @@ pipeline {
}
}
stages {
stage('Create environment') {
stage('Create python environment') {
steps {
sh """
python3 -m venv env
source env/bin/activate
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
"""
}
}
stage('Create uenv') {
steps {
sh """
git clone -b fix/jenkins https://github.com/eth-cscs/uenv.git
./uenv/install --yes --destdir=$WORKSPACE
source $WORKSPACE/etc/profile.d/uenv.sh
uenv repo create
uenv image pull mch/v8:rc4
"""
}
}
stage('Bootstrap spack') {
// Bootstrapping spack is a separate stage to avoid problems with concurrently bootstrapping spack in the tests.
steps {
sh """
source env/bin/activate
source ./setup-env.sh
spack spec gnuconfig
"""
@@ -42,26 +52,28 @@ pipeline {
stage('Unit Tests') {
steps {
sh """
source env/bin/activate
source .venv/bin/activate
python3 test/unit_test.py
"""
}
}
stage('Integration Tests') {
steps {
sh """
source env/bin/activate
source ./setup-env.sh $USER_ENV_ROOT
pytest -v -n auto test/integration_test.py
source $WORKSPACE/etc/profile.d/uenv.sh
source ./setup-env.sh /user-environment
source .venv/bin/activate
uenv run mch/v8:rc4 -- pytest -v -n auto test/integration_test.py
"""
}
}
stage('System Tests') {
steps {
sh """
source env/bin/activate
source ./setup-env.sh $USER_ENV_ROOT
pytest -v -n auto test/system_test.py
source $WORKSPACE/etc/profile.d/uenv.sh
source ./setup-env.sh /user-environment
source .venv/bin/activate
uenv run mch/v8:rc4 -- pytest -v -n auto test/system_test.py
"""
}
}
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -17,6 +17,8 @@ Spack is the package manager used by C2SM and MeteoSwiss to install and deploy s
* [Official spack v0.20.1](https://spack.readthedocs.io/en/v0.20.1/)
* [Official spack v0.18.1](https://spack.readthedocs.io/en/v0.18.1/) [deprecated]

The first 3 numbers of every spack-c2sm version match with the version of spack it uses as a submodule.

## Workflow
We suggest local/individual spack instances and the use of spack environments.

48 changes: 12 additions & 36 deletions docs/CodeDevelopment.rst
Original file line number Diff line number Diff line change
@@ -3,42 +3,6 @@ Develop packages

Spack offers several options for package development.
Depending on your workflow, one or the other option is preferred.
Also some packages like ICON or COSMO have their own
development workflow which is maintained by C2SM.

Plain dev-build
---------------

This is the easiest way to build local sources.
Enter the root of your source repository and execute:

.. code-block:: console
$ spack dev-build --until build <package> @<version>
This will build the package as is. The downside of this approach is that
you need to go through all phases of a package build.

Dev-build in combination with build-env
---------------------------------------

We assume that developers of a package are familiar with its build system.
Therefore, we reccomend to use Spack to set up the environment for the package.
Building and testing should be done with the package's build and test system.

.. code-block:: console
# Load Spack!
$ spack dev-build --before build <package> @develop <variant> # stops dev-build before executing the phase 'build'
$ spack build-env <package> @develop <variant> -- bash # nests a bash shell with the build env vars loaded
# Work on the package!
# Use the build system of the package! (e.g. 'make')
# Use the testing infrastructure of the package!
$ exit # to exit the nested bash
If you want multiple dev-builds at the same time, label them with separate ``@<your-label>``.
The identifier ``@develop`` is common in the Spack documentation but you can use any string.


Environments with Spack develop
-------------------------------
@@ -95,3 +59,15 @@ To deactivate a Spack environment, type
Most of the Spack commands are sensitive to environments, see
`Spack docs <https://spack.readthedocs.io/en/latest/environments.html#environment-sensitive-commands>`__.

Plain dev-build
---------------

This is the easiest way to build local sources.
Enter the root of your source repository and execute:

.. code-block:: console
$ spack dev-build --until build <package> @<version>
This will build the package as is. The downside of this approach is that
you need to go through all phases of a package build.
83 changes: 46 additions & 37 deletions docs/QuickStart.rst
Original file line number Diff line number Diff line change
@@ -11,19 +11,17 @@ To set up a Spack instance, clone the repository using a specific Spack tag (lat
$ git clone --depth 1 --recurse-submodules --shallow-submodules -b $SPACK_TAG https://github.com/C2SM/spack-c2sm.git
To load it into your command line, execute
To load it into your command line, execute one of the following commands:

.. code-block:: console
$ . spack-c2sm/setup-env.sh
$ . spack-c2sm/setup-env.sh /user-environment
$ . spack-c2sm/setup-env.sh /mch-environment/v6
$ . spack-c2sm/setup-env.sh /mch-environment/v7
$ . spack-c2sm/setup-env.sh any_other_upstream
This auto-detects your machine and configures your instance for it.
You can force a machine with an argument. The name has to match a folder in sysconfigs.

.. code-block:: console
$ . spack-c2sm/setup-env.sh balfrin
This will make upstream installation from user-environment available in spack-c2sm.

Local machines and Containers
-----------------------------
@@ -84,21 +82,31 @@ ICON
----

ICON is built using environments.

For convenience, ICON provides bash scripts to set up the environment and install ICON for in-source
and out-of-source builds.
These scripts are located in ``config/cscs``, e.g. ``config/cscs/alps_mch.cpu.nvidia``.

For in-source builds, you need the run the configure scripts from your ICON root folder:

.. code-block:: console
$ ./config/cscs/alps_mch.cpu.nvidia
For out-of-source builds, navigate into your out-of-source directory (e.g., `cd cpu`) and run the configure scripts from there:

.. code-block:: console
$ ./../config/cscs/alps_mch.cpu.nvidia
For development, sometimes it is necessary to build ICON in a more customized way.
To do so please follow the instructions below.

Environments are located in a folder named after the environment and are defined in a ``spack.yaml`` file.
For ICON, they are located in ``config/cscs/spack/<version>/<machine>_<target>_<compiler>``.
For ICON, they are located in ``config/cscs/spack/<machine>_<target>_<compiler>``.
They work with a special Spack tag, that is provided in the ICON repository at ``config/cscs/SPACK_TAG_*``.
So make sure you clone Spack with the specified tag.

.. tip::
**On Balfrin:**
In case your Spack environment requires Python, a compatability issue
with `openssl` and `git` appears.

``/usr/bin/ssh: symbol lookup error: /usr/bin/ssh: undefined symbol: EVP_KDF_CTX_free, version OPENSSL_1_1_1d``

To circumvent that simply do
``spack load git`` prior to activation of the environment.

To activate the Spack environment, type

.. code-block:: console
@@ -109,14 +117,16 @@ To install the environment and so ICON, type

.. code-block:: console
$ spack develop --path $(pwd) icon@develop
$ spack install
Example to build ICON for CPU with NVHPC on Piz Daint:
Example to build ICON for CPU with NVHPC on Balfrin:

.. code-block:: console
$ SPACK_TAG=$(cat "config/cscs/SPACK_TAG_DAINT")
$ spack env activate -d config/cscs/spack/$SPACK_TAG/daint_cpu_nvhpc
$ SPACK_TAG=$(cat "config/cscs/SPACK_TAG_MCH")
$ spack env activate -d config/cscs/spack/mch_cpu_double
$ spack develop --path $(pwd) icon@develop
$ spack install
.. attention::
@@ -127,23 +137,22 @@ Out-of-source builds are possible as follows:

.. code-block:: console
$ mkdir cpu && cd cpu
$ cp -r ../config .
$ spack env activate -d config/cscs/spack/v0.20.1.5/daint_cpu_nvhpc
$ mkdir cpu
$ spack env activate -d config/cscs/spack/mch_cpu_double
$ # tell spack to build icon in folder cpu
$ spack develop --path $(pwd) --build-directory cpu icon@develop
$ spack install
.. attention::
Out-of-source build for AutotoolsPackages is not supported by Spack.
The implementation for ICON relies on some hacks inside package.py and
only works if the build-folder is located inside the Git repo of ICON.

COSMO
-----
By executing the commands above, spack will add some lines directly into ``spack.yaml``:

Building COSMO is not supported anymore starting with spack-c2sm v0.20.1.0 for CSCS.
.. code-block:: yaml
For Euler Cluster a CPU-only setup is supported using GCC 8.5.0

.. code-block:: console
spack:
packages:
icon:
package_attributes:
build_directory: /scratch/mch/juckerj/icon-nwp/cpu
$ spack install cosmo%gcc@8.5.0
Any further ``spack install`` command will use the build directory specified in the ``spack.yaml`` file.
In case you want to change the build directory, edit the ``spack.yaml`` file or remove the ``build_directory`` line
and run ``spack concretize -f`` afterwards.
26 changes: 26 additions & 0 deletions docs/Testing.rst
Original file line number Diff line number Diff line change
@@ -29,3 +29,29 @@ To test a PR, create a comment ``launch jenkins``.
Supported machines:

* balfrin

Jenkins test with uenv
----------------------
To test spack-c2sm with an uenv, add a stage

.. code-block:: bash
stage('Create uenv') {
steps {
sh """
git clone -b fix/jenkins https://github.com/eth-cscs/uenv.git
./uenv/install --yes --destdir=$WORKSPACE
source $WORKSPACE/etc/profile.d/uenv.sh
uenv repo create
uenv image pull mch/v8:rc2
"""
}
}
and run the tests in the uenv

.. code-block:: bash
source $WORKSPACE/etc/profile.d/uenv.sh
source ./setup-env.sh /user-environment
uenv run mch/v8:rc2 -- pytest -v -n auto test/integration_test.py
40 changes: 6 additions & 34 deletions repos/c2sm/packages/cosmo-eccodes-definitions/package.py
Original file line number Diff line number Diff line change
@@ -1,25 +1,3 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)

# ----------------------------------------------------------------------------
# If you submit this package back to Spack as a pull request,
# please first remove this boilerplate and all FIXME comments.
#
# This is a template package file for Spack. We've put "FIXME"
# next to all the things you'll want to change. Once you've handled
# them, you can save this file and test your package like this:
#
# spack install cosmo-eccodes-definitions
#
# You can edit this file again by typing:
#
# spack edit cosmo-eccodes-definitions
#
# See the Spack documentation for more information on packaging.
# ----------------------------------------------------------------------------

from spack import *


@@ -30,27 +8,21 @@ class CosmoEccodesDefinitions(Package):
url = "https://github.com/COSMO-ORG/eccodes-cosmo-resources.git"
git = 'https://github.com/COSMO-ORG/eccodes-cosmo-resources.git'

maintainers = ['dominichofer']
maintainers = ['huppd,lxavier']

version('2.36.0.3', tag='v2.36.0.3')
version('2.25.0.3', tag='v2.25.0.3')
version('2.25.0.2', tag='v2.25.0.2')
version('2.25.0.1', tag='v2.25.0.1')
version('2.19.0.7', tag='v2.19.0.7')
version('2.19.0.6', tag='v2.19.0.6')
version('2.19.0.5', tag='v2.19.0.5')
version('2.19.0.4', tag='v2.19.0.4')
version('2.19.0.3', tag='v2.19.0.3')
version('2.19.0.2', tag='v2.19.0.2')
version('2.19.0.1', tag='v2.19.0.1')
version('2.18.0.1', tag='v2.18.0.1')

depends_on('eccodes')
depends_on('eccodes@2.25.0',
depends_on('eccodes@2.36.4',
type=('build', 'link', 'run'),
when='@2.25.0.1:')
depends_on('eccodes@2.19.0',
when='@2.36.0.3')
depends_on('eccodes@2.25.0',
type=('build', 'link', 'run'),
when='@2.19.0.1:2.19.0.7')
when='@2.25.0.1:2.25.0.3')
depends_on('eccodes@2.18.0',
type=('build', 'link', 'run'),
when='@2.18.0.1')
Loading