Develop Spack build infrastructure #6
Comments
|
Edited: 16-09-22
Scratch above. Should use the existing ACCESS-OM2 reproducibility test, which is run weekly on Jenkins https://accessdev.nci.org.au/jenkins/job/ACCESS-OM2/job/reproducibility/ It clones https://github.com/COSIMA/access-om2.git and runs: module use /g/data/hh5/public/modules && module load conda/analysis3-unstable && python -m pytest -s test/test_bit_reproducibility.pyThe test is in the repository https://github.com/COSIMA/access-om2/blob/master/test/test_bit_reproducibility.py Specifically this is the part that opens the existing log file, pulls out the checksums and compares them to the checksums just produced This is the truth output |
|
Discussion - If we do NOT use shared libraries, what's the practical difference between using Spack Vs CMake's FetchContent |
|
We need to support methods to easily locate and use ACCESS-NRI developed software, including models. Spack has sophisticated support for modulefiles, and has it's own environment system. |
|
Notes https://spack.readthedocs.io/en/latest/repositories.html
|
|
Notes https://spack.readthedocs.io/en/latest/packaging_guide.html
|
|
Notes https://spack.readthedocs.io/en/latest/build_settings.html#package-requirements
|
|
Notes https://spack.readthedocs.io/en/latest/build_settings.html#package-preferences |
|
Notes The aforementioned forum thread might be misleading. I followed the 2 step process (first build, then install): |
|
Notes If we want to statically link Parallelio, then Spack's |
|
Notes How to create a variant: How to depend on a variant: |
|
Notes
|
I realised I was incorrect when I stated (IRL) that libraries were statically linked in ACCESS-OM2. The libraries that were compiled were statically linked, but other dependencies like Now we're building most dependencies with We could do it on a case-by-case basis, e.g. it is desirable to have OpenMPI dynamically linked, as it allows for, say, a container built on a GitHub runner with the correct base OS to also run on Probably better is to have the same policy as before, to statically link those libraries for which we have source code responsibility, e.g. Edit: inserted blank line to make code snippet render correctly. |
Now that I know how to switch between the two types of dependencies in Spack, we can do either option at build time. However, the implications at runtime may need some attention. |
|
Notes When calling find_libraries(), don't include the ".a" or ".so" of the filename: lib/spack/llnl/util/filesystem.py: e.g. |
|
One method to package and deploy ACCESS-OM is to make a simple bundle spack package https://spack.readthedocs.io/en/latest/build_systems/bundlepackage.html Then the this repo could contain a spack.yaml file defining an environment with a specific version of the access-om package. Alternatively the spack.yaml could just contain all the individual packages that make up access-om2. The advantage of making it a bundle is that it could be installed independently for development purposes. |
|
Notes This is a Python Context Manager that makes it easier to work with subdirectories in builds. You use this with the Python with statement to change into a working directory, and when the with block is done, you change back to the original directory. Think of it as a safe pushd / popd combination, where popd is guaranteed to be called at the end, even if exceptions are thrown. |
|
Notes |
|
Installing Perl package in Spack with an old Intel compiler (2019.5.281)
|
|
How do we want to refer to ACCESS-OM versions? Clearly there is a "major version" (ACCESS-OM2, ACCESS-OM3), but below that we have no standard for versioning. Options: This is designed with a particular model of software development in mind. The first requirement is
I'm unsure how well this fits a complex scientific model such as this. The API in this case would be the configuration which is used to run the model. Regardless this is largely the default method of versioning. Calendar versioning has a venerable history. I personally think it is probably a better fit for a project likes ACCESS-OM2 which lacks the cohesive nature of many traditional software projects. It has the advantage of encoding useful information in the version about how old it is. It is supported in spack.
Probably best avoided. It doesn't provide much semantic meaning, and calendar versioning can easily be mapped on to specific commit hashes with spack or via tags in the repo. |
|
Calendar sounds good! I would use YYYY.MM.MICRO, so it looks like this: ACCESS-OM3-2023.03.002 This is the version of ACCESS-OM3 initially released in March 2023, with micro version 2. Note the use of three digits for the micro version, so that it's not mistaken for the day. You can also introduce a release tag for the Spack built stuff. For example: ACCESS-OM3-2023.03.002-1 This is the same version of ACCESS-OM3 as before, but build with Spack, release 1. Why the Spack release version? So that you can signal that the way in which the package is built has changed, but not the sources. So ACCESS-OM3-2023.03.002-2 would correspond to exactly the same sources, but with some changes in how it was compiled. |
|
I think I am in agreement with @micaeljtoliveira, and thanks for providing the detailed justification. I can see the logic of versioning the spack release, but I don't know how much utility it has in practice. If we changed the way My thought was that a model configuration would target a specific build of the model to ensure reproducibility. It would not automatically pick up a new build (with a new hash) unless it was deliberately changed to do so. |
|
Notes Using the system openmpi as an "external" from Spack appears to trip over the non-standard directory structure on gadi. e.g. |
What is the error message for this? It is valuable to have it explicitly documented in case there is a similar error again (likely). |
|
|
Notes
|
|
Notes |
|
[Updated: 10/08/2023] Notes For example, the following compiles on |
|
|
Using the system openmpi 4.1.5 as an "external" from Spack: |
|
How have you defined the openmpi external? Does it include a module load? I don't think the CMake is very robust here Worth sprinkling some |
|
I presume the above is due to the non-standard directory structure that is not understood by CMake. e.g. Module loading is enabled in
|
|
Maybe you could try to set these variables
export CC=mpicc
export FC=mpif90
export CXX=mpicxx
On 4 Jul 2023, at 4:21 pm, Harshula Jayasuriya ***@***.******@***.***>> wrote:
I presume the above is due to the non-standard directory structure that is not understood by CMake. e.g. /apps/openmpi/4.1.5/lib/Intel/
Module loading is enabled in etc/spack/defaults/packages.yaml:
- spec: ***@***.***
prefix: /apps/openmpi/4.1.5
modules:
- openmpi/4.1.5
LD_LIBRARY_PATH=/apps/openmpi/4.1.5/lib:/apps/openmpi/4.1.5/lib/profilers:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:/apps/intel-ct/2021.6.0/compiler/linux/lib; export LD_LIBRARY_PATH
—
Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AESFTSUV4CXKUW7DKPAW4WTXOOY53ANCNFSM6AAAAAAQHMDQLY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi @rxy900 , Same error: The above environment variables are likely overridden by Spack: spack-build-env-mods.txt: spack-build-env.txt: |
|
[Updated: 28/07/2023] Notes Convert the Spack built executables link against openmpi via That's the reason we need to use |
For context - cmake gets the library and include paths by inspecting |
harshula
changed the title
|
[Updated: 07/07/2023] There are three scenarios of Spack built ACCESS-OM2 being discussed here: a) Built with Spack built openmpi and runs with Spack built openmpi.
b) Built with Spack built openmpi and runs with system openmpi.
c) Built with system openmpi and run with system openmpi.
|
|
Hi @ScottWales, What are you thoughts about adding nci-openmpi to https://github.com/ACCESS-NRI/spack_packages ? If you would like to, please create a PR. |
|
Installation instructions for model developers on Gadi v3
|
|
It's the Intel |
Thanks @harshula. No criticism, but I think we really want to get to a point where people can use I guess something like this, or a development of this https://forum.access-hive.org.au/t/shared-access-spack-configuration/227/ |
|
@aidanheerdegen You should open a separate issue, outside of the ACCESS-OM repository, instead of adding additional requests on a closed issue. |
Develop spack based build infrastructure.
The text was updated successfully, but these errors were encountered: