Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misc theta-gpu job script cleanups #58

Merged
merged 1 commit into from
Nov 22, 2022
Merged

Conversation

carns
Copy link
Collaborator

@carns carns commented Nov 22, 2022

  • uniquify sandbox directory so that duplicate runs don't collide
  • remove dummy spack repo path from spack env recipe to avoid warning
  • don't explicitly install mochi-margo (this is not allowed in a spack env without adding to root specs); it is pulled in via mochi-ssg deependency anyway

@jhendersonHDF and @vchoi-hdfgroup does this look Ok to you? I'm trying to get it running and got closer after these changes, but then I still hit errors that look like this before it finishes installing everything:

==> Error: AttributeError: Query of package 'openmpi' for 'headers' failed
        prefix : None
        spec : [email protected]%[email protected]~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none patches=60ce20b schedulers=none arch=linux-ubuntu20.04-zen2
        queried as : openmpi
        extra parameters : []

The 'mochi-ssg' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure
==> Error: mochi-ssg-develop-pkmj42xollb5zhbxervo22vlt3rln36o: AttributeError: Query of package 'openmpi' for 'headers' failed
        prefix : None

Possibly this is a problem triggered by a recent spack update if you are not seeing it.

- uniquify sandbox directory so that duplicate runs don't collide
- remove dummy spack repo path from spack env recipe to avoid warning
- don't explicitly install mochi-margo (this is not allowed in a spack
  env without adding to root specs); it is pulled in via mochi-ssg
  deependency anyway
@jhendersonHDF
Copy link
Contributor

Looks good to me as far as my familiarity with spack goes. I'm not sure on the MPI issue though. So far I've only run things on Polaris and there I often run into issues with needing to load the CUDA modules. I'm not sure if that's also the case for theta-gpu, but spack/spack#12520 seemed to have the same issue.

@carns
Copy link
Collaborator Author

carns commented Nov 22, 2022

Interesting; thanks for the pointer to the spack issue. I don't think the explicit cuda modules are needed on Polaris, but from the discussion on that issue you linked to, it looks like this might potentially be solvable by specifying the prefix in addition to the module for the external OpenMPI package. I'm going to go ahead and merge the changes in this PR (they are orthogonal I believe) and try out that workaround.

@carns carns merged commit 3724955 into main Nov 22, 2022
@carns carns deleted the carns/theta-gpu-script-cleanups branch November 22, 2022 20:35
@github-actions github-actions bot locked and limited conversation to collaborators Nov 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants