release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates, bug fix in depencies for awscli-v2, bump wgrib2 to 3.5.0 and re-enable for all compilers #1486

climbfuji · 2025-01-31T03:32:15Z

Summary

Add [email protected] to templates skylab-dev and unified-dev (new version was added in recently merged PR Update crtm(-fix), wgrib2 spack#510)
Bump wgrib2 from 3.1.1 to 3.5.0 and re-enable for all compilers in spack-ext packages (new version was added in recently merged PR Update crtm(-fix), wgrib2 spack#510). Note. [email protected] doesn't compile on macOS with apple-clang (version 14.0.3 on the CI runner), see wgrib 3.5.0 does not compile with apple-clang 14.0.3 on macOS NOAA-EMC/wgrib2#312. But 3.4.0 does compile, thereforeuse this version on macOS only
Update spack submodule pointer for PR Update crtm(-fix), wgrib2 spack#510 and the changes in release/1.9.0: Fix bug in awcli-v2, add upper bound for py-cryptography spack#511 (fix upper bound for py-cryptography in awscli-v2) and release/1.9.0: Bug fix in wgrib2: apply '-Wno-error=implicit-function-declaration' for LLVM clang spack#513 (bug fix for wgrib2 with apple-clang)

Testing

CI

Applications affected

All applications using unified-dev or skylab-dev templates

Systems affected

None

Dependencies

Issue(s) addressed

Closes #967

Checklist

This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
These changes have been tested on the affected systems and applications.
All dependency PRs/issues have been resolved and this PR can be merged.

…d testing

…ompilers

…into bugfix/awscliv2_crytography_PLUS_crtm_updates

…sites/tier2/macos.default/packages.yaml

climbfuji · 2025-01-31T12:47:39Z

@AlexanderRichert-NOAA Compile error for the new wgrib2 on macOS with apple-clang, both with the extra variant ~openmp and without it: https://github.com/JCSDA/spack-stack/actions/runs/13072810448/job/36477973444?pr=1486 - note that [email protected] built fine on macOS with that compiler. Are 3.5.0 and 3.1.1 functionally equivalent and have the same API? If so, we could keep the version on macOS as 3.1.1 for this release, file a bug in the wgrib2 repository, and deal with it after the 1.9.0 release.

climbfuji · 2025-01-31T14:57:02Z

Testing workaround to use [email protected] on macOS, but 3.5.0 everywhere else.

climbfuji · 2025-01-31T15:05:01Z

Next error: With the updated spack recipe, can no longer fetch [email protected]. @AlexanderRichert-NOAA

===
No duplicates found.
==> Warning: Error while fetching [email protected]
  Unable to parse extension from https://www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/wgrib2.tgz.v3.1.1.

If this URL is for a tarball but does not include the file extension
in the name, you can explicitly declare it with the following syntax:

    version('1.2.3', 'hash', extension='tar.gz')

If this URL is for a download like a .jar or .whl that does not need
to be expanded, or an uncompressed installation script, you can tell
Spack not to expand it with the following syntax:

    version('1.2.3', 'hash', expand=False)

==> Error: Failed downloads:
==> Skipping [email protected]%[email protected]~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release patches=dbc3892 arch=darwin-ventura-m1/n5bd7lt as it is an external spec.
==> Skipping [email protected]%[email protected]+gssapi+ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=secure_transport arch=darwin-ventura-m1/j7kte32 as it is an external spec.
==> Skipping [email protected]%[email protected]+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=darwin-ventura-m1/hf4bewp as it is an external spec.
==> Skipping [email protected]%[email protected]+sigsegv build_system=autotools arch=darwin-ventura-m1/2t2jx64 as it is an external spec.
==> Skipping [email protected]%[email protected]~cpanm+opcode+open+shared+threads build_system=generic arch=darwin-ventura-m1/7ba2xam as it is an external spec.
==> Skipping [email protected]%[email protected] build_system=autotools libs=shared,static arch=darwin-ventura-m1/iiwylwe as it is an external spec.
==> Skipping [email protected]%[email protected]+gssapi build_system=autotools patches=3505c58,d886b98 arch=darwin-ventura-m1/v3o6rh7 as it is an external spec.
==> Skipping [email protected]%[email protected] build_system=bundle arch=darwin-ventura-m1/4gj24zj as it is an external spec.
==> Skipping [email protected]%[email protected]+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=darwin-ventura-m1/vlzoy2e as it is an external spec.
==> Skipping [email protected]%[email protected]~libpsl~pcre~python+zlib build_system=autotools ssl=openssl arch=darwin-ventura-m1/zqrupjl as it is an external spec.
==> Skipping [email protected]%[email protected]+lex~nls build_system=autotools patches=f8b85a0 arch=darwin-ventura-m1/nbss4q6 as it is an external spec.
==> Skipping [email protected]%[email protected] build_system=makefile arch=darwin-ventura-m1/6wkzjjc as it is an external spec.
==> Skipping [email protected]%[email protected]~dbus~debug~doc~examples+framework~gtk+gui~location~opengl~phonon+shared+sql+ssl+tools~webkit build_system=generic patches=51aeba5,7f34d48,c2d75d9 arch=darwin-ventura-m1/6ubtqzf as it is an external spec.
==> Skipping [email protected]%[email protected]~pcre build_system=autotools arch=darwin-ventura-m1/tmocriw as it is an external spec.

AlexanderRichert-NOAA · 2025-01-31T16:35:48Z

I'll push a fix momentarily, looks like a couple lines got lost between main Spack and wgrib2 repo copies of the recipe...

…crytography_PLUS_crtm_updates

AlexanderRichert-NOAA · 2025-01-31T17:02:44Z

fix: JCSDA/spack#512

…://github.com/climbfuji/spack-stack into bugfix/awscliv2_crytography_PLUS_crtm_updates

climbfuji · 2025-01-31T17:11:49Z

Issue for wgrib2 build error on macOS: NOAA-EMC/wgrib2#312

climbfuji · 2025-01-31T17:15:31Z

Sigh, now it fails again in fetching wgrib2 3.1.1. ...

==> Warning: Error while fetching [email protected]
  All fetchers failed for spack-stage-crtm-3.1.1-build1-5uhgqjzjxhmtgt57hfdxtmxwcvxljgtx

configs/templates/unified-dev/spack.yaml

AlexanderRichert-NOAA · 2025-01-31T17:28:32Z

I think it's a separate issue, but the [email protected] in the unified dev template should be crtm@=3.1.1-build1.

…tead of crtm@=3.1.1-build1, and add back in [email protected]

…into bugfix/awscliv2_crytography_PLUS_crtm_updates

AlexanderRichert-NOAA · 2025-01-31T18:12:20Z

I can see that the crtm recipe defines a version for v3.1.1-build1, I'm not sure if that was an oversight on my part or a change you made, but in any case if you can update it to be self consistent one way or the other we should be good.

climbfuji · 2025-01-31T18:28:34Z

I can see that the crtm recipe defines a version for v3.1.1-build1, I'm not sure if that was an oversight on my part or a change you made, but in any case if you can update it to be self consistent one way or the other we should be good.

I didn't add the v, but it's fine, it works with the v now. The wgrib2 3.1.1 for macOS still fails, because the new logic doesn't work as intended for apple-clang

…d testing

…e packages

climbfuji · 2025-02-03T17:04:06Z

I can see that the crtm recipe defines a version for v3.1.1-build1, I'm not sure if that was an oversight on my part or a change you made, but in any case if you can update it to be self consistent one way or the other we should be good.

I didn't add the v, but it's fine, it works with the v now. The wgrib2 3.1.1 for macOS still fails, because the new logic doesn't work as intended for apple-clang

v3.4.0 of wgrib2 works on macOS, so let's use that

…elop * Update .gitmodules and doc/source/conf.py for spack-stack release/1.9.0 * Avoid linking to libirc.so in spack (parallel-netcdf), turn off crypt variant for Python, and update Orion site config to fix tar issue (#1435) 1. Applications built with spack-stack packages esmf, parallelio, parallel-netcdf have libirc.so dynamically linked. Applications linked against libirc.so fail to start up. See Avoid linking to Intel's libirc.so library (aka bad configure script of package parallel-netcdf) #1436. The spack PR that is part of the suggested changes here fixes this by replacing libirc.so with libintlc.so in the parallel-netcdf build. See Bug fix in parallel-netcdf to avoid linking to libirc.so AND cherry-pick spack develop PR 48251 (conflict Intel Classic with [email protected]) spack#495. 2. Turn off crypt variant for Python; this variant leads to build errors with Intel in py-cryptography unless external curl and openssl are removed, which itself is problematic. 3. Add external wget on Orion, latest versions don't build with Intel on the machine. --------- Co-authored-by: Stephen Herbener <[email protected]> * Update ectrans from 1.2.0 to 1.5.0 in configs/common/packages.yaml (#1474) * Update .gitmodules and submodule pointer for spack for code review and testing * In spack-ext/lib/jcsda-emc/spack-stack/stack, update meta_modules.py and templates/{mpi,mpi.lua}: set compiler paths in MPI meta modules directly using SUBSTITUTES_SAVE, not using environment variables (#1479) * Revert .gitmodules and update submodule pointer for spack * release/1.9.0: Update instructions for setting up spack-stack with Nvidia compilers (#1462) This PR brings the Nvidia instructions a bit more up-to-date. On develop, the instructions only worked with Ubuntu 22.04 spack-stack 34bfda1 [email protected] With this PR, these constraints are updated to the slightly more recent Ubuntu 24.04 spack-stack 26901af [email protected] * For orion, intel config, pin py-numpy to version 1.26. This prevents (#1482) getting unwanted duplicate packages during concretize. * release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates, bug fix in depencies for awscli-v2, bump wgrib2 to 3.5.0 and re-enable for all compilers (#1486) 1. Add [email protected] to templates skylab-dev and unified-dev (new version was added in recently merged PR Update crtm(-fix), wgrib2 spack#510) 2. Bump wgrib2 from 3.1.1 to 3.5.0 and re-enable for all compilers in spack-ext packages (new version was added in recently merged PR Update crtm(-fix), wgrib2 spack#510). Note. [email protected] doesn't compile on macOS with apple-clang (version 14.0.3 on the CI runner), see wgrib 3.5.0 does not compile with apple-clang 14.0.3 on macOS NOAA-EMC/wgrib2#312. But 3.4.0 does compile, thereforeuse this version on macOS only 3. Update spack submodule pointer for PR Update crtm(-fix), wgrib2 spack#510 and the changes in release/1.9.0: Fix bug in awcli-v2, add upper bound for py-cryptography spack#511 (fix upper bound for py-cryptography in awscli-v2) and release/1.9.0: Bug fix in wgrib2: apply '-Wno-error=implicit-function-declaration' for LLVM clang spack#513 (bug fix for wgrib2 with apple-clang) --------- Co-authored-by: Alex Richert <[email protected]> * Update .gitmodules and submodule pointer for spack for code review and testing * For release/1.9.0: cherry-pick `[email protected]: ~mpi` instead of `+mpi` from #1489 (#1491) In PR #1489 we are changing the requirements for py-netcdf4 from [email protected]: +mpi to [email protected]: ~mpi in configs/common/packages.yaml. This change is required to fix an error with py-netcdf4 on certain systems when built with +mpi. We used to build py-netcdf4 without mpi, but for a period this wasn't possible until we added a patch to disable the py-netcdf4 auto-detect parallel feature. That patch allows us to build py-netcdf4 ~mpi even if netcdf-c was built with +mpi. --------- Co-authored-by: Alex Richert <[email protected]> * Revert .gitmodules and update submodule pointer for spack --------- Co-authored-by: Stephen Herbener <[email protected]> Co-authored-by: Francois Hebert <[email protected]> Co-authored-by: Stephen Herbener <[email protected]> Co-authored-by: Alex Richert <[email protected]>

climbfuji added 4 commits January 30, 2025 20:20

Update .gitmodules and submodule pointer for spack for code review an…

2cca3da

…d testing

Add [email protected] to templates skylab-dev and unified-dev

3b6e18d

In configs/common/packages.yaml, bump wgrib2 from 3.1.1 to 3.5.0

99b95b9

In spack-ext/repos/spack-stack/packages/*, re-enable wgrib2 for all c…

e40dbda

…ompilers

climbfuji self-assigned this Jan 31, 2025

climbfuji and others added 5 commits January 30, 2025 20:32

Merge branch 'release/1.9.0' of https://github.com/jcsda/spack-stack …

735563c

…into bugfix/awscliv2_crytography_PLUS_crtm_updates

crtm & wgrib2 version updates

d07a817

update unified-dev template

cb561dd

For testing only: remove ~openmp requirement for wgrib2 from configs/…

05f1830

…sites/tier2/macos.default/packages.yaml

Revert .gitmodules and update submodule pointer for spack

3af2c8d

climbfuji mentioned this pull request Jan 31, 2025

crtm & wgrib2 version updates #1487

Closed

3 tasks

climbfuji marked this pull request as ready for review January 31, 2025 12:43

Use [email protected] on macOS, but 3.5.0 everywhere else

1a38d2c

Merge remote-tracking branch 'ajwr/crtm_wgrib2' into bugfix/awscliv2_…

ddc90ab

…crytography_PLUS_crtm_updates

climbfuji added 3 commits January 31, 2025 10:06

Update submodule pointer for spack

2e022f5

Merge branch 'bugfix/awscliv2_crytography_PLUS_crtm_updates' of https…

32247e9

…://github.com/climbfuji/spack-stack into bugfix/awscliv2_crytography_PLUS_crtm_updates

Update submodule pointer for spack

c69978b

climbfuji mentioned this pull request Jan 31, 2025

release/1.9.0: Pin py-numpy version to 1.26 for Orion Intel configuration #1482

Merged

5 tasks

climbfuji commented Jan 31, 2025

View reviewed changes

configs/templates/unified-dev/spack.yaml Show resolved Hide resolved

climbfuji added 2 commits January 31, 2025 10:34

Fix configs/templates/unified-dev/spack.yaml: crtm@=v3.1.1-build1 ins…

4437966

…tead of crtm@=3.1.1-build1, and add back in [email protected]

Merge branch 'release/1.9.0' of https://github.com/jcsda/spack-stack …

f7a59fb

…into bugfix/awscliv2_crytography_PLUS_crtm_updates

climbfuji mentioned this pull request Jan 31, 2025

release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates #1454

Closed

5 tasks

climbfuji added 4 commits January 31, 2025 11:31

Update .gitmodules and submodule pointer for spack for code review an…

2f18b5d

…d testing

Rename crtm version from v3.1.1-build1 to 3.1.1-build1

9343ae7

In .github/workflows/*.yaml, add crtm-fix to list of allowed duplicat…

9430415

…e packages

On macOS, use [email protected]~openmp

1c1e983

climbfuji mentioned this pull request Feb 3, 2025

release/1.9.0: Bug fix in wgrib2: apply '-Wno-error=implicit-function-declaration' for LLVM clang JCSDA/spack#513

Merged

climbfuji requested review from AlexanderRichert-NOAA, stiggy87, BenjaminTJohnson and RatkoVasic-NOAA February 3, 2025 17:02

AlexanderRichert-NOAA approved these changes Feb 3, 2025

View reviewed changes

Revert .gitmodules and update submodule pointer for spack

7616fff

stiggy87 approved these changes Feb 3, 2025

View reviewed changes

BenjaminTJohnson approved these changes Feb 3, 2025

View reviewed changes

climbfuji merged commit 81969a3 into JCSDA:release/1.9.0 Feb 3, 2025
9 checks passed

climbfuji deleted the bugfix/awscliv2_crytography_PLUS_crtm_updates branch February 3, 2025 21:12

This was referenced Feb 3, 2025

[INSTALL]: CRTM 3.1.1 for emc #1185

Closed

wgrib2 2.0.8/3.1.1 do not build with Intel oneAPI LLVM compilers #967

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates, bug fix in depencies for awscli-v2, bump wgrib2 to 3.5.0 and re-enable for all compilers #1486

release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates, bug fix in depencies for awscli-v2, bump wgrib2 to 3.5.0 and re-enable for all compilers #1486

climbfuji commented Jan 31, 2025 •

edited

Loading

climbfuji commented Jan 31, 2025

climbfuji commented Jan 31, 2025

climbfuji commented Jan 31, 2025

AlexanderRichert-NOAA commented Jan 31, 2025

AlexanderRichert-NOAA commented Jan 31, 2025

climbfuji commented Jan 31, 2025

climbfuji commented Jan 31, 2025 •

edited

Loading

AlexanderRichert-NOAA commented Jan 31, 2025

AlexanderRichert-NOAA commented Jan 31, 2025

climbfuji commented Jan 31, 2025

climbfuji commented Feb 3, 2025

release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates, bug fix in depencies for awscli-v2, bump wgrib2 to 3.5.0 and re-enable for all compilers #1486

release/1.9.0: Add [email protected] to unified-dev and skylab-dev templates, bug fix in depencies for awscli-v2, bump wgrib2 to 3.5.0 and re-enable for all compilers #1486

Conversation

climbfuji commented Jan 31, 2025 • edited Loading

Summary

Testing

Applications affected

Systems affected

Dependencies

Issue(s) addressed

Checklist

climbfuji commented Jan 31, 2025

climbfuji commented Jan 31, 2025

climbfuji commented Jan 31, 2025

AlexanderRichert-NOAA commented Jan 31, 2025

AlexanderRichert-NOAA commented Jan 31, 2025

climbfuji commented Jan 31, 2025

climbfuji commented Jan 31, 2025 • edited Loading

AlexanderRichert-NOAA commented Jan 31, 2025

AlexanderRichert-NOAA commented Jan 31, 2025

climbfuji commented Jan 31, 2025

climbfuji commented Feb 3, 2025

climbfuji commented Jan 31, 2025 •

edited

Loading

climbfuji commented Jan 31, 2025 •

edited

Loading