Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INSTALL]: g2 v3.5.1 and g2tmpl v1.13.0 for spack-stack-1.8.0 #1180

Closed
1 of 14 tasks
WenMeng-NOAA opened this issue Jul 8, 2024 · 37 comments
Closed
1 of 14 tasks

[INSTALL]: g2 v3.5.1 and g2tmpl v1.13.0 for spack-stack-1.8.0 #1180

WenMeng-NOAA opened this issue Jul 8, 2024 · 37 comments
Assignees
@climbfuji @srherbener @RatkoVasic-NOAA @jkbk2004 @Hang-Lei-NOAA @AlexanderRichert-NOAA
Labels
NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center

Comments

@WenMeng-NOAA
Copy link

Package name

g2 and g2tmpl

Package version/tag

3.5.1 for g2; 1.13.0 for g2tmpl

Build options

None

Installation timeframe

For GEFS v13 development, the UPP updates require g2 v3.5.1 and g2tmpl v1.13.0 installations.

Other information

No response

WCOSS2

  • Check this box if and only if your package should be installed on WCOSS2 Cactus and Dogwood (all spack-stack packages will be installed on Acorn). If not, you may disregard the rest of the items below and submit this request.

WCOSS2: General questions

No response

WCOSS2: Installation and testing

No response

WCOSS2: Technical & security review list

  • The code is mature, stable, and production ready
  • The code is does not and cannot use the internet, and does not contain URLs (http, https, ftp, etc.) except in comments
  • The package does not contain prebuilt binary files that have not been approved by NCO security review
  • The code has no publicly disclosed cybersecurity vulnerabilities and exposures (search https://cve.mitre.org/cve/)
  • The code is not prohibited by DHS, DOC, NOAA, or NWS
  • The code comes from a trusted source. Trusted sources include other NWS, NOAA, or DOC, agencies, or other Federal agencies that operate at a FISMA high or equivalent level. Additionally, trusted sources could be third-party agencies through which there is an existing SLA on file (such as RedHat).
  • The code is actively maintained and supported (it continues to get updates, patches, etc.)
  • The code is not maintained by a private entity operating in a foreign country (if it is, make a note below)
  • There is sufficient documentation to support maintenance
  • There are no known security vulnerabilities or weaknesses
  • Installing and running the code does not require privileged processes/users
  • There are no software dependencies that are unapproved or have security concerns (if there are, make a note below)
  • There are no concerns related to SA, SI, and SC NIST control families

WCOSS2: Additional comments

No response
@climbfuji
Copy link
Collaborator

We are limiting the number of installations/modifications for existing spack-stack releases. Also, as far as I know the UFS applications are going to skip spack-stack-1.7.0 and go straight to 1.8.0 (from 1.6.0). Thus, we will include this update in the spack-stack-1.8.0 release, but not in the already installed 1.7.0 release.

@climbfuji climbfuji changed the title [INSTALL]: g2 v3.5.1 and g2tmpl v1.13.0 for the spack-stack 1.7.0 and above [INSTALL]: g2 v3.5.1 and g2tmpl v1.13.0 for the spack-stack 1.7. 0 and above - no, for spack-stack-1.8.0 Jul 8, 2024
@climbfuji climbfuji added NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center labels Jul 8, 2024
@WenMeng-NOAA WenMeng-NOAA mentioned this issue Jul 8, 2024
Closed
@AndrewBenjamin-NOAA
Copy link

@climbfuji How will this impact the timing of the installs of the g2 and g2tmpl library on RDHPCS machines?

@Hang-Lei-NOAA have you contacted anyone from the SPA team at NCO yet regarding the WCOSS2 installation?

My concern is giving @WenMeng-NOAA enough time to update and properly test UPP prior to the GEFS code freeze.

@climbfuji
Copy link
Collaborator

@climbfuji How will this impact the timing of the installs of the g2 and g2tmpl library on RDHPCS machines?

@Hang-Lei-NOAA have you contacted anyone from the SPA team at NCO yet regarding the WCOSS2 installation?

My concern is giving @WenMeng-NOAA enough time to update and properly test UPP prior to the GEFS code freeze.

If time for testing is a concern, then it's possible to request a test install on a single RDHPCS platform before the spack-stack-1.8.0 release. It's simply not feasible to amend existing spack-stack installations on all systems multiple times per week, and right now we get hammered with such requests from NOAA.

@Hang-Lei-NOAA
Copy link
Collaborator

Hang-Lei-NOAA commented Jul 9, 2024 via email

@AndrewBenjamin-NOAA
Copy link

@climbfuji if the schedule to install on all R&D platforms is still Q3 2024, that should work for the UPP group. If that were to get pushed back, then we would probably need to do a test install.

@AndrewBenjamin-NOAA
Copy link

@Hang-Lei-NOAA Thanks for the explanation. Let us know when the test area is staged on Dogwood.

@climbfuji
Copy link
Collaborator

@AndrewBenjamin-NOAA The plan is to release spack-stack-1.8.0 end of August/beginning of September and then roll it out. That would mean the new packages will be on all systems in the first 1-2 weeks of September. Thus, if you are referring to calendar years and not fiscal years, that would fit. Nonetheless, I would encourage a test install earlier on one platform so that we know that things work - last thing we want is to redo entire spack-stack installs.

@AndrewBenjamin-NOAA
Copy link

I would encourage a test install earlier on one platform so that we know that things work - last thing we want is to redo entire spack-stack installs.

@climbfuji That makes sense and I agree. I think the best course is to go ahead with the test install on Hera for UPP testing. Is that something you can set up or will the UPP group need to stage the testing area?

@climbfuji
Copy link
Collaborator

We have a spack-stack meeting today- will get back with you after that. Thanks!

@AlexanderRichert-NOAA
Copy link
Collaborator

@WenMeng-NOAA are you sure you want this under 1.7.0 as opposed to 1.6.0?

@WenMeng-NOAA
Copy link
Author

I would encourage a test install earlier on one platform so that we know that things work - last thing we want is to redo entire spack-stack installs.

@climbfuji That makes sense and I agree. I think the best course is to go ahead with the test install on Hera for UPP testing. Is that something you can set up or will the UPP group need to stage the testing area?

@climbfuji The test installation on Hera only is not sufficient for the UPP updates. That will break down the UPP support on other R&D platforms.

@AndrewBenjamin-NOAA
Copy link

@AlexanderRichert-NOAA, would you mind clarifying here?

@WenMeng-NOAA are you sure you want this under 1.7.0 as opposed to 1.6.0?

We are limiting the number of installations/modifications for existing spack-stack releases. Also, as far as I know the UFS applications are going to skip spack-stack-1.7.0 and go straight to 1.8.0 (from 1.6.0). Thus, we will include this update in the spack-stack-1.8.0 release, but not in the already installed 1.7.0 release.

We were under the impression that modification of an existing release is not possible.

Given UPP's need to support multiple R&D platforms, would the most likely solution be to have testing areas set up on all platforms UPP supports prior to 1.8.0's release?

@AlexanderRichert-NOAA
Copy link
Collaborator

@AndrewBenjamin-NOAA sure-- We can create "add-on" environments in each system on top of the unified environment (which is the piece that we don't want to go back and directly modify). So in this case, we could create another chained environment under whatever release is desired which would use [email protected] and [email protected] and rebuild their dependents, with the rest of the packages coming from that release's unified environment. So I'm assuming you'll want 1.6.0 since that's what UPP currently uses (though I don't know how big of a leap it would be to go to 1.7.0 in terms of how many UPP dependencies have changed versions).

@WenMeng-NOAA
Copy link
Author

@AndrewBenjamin-NOAA sure-- We can create "add-on" environments in each system on top of the unified environment (which is the piece that we don't want to go back and directly modify). So in this case, we could create another chained environment under whatever release is desired which would use [email protected] and [email protected] and rebuild their dependents, with the rest of the packages coming from that release's unified environment. So I'm assuming you'll want 1.6.0 since that's what UPP currently uses (though I don't know how big of a leap it would be to go to 1.7.0 in terms of how many UPP dependencies have changed versions).

@AlexanderRichert-NOAA The installations at the "add-on" environment under 1.6.0 should work for the UPP standalone (offline post). Eventually when ufs-weather-model is updated to 1.8.0, I will update the upp submodule for inline post.

@edwardhartnett
Copy link
Collaborator

Once these new NCEPLIBS releases are installed, the UPP crew can test against them. If they find a problem, we repeat this whole process.

Imagine a world in which UPP runs unit tests on GitHub, and confirms that new releases of NCEPLIBS work. In that case, all this work would not be needed. Tests would have proceeded within minutes of the NCEPLIBS releases, without involving Hang, Alex, or Andrew. Tests would have run on a computer in Bill Gate's closet, instead of NOAA machines.

Only after everything had been thoroughly tested would we ask for an update on NOAA machines. We would be much less likely to need to fix something and install again. This would be a significant savings for NOAA, the NCEPLIBS team, and the UPP team. Bugs that currently take more than a week to find, could be found within minutes.

@Hang-Lei-NOAA
Copy link
Collaborator

Hang-Lei-NOAA commented Jul 10, 2024 via email

@climbfuji
Copy link
Collaborator

@Hang-Lei-NOAA and @WenMeng-NOAA Didn't we agree that WCOSS2 specific communications, as long as WCOSS2 is not using spack-stack, will happen in its own repository? It's making it harder for us to track what needs to be done for spack-stack on the other systems. Apologies if I misunderstood previous conversations.

@WenMeng-NOAA
Copy link
Author

@climbfuji Sure. We will communicate with @Hang-Lei-NOAA for WCOSS2 testing for offline.

@Hang-Lei-NOAA
Copy link
Collaborator

Hang-Lei-NOAA commented Jul 10, 2024 via email

@climbfuji
Copy link
Collaborator

Sorry Dom, will pay more attention on it.
…
On Wed, Jul 10, 2024 at 12:41 PM WenMeng-NOAA @.> wrote: @climbfuji https://github.com/climbfuji Sure. We will communicate with @Hang-Lei-NOAA https://github.com/Hang-Lei-NOAA for WCOSS2 testing for offline. — Reply to this email directly, view it on GitHub <#1180 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKWSMFH2VNWEOKCGEGVXN7DZLVPZZAVCNFSM6AAAAABKQYNGGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQHE4TGNBTHE . You are receiving this because you were mentioned.Message ID: @.>

No problem! Just wanted to make sure whether I remembered correctly (sometimes, no, often, my memory is weak ...)

@AndrewBenjamin-NOAA
Copy link

Bringing this back to our attention: What is the status of installing "add-on" environments on the R&D machines for Wen to test UPP?

@AlexanderRichert-NOAA
Copy link
Collaborator

I've updated the upp-addon-env's under spack-stack-1.6.0 (per above discussion) on Hera, Jet, Gaea C5, Orion, and Hercules to include g2 3.4.5 and 3.5.1, and g2tmpl 1.12.0 and 1.13.0.

@WenMeng-NOAA
Copy link
Author

I've updated the upp-addon-env's under spack-stack-1.6.0 (per above discussion) on Hera, Jet, Gaea C5, Orion, and Hercules to include g2 3.4.5 and 3.5.1, and g2tmpl 1.12.0 and 1.13.0.

@AlexanderRichert-NOAA I conducted the UPP test on Hera and confirmed the expected changes with g2/3.5.1 and g2tmpl/1.13.0. Could you also install g2 3.4.5 and 3.5.1, and g2tmpl 1.12.0 and 1.13.0 under the upp-addon-env's of spack-stack-1.6.0 on s4 and noaacloud? Thanks!

@AlexanderRichert-NOAA
Copy link
Collaborator

Those are both JCSDA platforms (as far as spack-stack maintenance goes), and I don't have access to either. @srherbener @RatkoVasic-NOAA @natalie-perlin would you be able to assist?

@RatkoVasic-NOAA
Copy link
Collaborator

We don't have access to S4. @natalie-perlin might do that on cloud when she's back from conference.

@srherbener
Copy link
Collaborator

@AlexanderRichert-NOAA, I can help with S4, but I'm not sure I fully understand what needs to be done. Would it work for me to simply replicate what was done on Orion on S4? Would I be looking for spack-stack-1.6.0 as the upstream environment, and "upp-addon-env" as the chained environment? It looks like only Intel compiler is supported on S4, so will that be sufficient to do only Intel? Thanks!

@AlexanderRichert-NOAA
Copy link
Collaborator

Thanks @srherbener. That's correct, only Intel, and yes, we're updating the existing upp-addon-env under spack-stack-1.6.0, which chains to the unified env. I made the installation probably a bit overly elaborate, but here's the idea:

  • Update the spack.yaml based on what's on Orion (leaving out the '%gcc' specs); use the existing upstream setting that should already be there pointing to the 1.6.0 unified env. It should also contain the repos: [$env/envrepo] setting.
  • Copy in the envrepo/ directory from upp-addon-env on Orion so that the updated g2, g2tmpl, and grib-util recipes are available.
  • In common/modules.yaml, under 'modules:lmod:hierarchy' (currently there's just the one entry for 'mpi'), add 'g2virt' and 'g2tmplvirt' to that list.
    That should do it. Feel free to Slack me if it doesn't work on account of I forgot something...

@srherbener
Copy link
Collaborator

Thanks @AlexanderRichert-NOAA! I'll let you know if/when questions come up.

@srherbener srherbener self-assigned this Jul 26, 2024
@srherbener
Copy link
Collaborator

@AlexanderRichert-NOAA I have updated S4 and I can see the new g2 and g2tmpl versions. I made it all the way through to the end of updating the chained environment (upp-addon-env) including the lmod refresh and the building of the setup-meta-modules steps.

I think S4 is done, but it would be great if someone who knows what the new environment should look like to test my updates. Thanks!

@climbfuji climbfuji changed the title [INSTALL]: g2 v3.5.1 and g2tmpl v1.13.0 for the spack-stack 1.7. 0 and above - no, for spack-stack-1.8.0 [INSTALL]: g2 v3.5.1 and g2tmpl v1.13.0 for spack-stack-1.8.0 Aug 5, 2024
@jkbk2004
Copy link

@RatkoVasic-NOAA can you make sure that g2tmpl-1.13.0/g2-3.5.1 is available on Derecho with spack 1.6.0? New version of g2tmpl-1.13.0 version will be available on WCOSS2 anytime this week. We may update UPP directly with g2tmpl-1.13.0 for ufs-community/ufs-weather-model#2326.

@InnocentSouopgui-NOAA
Copy link
Collaborator

@AlexanderRichert-NOAA I have updated S4 and I can see the new g2 and g2tmpl versions. I made it all the way through to the end of updating the chained environment (upp-addon-env) including the lmod refresh and the building of the setup-meta-modules steps.

I think S4 is done, but it would be great if someone who knows what the new environment should look like to test my updates. Thanks!

@srherbener
I am testing a PR ufs-community/ufs-weather-model#2326 on S4. It uses this upp-addon-env that you installed. The compiling of ufs-weather-model is failing with the following error message

Could NOT find PIO (missing: C Fortran) (Required is at least version "2.5.3")

You can see error details in the pull request.

@InnocentSouopgui-NOAA
Copy link
Collaborator

@srherbener
The issue on S4 is solved.

@climbfuji
Copy link
Collaborator

Yes, I fixed it

@climbfuji
Copy link
Collaborator

was about to test but you were faster

@climbfuji
Copy link
Collaborator

The issue on S4 was that the spack module lmod refresh command was run without --upstream-modules, which is required for chained environments.

I am going to close this issue as completed.

@climbfuji
Copy link
Collaborator

@RatkoVasic-NOAA @AlexanderRichert-NOAA I noticed that configs/common/packages.yaml still lists the old g2/g2tmpl versions. I thought 1.8.0 should use 3.5.1 and 1.13.0.

@RatkoVasic-NOAA
Copy link
Collaborator

@RatkoVasic-NOAA @AlexanderRichert-NOAA I noticed that configs/common/packages.yaml still lists the old g2/g2tmpl versions. I thought 1.8.0 should use 3.5.1 and 1.13.0.

@climbfuji Good catch, I'll open PR.

@climbfuji climbfuji mentioned this issue Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@climbfuji climbfuji

@srherbener srherbener

@RatkoVasic-NOAA RatkoVasic-NOAA

@jkbk2004 jkbk2004

@Hang-Lei-NOAA Hang-Lei-NOAA

@AlexanderRichert-NOAA AlexanderRichert-NOAA

Labels
NOAA-EMC OAR-EPIC NOAA Oceanic and Atmospheric Research and Earth Prediction Innovation Center
Projects
No open projects
spack-stack-1.8.0 (2024 Q3)
Status: Done
Milestone
No milestone
Development

No branches or pull requests
10 participants
@climbfuji @srherbener @RatkoVasic-NOAA @jkbk2004 @edwardhartnett @Hang-Lei-NOAA @WenMeng-NOAA @AlexanderRichert-NOAA @AndrewBenjamin-NOAA @InnocentSouopgui-NOAA