Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test MAPL v2.46.3 in UFS weather model #2346

Open
Tracked by #2984
junwang-noaa opened this issue Jun 28, 2024 · 25 comments
Open
Tracked by #2984

Test MAPL v2.46.3 in UFS weather model #2346

junwang-noaa opened this issue Jun 28, 2024 · 25 comments
Assignees
@RatkoVasic-NOAA @ulmononian @natalie-perlin
Labels
enhancement New feature or request

Comments

@junwang-noaa
Copy link
Collaborator

junwang-noaa commented Jun 28, 2024 •
edited
Loading

Description

MAPL 2.46.2 has fixes for issue #2162. UFS weather model needs to be tested and updated with this version.

20240830:
MAPL 2.46.2 has a bug. MAPL 2.46.3 should be installed and tested in UFS weather model. The issue title is updated.

Solution

Alternatives

Related to
@junwang-noaa junwang-noaa added the enhancement New feature or request label Jun 28, 2024
@junwang-noaa
Copy link
Collaborator Author

@lipan-NOAA Can you confirm this version of MAPL works in GOCART for GEFSv13? Thanks

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA is MAPL 2.46.2 installed on Acorn/WCOSS2?

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jul 1, 2024 via email

@lipan-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA Can you tell me where you installed it?

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Jul 2, 2024 via email

@ulmononian
Copy link
Collaborator

do you want this installed in spack-stack/1.6.0? and on which machine?

@ulmononian ulmononian mentioned this issue Jul 2, 2024
Merged
@junwang-noaa junwang-noaa mentioned this issue Aug 12, 2024
Open
@junwang-noaa junwang-noaa changed the title Test MAPL v2.46.2 in UFS weather model Test MAPL v2.46.3 in UFS weather model Aug 30, 2024
@jkbk2004
Copy link
Collaborator

@DusanJovic-NOAA @junwang-noaa new maple and esmf version are available on hercules and orion for the test and debug activities. @RatkoVasic-NOAA thanks for the installation!

Hercules: /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/ue-esmf-8.6.1-mapl-2.46.3/install/modulefiles/Core
Orion: /work/noaa/epic/role-epic/spack-stack/orion/spack-stack-1.6.0/envs/ue-esmf-8.6.1-mapl-2.46.3/install/modulefiles/Core

@jkbk2004 jkbk2004 mentioned this issue Sep 24, 2024
Open
14 tasks
@mathomp4
Copy link

As an FYI, the MAPL 2.46.3 fix was when using externally initialized MPI. (Which is something you all do but we don't do internally).

We also had this notice to users:

Notice to users: External code should initialize MPI with MPI_THREAD_MULTIPLE as that is what MAPL expects. Also, code might need to call:

call ESMF_InitializePreMPI()

for all features of MAPL to be supported, namely if ESMF Single System Image (SSI) code is enabled. If users do not use this (or know what it is), it most likely is not needed.

My guess is you do not need to call ESMF_InitializePreMPI() as I don't think even we internally quite use all the SSI code yet.

@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Oct 4, 2024
Open
36 tasks
@DavidHuber-NOAA DavidHuber-NOAA mentioned this issue Nov 19, 2024
Open
@junwang-noaa junwang-noaa mentioned this issue Dec 2, 2024
Draft
14 tasks
@junwang-noaa
Copy link
Collaborator Author

@weiyuan-jiang may I ask what compiler/mpi versions you want to test? Thanks

@mathomp4
Copy link

mathomp4 commented Dec 2, 2024

@weiyuan-jiang may I ask what compiler/mpi versions you want to test? Thanks

@junwang-noaa Well for that version of MAPL, we would have been testing with:

  • Intel ifort 2021.6
  • Intel ifort 2021.12 (or .13)
  • GCC 13.2.0

We never had Intel ifort 2021.9 on any machines we have. Are any of these possible?

@junwang-noaa
Copy link
Collaborator Author

@AlexanderRichert-NOAA do we have these intel and GCC version on Hera or Hercules orGaea?

@AlexanderRichert-NOAA
Copy link
Collaborator

I can't get onto Hera at the moment but for the others:

  • Gaea (C5 & C6):
    • ifort 2021.6 (module load intel-classic/2022.1.0)
    • GCC 13.2 (module load gcc-native/13.2)
  • Hercules & Orion:
    • ifort 2021.12.0 (module load intel-oneapi-compilers/2024.1.0)

@mathomp4
Copy link

mathomp4 commented Dec 3, 2024

As we have access to Hercules/Orion, I guess ifort 2021.12 is our best bet at the moment (though I'd have to imagine there must be a recent-ish gcc on there).

I'm fairly certain we can run with 2021.12. There were some bugfixes needed in places of GEOS (I don't think MAPL, but I can find out), so by the time we got them all in, 2021.13 was out, but I think 2021.12 works.

Plus, we have 2021.12 on discover, so we can do some matching if need be.

@junwang-noaa
Copy link
Collaborator Author

@weiyuan-jiang @mathomp4 Which MAPL version would you like us to test in UFS? I assume the ESMF version is 8.6.1

@weiyuan-jiang
Copy link
Collaborator

@junwang-noaa It doesn't matter. I can always replace the MAPL as long as it can be built and run under new compiler on Hercules

@junwang-noaa
Copy link
Collaborator Author

@AlexanderRichert-NOAA would you please build ifort 2021.12.0 with ESMF 8.6.1 and MAPL v2.50.2 on Hercules in spack-stack 1.6.0? Thank you!

@AlexanderRichert-NOAA
Copy link
Collaborator

Since ifort 2021.12.0 is only available with oneAPI, will it work to use icx/icpx from the same module? If not, I can try mixing and matching to use icc/icpc, but it might take some fiddling with your modulefile to get it to work since the compilers won't all be available in the same module.

@junwang-noaa
Copy link
Collaborator Author

@AlexanderRichert-NOAA I actually do not know. Maybe we need to ask Hercules admin.

@mathomp4
Copy link

mathomp4 commented Dec 6, 2024

Since ifort 2021.12.0 is only available with oneAPI, will it work to use icx/icpx from the same module? If not, I can try mixing and matching to use icc/icpc, but it might take some fiddling with your modulefile to get it to work since the compilers won't all be available in the same module.

@AlexanderRichert-NOAA MAPL should be good with icx and icpx at least in recent times from my memory. A few other repos needed updates, but I don't think MAPL did...

@DusanJovic-NOAA
Copy link
Collaborator

@climbfuji has open PR with changes to support so-called mixed Intel compilers (icx, icpx, ifort) #2504

@DusanJovic-NOAA
Copy link
Collaborator

Actually, this will be a good test to confirm that we can use a spack_stack build libraries with llvm c/c++ compiler and classic Fortran compiler during the transition period.

@AlexanderRichert-NOAA
Copy link
Collaborator

I actually ended up running into some difficulties when using icx/icpx (I forget which exact package/s failed), so it ended up being easiest to use icc and icpc with the newer ifort. hercules:/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/modulefiles/Core

@DusanJovic-NOAA
Copy link
Collaborator

I've updated my branch to use this spack environment, but now compilation fails with errors like this:

/work/noaa/fv3-cam/djovic/ufs/update_gocart_mapl/ufs-weather-model/GOCART/Process_Library/GOCART2G_MieMod.F90(12): error #7012: The module file cannot be read.  Its format requires a more recent F90 compiler.   [NETCDF]
  use netcdf
-------^
/work/noaa/fv3-cam/djovic/ufs/update_gocart_mapl/ufs-weather-model/GOCART/Process_Library/GOCART2G_MieMod.F90(182): error #6404: This name does not have a type, and must have an explicit type.   [NF90_NOWRITE]
     rc = nf90_open(this%table_name, NF90_NOWRITE, ncid)
--------------------------------------^
/work/noaa/fv3-cam/djovic/ufs/update_gocart_mapl/ufs-weather-model/GOCART/Process_Library/GOCART2G_MieMod.F90(182): error #6404: This name does not have a type, and must have an explicit type.   [NF90_OPEN]
     rc = nf90_open(this%table_name, NF90_NOWRITE, ncid)
-----------^

I'm confused about which version of the compiler this environment is supposed to use. When I list available modules I see:

$ module use /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/modulefiles/Core

$ module avail

--------------------------------------- /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/modulefiles/Core ----------------------------------------
   stack-intel/2021.9.0

The stack-intel/2021.9.0 module is:

$ module show stack-intel/2021.9.0
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   /work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/modulefiles/Core/stack-intel/2021.9.0.lua:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
help([[]])
family("MetaCompiler")
conflict("stack-apple-clang")
conflict("stack-clang")
conflict("stack-gnu")
conflict("stack-intel")
conflict("stack-intel-oneapi-compilers")
load("intel-oneapi-compilers/2023.1.0")
prereq("intel-oneapi-compilers/2023.1.0")
prepend_path("MODULEPATH","/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/modulefiles/intel/2021.9.0")
setenv("F77","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/bin/ifort")
setenv("FC","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/bin/ifort")
setenv("CC","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2023.1.0-sb753366rvywq75zeg4ml5k5c72xgj72/compiler/2023.1.0/linux/bin/intel64/icc")
setenv("CXX","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2023.1.0-sb753366rvywq75zeg4ml5k5c72xgj72/compiler/2023.1.0/linux/bin/intel64/icpc")
setenv("SERIAL_F77","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/bin/ifort")
setenv("SERIAL_FC","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/bin/ifort")
setenv("SERIAL_CC","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2023.1.0-sb753366rvywq75zeg4ml5k5c72xgj72/compiler/2023.1.0/linux/bin/intel64/icc")
setenv("SERIAL_CXX","/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2023.1.0-sb753366rvywq75zeg4ml5k5c72xgj72/compiler/2023.1.0/linux/bin/intel64/icpc")
setenv("I_MPI_EXTRA_FILESYSTEM","ON")
whatis("Name: stack-intel")
whatis("Version: 2021.9.0")
whatis("Category: compiler")
whatis("Description: stack-intel compiler family and module access")

When I load it:

$ module load stack-intel/2021.9.0

$ echo $FC
/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2024.1.0-76aou5yu2oqfcz72vq7lp77bp3rqcoht/compiler/2024.1/bin/ifort

$ which ifort
/apps/spack-managed/gcc-11.3.1/intel-oneapi-compilers-2023.1.0-sb753366rvywq75zeg4ml5k5c72xgj72/compiler/2023.1.0/linux/bin/intel64/ifort

It sets $FC environment variable to ifort 2024.1, but since it also loads intel-oneapi-compilers/2023.1.0 the ifort in the $PATH is 2023.1. 

Module stack-intel version 2021.9.0 sets FC variable to the compiler version 2024.1.0 but also adds path to compiler version 2023.1.0. Very confusing, and inconsistent.

Is there anything I need to change in the ufs-weather-model modulefiles in order to fix this? Set some variable, remove or add something?

@AlexanderRichert-NOAA
Copy link
Collaborator

Can you run make verbosely to verify which ifort it's using in that step?

@DusanJovic-NOAA
Copy link
Collaborator 

[  0%] Building Fortran object GOCART/Process_Library/CMakeFiles/Process_Library.dir/GOCART2G_MieMod.F90.o
cd /work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_751888/compile_s2swa_intel/build_fv3_s2swa_intel/GOCART/Process_Library && /apps/spack-managed/oneapi-2023.1.0/intel-oneapi-mpi-2021.9.0-a66eaipzsnyrdgaqzxmqmqz64qzvhkse/mpi/2021.9.0/bin/mpiifort -DESMA64 -DHAS_NETCDF3 -DsysLinux -I/work/noaa/fv3-cam/djovic/ufs/update_gocart_mapl/ufs-weather-model/GOCART/Process_Library -I/work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_751888/compile_s2swa_intel/build_fv3_s2swa_intel/GOCART/Process_Library -I/work2/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_751888/compile_s2swa_intel/build_fv3_s2swa_intel/include/Process_Library -I/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/intel/2021.9.0/esmf-8.6.1-7z7d4jp/include -I/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/intel/2021.9.0/netcdf-c-4.9.2-tttotzi/include -I/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/intel/2021.9.0/netcdf-fortran-4.6.1-4qjzskp/include -I/work/noaa/epic/role-epic/spack-stack/hercules/spack-stack-1.6.0/envs/esmf-8.6.1-mapl-2.50.2-intel-mixed/install/intel/2021.9.0/parallelio-2.5.10-b7pdfd6/include -g -traceback -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -sox -align array64byte -qno-opt-dynamic-align -diag-disable 5462 -diag-disable 7712 -diag-disable 8291  -O2 -debug minimal -qoverride-limits -fp-model consistent -march=core-avx2 -module ../../include/Process_Library -qopenmp -c /work/noaa/fv3-cam/djovic/ufs/update_gocart_mapl/ufs-weather-model/GOCART/Process_Library/GOCART2G_MieMod.F90 -o CMakeFiles/Process_Library.dir/GOCART2G_MieMod.F90.o

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RatkoVasic-NOAA RatkoVasic-NOAA

@ulmononian ulmononian

@natalie-perlin natalie-perlin

Labels
enhancement New feature or request
Projects
Model infrastructure development FY25Q1
Status: In Progress
ufs-community User Support
Status: No status
Model infrastructure development FY2024Q4
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests