Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update cmake build with GTG code #272

Closed
KateFriedman-NOAA opened this issue Mar 2, 2021 · 17 comments · Fixed by #294
Closed

Update cmake build with GTG code #272

KateFriedman-NOAA opened this issue Mar 2, 2021 · 17 comments · Fixed by #294
Labels
bug Something isn't working

Comments

@KateFriedman-NOAA
Copy link
Member

The current EMC_post develop branch does not build with GTG code.

Tried to build develop (33eb60e) on WCOSS-Dell with GTG code copied in; build failed:

/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/logs/build_ncep_post_33eb60e.log

...
[ 19%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/TABLE.f.o
[ 19%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/TABLEQ.f.o
[ 19%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/UPP_MATH.f.o
[ 19%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/xml_perl_data.f.o
[ 20%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/gtg_ctlblk.f90.o
[ 21%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/native_endianness.f.o
[ 21%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/AllGETHERV_GSD.f.o
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(13): warning #5117: Bad # preprocessor line
#ifndef NCAR
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(15): warning #5117: Bad # preprocessor line
#endif
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(19): warning #5117: Bad # preprocessor line
#ifdef NCAR
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(23): warning #5117: Bad # preprocessor line
#endif
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(22): error #6401: The attributes of this name conflict with those made accessible by a USE statement.   [SPVAL]
  real,parameter :: SPVAL=9.9E10
--------------------^
compilation aborted for /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90 (code 1)
make[2]: *** [sorc/ncep_post.fd/CMakeFiles/upp.dir/gtg_ctlblk.f90.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [sorc/ncep_post.fd/CMakeFiles/upp.dir/all] Error 2
make: *** [all] Error 2
+ make install
[  1%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/physcons.f.o
[  2%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/GFIP3.f.o
[  3%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/UPP_PHYSICS.f.o
[  4%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/grib2_module.f.o
[  5%] Building Fortran object sorc/ncep_post.fd/CMakeFiles/upp.dir/gtg_ctlblk.f90.o
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(13): warning #5117: Bad # preprocessor line
#ifndef NCAR
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(15): warning #5117: Bad # preprocessor line
#endif
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(19): warning #5117: Bad # preprocessor line
#ifdef NCAR
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(23): warning #5117: Bad # preprocessor line
#endif
-^
/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90(22): error #6401: The attributes of this name conflict with those made accessible by a USE statement.   [SPVAL]
  real,parameter :: SPVAL=9.9E10
--------------------^
compilation aborted for /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/sorc/ncep_post.fd/gtg_ctlblk.f90 (code 1)
make[2]: *** [sorc/ncep_post.fd/CMakeFiles/upp.dir/gtg_ctlblk.f90.o] Error 1
make[1]: *** [sorc/ncep_post.fd/CMakeFiles/upp.dir/all] Error 2
make: *** [all] Error 2
+ rm -rf /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/exec
+ mkdir /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/exec
+ cp /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/tests/install/bin/upp.x /gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/exec/.
cp: cannot stat ‘/gpfs/dell2/emc/modeling/save/Kate.Friedman/git/global-workflow/feature-hpc-stack/sorc/gfs_post.fd/tests/install/bin/upp.x’: No such file or directory
@WenMeng-NOAA
Copy link
Collaborator

@YaliMao-NOAA Please see the compiling errors from gtg code above.

@YaliMao-NOAA
Copy link
Contributor

Kate,
Did you use '-fpp' for compiling?

@kgerheiser
Copy link
Contributor

The lowercase .f90 means the pre-processor won't be called. If there are #ifdefs in the code it should be .F90 or -fpp should be used.

@WenMeng-NOAA
Copy link
Collaborator

@YaliMao-NOAA Could you try Kyle's suggestion for changing gtg code from .f90 to .F90 via cmake build?

@WenMeng-NOAA
Copy link
Collaborator

@YaliMao-NOAA You may find the instruction of build UPP via camke at
https://github.com/NOAA-EMC/EMC_post/wiki/Building-UPP#build-standalone-upp-via-cmake

@YaliMao-NOAA
Copy link
Contributor

@kgerheiser I was able to duplicate the error messages as Kate did, so I took your suggestion. After changing the file extensions from f90 to F90 and modifying sorc/ncep_post.fd/CMakeLists.txt, I ran into another error message:
/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/git/EMC_post_wafs/sorc/ncep_post.fd/gtg_indices.F90(7588): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [GDSWZD_MOD]
use GDSWZD_MOD
--------^

GDSWZD_MOD comes from module ip:
/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/src/hpc-stack/pkg/ip-v3.3.3/src/gdswzd_mod.f90

Looking into tests/build/sorc/ncep_post.fd/CMakeFiles/upp.dir/flags.make, I didn't see ip/3.3.3/include_4 included in Fortran_INCLUDES. Is this the reason why I got the error message as above?
Fortran_INCLUDES = -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/bacio/2.4.1/include_4 -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/crtm/2.3.0/include -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/g2/3.4.1/include_4 -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/png/1.6.35/include -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/zlib/1.2.11/include -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/g2tmpl/1.9.1/include -I/gpfs/dell1/usrx/local/prod/intel/2018UP01/compilers_and_libraries_2018.1.163/linux/mpi/intel64/include -I/gpfs/dell2/usrx/local/nceplibs/dev/hpc-stack/libs/hpc-stack/ips-18.0.1.163/impi-18.0.1/netcdf/4.7.4/include

@kgerheiser
Copy link
Contributor

kgerheiser commented Mar 5, 2021

Is there a missing target_link_library(ip::ip_4) in CMake?

@YaliMao-NOAA
Copy link
Contributor

@kgerheiser Are you talking about cmake/PackageConfig.cmake.in? There is a line:
#find_dependency(ip CONFIG)

@YaliMao-NOAA
Copy link
Contributor

@kgerheiser May you please take a look when you get a chance? The package is on Venus at /gpfs/dell2/emc/modeling/noscrub/Yali.Mao/git/EMC_post_wafs. ip/v3.3.3 is loaded successfully, somehow IP_INC4 is not included in Fortran_INCLUDES for compiling. Thank you!

@kgerheiser
Copy link
Contributor

Is there a branch or somewhere I can get the code from and build myself?

@YaliMao-NOAA
Copy link
Contributor

@kgerheiser I followed Wen's instruction earlier, https://github.com/NOAA-EMC/EMC_post/wiki/Building-UPP#build-standalone-upp-via-cmake.
If you copy my directory, you may only need to do 2 steps:
cd tests
./compile_upp.sh
Thank you!

@WenMeng-NOAA WenMeng-NOAA added the bug Something isn't working label Mar 8, 2021
@YaliMao-NOAA
Copy link
Contributor

@kgerheiser Hi Kyle, not sure whether you caught the message earlier. I have a copy of UPP GTG code on Venus at /gpfs/dell2/emc/modeling/noscrub/Yali.Mao/git/EMC_post_wafs.

There are two differences from the general UPP that you can clone from github.

  1. The f90 files were overwritten with different code and renamed as F90.
  2. sorc/ncep_post.fd/CMakeLists.txt was modified from f90 extension to F90 extension.

@kgerheiser
Copy link
Contributor

kgerheiser commented Mar 10, 2021

You need an ip::ip_4 here:

https://github.com/NOAA-EMC/EMC_post/blob/b78ef1058acb1822e755af9e9e43650f2c6111de/sorc/ncep_post.fd/CMakeLists.txt#L202

The library no longer links to ip because of the OpenMP dependency, but now it needs to compile with it.

@YaliMao-NOAA
Copy link
Contributor

@kgerheiser Great! It works now. Thank you for your help.

@WenMeng-NOAA
Copy link
Collaborator

@kgerheiser

You need an ip::ip_4 here:

With adding the change you suggest, I tested upp lib building with stub gtg code and didn't see -qopenmp in the compiling log. I would assume this change would not have impact on the upp lib building with openmp disabled, right?

@kgerheiser
Copy link
Contributor

I expected OpenMP to propagate to UPP if you link to ip::ip_4, but maybe not. Let me look into it a little.

@YaliMao-NOAA
Copy link
Contributor

@kgerheiser Both Wen and I ran into runtime error on Hera, while Wen got runtime error on Dell but I didn't. So I debugged on Hera, found a temporary fix but I really doubt the runtime error coming from cmake compiling, which doesn't handle optional arguments well.

Error messages as shown in /scratch2/NCEPDEV/stmp3/Yali.Mao/run_gfs_post.oe18041469
upp.x 0000000000A82B9A gtg_algo_ 179 gtg_algo.F90
upp.x 000000000098C283 mdlfld_ 3745 MDLFLD.f
upp.x 000000000070255D process_ 82 PROCESS.f
upp.x 000000000065E6AD MAIN__ 931 WRFPOST.f
...

  1. How to duplicate the error messages on Hera:
    git clone [email protected]:NOAA-EMC/EMC_post.git
    go to the EMC_post folder you just cloned
    cp /scratch2/NCEPDEV/ovp/Yali.Mao/git/UPP_GTG/*90 sorc/ncep_post.fd/.
    git submodule update --init CMakeModules
    cd tests
    ./compile_upp.sh

I have a jobcard and you can just modify it for your test run:
/scratch2/NCEPDEV/ovp/Yali.Mao/git/EMC_post_fork/driver/run_gtg_post.sh.hera.cmake

  1. How to temporarily fix the runtime error:
    go to the EMC_post folder you just cloned
    cp /scratch2/NCEPDEV/ovp/Yali.Mao/git/UPP_GTG_fork/gtg_algo.F90 sorc/ncep_post.fd/.
    cd tests
    ./compile_upp.sh

Please re-run the jobcard again.
My sample of a successful test is /scratch2/NCEPDEV/stmp3/Yali.Mao/run_gfs_post.success.oe18038495

  1. How does the temporary fix work? This is the difference between the modified version and the original version:
    diff /scratch2/NCEPDEV/ovp/Yali.Mao/git/UPP_GTG_fork/gtg_algo.F90 /scratch2/NCEPDEV/ovp/Yali.Mao/git/UPP_GTG/gtg_algo.F90
    As you can see, it just removes 'present' conditions of the optional argument, and adds default initialization of the arrays, despite whether the arguments are present or not. So I really doubt the cmake doesn't compile the optional arguments well.
    < if(MODELNAME(1:3)=='GFS' .or. MODELNAME(1:4)=='FV3R') then
    < cvalidtime = ""
    < tkem = SPVAL
    < edrm = SPVAL
    < LMOm = SPVAL
    < hvarm = SPVAL
    < msfxm = -1.
    < msfym = -1.
    < endif

! take care of optional arguments first
if(present(validtime)) then
cvalidtime = validtime
else
cvalidtime = ""
end if
if(present(tke)) then
tkem = tke
else
tkem = SPVAL
end if
if(present(edr)) then
edrm = edr
else
edrm = SPVAL
end if
if(present(LMO)) then
LMOm = LMO
else
LMOm = SPVAL
end if
if(present(hvar)) then
hvarm = hvar
else
hvarm = SPVAL
end if
if(present(msfx)) then
msfxm = msfx
else
msfxm = -1.
end if
if(present(msfy)) then
msfym = msfy
else
msfym = -1.
end if

YaliMao-NOAA added a commit to YaliMao-NOAA/UPP that referenced this issue Apr 26, 2021
WenMeng-NOAA pushed a commit that referenced this issue Apr 26, 2021
* To make 'cmake' go through for GTG,
1. GTG source code file extensions are changed from f90 to F90
2. Add ip::ip_4 back to sorc/ncep_post.fd/CMakeLists.txt

* Made a new tag of NCAR GTG to change file extension from f90 to F90,
then updated Externals.cfg with this new tag

* Update GTG tag to fix two issues:
  1. Fix buzillar #1207 http://www2.spa.ncep.noaa.gov/bugzilla/show_bug.cgi?id=1207
  2. EMC_post issue #272
EricJames-NOAA pushed a commit to EricJames-NOAA/UPP that referenced this issue Dec 14, 2022
WenMeng-NOAA pushed a commit that referenced this issue Feb 10, 2023
* To make 'cmake' go through for GTG,
1. GTG source code file extensions are changed from f90 to F90
2. Add ip::ip_4 back to sorc/ncep_post.fd/CMakeLists.txt

* Made a new tag of NCAR GTG to change file extension from f90 to F90,
then updated Externals.cfg with this new tag

* Update GTG tag to fix two issues:
  1. Fix buzillar #1207 http://www2.spa.ncep.noaa.gov/bugzilla/show_bug.cgi?id=1207
  2. EMC_post issue #272

* Pick up GTG new tag fixing the failure because of strong wind

* Add debug mode option to tests/compile_upp.sh

---------

Co-authored-by: wx20yam <[email protected]>
Co-authored-by: yali mao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants