Cannot find -lhdf5_hl, -lhdf5 when building ufs-s2s-model prototype5.0 on stampede2

[benjamin-cash]

I installed hpc-stack on stampede2 and attempted to use it in place of NCEPLIBS-develop in my successful build of prototype5.0. It failed with the following:

mpiifort -o /work/02441/bcash/stampede2/s2s_p5_port/NEMS/exe/NEMS.x MAIN_NEMS.o module_NEMS_UTILS.o module_MEDIATOR_methods.o module_MEDIATOR.o module_MEDIATOR_SpaceWeather.o module_EARTH_INTERNAL_STATE.o module_EARTH_GRID_COMP.o module_NEMS_INTERNAL_STATE.o module_NEMS_GRID_COMP.o module_NEMS_Rusage.o nems_c_rusage.o /work/02441/bcash/stampede2/s2s_p5_port/CMEPS_INSTALL/libcmeps.a /work/02441/bcash/stampede2/s2s_p5_port/CMEPS_INSTALL/libcmeps_util.a /work/02441/bcash/stampede2/s2s_p5_port/CMEPS_INSTALL/libpiof.a /work/02441/bcash/stampede2/s2s_p5_port/CMEPS_INSTALL/libpioc.a /work/02441/bcash/stampede2/s2s_p5_port/WW3/model/obj_HYB/libww3_multi_esmf.a /work/02441/bcash/stampede2/s2s_p5_port/CICE-interface/CICE_INSTALL/libcice6.a /work/02441/bcash/stampede2/s2s_p5_port/MOM6-interface/MOM6_INSTALL/libmom.a /work/02441/bcash/stampede2/s2s_p5_port/MOM6-interface/MOM6_INSTALL/lib_ocean.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libfv3cap.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libccppdriver.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libfv3core.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libfv3io.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libipd.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libgfsphys.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libfv3cpl.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libstochastic_physics_wrapper.a /work/02441/bcash/stampede2/s2s_p5_port/FV3/FV3_INSTALL/libstochastic_physics.a /work/02441/bcash/stampede2/s2s_p5_port/FMS/FMS_INSTALL/libfms.a -L/work/02441/bcash/stampede2/s2s_p5_port/FV3/ccpp/lib -lccpp -lccppphys ENS_Cpl/ENS_Cpl.a /work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/nemsio/2.5.2/lib/libnemsio.a /work/02441/bcash/stampede2/intel-18.0.2/bacio/2.4.1/lib/libbacio_4.a /work/02441/bcash/stampede2/intel-18.0.2/sp/2.3.3/lib/libsp_d.a /work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/w3emc/2.7.3/lib/libw3emc_d.a /work/02441/bcash/stampede2/intel-18.0.2/w3nco/2.4.1/lib/libw3nco_d.a -L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/esmf/8_1_0_beta_snapshot_27/lib -Wl,-rpath,/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/esmf/8_1_0_beta_snapshot_27/lib -lesmf -cxxlib -lrt -ldl -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz -ldl -lm -qopenmp -L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib -lnetcdff -lnetcdf -L -lpnetcdf
/opt/apps/gcc/6.3.0/bin/ld: cannot find -lhdf5_hl
/opt/apps/gcc/6.3.0/bin/ld: cannot find -lhdf5
gmake[1]: *** [nems] Error 1

[kgerheiser]

You need to update your FindNetCDF.

See Dusan's PR to make ufs-weather-model work (and his modifications to FindNetCDF.cmake): https://github.com/ufs-community/ufs-weather-model/pull/193/files

[aerorahul]

@kgerheiser The S2S model does not use CMake, so the PR from @DusanJovic-NOAA is of no help here.
The bug is in how the S2S model GNUMake sets the NetCDF include and lib paths.
I believe @binli2337 has a fix/solution.

[kgerheiser]

Oh, ok. I didn't look super closely and thought it was the same problem I had yesterday.

[aerorahul]

@benjamin-cash
I don't know how the S2S model builds on Stampede. Does it use the file conf/configure.fv3.cheyenne.intel?
If so, replace lines 78-84 below:

INCLUDE = -I$(NETCDF_ROOT)/include
NETCDF_INC = -I$(NETCDF_ROOT)/include
ifneq ($(findstring netcdf/4,$(LOADEDMODULES)),)
  NETCDF_LIB += -L$(NETCDF)/lib -lnetcdff -lnetcdf
else
  NETCDF_LIB = -L$(NETCDF)/lib -lnetcdff -lnetcdf
endif

with

INCLUDE = -I$(nc-config --includedir)
NETCDF_INC = $(INCLUDE)
NETCDF_LIB = $(nc-config --libs)
NETCDF_LIB += $(nc-config --flibs)

[benjamin-cash]

@aerorahul It uses a file based on conf/configure.fv3.hera.intel. I found those lines and made the changes, testing now.

[benjamin-cash]

That failed with a bunch of cannot open include file 'netcdf.inc' errors in ../drifters/drifters_io.F90. It also looks like ./comp_ice.backend.libcice has my old esmf.mk path, so probably that needs to be changed as well?

[benjamin-cash]

There are also errors like this:
mpiicc -Duse_libMPI -Duse_netCDF -DSPMD -DUSE_LOG_DIAG_FIELD_INFO -DUSE_GFSL63 -DGFS_PHYS -Duse_WRTCOMP -DNEW_TAUCTMAX -DINTERNAL_FILE_NML -DNO_INLINE_POST -DMOIST_CAPPA -DUSE_COND -DOPENMP -DCCPP -I -xCORE-AVX2 -qno-opt-dynamic-align -D__IFC -sox -fp-model source -O2 -debug minimal -qopenmp -I/work/02441/bcash/stampede2/s2s_p5_port/FV3/ccpp/include -c ../mpp/nsclock.c -o ../mpp/nsclock.o
../mosaic/read_mosaic.c(27): catastrophic error: cannot open source file "netcdf.h"
#include <netcdf.h>
^
compilation aborted for ../mosaic/read_mosaic.c (code 4)

mpiifort -Duse_libMPI -Duse_netCDF -DSPMD -DUSE_LOG_DIAG_FIELD_INFO -DUSE_GFSL63 -DGFS_PHYS -Duse_WRTCOMP -DNEW_TAUCTMAX -DINTERNAL_FILE_NML -DNO_INLINE_POST -DMOIST_CAPPA -DUSE_COND -DOPENMP -DCCPP -fpp -Wp,-w -I -I -fno-alias -auto -safe-cray-ptr -save-temps -ftz -assume byterecl -nowarn -sox -align array64byte -i4 -real-size 64 -no-prec-div -no-prec-sqrt -xCORE-AVX2 -qno-opt-dynamic-align -O2 -debug minimal -fp-model source -qoverride-limits -qopt-prefetch=3 -qopenmp -I/work/02441/bcash/stampede2/s2s_p5_port/FV3/ccpp/include -I../include -I../mpp/include -I../fms -c ../block_control/block_control.F90 -o ../block_control/block_control.o
mpp_io.F90(371): #error: can't find include file: netcdf.inc

[aerorahul]

seems like include paths are missing.
Can you post the output of:

nc-config --includedir
nc-config --libs
nc-config --flibs

here?

[benjamin-cash]

login4.stampede2(987)$ nc-config --includedir
/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/include
login4.stampede2(988)$ nc-config --libs
-L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib -lnetcdf
login4.stampede2(989)$ nc-config --flibs
-L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib -lnetcdff

[aerorahul]

Not entirely what I was expecting.
And nc-config --all

[benjamin-cash]

Looking back through the build log, it looks like at some point the new modules got purged and replaced with the older setup. I'm going to break for dinner and then I will keep looking to see where that might have happened.

[aerorahul]

Likewise. Happy to help you out tomorrow. Initiate a call on slack.

[benjamin-cash]

login4.stampede2(1009)$ nc-config --all

This netCDF 4.7.4 has been built with the following features:

--cc -> mpiicc
--cflags -> -I/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/include -I/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/hdf5/1.10.6/include
--libs -> -L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib -lnetcdf
--static -> -lm

--has-c++ -> no
--cxx ->

--has-c++4 -> yes
--cxx4 -> mpiicpc
--cxx4flags -> -I/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/include -I/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/hdf5/1.10.6/include
--cxx4libs -> -L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib -lnetcdf_c++4 -lnetcdf

--has-fortran -> yes
--fc -> mpiifort
--fflags -> -I/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/include -I/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/include
--flibs -> -L/work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib -lnetcdff
--has-f90 ->
--has-f03 -> yes

--has-dap -> no
--has-dap2 -> no
--has-dap4 -> no
--has-nc2 -> yes
--has-nc4 -> yes
--has-hdf5 -> yes
--has-hdf4 -> no
--has-logging -> no
--has-pnetcdf -> no
--has-szlib -> no
--has-cdf5 -> yes
--has-parallel4 -> yes
--has-parallel -> yes

--prefix -> /work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4
--includedir -> /work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/include
--libdir -> /work/02441/bcash/stampede2/intel-18.0.2/impi-18.0.2/netcdf/4.7.4/lib
--version -> netCDF 4.7.4

[aerorahul]

From the looks of it, it appears to be a share library.

Could I ask how you built the stack?
What config.sh and what stack.yaml did you use?

[MinsukJi-NOAA]

@aerorahul, are -lhdf5 and -lnetcdf needed separately? Or does it depend on the version of netcdf?

[aerorahul]

NETCDF builds with HDF5.
It is needed.

[aerorahul]

I am also realizing that these flags are probably not set properly in the ESMF linker.
I am certain this is not a stack issue, but rather a S2S model issue.

[MinsukJi-NOAA]

Do you know if ESMF_F90LINKPATHS is defined from hpc-stack?

[aerorahul]

The modulefile for ESMF defines ESMFMKFILE.
It does not explicitly define any specific PATHS.

[arunchawla-NOAA]

Looks like we need to update the GNU build system to work with hpc-stack @binli2337 did you not have branch that works ? Maybe we need a PR back to the S2S model repo

[MinsukJi-NOAA]

I tried adding HDF5_LIBRARIES: NETCDF_LIB += -L$(NETCDF)/lib -lnetcdff -lnetcdf -L$(HDF5_LIBRARIES) in configure.*, but it instead gave me g2 errors:

8970 /work/07738/kgerheis/stampede2/hpc-stack/v1.0.0-beta1/intel-18.0.2/g2/3.4.1/lib/libg2_4.a(enc_jpeg2000.c.o): In function `enc_jpeg2000_':
8971 /work/07738/kgerheis/stampede2/hpc-stack/src/hpc-stack/pkg/g2-v3.4.1/src/enc_jpeg2000.c:167: undefined reference to `jas_stream_memopen'
8972 /work/07738/kgerheis/stampede2/hpc-stack/src/hpc-stack/pkg/g2-v3.4.1/src/enc_jpeg2000.c:174: undefined reference to `jas_stream_memopen'
8973 /work/07738/kgerheis/stampede2/hpc-stack/src/hpc-stack/pkg/g2-v3.4.1/src/enc_jpeg2000.c:179: undefined reference to `jpc_encode'
8974 /work/07738/kgerheis/stampede2/hpc-stack/src/hpc-stack/pkg/g2-v3.4.1/src/enc_jpeg2000.c:188: undefined reference to `jas_stream_close'
8975 /work/07738/kgerheis/stampede2/hpc-stack/src/hpc-stack/pkg/g2-v3.4.1/src/enc_jpeg2000.c:189: undefined reference to `jas_stream_close'

[aerorahul]

@MinsukJi-NOAA
Look at these differences from the branch hpc3 from @binli2337.

It doesn't address these g2 errors though.

Looking at the configure.fv3.<platform> files, I suspect more variables used in the S2S GNUMake may be missing.
Can you check if your NCEPLIBS variable is setting the JASPER_LIB and Z_LIB paths correctly. The modulefiles don't set the variables JASPER_LIB or Z_LIB. You will most likely need to construct that based on JASPER_ROOT and ZLIB_ROOT.
Other hidden/missing variables may include PNG_LIB.

[MinsukJi-NOAA]

@aerorahul @binli2337, modifying EXTLIBS = $(NCEPLIBS) $(ESMF_F90LINKPATHS) $(ESMF_LIB) $(LDFLAGS) $(NETCDF_LIB) in cofigure.fv3.* did not help.

[MinsukJi-NOAA]

@aerorahul, I do see undefined JASPER_LIB being used in NCEPLIBS. I will try $JASPER_ROOT/lib64/libjasper.a. Regarding, -lhdf5 and -lhdf5_hl, it's still not clear how it can be fixed.

[MinsukJi-NOAA]

@aerorahul, I do see undefined JASPER_LIB being used in NCEPLIBS. I will try $JASPER_ROOT/lib64/libjasper.a. Regarding, -lhdf5 and -lhdf5_hl, it's still not clear how it can be fixed.

This is how it's currently done.
NCEPLIBS = $(POST_LIB) $(NEMSIO_LIB) $(G2_LIB4) $(G2TMPL_LIB) $(BACIO_LIB4) $(SP_LIBd) $(W3EMC_LIBd) $(W3NCO_LIBd) $(CRTM_LIB) $(JASPER_LIB) -lpng -lz

[aerorahul]

@aerorahul, I do see undefined JASPER_LIB being used in NCEPLIBS. I will try $JASPER_ROOT/lib64/libjasper.a. Regarding, -lhdf5 and -lhdf5_hl, it's still not clear how it can be fixed.

This is how it's currently done.
NCEPLIBS = $(POST_LIB) $(NEMSIO_LIB) $(G2_LIB4) $(G2TMPL_LIB) $(BACIO_LIB4) $(SP_LIBd) $(W3EMC_LIBd) $(W3NCO_LIBd) $(CRTM_LIB) $(JASPER_LIB) -lpng -lz

The -lpng -lz need paths to PNG_ROOT/lib64 and ZLIB_ROOT/lib if they are not in system locations.

What does the variable EMSF_F90LINKPATHS report?

[MinsukJi-NOAA]

The -lpng -lz need paths to PNG_ROOT/lib64 and ZLIB_ROOT/lib if they are not in system locations.

What does the variable EMSF_F90LINKPATHS report?

I am trying to figure out where EMSF_F90LINKPATHS is set.

[aerorahul]

@MinsukJi-NOAA Look at one of the older modulefiles for ESMF. It is possible that modulefile is setting that variable.
That information is in ESMFMKFILE.
Several places "include" this file, but I am not sure how the variables are set.

[aerorahul]

These variables are set in a file ESMFMKFILE=esmf.mk

[aerorahul]

@benjamin-cash Is this issue still relevant?

[benjamin-cash]

No, I have just borrowed the functioning setup Minsuk has on stampede so it isn't holding me up. I don't know if/how the issue hpc-stack not working with s2s was resolved, but I can go ahead and close this specific issue.

[MinsukJi-NOAA]

No, I have just borrowed the functioning setup Minsuk has on stampede so it isn't holding me up. I don't know if/how the issue hpc-stack not working with s2s was resolved, but I can go ahead and close this specific issue.

This issue was solved by using the approach taken here: ufs-community/ufs-s2s-model#191
Files related to porting to stampede 2 will be checked into the S2S repo: ufs-community/ufs-s2s-model#213