Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mlevy/mapping on geyser #2589

Merged
merged 7 commits into from
May 14, 2018
Merged

Mlevy/mapping on geyser #2589

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
abd31ab
Mapping tools clean-up
mnlevy1981 May 11, 2018
b1e484e
Clean up mapping documentation
mnlevy1981 May 11, 2018
c15b19a
Use CIME configure to get proper modules
mnlevy1981 May 14, 2018
f432dda
PR #2591 allows a little more code clean-up
mnlevy1981 May 14, 2018
609221e
Better README files for mapping tools
mnlevy1981 May 14, 2018
2d4c27a
Parallel map generation on cheyenne
mnlevy1981 May 14, 2018
ee287e8
Better error message on cheyenne login node
mnlevy1981 May 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions tools/mapping/check_maps/README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@ BUILD
=====

To compile this tool, you need to have the ESMFMKFILE environment variable set.
The easiest way to do that is to load the ESMF modules. On yellowstone and
cheyenne, you must already have the intel module loaded. Then load the version
of ESMF you want to use (as of April 2017, the most recent version available is
7.0.0):
The easiest way to do that is to load the ESMF modules. On cheyenne, caldera,
geyser, or pronghorn you must already have the intel module loaded. Then load the
version of ESMF you want to use (as of May 2018, the most recent version on the
NCAR machines is 7.0.0):

Yellowstone:
Caldera, Geyser, Pronghorn:
$ module load esmf
$ module load esmf-7.0.0-ncdfio-uni-O

Expand Down Expand Up @@ -78,6 +78,6 @@ listed in FILELIST
--help, -h Output this usage information

Notes:
1) For use on cheyenne, yellowstone, geyser, or caldera only!
1) For use on cheyenne, geyser, caldera, or pronghorn only!
2) Need to set ESMFMKFILE (see comments in Makefile) or compilation will fail
3) If -rc option is not enabled, -v flag is ignored and verbose / concise will depend on previous compilation
2 changes: 1 addition & 1 deletion tools/mapping/check_maps/check_map.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ usage() {
echo ' --help, -h Output this usage information'
echo ''
echo 'Notes:'
echo ' 1) For use on yellowstone, geyser, or caldera only!'
echo ' 1) For use on cheyenne, caldera, geyser, or pronghorn only!'
echo ' 2) Need to set ESMFMKFILE (see comments in Makefile)'\
'or compilation will fail'
echo ' 3) If -rc option is not enabled, -v flag is ignored and verbose /'\
Expand Down
8 changes: 4 additions & 4 deletions tools/mapping/check_maps/src/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@

################################################################################
### Finding and including esmf.mk ######
### NOTE: On Yellowstone, ESMF is only available with the intel compilers ######
### NOTE: On Cheyenne, ESMF is only available with the intel compilers ######
### And require running the following: ######
### $ module load esmf ######
### $ module load esmf-6.3.0r-ncdfio-uni-O ######
### The executable should run on the yellowstone login node ######
### $ module load esmf_libs/7.0.0 ######
### $ module load esmf-7.0.0-ncdfio-uni-O ######
### The executable should run on the cheyenne login node ######
################################################################################

# Don't require ESMF to be loaded to run "make clean":
Expand Down
1 change: 1 addition & 0 deletions tools/mapping/gen_domain_files/src/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ Depends*
Macros.make
*.o
*.mod
*.optrpt
2 changes: 1 addition & 1 deletion tools/mapping/gen_domain_files/src/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ $(EXENAME): $(OBJS)
$(LINKER) -o $@ $(OBJS) $(SLIBS) $(LDFLAGS)

clean:
$(RM) -f $(OBJS) $(EXENAME) *.mod
$(RM) -f $(OBJS) $(EXENAME) *.mod *.optrpt

distclean: clean
../../../configure --clean
Expand Down
44 changes: 9 additions & 35 deletions tools/mapping/gen_mapping_files/README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,12 @@ gen_cesm_maps.sh
--fileocn|-focn input ocn_grid_filename
--filelnd|-flnd input lnd_grid_filename
--filertm|-frtm input rtm_grid_filename
--fileglc|-fglc input glc_grid_filename
--nameocn|-nocn output ocn_name in mapping file
--nameatm|-natm output atm_name in mapping file
--namelnd|-nlnd output lnd_name in mapping file
--namertm|-nrtm output rtm_name in mapping file
--nameglc|-nglc output glc_name in mapping file
[ --typeocn|tocn ] [regional|global]
[ --typeatm|tatm ] [regional|global]
[ --nogridcheck ]
Expand All @@ -84,6 +86,8 @@ where
SCRIP grid format land filename (full pathname), must be global
--filertm (or -frtm)
SCRIP grid format runoff filename (full pathname)
--fileglc (or -fglc)
SCRIP grid format glc filename (full pathname), assumed to be regional
--nameatm (or -natm)
Shortname to use for atm in mapping filename
--nameocn (or -nocn)
Expand All @@ -92,6 +96,8 @@ where
Shortname to use for lnd in mapping filename
--namertm (or -nrtm)
Shortname to use for rtm in mapping filename
--nameglc (or -nglc)
Shortname to use for glc in mapping filename
--typeocn (or -tocn)
ocean grid type, valid values are regional or global
default is global
Expand All @@ -115,44 +121,12 @@ where
--help or -h
displays this help message

Note: if rtm is specified and lnd is not, then this tool will
Note: if rtm or glc are specified and lnd is not, then this tool will
assume lnd and atm are on the same grid.

You can also set the following env variables:
ESMFBIN_PATH - Path to ESMF binaries
(Leave unset on cheyenne, yellowstone, and caldera and the
tool will be loaded from modules)
(Known machines will load tools from modules)
MPIEXEC ------ Name of mpirun executable
(default is mpirun.lsf on yellowstone and caldera; if
you run interactively on yellowstone, mpi is not used)
REGRID_PROC -- Number of MPI processors to use
(default is 8)

=================
USAGE: BATCH MODE
=================

Currently, batch mode is only set up for yellowstone. This uses the file
regridbatch.yellowstone.sh as a wrapper to gen_cesm_maps.sh.

IMPORTANT NOTE: check_maps is NOT run when you run in batch mode. This must be
run as a separate step after you have created the mapping files. See below for
an example of how to do this.

(1) Modify the #BSUB lines at the top of regridbatch.yellowstone.sh. You need to
at least specify a project number; you may want to change some of the other
settings, too.

(2) Modify the variables in the section labeled "Set user-defined parameters
here". See the documentation for gen_cesm_maps.sh above for more details on
the parameters that can be set here.

(3) Submit with:

bsub < regridbatch.yellowstone.sh

(4) Run the check_maps tool on the generated mapping files; e.g.:

files=*.nc
../check_maps/check_map.sh $files
(currently tools only run in serial due to module issues)

44 changes: 8 additions & 36 deletions tools/mapping/gen_mapping_files/gen_ESMF_mapping_file/README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,11 @@ where
--clm_name
Use the CLM naming convention
--serial
For yellowstone batch jobs only! Load the serial ESMF tools rather
than the parallel tools (necessary for mapping grids with a single
point).
Load the serial ESMF tools rather than the parallel tools
(necessary for mapping grids with a single point).
--machine (or -mach)
Name of the machine you are running on. Currently supports yellowstone,
geyser, caldera, pronghorn, and jaguar. Note that this script will
Name of the machine you are running on. Currently supports cheyenne,
geyser, caldera, and pronghorn. Note that this script will
determine the machine name automatically from the hostfile command.
-d
toggle debug-only
Expand All @@ -82,36 +81,9 @@ where

You can also set the following env variables:
ESMFBIN_PATH - Path to ESMF binaries
(Leave unset on cheyenne, yellowstone, and caldera and the
tool will be loaded from modules)
(Leave unset on cheyenne/caldera/geyser/pronghorn
and the tool will be loaded from modules)
MPIEXEC ------ Name of mpirun executable
(default is mpirun.lsf on yellowstone and caldera; if
you run interactively on yellowstone, mpi is not used)
REGRID_PROC -- Number of MPI processors to use (jaguar only!)
(default is 8)

=================
USAGE: BATCH MODE
=================

Currently, batch mode is only set up for yellowstone. This uses the file
regridbatch.yellowstone.sh as a wrapper to create_ESMF_map.sh. Before submitting,
make sure you do the following:

(1) Modify the #BSUB lines at the top of regridbatch.yellowstone.sh. You need to
at least specify a project number; you may want to change some of the other
settings, too.

(2) Modify the variables in the section labeled "Set user-defined parameters
here". See the documentation for create_ESMF_map.sh above for more details on
the parameters that can be set here.

(3) Submit with:

bsub < regridbatch.yellowstone.sh

(4) Run the check_maps tool on the generated mapping file:

out_file=*.nc
../../check_maps/check_map.sh ${out_file}
(ignored if --serial, which is default on cheyenne
login nodes

90 changes: 33 additions & 57 deletions tools/mapping/gen_mapping_files/gen_ESMF_mapping_file/create_ESMF_map.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# SVN $URL: https://svn-ccsm-models.cgd.ucar.edu/tools/mapping/trunk_tags/mapping_141106/gen_mapping_files/gen_ESMF_mapping_file/create_ESMF_map.sh $
#
# Create needed mapping files for gen_domain and coupler mapping
# Currently supported on yellowstone, geyser, caldera, pronghorn, and jaguarpf
# Currently supported on cheyenne, geyser, caldera, and pronghorn
#
#===============================================================================
echo $0
Expand All @@ -18,14 +18,14 @@ usage() {
echo ''
echo '**********************************************************'
echo 'usage:'
echo './create_ESMF_map.sh '
echo './create_ESMF_map.sh'
echo ' A wrapper for the ESMF mapping tool that creates a mapping file'
echo ' from the source grid to the destination grid. Specify what type'
echo ' of mapping to use with the -maptype flag (aave, blin, bilin,patc,'
echo ' nearestdtos, or neareststod)'
echo ''
echo 'create_ESMF_map.sh '
echo ' --filesrc|-fsrc input source grid_filename (required) '
echo 'create_ESMF_map.sh'
echo ' --filesrc|-fsrc input source grid_filename (required)'
echo ' --filedst|-fdst input destination grid_filename (required)'
echo ' --namesrc|-nsrc output source name in mapping file (required)'
echo ' --namedst|-ndst output destination name in mapping file (required)'
Expand All @@ -40,13 +40,13 @@ usage() {
echo ' [ --large_file|-big ]'
echo ' [ --help|-h ]'
echo ' [ -v ]'
echo ' '
echo 'where '
echo ''
echo 'where'
echo ' --filesrc (or -fsrc)'
echo ' SCRIP grid format source filename (full pathname)'
echo ' --filedst (or -fdst)'
echo ' SCRIP grid format destination filename (full pathname)'
echo ' --namesrc (or -nsrc) and --namesrc (or -nsrc) will result in the '
echo ' --namesrc (or -nsrc) and --namesrc (or -nsrc) will result in the'
echo ' following mapping files'
echo ' namesrc_TO_namedst_maptype.cdate.nc'
echo ''
Expand All @@ -66,27 +66,24 @@ usage() {
echo ' --clm_name'
echo ' Use the CLM naming convention'
echo ' --serial'
echo ' For yellowstone batch jobs only! Load the serial ESMF tools rather'
echo ' than the parallel tools (necessary for mapping grids with a single'
echo ' point).'
echo ' Load the serial ESMF tools rather than the parallel tools'
echo ' (necessary for mapping grids with a single point).'
echo ' --machine (or -mach)'
echo ' Name of the machine you are running on. Currently supports yellowstone,'
echo ' geyser, caldera, pronghorn, and jaguar. Note that this script will'
echo ' Name of the machine you are running on. Currently supports cheyenne,'
echo ' geyser, caldera, and pronghorn. Note that this script will'
echo ' determine the machine name automatically from the hostfile command.'
echo ' -d'
echo ' toggle debug-only '
echo ' --help or -h '
echo ' toggle debug-only'
echo ' --help or -h'
echo ' displays this help message'
echo ''
echo 'You can also set the following env variables:'
echo ' ESMFBIN_PATH - Path to ESMF binaries '
echo ' (Leave unset on cheyenne/yellowstone/caldera/pronghorn'
echo ' ESMFBIN_PATH - Path to ESMF binaries'
echo ' (Leave unset on cheyenne/caldera/geyser/pronghorn'
echo ' and the tool will be loaded from modules)'
echo ' MPIEXEC ------ Name of mpirun executable'
echo ' (default is mpirun.lsf on yellowstone/caldera/pronghorn; if'
echo ' you run interactively on yellowstone, mpi is not used)'
echo ' REGRID_PROC -- Number of MPI processors to use (jaguar only!)'
echo ' (default is 8)'
echo ' (ignored if --serial, which is default on cheyenne'
echo ' login nodes'
echo '**********************************************************'
}

Expand Down Expand Up @@ -221,10 +218,6 @@ done
if [ $MACH == "UNSET" ]; then
hostname=`hostname`
case $hostname in
## yellowstone
ys* )
MACH="yellowstone"
;;
cheyenne* )
MACH="cheyenne"
;;
Expand All @@ -237,26 +230,25 @@ if [ $MACH == "UNSET" ]; then
pronghorn* )
MACH="pronghorn"
;;
jaguarpf* )
MACH="jaguar"
;;
*)
echo "Machine $hostname NOT recognized"
;;
esac
fi

# Machine specific settings:
# 1) can not run in parallel interactively on yellowstone or cheyenne
if [ $MACH == "yellowstone" ] && [ $interactive == "YES" ]; then
serial="TRUE"
fi
# 1) can not run in parallel interactively on cheyenne
if [ $MACH == "cheyenne" ] && [ $interactive == "YES" ]; then
serial="TRUE"
fi
# 2) jaguar requires additional environment var
if [ $MACH == "jaguar" ] && [ -z "$REGRID_PROC" ]; then
REGRID_PROC=8
# 2) FIXME: can not run in parallel on NCAR machines
if [ $MACH == "cheyenne" ] || \
[ $MACH == "geyser" ] || \
[ $MACH == "caldera" ] || \
[ $MACH == "pronghorn" ] ; then
echo "Using serial implementation of ESMF because module changes"
echo "have made it difficult to run in parallel."
serial="TRUE"
fi

# check for required arguments
Expand Down Expand Up @@ -315,17 +307,15 @@ fi
#-------------------------------------------------------------------------------

case $MACH in
## yellowstone, geyser, caldera, or pronghorn
"cheyenne" | "yellowstone" | "geyser" | "caldera" | "pronghorn" )
# From tcsh, script will not find module command
# So check to see if module works, otherwise source an init file
module list > /dev/null 2>&1 || source /etc/profile.d/modules.sh
## cheyenne, geyser, caldera, or pronghorn
"cheyenne" | "geyser" | "caldera" | "pronghorn" )
module purge
module load intel
module load nco
if [ $MACH == "cheyenne" ]; then
module load intel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this tool were to invoke the cime/tools/configure script maybe it wouldn't be restricted to only run on this limited set of machines? The configure script would provide a env_machine_specific file which should load a proper esmf environment if one is defined for that system.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea, but ran into two issues with it:

  1. on cheyenne ../../../configure --mpilib mpi-serial worked fine but specifying mpt did not:
$ ../../../configure --mpilib mpt
ERROR: Multiple matches

I get the same error with --mpilib openmpi and --mpilib mpich2. Is there another mandatory argument? Adding --macros-format and --compiler doesn't change the error message. (Just a note: --clean works, so this branch has your fix for that issue)

  1. on pronghorn, I needed to remove trilinos to avoid this error:
$ ../../../configure --mpilib mpi-serial
ERROR: module command /glade/apps/opt/lmod/lmod/libexec/lmod python load ncarenv/1.0 ncarbinlibs/1.1 perlmods gmake/4.1 python all-python-libs git intel/15.0.3 mkl/11.1.2 trilinos/11.10.2 esmf esmf-7.0.0-ncdfio-uni-O netcdf/4.3.3.1 ncarcompilers/1.0 cmake/3.0.2 all-python-libs failed with message:
Lmod Error: Cannot find Trilinos package built for intel version: 15.0.3

But then it looks like changing --mpilib causes different problems:

$ ../../../configure --mpilib mpich2
ERROR: module command /glade/apps/opt/lmod/lmod/libexec/lmod python load ncarenv/1.0 ncarbinlibs/1.1 perlmods gmake/4.1 python all-python-libs git intel/15.0.3 mkl/11.1.2 esmf esmf-7.0.0-defio-mpi-O netcdf-mpi/4.3.3.1 pnetcdf/1.6.1 ncarcompilers/1.0 cmake/3.0.2 all-python-libs failed with message:
Lmod Warning: Did not find: netcdf-mpi/4.3.3.1 pnetcdf/1.6.1

Try: "module spider netcdf-mpi/4.3.3.1 pnetcdf/1.6.1"

netcdf-mpi/4.5.0 looks to be the only module built with intel, except it doesn't actually work:

$ module purge
$ module load intel/15.0.3
$ module avail netcdf-mpi

------------------------------------------------------- /glade/apps/opt/modulefiles/ca/cdep/intel -------------------------------------------------------
   netcdf-mpi/4.5.0

------------------------------------------------------- /glade/apps/opt/modulefiles/pr/cdep/intel -------------------------------------------------------
   netcdf-mpi/4.5.0

------------------------------------------------------- /glade/apps/opt/modulefiles/ys/cdep/intel -------------------------------------------------------
   netcdf-mpi/4.5.0
$ ../../../configure --mpilib mpich2
ERROR: module command /glade/apps/opt/lmod/lmod/libexec/lmod python load ncarenv/1.0 ncarbinlibs/1.1 perlmods gmake/4.1 python all-python-libs git intel/15.0.3 mkl/11.1.2 esmf esmf-7.0.0-defio-mpi-O netcdf-mpi/4.5.0 ncarcompilers/1.0 cmake/3.0.2 all-python-libs failed with message:
Lmod Error: Cannot find netcdf-mpi-4.5.0 built for intel version: 15.0.3

So if I can remove trilinos from the yellowstone entry in config_machines.xml then I think I'm good to go with using --mpilib mpi-serial... but there are more issues to work out before CIME's configure lets us use parallel.

module load esmf_libs/7.0.0
else
module load intel/12.1.5
module load esmf
fi

Expand All @@ -334,25 +324,11 @@ case $MACH in
if [ -z "$MPIEXEC" ]; then
MPIEXEC=""
fi
else
module load esmf-7.0.0-ncdfio-mpi-O
if [ -z "$MPIEXEC" ]; then
MPIEXEC="mpirun.lsf"
fi
fi

;;
##jaguarpf
## NOTE that for jaguarpf there is no batch script for now
"jaguar" )
if [ -z "$ESMFBIN_PATH" ]; then
module load esmf/5.2.0-p1_with-netcdf_g
ESMFBIN_PATH=$ESMF_BINDIR
# FIXME: get parallel tools working
# else
# module load esmf-7.0.0-ncdfio-mpi-O
fi

if [ -z "$MPIEXEC" ]; then
MPIEXEC="aprun -n $REGRID_PROC"
fi
;;
*)
echo "Machine $MACH NOT recognized"
Expand Down
Loading