Compiler instructions (aka the include.mk files)

Compiler instruction file

These files are located in directory ED2/ED/build/make. In case you are part of a group that already uses ED2, the correct configuration file may be already included in the ED2 distribution:

• include.mk.bnl. Brookhaven National Labs.
• include.mk.bu_geo. Boston University.
• include.mk.cannon. Harvard's Cannon cluster (formerly known as Odyssey).
• include.mk.sdumont. Santos Dumont Cluster (LNCC).
• include.mk.ugent. Ghent University.

Otherwise, copy include.mk.gfortran or include.mk.intel to a unique file (e.g. include.mk.mylab). Below is a list of variables that must be set. You can add other variables for convenience (for example, create variables that define paths to HDF5 and other libraries).

Name	Description
MAKE	The location of the make command. This is system-dependent, but typically it is set to /usr/bin/make. Consider changing this only if you get messages on make command not being found.
BASE	The base path where ED2 compiling instructions are located. The default value $(ED_ROOT)/build should work for virtually all cases.
HDF5_INCS	The path where the HDF5 include files are located. If the include files are located in a standard directory (e.g., /usr/include), it is fine to leave this variable empty (i.e., HDF5_INCS=).
HDF5_LIBS	The path where to find HDF5 libraries and dependencies. A few examples: All libraries (zlib and HDF5) are located in standard directories (i.e., /usr/lib). this variable should be set as: HDF5_LIBS= -lz -lm -lhdf5 -lhdf5_fortran -lhdf5_hl HDF5 libraries are located in a non-standard directories (for example /home/ajlotka/Modules/hdf5/1.10.2_intel), but zlib is installed in a standard directory: HDF5_LIBS= -lz -lm -L/home/ajlotka/Modules/hdf5/1.10.2_intel/lib -lhdf5 -lhdf5_fortran -lhdf5_hl Both zlib and HDF5 libraries are located in a non-standard directories. For example zlib is installed in /home/ajlotka/Modules/zlib/1.2.11_intel and HDF5 is installed in /home/ajlotka/Modules/hdf5/1.10.2_intel), but zlib is installed in a standard directory: ZLIB_HOME=/home/ajlotka/Modules/zlib/1.2.11_intel HDF5_HOME=/home/ajlotka/Modules/hdf5/1.10.2_intel HDF5_LIBS= -lm -L$(ZLIB_HOME)/lib -lz -L$(HDF5_HOME)/lib -lhdf5 -lhdf5_fortran -lhdf5_hl
USE_COLLECTIVE_MPIO	If you have a version of HDF5 compiled in parallel, and you are planning to use parallel runs (i.e., a gridded, multi-polygon simulation, or a coupled ED2-BRAMS simulation), then you may benefit from collective I/O. In this case, you may set this variable to 1. For single-site runs, or in case your HDF5 file is not compiled with parallel, you can set this variable to 0.
CMACH	This informs the compiler of which system you are using. Possible options include: CMACH=PC_LINUX1. Option for most Linux systems. CMACH=PC_GFORTRAN. Alternative to above in case you are using gfortran and the option above doesn't work (typically when compiling file rsys.F90 or utils_c.c). CMACH=MAC_OS_X. In case you are using Mac OS and compiling with gfortran. Previous users indicated issues compiling ED2 on Mac systems, so you may need additional changes. Note. It is unlikely, but depending on the system and compiler options, you may need to edit rsys.F90 and utils_c.c and provide new, system-specific, settings.
FC_TYPE	The type of Fortran/C compilers being used. Options are: INTEL, if using ifort and icc compilers GNU, if using gfortran and gcc compilers PGI, if using pgf90 and pgcc compilers Note. In case you are using mpi compilers (mpif90 and mpicc), you must set FC_TYPE consistently to the compiler type used to build mpif90 and mpicc. In case you do not know, you can search for the location of mpif90 in your system (for example, in Linux or Mac you can use command which mpif90), then open the file in any text editor (mpif90 is a script) and look for variable F90BASE.
F_COMP	The Fortran compiler. Typical options are: ifort, if using Intel Fortran compilers without MPI capabilities. gfortran, if using GNU Fortran compilers without MPI capabilities. pgf90, if using PGI Fortran compilers without MPI capabilities. mpif90, if using MPI capabilities, regardless of the Fortran compiler.
C_COMP	The C compiler. Typical options are: icc, if using Intel C compilers without MPI capabilities. gcc, if using GNU C compilers without MPI capabilities. pgcc, if using PGI C compilers without MPI capabilities. mpicc, if using MPI capabilities, regardless of the C compiler. Note.Do not mix different families of Fortran and C compilers. For example, if F_COMP=ifort, then ensure that C_COMP=icc, otherwise the code may have unexpected behaviour.
LOADER	The Fortran compiler used to generate the executable. Unless otherwise instructed by system administrators, this variable should be set with the same value as F_COMP.
MOD_EXT	The extension of module files compiled with Fortran. Unless otherwise instructed by system administrators, use the default option (MOD_EXT=mod).
KIND_COMP	This sets compiler flags in case your include.mk file is derived from include.mk.gfortran or include.mk.intel. Intel-based options. A. Full debugging options. This is strongly recommended whenever you change the source code. This option will check for uninitialised variables, matrix bound checks, mismatches in argument types between subroutine/function declaration and calls, floating point exceptions, and unused variables and functions. You will need to compile the code two times. If using the script install.sh, use option -s 1 for the first compilation, -s 2 for preparing the second compilation, and -s 3 for the second compilation. E. Use this option for research simulations. This option will skip all the checks and use the highest possible optimisation level for maximum simulation speed. GNU-based options. A. Full debugging options. This is strongly recommended whenever you change the source code. This option will check for uninitialised variables, matrix bound checks, floating point exceptions, and unused variables and functions. E. Use this option for research simulations. This option will skip all the checks and use the highest possible optimisation level for maximum simulation speed. Note. Options B, C, and D are currently deprecated. They were formerly used to provide intermediate levels of debugging.
F_OPTS	The list of compilation options for Fortran. Check the sections on Fortran compilation flags for ifort and gfortran for suggestions. Note>/br>If using include.mk based on include.mk.gfortran or include.mk.intel, we recommend keeping the default options and set the type of compilation through KIND_COMP.
C_OPTS	The list of compilation options for C. Note>/br>If using include.mk based on include.mk.gfortran or include.mk.intel, we recommend keeping the default options and set the type of compilation through KIND_COMP.
LOADER_OPTS	The list of compilation options for the step that generates the executable file. This is almost always the same options as F_OPTS, except that LOADER_OPTS rarely has the -static option. Note>/br>If using include.mk based on include.mk.gfortran or include.mk.intel, we recommend keeping the default options and set the type of compilation through KIND_COMP.
F_LOWO_OPTS	Optional variable, that allows compiling a few Fortran subroutines with slightly lower optimisation level. This is useful when compiling the code with ifort. Note>/br>If using include.mk based on include.mk.gfortran or include.mk.intel, we recommend keeping the default options and set the type of compilation through KIND_COMP.
PAR_DEFS	This flag defines whether or not the code will be compiled with MPI support. In case you want MPI support, set PAR_DEFS=-DRAMS_MPI, otherwise, leave it blank (PAR_DEFS=). Note. In case you want MPI support, you should also set the Fortran and C compilers to mpif90 and mpicc, respectively. Check variables F_COMP, C_COMP, and LOADER_COMP.
ARCHIVE	Command to create the ED2 library. Unless otherwise instructed by system administrators, this variable should be set to ARCHIVE=ar rs.

Useful Fortran compilation flags for Intel (ifort).

This provides a short summary of compilation options. Most of the descriptions were taken directly from the Intel compiler manual. For additional information, type man ifort in the computer you plan to run ED2.

Required/strongly recommended flags for all compilations

Flag	Description
-assume byterecl	Specifies that the units for the OPEN statement RECL specifier (record length) value are in bytes for unformatted data files, not longwords (four-byte units). This is the case in ED2.
-FR	Specifies source files are in free format (as opposed to F77 format). All code in ED2 uses free format.
-qopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel. In some ifort systems, the option may be -openmp instead of -qopenmp.
-recursive	Ensures that subroutines can call themselves recursively (ED2 contains a few recursive subroutines).
-static	This prevents linking with shared libraries. Use this in F_OPTS only (the ED2 library compilation), not LOADER_OPTS (the ED2 executable compilation).
-traceback	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.

Debugging flags (KIND_COMP=A)

Flag	Description
-check all,noarg_temp_created	Enables all check options, with the exception of whether arguments were copied to temporary storage. This option will check the following: Bounds. Checks on whether array sub-script and substring references are within declared bounds. Format. This will ensure data formatted for output is consistent with variable type, and if output variable is too large to fit in the designated format description field. Uninitialised variables. Checks that the code does not attempt to use variables that have not been initialized (i.e., assigned a valid initial value). Also, see -ftrapuv
-debug extended.	This will produce enhanced debug information useful for breakpoints and stepping, and information useful in finding scalar local variables.
-debug-parameters all	Generates debug information for all parameters defined in the program.
-fpe0	This will issue errors whenever a floating point exception occurs. This includes: invalid values, division by zero, overflow, and underflow (the latter only if -no-ftz is set).
-fp-stack-check	Generates extra code after every function call to ensure that the floating-point variables are all in the expected state. In some systems, the option is -fpstkchk
-ftrapuv	This assigns non-sensical initial values to all local variables, which helps identifying variables that were not properly initialised.
-g	Produces symbolic debug information in the object file. Useful when running a debugger with the model (idb, gdb, TotalView, ddd)
-gen-interface	Tells the compiler to generate an interface block for each routine (that is, for each SUBROUTINE and FUNCTION statement) defined in the source file, even if they are not inside modules.
-implicitnone	Sets the default type of a variable to undefined (IMPLICIT NONE).
-O0	Disables all optimisations. Debugger works much better when the code optimisation is disabled
-no-ftz	Prevents the code to flush very small values to zero. This can be useful to spot incorrectly set variables.
-warn declarations	Enables warnings about any undeclared names. The compiler will use the default implicit data typing rules for such undeclared names. The IMPLICIT and IMPLICIT NONE statements override this option.
-warn errors	Tells the compiler to change all warning-level messages to error-level messages; this includes warnings about Fortran standards violations.
-warn interfaces	Tells the compiler to check the interfaces of all subroutines called and functions invoked in your compilation against a set of interface blocks stored separately from the source being compiled.
-warn uncalled	Enables warnings when a statement function is never called.
-warn unused	Enables warnings about variables that are declared but never used.
-warn usage	Enables warnings about questionable programming practices.

Performance flags (KIND_COMP=E)

Flag	Description
-O2	This option is the default for optimisations. The main distribution of ED2 is routinely tested for bugs that would prevent using the aggressive optimisation; therefore, you can take advantage of option -O3 (see below) for most sub-routines.
-O3	Enables -O2 optimizations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance. Note. In case the compilation is taking extraordinarily long (several hours), you may want to set an additional variable F_LOWO_OPTS in your include.mk file and use -O2 instead of -O3 (plus the other options used in F_OPTS). This will compile the files known to take notoriously long to compile with -O3 at lower optimisation level, whilst compiling most subroutines with the most aggressive optimisation.

Useful Fortran compilation flags for GNU (gfortran).

This provides a short summary of compilation options. Most of the descriptions were taken directly from the GNU compiler manual. For additional information, type man gfortran in the computer you plan to run ED2.

Required/strongly recommended flags for all compilations

Flag	Description
-fbacktrace	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.
-ffree-line-length-none	Specifies source files are in free format (as opposed to F77 format), with no line length limit. All code in ED2 uses free format.
-fopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel.
-frecursive	Ensures that subroutines can call themselves recursively (ED2 contains a few recursive subroutines).
-static	This prevents linking with shared libraries. Use this in F_OPTS only (the ED2 library compilation), not LOADER_OPTS (the ED2 executable compilation).

Debugging flags (KIND_COMP=A)

Flag	Description
-fcheck=all,no-array-temps	Enables all check options, with the exception of whether arguments were copied to temporary storage. This option will check the following: Bounds. Checks on whether array sub-script and substring references are within declared bounds. Do. Checks for invalid modifications of loop iteration variables Memory. Checks for valid memory allocation during run-time, including pointers and allocatable variables.
-ffpe-trap=invalid,zero,overflow,underflow	This will issue errors whenever a floating point exception occurs. This includes: invalid values, division by zero, overflow, and underflow.
-fimplicit-none	Sets the default type of a variable to undefined (IMPLICIT NONE).
-finit-integer=-2147483648	This assigns nearly NaN initial values to all integer variables, which helps identifying variables that were not properly initialised.
-finit-real=snan	This assigns NaN initial values to all real variables, which helps identifying variables that were not properly initialised.
-Wall	Enables commonly used warning options pertaining to questionable programming practices that we believe are easy to avoid. This currently includes (but is not limited to): Missing ampersand in continued character constants Type, rank, and other mismatches between formal parameters and actual arguments to functions and subroutines. Character assignment that would truncate the assigned string. Implicit conversions that are likely to change the value of the expression after conversion, and implicit conversions between different types and kinds Procedures that may be called without an explicit interface. Constant integer division that would truncate its result (e.g., 3/5 evaluates to 0). "Suspicious" code constructs, which are technically legal but usually indicate coding error. DO loop with step either 1 or -1 that would yield an underflow or an overflow during iteration of an induction variable of the loop. User-defined procedure or module procedures that have the same name as an intrinsic. Unused arguments, parameters, or variables.
-Werror	Turns all warnings into errors.

Performance flags (KIND_COMP=E)

Flag	Description
-O2	This option is the default for optimisations. The main distribution of ED2 is routinely tested for bugs that would prevent using the aggressive optimisation; therefore, you can take advantage of option -O3 (see below) for most sub-routines.
-O3	Enables -O2 optimizations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance.

Useful C compilation flags for Intel (icc).

Flag	Description
-DLITTLE	Not a native icc option, but this must be provided to compilation instructions, to ensure the code assumes little endian variables.
-g	Produces symbolic debug information in the object file. Useful only if compiling the code for debugging, especially if you plan to use debuggers (idb, gdb, TotalView, ddd).
-O0 or -O3	Optimisation level: the recommended option depends on the intent. -O0. Disables all optimisations. Debugger works much better when the code optimisation is disabled. -O3. Enables -O2 optimizations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance. Recommended option for science and high-performance simulations.
-qopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel. In some ifort systems, the option may be -openmp instead of -qopenmp.
-static	This prevents linking with shared libraries.
-traceback	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.

Useful C compilation flags for GNU (gcc).

Flag	Description
-DLITTLE	Not a native gcc option, but this must be provided to compilation instructions, to ensure the code assumes little endian variables.
-g	Produces symbolic debug information in the object file. Useful only if compiling the code for debugging, especially if you plan to use debuggers (idb, gdb, TotalView, ddd).
-fopenmp	Generates multithreaded code based on OpenMP directives. This is very useful to speed up simulations as it allows patches to be solved in parallel. In some ifort systems, the option may be -openmp instead of -qopenmp.
-mtraceback=full	Displays the source file traceback information (e.g., line numbers and the tree of calls) at run time when a severe error occurs.
-O0 or -O3	Optimisation level: the recommended option depends on the intent. -O0. Disables all optimisations. Debugger works much better when the code optimisation is disabled. -O3. Enables -O2 optimizations plus more aggressive optimisations, such as prefetching, scalar replacement, and loop transformations. Enables optimisations for maximum speed, but does not guarantee higher performance. Recommended option for science and high-performance simulations.
-static	This prevents linking with shared libraries.

A note about compilers

It is possible that your system has multiple compilers available. This list has a few points to consider when choosing which compiler to use based on previous experience. Please mind that things can change over time and some of these remarks may need to be revisited.

• Speed From former comparisons at the Harvard Cluster, the model takes considerably longer to compile when using ifort and icc, but the simulation runtime is generally faster compared to gfortran/gcc.
• Debugging None of the compiler settings catch all the bugs or warnings. When developing the code, it is advisable to compile the code with both ifort/icc and gfortran/gcc using the debugging flags.
  • One feature that is unique to ifort is the possibility of generating interfaces. This capability is very useful when the code development involves changing the list of arguments in functions and subroutines, especially when these functions and subroutine are not contained inside a module.
• Portability The GNU compilers (gfortran/gcc) are free, and therefore it is a good practice to ensure that the model can compile and run using these compilers, even when your preferred compilers are ifort/icc. Whenever a pull request is submitted, GitHub will try to compile the pull request with gfortran, so it is advisable to run tests in gfortran before submitting the pull request.