Skip to content

Commit

Permalink
Update machine files for wolf and conejo
Browse files Browse the repository at this point in the history 
Adds the machine environment files for the wolf and conejo machines at
LANL.

Fixes: none, feature addition only

User interface changes?: yes, "wolf" and "conejo" machines are now
available. Changes were designed with minimal to no impact on other
machine setups.

Code review: Chonggang Xu, Ryan Knox, bandre

Test suite: ed : lawrencium_intel, conejo_intel, yellowstone_{intel,pgi,gnu}
Test baseline: lrc: a066165; conejo: none; yellowstone: 8740a1a
Test namelist changes: None
Test answer changes: None

Test summary: All test functionality and restarts pass except f09
and f19 restarts (github issue #14)
  • Loading branch information
@bandre-ucar
bandre-ucar committed Mar 4, 2016
2 parents 8740a1a + e2b2f4e commit 19fe567
Show file tree
Hide file tree
Showing 9 changed files with 322 additions and 7 deletions.
43 changes: 42 additions & 1 deletion cime/machines/config_batch.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,33 @@
<directive name=""></directive>
</directives>
</batch_system>

<batch_system type="none" version="x.y">
<batch_query args=""></batch_query>
<batch_submit></batch_submit>
<batch_redirect></batch_redirect>
<batch_directive></batch_directive>
<directives>
<directive name=""></directive>
</directives>
</batch_system>

<batch_system MACH="wolf" version="x.y">
<batch_directive>#MSUB</batch_directive>
<directives>
<directive>-l nodes={{ num_nodes }}:ppn={{ tasks_per_node }}</directive>
</directives>
</batch_system>

<batch_system MACH="conejo" version="x.y">
<batch_directive>#MSUB</batch_directive>
<directives>
<directive>-l nodes={{ num_nodes }}:ppn={{ tasks_per_node }}</directive>
</directives>
</batch_system>


<batch_system type="cobalt" version="x.y">
<batch_system type="cobalt" version="x.y">
<batch_query>qstat</batch_query>
<batch_submit>bash</batch_submit>
<batch_directive></batch_directive>
Expand All @@ -37,6 +62,22 @@
<arg flag="--mode script"/>
</submit_args>
</batch_system>

<batch_system type="moab" version="x.y">
<batch_query>showq</batch_query>
<batch_submit>msub </batch_submit>
<batch_directive>#MSUB</batch_directive>
<jobid_pattern>(\d+)$</jobid_pattern>
<depend_string> -W depend=afterok:jobid</depend_string>
<directives>
<directive> -N {{ job_id }}</directive>
<directive> -l walltime={{ wall_time }}</directive>
<directive> -j oe </directive>
<directive default="n"> -r {{ rerunnable }} </directive>
<directive default="ae" > -m {{ mail_options }} </directive>
<directive default="/bin/bash" > -S {{ shell }}</directive>
</directives>
</batch_system>

<batch_system type="lsf" version="9.1">
<batch_query args=" -w" >bjobs</batch_query>
Expand Down
64 changes: 64 additions & 0 deletions cime/machines/config_compilers.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,70 @@ for mct, etc.

</compiler>

<compiler MACH="conejo" COMPILER="intel">
<MPICC>mpicc</MPICC>
<MPIFC>mpif90</MPIFC>
<MPICXX>mpic++</MPICXX>
<SFC>ifort</SFC>
<SCC>icc</SCC>
<SCXX>icpc</SCXX>
<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -llapack -lblas</ADD_SLIBS>
<GPTL_CPPDEFS> -DHAVE_VPRINTF -DHAVE_GETTIMEOFDAY </GPTL_CPPDEFS>
<TRILINOS_PATH>$(TRILINOS_PATH)</TRILINOS_PATH>
<ADD_SLIBS MPILIB="mpich"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpich2"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpt"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="openmpi"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mvapich"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="impi"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpi-serial"> -mkl </ADD_SLIBS>
</compiler>

<compiler MACH="conejo" COMPILER="gnu">
<MPICC>mpicc</MPICC>
<MPIFC>mpif90</MPIFC>
<MPICXX>mpic++</MPICXX>
<SFC>gfortran</SFC>
<SCC>gcc</SCC>
<SCXX>g++</SCXX>
<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -llapack -lblas</ADD_SLIBS>
<CXX_LIBS>-lstdc++ -lmpi_cxx</CXX_LIBS>
<TRILINOS_PATH>$(TRILINOS_PATH)</TRILINOS_PATH>
<ALBANY_PATH>$(ALBANY_PATH)</ALBANY_PATH>
</compiler>

<compiler MACH="wolf" COMPILER="intel">
<MPICC>mpicc</MPICC>
<MPIFC>mpif90</MPIFC>
<MPICXX>mpic++</MPICXX>
<SFC>ifort</SFC>
<SCC>icc</SCC>
<SCXX>icpc</SCXX>
<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -llapack -lblas</ADD_SLIBS>
<TRILINOS_PATH>$(TRILINOS_PATH)</TRILINOS_PATH>
<ADD_SLIBS MPILIB="mpich"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpich2"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpt"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="openmpi"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mvapich"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="impi"> -mkl=cluster </ADD_SLIBS>
<ADD_SLIBS MPILIB="mpi-serial"> -mkl </ADD_SLIBS>
</compiler>

<compiler MACH="wolf" COMPILER="gnu">
<MPICC>mpicc</MPICC>
<MPIFC>mpif90</MPIFC>
<MPICXX>mpic++</MPICXX>
<SFC>gfortran</SFC>
<SCC>gcc</SCC>
<SCXX>g++</SCXX>
<ADD_SLIBS>$(shell $(NETCDF_PATH)/bin/nf-config --flibs) -llapack -lblas</ADD_SLIBS>
<CXX_LIBS>-lstdc++ -lmpi_cxx</CXX_LIBS>
<TRILINOS_PATH>$(TRILINOS_PATH)</TRILINOS_PATH>
<ALBANY_PATH>$(ALBANY_PATH)</ALBANY_PATH>
</compiler>


<compiler COMPILER="intelmic">
<!-- http://software.intel.com/en-us/articles/intel-composer-xe/ -->
<ADD_CPPDEFS> -DFORTRANUNDERSCORE -DNO_R16</ADD_CPPDEFS>
Expand Down
104 changes: 104 additions & 0 deletions cime/machines/config_machines.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,110 @@
<MAX_TASKS_PER_NODE>USERDEFINED_required_build</MAX_TASKS_PER_NODE>
</machine>

<machine MACH="wolf">
<DESC>LANL Linux Cluster, 16 pes/node, batch system Moab</DESC>
<NODENAME_REGEX>wolf</NODENAME_REGEX>
<TESTS>acme_developer</TESTS>
<COMPILERS>intel,gnu</COMPILERS>
<MPILIBS>openmpi,mvapich,mpi-serial</MPILIBS>
<OS>LINUX</OS>
<RUNDIR>/lustre/scratch1/turquoise/$ENV{USER}/ACME/cases/$CASE/run</RUNDIR>
<EXEROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME/cases/$CASE/bld</EXEROOT>
<DIN_LOC_ROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME/input_data</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/lustre/scratch1/turquoise/$ENV{USER}/ACME/input_data/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME/archive/$CASE</DOUT_S_ROOT>
<DOUT_L_MSROOT>UNSET</DOUT_L_MSROOT>
<CCSM_BASELINE>/lustre/scratch1/turquoise/$ENV{USER}/ACME/input_data/ccsm_baselines</CCSM_BASELINE>
<CESMSCRATCHROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME</CESMSCRATCHROOT>
<CCSM_CPRNC>/turquoise/usr/projects/climate/SHARED_CLIMATE/software/wolf/cprnc/v0.40/cprnc</CCSM_CPRNC>
<BATCHQUERY>mshow</BATCHQUERY>
<BATCHSUBMIT>msub</BATCHSUBMIT>
<BATCHREDIRECT></BATCHREDIRECT>
<batch_system type="moab" version="x.y">
<walltimes>
<walltime default="true">00:30:00</walltime>
<walltime ccsm_estcost="1">00:30:00</walltime>
<walltime ccsm_estcost="3">00:30:00</walltime>
</walltimes>
</batch_system>
<mpirun mpilib="default">
<executable>mpirun</executable>
<arguments>
<arg name="num_tasks"> -n {{ num_tasks }}</arg>
</arguments>
</mpirun>
<mpirun mpilib="openmpi">
<executable>mpirun</executable>
<arguments>
<arg name="num_tasks"> -n {{ num_tasks }}</arg>
</arguments>
</mpirun>
<mpirun mpilib="mvapich">
<executable>srun</executable>
<arguments>
<arg name="num_tasks"> -n {{ num_tasks }}</arg>
</arguments>
</mpirun>
<mpirun mpilib="mpi-serial">
<executable></executable>
</mpirun>
<GMAKE_J>4</GMAKE_J>
<MAX_TASKS_PER_NODE>16</MAX_TASKS_PER_NODE>
<SUPPORTED_BY>jacobsen.douglas -at- gmail.com</SUPPORTED_BY>
</machine>

<machine MACH="conejo">
<DESC>LANL Linux Cluster, 8 pes/node, batch system Moab</DESC>
<COMPILERS>intel,gnu</COMPILERS>
<MPILIBS>openmpi,mvapich,mpi-serial</MPILIBS>
<OS>LINUX</OS>
<RUNDIR>/lustre/scratch1/turquoise/$ENV{USER}/ACME/cases/$CASE/run</RUNDIR>
<EXEROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME/cases/$CASE/bld</EXEROOT>
<DIN_LOC_ROOT>/usr/projects/cesm/input_data</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/usr/projects/cesm/input_data/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME/archive/$CASE</DOUT_S_ROOT>
<DOUT_L_MSROOT>UNSET</DOUT_L_MSROOT>
<CCSM_BASELINE>/usr/projects/cesm/input_data/ccsm_baselines</CCSM_BASELINE>
<CESMSCRATCHROOT>/lustre/scratch1/turquoise/$ENV{USER}/ACME</CESMSCRATCHROOT>
<CCSM_CPRNC>/turquoise/usr/projects/climate/SHARED_CLIMATE/software/wolf/cprnc/v0.40/cprnc</CCSM_CPRNC>
<BATCHQUERY>mshow</BATCHQUERY>
<BATCHSUBMIT>msub</BATCHSUBMIT>
<BATCHREDIRECT></BATCHREDIRECT>
<batch_system type="moab" version="x.y">
<walltimes>
<walltime default="true">00:30:00</walltime>
<walltime ccsm_estcost="1">00:30:00</walltime>
<walltime ccsm_estcost="3">00:30:00</walltime>
</walltimes>
</batch_system>
<mpirun mpilib="default">
<executable>mpirun</executable>
<arguments>
<arg name="num_tasks"> -n {{ num_tasks }}</arg>
</arguments>
</mpirun>
<mpirun mpilib="openmpi">
<executable>mpirun</executable>
<arguments>
<arg name="num_tasks"> -n {{ num_tasks }}</arg>
</arguments>
</mpirun>
<mpirun mpilib="mvapich">
<executable>srun</executable>
<arguments>
<arg name="num_tasks"> -n {{ num_tasks }}</arg>
</arguments>
</mpirun>
<mpirun mpilib="mpi-serial">
<executable></executable>
</mpirun>
<GMAKE_J>4</GMAKE_J>
<MAX_TASKS_PER_NODE>8</MAX_TASKS_PER_NODE>
<SUPPORTED_BY>jacobsen.douglas -at- gmail.com</SUPPORTED_BY>
</machine>



<machine MACH="babbage">
<DESC>NERSC , 16 pes/node, batch system is PBS</DESC>
<OS>LINUX</OS>
Expand Down
49 changes: 49 additions & 0 deletions cime/machines/env_mach_specific.conejo
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#! /bin/csh -f

#-------------------------------------------------------------------------------
# Modules and library paths
#-------------------------------------------------------------------------------
source /usr/share/Modules/init/csh
module purge
module use /usr/projects/cesm/software/local/modulefiles/all

if ( $COMPILER == "intel" ) then
# module load intel/13.1.3
module load intel/15.0.5
endif

if ( $COMPILER == "pgi" ) then
# 13.10 is too old to support
# PGI support is currently disabled until we have access to a newer version
# (i.e. 14.10)
module load pgi/13.10
endif

if ( $COMPILER == "gnu" ) then
module load gcc/4.8.2
endif

if ( $MPILIB == "openmpi" ) then
module load openmpi/1.6.5
endif

if ( $MPILIB == "mvapich" ) then
module load mvapich2/1.8
endif

module load netcdf/4.4.0
module load parallel-netcdf/1.5.0
module load cmake/3.0.0
module load mkl/11.2.4
module load totalview

if ( $COMPILER == "gnu" && $MPILIB == "openmpi" ) then
module load trilinos
module load albany
endif

if ( $?PERL ) then
printenv
endif

module list
44 changes: 44 additions & 0 deletions cime/machines/env_mach_specific.wolf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#! /bin/csh -f

#-------------------------------------------------------------------------------
# Modules and library paths
#-------------------------------------------------------------------------------
source /usr/share/Modules/init/csh
module purge
module use /usr/projects/climate/SHARED_CLIMATE/modulefiles/all

if ( $COMPILER == "intel" ) then
module load intel/13.1.3
endif

if ( $COMPILER == "pgi" ) then
# 13.10 is too old to support
# PGI support is currently disabled until we have access to a newer version
# (i.e. 14.10)
module load pgi/13.10
endif

if ( $COMPILER == "gnu" ) then
module load gcc/4.8.2
endif

if ( $MPILIB == "openmpi" ) then
module load openmpi/1.6.5
endif

if ( $MPILIB == "mvapich" ) then
module load mvapich2/1.8
endif

module load netcdf/4.4.0
module load parallel-netcdf/1.5.0
module load cmake/3.0.0

if ( $COMPILER == "gnu" && $MPILIB == "openmpi" ) then
module load trilinos
module load albany
endif

if ( $?PERL ) then
printenv
endif
10 changes: 8 additions & 2 deletions cime/utils/perl5lib/Batch/BatchUtils.pm
View file
Original file line number Diff line number Diff line change
Expand Up @@ -216,8 +216,14 @@ sub submitSingleJob()

eval {
open (my $RUN, "-|", $runcmd) // die "job submission failed, $!";
$output = <$RUN>;
close $RUN or die "job submission failed: |$?|, |$!|"
# $output = <$RUN>;
foreach (<$RUN>) {
chomp;
print "$_\n";
$output .= $_;
}

close $RUN or die "job submission failed: |$?|, |$!|"
};
my $exitstatus = ($?>>8);
if($exitstatus != 0)
Expand Down
6 changes: 4 additions & 2 deletions components/clm/src/ED/biogeochem/EDGrowthFunctionsMod.F90
View file
Original file line number Diff line number Diff line change
Expand Up @@ -350,8 +350,10 @@ subroutine mortality_rates( cohort_in,cmort,hmort,bmort )
real(r8) :: frac ! relativised stored carbohydrate

! 'Background' mortality (can vary as a function of density as in ED1.0 and ED2.0, but doesn't here for tractability)
bmort = 0.014_r8

! bmort = EDecophyscon%b_mort(cohort_in%pft) !0.014_r8
! RGK:/CX HOLDING OFF ON SENS-ANALYSIS UNTIL MACHINE CONFIGS SQUARED AWAY
bmort = 0.014_r8

! Proxy for hydraulic failure induced mortality.
if(cohort_in%patchptr%btran_ft(cohort_in%pft) <= 0.000001_r8)then
hmort = ED_val_stress_mort
Expand Down
4 changes: 2 additions & 2 deletions components/clm/src/ED/biogeochem/EDPhysiologyMod.F90
View file
Original file line number Diff line number Diff line change
Expand Up @@ -309,13 +309,13 @@ subroutine phenology( currentSite, ed_phenology_inst, temperature_inst, watersta
a = -68.0_r8
b = 638.0_r8
c = -0.001_r8
coldday = 5.0_r8
coldday = 5.0_r8 !ed_ph_chiltemp

mindayson = 30

!Parameters from SDGVM model of senesence
ncolddayslim = 5
cold_t = 7.5_r8
cold_t = 7.5_r8 ! ed_ph_coldtemp

t = udata%time_period
temp_in_C = t_veg24(currentSite%oldest_patch%clm_pno-1) - tfrz
Expand Down
Loading

0 comments on commit 19fe567

Please sign in to comment.