Skip to content

Commit

Permalink
Merge pull request #3713 from climbfuji/cheyenne_intel_18
Browse files Browse the repository at this point in the history 
ufs_release_v1.1: remove unnecessary/incorrect configuration options for Cheyenne for the UFS; downgrade Intel 19.x.y to 18.m.n on Cheyenne, Gaea, Orion

### Description

This PR removes a whole bunch of unnecessary/incorrect configuration options for Cheyenne for the UFS and reverts Intel back to Intel-18.0.5 on Cheyenne, Gaea, Orion.

Beforehand, the CIME regression tests failed frequently in `chgres_cube.exe` for the highest resolution cases, see description here: ufs-community/ufs-mrweather-app#190

With this change and the associated change in the ufs-weather-model and NCEPLIBS-external (documentation only), the regression tests ran successfully (tried one time thus far, will conduct multiple runs to make sure I wasn't just lucky).
```
export UFS_DRIVER=nems
export UFS_INPUT=$CESMDATAROOT
export UFS_SCRATCH=/glade/work/heinzell/fv3/ufs-mrweather-app/ufs_scratch
qcmd -l walltime=3:00:00 -- "export UFS_DRIVER=nems; CIME_MODEL=ufs ./create_test --xml-testlist ../../src/model/FV3/cime/cime_config/testlist.xml --xml-machine cheyenne --xml-compiler intel --workflow ufs-mrweather_wo_post -j 4 --walltime 03:00:00"

...

heinzell@cheyenne3:/glade/work/heinzell/fv3/ufs-mrweather-app/ufs-mrweather-app-release-public-v1/cime/scripts> /glade/work/heinzell/fv3/ufs-mrweather-app/ufs_scratch/cs.status.20200917_210120_2ap63n | grep Overall
  ERS_Lh11.C96.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  ERS_Lh11.C96.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  PET_Lh11.C96.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C192.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C192.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C384.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C384.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C768.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C768.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C96.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3.C96.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C192.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C192.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C384.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C384.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C768.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C768.GFSv16beta.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C96.GFSv15p2.cheyenne_intel (Overall: PASS) details:
  SMS_Lh3_D.C96.GFSv16beta.cheyenne_intel (Overall: PASS) details:
```

The PR also replaces a few tabs with whitespaces for consistent formatting in `config/ufs/machines/config_machines.xml`.

Note that we will need to (re-)create tag `ufs-v1.1.0` (or whatever the CIME convention is; this is what we use for all the UFS components and external libraries, and what is also consistent with MRW App release 1.0).
  • Loading branch information
@billsacks
billsacks authored Sep 22, 2020
2 parents e0dcc59 + 370072f commit cac6a3e
Show file tree
Hide file tree
Showing 2 changed files with 29 additions and 68 deletions.
36 changes: 0 additions & 36 deletions config/ufs/machines/config_compilers.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -276,16 +276,6 @@ using a fortran linker.
<SCC> icc </SCC>
<SCXX> icpc </SCXX>
<SFC> ifort </SFC>
<SLIBS>
<append MPILIB="mpich"> -mkl=cluster </append>
<append MPILIB="mpich2"> -mkl=cluster </append>
<append MPILIB="mvapich"> -mkl=cluster </append>
<append MPILIB="mvapich2"> -mkl=cluster </append>
<append MPILIB="mpt"> -mkl=cluster </append>
<append MPILIB="openmpi"> -mkl=cluster </append>
<append MPILIB="impi"> -mkl=cluster </append>
<append MPILIB="mpi-serial"> -mkl </append>
</SLIBS>
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
</compiler>

Expand Down Expand Up @@ -491,39 +481,13 @@ using a fortran linker.
</FFLAGS>
</compiler>

<compiler MACH="cheyenne">
<CPPDEFS>
<!-- these flags enable nano timers -->
<append MODEL="gptl"> -DHAVE_NANOTIME -DBIT64 -DHAVE_VPRINTF -DHAVE_BACKTRACE -DHAVE_SLASHPROC -DHAVE_COMM_F2C -DHAVE_TIMES -DHAVE_GETTIMEOFDAY </append>
</CPPDEFS>
<NETCDF_PATH>$ENV{NETCDF}</NETCDF_PATH>
<PIO_FILESYSTEM_HINTS>gpfs</PIO_FILESYSTEM_HINTS>
<PNETCDF_PATH>$ENV{PNETCDF}</PNETCDF_PATH>
</compiler>

<compiler MACH="cheyenne" COMPILER="gnu">
<CPPDEFS>
<append MODEL="pio1"> -DNO_MPIMOD </append>
</CPPDEFS>
<SLIBS>
<append> -ldl </append>
</SLIBS>
</compiler>

<compiler MACH="cheyenne" COMPILER="intel">
<CFLAGS>
<append> -qopt-report -xCORE-AVX2 </append>
</CFLAGS>
<FFLAGS>
<append> -qopt-report -xCORE-AVX2 </append>
</FFLAGS>
<CMAKE_OPTS>
<append DEBUG="TRUE"> -DPIO_ENABLE_LOGGING=ON </append>
</CMAKE_OPTS>
<PFUNIT_PATH MPILIB="mpi-serial" compile_threaded="FALSE">$ENV{CESMDATAROOT}/tools/pFUnit/pFUnit3.2.8_cheyenne_Intel17.0.1_noMPI_noOpenMP</PFUNIT_PATH>
<PFUNIT_PATH MPILIB="mpt" compile_threaded="TRUE">$ENV{CESMDATAROOT}/tools/pFUnit/pFUnit3.2.8_cheyenne_Intel17.0.1_MPI_openMP</PFUNIT_PATH>
<!-- SET to FALSE for intel 17 and 18 TRUE otherwise -->
<HAS_F2008_CONTIGUOUS>TRUE</HAS_F2008_CONTIGUOUS>
</compiler>

<compiler MACH="hera" COMPILER="intel">
Expand Down
61 changes: 29 additions & 32 deletions config/ufs/machines/config_machines.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,42 +105,40 @@ This allows using a different mpirun command to launch unit tests
<cmd_path lang="sh">module</cmd_path>
<cmd_path lang="csh">module</cmd_path>
<modules>
<command name="purge"/>
<command name="load">ncarenv/1.3</command>
<command name="load">cmake/3.16.4</command>
<command name="purge"/>
<command name="load">ncarenv/1.3</command>
<command name="load">cmake/3.16.4</command>
</modules>
<modules compiler="intel">
<command name="load">intel/19.0.5</command>
<command name="load">mkl</command>
<command name="load">intel/18.0.5</command>
</modules>
<modules compiler="gnu">
<command name="load">gnu/8.3.0</command>
<command name="load">openblas/0.3.6</command>
</modules>
<modules mpilib="mpt" compiler="gnu">
<command name="load">mpt/2.19</command>
<command name="load">netcdf/4.7.3</command>
<command name="load">mpt/2.19</command>
<command name="load">netcdf/4.7.3</command>
</modules>
<modules mpilib="mpt" compiler="intel">
<command name="load">mpt/2.19</command>
<command name="load">netcdf/4.7.3</command>
<command name="load">pnetcdf/1.12.1</command>
<command name="load">mpt/2.19</command>
<command name="load">netcdf/4.7.3</command>
<command name="load">pnetcdf/1.12.1</command>
</modules>
<modules mpilib="impi" compiler="intel">
<command name="load">impi/2019.6.154</command>
<command name="load">netcdf-mpi/4.7.3</command>
<command name="load">pnetcdf/1.12.1</command>
</modules>
<modules>
<command name="load">ncarcompilers/0.5.0</command>
<command name="load">ncarcompilers/0.5.0</command>
</modules>
<modules mpilib="mpt" compiler="gnu">
<command name="use">/glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19</command>
<command name="load">NCEPlibs/1.1.0</command>
<command name="use">/glade/p/ral/jntp/GMTB/tools/modulefiles/gnu-8.3.0/mpt-2.19</command>
<command name="load">NCEPlibs/1.1.0</command>
</modules>
<modules mpilib="mpt" compiler="intel">
<command name="use">/glade/p/ral/jntp/GMTB/tools/modulefiles/intel-19.0.5/mpt-2.19</command>
<command name="load">NCEPlibs/1.1.0</command>
<command name="use">/glade/p/ral/jntp/GMTB/tools/modulefiles/intel-18.0.5/mpt-2.19</command>
<command name="load">NCEPlibs/1.1.0</command>
</modules>
<modules compiler="intel" mpilib="impi" DEBUG="FALSE" comp_interface="nuopc">
<command name="use">/glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/intel/19.0.5</command>
Expand All @@ -151,10 +149,10 @@ This allows using a different mpirun command to launch unit tests
<command name="load">esmf-8.1.0b24-ncdfio-intelmpi-g</command>
</modules>
<modules compiler="gnu" mpilib="mpi-serial">
<command name="load">netcdf/4.7.1</command>
<command name="load">netcdf/4.7.1</command>
</modules>
<modules compiler="intel" mpilib="mpi-serial">
<command name="load">netcdf/4.7.1</command>
<command name="load">netcdf/4.7.1</command>
</modules>
</module_system>
<environment_variables queue="share">
Expand All @@ -164,10 +162,9 @@ This allows using a different mpirun command to launch unit tests
<env name="ESMF_RUNTIME_PROFILE">ON</env>
<env name="ESMF_RUNTIME_PROFILE_OUTPUT">SUMMARY</env>
<env name="OMP_NUM_THREADS">1</env>
<env name="OMP_STACKSIZE">1024M</env>
<env name="MPI_TYPE_DEPTH">16</env>
<env name="MPI_IB_CONGESTED">1</env>
<env name="MPI_USE_ARRAY"/>
<env name="OMP_STACKSIZE">512M</env>
<env name="MPI_TYPE_DEPTH">20</env>
<env name="ESMF_RUNTIME_COMPLIANCECHECK">OFF:depth=4</env>
</environment_variables>
<resource_limits>
<resource name="RLIMIT_STACK">-1</resource>
Expand Down Expand Up @@ -204,23 +201,23 @@ This allows using a different mpirun command to launch unit tests
</arguments>
</mpirun>
<module_system type="module">
<init_path lang="perl">/opt/cray/pe/modules/3.2.10.5/init/perl.pm</init_path>
<init_path lang="python">/opt/cray/pe/modules/3.2.10.5/init/python.py</init_path>
<init_path lang="csh">/opt/cray/pe/modules/3.2.10.5/init/csh</init_path>
<init_path lang="sh">/opt/cray/pe/modules/3.2.10.5/init/sh</init_path>
<cmd_path lang="perl">/opt/cray/pe/modules/4.1.3.1/bin/modulecmd perl</cmd_path>
<cmd_path lang="python">/opt/cray/pe/modules/4.1.3.1/bin/modulecmd python</cmd_path>
<init_path lang="perl">/opt/cray/pe/modules/default/init/perl.pm</init_path>
<init_path lang="python">/opt/cray/pe/modules/default/init/python.py</init_path>
<init_path lang="csh">/opt/cray/pe/modules/default/init/csh</init_path>
<init_path lang="sh">/opt/cray/pe/modules/default/init/sh</init_path>
<cmd_path lang="perl">/opt/cray/pe/modules/default/bin/modulecmd perl</cmd_path>
<cmd_path lang="python">/opt/cray/pe/modules/default/bin/modulecmd python</cmd_path>
<cmd_path lang="sh">module</cmd_path>
<cmd_path lang="csh">module</cmd_path>
<modules compiler="intel">
<command name="load">PrgEnv-intel/6.0.5</command>
<command name="unload">intel</command>
<command name="load">intel/19.0.5.281</command>
<command name="load">intel/18.0.6.288</command>
<command name="unload">cray-mpich</command>
<command name="load">cray-mpich/7.7.11</command>
<command name="unload">cray-netcdf</command>
<command name="load">cmake/3.17.0</command>
<command name="use">/lustre/f2/pdata/esrl/gsd/ufs/modules/modulefiles/intel-19.0.5.281/cray-mpich-7.7.11</command>
<command name="use">/lustre/f2/pdata/esrl/gsd/ufs/modules/modulefiles/intel-18.0.6.288/cray-mpich-7.7.11</command>
<command name="load">NCEPlibs/1.1.0</command>
<command name="load">alps</command>
</modules>
Expand Down Expand Up @@ -332,12 +329,12 @@ This allows using a different mpirun command to launch unit tests
<cmd_path lang="python">/apps/lmod/lmod/libexec/lmod python</cmd_path>
<modules compiler="intel">
<command name="purge"/>
<command name="load">intel/2020</command>
<command name="load">intel/2018.4</command>
<command name="load">netcdf/4.7.2</command>
<command name="load">cmake/3.15.4</command>
</modules>
<modules mpilib="impi">
<command name="load">impi/2020</command>
<command name="load">impi/2018.4</command>
</modules>
</module_system>
<environment_variables>
Expand Down

0 comments on commit cac6a3e

Please sign in to comment.