Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing suite #9

Merged
merged 30 commits into from
Jun 27, 2017
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
095f3bf
Test scripts now write to README.case instead of a separate README.te…
Jun 12, 2017
34cd5f0
when checking for compile success, write PASS/FAIL to new test_output…
Jun 12, 2017
cfabe15
update cice.run.setup.csh to add the option to write PASS/FAIL inform…
Jun 12, 2017
0c87661
delete old cice.<test_nae>.csh scripts and replace with cice.test.csh…
Jun 12, 2017
576c0cb
Merge branch 'master' of https://github.com/mattdturner/CICE into tes…
Jun 12, 2017
b2aafa2
Move batch logic from cice.run.setup.csh and cice.test.csh to new cic…
Jun 12, 2017
5668668
Remove the requirement for -b or -bd to be provided when using create…
Jun 12, 2017
17d722d
Add the option to pass a test ID to create.case, which gets included …
Jun 12, 2017
f52288d
Modification to create.case to properly handle running tests where -b…
Jun 12, 2017
d2dbf28
Add a check in create.case to ensure that the test requested is one o…
Jun 12, 2017
0db5c73
Update cice.run.setup.csh to not overwrite the ice.restart_file point…
Jun 13, 2017
d224516
Create new exact restart test case.
Jun 13, 2017
2ddbdd9
Add new README.test file that gives instructions on running the test …
Jun 13, 2017
4b7b976
add CICE_MACHINE_BASELINE default baseline directory to env.<machine>…
Jun 15, 2017
a3c6c85
bugfix for cheyenne machine. Also added error if using unknown machine
Jun 15, 2017
473d158
Check return status of calls to cice.launch.csh and cice.batch.csh. …
Jun 15, 2017
91e432f
Update the documentation for the recent modifications to create.case
Jun 15, 2017
16b7c4b
Add output to beginning of cice.tests.csh that prints information abo…
Jun 15, 2017
ff71bdb
add default CICE_TEST and CICE_BASELINE variables to cice.settings file
Jun 15, 2017
5729c95
Assign a default testid. Add new baseCom (-bc) and baseGen (-bg) arg…
Jun 15, 2017
6482090
Add PASS/FAIL commands to cice.restart.csh after calls to cice.run. …
Jun 15, 2017
f5e4543
For restart test case, create.case now copies new test_nml.restart2 f…
Jun 15, 2017
374955d
add new test_nml.restart2 options file
Jun 15, 2017
356f958
Modify README.test to account for the recent updates to the testing s…
Jun 15, 2017
7716032
Add logic to handle cases where -bg and -bc are both passed. Bugfix …
Jun 15, 2017
39b3e56
Add test name as an argument. Add logic to handle regression runs. …
Jun 15, 2017
204fd6c
A few corrections in README.test. Also added a few clarifying statem…
Jun 22, 2017
cb845ef
Change diagfreq from once per year to daily for annual test case
Jun 22, 2017
5445671
Update the list of available tests in README.test to also specify the…
Jun 22, 2017
3984d99
Add information about the -testid flag to README.test
Jun 27, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions README.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
CICE version 6 testing
June 2017

This README file documents how to use the testing suite developed for the
CICE Consortium sea ice model. The tests currently supported are:
1. "Smoke" test - Ensures that the model compiles successfully and runs
to completion for a 1 day simulation. Validation is
determined by performing a binary comparison on restart
files to a baseline dataset.
2. 10-day test - Ensures that the model compiles successfully and runs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the actual arguments needed for -t (smoke, 10day, restart, annual)

to completion for a 10 day simulation. Validation is
determined by performing a binary comparison on restart
files to a baseline dataset.
3. Exact restart - Ensures that the output from a cold-start simulation
and a restart simulation provide bit-for-bit identical
results.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

State explicitly that a baseline does not need to be created for restart tests

4. Annual test - Ensures that the model compiles successfully and runs
to completion for an annual simulation. Validation is
determined by performing a binary comparison on restart
files to a baseline dataset.

--- To generate a baseline dataset for a test case ---

Quickstart (example):

./create.case -t smoke -m conrad -b
cd smoke_baseline_gx3_conrad_4x1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove _baseline from first directory and add .t00 to second one

./cice.build
./cice.submit
# After job finishes, check output
cat test_output


--- To run a test case and compare to a baseline dataset ---

Quickstart (example):

./create.case -t smoke -m conrad
cd smoke_gx3_conrad_4x1
./cice.build
./cice.submit
# After job finishes, check output
cat test_output


--- To run a regression test (compare cicev6.0.1 output to cicev6.0.0 output) ---

Quickstart (example):

./create.case -t smoke -m conrad -b -bg cicev6.0.1 -bc cicev6.0.0
cd smoke_gx3_conrad_4x1
./cice.build
./cice.submit
# After job finishes, check output
cat test_output


Additional Details:
- The '-b' flag can be combined with '-bd' to specify the location where you
want the baseline dataset to be written. Without specifying '-bd', the
baseline dataset will be written to the default baseline directory found
in the env.<machine> file (CICE_MACHINE_BASELINE).
- If '-b' is not passed, the '-bd' option will specify the location of the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be -bc instead of -bd?

baseline dataset that you want to compare to. Without specifying '-bd',
the scripts will look for baseline datasets in the default baseline
directory found in the env.<machine> file (CICE_MACHINE_BASELINE).
- To generate a baseline dataset for a specific version (for regression testing),
use '-b' in conjunction to '-bg <version_name>'. The scripts will then place
the baseline dataset in $CICE_MACHINE_BASELINE/<version_name>/
5 changes: 4 additions & 1 deletion README_v6
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,17 @@ cd ~/mycase1
A bit more information:

create.case generates a case, use "create.case -h" for help with the tool.
-c is the case name and location (required)
-c is the case name and location
-m is the machine name (required)
-g is the resolution (default is gx3)
-p is the task x thread/task values (default is 4x1)
-s are comma separated optional env or namelist settings (default is "null")
-t is the test name and location (cannot be used with -c).
-b is used to specify that a baseline dataset is to be generated (only used with -t)
-bd is used to specify the location of the baseline datasets (only used with -t)
-bg is used to specify the cice version name for generating baseline datasets (only used with -t)
-bc is used to specify the cice versoin name for comparison. I.e., the version name for the baseline dataset (only used with -t)
-testid is used to specify a test ID (used only with -t)

Several files are placed in the case directory
- env.${machine} defines the environment
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
#! /bin/csh -f

echo ${0}
if ( $1 != "" ) then
echo ${0} ${1}
else
echo ${0}
endif

source ./cice.settings
source ${CICE_CASEDIR}/env.${CICE_MACHINE} || exit 2

set jobfile = cice.10day
set subfile = cice.submit
set jobfile = $1

set ntasks = ${CICE_NTASKS}
set nthrds = ${CICE_NTHRDS}
Expand All @@ -24,9 +27,6 @@ if ($ptile > ${maxtpn} / 2) @ ptile = ${maxtpn} / 2

#==========================================

# Create test script that runs cice.build, cice.run, and validates
#==========================================

cat >! ${jobfile} << EOF0
#!/bin/csh -f
EOF0
Expand All @@ -48,6 +48,20 @@ cat >> ${jobfile} << EOFB
#BSUB -P ${acct}
EOFB

#==========================================

else if (${CICE_MACHINE} =~ cheyenne*) then
cat >> ${jobfile} << EOFB
#PBS -j oe
#PBS -m ae
#PBS -V
#PBS -q regular
#PBS -N ${CICE_CASENAME}
#PBS -A ${CICE_ACCT}
#PBS -l select=${nnodes}:ncpus=${ntasks}:mpiprocs=${ntasks}
#PBS -l walltime=02:00:00
EOFB

else if (${CICE_MACHINE} =~ thunder* || ${CICE_MACHINE} =~ gordon* || ${CICE_MACHINE} =~ conrad*) then
cat >> ${jobfile} << EOFB
#PBS -N ${CICE_CASENAME}
Expand Down Expand Up @@ -78,88 +92,30 @@ EOFB
else if (${CICE_MACHINE} =~ wolf*) then
cat >> ${jobfile} << EOFB
#SBATCH -J ${CICE_CASENAME}
#SBATCH -t 0:10:00
#SBATCH -t 0:45:00
#SBATCH -A ${acct}
#SBATCH -N ${nnodes}
#SBATCH -e slurm%j.err
#SBATCH -o slurm%j.out
#SBATCH --mail-type FAIL
#SBATCH [email protected]
###SBATCH --mail-type END,FAIL
###SBATCH [email protected]
#SBATCH --qos=low
EOFB

endif

cat >> ${jobfile} << EOF2
cd ${CICE_CASEDIR}
source ./cice.settings || exit 2
source ./env.\${CICE_MACHINE} || exit 2

# Compile the CICE code
./cice.build
set rc_build = \$?

# Run the CICE model
./cice.run
set rc_run = \$?

EOF2

if ($1 != "") then
cat >> ${jobfile} << EOF3
# Get the final output filename
foreach file (\${CICE_RUNDIR}/restart/*)
set test_data = \$file
end

set baseline_data = $1/\$test_data:t

if ( { cmp -s \$test_data \$baseline_data } ) then
set rc_valid = 0
else
set rc_valid = 1
endif
EOF3
endif

cat >> ${jobfile} << EOF4

if (\$rc_build == 0) then
echo "Build: PASS"
else
echo "Build: FAIL"
endif

if (\$rc_run == 0) then
echo "Run: PASS"
else
echo "Run: FAIL"
endif
EOF4

if ($1 != "") then
cat >> ${jobfile} << EOF5
else if (${CICE_MACHINE} =~ pinto*) then
cat >> ${jobfile} << EOFB
#SBATCH -J ${CICE_CASENAME}
#SBATCH -t 0:45:00
#SBATCH -A ${acct}
#SBATCH -N ${nnodes}
#SBATCH -e slurm%j.err
#SBATCH -o slurm%j.out
###SBATCH --mail-type END,FAIL
###SBATCH [email protected]
#SBATCH --qos=standby
EOFB

if (\$rc_valid == 0) then
echo "Validation: PASS"
else
echo "Validation: FAIL"
endif
EOF5
echo "${0} ERROR ${CICE_MACHINE} unknown"
exit -1
endif

#==========================================

chmod +x ${jobfile}

#==========================================

cat >! ${subfile} << EOFS
#!/bin/csh -f

${CICE_MACHINE_SUBMIT} ${jobfile}
echo "\`date\` \${0}: ${CICE_CASENAME} job submitted" >> ${CICE_CASEDIR}/README.test

EOFS

chmod +x ${subfile}
7 changes: 7 additions & 0 deletions configuration/scripts/cice.build
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,18 @@ gmake -j ${CICE_MACHINE_BLDTHRDS} VPFILE=Filepath EXEC=${CICE_RUNDIR}/cice CICE_
if ($status != 0) then
echo "${0}: COMPILE FAILED, see"
echo " cat ${CICE_OBJDIR}/${CICE_BLDLOG_FILE}"
if ( $?CICE_TEST ) then
# This is a test case. Write output to test_output file
echo "FAIL $CICE_CASENAME build" >> ${CICE_CASEDIR}/test_output
endif
exit 99
endif

if !(-d ${CICE_LOGDIR}) mkdir -p ${CICE_LOGDIR}
cp -p ${CICE_BLDLOG_FILE} ${CICE_LOGDIR}/
echo "`date` ${0}:${CICE_CASENAME} build completed ${CICE_BLDLOG_FILE}" >> ${CICE_CASEDIR}/README.case
echo "${0}: COMPILE SUCCESSFUL, ${CICE_LOGDIR}/${CICE_BLDLOG_FILE}"
if ( $?CICE_TEST ) then
echo "PASS $CICE_CASENAME build" >> ${CICE_CASEDIR}/test_output
endif

56 changes: 56 additions & 0 deletions configuration/scripts/cice.launch.csh
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#! /bin/csh -f

echo ${0}

source ./cice.settings

set jobfile = $1

set ntasks = ${CICE_NTASKS}
set nthrds = ${CICE_NTHRDS}

#==========================================

if (${CICE_MACHINE} =~ yellowstone*) then
cat >> ${jobfile} << EOFR
setenv MP_TASK_AFFINITY core:\${OMP_NUM_THREADS}
mpirun.lsf ./cice >&! \$CICE_RUNLOG_FILE
EOFR

else if (${CICE_MACHINE} =~ cheyenne*) then
cat >> ${jobfile} << EOFR
mpiexec_mpt -n ${ntasks} ./cice >&! \$CICE_RUNLOG_FILE
EOFR

else if (${CICE_MACHINE} =~ thunder*) then
cat >> ${jobfile} << EOFR
mpiexec_mpt -np ${ntasks} omplace ./cice >&! \$CICE_RUNLOG_FILE
EOFR

else if (${CICE_MACHINE} =~ gordon* || ${CICE_MACHINE} =~ conrad*) then
cat >> ${jobfile} << EOFR
aprun -n ${ntasks} -N ${ntasks} -d ${nthrds} ./cice >&! \$CICE_RUNLOG_FILE
EOFR

else if (${CICE_MACHINE} =~ cori*) then
cat >> ${jobfile} << EOFR
srun -n ${ntasks} -c ${nthrds} ./cice >&! \$CICE_RUNLOG_FILE
EOFR

else if (${CICE_MACHINE} =~ wolf*) then
cat >> ${jobfile} << EOFR
mpirun -np ${ntasks} ./cice >&! \$CICE_RUNLOG_FILE
EOFR

else if (${CICE_MACHINE} =~ pinto*) then
cat >> ${jobfile} << EOFR
mpirun -np ${ntasks} ./cice >&! \$CICE_RUNLOG_FILE
EOFR
#cat >> ${jobfile} << EOFR
#srun -n ${ntasks} -c ${nthrds} ./cice >&! \$CICE_RUNLOG_FILE
#EOFR

else
echo "${0} ERROR ${CICE_MACHINE} unknown"
exit -1
endif
Loading