Skip to content

Commit

Permalink
Corrections and updates for installation
Browse files Browse the repository at this point in the history
  • Loading branch information
@aekiss
aekiss committed Oct 27, 2017
1 parent 388659f commit 457d216
Show file tree
Hide file tree
Showing 3 changed files with 230 additions and 24 deletions.
118 changes: 94 additions & 24 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# ACCESS-OM2 pre-release

ACCESS-OM2 is a coupled ice and ocean global model. It is being developed through a collaborate with [COSIMA](www.cosima.org.au), [ARCCSS](www.arccss.org.au) and [CSIRO](www.csiro.au). It builds on the ACCESS-OM model orinially developed at CSIRO [1].
ACCESS-OM2 is a coupled ice and ocean global model. It is being developed through a collaboration with [COSIMA](http://www.cosima.org.au), [ARCCSS](http://www.arccss.org.au) and [CSIRO](http://www.csiro.au). It builds on the ACCESS-OM model originally developed at CSIRO [1].

The model consists of [MOM5.1](mom-ocean.science), [CICE5.1](http://oceans11.lanl.gov/trac/CICE), and a file-based atmosphere. The models are coupled together using the [OASIS3-MCT](https://portal.enes.org/oasis) coupler and regridding is done using ESMF_RegridWeightGen from [ESMF](https://www.earthsystemcog.org/projects/esmf/) and [KDTREE2](https://github.com/jmhodges/kdtree2).
The model consists of [MOM5.1](http://mom-ocean.science), [CICE5.1](http://oceans11.lanl.gov/trac/CICE), and a file-based atmosphere. The models are coupled together using the [OASIS3-MCT](https://portal.enes.org/oasis) coupler and regridding is done using ESMF_RegridWeightGen from [ESMF](https://www.earthsystemcog.org/projects/esmf/) and [KDTREE2](https://github.com/jmhodges/kdtree2).

ACCESS-OM2 comes with a number of standard experiments. These configurations include ice and ocean at 1, 1/4 and 1/10th degree resolution. [JRA55](http://jra.kishou.go.jp/JRA-55/index_en.html) and [CORE2](http://www.clivar.org/clivar-panels/omdp/core-2) forcing datasets are supported. These instruction are to build and run the 1 degree configration with JRA55 forcing.
ACCESS-OM2 comes with a number of standard experiments. These configurations include ice and ocean at 1, 1/4 and 1/10th degree resolution. [JRA-55](http://jra.kishou.go.jp/JRA-55/index_en.html) and [CORE2](http://www.clivar.org/clivar-panels/omdp/core-2) forcing datasets are supported. These instruction are to build and run the 1 degree configration with JRA-55 repeat-year forcing forcing (RYF).

This document describes how to download, compile and run the model. The instructions have only been tested on the [NCI](www.nci.org.au) raijin supercomputer.
This document describes how to download, compile and run the model. The instructions have only been tested on the [NCI](http://www.nci.org.au) raijin supercomputer.

## Prerequisites

Expand All @@ -16,22 +16,24 @@ The ACCESS-OM2 depends on the following software:
* git distributed version control software.
* a fortran compiler such as gfortran or intel-fc.
* an MPI implementation such as OpenMPI.
* Python and [pytest](https://docs.pytest.org) to run the tests.
* Python and [pytest](https://docs.pytest.org) to run the tests (optional).

To use JRA-55 on NCI you need to be a member of the ua8 project - apply via <https://my.nci.org.au>

To use JRA55 on NCI you need to be a member of the ua8 project - apply via <https://my.nci.org.au>
## Install

Start by downloading the experiment configurations and the source repositories. This should be downloaded to a place which has enough disk space for the model inputs and output. On raijin it should be downloaded to `/short/${PROJECT}/${USER}`.

For a new install:
```{bash}
cd /short/${PROJECT}/${USER}
git clone --recursive https://github.com/OceansAus/access-om2.git
cd access-om2
```

or if you have an existing download and would like to update to the latest version:

```{bash}
cd /short/${PROJECT}/${USER}
cd access-om2
git pull
git submodule update
Expand All @@ -53,24 +55,41 @@ setenv OASIS_ROOT $ACCESS_OM_DIR/src/oasis3-mct/
The next step is to create the 'lab' by downloading experiment input data and creating directories.
```{bash}
./get_input_data.py
mkdir -p bin
```

If the above script does not work for any reason the input can be setup manually with:

```{bash}
cp /short/public/access-om2/input_b8053e87.tar.gz ./
tar zxvf input_b8053e87.tar.gz
```

## Compile

Now to build each model. There are two ways to do this. If you have pytest installed, an easier way to compile all models is to use the build test:
Now to build each model. There several ways to do this.

### The easy way
The easiest is simply
```{bash}
./install.sh
```
which should will build and install all models, at 1, 1/4 and 1/10th degree resolution, with JA55 forcing. You can then skip to the "Run" section below.

### With payu **currently not working**
Alternatively, if you have [pytest](https://docs.pytest.org) available, it can compile all models, at 1, 1/4 and 1/10th degree resolution. This is the easiest option if available.

On NCI you can load pytest with:
```{bash}
module use ~access/modules
module load pythonlib/pytest
```

Then do compilation (might take 10-15 min):
```{bash}
python -m pytest test/test_build.py
```
You can then skip to the "Pre-run setup and checks" section below.

### The hard way
Otherwise, the models can be build individually. Start with OASIS because it is needed by the others:

```
Expand Down Expand Up @@ -109,6 +128,8 @@ make jra55
```

## Pre-run setup and checks:
You should skip this if you built the model using `install.sh`.

First check that the executables exist:

```{bash}
Expand All @@ -123,10 +144,6 @@ ls $ACCESS_OM_DIR/src/cice5/build_auscom_1440x1080_480p/cice_auscom_1440x1080_48
ls $ACCESS_OM_DIR/src/cice5/build_auscom_3600x2700_1200p/cice_auscom_3600x2700_1200p.exe
```

You will also have a directory `$ACCESS_OM_DIR/control` whose subdirectories are model configurations at 1, 1/4 and 1/10 degree resolution. `config.yaml` within each of these subdirectories gives the PBS specification for the job, including executable names. Details are here: <http://payu.readthedocs.io/en/latest/config.html>

**You will need to edit `config.yaml` to set `project` and `shortpath` appropriately.** Service units are charged to `project` and output is saved in `shortpath`.

The executables then need to be copied to the directory `$ACCESS_OM_DIR/bin`.

First copy and rename mppnccombine:
Expand All @@ -137,24 +154,24 @@ cp $ACCESS_OM_DIR/src/mom/bin/mppnccombine.nci $ACCESS_OM_DIR/bin/mppnccombine

Now for the model executables. As an added complication, they need to be renamed to match the `exe` names used in the experiment configuration file `config.yaml` --- check this with `grep exe $ACCESS_OM_DIR/control/*/config.yaml` (for uniqueness the name of each executable is changed to include the hash/id of the git commit from which they were built).

All three JRA55 models need this (but **you may need to update the hash/ids**):
All three JRA-55 models need this (but **you may need to update the hash/ids**):
```{bash}
cp $ACCESS_OM_DIR/src/mom/exec/nci/ACCESS-OM/fms_ACCESS-OM.x $ACCESS_OM_DIR/bin/fms_ACCESS-OM_160db8a0.x
cp $ACCESS_OM_DIR/src/matm/build_jra55/matm_jra55.exe $ACCESS_OM_DIR/bin/matm_jra55_2318e909.exe
```
plus one of the following:

For the 1 degree JRA55 model (but **you may need to update the hash/id**):
For the 1 degree JRA-55 model (but **you may need to update the hash/id**):
```{bash}
cp $ACCESS_OM_DIR/src/cice5/build_auscom_360x300_24p/cice_auscom_360x300_24p.exe $ACCESS_OM_DIR/bin/cice_auscom_360x300_24p_fe730022.exe
```

For the 1/4 degree JRA55 model (but **you may need to update the hash/id**):
For the 1/4 degree JRA-55 model (but **you may need to update the hash/id**):
```{bash}
cp $ACCESS_OM_DIR/src/cice5/build_auscom_1440x1080_480p/cice_auscom_1440x1080_480p.exe $ACCESS_OM_DIR/bin/cice_auscom_1440x1080_480p_fe730022.exe
```

For the 1/10 degree JRA55 model (but **you may need to update the hash/id**):
For the 1/10 degree JRA-55 model (but **you may need to update the hash/id**):
```{bash}
cp $ACCESS_OM_DIR/src/cice5/build_auscom_3600x2700_1200p/cice_auscom_3600x2700_1200p.exe $ACCESS_OM_DIR/bin/cice_auscom_3600x2700_1200p_fe730022.exe
```
Expand All @@ -166,27 +183,80 @@ cp $ACCESS_OM_DIR/src/mom/exec/nci/ACCESS-OM/fms_ACCESS-OM.x $ACCESS_OM_DIR/bin/
cp $ACCESS_OM_DIR/src/matm/build_nt62/matm_nt62.exe $ACCESS_OM_DIR/bin/matm_nt62_b3a80f3b.exe
```

### load payu
## Run
[payu](http://payu.readthedocs.io) needs to be available. On NCI you just do:
```{bash}
module load payu/dev
module load payu
```

## Run

Each of the model configurations is run by payu from within its respective directory in `$ACCESS_OM_DIR/control/`.
`config.yaml` within each of these subdirectories gives the PBS specification for the job, including executable names. Details are here: <http://payu.readthedocs.io/en/latest/config.html>

**You will need to edit `config.yaml` to set `project` and `shortpath` appropriately.** Service units are charged to `project` and output is saved in `shortpath`.


For example, to run the 1 degree JRA55 RYF experiment:
For example, to run the 1 degree JRA-55 RYF experiment:
```{bash}
cd $ACCESS_OM_DIR/control/1deg_jra55_ryf/
payu run
```

or to do N runs:
```{bash}
cd $ACCESS_OM_DIR/control/1deg_jra55_ryf/
payu run -n N
```
See <http://payu.readthedocs.io/en/latest/usage.html> for more details.

On NCI, status of submitted runs can be checked with `qstat -u ${USER}`.

See <http://payu.readthedocs.io/en/latest/design.html> for explanation of where run output is stored.
Output will be stored in `$ACCESS_OM_DIR/control/1deg_jra55_ryf/archive` if the run is successful.

If the run crashes you can find diagnostics in `$ACCESS_OM_DIR/control/1deg_jra55_ryf/access-om2.err` and any output from the run will be in `$ACCESS_OM_DIR/control/1deg_jra55_ryf/work`. To rerun after a crash, do `payu sweep` before `payu run` to clear out all the debris from the crash.

## NCI users only: Integration with cosima cookbook
Here's how you can automatically copy your model output to `/g/data3/` to make it available for analysis via the [cosima cookbook](http://cosima-cookbook.readthedocs.io/en/).

You will need write access to `/g/data3/hh5/tmp/cosima/`.

**This example is for `1deg_jra55_ryf`; you'll need to make the obvious changes for other models.**

First check the existing directories in `/g/data3/hh5/tmp/cosima/access-om2/` and create a new one with a unique name, e.g.
```
mkdir /g/data3/hh5/tmp/cosima/access-om2/1deg_jra55_ryf_spinupN/
```

See <http://payu.readthedocs.io/en/latest/usage.html> for how to continue additional runs from restarts.
Then copy ``sync_to_gdata.sh`` to your control directory, e.g.
```
cp /short/v45/amh157/access-om2/control/1deg_jra55_ryf/sync_to_gdata.sh $ACCESS_OM_DIR/control/1deg_jra55_ryf/sync_to_gdata.sh
```
*TODO: source `sync_to_gdata.sh` from somewhere else?*

Now edit ``$ACCESS_OM_DIR/control/1deg_jra55_ryf/sync_to_gdata.sh`` to set ``GDATADIR`` to the directory in `g/data3` you created above, e.g.
```
GDATADIR=/g/data3/hh5/tmp/cosima/access-om2/1deg_jra55_ryf_spinupN/
```

Finally, edit ``$ACCESS_OM_DIR/control/1deg_jra55_ryf/config.yaml`` to add
```
postscript: sync_to_gdata.sh
```
as the final line.
This will run ``sync_to_gdata.sh`` after each run, automatically rsynching collated output from all previous runs to ``/g/data3/hh5/tmp/cosima/access-om2/1deg_jra55_ryf_spinupN/``, where cosima cookbook can find it and add it to the cookbook database.

In python, you need to use `build_index` to update the cookbook index to see your new runs:
```{python}
import cosima_cookbook as cc
cc.build_index()
```

The `/g/data3` directory you created above (e.g. `1deg_jra55_ryf_spinupN`) will be the experiment's name in cosima cookbook, i.e. in the list returned by `cc.get_experiments(configuration)`, where `configuration` is the parent directory name (e.g. `access-om2`):
```{python}
configuration = 'access-om2'
expts = cc.get_experiments(configuration)
```
For further cosima cookbook usage instructions and examples see <http://cosima-cookbook.readthedocs.io/en/>.

## Testing

Expand Down
87 changes: 87 additions & 0 deletions hashexe.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
#!/usr/bin/env sh

# Copy ACCESS-OM2 JRA55 executables to bin with githash in name, and config.yaml to match.
# All changes to config.yaml are reported.
# NB: requires ACCESS_OM_DIR environment variable.
# NB: doesn't handle CORE2 experiments (though this could easily be added).
# Andrew Kiss https://github.com/aekiss

set -e

matmpath=${ACCESS_OM_DIR}/src/matm/build_jra55/matm_jra55.exe
fmspath=${ACCESS_OM_DIR}/src/mom/exec/nci/ACCESS-OM/fms_ACCESS-OM.x
cice1path=${ACCESS_OM_DIR}/src/cice5/build_auscom_360x300_24p/cice_auscom_360x300_24p.exe
cice025path=${ACCESS_OM_DIR}/src/cice5/build_auscom_1440x1080_480p/cice_auscom_1440x1080_480p.exe
cice010path=${ACCESS_OM_DIR}/src/cice5/build_auscom_3600x2700_1200p/cice_auscom_3600x2700_1200p.exe
mppnccombinepath=${ACCESS_OM_DIR}/src/mom/bin/mppnccombine.nci

config1path=${ACCESS_OM_DIR}/control/1deg_jra55_ryf/config.yaml
config025path=${ACCESS_OM_DIR}/control/025deg_jra55_ryf/config.yaml
config010path=${ACCESS_OM_DIR}/control/01deg_jra55_ryf/config.yaml

bindir=${ACCESS_OM_DIR}/bin

mkdir -p ${bindir}

echo "Getting executable hashes..."
cd $(dirname "${matmpath}") && matmhash=`git rev-parse --short=8 HEAD`
cd $(dirname "${fmspath}") && fmshash=`git rev-parse --short=8 HEAD`
cd $(dirname "${cice1path}") && cicehash=`git rev-parse --short=8 HEAD` # NB only one hash for all three cice builds

echo "Copying executables to "${bindir}" with hashes added to names..."
matmbn=$(basename "${matmpath}")
matmhashexe="${matmbn%.*}"_${matmhash}."${matmpath##*.}"
echo " cp ${matmpath} ${bindir}/${matmhashexe}"
cp ${matmpath} ${bindir}/${matmhashexe}

fmsbn=$(basename "${fmspath}")
fmshashexe="${fmsbn%.*}"_${fmshash}."${fmspath##*.}"
echo " cp ${fmspath} ${bindir}/${fmshashexe}"
cp ${fmspath} ${bindir}/${fmshashexe}

cice1bn=$(basename "${cice1path}")
cice1hashexe="${cice1bn%.*}"_${cicehash}."${cice1path##*.}"
echo " cp ${cice1path} ${bindir}/${cice1hashexe}"
cp ${cice1path} ${bindir}/${cice1hashexe}

cice025bn=$(basename "${cice025path}")
cice025hashexe="${cice025bn%.*}"_${cicehash}."${cice025path##*.}"
echo " cp ${cice025path} ${bindir}/${cice025hashexe}"
cp ${cice025path} ${bindir}/${cice025hashexe}

cice010bn=$(basename "${cice010path}")
cice010hashexe="${cice010bn%.*}"_${cicehash}."${cice010path##*.}"
echo " cp ${cice010path} ${bindir}/${cice010hashexe}"
cp ${cice010path} ${bindir}/${cice010hashexe}

echo " cp ${mppnccombinepath} ${bindir}/mppnccombine"
cp ${mppnccombinepath} ${bindir}/mppnccombine # no hash for mppnccombine


echo "Fixing exe in "${config1path}" to match executable names..."
sed "s/${matmbn%.*}.*/${matmhashexe}/g" < ${config1path} > ${config1path}-tmp
sed "s/${fmsbn%.*}.*/${fmshashexe}/g" < ${config1path}-tmp > ${config1path}-tmp2
sed "s/${cice1bn%.*}.*/${cice1hashexe}/g" < ${config1path}-tmp2 > ${config1path}-tmp3
diff ${config1path} ${config1path}-tmp3 || true
mv ${config1path}-tmp3 ${config1path}
rm ${config1path}-tmp*

echo "Fixing exe in "${config025path}" to match executable names..."
sed "s/${matmbn%.*}.*/${matmhashexe}/g" < ${config025path} > ${config025path}-tmp
sed "s/${fmsbn%.*}.*/${fmshashexe}/g" < ${config025path}-tmp > ${config025path}-tmp2
sed "s/${cice025bn%.*}.*/${cice025hashexe}/g" < ${config025path}-tmp2 > ${config025path}-tmp3
diff ${config025path} ${config025path}-tmp3 || true
mv ${config025path}-tmp3 ${config025path}
rm ${config025path}-tmp*

echo "Fixing exe in "${config010path}" to match executable names..."
sed "s/${matmbn%.*}.*/${matmhashexe}/g" < ${config010path} > ${config010path}-tmp
sed "s/${fmsbn%.*}.*/${fmshashexe}/g" < ${config010path}-tmp > ${config010path}-tmp2
sed "s/${cice010bn%.*}.*/${cice010hashexe}/g" < ${config010path}-tmp2 > ${config010path}-tmp3
diff ${config010path} ${config010path}-tmp3 || true
mv ${config010path}-tmp3 ${config010path}
rm ${config010path}-tmp*

echo "Success."
exit 0

49 changes: 49 additions & 0 deletions install.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/usr/bin/env sh

# Compile ACCESS-OM2 at 1, 1/4 and 1/10 degree resolution wth JRA-55 forcing
# NB: requires ACCESS_OM_DIR environment variable.
# Andrew Kiss https://github.com/aekiss

set -e

export OASIS_ROOT=${ACCESS_OM_DIR}/src/oasis3-mct/

cd ${ACCESS_OM_DIR}

# echo "Downloading experiment input data and creating directories..."
# ./get_input_data.py

echo "Compiling OASIS3-MCT..."
cd ${OASIS_ROOT}
make

echo "Compiling MOM5.1..."
cd ${ACCESS_OM_DIR}/src/mom/exp
./MOM_compile.csh --type ACCESS-OM --platform nci

echo "Compiling CICE5.1 at 1 degree..."
cd ${ACCESS_OM_DIR}/src/cice5
make # 1 degree
echo "Compiling CICE5.1 at 1/4 degree..."
make 025deg
echo "Compiling CICE5.1 at 1/10 degree..."
make 01deg

echo "Compiling MATM JRA-55 file-based atmosphere... "
cd ${ACCESS_OM_DIR}/src/matm
make jra55

echo "Checking all executables have been built..."
ls ${ACCESS_OM_DIR}/src/mom/exec/nci/ACCESS-OM/fms_ACCESS-OM.x
ls ${ACCESS_OM_DIR}/src/matm/build_jra55/matm_jra55.exe
ls ${ACCESS_OM_DIR}/src/cice5/build_auscom_360x300_24p/cice_auscom_360x300_24p.exe
ls ${ACCESS_OM_DIR}/src/cice5/build_auscom_1440x1080_480p/cice_auscom_1440x1080_480p.exe
ls ${ACCESS_OM_DIR}/src/cice5/build_auscom_3600x2700_1200p/cice_auscom_3600x2700_1200p.exe

source ${ACCESS_OM_DIR}/hashexe.sh

echo "Success."
echo "You will need to edit project, shortpath and possibly timestep in ${ACCESS_OM_DIR}/control/*/config.yaml"

exit 0

0 comments on commit 457d216

Please sign in to comment.