Leap year offsets with multiple IAF forcing cycles

[aekiss]

In our 1deg and 0.25deg IAF runs we cycle through 60 years of JRA55-do data starting 1 Jan 1958. This uses a Gregorian calendar (set in the JRA55 forcing files), i.e. years divisible by 400 are always leap years, as are years divisible by 4 unless they are divisible by 100.

So 2000, 2400 are leap years but 2100, 2200, 2300 are not.

Here's how the 5 cycles pan out:

• Cycle 1: 1958-2018 using 1958-2018 forcing. Nothing to go wrong.
• Cycle 2: 2018-2078 using 1958-2018 forcing. Leap years fortuitously align because 60 is divisible by 4 and 2000 wasn't a leap year.
• Cycle 3: 2078-2138 using 1958-2018 forcing. 2100 is not a leap year but the forcing is from 1980, which is a leap year.
• Cycle 4: 2138-2198 using 1958-2018 forcing. Leap years fortuitously align because 60 is divisible by 4 and 2000 wasn't a leap year.
• Cycle 5: 2198-2258 using 1958-2018 forcing. 2200 is not a leap year but the forcing is from 1960, which is a leap year.

In cycle 3 in /g/data3/hh5/tmp/cosima/access-om2-025/025deg_jra55v13_iaf_gmredi6/output071/atmosphere/log/matmxx.pe00000.log
we have

 cur_exp_date 2100-01-01T03:00:00
 cur_forcing_date 1980-01-01T03:00:00
...
 cur_exp_date 2100-02-28T21:00:00
 cur_forcing_date 1980-02-28T21:00:00
...
 cur_exp_date 2100-03-01T00:00:00
 cur_forcing_date 1980-02-29T00:00:00
...
 cur_exp_date 2102-01-01T00:00:00
 cur_forcing_date 1981-12-31T00:00:00

i.e. the dates slip out of sync on 1 Mar 2100, which uses 29 Feb 1980 forcing.
This offset persists in all subsequent runs, e.g.

grep cur_[ef] /g/data3/hh5/tmp/cosima/access-om2-025/025deg_jra55v13_iaf_gmredi6/output072/atmosphere/log/matmxx.pe00000.log | head
 cur_exp_date 2102-01-01T03:00:00
 cur_forcing_date 1981-12-31T03:00:00

Cycle 4 has a 1-day offset throughout.

In cycle 5 we start with a 1-day offset and then have a 2-day offset from 1 March 2200 to the end of the cycle:

grep cur_[ef] /g/data3/hh5/tmp/cosima/access-om2-025/025deg_jra55v13_iaf_gmredi6/output121/atmosphere/log/matmxx.pe00000.log
 cur_exp_date 2200-01-01T03:00:00
 cur_forcing_date 1959-12-31T03:00:00
...
 cur_exp_date 2200-02-28T21:00:00
 cur_forcing_date 1960-02-27T21:00:00
 cur_exp_date 2200-03-01T00:00:00
 cur_forcing_date 1960-02-28T00:00:00
...
 cur_exp_date 2200-03-02T03:00:00
 cur_forcing_date 1960-02-29T03:00:00
...
 cur_exp_date 2202-01-01T00:00:00
 cur_forcing_date 1961-12-30T00:00:00

This would have been a much bigger problem if the cycle length hadn't been divisible by 4.

To avoid this issue it may be better practice to restart each cycle as a new experiment so the forcing and model dates stay aligned. However timeseries of all cycles would then require post-processing with date shifts (which also produces the same kind of problems).

This is not an issue with the RYF runs because they use a noleap calendar (at least, this set in the JRA55 forcing files and presumably applies to all model components?)

@nichannah - any thoughts?

[russfiedler]

My feeling is that these runs should be treated as an ensemble of 60 year experiments with different initial conditions rather than a single time series. Doing this allows each experiment to be extended whenever new forcing becomes available. This isn't really possible with the single time series approach. It also makes it easy to compare across cycles.

Also note that the ocean outputs from the current experiments are already out of whack by 2 days from the start due to using 0001-01-01 as the offset time.

[aekiss]

Thanks @russfiedler. I'll link the start date issue for reference: #117

[aekiss]

Here's how to restart a new cycle from the final restart of the previous one: https://github.com/COSIMA/access-om2/wiki/Tutorials#Starting-an-experiment-from-existing-restarts

[nichannah]

In theory (although perhaps not with this forcing date range), the problem could be the other way around as well - i.e. a leap year in the experiment date that does not exist in the forcing. So, I think the fix will be in two parts:

1. For the case that the experiment date is not a leap year but the forcing is: skip the leap day in the forcing.
2. When the experiment date is a leap year but the forcing is not: replay Feb 28.

In addition to fixing the bug, I'd like to put in a check to make sure that the experiment and forcing times only differ on the year. The check can be switched to a warning with a namelist parameter for use cases like rerunning the same month etc.

[nichannah]

This has been fixed and a test added which checks dates for a full IAF 5 cycle run.

[aekiss]

Reopening - see COSIMA/cice5#38 (comment)

[aekiss]

For the record, here's how I set up a new cycle for 01deg_jra55v140_iaf_cycle2 to start from the end of 01deg_jra55v140_iaf, based on https://github.com/COSIMA/access-om2/wiki/Tutorials#starting-a-new-experiment-using-restarts-from-a-previous-experiment

• clone final commit of 01deg_jra55v140_iaf and make a new branch for this run:

cd ~/payu
git clone --no-hardlinks 01deg_jra55v140_iaf 01deg_jra55v140_iaf_cycle2
cd ~/payu/01deg_jra55v140_iaf_cycle2
git checkout 01deg_jra55v140_iaf
git checkout -b 01deg_jra55v140_iaf_cycle2

• set SYNCDIR to
  /g/data/ik11/outputs/access-om2-01/01deg_jra55v140_iaf_cycle2

• set up initial condition, in this case the end of 01deg_jra55v140_iaf (1 Jan 2019):

cd ~/payu/01deg_jra55v140_iaf_cycle2
mkdir /scratch/x77/aek156/access-om2/archive/01deg_jra55v140_iaf_cycle2
ln -s /scratch/x77/aek156/access-om2/archive/01deg_jra55v140_iaf_cycle2 archive
cp -r /scratch/x77/aek156/access-om2/archive/01deg_jra55v140_iaf/restart243 archive/restart243
ln -s /scratch/x77/aek156/access-om2/archive/01deg_jra55v140_iaf/output243 archive/output243

Note: needed to have archive/output243/ice/cice_in.nml to avoid payu: error: Cannot find prior namelist cice_in.nml

reset run start date, in this case to 1 Jan 1958:

• edited archive/restart243/accessom2_restart.nml to have

 FORCING_CUR_DATE        = 1958-01-01T00:00:00,
 EXP_CUR_DATE    = 1958-01-01T00:00:00

• edited archive/restart243/ocean/ocean_solo.res to have

1958     1     1     0     0     0        Model start time:   year, month, day, hour, minute, second
1958     1     1     0     0     0	    Current model time: year, month, day, hour, minute, second

• set use_restart_time = .false. in ice/cice_in.nml

• did one run: payu run

• run completed successfully, so set use_restart_time = .true. in ice/cice_in.nml

• carry on experiment: payu run -n 10