incorrect initialisation for age_global

[aekiss]

/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3/output488/ocean/ocean-2d-surface_pot_temp-1-monthly-min-ym_1958_01.nc
has a maximum of 273.15K for January 1958

but Feb 1958 (and all other dates I've looked at) seem OK

This file is from the start of a new cycle, so it looks like an initial value of 0C is used for calculating the minimum. This should be replaced by an unphysically large value (maybe +Inf if we want something that applies regardless of the diagnostic).

Thanks to @russfiedler for pointing this out.

[aekiss]

Russ also commented

After the first run it seems to sort itself out. I note that there was a change in outputting surface conservative temp for cycle 2 and PT for cycle 3. Maybe the diagnostic temp (PT) restart file wasn't handled correctly across cycles. It might also be related to this bfd21d0

...so if that's the case, the diagnostic is correct, and it's the initial condtion that's incorrect

[aekiss]

This issue doesn't occur at the start of cycle 2
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle2/output244/ocean/ocean-2d-surface_temp-1-monthly-min-ym_1958_01.nc
so there's definitely something fishy about the cycle2-3 transition

[aekiss]

These min and max dailies from the start of cycle 3 show this pot_temp issue affects only the first time sample
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3/output488/ocean/ocean-2d-surface_pot_temp-1-daily-min-ym_1958_01.nc
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3/output488/ocean/ocean-2d-surface_pot_temp-1-daily-max-ym_1958_01.nc

And there seems to be no problem with daily max bottom conservative temperature
/g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3/output488/ocean/ocean-2d-bottom_temp-1-daily-max-ym_1958_01.nc

So this is all consistent with @russfiedler's suggestion

there was a change in outputting surface conservative temp for cycle 2 and PT for cycle 3. Maybe the diagnostic temp (PT) restart file wasn't handled correctly across cycles

[russfiedler]

@aekiss An examination of the log files shows that the diagnostic temperature (pot_temp) is being set to zero for the start of new cycles. The same goes for frazil and age. In /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle2/output244/access-om2.out and /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf_cycle3/output488/access-om2.out we see the following

Initializing tracer number 1 at time level tau. This tracer is called pot_temp Initializing diagnostic tracer pot_temp to constant 0.000000000000000E+000

This is because MOM thinks you are at the start of a new experiment and the field_table isn't telling it not to start with a constant initial value (default). In other words we need an entry in the field table to override the value of const_init_tracer

I think an entry like

`"diag_tracers","ocean_mod","pot_temp"
const_init_tracer = .false.
/

is required.

[russfiedler]

Note the logic for reading a restart file https://github.com/mom-ocean/MOM5/blob/master/src/mom5/ocean_tracers/ocean_tracer.F90#L2121-L2128

[aekiss]

Aha, thanks @russfiedler, sounds like that will fix this issue, and also COSIMA/access-om2#235

I guess from the code that we should leave field_table as it is for the standard configurations so they can do a cold start, but if we want to start a 2nd cycle we would need to remember to change it.
Or maybe it would be more gotcha-proof to set field_table to read a file, and make some restarts with the default initial condition?

[russfiedler]

I wouldn't recommend that if you're outputting any diagnostics wrt diagnostic tracers unless you know them to be zero.
There's no real reason for not supplying zero tracer restart files to start with. They compress to nothing.

[aekiss]

OK thanks, I'll do the second option then.

[aekiss]

Hi @russfiedler - just getting back to this for cycle 4. Is this what you'd recommend?
COSIMA/01deg_jra55_iaf@96876c6
not sure if I need to specify restart_file.

[russfiedler]

@aekiss That looks sensible to me.

[aekiss]

Hi @russfiedler

The changes in COSIMA/01deg_jra55_iaf@ae32f24 fixed the initialization of frazil and pot_temp, but age is still being initialized to zero.

My cycle 4 test run here
/home/156/aek156/payu/01deg_jra55v140_iaf_cycle4/archive/output732
uses this initial condition from the end of cycle 3, which (as expected) has a maximum of 61 years:
/scratch/v45/aek156/access-om2/archive/01deg_jra55v140_iaf_cycle4/restart731/ocean/ocean_age.res.nc
and the log from the test run
/scratch/v45/aek156/access-om2/archive/01deg_jra55v140_iaf_cycle4/output732/access-om2.out
says

Initializing tracer number          13
 at time level tau. This tracer is called age_global
NOTE from PE     0: GLOBAL ATT too long - not reading this metadata
Reading restart for prog tracer age_global from file ocean_age.res.nc
NOTE from PE     0: GLOBAL ATT too long - not reading this metadata
After reading ic, linearly interpolate age_global to partial cell bottom.
Completed initialization of tracer age_global at time level tau

so that all looks OK, but the output file from the test run
/scratch/v45/aek156/access-om2/archive/01deg_jra55v140_iaf_cycle4/output732/ocean/ocean-3d-age_global-1-monthly-mean-ym_1958_01.nc
has maximum age 0.05yr, clearly not picking up the initial condition.

I'm a bit stumped. Am I missing something obvious?

[aekiss]

The problem was that I was using

&ocean_tracer_nml
     age_tracer_max_init = 0.0
     limit_age_tracer  = .true.

which limits the age to at most the run length of the cycle plus age_tracer_max_init. It also ensures age is non-negative.

I tried using limit_age_tracer = .false.. This fixes the initialisation of the age (ie it starts with a maximum of 61 years, as expected), but unfortunately the numerics readily generates negative values: the minimum age after a 1 month run is -0.82965 years (unclear where this occurs, but it isn't widespread). The input file has no negative values, so this is definitely due to the numerics. This occurs with these settings in field_table:

"tracer_packages","ocean_mod","ocean_age_tracer"
names = global
const_init_tracer = .false.
horizontal-advection-scheme = mdppm
vertical-advection-scheme = mdppm
ppm_hlimiter = 3
ppm_vlimiter = 3
restart_file = ocean_age.res.nc
min_tracer_limit=0.0
/

A better solution was to use

&ocean_tracer_nml
     age_tracer_max_init = 61.0
     limit_age_tracer  = .true.

This worked as expected, with 0<= age <= 61.042yr (61yr+0.5mo) after 1 mo (as 1mo average).
The only awkwardness with this solution is that age_tracer_max_init needs to be updated for each cycle to reflect the total run length.