Change MPAS-Ocean high-frequency output mode to append

[xylar]

Without this, the first output is missing after each restart

fixes #6432

[xylar]

@zhangshixuan1987 and @wlin7, can you verify that this fixes #6432? I am happy to do some testing but I think you are in a better position to verify this.

[xylar]

Note: an alternative fix would be to change:
https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/cime_config/buildnml#L862
to append instead of truncate. This may be the preferred solution, since we're not 100% sure that the fields typically included in the high frequency stream are available after a restart.

[zhangshixuan1987]

Note: an alternative fix would be to change: https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/cime_config/buildnml#L862 to append instead of truncate. This may be the preferred solution, since we're not 100% sure that the fields typically included in the high frequency stream are available after a restart.

Hi @xylar: I conducted a set of 2-month short-term simulations on Compy with the code changes you suggested:

• The code for this simulation is master branch on March 21 2024 (commit 2cfa5f184848e9de083d71a7c670a8af8efcfde4).
• Sensitivity experiments were conducted without (/compyfs/zhan391/e3sm_project/E3SMv3_testings/20240521.v3.default.WCYCL1850.ne30pg2_r05_IcoswISC30E3r5.compy) and with (see /compyfs/zhan391/e3sm_project/E3SMv3_testings/20240521.v3.f6434.WCYCL1850.ne30pg2_r05_IcoswISC30E3r5.compy)) the changes in the buildnml suggested above.

The changes here indeed solved the issues:

• Without change: ncdump -t -v xtime 20240521.v3.default.WCYCL1850.ne30pg2_r05_IcoswISC30E3r5.compy.0161-01-01_00.00.00.nc

data:

xtime =
 "0161-01-06_00:00:00",
 "0161-01-11_00:00:00",
 "0161-01-16_00:00:00",
 "0161-01-21_00:00:00",
 "0161-01-26_00:00:00",
 "0161-01-31_00:00:00" ;

• With change: ncdump -t -v xtime 20240521.v3.f6434.WCYCL1850.ne30pg2_r05_IcoswISC30E3r5.compy.mpaso.hist.am.highFrequencyOutput.0161-01-01_00.00.00.nc

data:
xtime =
 "0161-01-01_00:00:00",
 "0161-01-06_00:00:00",
 "0161-01-11_00:00:00",
 "0161-01-16_00:00:00",
 "0161-01-21_00:00:00",
 "0161-01-26_00:00:00",
 "0161-01-31_00:00:00" ;

[xylar]

Those error messages are annoying for sure but ultimately harmless I think. Did you make both changes? compute/write on startup and append? If so, you only want append and not compute/write on startup.

[xylar]

Do the high frequency files have all the expected time entries?

[zhangshixuan1987]

Do the high frequency files have all the expected time entries?

Hi @xylar : Yes, I think that this change can solve the issues I originally encountered. However, when I employ this solution in a new G-case simulation, it seems to me that it can potentially trigger a new potential issue for the following scenario:

• When you restart the model to rerun a piece of simulation. For example, my simulation stopped at 0199-03-01_00.00.00.nc due to hitting the job walltime limit. So I restarted the simulation at 0199-01-01_00.00.00.nc to continue my simulation. Note that previous run already generated files at 0199-02-01_00.00.00 and 0199-03-01_00.00.00. The restart simulation are supposed to overwrite these files (as we restart the model at 0199-01-01_00.00.00). However, using append instead of truncate triggers the following error from mpaso:

ERROR: Writing to stream 'highFrequencyOutput' would overwrite record    1 in file '20240524.v3-LR.GMPAS-BC.ne30pg2_r05_icos30.anvil.mpaso.hist.am.high-frequency output.0199-02-01_00.00.00.nc',
ERROR:     but clobber_mode is set to 'append'.

I think this error is expected from the purpose of append (not allowed to overwrite) and truncate (allowed to overwrite). I believe that truncate is what we want in this scenario. Overall, my point is that truncate seems to be a better choice instead of append for most of the cases.

[zhangshixuan1987]

Those error messages are annoying for sure but ultimately harmless I think. Did you make both changes? compute/write on startup and append? If so, you only want append and not compute/write on startup.

Hi @xylar : Actually this is not the case. For the test simulation as I mentioned above, I ONLY made changes for https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/cime_config/buildnml#L862. No other changes have been made.

[zhangshixuan1987]

@xylar: I think I overlooked the changes in this pull request. I thought that #6434 and #6432 were different. In fact, the pull request along with the changes here is indeed the correct one. To avoid confusion, I would like to clarify:

1. changing the two flags in mpas-ocean/bld/namelist_files/namelist_defaults_mpaso.xml as suggested in Potential bug in MPAS ocean output  #6432 is the solution
2. the alternative fix with changes in https://github.com/E3SM-Project/E3SM/blob/master/components/mpas-ocean/cime_config/buildnml#L862 is NOT a preferred solution to me.

Thank you!

[xylar]

I believe that append mode is functioning as I should, and that one simply has to ignore the disconcerting error messages if one is restarting from an earlier point and cannot overwrite the data.

[xylar]

I think the append mode change remains the preferred solution despite the annoying error messages. In general, we cannot guarantee that fields selected for output are available at startup. Fields may contain all zeros. So it is safer to keep the results written before the last run wrote its final restart than to overwrite with new values after restart.

You all may use the other solution of computing/writing at startup for the alternating B- and G-cases but that won't be what we ultimately do in E3SM.

[xylar]

@wlin7, @zhangshixuan1987 and @mark-petersen, as discussed above, I'm going with the "alternative" method of fixing this issue where the output is in "append" mode rather than "truncate". This has the annoying side effect that error messages may appear after each restart (particularly if a run timed out) saying that entries could not be overwritten because the mode is "append". This was the preferred fix in our MPAS DevOps call last week.

[mark-petersen]

Yes, I agree. The inconvenience of extra messages from a bad restart is preferred to having a time slice missing. Thanks for everyone's detailed review and testing above.

[xylar]

The inconvenience of extra messages from a bad restart is preferred to having a time slice missing. Thanks for everyone's detailed review and testing above.

@mark-petersen, do you also agree that there's some risk in setting things to compute/write at startup? That worked for what @zhangshixuan1987 and @wlin7 need for their purposes but I think there are enough fields that start out as all zeros until after the first time step of a restart run, so that the solution I now have in this branch is preferable. But I would be keen to make sure you agree.

[xylar]

@wlin7, I'm taking you off as a reviewer. I would still welcome your input if you want to provide it but I don't think it is necessary before we merge this.

[rljacob]

@jonbob please start merging this.

[jonbob]

Passes:

• ERS_Ld5.T62_oQU120.CMPASO-NYF.chrysalis_intel
• ERP_Ld3.ne30pg2_r05_IcoswISC30E3r5.WCYCL1850.chrysalis_intel.allactive-pioroot1

merged to next

[jonbob]

merged to master

[xylar]

Thank you, @jonbob!