Short term archiving removing MPAS restart files

[rljacob]

Issue #1252 is nearly solved. But testing with some different restart frequencies is showing problems.

Test 1: low-res fully coupled ACME config.
Run for 1 year with restart and end of 1 year.
Everything works fine. See https://gist.github.com/rljacob/ca750c3d4a2ea262db5b728b78e59acc

Test 2: same case but run for 2 years and keep the restart frequency the same.
The interim restarts have the string "0002-01-01" while the final restarts have "0003-01-01".

The archiver deleted both sets for the active MPAS components, thinking they were all interim:

-------------------------------------------
doing short term archiving for mpaso (ocn)
-------------------------------------------
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsDaily.0003-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsCustom.0003-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.0002-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsClimatology.0003-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsClimatology.0002-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsMonthly.0003-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsDaily.0002-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsMonthly.0002-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.0003-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.rst.am.timeSeriesStatsCustom.0002-01-01_00000.nc
histfiles_savein_rundir []
moving
/lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpaso.hist.am.globalStats.0003-01-01.nc to
/lcrc/group/acme/jacob/archive/Wlowrmaster/ocn/hist/mpaso.hist.am.globalStats.0003-01-01.nc

and also

-------------------------------------------
doing short term archiving for mpascice (ice)
-------------------------------------------
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpascice.rst.0003-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpascice.rst.0002-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpascice.rst.am.timeSeriesStatsMonthly.0002-01-01_00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/mpascice.rst.am.timeSeriesStatsMonthly.0003-01-01_00000.nc

[rljacob]

Compare to land model which does the right thing:

-------------------------------------------
doing short term archiving for clm (lnd)
-------------------------------------------
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/Wlowrmaster.clm2.r.0002-01-01-00000.nc
removing interim restart file /lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/Wlowrmaster.clm2.rh0.0002-01-01-00000.nc
copying
/lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/Wlowrmaster.clm2.r.0003-01-01-00000.nc to
/lcrc/group/acme/jacob/archive/Wlowrmaster/rest/0003-01-01-00000/Wlowrmaster.clm2.r.0003-01-01-00000.nc
copying
/lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/Wlowrmaster.clm2.h0.0002-12.nc to
/lcrc/group/acme/jacob/archive/Wlowrmaster/rest/0003-01-01-00000/Wlowrmaster.clm2.h0.0002-12.nc
copying
/lcrc/group/acme/jacob/acme_scratch/Wlowrmaster/run/Wlowrmaster.clm2.rh0.0003-01-01-00000.nc to
/lcrc/group/acme/jacob/archive/Wlowrmaster/rest/0003-01-01-00000/Wlowrmaster.clm2.rh0.0003-01-01-00000.nc

[rljacob]

Reporting here but we need to fix/test within ACME.

[rljacob]

Removed "bug" because this isn't a problem with code on master, just my (ACME) branch.

[rljacob]

This has been fixed on the ACME side. Will be brought in the next time we subtree split from ACME.