Make sure restart works correctly

[szy21]

We should make sure the simulation results are identical when running from a restart file and when running from a cold start (freshly). It would be good if we can add a test for this in ci. cc @charleskawczynski

Some thoughts from @Sbozzolo :

• How are the new callback initialized? For example, if you have a callback called every N/dt, does it pick up from the old ones has left or is a new counter?
• We have to test MPI/GPU cases as well
• What happens when you change settings across restarts?
• Accmulated diagnostics will not work across restarts
• Progress log will probably be off across restarts
• Where are restart/output files saved? If we use the ActiveLink style and keep restart_dir = .../output_active, we will need to be careful because the base dir associated to output_active changes with the new simulation.

[Sbozzolo]

We should also think about what we want to do for the output. Should output from different restarts be in the same folder? Should we append NetCDF files and make very large files? Should we split into different folders?

[Sbozzolo]

Accumulated diagnostics are not saved in the HDF5 file, so we should think on what to do for them.

[Sbozzolo]

This has largely been implemented and verified, with some caveats (e.g., diagnostics having to line up with checkpoints)