Restart reproducibility tests are failing

[aidanheerdegen]

The tests for reproducibility across restarts are currently not being run in the CI tests because they fail.

https://github.com/ACCESS-NRI/access-om2-configs/blob/main/test/test_bit_reproducibility.py#L78

The original COSIMA ACCESS-OM2 tests from which these were adapted used reproducibility flags turned on for their builds

https://github.com/COSIMA/access-om2/blob/master/test/exp_test_helper.py#L217-L218

though this was not used in production builds

https://github.com/COSIMA/access-om2/blob/master/install.sh#L49

We need a repro variant for the spack package to test if we can replicate the COSIMA reproducibility results, and see what effect that option has on performance.

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/modifying-access-om2-spack-package/1847/6

[aidanheerdegen]

We have a fix for the restart repro.

1. The first step is to add the CI restart repro checks to CI so they will fire for the next step.

2. Once there is a PR to add this to ACCESS-OM2 the prerelease deployment can be used in a PR to a config, say release-1deg_jra55_ryf.

3. Other performance tests will also be done with the pre-release PR.

4. If the CI tests pass and performance tests are acceptable the ACCESS-OM2 PR should be merged, then the release-1deg_jra55_ryf PR should be updated with the paths to the properly deployed ACCESS-OM2, then merged (assuming all tests still pass)