Fix Persistence.Healthcheck probe stall because of failed warmup

[Arkatufus]

Persistence.Healthcheck will stall if the first warmup cycle failed.

• Warmup cycle need to be extended every time snapshot save or journal persist fail until both succeeded because recovery will never be defined until both succeeded for the first time.
• Recovery status should not be a required check to declare that the persistence probe is a success.

NOTE: this would mean that if snapshot save and journal persist never succeed, the probe will be stuck in a perpetual warmup state because recovery would never be defined

[Aaronontheweb]

Left some questions

[Aaronontheweb]

So this value goes up each time the warmup probe recovers without knowing whether or not the two persist operations have both completed successfully?

[Arkatufus]

Warmup is a property we pass back to the probe that the suicide actor is in "first attempt" mode, if any of the snapshot save or journal persist failed, we extend the "first attempt" time to the next cycle until both succeed.

This is because the liveness status does not allow for undefined/null recovery status for the probe to detect if recovery was performed on both snapshot and journal.

Maybe we should change the recovery properties of the persistence liveness status to be nullable instead?

[Aaronontheweb]

Maybe we should change the recovery properties of the persistence liveness status to be nullable instead?

Yes, if there's a better / more intuitive way of modeling this we should do that instead.