e2e: Test more complicated flows

[nirs]

We test only very basic flow:

1. Deploy workload
2. Protect
3. Failover
4. Relocate
5. Unprotect
6. Undeploy

This does not cover enough code, hiding issues. For example if a PVC is deleted by mistake when unprotecting the workload, the workload will continue to run normally (with pvc in terminating state), but we cannot protect the workload again.

We need to cover more flows:

• failover after failover (failure of secondary cluster after primary cluster failed)
• relocate after relocate (planned relocate to secondary planned relocate back to primary)
• failover after relocate (handling second failure in after recovering from failure)
• Protecting after unprotect (used when replacing a cluster)

We can enhance the basic flow, or create an alternative flows. A flow or multiple flows testing all possible combinations may be too slow to run for every PR.

[ShyamsundarR]

The e2e intention is to run an exhaustive matrix of states, even if that takes too much time per PR. This is to capture any breakage in the various state transitions overall. So overall I disagree with the end statement around slowness above.

The intention of Exhaustive was to add more flows into it as a matrix to exhaustively test the state changes.

The intention was also to provide for a Simple flow, that has the basic states, and runs through them once, or one can step though them in any valid order. This is still pending.

So creating alternative flows as of now would not be useful. Adding more to the current Exhaustive till we arrive at a full state transition maybe useful at present.