Observed generation

[ChunyiLyu]

This closes #311

Note to reviewers: remember to look at the commits in this PR and consider if they can be squashed

Summary Of Changes

• Add status.observedGeneration to rabbitmqcluster
• Only set status.observedGeneration to the generation of the resource when the controller has successfully reconciled the version
• Will squash when merging

Additional Context

Local Testing

[ChunyiLyu]

Needs to make the test less flaky. Putting is down in favor of messaging topology operator, will pick up in couple days.

[Zerpet]

Two things that come to mind:

1. Do we really need to test this at system level? I'll be much more in favour of testing this at integration level with testEnv if it's possible.
2. Does it make sense to add a test for observed generation transitions/increments?
   It might be a bit in the edge between testing Kubernetes and testing our code, but I'd make an argument to say that we want to test that any update to the generation in metadata is reflected in the status of RabbitMQ.

[ChunyiLyu]

1. Do we really need to test this at system level? I'll be much more in favour of testing this at integration level with testEnv if it's possible.

Sadly you cannot... Observed generation is set only after a successful reconcile. Our integration tests controller never successfully exist because no pods ever becomes ready. It's the same as the status ReconcileSuccess where it can only be tested in an actual k8s cluster :(((((

2. Does it make sense to add a test for observed generation transitions/increments?

Assuming that k8s api is updating the metadata.generation successfully (which is not in scope for our operator), the system test is testing that observedGeneration is set to a correct value: https://github.com/rabbitmq/cluster-operator/pull/637/files#diff-b18e65a50201ede1c97f44776a33955bd5c2dad93e055102833ddebe07fa647bR114. It's enough imo because we are not incrementing the counter ourselves, but rather setting it to the value of the object generation.