Upgrade of bookkeeper cluster is failing intermittently

[anishakj]

Description

Upgrade of bookkeeper cluster is failing intermittently. In operator logs, seeing below output

time="2022-06-23T13:56:25Z" level=info msg="syncing cluster version from 4.14.3-8-de07afb7f to 4.14.3-13-378101a"
time="2022-06-23T13:56:25Z" level=info msg="Reconciling BookkeeperCluster store-ns/openshift-upgrade-bk\n"
time="2022-06-23T13:56:25Z" level=info msg="updating statefulset (openshift-upgrade-bk-bookie) template image to 'asdrepo.isus.emc.com:8099/bookkeeper:4.14.3-13-378101a'"
time="2022-06-23T13:56:25Z" level=info msg="failed to reconcile bookkeeper cluster (openshift-upgrade-bk): failed to reconcile cluster status: failed to update cluster status: Operation cannot be fulfilled on bookkeeperclusters.bookkeeper.pravega.io \"openshift-upgrade-bk\": the object has been modified; please apply your changes to the latest version and try again"
{"level":"error","ts":1655992585.8642533,"logger":"controller.bookkeepercluster","msg":"Reconciler error","reconciler group":"bookkeeper.pravega.io","reconciler kind":"BookkeeperCluster","name":"openshift-upgrade-bk","namespace":"store-ns","error":"failed to reconcile cluster status: failed to update cluster status: Operation cannot be fulfilled on bookkeeperclusters.bookkeeper.pravega.io \"openshift-upgrade-bk\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
time="2022-06-23T13:56:25Z" level=info msg="Reconciling BookkeeperCluster store-ns/openshift-upgrade-bk\n"
time="2022-06-23T13:56:25Z" level=info msg="statefulset (openshift-upgrade-bk-bookie) status: 4 updated, 4 ready, 4 target"
2022/06/23 13:56:25 Updating version history adding version 4.14.3-13-378101a

Importance

must-have

Location

upgrade.go

Suggestions for an improvement

Handle upgrade in this scenario as well