Unable to upgrade the collector using operator before v0.52.0 to the newer version

[pureklkl]

When try to upgrade the collector deployed by operator < 0.52 to a newer version, I met
selector` does not match template `labels

#849 defined a set of static selector labels to fix that mutating the selectors will fail the upgrade reconciliation.

However, it will only work for the versions that are after the fix. I want to upgrade a deployment which is before the fix, to the latest.
It will still fail, because the selector remains unchanged during the reconciliation, i.e. the old selector which included app.kubernetes.io/version got inherited and the new collector labels simply won't match.
https://github.com/open-telemetry/opentelemetry-operator/blob/main/pkg/collector/reconcile/daemonset.go#L93

[pureklkl]

keycloak/keycloak-operator#567
Similar scenario happened for keycloak, their solution is to delete the old deployment and recreate a new one when the immutable field need to be changed.
I would like to implement it in OT operator if possible.

[pavolloffay]

Similar scenario happened for keycloak, their solution is to delete the old deployment and recreate a new one when the immutable field need to be changed.

IIRC this was recommended here in the past as well.

I would like to implement it in OT operator if possible.

PRs are welcome to fix it.