Delete stale metrics on object delete

[darkowlzz]

Depends on fluxcd/pkg#612

Use the metrics helper to record all the metrics. Metrics helper ensures that the metrics for deleted objects are deleted as well.

Move all the metrics recording to be performed at the very end of the reconciliation. Realtime metrics for readiness is no longer recorded as it will be removed in a future version for CRD metrics collected using kube-state-metrics. Updating the object status with realtime readiness should provide the readiness to CRD metrics watchers.

HelmReleaseReconciler.reconcileDelete() is modified to receive a pointer HelmRelease object so that any modifications on the object is reflected on the object instance that's passed to the metrics recorder. This is not needed for HelmReleaseReconciler.reconcile() as it returns a new copy of the object that's saved in the same object variable, overwriting the object instance with the updates.

Before this change, the following metrics continued to be exported even after the associated object is deleted for HelmRelease:

gotk_reconcile_condition{kind="HelmRelease",name="podinfo",namespace="default",status="False",type="Ready"} 0
gotk_reconcile_condition{kind="HelmRelease",name="podinfo",namespace="default",status="True",type="Ready"} 1
gotk_reconcile_condition{kind="HelmRelease",name="podinfo",namespace="default",status="Unknown",type="Ready"} 0
...
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.01"} 0
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.038363583488692544"} 1
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.1471764538093883"} 1
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="0.5646216173286169"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="2.166090855590701"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="8.309900738254731"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="31.879757075478317"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="122.30217221643493"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="469.19495946736544"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="1799.9999999999986"} 2
gotk_reconcile_duration_seconds_bucket{kind="HelmRelease",name="podinfo",namespace="default",le="+Inf"} 2
gotk_reconcile_duration_seconds_sum{kind="HelmRelease",name="podinfo",namespace="default"} 0.227111736
...
gotk_suspend_status{kind="HelmRelease",name="podinfo",namespace="default"} 0

With this change, they get deleted once the associated object is deleted.

Also, the metrics helper no longer exports ConditionDelete for readiness metrics.