Metric last_state persists after ExternalSecret deleted

[EricHorst]

Great new metric last_state, thanks!

We now have a monitoring rule is in place to watch for failed sync. A sync was failing so I deleted the ExternalSecret. However, last_state does not forget and continues to report -1 on the ExternalSecret that no longer exists. It continues to report success on deleted objects as well.

Clearly this is bad and last_state should stop reporting ExternalSecrets status when they go away.

[cep21]

Would love a fix for this too :)

The line is here:

https://github.com/godaddy/kubernetes-external-secrets/blob/af9147548147a011230850f57c98555c0bbd9325/lib/metrics.js#L33

Once the gauge is set, it needs to be explicitly unset or removed when the external secret is removed. I think observeSync needs a similar "not monitored anymore" callback. I think prom-client remove will cause the metric to stop being reported https://github.com/siimon/prom-client/blob/8d69d1f615bd0f82518292720168c368937fb0dc/lib/gauge.js#L114

One workaround is to restart the external secret daemon when you see the bad gauge.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.

[cep21]

This bug is still around

[loreleimccollum-work]

Yeah would love to see a fix for this, its impacting the monitoring of our secrets

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.

[druesendieb]

This still haunts our monitoring checks - could you please look into this?

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[Flydiverny]

While this is still unfortunate please see #864

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.