Improved pipeline resource cleanup

[rakesh-garimella]

Description
Currently we dont have a proper decision on restarting the services for logging, trace and metrics for corner cases. Eg. see below the corner cases

Logging

When there is a logpipeline which uses a secret and if the secret is deleted then in the reconciliation we remove this logpipeline from the sections configmap and restart the fluentbit. Currently we cannot delete the fluentbit daemonset because the current decision making does not allow deleting the daemonset if the logpipeline is inpending state

tracing/metrics

Same scenario with trace pipeline we return when dont have a deployable pipelines. Thus we dont update the config and we keep running with config which might not work eventually (as the secret has beed deleted).

Same is the case with metrics as well.

Expected result

When we have logpipeline/tracepipeline/metricpipeline in pending state (which was running but moved to pending because secret has been deleted) then we should not create resources.

Ideally we should think something like this

if (deployablePipelines>0) {
  for i := range deployablePipelines {
    generateConfig()
  }
  deployOrRestartCollector()
} else {
  deleteCollector()
}

[a-thaler]

There was following situation on one cluster which needs to get addressed in the same run:

there was a healthy tracepieline till the referenced secret got deleted. The pipeline false in pending state, the collector and config stays unchanged and continues to work healthy.
A new version rollout of the operator and collector is not getting applied in that situation, the collector stays in the old version which is a serious bug

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs.
Thank you for your contributions.