Detect unexpected field managers

[c-pius]

As aligned in kyma-project/community#953, we eventually want to revert changes to resources we manage by unexpected field managers.
Before actually reverting the changes, we agreed to first implement a detection of unexpected field managers. We want to collect data about:

• what objects/resources have unexpected field managers
• what part of the object they manage
  • if this part is exclusive to the unexpected manager

The approach for how to detect the unexpected managers can be derived from the ADR (see Option 2). To keep it simple in the beginning without the need to rename managers, we can first start with an allow list instead of the naming pattern. This allow list should include:

• the values KLM uses (values are differing between the reconcile loops)
• the values K8s uses (e.g., k3d or kube-controller-manager values for the status)

Reason

See context of linked ADR.

AC

• determined approach how to collect the needed data
  • e.g., where to collect them (existing control loops or separate, e.g., via Job)
  • e.g., where to write them (metrics, local file that can be extracted, ...)
• data collection is happening on an opt-in basis
  • e.g., when a certain label is set to Kyma/Manifest
• collected data includes
  • all objects managed by lifecycle-manager
  • indicator whether the object as other field managers (except the known ones like from k3d/k8s)
  • name(s) of the unexpected field manager(s)
  • fields that the unexpected field manager manages
  • indicator whether lifecycle-manager is also managing those

[Tomasz-Smelcerz-SAP]

We have two groups of objects to analyze:

• Manifest (kyma module) related objects
• kyma-system namespace, the kyma object itself and skr-webhook (and possibly others). These are not related to any single kyma module (Manifest)

[Tomasz-Smelcerz-SAP]

How to expose the collected data: After considering different options for "storage", we decided to just use cloud logging (stdout) as the storage mechanism.
Other alternatives:

• Local files: Easy to write, hard to read - we'd have to extend Lifecycle Manager with a HTTP endpoint that exposes these files (normally we don't have permission to exec to a Pod or copy files from it)
• ConfigMaps: It's against the best practices: etcd is a configuration database, not a storage mechanism. Of course for limited amout of data it "would work", but we shouldn't go for such solution if other options are available.
• Redis: There is a redis instance available in KCP. A bit better than local files (built-in TTL), but also requires adding a HTTP endpoint to expose stored data.