NotManagedByMCM annotation incorrectly added by MCM to Node

[elankath]

How to categorize this issue?

/area quality
/kind bug
/priority 2

What happened:
Node associated with machines that are managed by the machine-controller are being annotated with the NotManagedByMCM (node.machine.sapcloud.io/not-managed-by-mcm="1") annotation. This can occur plane migration when the Node creation time stamps predate the Machine creation time stamps after control plane migration.

kubectl get no -o custom-columns="NAME":.metadata.name,"CREATION":.metadata.creationTimestamp
NAME                                                        CREATION
shoot--di-xm8--dh-tt23jtpe4f2f-static-dynamic-76487-89zwb   2023-08-16T16:42:15Z
shoot--di-xm8--dh-tt23jtpe4f2f-system-57c59-g2f8q           2023-08-16T16:42:17Z
shoot--di-xm8--dh-tt23jtpe4f2f-system-57c59-k9jwh           2023-08-16T16:42:13Z

k get mc  -o custom-columns="NAME":.metadata.name,"CREATION":.metadata.creationTimestamp
NAME                                                        CREATION
shoot--di-xm8--dh-tt23jtpe4f2f-static-dynamic-76487-89zwb   2023-10-22T05:56:53Z
shoot--di-xm8--dh-tt23jtpe4f2f-system-57c59-g2f8q           2023-10-22T05:56:53Z
shoot--di-xm8--dh-tt23jtpe4f2f-system-57c59-k9jwh           2023-10-22T05:56:53Z

What you expected to happen:
Ideally, Nodes should not be incorrectly tagged with this annotation. If they have this annotation, the annotation must be removed. A proper fix is described at #774 . Unfortunately, the solution proposed there will take more time since now even gardener code shoot health check is relying on the process of this NotManagedByMCM annotation.

Solution Proposal

1. In AnnotateNodesUnmanagedByMCM (invoked by the safety reconciler), remove annotation if the Node is already associated with Machine , yet has this annotation set.
2. Fix needs vendoring and release in all providers.
3. Current default period of safety reconciler job is 30m. Consider reducing to 5m after checking that all calls go through informer cache.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:
all mcm versions

• Kubernetes version (use kubectl version): all
• Cloud provider or hardware configuration: all
• Others:

What you expected to happen:

[gardener-robot]

@elankath You have mentioned internal references in the public. Please check.

[gardener-robot]

@elankath You have mentioned internal references in the public. Please check.