AWSMachine metadata can exceed safe sizes due to cluster-api-provider-aws-last-applied-tags values

[cnmcavoy]

/kind bug

What steps did you take and what happened:
AWSMachine annotation values for sigs.k8s.io/cluster-api-provider-aws-last-applied-tags seem to accumulate and never be cleared or reset when volumes are removed for a machine. This results in problems for longer-lived awsmachines, especially if they are used in a context with frequent volume churn (Gitlab CI), because this annotation value can become excessively large. We have several dozen AWSMachines where this value exceeds 20k length:

kubectl --cluster awscmhinfra2 -n capi-awscmhinfra2 get -o json awsmachine | jq -r '.items[] | select(.metadata.annotations | to_entries | any(.value | length > 20000)) | .metadata.name' | wc -l
28

Today we noticed three cases where the value exceeded 250k length, which prevented the resources from being converted to older versions by the conversion webhook:

kubectl get awsmachine.v1beta1.infrastructure.cluster.x-k8s.io worker-private-m6-mixed-az1-0a772868-9b8lf -oyaml -n capi-awscmhinfra2

Error from server: conversion webhook for infrastructure.cluster.x-k8s.io/v1beta2, Kind=AWSMachine returned invalid metadata: metadata.annotation: Too long: must have at most 262144 bytes

What did you expect to happen:
The number of elements stored in the sigs.k8s.io/cluster-api-provider-aws-last-applied-tags should match the number of volumes the corresponding node resource has attached. On one of the problematic AWSMachine resources, there were 1907 distinct volumes ids saved in the annotation values, but only 2 volumes present on the node.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version: 2.2.1
• Kubernetes version: (use kubectl version): 1.24.14
• OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS

[Ankitasw]

/triage accepted
/priority important-soon