add transient error handling to AMMP delete

[nojnhuh]

What type of PR is this?
/kind bug
/area managedclusters

What this PR does / why we need it: This PR adds logic to handle transient errors in the AzureManagedMachinePool controller like in other controllers. Without this, transient errors (including "operation not done") cause a resource to get requeued as if any other error occurred, which is likely more frequent than necessary.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Possibly related to the flake described in #3970

Special notes for your reviewer:

• cherry-pick candidate

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

Fixed a bug causing transient errors in AzureManagedMachinePool reconciliation to requeue too aggressively

[nojnhuh]

/cherry-pick release-1.10
/cherry-pick release-1.11

[k8s-infra-cherrypick-robot]

@nojnhuh: once the present PR merges, I will cherry-pick it on top of release-1.10 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.10
/cherry-pick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (79b63b5) 56.57% compared to head (74a59ba) 56.60%.
Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4039      +/-   ##
==========================================
+ Coverage   56.57%   56.60%   +0.02%     
==========================================
  Files         187      187              
  Lines       19130    19138       +8     
==========================================
+ Hits        10823    10833      +10     
+ Misses       7676     7674       -2     
  Partials      631      631              

Files	Coverage Δ
controllers/azuremanagedmachinepool_controller.go	56.14% <66.66%> (+1.14%)	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nojnhuh]

Role assignments issue should be ok now
/retest

[CecileRobertMichon]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: de42eaee4dc0bd8f2b7767ff958913f3fabf0f6b

[willie-yao]

/lgtm

[CecileRobertMichon]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CecileRobertMichon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CecileRobertMichon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-infra-cherrypick-robot]

@nojnhuh: #4039 failed to apply on top of branch "release-1.10":

Applying: add transient error handling to AMMP delete
Using index info to reconstruct a base tree...
M	controllers/azuremanagedmachinepool_controller.go
M	controllers/azuremanagedmachinepool_controller_test.go
Falling back to patching base and 3-way merge...
Auto-merging controllers/azuremanagedmachinepool_controller_test.go
CONFLICT (content): Merge conflict in controllers/azuremanagedmachinepool_controller_test.go
Auto-merging controllers/azuremanagedmachinepool_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 add transient error handling to AMMP delete
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.10
/cherry-pick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-infra-cherrypick-robot]

@nojnhuh: new pull request created: #4047

In response to this:

/cherry-pick release-1.10
/cherry-pick release-1.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.