Restarting cilium daemonset on upgrade workflow to address upgrade issue

[maxdrib]

Issue #, if available:
#1888

Description of changes:
This PR restarts the cilium pods in a CloudStack upgrade workflow in order to prevent the issues observed in #1888. We are still waiting to hear what the root cause may be from Isovalent but expect this change to resolve the majority of the issues.

Will create issue to make the logic non CloudStack specific

Testing (if applicable):
Unit tests. Executed the TestCloudStackKubernetes120RedhatTo121Upgrade locally several times and observed the networking issues recover during the upgrade

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

Merging #2057 (7418d0f) into main (dc7e82e) will decrease coverage by 0.19%.
The diff coverage is 82.35%.

@@            Coverage Diff             @@
##             main    #2057      +/-   ##
==========================================
- Coverage   49.81%   49.61%   -0.20%     
==========================================
  Files         306      308       +2     
  Lines       25810    25911     +101     
==========================================
  Hits        12856    12856              
- Misses      11632    11734     +102     
+ Partials     1322     1321       -1     

Impacted Files	Coverage Δ
pkg/clustermanager/cluster_manager.go	53.50% <57.14%> (+0.03%)	⬆️
pkg/executables/kubectl.go	37.81% <100.00%> (+0.57%)	⬆️
pkg/curatedpackages/curatedpackages.go	26.73% <0.00%> (-35.77%)	⬇️
pkg/curatedpackages/bundle.go	62.19% <0.00%> (-11.72%)	⬇️
pkg/curatedpackages/packageclient.go	76.37% <0.00%> (-0.55%)	⬇️
pkg/manifests/reader.go	50.00% <0.00%> (ø)
pkg/curatedpackages/packagecontrollerclient.go	100.00% <0.00%> (ø)
...l-anywhere/cmd/internal/commands/artifacts/noop.go	0.00% <0.00%> (ø)
pkg/curatedpackages/reader.go	0.00% <0.00%> (ø)
pkg/dependencies/factory.go	49.75% <0.00%> (+0.08%)	⬆️
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc7e82e...7418d0f. Read the comment docs.

[vivek-koppuru]

We should change the unit tests to not include cilium also right?

[vivek-koppuru]

/approve

[eks-distro-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vivek-koppuru

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vivek-koppuru]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[maxdrib]

/cherrypick release-0.9

[eks-distro-pr-bot]

@maxdrib: new pull request created: #2068

In response to this:

/cherrypick release-0.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.