charts(osm-*): add pod and node affinities to control plane pods

[SanyaKochhar]

Description:

• Adds pod anti-affinities to osm-controller, osm-bootstrap,
  osm-injector to ensure better resilience for pod distribution
  across nodes
• Replaces NodeSelectors with NodeAffinities for consistency

Testing done:
Pod anti-affinities:

• without anti-affinities: >= 2 replicas may or may not schedule on different nodes
• with anti-affinities: >= 2 replicas schedule on different nodes if possible

NodeAffinities:

• verified that a pod whose label values do not match node's label values fails to schedule
• verified that a node that does not have label for a pod's matchExpression causes the pod to fail to schedule

Affected area:

Functional Area
New Functionality	[x]

Please answer the following questions with yes/no.

1. Does this change contain code from or inspired by another project? no

   • Did you notify the maintainers and provide attribution?

2. Is this a breaking change? no

3. Has documentation corresponding to this change been updated in the osm-docs repo (if applicable)?
   yes, PR open here: guides(ha): add note about pod anti-affinities osm-docs#330

[codecov-commenter]

Codecov Report

Merging #4527 (20e5484) into main (c3b6739) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #4527   +/-   ##
=======================================
  Coverage   69.29%   69.29%           
=======================================
  Files         216      216           
  Lines       14709    14709           
=======================================
  Hits        10192    10192           
  Misses       4466     4466           
  Partials       51       51           

Flag	Coverage Δ
unittests	69.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3b6739...20e5484. Read the comment docs.

[jaellio]

#4496 is a related issue to consider (perhaps in separate PR)

[SanyaKochhar]

#4496 is a related issue to consider (perhaps in separate PR)

yeah, I agree this could be addressed in a separate PR. The main objective of this PR is to add pod anti-affinities for better resilience of distribution of replicas across nodes when HPA is enabled, the nodeselector -> affinity change is more for consistency