Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expose anti-affinity for coredns deployment #3574

Closed
TeddyAndrieux opened this issue Oct 21, 2021 · 1 comment · Fixed by #3579
Closed

Expose anti-affinity for coredns deployment #3574

TeddyAndrieux opened this issue Oct 21, 2021 · 1 comment · Fixed by #3579
Labels
complexity:medium Something that requires one or few days to fix kind:enhancement New feature or request severity:major Major impact on live deployments (e.g. some non-critical feature is not working at all) topic:deployment Bugs in or enhancements to deployment stages topic:networking Networking-related issues

Comments

@TeddyAndrieux
Copy link
Collaborator

TeddyAndrieux commented Oct 21, 2021 •
edited
Loading

Component:

'kubernetes', 'dns'

Why this is needed:

Today we deploy coredns with 2 replicas and no affinity which means that all coredns may sit on the same node, which mean no HA for the Kubernetes DNS which likely has an impact on the workload running in the cluster.

What should be done:

In order to have HA of coredns we need to make sure the coredns replica sits in different "failure zone".
But this cannot be done automatically and need to be input from the user, so we need to expose a way to change this.

We likely want to also expose the number of coredns replicas.

Implementation proposal (strongly recommended):

Add an entry in the bootstrap config for coredns so that user can input:

  • the number of replicas for coredns (default to 2)
  • the topologyKey to use for anti-affinity (soft and hard) of coredns Pods
    suggested default (TBD) (soft anti-affinity on hostname): 
     podAntiAffinity:
   preferredDuringSchedulingIgnoredDuringExecution:
   - weight: 100
     podAffinityTerm:
       labelSelector:
         matchLabels:
           k8s-app: kube-dns
       namespaces:
       - kube-system
       topologyKey: kubernetes.io/hostname

Since coredns pod will not be re-scheduled during the life of the platform in order to be sure our "soft" anti-affinity gets applied when it's possible we will need to trigger a "rollout restart" of the coredns when we expand MetalK8s cluster with a new master node.

Check kubernetes docs for more information about affinity and anti-affinity https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
@TeddyAndrieux TeddyAndrieux added kind:enhancement New feature or request topic:networking Networking-related issues topic:deployment Bugs in or enhancements to deployment stages complexity:medium Something that requires one or few days to fix severity:major Major impact on live deployments (e.g. some non-critical feature is not working at all) labels Oct 21, 2021
@TeddyAndrieux
Copy link
Collaborator Author

We need to do the same for Control Plane Ingress Controller deployment (when we use MetalLB) and for Dex

TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,tests: Add podAntiAffinity support for CoreDNS
d9eba43 
Add ability for the user to change podAntiAffinity for CoreDNS
deployment and also have a default soft podAntiAffinity on hostname so
that if it's possible each CoreDNS replica will sit on a different node
by default.
Trigger a rollout restart of CoreDNS deployment after deploying a new
master node in order to "apply" soft anti-affinity if possible

Fixes: #3574
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,tests: Add podAntiAffinity support for CoreDNS
1dad9f3 
Add ability for the user to change podAntiAffinity for CoreDNS
deployment and also have a default soft podAntiAffinity on hostname so
that if it's possible each CoreDNS replica will sit on a different node
by default.
Trigger a rollout restart of CoreDNS deployment after deploying a new
infra node in order to "apply" soft anti-affinity if possible

Fixes: #3574
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,tests,docs: Add podAntiAffinity support for CoreDNS
e416acc 
Add ability for the user to change podAntiAffinity for CoreDNS
deployment and also have a default soft podAntiAffinity on hostname so
that if it's possible each CoreDNS replica will sit on a different node
by default.
Trigger a rollout restart of CoreDNS deployment after deploying a new
infra node in order to "apply" soft anti-affinity if possible

Fixes: #3574
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,docs: Allow to manage number of CoreDNS replica
3939171 
Refs: #3574
@TeddyAndrieux TeddyAndrieux mentioned this issue Oct 26, 2021
Merged
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,tests,docs: Add podAntiAffinity support for CoreDNS
71e87a1 
Add ability for the user to change podAntiAffinity for CoreDNS
deployment and also have a default soft podAntiAffinity on hostname so
that if it's possible each CoreDNS replica will sit on a different node
by default.
Trigger a rollout restart of CoreDNS deployment after deploying a new
infra node in order to "apply" soft anti-affinity if possible

Fixes: #3574
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,docs: Allow to manage number of CoreDNS replica
cd79b6c 
Refs: #3574
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,tests,docs: Add podAntiAffinity support for CoreDNS
d8d03ee 
Add ability for the user to change podAntiAffinity for CoreDNS
deployment and also have a default soft podAntiAffinity on hostname so
that if it's possible each CoreDNS replica will sit on a different node
by default.
Trigger a rollout restart of CoreDNS deployment after deploying a new
infra node in order to "apply" soft anti-affinity if possible

Fixes: #3574
TeddyAndrieux added a commit that referenced this issue Oct 26, 2021
@TeddyAndrieux
salt,docs: Allow to manage number of CoreDNS replica
9b1a8e9 
Refs: #3574
TeddyAndrieux added a commit that referenced this issue Oct 27, 2021
@TeddyAndrieux
salt,tests,docs: Add podAntiAffinity support for CoreDNS
9e8ad55 
Add ability for the user to change podAntiAffinity for CoreDNS
deployment and also have a default soft podAntiAffinity on hostname so
that if it's possible each CoreDNS replica will sit on a different node
by default.
Trigger a rollout restart of CoreDNS deployment after deploying a new
infra node in order to "apply" soft anti-affinity if possible

Fixes: #3574
TeddyAndrieux added a commit that referenced this issue Oct 27, 2021
@TeddyAndrieux
salt,docs: Allow to manage number of CoreDNS replica
a3c8dfa 
Refs: #3574
TeddyAndrieux added a commit that referenced this issue Nov 22, 2021
@TeddyAndrieux
salt,docs: Allow to manage number of CoreDNS replica
3615bbb 
Refs: #3574
@bert-e bert-e closed this as completed in b2e8764 Nov 23, 2021
TeddyAndrieux added a commit that referenced this issue Nov 24, 2021
@TeddyAndrieux
salt: Add execution module to "compute" affinity
a61ecce 
Since we want to expose a really simple way to setup Pod affinity we do
not use the exact same syntax as what need to provided in the Kubernetes
objects, in order to convert from "simple syntax" to "kubernetes syntax"
we add this execution module

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 24, 2021
@TeddyAndrieux
salt: Add execution module to "compute" affinity
5c69302 
Since we want to expose a really simple way to setup Pod affinity we do
not use the exact same syntax as what need to provided in the Kubernetes
objects, in order to convert from "simple syntax" to "kubernetes syntax"
we add this execution module

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 24, 2021
@TeddyAndrieux
salt: Add execution module to "compute" affinity
98770e0 
Since we want to expose a really simple way to setup Pod affinity we do
not use the exact same syntax as what need to provided in the Kubernetes
objects, in order to convert from "simple syntax" to "kubernetes syntax"
we add this execution module

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 25, 2021
@TeddyAndrieux
salt: Add execution module to "compute" affinity
050eddd 
Since we want to expose a really simple way to setup Pod affinity we do
not use the exact same syntax as what need to provided in the Kubernetes
objects, in order to convert from "simple syntax" to "kubernetes syntax"
we add this execution module

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 25, 2021
@TeddyAndrieux
charts,salt,tests: Add podAntiAffinity support for Dex
cc15346 
This commit add ability to configure `podAntiAffinity` for Dex from CSC.

Patche the Dex helm chart to add support for `strategy` on Dex
deployment, as the default one does not make sense for our Dex
deployment
See dexidp/helm-charts#66

Render chart to salt state using
```
./charts/render.py dex charts/dex.yaml charts/dex \
  --namespace metalk8s-auth \
  --service-config dex metalk8s-dex-config \
  metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \
  > salt/metalk8s/addons/dex/deployed/chart.sls
```

See: #3574
@TeddyAndrieux TeddyAndrieux mentioned this issue Nov 25, 2021
Merged
TeddyAndrieux added a commit that referenced this issue Nov 25, 2021
@TeddyAndrieux
charts,salt,tests: Add podAntiAffinity support for Dex
d2b786a 
This commit add ability to configure `podAntiAffinity` for Dex from CSC.

Patche the Dex helm chart to add support for `strategy` on Dex
deployment, as the default one does not make sense for our Dex
deployment
See dexidp/helm-charts#66

Render chart to salt state using
```
./charts/render.py dex charts/dex.yaml charts/dex \
  --namespace metalk8s-auth \
  --service-config dex metalk8s-dex-config \
  metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \
  > salt/metalk8s/addons/dex/deployed/chart.sls
```

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 29, 2021
@TeddyAndrieux
salt: Add execution module to "compute" affinity
9e1f5fc 
Since we want to expose a really simple way to setup Pod affinity we do
not use the exact same syntax as what need to provided in the Kubernetes
objects, in order to convert from "simple syntax" to "kubernetes syntax"
we add this execution module

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 29, 2021
@TeddyAndrieux
charts,salt,tests: Add podAntiAffinity support for Dex
5edb900 
This commit add ability to configure `podAntiAffinity` for Dex from CSC.

Patche the Dex helm chart to add support for `strategy` on Dex
deployment, as the default one does not make sense for our Dex
deployment
See dexidp/helm-charts#66

Render chart to salt state using
```
./charts/render.py dex charts/dex.yaml charts/dex \
  --namespace metalk8s-auth \
  --service-config dex metalk8s-dex-config \
  metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \
  > salt/metalk8s/addons/dex/deployed/chart.sls
```

See: #3574
TeddyAndrieux added a commit that referenced this issue Nov 29, 2021
@TeddyAndrieux
charts,salt,tests,docs: Add podAntiAffinity for CP ingress controller
72a15a6 
This commit add ability to configure `podAntiAffinity` for the control
plane ingress controller from bootstrap config.

NOTE: This is only used when MetalLB is enabled as otherwise the control
plane ingress controller is deployed as a DaemonSet.

Render chart to salt state using
```
./charts/render.py ingress-nginx-control-plane --namespace metalk8s-ingress \
  charts/ingress-nginx-control-plane-deployment.yaml charts/ingress-nginx/ \
  > salt/metalk8s/addons/nginx-ingress-control-plane/deployed/chart-deployment.sls
```

See: #3574
@TeddyAndrieux TeddyAndrieux mentioned this issue Nov 30, 2021
Merged
TeddyAndrieux added a commit that referenced this issue Nov 30, 2021
@TeddyAndrieux
charts,salt,tests,docs: Add podAntiAffinity for CP ingress controller
5f5b4d2 
This commit add ability to configure `podAntiAffinity` for the control
plane ingress controller from bootstrap config.

NOTE: This is only used when MetalLB is enabled as otherwise the control
plane ingress controller is deployed as a DaemonSet.

Render chart to salt state using
```
./charts/render.py ingress-nginx-control-plane --namespace metalk8s-ingress \
  charts/ingress-nginx-control-plane-deployment.yaml charts/ingress-nginx/ \
  > salt/metalk8s/addons/nginx-ingress-control-plane/deployed/chart-deployment.sls
```

See: #3574
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
complexity:medium Something that requires one or few days to fix kind:enhancement New feature or request severity:major Major impact on live deployments (e.g. some non-critical feature is not working at all) topic:deployment Bugs in or enhancements to deployment stages topic:networking Networking-related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Manage podAntiAffinity and number of replicas for CoreDNS scality/metalk8s
1 participant
@TeddyAndrieux