Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add arch, os, and provisioner anti-affinities for the karpenter deployment #468

Merged
merged 3 commits into from
Jun 23, 2021
Merged

add arch, os, and provisioner anti-affinities for the karpenter deployment #468

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a85395c
add arch, os, and provisioner anti-affinities for the karpenter deplo…
bwagner5 Jun 22, 2021
9976a1d
remove provisioner anti-affinity
bwagner5 Jun 22, 2021
b2b1443
remove arch affinity
bwagner5 Jun 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions charts/karpenter/templates/controller/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,25 @@ spec:
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.controller.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.controller.affinity }}
{{- toYaml .Values.controller.affinity | nindent 8 }}
{{- else }}
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't actually think we want this. Imagine if someone wants to run karpenter in a management cluster, who's nodes are themselves managed by a separate karpenter installation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I hadn't considered the case of a management cluster using karpenter... But a karpenter installation should never be hosted on a node provisioned by the installation, right? I can remove this one since it's not super important for people trying it out right now, only slightly annoying when developing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and removed this one.

- key: provisioning.karpenter.sh/name
operator: DoesNotExist
- key: kubernetes.io/arch
operator: In
values:
- amd64
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we currently build arm images w/ ko?

Copy link
Contributor Author

@bwagner5 bwagner5 Jun 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The makefile has the flag to generate for all platforms, but I only see the amd64 one in the public registry.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be that ecr public just doesn't show the other arch ones ... the manifest appears to have the hashes listed, let me do some digging real quick.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested on an arm64 instance and it does appear to pull down an arm64 variant, so I removed the arch affinity.

- key: kubernetes.io/os
operator: In
values:
- linux
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

{{- end }}
{{- with .Values.controller.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
Expand Down
21 changes: 18 additions & 3 deletions charts/karpenter/templates/webhook/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,25 @@ spec:
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.webhook.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.webhook.affinity }}
{{- toYaml .Values.webhook.affinity | nindent 8 }}
{{- else }}
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: provisioning.karpenter.sh/name
operator: DoesNotExist
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: kubernetes.io/os
operator: In
values:
- linux
{{- end }}
{{- with .Values.webhook.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
Expand Down