Topology spread constraint with topologyKey: topology.kubernetes.io/zone is not working as expected

[adjain131995]

Version

Karpenter: v0.6.3

Expected Behavior

Pods should be equally distributed across AZ and if nodes are already running but nodes are not spread equally in AZ. Karpenter should spin up new nodes to justify AZ spread topology constraint and do not deploy in an unbalanced AZ manner.

Actual Behavior

if three are nodes already present and are equally AZ spread like

• us-west-2a 4 nodes
• us-west-2b 4 nodes
• us-west-2c 1 node

Assuming karpenter does not kick in kube scheduler schedules the pods in this manner.

• 10 pods in 2a
• 9 pods in 2b
• 3 pods in 2c

because 2a and 2b has more capacity and hence pods are scheduled there instead of karpenter spinning new nodes in 2c and justify the AZ spread

This is a really genuine scenario that can happen when running karpenter in production with hundreds of clusters where one wants to ensure the workload is spread in a balanced manner across AZ.
kubernetes.io/hostname topology was also tried to resolve this situation but it results in spinning tons of nodes which are not required.

Steps to Reproduce the Problem

Deploy the following piece of code with already running nodes which are not zone spread equally

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          run: nginx2
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          run: nginx2-

Resource Specs and Logs

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: general-provisioner
spec:
labels:
provisioner-type: "standard"
limits:
resources:
cpu: 1k
provider:
tags:
clusterId: {{ .Values.clusterFullName }}
apiVersion: extensions.karpenter.sh/v1alpha1
instanceProfile: KarpenterNodeInstanceProfile-Test
kind: AWS
securityGroupSelector:
kubernetes.io/cluster/{{ include "karpenter.cluster.name" . }}: "*"
subnetSelector:
purpose_app: "true"
requirements:
- key: topology.kubernetes.io/zone
operator: In
values:
- us-west-2a
- us-west-2b
- us-west-2c
operator: In
values:
- amd64
- key: karpenter.sh/capacity-type
operator: In
values:
- on-demand
- spot
ttlSecondsAfterEmpty: 30

[tzneal]

Assuming karpenter does not kick in kube scheduler schedules the pods in this manner.

Karpenter begins by watching for pods that the Kubernetes scheduler has marked as unschedulable. It's not designed to replace kube-scheduler, but instead to work along side it provisioning capacity. For Karpenter to have an opportunity to provision new nodes for pods, kube-scheduler will first have to fail to schedule those pods.

This KEP may be relevant in the future as it is an enhancement that allows specifying a minimum number of domains to consider for the topology. kube-scheduler is currently limited in that it is only aware of a domain by the existence of nodes (e.g. if there is no node in zone-c, kube-scheduler doesn't know that zone-c exists).

I'm going to close this, but feel free to re-open it if I've misunderstood your question.