docs: add note on domain discovery for TSC

aws · Dec 3, 2024 · 14c023d · 14c023d
1 parent bbb4996
commit 14c023d
Showing 1 changed file with 82 additions and 0 deletions.
diff --git a/website/content/en/docs/concepts/scheduling.md b/website/content/en/docs/concepts/scheduling.md
@@ -381,6 +381,88 @@ See [Pod Topology Spread Constraints](https://kubernetes.io/docs/concepts/worklo
 NodePools do not attempt to balance or rebalance the availability zones for their nodes. Availability zone balancing may be achieved by defining zonal Topology Spread Constraints for Pods that require multi-zone durability, and NodePools will respect these constraints while optimizing for compute costs.
 {{% /alert %}}
 
+{{% alert title="Warning" color="secondary" %}}
+Pods do not transitively receive requirements from compatible NodePools. Consider the following NodePool and pod specs:
+
+```yaml
+appVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+  name: default
+spec:
+  template:
+    spec:
+      requirements:
+      - key: topology.kubernetes.io/zone
+        operator: Exists
+---
+appVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+  name: np-zonal-constraint
+  labels:
+    project: zone-specific-project
+spec:
+  template:
+    spec:
+      requirements:
+      - key: topology.kubernetes.io/zone
+        operator: In
+        values: ['us-east-1a', 'us-east-1b']
+      # ...
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: inflate
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: inflate
+  template:
+    metadata:
+      labels:
+        app: inflate
+    spec:
+      nodeSelector:
+        project: zone-specific-project
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                  - key: topology.kubernetes.io/zone
+                    operator: In
+                    values: ['us-west-2d', 'us-west-2c']
+      topologySpreadConstraints:
+        - maxSkew: 1
+          topologyKey: topology.kubernetes.io/zone
+          whenUnsatisfiable: DoNotSchedule
+          labelSelector:
+            matchLabels:
+              app: inflate
+```
+
+This cluster has subnets in three availability zones: `us-east-1a`, `us-east-1b`, and `us-east-1c`.
+NodePool `default` can launch instance types in all three zones, but `np-zonal-constraint` is constrained to two.
+Although the pod's `nodeSelector` restricts the pod to NodePool `np-zonal-constraint`, Karpenter uses the pod's constraints to determine compatible domains.
+Since the pod does not have any zonal constraints, Karpenter will attempt to spread the deployment across all three availability zones.
+Since the NodePool is only compatible with two, Karpenter will fail to satisfy the constraints for all but the first two pods.
+
+To resolve this issue, zonal constraints should be added to the pod spec:
+```yaml
+nodeAffinity:
+  requiredDuringSchedulingIgnoredDuringExecution:
+    nodeSelectorTerms:
+      - matchExpressions:
+          - key: topology.kubernetes.io/zone
+            operator: In
+            values: ['us-east-1a', 'us-east-1b']
+```
+
+{{% /alert %}}
+
 ### Pod affinity/anti-affinity
 
 By using the `podAffinity` and `podAntiAffinity` configuration on a pod spec, you can inform the Karpenter scheduler of your desire for pods to schedule together or apart with respect to different topology domains.