Support PodAntiAffinity

[Noksa]

Tell us about your request

As you know k8s has great podAntiAffinity mechanism that allows us to place the same pods on different nodes.
As example:

podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
          - key: "kazooService"
            operator: In
            values:
              - "kazoo-sbc"
      topologyKey: "kubernetes.io/hostname"

At the same time Karpenter doesn't support both podAffinity and podAntiAffinity

We can use topologySpreadConstraints instead but it has maxSkew 1 as minimum value.
It means that two same pods can be placed on the same node.

So how can we tell Karpenter to create a new node for each specific pod instead of trying to place them to an existing node if it still has capacity?

I know that I can use

resources:
  requests:

to let pod acquire all resources on a node but if I do this all other pods (from other deploy/sts) won't be able to be placed on the node if they satisfy requirements.

Can you please tell me if there is a workaround?

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ellistarn]

Closing in favor of #942. Can you post your question there?

[ellistarn]

We can use topologySpreadConstraints instead but it has maxSkew 1 as minimum value.
It means that two same pods can be placed on the same node.

If you use hostname topology with max skew of one, you will only get one pod per node.

edit: Thinking further, in single node cluster, the kube scheduler will place all pods on that node if there is room, since provisioning isn't part of the decision making.

I think this convinces me that we need antiaffinity support and my previous statements about topology being an abject replacement do not hold in all use cases.

[Noksa]

@ellistarn
Actually, the topologySpreadConstraints doesn't work properly when we want to place one pod per node because of that:
kubernetes/kubernetes#94627 (comment) (but in my case maxSkew is 1).
I mean the only one node always satisfies maxSkew

So let's say if I have only one node with label db-only=true ALL pods that tolerate this label and have nodeSelector or nodeAffinity equals to this label will be placed on the only one node.
And new nodes won't be provisioned.
topologySpreadConstraints starts to work properly only if we have >=2 nodes that are included in calculation of topologySpreadConstraints.

That's why I asked PodAntiAffinity support.

Currently it is impossible to place one pod per node with Karpenter when we use nodeSelector or nodeAffinity. Karpenter will create only one node and then kube-scheduler will place all pods on this node while it has capacity.

So please consider support for PodAntiAffinity. Thanks!