Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

topologySpreadConstriants with empty labelSelector makes karpenter crash #1021

Closed
olemarkus opened this issue Dec 19, 2021 · 1 comment · Fixed by #1095
Closed

topologySpreadConstriants with empty labelSelector makes karpenter crash #1021

olemarkus opened this issue Dec 19, 2021 · 1 comment · Fixed by #1095
Assignees
Labels
bug Something isn't working burning Time sensitive issues good-first-issue Good for newcomers

Comments

@olemarkus
Copy link
Contributor

Version

Karpenter: v0.5.3

Kubernetes: v1.22.4

Actual Behavior

Karpenter crash with an npr if a Pod has a topologySpreadConstraint with empty labelSelector. Empty labelSelectors doesn't make much sense, but still shouldn't crash karpenter.

Resource Specs and Logs

Just ignoring the spread constraint is probably fine if the labelSelector is empty.

2021-12-19T07:40:14.064Z        INFO    controller.provisioning Batched 2 pods in 1.000147818s  {"commit": "5047f3c", "provisioner": "karpenter-nodes"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19daf66]

goroutine 10410 [running]:
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.TopologyListOptions({0xc000bdb0b0, 0xb}, 0xc00410e820)
        github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:144 +0x66
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).countMatchingPods(0xc00376cf10, {0x27965b8, 0xc001184db0}, 0xc00410e820)
        github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:122 +0x85
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).computeZonalTopology(0xc00410e870, {0x27965b8, 0xc001184db0}, {0xc0024acea0, 0x0, 0x0}, 0x0)
        github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:114 +0x7e
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).computeCurrentTopology(0x0, {0x27965b8, 0xc001184db0}, 0xc000c5dda0, 0xc00410e820)
        github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:82 +0x93
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).Inject(0xc00118e2d8, {0x27965b8, 0xc001184db0}, 0xc000c5dda0, {0xc004902790, 0xc000c5dd68, 0x1})
        github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:45 +0xbf
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve(0xc003769740, {0x27965b8, 0xc001184db0}, 0xc00118e1c0, {0xc004902790, 0x2, 0x2})
        github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/scheduler.go:73 +0x1c8
github.com/aws/karpenter/pkg/controllers/provisioning.(*Provisioner).provision(0xc0003cc540, {0x27965b8, 0xc001184db0})
        github.com/aws/karpenter/pkg/controllers/provisioning/provisioner.go:116 +0x208
github.com/aws/karpenter/pkg/controllers/provisioning.NewProvisioner.func1()
        github.com/aws/karpenter/pkg/controllers/provisioning/provisioner.go:65 +0x73
created by github.com/aws/karpenter/pkg/controllers/provisioning.NewProvisioner
        github.com/aws/karpenter/pkg/controllers/provisioning/provisioner.go:63 +0x3a5
@olemarkus olemarkus added the bug Something isn't working label Dec 19, 2021
@ellistarn ellistarn added burning Time sensitive issues good-first-issue Good for newcomers labels Dec 19, 2021
@jalawala
Copy link

I also saw crash with below configs
nodeSelector:
kubernetes.io/arch: amd64
topologySpreadConstraints:
- maxSkew: 5
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: ScheduleAnyway

2021-12-21T11:47:25.368Z INFO controller.provisioning Batched 18 pods in 1.506678526s {"commit": "5047f3c", "provisioner": "default"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x19daf66]

goroutine 133726 [running]:
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.TopologyListOptions({0xc006bbf4e0, 0x7}, 0xc0046b1630)
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:144 +0x66
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).countMatchingPods(0xc003d2ea80, {0x27965b8, 0xc0046af710}, 0xc0046b1630)
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:122 +0x85
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).computeZonalTopology(0x0, {0x27965b8, 0xc0046af710}, {0xc0044ff6c0, 0x0, 0x0}, 0x0)
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:114 +0x7e
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).computeCurrentTopology(0x0, {0x27965b8, 0xc0046af710}, 0xc004737da0, 0xc0046b1630)
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:82 +0x93
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Topology).Inject(0xc003c070d8, {0x27965b8, 0xc0046af710}, 0xc004737da0, {0xc0046b5800, 0xc004737d68, 0xc0045e6320})
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/topology.go:45 +0xbf
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling.(*Scheduler).Solve(0xc0001830e0, {0x27965b8, 0xc0046af710}, 0xc003c06fc0, {0xc0046b5800, 0x12, 0x20})
github.com/aws/karpenter/pkg/controllers/provisioning/scheduling/scheduler.go:73 +0x1c8
github.com/aws/karpenter/pkg/controllers/provisioning.(*Provisioner).provision(0xc002f60cb0, {0x27965b8, 0xc0046af710})
github.com/aws/karpenter/pkg/controllers/provisioning/provisioner.go:116 +0x208
github.com/aws/karpenter/pkg/controllers/provisioning.NewProvisioner.func1()
github.com/aws/karpenter/pkg/controllers/provisioning/provisioner.go:65 +0x73
created by github.com/aws/karpenter/pkg/controllers/provisioning.NewProvisioner
github.com/aws/karpenter/pkg/controllers/provisioning/provisioner.go:63 +0x3a5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working burning Time sensitive issues good-first-issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants