MacOS Big Sur killing skaffold process -- violates CPU wakes limit

[vinceferro]

Expected behavior

Skaffold run should deploy correctly

Actual behavior

The process gets killed by the OS, i can see warnings in the console saying skaffold is waking up the CPU too often

Received CPU wakes trigger:
  skaffold[15156] () woke the CPU 45001 times over 29.59 seconds (average 1520/sec), violating a CPU wakes limit of 45000 over 300 seconds.
RESOURCE_NOTIFY trigger for skaffold [15156] (45001 CPU wakes over 29.00s seconds, violating limit of 45000 CPU wakes over 300.00s seconds)

Information

• Skaffold version: v1.17.2
• Operating system: macOS BigSur 11.1
• Contents of skaffold.yaml: company policy, won't share :(

[briandealwis]

Very odd @vinceferro! Could you provide some more details please? Does this happen immediately on every run, or just sometimes? What were you doing? Could you include some debug logs (skaffold ... -vdebug)?

This error message isn't new to Big Sur — I found several references to "violating a CPU wakes limit" going back to 2018. One report tied this to use of JAMF/SentinelOne.com. Do you have some anti-virus or file watcher running? Do you have frequent file changes going on?

[vinceferro]

hi @briandealwis, it happens as soon a little after the deploy phase starts, on the execution it's at this stage:
kubectl --context test-us-west-2 --namespace web-preprod get -f - --ignore-not-found -ojson
The command is a simple skaffold run. The project is composed by many different deployments, configuration maps, secrets deployed together.
I've looked at the activity monitor, I saw spikes in Idle Wakes up for skaffold, docker and kubectl, but nothing super suspicious..

Anyways, there is the debug log, thanks for the help!
skaffold.log

[briandealwis]

There's a lot of work happening in parallel according to your logs. Could you try making Skaffold run the build sequentially by altering your skaffold.yml to set build.local.concurrency to 1 and see if that makes a difference?

Did this happen with previous versions of Skaffold? Did you only notice this with the update to BigSur?

[vinceferro]

Same, I've tried with concurrency 1 but still no luck deploying. I think it started happening with skaffold v1.17.2, but really a lot changed.. For some reasons, with skaffold dev it doesn't happen to get the sigkill.. Mistery 😅

[vinceferro]

It was kubectl version, I downgraded it from v1.20 to v1.7.4!
Now I wonder if that is uniquely an issue with kubectl or the issue is combined with skaffold..

[briandealwis]

@vinceferro thanks for the update. 1.7.4 is pretty old, so you might be better off using the version corresponding to your kubernetes cluster.

I've since upgraded my personal laptop to Big Sur, and I'm unable to reproduce this problem with the officially provided kubectl 1.20.0. Could you let us know:

• Which kubernetes distribution are you using?
• Where did you get kubectl?
• What Skaffold command-line did you actually run? (skaffold run --tail?)

[vinceferro]

hey @bryanlarsen, just to give you more context, this is only happening with our fairly big application, with other smaller one this isn't happening.
Here's the answers:

• kubernetes 1.17+ / v1.17.12-eks-7684af, EKS managed kubernetes cluster
• the problematic one from brew, the working one from google's hosted binaries https://storage.googleapis.com/kubernetes-release/release/v1.17.4/bin/darwin/amd64/kubectl
• the command is skaffold run -p production -n *****-web-prod --status-check=false

[briandealwis]

I'm baffled — your debug log only shows kubectl being executed a single time. And it doesn't explain why Skaffold is being killed.

I'll leave this open in case others have a similar issue and can shed some light.

[cherudim]

I had the same issue, this is my environment:

OS X Catalina (10.15.7)
kubectl 1.20 (from homebrew)
skaffold 1.27.0 (from homebew)

Command + output:

$ skaffold -p pricing deploy
Tags used in deployment:
Starting deploy...
running [kubectl --context the-context get -f - --ignore-not-found -ojson]
 - stdout: ""
 - stderr: ""
 - cause: signal: killed

Resolved by downloading the latest kubectl (1.21) from the official site (curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl")

The common denominator seems to be kubectl 1.20 from homebrew