Docker 2.1.5.0 / hyperkit eats CPU cycles, idle wakeups

[bric3]

• I have tried with the latest version of my channel (Stable or Edge)
• I have uploaded Diagnostics
• Diagnostics ID: 3C0BA74C-547A-4215-9587-1B01588B78D2/20191105112331

Expected behavior

hyperkit uses a reasonable amount of CPU when docker is not used, e.g. when Docker for mac is started.

Actual behavior

Hyperkit uses 102% of CPU on Docker app start.

Information

• macOS Version: 10.15.1 (build: 19B88)

Just upgraded Docker for Mac to the last version on the Edge channel. Upon start, hyperkit starts to consume a lot of CPU. However the app says docker is running, and command line usage works. eg.

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Restarting didn't help.

The Console app, outputs a log of logs from hyperkit and com.docker.driver.amd64-linux, so I'm not sure what's wrong at this time. However I did notice this

default	12:43:52.408158+0100	symptomsd	Received CPU wakes trigger:
  com.docker.hyperkit[15009] () woke the CPU 45001 times over 78.01 seconds (average 576/sec), violating a CPU wakes limit of 45000 over 300 seconds.
default	12:43:52.411420+0100	symptomsd	RESOURCE_NOTIFY trigger for com.docker.hyperkit [15009] (45001 CPU wakes over 78.00s seconds, violating limit of 45000 CPU wakes over 300.00s seconds)
default	12:43:52.412062+0100	symptomsd	  Generated RESOURCE_NOTIFY report for com.docker.hyperkit within the past 300 seconds (1572954119.1545). Disallow report.

Diagnostic logs

Docker for Mac: (Edge) 2.1.5.0

The only thing that the UI displays is "Diagnose succeeded".

Steps to reproduce the behavior

1. Have macos Catalina 10.15.1
2. Have Docker for Mac Edge 2.1.5.0
3. Starts Docker for mac, look at the CPU

[bric3]

while it seems to be another issue, there could be interesting info in the late comments of #3499

[bric3]

Just upgraded to 2.1.6.0, and I got the same issue.

COMMAND
com.docker.hyperkit \
  -A \
  -u \
  -F vms/0/hyperkit.pid \
  -c 2 \
  -m 2048M \
  -s 0:0,hostbridge \
  -s 31,lpc \
  -s 1:0,virtio-vpnkit,path=vpnkit.eth.sock,uuid=9eb5729f-c320-4e63-81b5-95133921d6cd \
  -U a553145a-5447-419e-bacc-68bb5c3c4ece \
  -s 2:0,ahci-hd,/Users/bric3/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw \
  -s 3,virtio-sock,guest_cid=3,path=vms/0,guest_forwards=2376;1525 \
  -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-desktop.iso \
  -s 5,ahci-cd,vms/0/config.iso \
  -s 6,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker.iso \
  -s 7,virtio-rnd -l com1,autopty=vms/0/tty,asl \
  -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,

hyperkit options

$ /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit -h
Usage: com.docker.hyperkit [-behuwxMACHPWY] [-c vcpus] [-F <pidfile>] [-g <gdb port>] [-l <lpc>]
                           [-m mem] [-p vcpu:hostcpu] [-s <pci>] [-U uuid] -f <fw>
       -A: create ACPI tables
       -c: # cpus (default 1)
       -C: include guest memory in core file
       -e: exit on unhandled I/O access
       -f: firmware
       -F: pidfile
       -g: gdb port
       -h: help
       -H: vmexit from the guest on hlt
       -l: LPC device configuration. Ex: -l com1,stdio -l com2,autopty -l com2,/dev/myownpty
       -m: memory size in MB, may be suffixed with one of K, M, G or T
       -M: print MAC address and exit if using vmnet
       -P: vmexit from the guest on pause
       -s: <slot,driver,configinfo> PCI slot config
       -u: RTC keeps UTC time
       -U: uuid
       -v: show build version
       -w: ignore unimplemented MSRs
       -W: force virtio to use single-vector MSI
       -x: local apic is in x2APIC mode
       -Y: disable MPtable generation

Not sure if that helps, here's what hyperkit sampling for 30 seconds gives me :

sudo /usr/bin/sample 36111 30 -f 2019-11-20T1325-hyperkit-sample.txt

2019-11-20T1325-hyperkit-sample.txt

cat 2019-11-20T1325-hyperkit-sample.txt | stackcollapse-sample.awk | flamegraph.pl

Sorry the flamegraph is zipped, as GitHub does not allow to share svg file types directly.

I see that most sample are from libdyld`start ~81% which invokes something in hyperkit, but without debug symbols I don't know what that could be, and if we go up in the flame there's a significant part of libsystem_kernel`cerror ~11% , which could be a reason why hyperkit eats cpu as it constantly retry something,due to this error?

---

I don't know if it's helpful, but I found those in /Library/Log/DiagnosticReports
com.docker.hyperkit.wakeups_resource.diag.zip

[jimmycallin]

Same problem here: 99141BD1-8E5C-4C53-B1F3-FCB7D3E37DC4/20191123161352

[bric3]

In a related twitter thread I mentioned JAMF and SentinelOne, a colleague of mine tried docker without sentinel and it worked without the high CPU usage.

$ kextstat
...
  128    0 0xffffff7f84276000 0x3b000    0x3b000    com.sentinelone.sentinel-kext (2808) 19EA05C2-D73C-3BDC-B5D9-C559009A0062 <65 21 6 5 3 2 1>
...

Unfortunately I cannot get rid of sentinel at that time on my laptop to try without it.

[bric3]

So after excluding $HOME/Library/Containers/com.docker.docker/ from SentinelOne the CPU was fine again.

---

EDIT: Actually we had to exclude additional paths too

• ${HOME}/Library/Group Containers/group.com.docker
• ${HOME}/Library/Containers/com.docker.docker
• ${HOME}/Library/Containers/com.docker.helper
• ${HOME}/Library/Application Support/Docker Desktop
• /Applications/Docker.app

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked