Possible regression with podman system service causing high cpu usage #12308
Comments
|
Additionaly, when a service using docker-compose uses rabbitmq, it triggers a cpu usage of 120-160% in 4 CPUs. |
rod260185
changed the title
|
Can you share the API calls to get into this situation? Also the server debug log would be helpful. |
|
I can't inspect what call are causing it, but it happens when I use the default VSCode docker extension, which I think does some regular api calls, like "podman image ls", "podman ps -a" etc. As also the taiga-docker running through docker-compose, which, first get the 2 rabbitmq services down at start up, but then when I restart just then, they work, but causes this eternal high cpu consuming loop with podman API. I will need a help with how I get this server debug log. I noticed that the rootless API also have the same problem, as I have now two APIs with high CPU usage, that are taking me 2/3 of the 4 3GHz cores. |
|
Could you turn on logging at the server side and see if you can figure out what the server is doing |
|
I did it and exported a system log after I start rabbitmq services on taiga and used VS Code Extension "Docker", it seems related with conmon. |
|
Nov 17 08:18:51 ev00000897 podman[972]: @ - - [17/Nov/2021:08:18:51 -0300] "GET /containers/json?all=true HTTP/1.1" 200 45653 "" "" |
|
@giuseppe PTAL |
|
I think I have the same problem. rootless podman on Fedora 35. But when I visit the Docker tab in VSCode and it starts querying the podman daemon I get a lot of these messages in the debug log. Even after VSCode runs into a timeout error daemon keeps logging these for a while until it stops, causing a long spike in cpu usage. When I have multiple instances of VSCode open all querying podman daemon my system becomes very sluggish. Is this a problem with VSCode hitting the API too hard or should podman be faster in serving image information? Is 134 images too much? Should I prune them more often? |
|
Here are all the |
|
Ok after pruning my podman images down to 23, performance is back to nominal. |
|
@vrothberg thoughts? |
|
There was a regression in c/common that was fixed with containers/common@f745869fbc76 which has made it into Podman's main branch. It has not been backported to Podman v3.4.x. @rhatdan please let me know if we should backport or wait for 4.0 |
|
Probably should back port. |
|
@mheon, WDYT, since you mentioned a 3.5 earlier today? |
|
I was going to focus 3.5 mostly on OS X backports and fixes, and this does seem to qualify as a |
|
Here's a PR to backport the fix along with some other libimage-related fixes that will come in handy: containers/common#866 |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@vrothberg Is this fixed in the main branch? |
Yes, we can close. |
github-actions
bot
added
the
locked - please file new issue/PR
I have the same problem now (#7946), with this configuration:
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.0.30-1.1.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.30, commit: unknown'
cpus: 4
distribution:
distribution: '"opensuse-microos"'
version: "20211107"
eventLogger: journald
hostname: ev00000895
idMappings:
gidmap: null
uidmap: null
kernel: 5.14.14-1-default
linkmode: dynamic
logDriver: journald
memFree: 11157291008
memTotal: 20200374272
ociRuntime:
name: runc
package: runc-1.0.2-1.2.x86_64
path: /usr/bin/runc
version: |-
runc version 1.0.2
spec: 1.0.2-dev
go: go1.13.15
libseccomp: 2.5.2
os: linux
remoteSocket:
exists: true
path: /run/podman/podman.sock
security:
apparmorEnabled: true
capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.11-1.2.x86_64
version: |-
slirp4netns version 1.1.11
commit: unknown
libslirp: 4.6.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 4m 7.01s
plugins:
log:
network:
volume:
registries:
search:
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 27
paused: 0
running: 24
stopped: 3
graphDriverName: btrfs
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphStatus:
Build Version: 'Btrfs v5.14.1 '
Library Version: "102"
imageStore:
number: 235
runRoot: /run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 3.4.1
Built: 1634688000
BuiltTime: Tue Oct 19 21:00:00 2021
GitCommit: ""
GoVersion: go1.13.15
OsArch: linux/amd64
Version: 3.4.1
It seems to be some kind of regression. Though my problem comes with the use of docker extension of VSCode, which is inspecting the api through the podman.sock.
Originally posted by @rod260185 in #7946 (comment)
The text was updated successfully, but these errors were encountered: