high load in combination with cockpit-podman

[dschier-wtd]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When running cockpit 229 and cockpit-podman 24, I can see some very high load, caused by podman system service.

Steps to reproduce the issue:

1. install Fedora 33
2. install podman
3. install cockpit and cockpit-podman
4. Login as privileged user
5. access the cockpit-podman page
6. run top or similar to see podman system service to cause very high load
7. Close cockpit, podman still causes high load

Describe the results you received:

podman system service is causing high load

Describe the results you expected:

Hard to say, but some api service should not lead to this behaviour:

top - 21:50:39 up  1:13,  1 user,  load average: 4.00, 3.18, 1.69
   7454 root      20   0 1945912  62332  33840 S 400.0   0.4  30:19.49 podman                                                                                                                                                                                                             

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.1.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 4
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: hodgins01
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.8.14-300.fc33.x86_64
  linkmode: dynamic
  memFree: 14940172288
  memTotal: 15962685440
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 1h 14m 39.75s (Approximately 0.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 5
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601853924
  BuiltTime: Sun Oct  4 23:25:24 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

cockpit-bridge-229-1.fc33.x86_64
cockpit-system-229-1.fc33.noarch
cockpit-networkmanager-229-1.fc33.noarch
cockpit-ws-229-1.fc33.x86_64
podman-plugins-2.1.1-10.fc33.x86_64
podman-2.1.1-10.fc33.x86_64
cockpit-packagekit-229-1.fc33.noarch
cockpit-229-1.fc33.x86_64
cockpit-podman-24-1.fc33.noarch
cockpit-selinux-229-1.fc33.noarch
cockpit-storaged-229-1.fc33.noarch
cockpit-kdump-229-1.fc33.noarch
cockpit-sosreport-229-1.fc33.noarch

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

[lenticularis39]

I also noticed this problem and a quick GDB lookup revealed that the process causing the high load is processing a request to libpod/events.

Subsequent testing revealed a way how to reproduce this without cockpit-podman:

• Make a stream request to libpod/events. This doesn't cause anything suspicious yet: # curl --unix-socket /run/podman/podman.sock -X GET http://x/v1.12/libpod/events
• Close the connection by typing Ctrl-C.
• Now the CPU usage of the system service process gradually increases, reaching 100 % after a short while.

[mheon]

I believe this was already fixed on master by @jwhonce - Jhon, can you confirm that?

[xomachine]

It is probably a duplicate of #7946
The problems with cockpit-podman was the reason why I started that issue

[lenticularis39]

It is probably a duplicate of #7946

That seems very likely, at least in my case, because upgrading podman to master fixed the issue for me.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

We can continue to look at #7946