API service is using 100% cpu

[xomachine]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Podman API service process starts using 100% after closing connection to the /v1/events endpoint

Steps to reproduce the issue:

1. Start podman API service podman system service -t0

2. Make a request to API service using curl curl -v --unix-socket /run/podman/podman.sock 'http:/v1/events'.

3. After a few seconds close the connection using Ctrl-C on the terminal with curl

Describe the results you received:
After ~5 seconds podman service starts using 100% cpu . For every new interrupted events request it takes another 100%.

Describe the results you expected:
Usual cpu usage

Additional information you deem important (e.g. issue happens only occasionally):
The issue can be reproduced regardless of API version. Another streaming API - stats - does not show such behavior.
perf top shows that we stuck in the selectgo call

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.13.15
Built:        Fri Oct  2 17:30:39 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: fa5f92225c4c95759d10846106c1ebd325966f91-dirty'
  cpus: 8
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: podman8.test.local
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  linkmode: dynamic
  memFree: 4052320256
  memTotal: 16643387392
  ociRuntime:
    name: runc
    package: runc-1.0.0-15.1.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: 2e01c9e4dfdab4b4c993a3698059fb824cd3286a
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 2135158784
  swapTotal: 2147479552
  uptime: 3859h 2m 10.33s (Approximately 160.79 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 6
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601649039
  BuiltTime: Fri Oct  2 17:30:39 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.1-4.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Test is performed in qemu virtual machine

[TomSweeneyRedHat]

@jwhonce thoughts?

[baude]

would you be willing to try this upstream main branch?

[xomachine]

Were able to reproduce with version from the master branch on Archlinux

host:
  arch: amd64
  buildahVersion: 1.16.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 35a2fa83022e56e18af7e6a865ba5d7165fa2a4a'
  cpus: 4
  distribution:
    distribution: '"antergos"'
    version: unknown
  eventLogger: journald
  hostname: xom-base
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65536
  kernel: 5.8.12-arch1-1
  linkmode: dynamic
  memFree: 4501020672
  memTotal: 15653163008
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc92
      commit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
      spec: 1.0.2-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 20480909312
  swapTotal: 20480909312
  uptime: 15h 11m 28.42s (Approximately 0.62 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /home/xomachine/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/xomachine/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 9
  runRoot: /run/user/1000/containers
  volumePath: /home/xomachine/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1602100434
  BuiltTime: Wed Oct  7 22:53:54 2020
  GitCommit: 0e1d01103e45430693736dac10be13c49cf23f03
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.2.0-dev

[rod260185]

I have the same problem now, with this configuration:
host:
arch: amd64
buildahVersion: 1.23.1
cgroupControllers:

• cpuset
• cpu
• io
• memory
• hugetlb
• pids
• rdma
• misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
  package: conmon-2.0.30-1.1.x86_64
  path: /usr/bin/conmon
  version: 'conmon version 2.0.30, commit: unknown'
  cpus: 4
  distribution:
  distribution: '"opensuse-microos"'
  version: "20211107"
  eventLogger: journald
  hostname: ev00000895
  idMappings:
  gidmap: null
  uidmap: null
  kernel: 5.14.14-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 11157291008
  memTotal: 20200374272
  ociRuntime:
  name: runc
  package: runc-1.0.2-1.2.x86_64
  path: /usr/bin/runc
  version: |-
  runc version 1.0.2
  spec: 1.0.2-dev
  go: go1.13.15
  libseccomp: 2.5.2
  os: linux
  remoteSocket:
  exists: true
  path: /run/podman/podman.sock
  security:
  apparmorEnabled: true
  capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
  rootless: false
  seccompEnabled: true
  seccompProfilePath: /etc/containers/seccomp.json
  selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
  executable: /usr/bin/slirp4netns
  package: slirp4netns-1.1.11-1.2.x86_64
  version: |-
  slirp4netns version 1.1.11
  commit: unknown
  libslirp: 4.6.1
  SLIRP_CONFIG_VERSION_MAX: 3
  libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 4m 7.01s
  plugins:
  log:
• k8s-file
• none
• journald
  network:
• bridge
• macvlan
  volume:
• local
  registries:
  search:
• registry.opensuse.org
• docker.io
  store:
  configFile: /etc/containers/storage.conf
  containerStore:
  number: 27
  paused: 0
  running: 24
  stopped: 3
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
  Build Version: 'Btrfs v5.14.1 '
  Library Version: "102"
  imageStore:
  number: 235
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
  version:
  APIVersion: 3.4.1
  Built: 1634688000
  BuiltTime: Tue Oct 19 21:00:00 2021
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 3.4.1

It seems to be some kind of regression. Though my problem comes with the use of docker extension of VSCode, which is inspecting the api through the podman.sock.

[mheon]

This issue is over a year old, please open a fresh issue.