Event stream api leaks file handles

[towe75]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Repetitiv event stream API requests leak file handles.

Steps to reproduce the issue:

1. Start a long running stats stream in a terminal. This avoids that systemd will restart the podman socket server between subsequent requests.

curl -s --unix-socket /run/podman/podman.sock http://localhost/v1/libpod/containers/stats

2. Watch number of file handles in a second terminal.

watch "lsof -p $(pgrep -f 'podman.*system service') | wc -l"

3. Subscribe to the event stream and cancel it after a second. Repeat this step multiple times.

curl -m 1 -s --unix-socket /run/podman/podman.sock http://localhost/v1/libpod/events

Describe the results you received:

Number of file handles is growing with each subsequent "events" subscription until the podman system service is restarted.

The problem can be monitored on a idle system without starting/stopping containers.

Additional information you deem important (e.g. issue happens only occasionally):

I found this while working on the nomad podman driver. We try to leverage event and stats streams to improve the performance.
Running our unit tests exhausted the open file limit and i started to hunt the problem.

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.14.10
Built:        Tue Dec  8 15:37:43 2020
OS/Arch:      linux/amd64

[Luap99]

What eventLogger backend do you use? (podman info | grep event)

[towe75]

It's a standard fedora setup with journald backend.

podman info | grep event
  eventLogger: journald

[baude]

nice report ... i assume we are talking about these ...

podman  7565 root   52r      REG              253,0  16777216 3942679 /var/log/journal/4804cba2dc9947b7b73488ba5c085ce9/[email protected]~
podman  7565 root   53r      REG              253,0   8388608 3944305 /var/log/journal/4804cba2dc9947b7b73488ba5c085ce9/user-1000@fa82a97192354f88b90977b068a0e6c9-00000000000b73ba-0005b5fa735cc913.journal
podman  7565 root   54r      REG              253,0  92274688 3941208 /var/log/journal/4804cba2dc9947b7b73488ba5c085ce9/system@4ae9c9bc979142ab87569286f52f245c-0000000000030da6-0005b5f9b3b52f6f.journal
podman  7565 root   55r      REG              253,0  92274688 3934590 /var/log/journal/4804cba2dc9947b7b73488ba5c085ce9/system@4ae9c9bc979142ab87569286f52f245c-0000000000000001-0005b5f5bbd57fd5.journal
podman  7565 root   56r      REG              253,0  16777216 3942645 /var/log/journal/4804cba2dc9947b7b73488ba5c085ce9/system.journal
podman  7565 root   57r      REG              253,0  92274688 3944302 /var/log/journal/4804cba2dc9947b7b73488ba5c085ce9/system@4ae9c9bc979142ab87569286f52f245c-00000000000a23fd-0005b5fa554d791d.journal

[towe75]

@baude yes, lots of journal descriptors. Thx for having a look! 🧯