[Support]: container hangs due to ffmpeg beeing unresponsive - "kill -9" not working since moving [Ryzen 5700G] #10902

Trashmee · 2024-04-09T12:41:46Z

Trashmee
Apr 9, 2024

Describe the problem you are having

Hi,

I moved from j4105 to Ryzen 5700G CPU two weeks ago and keep seeing situations that I can resolve via magic reboot only:

Frigate WebUI unresponsive
Shuting down the container hangs
Found an ffmpeg process that is unresponsive, even to "kill -9"
systemctl reboot hangs - so hard reboot or magic reboot are required
I'm somewhat suspecting that ffmpeg doesn't like the hw acceleration / configuration on the Ryzen CPU - but due to absense of facts I'm guessing only
I did not find good reference to stale ffmpeg via google

Situation:

#  podman rm frigate --force
WARN[0010] StopSignal SIGTERM failed to stop container homeassistant-frigate in 10 seconds, resorting to SIGKILL
Error: cannot remove container 4d379268f86a4d3e116f7b6424bb71b13d3bb49452f978f4a5bec7055a2e7049 as it could not be stopped: given PID did not die within timeout

#  ps -aef |grep 158802
root      158802 1029121  0 04:35 ?        00:00:00 ffmpeg -hide_banner -y -i /tmp/cache/garten@20240409043513+0200.mp4 -c copy -movflags +faststart /media/frigate/recordings/2024-04-09/02/garten/35.13.mp4

#  killall -9 ffmpeg
#  kill -9 158802
#  kill -9 158802

#  ps -aef |grep 158802
root      158802 1029121  0 04:35 ?        00:00:00 ffmpeg -hide_banner -y -i /tmp/cache/garten@20240409043513+0200.mp4 -c copy -movflags +faststart /media/frigate/recordings/2024-04-09/02/garten/35.13.mp4

# cat /proc/cpuinfo | [..]
model name      : AMD Ryzen 7 5700G with Radeon Graphics

ghcr.io/blakeblackshear/frigate:0.13.2

# cat config/config/config.yml| [..]

##########
ffmpeg:
##########
  # Optional: global ffmpeg args (default: shown below)
  global_args: -hide_banner -loglevel warning
  # Optional: global hwaccel args (default: shown below)
  hwaccel_args: preset-vaapi

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

Version

0.13.2

Frigate config file

##########
ffmpeg:
##########
  # Optional: global ffmpeg args (default: shown below)
  global_args: -hide_banner -loglevel warning
  # Optional: global hwaccel args (default: shown below)
  hwaccel_args: preset-vaapi


############################
go2rtc:
############################
  streams:
    eingang:
        - rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=0
    eingang_sub:
        #- rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=0
        # Reacl sub:
        - rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=1
    garten:
        - rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=0
    garten_sub:
        #- rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=0
        # Reacl sub:
        - rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=1
    gartentor:
        - rtsp://frigate:[email protected]:554/h265Preview_01_main
        #- ffmpeg:https://gartentor.domain/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=frigate&password=pass#video=copy#audio=copy#audio=opus
    gartentor_sub:
        #- rtsp://frigate:[email protected]:554/h264Preview_01_main
        # Reacl sub:
        - rtsp://frigate:[email protected]:554/h264Preview_01_sub
        #- ffmpeg:https://gartentor.domain/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=frigate&password=pass
    vorgarten:
        - rtsp://frigate:[email protected]:554/h264Preview_01_main
        #- ffmpeg:https://vorgarten.domain/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=frigate&password=pass#video=copy#audio=copy#audio=opus
    vorgarten_sub:
        - rtsp://frigate:[email protected]:554/h264Preview_01_main
        # Reacl sub:
        #- rtsp://frigate:[email protected]:554/h264Preview_01_sub
        #- ffmpeg:https://vorgarten.domain/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=frigate&password=pass#video=copy#audio=copy#audio=opus
    terasse:
        - rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=0
    terasse_sub:
        #- rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=0
        # Reacl sub:
        - rtsp://frigate:[email protected]/cam/realmonitor?channel=1&subtype=1


############################
cameras:
############################
  ############################
  gartentor:      # Reolink RLC-811WA
  ############################
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/gartentor?video=copy&audio=aac
          input_args: preset-rtsp-restream
       # - path: rtmp://vorgarten.domain/bcs/channel0_main.bcs?channel=0&stream=0&user=frigate&password=pass
       #   input_args: -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1 -f live_flv
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/gartentor_sub?video=copy
          input_args: preset-rtsp-restream
       # - path: rtmp://vorgarten.domain/bcs/channel0_sub.bcs?channel=0&stream=0&user=frigate&password=pass
       #   input_args: -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1 -f live_flv
          roles:
            - rtmp
            - detect

Relevant log output

2024-04-09 04:21:45.554532131  [2024-04-09 04:21:45] watchdog.gartentor             ERROR   : Ffmpeg process crashed unexpectedly for gartentor.
2024-04-09 04:21:45.554690167  [2024-04-09 04:21:45] watchdog.gartentor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-04-09 04:31:55.611642475  [2024-04-09 04:31:55] watchdog.gartentor             INFO    : gartentor exceeded fps limit. Exiting ffmpeg...
2024-04-09 04:31:55.611765495  [2024-04-09 04:31:55] watchdog.gartentor             INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-09 04:31:55.687075521  [2024-04-09 04:31:55] frigate.video                  ERROR   : gartentor: Unable to read frames from ffmpeg process.
2024-04-09 04:31:55.687205788  [2024-04-09 04:31:55] frigate.video                  ERROR   : gartentor: ffmpeg process is not running. exiting capture thread...
2024-04-09 04:32:05.689335154  [2024-04-09 04:32:05] watchdog.gartentor             ERROR   : Ffmpeg process crashed unexpectedly for gartentor.
2024-04-09 04:32:05.689479022  [2024-04-09 04:32:05] watchdog.gartentor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-04-09 04:42:15.814733673  [2024-04-09 04:42:15] watchdog.gartentor             INFO    : gartentor exceeded fps limit. Exiting ffmpeg...
2024-04-09 04:42:15.814876786  [2024-04-09 04:42:15] watchdog.gartentor             INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-09 04:42:15.911739530  [2024-04-09 04:42:15] frigate.video                  ERROR   : gartentor: Unable to read frames from ffmpeg process.
2024-04-09 04:42:15.911946432  [2024-04-09 04:42:15] frigate.video                  ERROR   : gartentor: ffmpeg process is not running. exiting capture thread...
2024-04-09 04:42:25.917425282  [2024-04-09 04:42:25] watchdog.gartentor             ERROR   : Ffmpeg process crashed unexpectedly for gartentor.
2024-04-09 04:42:25.917565266  [2024-04-09 04:42:25] watchdog.gartentor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-04-09 04:52:36.331042961  [2024-04-09 04:52:36] watchdog.gartentor             INFO    : gartentor exceeded fps limit. Exiting ffmpeg...
2024-04-09 04:52:36.331158559  [2024-04-09 04:52:36] watchdog.gartentor             INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-09 04:52:36.434382956  [2024-04-09 04:52:36] frigate.video                  ERROR   : gartentor: Unable to read frames from ffmpeg process.
2024-04-09 04:52:36.434532435  [2024-04-09 04:52:36] frigate.video                  ERROR   : gartentor: ffmpeg process is not running. exiting capture thread...
2024-04-09 04:52:46.445318611  [2024-04-09 04:52:46] watchdog.gartentor             ERROR   : Ffmpeg process crashed unexpectedly for gartentor.
2024-04-09 04:52:46.445470078  [2024-04-09 04:52:46] watchdog.gartentor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-04-09 05:02:57.157387991  [2024-04-09 05:02:57] watchdog.gartentor             INFO    : gartentor exceeded fps limit. Exiting ffmpeg...
2024-04-09 05:02:57.157554983  [2024-04-09 05:02:57] watchdog.gartentor             INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-09 05:02:57.193460072  [2024-04-09 05:02:57] frigate.video                  ERROR   : gartentor: Unable to read frames from ffmpeg process.
2024-04-09 05:02:57.193710534  [2024-04-09 05:02:57] frigate.video                  ERROR   : gartentor: ffmpeg process is not running. exiting capture thread...
2024-04-09 05:03:07.207663985  [2024-04-09 05:03:07] watchdog.gartentor             ERROR   : Ffmpeg process crashed unexpectedly for gartentor.
2024-04-09 05:03:07.207825447  [2024-04-09 05:03:07] watchdog.gartentor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
2024-04-09 05:13:18.221284167  [2024-04-09 05:13:18] watchdog.gartentor             INFO    : gartentor exceeded fps limit. Exiting ffmpeg...


2024-04-09 05:13:18.221407447  [2024-04-09 05:13:18] watchdog.gartentor             INFO    : Waiting for ffmpeg to exit gracefully...
2024-04-09 05:13:18.333393174  [2024-04-09 05:13:18] frigate.video                  ERROR   : gartentor: Unable to read frames from ffmpeg process.
2024-04-09 05:13:18.333569442  [2024-04-09 05:13:18] frigate.video                  ERROR   : gartentor: ffmpeg process is not running. exiting capture thread...
2024-04-09 05:13:28.355192737  [2024-04-09 05:13:28] watchdog.gartentor             ERROR   : Ffmpeg process crashed unexpectedly for gartentor.
2024-04-09 05:13:28.355306907  [2024-04-09 05:13:28] watchdog.gartentor             ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.


s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service go2rtc-healthcheck: stopping
2024-04-09 08:27:38.073142548  [INFO] The go2rtc-healthcheck service exited with code 256 (by signal 15)
s6-rc: info: service go2rtc-healthcheck successfully stopped
2024-04-09 08:27:41.541526296  2024/04/09 08:27:41 [alert] 113#113: worker process 521 exited on signal 9
2024-04-09 08:27:41.584923739  [INFO] Service NGINX exited with code 0 (by signal 0)
s6-rc: info: service nginx successfully stopped
s6-rc: info: service nginx-log: stopping
s6-rc: info: service frigate: stopping
s6-rc: info: service nginx-log successfully stopped
2024-04-09 08:27:41.596569610  [2024-04-09 08:27:41] frigate.app                    INFO    : Stopping...
2024-04-09 08:27:41.922395132  [2024-04-09 08:27:41] root                           INFO    : Waiting for detection process to exit gracefully...
2024-04-09 08:27:41.922618305  [2024-04-09 08:27:41] frigate.watchdog               INFO    : Exiting watchdog...
2024-04-09 08:27:41.922890239  [2024-04-09 08:27:41] frigate.ptz.autotrack          INFO    : Exiting autotracker...
2024-04-09 08:27:41.927396977  [2024-04-09 08:27:41] detector.coral                 INFO    : Signal to exit detection process...
2024-04-09 08:27:42.273398842  [2024-04-09 08:27:42] frigate.events.maintainer      INFO    : Exiting event processor...
2024-04-09 08:27:42.835456276  [2024-04-09 08:27:42] detector.coral                 INFO    : Exited detection process...
2024-04-09 08:27:43.265434554  [2024-04-09 08:27:43] root                           INFO    : Detection process has exited...
2024-04-09 08:27:43.382518132  [2024-04-09 08:27:43] frigate.comms.ws               INFO    : Exiting websocket client...

FFprobe output from your camera

N/A

Frigate stats

No response

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Multiple, this time REOLINK

Any other information that may be helpful

No response

NickM-27 · 2024-04-09T12:54:23Z

NickM-27
Apr 9, 2024
Collaborator Sponsor

a few things:

it would be good to know more information about the host os
I'd suggest not overriding global_args

8 replies

NickM-27 Apr 9, 2024
Collaborator Sponsor

one thing you could check is the version of the radeon driver on the host vs inside the frigate contianer

Trashmee Apr 9, 2024
Author

Hi @NickM-27,

I wouldn't know how to do that - and didn't use Google in the right way, it seems.
I don't have lsmod / modinfo in the Container, no glxinfo etc..?

container #

# dpkg --list |grep -i -e radeon
ii  libdrm-radeon1:amd64                 2.4.114-1+b1                   amd64        Userspace interface to radeon-specific kernel DRM services -- runtime
ii  radeontop                            1.4-2                          amd64        Utility to show Radeon GPU utilization

# rpm -qa |grep -i -e radeon -e amd
amd-gpu-firmware-20240312-1.fc39.noarch
amd-ucode-firmware-20240312-1.fc39.noarch

NickM-27 Apr 9, 2024
Collaborator Sponsor

you can use the vainfo button in the system page of frigate and run vainfo in the host

Trashmee Apr 9, 2024
Author

Yes, maybe disableing HW acceleration to check how that behaves is at least an interim solution. If i did it right (comment out hwaccel section) CPU usage is still bearable.

Container:

# vainfo
vainfo: VA-API version: 1.17 (libva 2.10.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (renoir, LLVM 15.0.6, DRM 3.57, 6.8.4-200.fc39.x86_64)

Host:

vainfo: VA-API version: 1.20 (libva 2.20.1)
vainfo: Driver version: Mesa Gallium driver 23.3.6 for AMD Radeon Graphics (radeonsi, renoir, LLVM 17.0.6, DRM 3.57, 6.8.4-200.fc39.x86_64)

NickM-27 Apr 9, 2024
Collaborator Sponsor

to be clear I use the exact same processor to run frigate along with other things in Unraid

Trashmee · 2024-04-09T13:44:22Z

Trashmee
Apr 9, 2024
Author

When commenting out the following line HW acceleration would be disabled entirely, correct?

From

  global_args: -hide_banner -loglevel warning
  hwaccel_args: preset-vaapi

  global_args: -hide_banner -loglevel warning
  #hwaccel_args: preset-vaapi

0 replies

rmahtoa · 2024-10-25T05:44:09Z

rmahtoa
Oct 25, 2024

Maybe this can help
https://github.com/ranjeetvit2012/h264-video-conversion-ffmpeg

0 replies

nejco · 2024-11-28T08:40:07Z

nejco
Nov 28, 2024

I am having the same issues. It seems removing

hwaccel_args: preset-vaapi

mitigated the issue.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Support]: container hangs due to ffmpeg beeing unresponsive - "kill -9" not working since moving [Ryzen 5700G] #10902

{{title}}

Replies: 4 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Support]: container hangs due to ffmpeg beeing unresponsive - "kill -9" not working since moving [Ryzen 5700G] #10902

Trashmee Apr 9, 2024

Describe the problem you are having

Version

Frigate config file

Relevant log output

FFprobe output from your camera

Frigate stats

Operating system

Install method

Coral version

Network connection

Camera make and model

Any other information that may be helpful

Replies: 4 comments · 8 replies

NickM-27 Apr 9, 2024 Collaborator Sponsor

NickM-27 Apr 9, 2024 Collaborator Sponsor

Trashmee Apr 9, 2024 Author

NickM-27 Apr 9, 2024 Collaborator Sponsor

Trashmee Apr 9, 2024 Author

NickM-27 Apr 9, 2024 Collaborator Sponsor

Trashmee Apr 9, 2024 Author

rmahtoa Oct 25, 2024

nejco Nov 28, 2024

Trashmee
Apr 9, 2024

Replies: 4 comments 8 replies

NickM-27
Apr 9, 2024
Collaborator Sponsor

NickM-27 Apr 9, 2024
Collaborator Sponsor

Trashmee Apr 9, 2024
Author

NickM-27 Apr 9, 2024
Collaborator Sponsor

Trashmee Apr 9, 2024
Author

NickM-27 Apr 9, 2024
Collaborator Sponsor

Trashmee
Apr 9, 2024
Author

rmahtoa
Oct 25, 2024

nejco
Nov 28, 2024