Kernel OOPS under load on camera video capture - bcm2835_v4l2 driver #2387
Comments
|
I have an important update on this: It seems that the issue only happens if the 'nice' value of the gstreamer process is +5 or more. In this case, I can reproduce the issue in a few seconds. If I use a nice value of +4 or less, I could not reproduce the issue until now. I was using a high nice value for the streaming process (started with +19, and tried lower values by binary search until I got to the +4), in order for the streaming to not disturb more important services like Octoprint (which sends commands to a 3D printer and must not be disturbed) |
|
That sounds strange if it is failing in videbuf2 allocation as that is standard code. That also shouldn't be being called whilst streaming, which is the implication of queue_setup being in the middle at all those buffer_cb lines. Two thoughts:
|
|
Thank you for your response. When letting the V4L2 driver spit out H264 720p25, the lag is around 15 seconds and the image is quite choppy (but the CPU utilization is very low), while when using the omxh264enc on raw camera output, the lag is around 300 milliseconds and the image is fluid (and the CPU utilization is around 20%). Is this normal ? Or maybe I am using the wrong settings ? About the driver compilation, I will try to do it over the next days. I am now using kernel 4.14.18+ (same issue). By the way, here is my gstreamer pipeline using h264 output from v4l2. Maybe I am doing something wrong ? I used this: gst-launch-1.0 -v v4l2src device=/dev/video0 do-timestamp=true ! video/x-h264,width=1280,height=720,framerate=25/1,bitrate=1500000 ! h264parse ! rtph264pay name=pay0 config-interval=10 pt=96 ! udpsink host=0.0.0.0 port=5004 and also this (same 15-second lag) raspivid -t 0 -h 720 -w 1280 -fps 25 -hf -b 1500000 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay name=pay0 config-interval=10 pt=96 ! udpsink host=0.0.0.0 port=5004 |
|
There's nothing in the firmware that would be introducing that sort of latency. Use v4l2-ctl and it'll be down at the normal 80ish ms, so there's nothing fundamentally wrong in the V4L2 driver. One thing to try with the raspivid line is adding I'm wondering if GStreamer is trying to increase the number of buffers that it has requested from V4L2 whilst streaming. That would explain the CREATE_BUFS ioctl call whilst streaming. It'll take a modest amount of digging in to the GST code to see if that is what is going on, and I'm not sure if it is a valid thing to do within V4L2 (need to reread the spec). |
|
I have tried adding the -fl flag to raspivid and nothing has changed. Is there any possibility that the v4l2 driver does software instead of hardware encoding ? Like, maybe (due to some strange issue) not finding the required capabilities for hardware encoding and silently falling back to software ? The choppiness and lag almost feel like software encoding taking place. The low CPU utilization in this case may be because the kernel activity is not reflected in the CPU stats (not sure about this thing though). However, the more worrying issue is that in this case (v4l2 doing the H264 encoding) I get another kernel OOPS (reproduced it multiple times), and it does not depend on the process priority. Below you can find the trace (unfortunately the debug flag is not enabled on the driver, please let me know if it would help you): Thanks ! |
|
Hello, I have compiled the kernel from the branch you have indicated and used your latest driver modifications and the flags in config.txt and cmdline.txt. Unfortunately, running the raspivid -> gstreamer pipeline gets the same results (extremely choppy video, a fraction of a second of movement every 10-15 seconds or so) and running the gstreamer with omx264enc doesn't work because the device /dev/video0 doesn't seem to exist (but a device probably does exist, since the raspivid works. Please note, vcgencmd get_camera reports detected=1 and supported=1.). ERROR: Pipeline doesn't want to pause. Please let me know if you would like me to try different things/new source code/debug flags, etc. |
|
Hello, I think I have solved the problem, at least I have a good solution. The extremely choppy video was caused by the "do-timestamp=true" flag in the v4l2src parameters for gstreamer. Once I removed that, everything worked great. The raspivid -> gstreamer pipeline still does not work. I think it is also connected with the above issue. The problem seems to be that gstreamer sees the framerate as being "0/1" upstream instead of the correct "25/1" in my case. I'm not sure what raspivid does that causes this, I have even compiled the latest available version with no luck. So, for reference and those needing a good pipeline for gstreamer on Pi Zero W and camera V2, this is the best one that works for me (with only 5-6% CPU usage on Pi Zero W): gst-launch-1.0 -v v4l2src ! video/x-h264,width=1280,height=720,framerate=25/1,profile=high,bitrate=1850000 ! h264parse ! rtph264pay name=pay0 config-interval=10 pt=96 ! udpsink host=127.0.0.1 port=5004 |
|
Not having /dev/video0 means something in the setup didn't work. You'd need to go through the device tree logs to confim that the relevant node under /soc/v4l2 (you can either check the firmware logs by adding "dtdebug=1" to /boot/config.txt, and using The bcm2835-camera V4L2 driver will never do software encoding - the code simply isn't there to do it.
That doesn't seem like a sensible option to use unless you have reason to distrust the timestamps from the source component. AFAIK the V4L2 driver is producing sensible timestamps, so removing them will potentially add A/V sync issues as you're throwing away any information on processing latency. raspivid->gstreamer framerate is likely to be down to the fact that storing a frame rate in H264 headers is optional. raspberrypi/firmware#245 added a parameter MMAL_PARAMETER_VIDEO_ENCODE_SPS_TIMING / OMX_IndexParamBrcmVideoAVCSPSTimingEnable to insert the timing section into the H264 SPS header. It's off by default, but a quick hack to raspivid function may result in GStreamer's h264parse being happier and picking up a frame rate. Either that or you should be able to override the frame rate in the GStreamer I'm glad you have a workaround, but please leave this issue open until I've had a chance to look into what GStreamer is actually up to with queue_setup whilst streaming. |
|
Hello 6by9, Thanks a lot for your insights on the issue. I now understand it a bit better. When I will have a bit of free time I will re-compile raspivid with the code you suggested and I will report here the outcome. About the custom kernel, I kind of gave up because it seemed like it was going nowhere (same issues). |
|
@6by9 Any recently changed on this, or leave open? |
JamesH65
added
the
Waiting for internal comment
|
Leave open for now. |
JamesH65
removed
the
Waiting for internal comment
Hello everyone,
I am using a Raspberry Pi Zero W with camera module V2 and the official 5.1V/2.5A PSU (tried other PSUs too - same result), and I'm trying to capture hardware-accelerated H264 video at 720p/25fps, via gstreamer/omx/v4l2, and the stream is made available in web browsers via Janus WebRTC Gateway. All the software used is compiled at the latest version available on GitHub (or via apt-get dist-upgrade).
Things work fine until enough load is applied to the Pi Zero W device. For example, starting Octoprint server (consumes a lot of CPU while starting up) and opening a second stream session in an Android Firefox browser) is sure to consistently trigger the issue.
A kernel OOPS error is generated, always at the same function with the same stack. The system remains active but camera capture is not possible again until reboot. I am thinking that if it was a hardware failure of some sort it would happen at slightly different points in the source code. Below you can find the relevant kernel logs (I have also activated debug flag on the bcm2835_v4l2 driver):
My gstreamer pipeline looks like this:
gst-launch-1.0 -v v4l2src device=/dev/video0 do-timestamp=true ! video/x-raw,width=1280,height=720,framerate=25/1 ! omxh264enc target-bitrate=1500000 control-rate=1 ! video/x-h264,profile=high ! h264parse ! rtph264pay name=pay0 config-interval=10 pt=96 ! udpsink host=0.0.0.0 port=5004
Thanks for looking into it !
The text was updated successfully, but these errors were encountered: