Kernel issue? #2557
Comments
|
Please give an indication of how repeatable this issue is, and what you are typically doing when it occurs. |
|
It occurs whenever I run rtorrent under tmux and when I run omxplayer. As you can see, I also tried to run atop to see what's happening, but it just did not run. The whole system becomes unresponsive and I am unable to kill -9 omxplayer. Simple reboot -f does not work either. I can only do emergency reboot via magic sysrq. |
|
I think it looks more similiar to #2555. On which Raspberry Pi can you reproduce this issue? |
|
Hardware : BCM2835 That's PI 3 B |
|
@mtausk Since this is on a 3B not a 3B+, I suspect this is unrelated to #2555, which does appears to be limited to the 3B+ model, which initially had some lockup issues. From the log above, its possible the VC4 has died in the omxplayer case, since it has failed during a bulk transfer, but not conclusive. I think it would be worth trying an rpi-update, which will get the very latest kernel and retesting. Exact circumstances of any failure, and the dmesg at the time would be useful. |
JamesH65
added
the
Waiting for external input
|
@JamesH65, thanks for your response. Attaching dmesg and syslog. Thanks. |
|
@mtausk Just got back on to this - when you say you are using rtorrent and omxplayer at the same time, are you using the Raspbian desktop and running these from separate terminal windows? Or some other way? Are you actually downloading with rtorrent at the time? |
|
It's run within tmux (terminal multiplexor similiar to screen). And the rtorrent has to be either downloading or doing checksum. |
|
I wonder if the system is running out of memory - seems to be dying trying to allocate. |
|
I was checking that and the memory seems fine. There was free memory (1/3
or more) and no swapping occuring.
However I was unable to dump the processes with gdb. I couldn't kill the
processes nor restart/shutdown.
It's annoying.
|
|
I experience very similar issue (almost the same). A QT app runs in background, which plays video thru omxplayer (the PIOmxtextures library) and when i start writing on the sd card - exact same thing happens like in the description. I can reproduce the issue pretty quick, whithin 30 minute or so. The last firmware release that works fine is 1.20171029, all others have this issue. The problem I have are on PI2. Not sure if PI3 is affected. If you need any other info or tests, I can do it, just let me know. |
|
The problem does seem to be a stalled CMA allocation. Can you try with "cma=64M" added to /boot/cmdline.txt? |
|
Hmm - have you tried with a kernel newer than 4.14.73 (October 3rd)? A patch went in that changed the VCHIQ driver to use a pool of CMA buffers rather than continuously allocating and deallocating. |
|
the last kernel I tried is 4.14.71 from the 1.20180924 release. But I will test the latest soon. I am testing now with "cma=64M" and will let you know soon how it goes. |
|
I suspect it will run for longer, but will eventually die due to CMA fragmentation. |
|
So cma=64M helps a bit, but doesn't fix it completely. Will let you know how 4.14.79 goes. |
|
So 4.14.79 seems to fix the issue so far, but I can not do extensive tests because other issues get in the way:
Both issues seem to happen only at boot (to be confirmed). @pelwell , do you know what is the oldest raspberrypi/firmware commit that contains the fix you mentioned? I hope, may be an older commit will lack the new found bugs. |
|
While doing more tests, i got a case where a PI did not boot and had the following:
So the problem is not completely gone. |
|
Although not listed in the commit message, I'm sure the fix is in the 4.14.74 release (rpi-update release Hexxeh/rpi-firmware@e0a0ba8). |
|
So far the release fbad6408c4596d3d worked fine on these revisions: But failed on this (hung tasks):
|
|
@dimovnike Could you please post the output in case of hung task? Is it always the same? |
|
attached is an example of PI3B+ output. Please note that some lines might not be in order (this is actually elasticsearch output) as for sameness I don't know yet, as it happened only once until now. |
|
I can't think of a reason why 3B+ would behave any differently with respect to CMA allocations. If I'm not mistaken, that looks like a custom kernel you are running - can you try with an "official" firmware release? |
yes its a custom recompiled kernel but its almost the same (I will try with the official) however it might be the sdcard problem because I moved that card to a PI2 and now that PI is causing hung tasks. I think more tests are needed so I keep testing. |
|
I really think the SD card and driver is an innocent bystander here - collateral damage - but there's no harm in trying another card (except your time). |
|
@dimovnike In case you customized the kernel config, please attach it to this issue. |
the config is not yet customized, its a standard config, just recompiled kernel (it will be customized later). |
|
So after replacing SD card, hung tasks are gone, tested for 24+ hours on 6 PI's (PI2 and PI3). Thanks for the help! |
|
Can you confirm that these error messages are harmless? (from different PI's):
|
|
Yes - the probe deferrals are an ugly but expected part of unordered device creation, and the firmware load failures are from the newer in-kernel mechanism, where Pis currently use hciattach. |
|
My initial problem has gone away. Although, I don't know what helped, as it works for some time already and I also used cma=64M as suggested above. |
|
Closing this issue as questions answered/issue resolved. |
Similiar to #1587
Not sure which update caused this.
Linux pi 4.14.17-v7+ #1090 SMP Mon Feb 5 21:02:18 GMT 2018 armv7l armv7l armv7l GNU/Linux
Ubuntu 16.04
The text was updated successfully, but these errors were encountered: