bluepy-helper crashes

[Trashwarez]

bluepy-helper crashes on a PI 4 after some time. Sometimes a few hours, sometimes longer. Quite difficult to track. bluepy-helper then uses 100 % CPU and connection to this MAC address. Watchdog does not seem to work.

[JsBergbau]

Can you post your command line and log of script output while this error occurs?

[Trashwarez]

I run the following script from cron.hourly:

`#!/bin/bash

Read Mijia bluetooth temperature sensors. Trashwarez, 28.7.2020

(
/usr/bin/flock -w 5 -x 99 || exit 1
/usr/bin/logger "Abfrage der Bluetooth-Temperatursensoren gestartet"
/usr/local/bin/LYWSD03MMC.py -c 1 -d 11:11:11:11:11:11 -b 1000 -r --debounce --unreachable-coount 25 --skipidentical 50 --name Mijia-Bluez-1 --callback voodoo-temperature-mqtt
/usr/local/bin/LYWSD03MMC.py -c 1 -d 22:22:22:22:22:22 -b 1000 -r --debounce --unreachable-coount 25 --skipidentical 50 --name Mijia-Bluez-2 --callback voodoo-temperature-mqtt
/usr/local/bin/LYWSD03MMC.py -c 1 -d 33:33:33:33:33:33 -b 1000 -r --debounce --unreachable-coount 25 --skipidentical 50 --name Mijia-Bluez-3 --callback voodoo-temperature-mqtt
/usr/local/bin/LYWSD03MMC.py -c 1 -d 44:44:44:44:44:44 -b 1000 -r --debounce --unreachable-coount 25 --skipidentical 50 --name Mijia-Bluez-4 --callback voodoo-temperature-mqtt
/usr/bin/logger "Abfrage der Bluetooth-Temperatursensoren beendet"
/bin/rm /var/run/LYWSD03MMC.lck
) 99>/var/run/LYWSD03MMC.lck

exit
`
The callback of /usr/local/bin/voodoo-temperature-mqtt sends the values to openHAB.
Works fine for some hours. But gets hung up as mentioned above.
It does not show up any problems. It just gets stuck at "Trying to connect to 11:22:33:44:55:66".

Top shows then:

` " PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

29309 root 20 0 3744 672 584 R 99,7 0,0 21:36.03 bluepy-helper`

Killing the process leaves this:

Exception ignored in: <function Peripheral.__del__ at 0xb659ca08> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 630, in __del__ self.disconnect() File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 453, in disconnect self._writeCmd("disc\n") File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 305, in _writeCmd self._helper.stdin.flush() BrokenPipeError: [Errno 32] Broken pipe

[JsBergbau]

Please pipe the whohle output to a logfile and post it here when it crashes.

When it gets stuck at "Trying to connect" is killing bluepy really sufficient? On one RPi I have the problem that I have to reboot it completely otherwise no sensors are polled at all.

[Trashwarez]

As already told above it just gets stuck. NO MORE OUTPUT

Trying to connect to 11:11:11:1:11:11 Temperature: 25.4 Humidity: 52 Battery voltage: 3.11 Battery level: 100 1 measurements collected. Exiting in a moment. /usr/local/bin/voodoo-temperature-mqtt sensorname,temperature,humidity,voltage,batteryLevel,timestamp Mijia-Bluez-1 25.4 52 3.11 100 1596210573 Trying to connect to 22:22:22:22:22:22 Connection lost Waiting... Trying to connect to 22:22:22:22:22:22 Connection lost Waiting... Trying to connect to 22:22:22:22:22:22

It just is stuck then.

And yes, it works fine again after having killed bluepy. No reboot or hci0 up / down necessary.

The script just continues after having killed the process and even finally succeeds on the connection...

Exception ignored in: <function Peripheral.__del__ at 0xb65a0a08> Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 630, in __del__ self.disconnect() File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 453, in disconnect self._writeCmd("disc\n") File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 305, in _writeCmd self._helper.stdin.flush() BrokenPipeError: [Errno 32] Broken pipe Waiting... Trying to connect to 22:22:22:22:22:22 Connection lost Waiting... Trying to connect to 22:22:22:22:22:22 Connection lost Waiting... Trying to connect to 22:22:22:22:22:22 Temperature: 25.1 Humidity: 54 Battery voltage: 3.027 Battery level: 93 1 measurements collected. Exiting in a moment. /usr/local/bin/voodoo-temperature-mqtt sensorname,temperature,humidity,voltage,batteryLevel,timestamp Mijia-Bluez-2 25.1 54 3.027 93 1596211181 Trying to connect to 33:33:33:33:33:33

[JsBergbau]

Hi Trashwarez,

thanks for the log. That helps debugging the problem. It seems bluepy-helper is not killed/terminated even the program has code for that case. You seem to run the script as root user, that seems to cause the problem. Querying the sensor doesn't require root priviledges and thus it shouldn't be executed as root. Executing as normal user worked perfect in my test cases.

For your usecase and problem I suggest adding a line after /usr/bin/logger "Abfrage der Bluetooth-Temperatursensoren beendet"
sudo killall bluepy-helper

[Trashwarez]

Running the script as root does not cause the problem because it happens on other users as well, e.g. user pi:

4293 pi 20 0 3744 664 580 R 99,7 0,0 4:10.30 bluepy-helper

And the suggested "sudo killall bluepy-helper" will never be executed when the python script gets stuck in the middle by bluepy-helper.

[JsBergbau]

I think the stuck bluepy helper is from a previous execution of the script. My scripts run 24/7 and never hang because of bluepy-helper. So when you kill any remaining bluepy-helper after the first execution round of all scripts it should work.

[Trashwarez]

As it just got stuck on the very first run after reboot as user pi, I killed it. Let's hope that your recommendation works on later cron calls.

Edit: No, it does not work. It even got stuck after second call using user pi:

3465 pi 20 0 3744 628 540 R 99,7 0,0 0:14.23 bluepy-helper

What is your system on which the script runs stable 24/7? Mine is a Pi 4 with 4 GB running on raspbian buster with latest updates. Does your script run like mine 4 times in a row, requesting data from 4 sensors?

[JsBergbau]

My system is a RPi Zero W currently running Raspbian Buster 4.19.93+ the other stable system is running 4.19.97+
The start the script for each sensor at boot and then the script runs all the time, no more execution via cron, only once at boot.

RPi4 has compared to Zero W a reduced Bluetooth range. Could cause problems, but I don't think in your case.

RPi4 has a lot of problems with Bluetooth and or Wifi according to this RPi-Distro/firmware-nonfree#8
You can try to install one of those mentioned bluez packages and test. I once had the problem with the RPi Zero that bluetooth range was massively reduced with a bluezupdate and so I found out about this issue and now after every update I test thoroughly whether the bluetooth sensors collect data accordingly.

On a PI 4 which I used to update the script in the last few days, I'm on 4.19.118-v7l+ There I've only polled for a few seconds, but it worked.

bluez-firmware/testing,now 1.2-4+rpt3 all [installed,automatic]
bluez/testing,now 5.50-1.2~deb10u1+rpt1 armhf [installed]
pi-bluetooth/testing,now 0.1.13 all [installed]

Perhaps that helps. But as I only used for a few seconds, I recommend trying other bluez versions.

[Trashwarez]

Hmm. Here is running kernel 5.4.51-v7l+
So, it really seems to be the hardware or the kernel.

Your sentence "The start the script for each sensor at boot and then the script runs all the time, no more execution via cron, only once at boot." tells me that we are using a different approach.
I am calling your routine each hour via cron four times to read all four sensors. How do you call your routines to keep them up 24/7 w/o execution via cron?

[JsBergbau]

Try another bluez version, like described in the linked github issue.

Yeah thats a different approach.
I use that script. One for every sensor which its name. So the script never exits and needs only to be called once.

#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
ort=$(basename -s .sh $0)
$DIR/LYWSD03MMC.py -d 11:22:33:44:55:66 -2p -p2 75 -o2 -4 -p1 33 -o1 -3 -r --debounce --name $ort --callback sendToInflux.sh

[longos]

I've the same issue, when call /usr/bin/python3 /home/pi/domoticz/scripts/python/MiTemperature2/LYWSD03MMC.py -d A4:C1:38:12:51:85 -r -deb -b -c 1 -urc 2 -call ./sendToDomoticz.py 2>&1 every 30 minutes in crone as root.
Sometimes it's broken after few calls, sometimes after few days. Always bluepy-helper loads 100% CPU.

[JsBergbau]

I suggest trying the new ATC firmware mode #44