demo.py & examples fail with "Failed to obtain all builtin devices"

[mivola]

Hi Andrey,

first of all: thanks a lot for all the effort you put into this library & the related Youtube videos! Those got me triggered in buying a Raspi Zero W and a BLED112 (for testing on Windows) and trying to learn Python :-)

Since I'm a complete noob to Python and this whole BLE topic, I had a hard time to setup the library and its requirements. I didnt get it to work on Windows (using PyCharm), but on the Raspi I'm now at a point where I think I have everything correctly setup - but still cannot "talk" to the Lego Boost.

I'm using Python3 and installed bluepy and the pylgbst lib:

sudo pip3 install https://github.com/undera/pylgbst/archive/0.10.tar.gz
sudo pip3 install bluepy

I can use bluetoothctl to discover, pair and connect to the Boost Hub. However, when I try to run the demo.py, I get the following error:

pi@zeropi01:~/projects/pylgbst/examples $ sudo python3 demo.py
INFO:root:Trying get_connection_bluepy
INFO:comms-bluepy:Discovering devices...
INFO:comms-bluepy:Discovering devices...
INFO:comms-bluepy:Discovering devices...
INFO:comms-bluepy:Discovering devices...
INFO:root:Found LEGO Move Hub at 00:16:53:aa:7b:f2
INFO:movehub:Attached IMOTOR on port C
INFO:movehub:Attached DISTANCE_COLOR_SENSOR on port D
INFO:movehub:Attached MOTOR on port A
INFO:movehub:Attached MOTOR on port B
INFO:movehub:Attached MOTOR on port AB
INFO:movehub:Attached LED on port LED
WARNING:movehub:Got only these devices: (LED on port LED, EncodedMotor on port A, EncodedMotor on port B, EncodedMotor on port AB, None, Button on port 0, None, None)
Traceback (most recent call last):
  File "demo.py", line 194, in <module>
    hub = MoveHub(connection)
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/movehub.py", line 63, in __init__
    self._wait_for_devices()
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/movehub.py", line 83, in _wait_for_devices
    raise RuntimeError("Failed to obtain all builtin devices")
RuntimeError: Failed to obtain all builtin devices

I have all motors and the sensor connected. I also tried without any connected motors/sensors, but the error is the same.

I can launch the DebugServer as described in the readme, and that starts successfully - but running demo.py still fails:

pi@zeropi01:~/projects/pylgbst/examples $ sudo python3 demo.py
Traceback (most recent call last):
  File "demo.py", line 194, in <module>
    hub = MoveHub(connection)
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/movehub.py", line 63, in __init__
    self._wait_for_devices()
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/movehub.py", line 71, in _wait_for_devices
    self.connection.enable_notifications()
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/comms/__init__.py", line 41, in enable_notifications
    self.write(ENABLE_NOTIFICATIONS_HANDLE, ENABLE_NOTIFICATIONS_VALUE)
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/comms/__init__.py", line 163, in write
    self._send(payload)
  File "/usr/local/lib/python3.5/dist-packages/pylgbst/comms/__init__.py", line 167, in _send
    self.sock.send(json.dumps(payload) + "\n")
TypeError: a bytes-like object is required, not 'str'

Any idea what I'm doing wrong here? I also cloned your repo to the Raspi, so I'd happily try any suggested code change (if you tell me how to use the local repo instead of the installed pip3 pylgbst lib... ;-)

Furthermore I tried to run the android_remote.py but that fails with another error message:

pi@zeropi01:~/projects/pylgbst/examples $ sudo python3 vernie/android_remote.py
Traceback (most recent call last):
  File "android_remote.py", line 12, in <module>
    from . import Vernie
SystemError: Parent module '' not loaded, cannot perform relative import

What do I need to do to make that work?

Regards
Michael

[undera]

Hi,
The builtin devices message is about some things Hub expects to appear at the start. It seems you don't get Voltage and Currrent sensors up, and also internal Tilt sensor.
I have no idea why your hardware part misses these sensors, maybe it's a new HW version where they decided these sensors are not needed. I'd agree, I found no use to these sensors myself.
To fix issue, take this line: https://github.com/undera/pylgbst/blob/master/pylgbst/movehub.py#L75
And remove tilt, voltage and amperage sensors from the list:

 builtin_devices = (self.led, self.motor_A, self.motor_B, self.motor_AB, self.button)

This way it will pass startup health check.

[mivola]

Hi Andrey,

thanks a lot for your help! I changed this line, but when I start the demo, it still fails with the same message. I guess I need to publish/install a new version of the pylgbst module using "pip3 install ..."?
Or is there a way to use the local sources directly without the need to install a module?

Michael

[undera]

Two ways:

1. Do pip3 install <path to root of local pylgbst>
2. set PYTHONPATH environment variable to point to your version of pylgbst so python finds it first.

I did not want to bother you with subclassing way, where you can create a class that would inherit from MoveHub and only will override method that waits for devices to attach. This way would be the most elegant, though.

[mivola]

OK, reinstalling using pip3 works when using "--upgrade":
/projects $ sudo pip3 install --upgrade ./pylgbst/

I still have a problem using PYTHONPATH which is related to another problem: I can only connect to the MoveHub with sudo. Running it as user "pi" fails:

pi@zeropi01:~/projects $ python3 pylgbst/examples/demo.py
INFO:root:Trying get_connection_bluepy
INFO:comms-bluepy:Discovering devices...
INFO:root:Trying get_connection_bluegiga
INFO:root:Trying get_connection_gatt
INFO:root:Trying get_connection_gattool
INFO:root:Trying get_connection_gattlib
Traceback (most recent call last):
  File "pylgbst/examples/demo.py", line 188, in <module>
    connection = DebugServerConnection()
  File "/home/pi/projects/pylgbst/pylgbst/comms/__init__.py", line 146, in __init__
    self.sock.connect(('localhost', port))
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pylgbst/examples/demo.py", line 191, in <module>
    connection = get_connection_auto()
  File "/home/pi/projects/pylgbst/pylgbst/__init__.py", line 58, in get_connection_auto
    raise Exception("Failed to autodetect connection, make sure you have installed prerequisites")
Exception: Failed to autodetect connection, make sure you have installed prerequisites

Now, when running it with sudo (even with -E), the PYTHONPATH variable is not used:

pi@zeropi01:~/projects $ sudo -E python3 pylgbst/examples/demo.py
Traceback (most recent call last):
  File "pylgbst/examples/demo.py", line 5, in <module>
    from pylgbst import *
ImportError: No module named 'pylgbst'

Any idea how to solve this?

Well, now to the actual problem with BLE connection/communication. After some trial & error I found the following:

• the Hub does indeed have all the sensors mentioned in builtin_devices
• but they are not (or at least not always) found at startup
  => I had 2 successful starts in ~50 attempts
• I increased the sleep time https://github.com/undera/pylgbst/blob/master/pylgbst/movehub.py#L81 but that didnt help immediately, but it gave me some time to connect/disconnect a sensor/motor from the hub and then all of a sudden the missing sensors had been found:

pi@zeropi01:~/projects/pylgbst $ sudo python3 examples/demo.py
INFO:root:Trying get_connection_bluepy
INFO:comms-bluepy:Discovering devices...
INFO:root:Found LEGO Move Hub at 00:16:53:aa:7b:f2
INFO:movehub:Attached IMOTOR on port C
INFO:movehub:Attached DISTANCE_COLOR_SENSOR on port D
INFO:movehub:Attached MOTOR on port A
INFO:movehub:Attached MOTOR on port B
INFO:movehub:Attached MOTOR on port AB
INFO:movehub:Attached LED on port LED
INFO:movehub:Attached TILT_SENSOR on port TILT_SENSOR
INFO:movehub:Attached AMPERAGE on port AMPERAGE
INFO:movehub:Attached VOLTAGE on port VOLTAGE
INFO:movehub:Detached ColorDistanceSensor on port D
INFO:movehub:b'LEGO Move Hub' by b'LEGO System A/S'
INFO:movehub:Detached EncodedMotor on port C
INFO:movehub:Voltage: 71%
INFO:movehub:Attached DISTANCE_COLOR_SENSOR on port D

Another way to make it work was to remove all (or: all undiscoverable) sensors from the builtin_devices in https://github.com/undera/pylgbst/blob/master/pylgbst/movehub.py#L75:

pi@zeropi01:~/projects $ sudo -E python3 pylgbst/examples/demo.py
INFO:root:Trying get_connection_bluepy
INFO:comms-bluepy:Discovering devices...
INFO:root:Found LEGO Move Hub at 00:16:53:aa:7b:f2
INFO:movehub:Attached IMOTOR on port C
INFO:movehub:Attached DISTANCE_COLOR_SENSOR on port D
INFO:movehub:Attached MOTOR on port A
INFO:movehub:Attached MOTOR on port B
INFO:movehub:Attached MOTOR on port AB
INFO:movehub:Attached LED on port LED
INFO:movehub:Attached TILT_SENSOR on port TILT_SENSOR
INFO:movehub:Attached AMPERAGE on port AMPERAGE
INFO:movehub:Attached VOLTAGE on port VOLTAGE
INFO:movehub:b'LEGO Move Hub' by b'LEGO System A/S'
INFO:movehub:Voltage: 71%

I'm not sure why my MoveHub behaves so strange, but at least I got it working now :-)

My next topic would be to get android_remote.py to work, but that fails with this error message:

pi@zeropi01:~/projects/pylgbst/examples $ sudo python3 vernie/android_remote.py
Traceback (most recent call last):
  File "android_remote.py", line 12, in <module>
    from . import Vernie
SystemError: Parent module '' not loaded, cannot perform relative import

What do I need to do to make that work?

Thanks a lot!
Michael

[undera]

The problem is with PYTHONPATH. You need to add source root to Pythonpath so interpreter will find the modules

[mivola]

Regarding PYTHONPATH: I tried the following:

pi@zeropi01:~/projects/pylgbst $ echo $PYTHONPATH
/home/pi/projects/pylgbst/pylgbst/
pi@zeropi01:~/projects/pylgbst $ ls -lsa $PYTHONPATH
total 72
 4 drwxr-xr-x 4 pi pi  4096 Mar 15 12:22 .
 4 drwxr-xr-x 6 pi pi  4096 Mar 12 09:28 ..
 4 drwxr-xr-x 3 pi pi  4096 Mar 15 10:15 comms
 4 -rw-r--r-- 1 pi pi  2646 Mar 12 09:28 constants.py
 4 -rw-r--r-- 1 pi pi  2738 Mar 12 09:38 constants.pyc
 4 -rw-r--r-- 1 pi pi  1895 Mar 12 09:28 __init__.py
 4 -rw-r--r-- 1 pi pi  2499 Mar 12 09:38 __init__.pyc
12 -rw-r--r-- 1 pi pi  8876 Mar 15 12:22 movehub.py
20 -rw-r--r-- 1 pi pi 16686 Mar 12 09:28 peripherals.py
 4 drwxr-xr-x 2 pi pi  4096 Mar 15 12:05 __pycache__
 4 -rw-r--r-- 1 pi pi   527 Mar 12 09:28 utilities.py
 4 -rw-r--r-- 1 pi pi   953 Mar 12 09:38 utilities.pyc
pi@zeropi01:~/projects/pylgbst $ python3 examples/demo.py
Traceback (most recent call last):
  File "examples/demo.py", line 5, in <module>
    from pylgbst import *
ImportError: No module named 'pylgbst'

I also tried to set PATHONPATH to /home/pi/projects/pylgbst but the error is the same. Why do I need to use sudo to establish a connection in the first place?

Regarding the android_remote.py: can you advise me how to get it working? My son is desperate to see Vernie move according to the movements of the phone/tablet ...

Thanks a lot!
Michael

[undera]

Documentation clearly states that some of BLE backend libs might require sudo: https://github.com/undera/pylgbst#bluetooth-backend-prerequisites . I did not develop those libs and can't explain why.

I have made small change to android_remote.py, should help with import error.

[mivola]

Andrey, thanks again for your great support!
With the change in android_remote.py, it works well in Windows. Unfortunately it still fails on the Pi Zero. With sudo I still get the same error, when run as user "pi" however, the module is found (just to fail with the known problem in bluepy):

pi@zeropi01:~/projects/pylgbst/examples $ sudo python3 ./vernie/android_remote.py
Traceback (most recent call last):
  File "./vernie/android_remote.py", line 12, in <module>
    from examples.vernie import Vernie
ImportError: No module named 'examples'

pi@zeropi01:~/projects/pylgbst/examples $ python3 ./vernie/android_remote.py
INFO:comms-bluepy:Discovering devices...
Traceback (most recent call last):
  File "./vernie/android_remote.py", line 53, in <module>
    robot = Vernie()
  File "/home/pi/projects/pylgbst/examples/vernie/__init__.py", line 73, in __init__
    conn = get_connection_bluepy()
  File "/home/pi/projects/pylgbst/pylgbst/__init__.py", line 35, in get_connection_bluepy
    return BluepyConnection(controller).connect(hub_mac)
  File "/home/pi/projects/pylgbst/pylgbst/comms/cbluepy.py", line 101, in connect
    scanner.scan(1)
  File "/home/pi/.local/lib/python3.5/site-packages/bluepy/btle.py", line 852, in scan
    self.start(passive=passive)
  File "/home/pi/.local/lib/python3.5/site-packages/bluepy/btle.py", line 790, in start
    self._mgmtCmd("le on")
  File "/home/pi/.local/lib/python3.5/site-packages/bluepy/btle.py", line 312, in _mgmtCmd
    raise BTLEManagementError("Failed to execute management command '%s'" % (cmd), rsp)
bluepy.btle.BTLEManagementError: Failed to execute management command 'le on' (code: 20, error: Permission Denied)

Any idea why the module is not found as sudo? The pylgbst module is properly installed and the demo.py works as well:

pi@zeropi01:~/projects/pylgbst/examples $ sudo pip3 list pylgbst
bluepy (1.3.0)
chardet (2.3.0)
cryptography (1.7.1)
idna (2.2)
keyring (10.1)
keyrings.alt (1.3)
pip (9.0.1)
pyasn1 (0.1.9)
pycrypto (2.6.1)
pygobject (3.22.0)
pylgbst (0.10)
python-apt (1.1.0b5)
pyxdg (0.25)
requests (2.12.4)
SecretStorage (2.3.1)
setuptools (33.1.1)
six (1.10.0)
ssh-import-id (5.6)
urllib3 (1.19.1)
wheel (0.29.0)

Thanks a lot!
Michael

[undera]

Probably, inside sudo-ed shell, the PYTHONPATH is not set. One way to try is to run sudo bash which will give you sudo-ed shell, and from that shell to set PYTHONPATH and then run python.
Sorry, I don't have RaspberryPi to try this myself.

[mivola]

Thank you for the hint about "sudo bash" - this works well :-)

And I even found a ticket for the issue why sudo is required in the first place:
IanHarvey/bluepy#313
IanHarvey/bluepy#190

Unfortunately non of the suggested solutions works for me :-(

I think we covered enough topics here now and can close this issue now.

Thanks again!

[undera]

Good stuff!