[Feature] Broadcast example for other devices

[pbochynski]

With broadcast implementation, you opened a lot more possibilities than described in the initial comment on this issue. The publish-subscribe model is much better for integrating many hubs than a direct connection. I tried already simple scenario of sharing sensor data between hubs, and now I have a few other ideas, but wanted first to double check with you if it makes sense and how hybrids project can support it.

1. I started with reading advertisement data from hubs to see the messages published by Broadcast.send on my PC using the bleak library you introduced in pc-communication example. I now analyze the code in the PR to decode the data, but as both BLE and micropython code are new to me it will take some time ;). Maybe you already have some code snippet that you can share? Something like python/bleak implementation of Broadcast receive.
2. Receiving should be easy, but sending probably requires something more (bleak is just the client). What about sending data from non-lego devices in the same format (using broadcast)? I was thinking about using ESP32 with different sensors (like this ultrasonic sensor) to communicate with lego hub. I could build a 4-direction ultra-precise wireless distance sensor for less than $5 and use it instead overpriced Lego 45604 ($45).

What do you think?

[NStrijbosch]

2. What about sending data from non-lego devices in the same format (using broadcast)?

This is definitely a possibility. Both sending and receiving. The protocol implemented follows the format used by the hub-to-hub communication blocks in the official MINDSTORMS app. This means each message is broadcasted by setting the advertisement data as:

0x0C, 0xFF, 0x97, 0x03, 0x01, 0x1D, 0xC6, 0x73, 0x49, 0x61, 0x62, 0x63, 0x64
  ^     ^     ^     ^     ^     ^     ^     ^     ^     ^
  |     |     |_____|     |     |_________________|     |_________________
  |     |     |           |     |                     Transmitted data (number of bytes depends on the length of the transmitted data, the maximum is 23 bytes)
  |     |     |           |    Topic name encoded as a CRC32 hash
  |     |     |         Index of the message (should be increased by 1 for a new message)
  |     |     LEGO Company identifier
  |     Indicating manufacturer data
  Length (number of bytes that follow, not including this one)

Receiving should indeed be trivial. For sending you need to be able to set the full advertisement message. This should be possible for an ESP32.

[pbochynski]

Thanks for the detailed description of the advertisement data. I will try to use it and will share the example code when I succeed.

[pbochynski]

@NStrijbosch I am not able to get transmitted data on my computer. I am able to scan the device but all the libraries I used do not report any manufacturer data. One example with bleak:

import asyncio
from bleak import BleakScanner

async def main():
    devices = await BleakScanner.discover(timeout=3,return_adv=True)
    for d in devices:
        _ , adv = devices[d]
        print(adv)
asyncio.run(main())

For my technic hub with the custom firmware and code sending pitch and roll on tilt topic the output is like this:

AdvertisementData(local_name='legoble', 
     service_data={'00002a50-0000-1000-8000-00805f9b34fb': b'\x01\x97\x03\x80\x00\x00\x00'}, 
     service_uuids=['c5f50001-8280-46da-89f4-6d8051e4aeef'], 
     tx_power=0, 
     rssi=-35)

It doesn't contain manufacturer data at all but I see x97 x03 sequence in the service data. I tried to use different library (https://github.com/abandonware/noble/blob/master/examples/advertisement-discovery.js), but the effect is similar:

{
  "localName": "legoble",
  "serviceData": [
    {
      "uuid": "2a50",
      "data": {
        "type": "Buffer",
        "data": [
          1,
          151,
          3,
          128,
          0,
          0,
          0
        ]
      }
    }
  ]
}

I know the sender code works because I have the second hub as a receiver and it gets the messages. Is it possible that the data is not structured properly and cannot be decoded on my computer? Manufacturer data from other devices are visible (I started experiments with ESP32). Have you tried to receive the data on other devices?

[NStrijbosch]

The times I used other devices to receive the data I had access to the raw advertisement data, e.g., using https://docs.micropython.org/en/latest/library/bluetooth.html

I did the decoding to obtain the data, topic, index myself. I was of the impression the advertisment data followed the normal conventions. But maybe it is not so conventional to only send manufacturing data...

Anyhow, given that you recognize the company identifier there must be something.

What is your topic name?
And are you using send_bytes? This could be most intuitive in order to recognize where the data ends up.

[pbochynski]

The topic name is tilt.
I used such example:

    pitch, roll = hub.imu.tilt()
    radio.send("tilt", (pitch,roll))

Full code is here: https://gist.github.com/pbochynski/c8336c885251c77ed9fbf8e75cd4defc

I tried your advice and I send raw data:

    radio.send_bytes("tilt", b"1234")

Anyway, whatever I send, whatever the topic is the received looks the same:

AdvertisementData(local_name='legoble', 
 service_data={'00002a50-0000-1000-8000-00805f9b34fb': b'\x01\x97\x03\x80\x00\x00\x00'}, 
  rssi=-48)

The only difference is that now when I send static data in the loop (always the same value 1234) the advertisement data doesn't have service_uuids array. It looks like my message and topic are not popping up through the libraries on my MacBook.
I suspect that maybe the length of the advertising data is not calculated properly and doesn't include the hub name, which is also part of the advertising data, and other libraries truncate the message just after the name.

[NStrijbosch]

Clear. Indeed somewhere in the decoding of the advertising data is going wrong. I am actually surprised that 'legoble' is there, it is not something that is explicitly advertised. Moreover, I don't recognize the crc32 hash in any of the data.

I think the only solution would be to search for a way to read the raw advertising data before the decoding in service data etc.

Just some experience I have with ble advertising with apple products:
Developers of iphone apps are not able to advertise raw messages as would be required for iPhone to pybricks communication. Apple only allows to advertise up to 2 bytes of manufacturing data. It could be that the decoding in your macbook is expecting advertising data that complies with such a strict format.

I have been in conversation with someone who has been able to send and receive data in this format from an ESP32. So it could be worth trying if you can read the advertising data from a pybricks hub on your ESP32.

[pbochynski]

legoble is how I named the hub when I flashed it with the custom firmware. The same result I've got on my Iphone with BLE scanmer:

But I will try also on ESP32.

[pbochynski]

I tried on ESP32 with this BLE scanner code:
https://raw.githubusercontent.com/nkolban/ESP32_BLE_Arduino/master/examples/BLE_scan/BLE_scan.ino

The result is a little bit better:

Advertised Device: Name: , Address: 26:31:15:f0:55:98, manufacturer data: 97034ff6893f1c6162636465
Advertised Device: Name: , Address: 26:31:15:f0:55:98, manufacturer data: 970356f6893f1c6162636465

But the message should be longer as the sender code is like this:

data = bytes("abcdefghijklmn","ascii")
print(data)
while True:
    radio.send_bytes("tilt", data)
    wait(1000)

The topic name encoded would be f6893f1c and then we have four characters 6162636465 (abcde). 9 characters are missing.

So I did another experiment and when I send shorter message:

data = bytes("abc","ascii")
radio.send_bytes("tilt", data)

the scanner reports:

Advertised Device: Name: , Address: 26:31:15:f0:55:98, manufacturer data: 97

Look! Also 9 characters missing (3 from string and 6 from message meta data).
Maybe the metadata size is not added properly to the length field?

#define PBIO_BROADCAST_META_SIZE (9)

[pbochynski]

I managed to send the data with ESP32 code in the format @NStrijbosch described. I am also able to decode that data on my computer (all data - not truncated). But what is important broadcast implementation for pybrics can read my signal from ESP!
Here is the example: https://youtu.be/HA532a0KH7U
ESP32 dev kit connected to HC-SR04 ultrasonic sensor is advertising distance as BLE manufacturer data. Lego hub with custom micro-python firmware from pybrics.com receives the signal and adjusts light color.

I will share the code soon - it requires a little bit of refactoring before publishing.

[pbochynski]

I have a prototype for micropython module for ESP32 that implements Broadcast (send and receive).
@dlech @laurensvalk Would you accept such a contribution to the pybrics-projects repository?

[laurensvalk]

Quite possibly, yes. We're still finding out the best ways to add projects to the site, though.

To save yourself some time, perhaps you could start off posting your proposed content as an issue instead of a pull request.

Once we know where everything will go, you can always just use the same markdown content for your pull request.

[pbochynski]

Quite possibly, yes. We're still finding out the best ways to add projects to the site, though.

To save yourself some time, perhaps you could start off posting your proposed content as an issue instead of a pull request.

To be honest, PR is easier than maintaining the code in the issue. I need 2-3 python code files plus a markdown description, maybe some images also. From my side, a PR is not a problem, and it is also not a problem to refactor/move it somewhere else later.
The question was if you are interested in such content, as it goes a little bit outside of the main scope (lego).
My preferred way would be to create a PR with a new folder here: https://github.com/pybricks/pybricks-projects/tree/master/tutorials/wireless/hub-to-device (or even one level up), even if you decide to change the structure and not merge the PR in the end.
If you think it is a bad idea I will put it in my own repo and just link it from the issue.

[cadnza]

@pbochynski I'd be very interested in perusing your code—were you able to make it public?

[laurensvalk]

We now have a library and tutorial for this here: https://pybricks.com/project/micropython-ble-communication/

Thanks for opening this issue!