BLE problems with latest(master) bluez-firmware in RPi-3B+ and compute modules

[gjorgjipetkovski]

Hi, I already post this topic on RPi forums, since we noticed the problem on those devices. So here it is again:

Hi,
When August 2020 (same with earlier Buster Lite images) Buster Lite image is used on RPi-3B+ and compute module based on 3B+, and when RPi is configured to work as BLE peripheral, there are significant problems to Connect and Pair+Bond to RPi from central device (Android or iOS based) using test application like nRF-Connect. The connection is usually timed-out and Pairing+Bonding is either timed-out or rejected.

steps to reproduce:

1. configure BLE peripheral via bluetoothctl:

pi@raspberrypi-gj:~ $ bluetoothctl
Agent registered
[bluetooth]# agent off
Agent unregistered
[bluetooth]# agent NoInputNoOutput
Agent registered
[bluetooth]# default-agent
Default agent request successful
[bluetooth]# discoverable on
Changing discoverable on succeeded
[bluetooth]# advertise on
[CHG] Controller B8:27:EB:5C:1F:9B SupportedInstances: 0x04
[CHG] Controller B8:27:EB:5C:1F:9B ActiveInstances: 0x01
Advertising object registered
Tx Power: off
Name: off
Apperance: off
Discoverable: off
[bluetooth]# show
Controller B8:27:EB:5C:1F:9B (public)
Name: raspberrypi-gj
Alias: raspberrypi-gj
Class: 0x00000000
Powered: yes
Discoverable: yes
Pairable: yes
UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
UUID: A/V Remote Control (0000110e-0000-1000-8000-00805f9b34fb)
UUID: A/V Remote Control Target (0000110c-0000-1000-8000-00805f9b34fb)
UUID: Generic Access Profile (00001800-0000-1000-8000-00805f9b34fb)
Modalias: usb:v1D6Bp0246d0532
Discovering: no

2. Scan the device using nRF-Connect, and try to Connect, then Bond from the application.

After some investigation, the problem seemed like is gone if rollback of bluez-firmware is done (replace /lib/firmware/brcm/BCM4345C0.hcd)
Default version that comes with Buster Lite OS is either 1.2-4+rpt4, or 1.2-4+rpt5 (from bluez-firmware "master" branch) and working version is from bluez-firmware "buster" branch. We used latest commit: 1.2-4+rpt2 release ( https://github.com/RPi-Distro/bluez-firmware/tree/buster/broadcom)

Can someone please check this and advise us, if it is ok to use bluez-firmware from "buster" branch on our RPi devices?
We cannot see change log or detailed release notes between bluez-firmware releases, to realize why this problem is present on master commits.
Also to check whether something need to be changed in Buster OS Lite image for future releases, to support this BLE mode for 3B+ based devices.

Note. No problems what so ever on RPi 3B, but they use different Bluetooth chipset than 3B+

Thanks!

[pelwell]

I just tried it and can see a disconnect. It will take a bit longer to identify this as a regression and gather evidence for Cypress.

[gjorgjipetkovski]

Thanks for the interest in looking at this issue.
We have collected some btmon and perhaps other logs during our investigation of this problem. We can share those, if needed to gather more evidence for Cypress.

[pelwell]

Thanks for the detailed report - it's easily reproducible, and the immediately previous version of the firmware suffers from the same problem.

A ticket is open with Cypress.

[gjorgjipetkovski]

Thanks for the effort Phill. We would really appreciate if we could somehow follow the progress of that ticket or get informed about the outcome.

[pelwell]

I'll report back the major findings.

[pelwell]

A quick update: Cypress have sent us a number of test firmware to establish when the problem appeared to help them track down the root cause, and they're working through the results.

[bojanpotocnik]

@pelwell any updates on this?

[pelwell]

By coincidence I chased Cypress yesterday, who confessed overnight to issuing a test firmware in an internal-only reply. I now have that firmware to try, so hopefully we can make some progress.

[pelwell]

One of the two trial firmwares issued yesterday worked when I tested it. You can download it from here: https://drive.google.com/file/d/1RJnhkV_IBjb1Lz2gsGKeF6YAcfa69IaK/view?usp=sharing

I have a feeling you'll know how to install it, but just in case:

$ sudo mv /lib/firmware/brcm/BCM4345C0.hcd{,.bak}
$ sudo cp /BCM4345C0_003.001.025.0171.0000_RaspberryPi_RPI3PLUS_TEST_ONLY.hcd /lib/firmware/brcm/BCM4345C0.hcd

To revert to the previous firmware:

$ sudo mv /lib/firmware/brcm/BCM4345C0.hcd{.bak,}

Please confirm that this trial firmware solves the problem for you and seems to be generally functional, then we can see about getting a proper release made.

[dhalbert]

I tried BCM4345C0_003.001.025.0171.0000_RaspberryPi_RPI3PLUS_TEST_ONLY.hcd on an RPi4 (not 3B+). I did an apt dist-upgrade just before. I rebooted after replacing the firmware with the test .hcd file.

[EDIT: see below]
Unfortunately it does not work for me. My test code is several layers of Python, including https://github.com/hbldh/bleak underneath. I'm getting an org.bluez.Error.Failed: Input/output error. With bluez-firmware versions 1.2-4+rpt5 and 1.2-4+rpt6 I would get an error higher up, indicating that the bluez layer was at least able to communicate with the adapter. 1.2-4+rpt2 works for me on RPi4 and RPi3B+.

The test program communicates with a BLE pulse oximeter. Referencing hbldh/bleak#332 for your interest.

[pelwell]

That's a bit vague - can you narrow down the failure any further? Has it ever worked for you? You could try running "sudo btmon -w bluez.bts" in the background while running your test - twice, ideally - then upload "bluez.bts" somewhere I can download (or if it might contain sensitive information, email it to [email protected]).

[dhalbert]

OK, I retested, and after some confusion, I got the new version to work! I had just been rebooting between changing the .hcd files, but it appears that I have to sudo poweroff and physically remove power to get it to reload the firmware properly.

Does that make sense? Under what conditions is the firmware reloaded? Why would simply rebooting after replacing the .hcd cause problems? Perhaps there was no communication with the radio chip. Thanks.

(I had tried swapping back to the working .hcd, which failed in the same way, which made me think I should try a real power-cycle.)

[pelwell]

Rebooting ought to be enough to cause the firmware to be reloaded - the firmware holds the BT modem in reset for quite some time - so I can't explain why it didn't appear to be reloaded for you, but I'm happy to hear that it's working now.

[pelwell]

Cypress have identified the root cause and provided an official release version of the firmware including the fix: https://drive.google.com/file/d/1DVOtBjrsoR2NhwEBVn3ei0sv-xTIBCxR/view?usp=sharing

[dhalbert]

Cypress have identified the root cause and provided an official release version of the firmware including the fix: https://drive.google.com/file/d/1DVOtBjrsoR2NhwEBVn3ei0sv-xTIBCxR/view?usp=sharing

This version works for me as well with the pulse oximeter example.

[pelwell]

Thanks for the feedback. @gjorgjipetkovski @bojanpotocnik Are you going to get a chance to try it in the next few days? If not, I'll get it into our BlueZ firmware repo sooner rather than later.

[gjorgjipetkovski]

Hi @pelwell. I already tried the basic procedure, which is described in this topic, with bluetoothctl and nRF, and that is working ok.
Furthermore I continued to test it with our actual application, and run upon advertisement deregistration/registration problem after changing the advertisement name, which is different than 1.2-4+rpt2 version behavior (no such problem there). But will do more testing in next few days to make firm conclusion.
(Failed to register advertisement: org.bluez.Error.Failed: Failed to register advertisement)
Question:

• is this firmware version somehow dependent from certain version of BlueZ library? We currently use BlueZ 5.50.

[pelwell]

• is this firmware version somehow dependent from certain version of BlueZ library? We currently use BlueZ 5.50.

If there is such a dependency on BlueZ versions, Cypress didn't mention it. Provided this version isn't a regression on the previous broken version and it fixes the known bug (which it seems to) then I'll release it.

[rickjansen-dev]

Is this firmware supposed to work on the pi 4 too? I just tried it and the only thing I seem to be able to do is turn discovery on and off. Connecting to a device fails (using bluetoothctl here):

[bluetooth]# connect 90:84:2B:5A:60:33
Attempting to connect to 90:84:2B:5A:60:33
Failed to connect: org.bluez.Error.Failed

output from btmon:

= bluetoothd: ATT bt_io_connect(90:84:2B:5A:60:33): connect: No route to host (113)                                      21.331779

I reverted back to the 1.2-4+rpt2 version, everything works again, I can connect fine.

[dhalbert]

I tested successfully on the Pi 4.

[dhalbert]

The failure you're seeing might be similar to what I had. Try power-cycling completely between swapping the firmware versions.

[rickjansen-dev]

The failure you're seeing might be similar to what I had. Try power-cycling completely between swapping the firmware versions.

I tried this, but didn't work

[dhalbert]

Is this change incorporated into the 1.2-4+rpt8 release, or was that solely a security fix?

[pelwell]

1.2-4+rpt7 contains the fix to this problem, and 1.2-4+rpt8 is a security patch for a vulnerability.

[sneko]

@dhalbert @pelwell I'm getting same error with Bleak library (Python one), I'm currently on 1.2-4+rpt6 thanks to apt-get but the rpt7 nor rpt8 are released.

Do you know if it has been released on any repository since then?

Otherwise do you have some notes/articles showing how to manually upgrade bluez-firmware. I'm not very familiar with this, sorry 😢

Thank you,

[pelwell]

I'm currently on 1.2-4+rpt6 thanks to apt-get but the rpt7 nor rpt8 are released.

I don't follow you - 1.2-4+rpt8 is the released version. To update:

$ sudo apt update
$ sudo apt install bluez-firmware

[sneko]

@pelwell I'm really sorry, thought I did the apt update... my bad I'm sorry! Probably should take a nape 😄 just retried... thank you!

[dhalbert]

I confirmed that rpt8 works now. Should this be closed?

[pelwell]

Yes, let's be optimistic.