AAVAA board: native BLE does not connect to device on MacOS

[aavaa-farnood]

Description

When compiled from source, the library connects to the board using BLE on Windows, but the same code compiled on MacOS does not connect to the same board and gets stuck in this method: simpleble_peripheral_connect().

To Reproduce

Please refer to this fork to see the changes:

Code:

BoardShim.enable_dev_board_logger()
params = BrainFlowInputParams()
params.mac_address = "8CE5F615-5CEA-35A3-499C-B85EC91B5802"

board = BoardShim(board_id=BoardIds.AAVAA_V3_BOARD, input_params=params)
board.enable_dev_board_logger()

board.set_log_level(log_level=LogLevels.LEVEL_TRACE)
board.set_log_file(log_file="test.log")

Logs:

[2023-08-17 16:33:12.267] [board_logger] [info] incoming json: {
    "file": "",
    "file_anc": "",
    "file_aux": "",
    "ip_address": "",
    "ip_address_anc": "",
    "ip_address_aux": "",
    "ip_port": 0,
    "ip_port_anc": 0,
    "ip_port_aux": 0,
    "ip_protocol": 0,
    "mac_address": "8CE5F615-5CEA-35A3-499C-B85EC91B5802",
    "master_board": -100,
    "other_info": "",
    "serial_number": "",
    "serial_port": "",
    "timeout": 0
}
[2023-08-17 16:33:12.267] [board_logger] [trace] Board object created 53
[2023-08-17 16:33:12.267] [board_logger] [info] Use timeout for discovery: 5
[2023-08-17 16:33:12.267] [board_logger] [debug] use dyn lib: ~/AAVAAflow/python_package/brainflow/lib/libsimpleble-c.dylib
[2023-08-17 16:33:13.532] [board_logger] [trace] Scan started
[2023-08-17 16:33:13.557] [board_logger] [trace] address 960817F6-1B5B-BD03-C3F3-3DF9EC4318A0
[2023-08-17 16:33:13.557] [board_logger] [trace] identifier 
[2023-08-17 16:33:13.559] [board_logger] [trace] address EBC182A6-C158-D76B-6482-0E57B2C40A1E
[2023-08-17 16:33:13.559] [board_logger] [trace] identifier 
[2023-08-17 16:33:13.563] [board_logger] [trace] address 8CE5F615-5CEA-35A3-499C-B85EC91B5802
[2023-08-17 16:33:13.563] [board_logger] [trace] identifier AAVAA001
[2023-08-17 16:33:13.563] [board_logger] [info] Found AAVAAv3 device
[2023-08-17 16:33:13.621] [board_logger] [trace] Scan stopped

Info (please complete the following information):

• Board Id : AAVAA V3 (53 in the fork)
• OS running on your PC: MacOS (13.5)
• Programming Language and its version: python3.10
• Architecture(x86, x64, ARM, etc): ARM64
• pip list:

brainflow  0.0.1   ~/AAVAAFlow/python_package
nptyping   1.4.4
numpy      1.25.2
pip        23.1.2
setuptools 67.8.0
typish     1.9.3
wheel      0.40.0

Expected behavior
The same compiled code works great on Windows, but not on MacOS

Additional context
The code on MacOS is compile using cmake and Ninja:

cmake -DCMAKE_INSTALL_PREFIX=~/AAVAAFlow/tools/../installed -DBRAINFLOW_VERSION=4.9.2 -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" -DCMAKE_OSX_DEPLOYMENT_TARGET=10.15 -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_BLE=ON ~/AAVAAFlow/tools/..

ninja

[github-actions]

Welcome to BrainFlow project and thanks for your contribution! We will try to fix your issue ASAP. Make sure that you have read about issue format in the docs.

[Andrey1994]

Hi!

I see that you are adding a new board, it will be great if you open a PR with these changes to upstream and I will be glad to merge it after review.

Regarding macos issue, in the latest release there were a lot of changes in macos simpleble backend, do you try it with them or without?

[aavaa-farnood]

Hi @Andrey1994,

Thanks for the quick reply.
I will, I wanted to compile and run successfully before a PR.
I pulled all the recent changes on your main branch.

[Andrey1994]

I am talking about this one #666
It will be good if you test it with these changes and without them, and based on results we will be able to ask simpleble developers, since I have no mac to test it

[aavaa-farnood]

I have #666 in my recent test. I will try to test without and see if it solves the issue.

[Andrey1994]

I had issues with connect method before and it looked like race condition somewhere inside simpleble(like device is found but not ready to connect). For some boards in previos versions I called it in a loop couple of times(it failed to connect but didnt stuck)

in code it looked like this:

    simpleble_adapter_scan_stop (muse_adapter);
    if (res == (int)BrainFlowExitCodes::STATUS_OK)
    {
        // for safety
        for (int i = 0; i < 3; i++)
        {
            if (simpleble_peripheral_connect (muse_peripheral) == SIMPLEBLE_SUCCESS)
            {
                safe_logger (spdlog::level::info, "Connected to Muse Device");
                res = (int)BrainFlowExitCodes::STATUS_OK;
                break;
            }
            else
            {
                safe_logger (spdlog::level::warn, "Failed to connect to Muse Device: {}/3", i);
                res = (int)BrainFlowExitCodes::BOARD_NOT_READY_ERROR;
#ifdef _WIN32
                Sleep (1000);
#else
                sleep (1);
#endif
            }
        }

I see that you do not have this ugly hack, maybe worth trying at least with previos version of simpleble

[aavaa-farnood]

I actually saw that too. My problem is that it never gets out of the method simpleble_peripheral_connect() so it never gets to try that loop hack.

[Andrey1994]

Another idea to try - do smth with permissions for macos and bluetooth. I have no idea how it works but somehow macos requires bluetooth permissions for certain applications and these permissions are inherited somehow as far as I understand

https://developer.apple.com/forums/thread/675773

[Andrey1994]

I actually saw that too. My problem is that it never gets out of the method simpleble_peripheral_connect() so it never gets to try that loop hack.

maybe will be different without this PR

[aavaa-farnood]

To make the matter even more weird, in some lucky tries the device connects successfully and streams but then gets stuck for the rest of the day! It seems really hectic.

[Andrey1994]

Could you add some prints in this method

https://github.com/brainflow-dev/brainflow/blob/master/third_party/SimpleBLE/simpleble/src/backends/macos/PeripheralBaseMacOS.mm#L137
?

There are multiple calls of WAIT_UNTIL_FALSE and I suppose it hangs in one of them, if you determine in which one that would be easier to debug

[aavaa-farnood]

Sorry, it was a busy day.

I put some loggings in that method and it gets stuck here.

I have no idea about SimpleBLE so it'd be great if you could help.

[Andrey1994]

Can you try it wo python also? Cpp example to get data will be the best option to test

[Andrey1994]

And did you try it wo this PR for ble update?

[aavaa-farnood]

I haven't checked the code with cpp yet, but with python and without #666 in the logs it prints that it failed:

[2023-08-21 16:36:02.797] [board_logger] [info] Found AAVAA3c8 device
[2023-08-21 16:36:02.855] [board_logger] [trace] Scan stopped
[2023-08-21 16:36:07.863] [board_logger] [error] Failed to connect to AAVAA3c8 Device

but the method never returns, meaning it gets stuck somewhere else.

[Andrey1994]

it can be an issue with calling release_session I saw smth similar on linux, and added such workaround https://github.com/brainflow-dev/brainflow/blob/master/src/board_controller/muse/muse.cpp#L177C4-L177C86

its currently under ifdef linux, maybe makes sense to expand it for macos..

if you build your cpp code with python tools\build.py --debug you will be able to connect to running process with gdb - https://stackoverflow.com/questions/2308653/can-i-use-gdb-to-debug-a-running-process and get backtrace for each thread

[Andrey1994]

I asked simpleble developers for advice, other than that wo mac I cannot do much. You can join their discord also - there is a link in readme https://github.com/OpenBluetoothToolbox/SimpleBLE

BTW did you check permissions for ble like here https://github.com/pauldemarco/flutter_blue/issues/1011#issuecomment-1008096832 ?

[aavaa-farnood]

Unfortunately, I don't have gdb on my M2 Mac, at least couldn't install with brew.

I'd appreciate it if you let me know about possible solutions or updates. In the mean time I will dig into the code and debug it.

[Andrey1994]

I had issues with gdb on mac too, bit I think I managed to install it via macports or by building it from the source code

[aavaa-farnood]

Since you brought up SimpleBLE, I tested with their code on their repo and without BrainFlow and only SimpleBLE everything looks good, I can scan and connect really fast. Does that help?

Also on the permissions side, I have already checked all those and I have the right permissions.
I changed the sleep times in the code and now it connects every once in 10 trials. But still not reliable.

I can easily connect and stream using Bleak and SimpleBLE.

[Andrey1994]

#671 this may help

[retiutut]

We are not seeing similar issues with BrainFlow 5.9.0 and OpenBCI Ganglion using Native Bluetooth on multiple Macs.

@aavaa-farnood can we rename this issue to include the name for the board you are using?

[Andrey1994]

yes, it makes sense.

Regarding further testing I can propose next steps to see at which layer it fails:

1. confirm that it doesnt work for your board with brainflow and cpp example
2. check if it works or not using C++ SimpleBLE, Take simpleble from https://github.com/brainflow-dev/brainflow/tree/master/third_party/SimpleBLE
3. check if it works or not using plain C binding for SimpleBLE. Take simpleble from https://github.com/brainflow-dev/brainflow/tree/master/third_party/SimpleBLE
4. check if it works or not using plain C binding for SimpleBLE loaded manually without linking(its how brainflow calls it). You will need to copypaste code from https://github.com/brainflow-dev/brainflow/blob/master/src/utils/inc/runtime_dll_loader.h https://github.com/brainflow-dev/brainflow/blob/master/src/utils/inc/get_dll_dir.h and https://github.com/brainflow-dev/brainflow/blob/master/src/board_controller/ble_lib_board.cpp

If it fails at steps 2 or 3 than its more likely an issue from simpleble or I copypasted it with mistake(its possible). If you see the issue at step 4 then we will need to debug whats wrong with my wrapper and loader