Flow deck and Loco deck in TWR mode makes Loco deck hang

[tobbeanton]

If the Flow deck and the Loco deck are mounted together after a while ~1min the Loco deck hangs and the leds turn off.

Guesses of this behavior is collision on the SPI bus, interrupts not services fast enough, or stat machine bug. Note also that flow deck continue to work as it should so SPI bus is not blocked.

[miracatici]

We're currently use same setup. Do you confirm that you're using latest firmware (not released)? You can clone the repo and compile yourself.

[krichardsson]

@miracatici This was probably on the codebase of Sep 14 (or there about).

[ataffanel]

I think I have verified that this is a timing problem (so a state machine bug) and not an SPI problem: I have had the problem when developing another deck that was only using the UART.

[dastoqc]

using the latest firmware on all our units, we also experienced the issue with our Crazyflies 2.0+Flow deck 1.0. We need to reboot 2 to 4 times the Crazyflie in order to get the UWB deck not to crash. The issue is not present with our newer Crazyflies 2.1+ Flow deck 2.0.

Any solution?

[krichardsson]

@dastoqc that is interesting. Are you running the same firmware on the CF 2.0 and 2.1?
Have you tried CF 2.0 + Flow 2.0 and CF2.1 + Flow 1.0?

There is a difference in how the IMU is read between CF 2.0 and 2.1 that could change the timing and support what @ataffanel found.

Unfortunately there is no solution available at the moment.

[dastoqc]

Hi @krichardsson

We made some more tests today, not sure what is related to this issue:

• we have a hard time writing to the anchors using the CF (neither positions, nor change mode), it works well only with CF2.1+Flow2 (or no flow deck)
• all anchors and CF are running the latest firmware (pulled this morning), compiled from source
• all CF are running on 2Mhz band and the UWB deck crashes happen in both TWR and TDoA2 modes
• CF2.1+Flow1: still experience crashes and unable to write position to anchors
• CF2.0+Flow2: seems to be all good...

[krichardsson]

Thanks @dastoqc for the testing!
We'll digest it and see what we can do

[shushuai3]

By adding a task delay in uwbTask function, the timing problem can be solved such that flow deck and loco deck can be used together.

static void uwbTask(void* parameters)
{
...
while(1) {
vTaskDelay(5); // the line to be added
...

[dastoqc]

thanks @shushuai3 it seems to fix the issue indeed! However, now the 'kalman.resetEstimation' always converge to a wrong position.... will look into it.

[dastoqc]

looks like a delay of 3 works to solve the LPS+Flow issue and still have the kalman reset working!

[krichardsson]

Thanks @dastoqc

[shushuai3]

Wow! The latest firmware solves this issue, the timing problem when using UWB and Flow decks at the same time. Tests are done with CF2.0+UWB+Flow2.0

[krichardsson]

@shushuai3 Thanks for the report! I'm happy it works for you :-)

[knmcguire]

I guess we need to test this out on a 2.1 and see if this has been completely fixed.

[youngbin-song]

I'm working on a Crazyflie 2.1 and it doesn't seem to be working for it. My current setup is CF2.1+Crazyradio+Flow2.0.

While waiting for the fix, where should I implement shushuai3's workaround for this issue?

[knmcguire]

Hi @youngbin-song . This Github issue is about the combination of the flow deck and the LPS deck, not only the Flowdeck. Could you please ask your question on forum.bitcraze.io, with an output of the console tab of the cfclient and a more detailed description of your problem (which LEDS on the crazyflie, when it happens etc etc).

[youngbin-song]

Apologies for being unclear. Yes, my issue also includes the LPS deck as well.

[knmcguire]

ahh oke yes then you are at the right place! Yes go for it, try out Shushuais fix, but it is good to know that apperently there is still thing to be done in this github issue. I will try to investigate this week a well.

[youngbin-song]

Sorry to ask but where should I integrate with Shushuai's fix, specifically which code should I add it into?

Edit: I've figured it out. Thanks!

[knmcguire]

Great! the fix still seems to work?

[knmcguire]

I also checked it out and it seems to work for me as well.

Although I'm not quite sure why this hasn't been implemented yet, I will probably still wait a bit. There are several issues to fix for LPS I think, so it is good to do it when the rest is back at the office as I haven't worked or developed for the LPS system myself.

[youngbin-song]

Sorry, I wanted to give my reply after I've tested and made sure that it works.

It seems to work fine for me. There are some occasions that it the flowdeck turns off after a crash/sequence but I assume that it's because of some connection issue.

[knmcguire]

thanks for letting us know. I was able to contact my colleagues about this and the reason that this hasn't been implemented is that this fix is just a patch on the symptom of the real issue that needs to be fixed. So we need to investigate it further.

[shushuai3]

Just a clarification: the fix of adding a task delay (vTaskDelay(5)) in the uwbTask function will decrease the ranging frequency roughly from 400Hz to 120Hz. Luckily, the latest firmware has no issue on CF2.0+Flow2.0+LocoDeck such that the ranging frequency is not influenced. But I am not sure if the latest firmware works on CF2.1.

[knmcguire]

Yeah indeed figured that it would slow it down. It does work on the CF2.1 but I will need to wait until the others are back to the office in August to go into this again, since it is believed that there is more to it.

I might also test out if the same happens with the zranger, to see if the motion sensor has anything to do with it or not.

[krichardsson]

I have put some time into this issue and have found some interesting information:

First some basics about the implementation:
There is a semaphore protecting access to the SPI bus to make sure it is only used to communicate with one chip at a time. While one SPI transaction is ongoing, other tasks have to wait for the semaphore to be given before they can access the SPI bus (tasks are blocked).
The LPS deck is implemented using an interrupt to indicate that there is new data from the DWM1000 module. When the interrupt is triggered, the service routine gives a binary semaphore, which in turn releases the uwb task that will handle the interrupt. The task does all the SPI communication, the interrupt service routine simply gives the semaphore.

It seems as the flow deck hogs the SPI bus for a fair amount of time when reading data from the flow sensor. The current theory is that this blocks the uwb task from accessing the SPI buss (and thus takes longer to complete), and that during this time a second interrupt from the DWM1000 chip is triggered, which is not handled properly.
The LPS TWR algorithm relies heavily on a continuous flow of events from the DWM1000 chip, and since no transmission is scheduled (due to the missed interrupt) the process stops.

If this is actually the problem, the best solution is probably to use task notification with counting functionality instead of the binary semaphore.

[krichardsson]

The previous comment was not too far off, but not correct.

There was already a mechanism for checking if new events arrived while the interrupt is handled, the problem was that it did not work.
The interrupt mechanism in the DWM1000 is based on a status register and a mask. A pin on the chip is set high whenever there is one or more bits in status register that is set and the corresponding bits are set in the mask. The pin is used to assert an interrupt in the STM CPU and start the interrupt handling routine. In the handling routine various events are detected and the appropriate status bits are cleared. At the end of the handling routine the interrupt pin is read and if it is still set the handler is run once more.
The problem was that the pin that was read at the end of the handler was the wrong pin, always set to 0. There was a mixup of pin definitions: there are definitions GPIO pins in the STM library, and a second definition in the deck pin API (deck_constants.h). A GPIO definition was used together with the deck pin API function which caused the wrong pin to be read.

We should possibly consider a check to make sure the correct definition is used to avoid similar problems in the future?