LPS deck stops working in the system test rig

[krichardsson]

When running the system tests we have observed that the LPS deck on the Roadrunners sometimes stops blinking when the devices are idle. This is observed after a few minutes on some devices, but not all.

In the test rig we have 10 RR running in parallell. By default they switch between TWR, TDoA2 and TDoA3. In TWR they transmit UWB messages that might be received by the other devices in the rig, and it is possible that this is related to this problem.

[ntamas]

For what it's worth, we have observed the same behaviour with 10 Crazyflies running in parallel, and all of them are configured to TDoA2 so they should not be transmitting any UWB messages (if my understanding is correct).

[krichardsson]

Thanks for the input!
Yes you are correct, if they are fixed on TDoA2 they should not transmit.
Another theory has been that it is related to the automatic mode switching, but that also sounds unlikely since you are not changing modes.

[ntamas]

Another two cents: I was charging 11 Crazyflies at home this morning, without any Loco anchors nearby, and the Loco deck on two of them has stopped responding after half an hour. So I think the issue is most likely not related to any transmission on the UWB channel, and would probably happen on a single Crazyflie as well if you watch it for long enough.

I hope this can get fixed soon; I'll also try to look into the firmware code shortly to see if I can figure out something on my own.

[ataffanel]

This problem seems related to the issue #368. It might actually be the same problem.

[ntamas]

In our case we have the Loco deck and the LED ring installed. Shall we try testing for a while with the LED ring disconnected?

(Note that the LED ring was turned off (i.e. effect 0) when I was charging them).

[krichardsson]

I don't think removing the LED ring will make any substantial difference, it should not use a lot of CPU.

There is a fix described in #368 adding a sleep. This could be interesting to test to see if it has any effect on this issue, if you have the time.

[krichardsson]

We have realized that the power supply for the Roadrunners in the test rig has been a bit too weak. This problem seems to be less likely to happen with proper power.

[krichardsson]

This issue is most likely connected to #368.
The LPS modules on the roadrunners are switching between TWR, tdoa2 and tdoa3. In TWR mode they burst UWB packets that are received by the other roadrunners in the rig. The problem with interrupt handing in #368 would cause TWR algorithm to stop in this scenario.

[krichardsson]

I have tested with #368 in TWR mode for 2-3 hours without problem. Will test with TDoA tomorrow to see if it fixed the issue observed by @ntamas

[krichardsson]

I ran TDoA2 only in the test rig for an hour without problems.
When I flashed a version that switches between modes some decks stoped. Seems as all problems are not solved yet.

[krichardsson]

It seems as it is working as long as the automatic mode switching is disabled. A guess is that it is possible to start an action (transmit for instance) in one mode and that another mode is activated before the event that is signalling the completion of the action is received. The new mode thus gets an unexpected event that possibly can cause this issue. This is a theory that has not been confirmed.
Since automatic system identification is not running for hours in normal use cases, it is deemed not to be worth investigating further. Closing the issue,.