ESTKALMAN: WARNING: Kalman prediction rate off

[TagirMuslimov]

Hello.

First of all I would like to thank you for this simulator.
I was able to install it and the example scripts are working. However I have a problem, in the form of an error "ESTKALMAN: WARNING: Kalman prediction rate off" that's constantly updated and never goes away.

I thought it was due to a weak laptop (I have a relatively old one with a 7th gen Core i5). However, the error is the same even when running two Crazyflies.
I also tested the hl-commander-swarm.py script and it works well on two Crazyflies. But already on three copters at some point the Crazyflies are crashing. I'm assuming the cause is this error with the Kalman filter.
Could you please advise what can be done to remove the error and to be able to work with more than two Crazyflies (would really like to work with four)?

[TagirMuslimov]

Ok, I tried running hl-commander-swarm.py on four Crazyflies on a much higher performance computer (Desktop, Core i7 13th Gen). The script now worked fine, with no crashes. However, the error "ESTKALMAN: WARNING: Kalman prediction rate off" kept popping up just as it did on the weaker laptop. Strange, I guess it's not related to performance, since this computer is quite powerful.

(Also got stderr on sim_cf2 during "colcon build", but there were only Warnings, as I understood it)

[TommiAux]

Hi @TagirMuslimov,

thanks for pointing out that issue. I am getting the same Kalman warnings and as you mentioned, even on a powerful computer, the problem remains. When I had a closer look at the sensor data arriving in the Kalman filter, I noticed that the frequency of the simulated IMU is set to 500Hz in contrast to the real IMU running at 1kHZ. I also had a short delay built in the task receiving the data from Gazebo. Removing this delay and changing the IMU frequency to 1kHz leads to a significant improvement and I mostly do not encounter the warning message anymore.

However, this will not bring any benefit when running on an older laptop with weaker hardware. The problem here is that gazebo requires a lot of computing power and when spawning too many Crazyflies, the real-time factor drops below 1.0. Since the software-in-the-loop part of the simulation still assumes real-time, the data coming from Gazebo is somehow outdated and therefore causing this behavior.
My suggestion in this case is to set the real time factor in Gazebo to a default 0.5 and adjust the TICK_RATE frequency in the Crazyflie firmware accordingly. The simulation time will run with half the speed in Gazebo and the SITL.
I tested this setting on my laptop (which is pretty old - 4th gen Core i7) and was able to run four Crazyflies.

I will push those adjustments to the 'sim_cf2' and 'crazyflie-firmware' repositories, creating a branch 'rtf05' in each.

For changing the real-time-factor in the crazyflie-firmware I added an option in the menuconfig of Kbuild:
SITL real time factor --> Use a real time factor of 0.5

I hope this helps!

[TagirMuslimov]

Thanks a lot.
This Warning seems to be appearing much less frequently now. I also occasionally get another WARNING: "STAB: WARNING: stabilizer loop rate is off" (see screenshot). Can this Warning significantly affect the simulation?