Flashing of GAP8 sometimes stops at 3-4%

[krichardsson]

Sometimes the flashing process is stopping around 3-4% when flashing a binary to the GAP8 from the cfloader.
I have not investigated this deeply, but it seems as if the GAP8 is blocking for some reason. From a STM fw point of view, buffers are pushed to CPX, and the expected behavior is that the ESP passes them on to the GAP8 where they should be written to flash. The first incoming buffer will also trigger an erase of the first flash page.

3-4% is most likely the amount of data that fits into the CPX buffers between the STM and GAP8, that is the GAP8 is not consuming (writing data to flash) as expected. When the CPX buffers are full, the STM can not write any more which blocks the memory mapping task. When the memory mapping task stops sending back ACK packets to the python lib, the memory mapping write in the python lib stops. This is finally what stops the cfloader.

We recently added asserts in the STM fw that are triggered if a write to the GAP8 takes too long, one of them should reboot the Crazyflie when this happens.
It can be a bit tricky to actually get the assert information out in this scenario, but there are two options

1. Connect to the CF with the python client and click the "Assert Info" button in the console log
2. Set enable_console_lo to True in the deck flashing code

[whoenig]

I have one AI-deck where this is very consistently the issue. This AI deck has the latest ESP firmware (flashed via cfclient update) and gap8-bootloader (flashed via JTAG). When flashing, cfloader hangs for a while and the Crazyflie reboots. The assert information (via cfclient Assert info) is surprisingly an unrelated assert (note that the firmware does not crash using the 2022.09 release):

SYS: Assert failed at ../src/hal/src/radiolink.c:164

[krichardsson]

Ah, the magic radiolink.c:164 assert!
We have seen this in relation to all sorts of problems and I think the general interpretation should be: "something in the STM is using too much CPU power". We get this assert when NRF is pushing radio packets faster than the STM is consuming them. I don't think the problem actually is radio link related, it is just a symptom.

Interesting that you get this assert though, I would have expected it to be one of the new asserts in the aideck code.

[krichardsson]

We just got this problem in the stability lab, but the assert was at line 136 in aideck.c

[krichardsson]

@whoenig do you still see this problem?

[whoenig]

No - we have only used the AI deck with the JTAG adapter lately.

[whoenig]

Had this again today with an assert in stream_buffer.c (unfortunately didn't write down the exact line). There was no way out, except of flashing the GAP8 directly via JTAG.

[knmcguire]

hmm this is strange... We need to find time at one point to stabilize the aideck again but it is not really on our priority list right now. We now say for everybody that a jtag programmer is crucial.

[gemenerik]

gap8 flashing stops at ~6% when having a flow deck attached

[williamleong]

I find that the success rate seems to be higher if we remove the multiranger and flow deck.

[gemenerik]

bitcraze/crazyflie-firmware#1381 fixes OTA flashing for aideck+flowdeck+multiranger

[gemenerik]

Still happens for aideck+lighthouse. Assert aideck.c ln 136

[gemenerik]

The lighthouse deck seems to add capacitance on VCOM, causing the GAP8 to not reset correctly

[gemenerik]

As a workaround, we tried a 4.5 kOhm resistor between VCOM and ground. This configuration drops the voltage quickly enough to successfully flash with the lighthouse attached. Note that we are using long wires, which might influence the outcome. Users may experiment with resistances around this range to achieve similar results.

[gemenerik]

One neat way of doing this would be to solder the resistor between the relevant pads on a battery holder.