Node crash on state sync

[bowenwang1996]

As reported by a validator, their node on testnet crashed while trying to do state sync:

Mar 18 05:33:04 neard[1590]: Mar 18 05:33:04.815  WARN sync: State sync didn't download the state for shard 0 in 60 seconds, sending StateRequest again
Mar 18 05:33:09 neard[1590]: Mar 18 05:33:09.870  INFO stats: State [0: parts]  16/15/40 peers ⬇ 6.1MiB/s ⬆ 72.1kiB/s 0.00 bps 0 gas/s CPU: 58%, Mem: 2.6 GiB
Mar 18 05:33:47 neard[1590]: fatal runtime error: Rust cannot catch foreign exceptions

I wonder whether something crashed in rocksdb @mikhailOK

[mikhailOK]

Reproduced the crash, it happens in rocksdb background compaction. Investigating

[mikhailOK]

crashing in CompactionPicker::GetRange, somehow inputs is empty when it's not supposed to be according to the assert above. The column is ColBlockHeader. Looking into rocksdb logic.

[bowenwang1996]

From some other reports we received, it seems that this error is triggered when the node runs out of memory. @mikhailOK

[mikhailOK]

The hypothesis is that an out of memory crash can put rocksdb in an inconsistent state, which is what leads to the following crash in compaction. The memory issue is fixed in 1.18.1, the next step would be to implement #3266 because current way of shutting down might also be a risk. Ending the investigation for now.

[bowenwang1996]

No action item here. Closing