daemon crash Merkle_mask.Masking_merkle_tree.Make(Inputs).Attached.Dangling_parent_reference

[jrwashburn]

QATF - https://gist.github.com/lk86/cff55a960262de9173741c4c49c4e250
Debian 8c/32G

After updating per instructions (gist) above, started daemon without a problem. After 5 mins (while node still bootstrapping) I stopped the daemon, renamed .coda-config director to a backup, and started daemon again. Crashed after 1-2 mins as reported by others in discord. Attached are crash files and logs.

After crash, restored original .coda-config directory and started daemon again. As of this writing, it has not crashed (15m uptime, still bootstrapping.)
coda_crash_report_2020-11-17_01-37-23.313960.tar.gz

Logs too large to attach; shared to logs@o1labs via google drive.

[jrwashburn]

https://drive.google.com/file/d/1cpIH7SIWXyqSgA9VI2AegdY2E5g7OuPG/view?usp=sharing

[boscro]

coda_crash_report_2020-11-17_01-52-10.884771.tar.gz

[deepthiskumar]

From the two attached logs, it seems like the nodes were catching up from length 851 and when they saw a block with length 1655 and 1957 respectively they closed their transition frontiers and switched to bootstrap. The old catchup process that was adding blocks to the transition frontier then fails with this error because the frontier is not there anymore.

There are also a few other hypotheses for this issue discussed here

[duccas]

The .coda-config folder was immediately deleted. After the update, the nodes crashed with the same error.
It was possible to start without crashing only the fourth time. Just restarting the node, no longer deleting the .coda-config folder.

The node has been working for over 7 hours.
Global number of accounts: 121
Block height: 1722
Max observed block length: 1722
Local uptime: 7h16m29s
Ledger Merkle root: jxefvn22HZXbV1hjGuJ68SJbuxXBL51q29CsB8tL278BMHzTD7W
Protocol state hash: 3NLrZLTHwibz5GiChitBmniv4ARbdb5A5AcPT2S648bBGDtbp6v5
Chain id: 3450411a053fc5417f639893b0ec1ca1616296c3696a590fac1da49c3dbdf861
Git SHA-1: [DIRTY]069333b5b08c48ea1df513edfa06cd40233843d6
Configuration directory: /root/.coda-config
Peers: 39

All info about the server and my logs in Google drive