Crashes due to ledger mutations

[mrmr1993]

At present, the ledger can be mutated without notifying or updating other parts of the program that depend on the ledger state.

For example the ledger may be mutated during Transaction_pool.apply, causing the ledger to become inconsistent. This has manifested in

• We need a mechanism to stop the async jobs of created by the transaction_pool and its attached transition frontier is outdated #6946 where the ledger is detached while a diff is being applied
  • addressed by PR Interrupt Transaction_pool.apply when the ledger is detached #6971
• Transaction pool may attempt to add transactions between ledger update and pool revalidation #7613, 1dffa7c: [Fatal] Unhandled top-level exception: $exn #4255, Unhandled exception: network client EOF #4274, Coda Daemon crashed: Assert_failure src/lib/network_pool/indexed_pool.ml:529:8 #4282 where the ledger has changed while a diff is being applied
  • addressed by Protect Indexed_pool.add_from_gossip_exn from changes to the ledger #7639 (in review)
• the ledger has changed but the pool hasn't yet been updated, violating invariants
  • partially addressed by Fix error when expired transactions are replaced in the pool #7705

It's not clear what is mutating the ledger to cause this. We should investigate, and consider using immutable ledgers by default. If we use mutable ledgers only at creation and destruction time, and only allow ledger mutation when no immutable references are held via some kind of locking, then we should be able to mitigate this.

There are related concerns around all of our uses of ephemeral data. For example, the transaction pool is valid at a particular slot, and modifying it before we update it to the current slot also breaks invariants. We need to think much more carefully about how we handle mutable and ephemeral data, especially in the context of the concurrent async primitives we use.

[mrmr1993]

We haven't seen this crash again since #7759 was merged. Tentatively closing.