Binaries compiled with MDBX exhibit memory corruption

[michaelsproul]

Description

We've found a very weird issue on one of our nodes where binaries compiled with LMDB MDBX cause memory corruption during execution. Even when the code paths that use MDBX (i.e. the slasher) are inactive, the corruption still occurs.

The nature of the corruption is described here:

• Expensive fork-choice queued attestation mutation #6206

Initially we identified UB in the LMDB bindings and worked around it in this PR:

• Work around UB in LMDB bindings #6211

Steps to resolve (MDBX)

1. Delete MDBX from LH, retaining LMDB as default for now (if it is untainted), and Redb.
2. Move to using the Reth bindings (assuming they are free of UB), or

Steps to resolve (LMDB)

This section was written assuming LMDB was buggy, which it still could be:

Option 1 is that we remove LMDB from Lighthouse completely, or at least from our compiled binaries. The downside is that it's the default slasher DB, so this will require everyone with a slasher to delete their slasher DB and start again (not really a big deal). The other difficulty is choosing whether to go with Redb or MDBX as the new default (I would prefer Redb).

Option 2 is to try to fix the UB in the bindings. I worry that this will be very involved and for little long-term benefit. There are no alternative LMDB bindings we can use because the bug exists in the original lmdb and its fork lmdb-rkv. There are no LMDB bindings that are being maintained (see https://crates.io/search?q=lmdb).

[dapplion]

Let's move on to Option 1 and not waste time

[michaelsproul]

Sounds good. I don't think we have the bandwidth to maintain LMDB bindings solo, and there's no real point pouring tonnes of resources in to a dead-end DB.

Documenting here so we have something to reference in the release notes when we delete it!

[michaelsproul]

Ok so it turns out it's actually the binaries compiled with MDBX that are corrupting their memory.

I think this means we need to either remove just MDBX, or remove MDBX and LMDB. Or we could try using the Reth bindings for MDBX.

[michaelsproul]

MDBX seems to make some attempt not to alias & pointers, by copying data if MDBX indicates that the returned value is "dirty":

https://github.com/sigp/libmdbx-rs/blob/e6ff4b9377c1619bcf0bfdf52bee5a980a432a1a/src/cursor.rs#L103-L111

https://github.com/sigp/libmdbx-rs/blob/e6ff4b9377c1619bcf0bfdf52bee5a980a432a1a/src/codec.rs#L31-L46

[michaelsproul]

That same node is crashing with LMDB again, but is also throwing I/O errors. Going to investigate the hardware failure angle some more.