block export and/or block import appear to be broken

[matthew1001]

Description

Some time between 23.4.2 and 23.10.2 the block export and block import commands appear to have stopped working.

Steps to Reproduce (Bug)

Working example with 23.4.2:

1. Run a single QBFT node with besu --data-path=./data --genesis-file=../genesis.json --min-gas-price=0 --rpc-http-apis=ETH,QBFT,TXPOOL,ADMIN,WEB3 --p2p-port=30403 --rpc-http-enabled --rpc-http-port=8545 --sync-mode=FULL --node-private-key-file=data/key --rpc-ws-enabled (most of those options are unnecessary but that's what I've been running with)
2. After a few blocks with transactions have been mined, stop the node
3. Run besu --genesis-file=../genesis.json --data-path=./data blocks export --to=blocks.bin

Output:

2024-03-04 12:49:06.160+00:00 | main | INFO  | BlocksSubCommand | Export RLP block data to file blocks.bin
2024-03-04 12:49:06.420+00:00 | main | INFO  | DatabaseMetadata | Lookup database metadata file in data directory: /Users/mwhitehead/qbft-besu/1validator/node1/./data
2024-03-04 12:49:06.448+00:00 | main | INFO  | RocksDBKeyValueStorageFactory | Existing database detected at /Users/mwhitehead/qbft-besu/1validator/node1/./data. Version 1. Compacting database...
2024-03-04 12:49:07.739+00:00 | main | INFO  | KeyPairUtil | Loaded public key 0xbeea7aa3a6e1f862870a456239dbf9a232d9a69e58edfeda92193f679bdbf3960bb7f41bc6f0d574c4d0c6baa2324c7279045405ef8a240906bc13d3145ccc14 from /Users/mwhitehead/qbft-besu/1validator/node1/./data/key
2024-03-04 12:49:08.012+00:00 | main | INFO  | ProtocolScheduleBuilder | Protocol schedule created with milestones: [Berlin: 0, London: 100]
2024-03-04 12:49:08.059+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction pool
2024-03-04 12:49:08.066+00:00 | main | INFO  | BesuControllerBuilder | TTD difficulty is not present, creating initial sync phase for PoW
2024-03-04 12:49:08.082+00:00 | main | INFO  | BlockExporter | Exporting blocks [0,93) to file blocks.bin (appending: false)
2024-03-04 12:49:08.082+00:00 | main | INFO  | BlockExporter | Export at block 0
2024-03-04 12:49:08.384+00:00 | main | INFO  | BlockExporter | Export complete at block 92

4. Create a new data directory data2, and run besu --genesis-file=../genesis.json --data-path=data2 blocks import --from=blocks.bin

Output:

2024-03-04 12:49:32.165+00:00 | main | INFO  | BlocksSubCommand | Import RLP block data from 1 files
2024-03-04 12:49:32.364+00:00 | main | INFO  | RocksDBKeyValueStorageFactory | No existing database detected at /Users/mwhitehead/qbft-besu/1validator/node1/data2. Using version 1
2024-03-04 12:49:33.909+00:00 | main | INFO  | KeyPairUtil | Loaded public key 0xbeea7aa3a6e1f862870a456239dbf9a232d9a69e58edfeda92193f679bdbf3960bb7f41bc6f0d574c4d0c6baa2324c7279045405ef8a240906bc13d3145ccc14 from /Users/mwhitehead/qbft-besu/1validator/node1/data2/key
2024-03-04 12:49:34.193+00:00 | main | INFO  | ProtocolScheduleBuilder | Protocol schedule created with milestones: [Berlin: 0, London: 100]
2024-03-04 12:49:34.245+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction pool
2024-03-04 12:49:34.251+00:00 | main | INFO  | BesuControllerBuilder | TTD difficulty is not present, creating initial sync phase for PoW
2024-03-04 12:49:34.266+00:00 | main | INFO  | BlocksSubCommand | Importing from blocks.bin
2024-03-04 12:49:35.121+00:00 | main | INFO  | RlpBlockImporter | Import at block 92 / 12405550 gas 837135 micros / Mgps 14.819055468950648 segment 14.819675068689524 cumulative

Broken example with 23.10.2:

1. Same as step 1 above
2. Same as step 2 above
3. Same as step 3 above

Output:

2024-03-04 11:46:24.297+00:00 | main | INFO  | BlocksSubCommand | Export RLP block data to file blocks.bin
2024-03-04 11:46:25.364+00:00 | main | INFO  | KeyPairUtil | Loaded public key 0xbeea7aa3a6e1f862870a456239dbf9a232d9a69e58edfeda92193f679bdbf3960bb7f41bc6f0d574c4d0c6baa2324c7279045405ef8a240906bc13d3145ccc14 from /Users/mwhitehead/qbft-besu/1validator/node1/./data/key
2024-03-04 11:46:25.773+00:00 | main | INFO  | ProtocolScheduleBuilder | Protocol schedule created with milestones: [Berlin:0, London:100]
2024-03-04 11:46:25.787+00:00 | main | INFO  | DatabaseMetadata | Lookup database metadata file in data directory: /Users/mwhitehead/qbft-besu/1validator/node1/./data
2024-03-04 11:46:25.819+00:00 | main | INFO  | RocksDBKeyValueStorageFactory | Existing database detected at /Users/mwhitehead/qbft-besu/1validator/node1/./data. Version 1. Compacting database...
2024-03-04 11:46:26.747+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction pool
2024-03-04 11:46:26.760+00:00 | main | INFO  | BesuControllerBuilder | TTD difficulty is not present, creating initial sync phase for PoW
2024-03-04 11:46:26.774+00:00 | main | INFO  | QbftBesuControllerBuilder | Starting QBFT mining coordinator
2024-03-04 11:46:26.915+00:00 | main | INFO  | BlockExporter | Exporting blocks [0,25) to file blocks.bin (appending: false)
2024-03-04 11:46:26.915+00:00 | main | INFO  | BlockExporter | Export at block 0
2024-03-04 11:46:26.998+00:00 | BftProcessorExecutor-QBFT-0 | INFO  | QbftBesuControllerBuilder | Produced #25 / 0 tx / 0 pending / 0 (0.0%) gas / (0x39dc01a1248f88d62b760d6a5c829fdecff85f7c1279d2bbdd4a25d015c743a0)
2024-03-04 11:46:29.072+00:00 | BftProcessorExecutor-QBFT-0 | INFO  | QbftBesuControllerBuilder | Produced #26 / 0 tx / 0 pending / 0 (0.0%) gas / (0x101ab098af18921e4b63088951bbb581f38d1937a6fae4b4a5a89bf64eaccea1)
(note - the command doesn't stop, and appears to be producing new blocks)

Killing the export command and trying to export the blocks.bin fails in a similar way.) from before and after the issue.

Versions (Add all that apply)

• See notes above

[matthew1001]

The main difference in the logs that leaps out is 2024-03-04 11:46:26.774+00:00 | main | INFO | QbftBesuControllerBuilder | Starting QBFT mining coordinator in the failing case. Wondering if the logic introduced in https://github.com/hyperledger/besu/pull/5861 which relates to starting/stopping the BFT mining coordinator could be related.

[matthew1001]

After investigating it appears that the actual export does work, but while it's taking place the mining coordinator mines new blocks (if it's the only validator, or if the other validators are online).

For clarity I've decided to close this issue and open #6674 which more clearly states the problem.