Validator PoH problems with replicators+erasure combo

[sakridge]

Problem

Erasure turned on seems to cause problems for validators when replicators are in the network: PoH fails:

2018-12-13T23:07:45.394379387Z WARN  solana::ledger] entry invalid!: x0: FnfLjmLQ95Q9NF5vfJDGwT3eicksbfwNUyf5APbSL2vH, x1: 4taJuyBoi9dhkNUx2pxis3ywgESvCCAtG3T3nhcBY5si num txs: 0
2018-12-13T23:07:45.394248461Z WARN  solana::entry] next_hash is invalid expected: DNeQXyCBTrbuLzcsaQsrYRutPwbHjruXwHehtfQ5ZkSv actual: E4TdoJYAVhpTHf3SdbtjT8FRADJBwauGjvFunctKrAaH
2018-12-13T23:07:45.394537316Z WARN  solana::ledger] entry invalid!: x0: 34PvZvYEkmv3Qp27pEqpYbZqxk2VKkNZnJdKCfjECoaZ, x1: BbZwaaKbH4L52tHfcHwnWf9bdT5erhthHHRNEuLD3JUF num txs: 0
2018-12-13T23:07:45.394625282Z WARN  solana::ledger] entry invalid!: x0: 3VzuYYJGPNBna85DQRcx79fz8MTChWEQ1aoSPrh4ZWtJ, x1: E4TdoJYAVhpTHf3SdbtjT8FRADJBwauGjvFunctKrAaH num txs: 0
2018-12-13T23:07:45.394780155Z WARN  solana::ledger] entry invalid!: x0: 32U5kBEE2mGWFn4991XQeZ8srQ7MHsMrfoimfL3QoGcC, x1: DNeQXyCBTrbuLzcsaQsrYRutPwbHjruXwHehtfQ5ZkSv num txs: 0
2018-12-13T23:07:45.395374885Z ERROR solana::replay_stage] BlobError(VerificationFailed)

Proposed Solution

[sakridge]

Testing with: cargo test --features=bpf_c,erasure,cuda,chacha --test=replicator

Root cause seems to be that db_window::try_erasure doesn't order the recovered data blobs into the consumed queue correctly.

[sakridge]

Fixed by #2185