backend(fix): Remove only finalized blocks from the event window

[lexnv]

This PR ensures that the window of events we feed to every new subscription contains the appropriate NewBlock and BestBlock events.

Before this PR, the block_events_from_last_finalized window of events was cleared out on every Finalized event.
However, this approach could miss reporting NewBlocks that will later become Finalized. Therefore, the unstable driver could report a block as Finalized and never report it as NewBlock first.

Considering the following edge case:

      0x1 -----> 0x2 -----> 0x3 -----> 0x4

T0    init       new         new       new
T1               fin                 

• T0: The chainHead subscription is started, 0x1 0x2 0x3 0x4 are announced
  • 0x1 is the Initialized event
  • block_events_from_last_finalized contains: 0x2 0x3 0x4 as NewBlocks
• T1: 0x2 is reported as Finalized
  • Before this PR the block_events_from_last_finalized is cleared our entirely

If a new subscription is started after T1; it will receive an Initialized event with the 0x2 block. However, it will never receive 0x3 and 0x4 as NewBlocks. After around 6 seconds, the 0x3 is reported in a Finalized event. This breaks the expectations of the rpc-v2 spec.

To mitigate this behavior, the window of events is not entirely cleaned out. Instead, only blocks reported as Finalized or pruned are removed from the window.
The last finalized block will be reported as Initialized by our driver, therefore there is no need to report NewBlock and BestBlock events for it. If the Finalized event reported multiple finalized hashes, we only care about the state at the head of the chain, therefore it is correct to remove those as well. Idem for the pruned hashes; they will never be reported again and we remove them from the window of events.

Testing Done

• This has been discovered by [DNM] Do not merge: Subxt UnstableBackend: Timeout issue repro #1318
• For more details see this comment
• The decode_signed_extensions_from_blocks with the panic patch is passing after this PR
• Added an unit test to ensure we don't clear out events of interest

[niklasad1]

Can you add a comment here why on NewBlock and BestBlockChanged are removed here if it's pruned or finalized?

The last finalized block will be reported as Initialized by our driver, therefore there is no need to report NewBlock and BestBlock events for it. If the Finalized event reported multiple finalized hashes, we only care about the state at the head of the chain, therefore it is correct to remove those as well. Idem for the pruned hashes; they will never be reported again and we remove them from the window of events.

[niklasad1]

yeah, alright when Initialized event is generated we can conclude that no messages regarding the next block are missed.

[jsdw]

I could only review on my phone but this looks like a good approach to me; good job!

[lexnv]

Thanks for the review guys! I'll keep my eyes on the CI for a couple of days to make sure no other errors appear 🙏