harden ReliableDeliveryShardingSpecs

[Aaronontheweb]

Changes

Fixes the following test race condition caused by forcing the test to wait until a probe gracefully stops, which can cause a re-delivery.

Akka.Cluster.Sharding.Tests.Delivery.ReliableDeliveryShardingSpec.ReliableDelivery_with_Sharding_must_deliver_unconfirmed_if_ShardingConsumerController_is_terminated
Expected delivery4b.Message to be Job(msg-4), but found Job(msg-3).

With configuration:
- Use declared types and members
- Compare enums by value
- Match member by name (or throw)
- Without automatic conversion.
- Be strict about the order of items in byte arrays

[Aaronontheweb]

This might not be the only fix this tests needs, FYI. No way to know for certain other than gathering more data.

[Aaronontheweb]

@Arkatufus so we have a hard bug in the async TestKit here:

akka.net/src/contrib/cluster/Akka.Cluster.Sharding.Tests/Delivery/ReliableDeliveryShardingSpec.cs

Line 131 in 4f27bac

var endMessages = await consumerEndProbe.ReceiveNAsync(3, TimeSpan.FromSeconds(25)).ToListAsync();

25 seconds is what we allot - it used to be 15 seconds prior to my last PR to harden this test (issue is: actors weren't being given enough time to complete the messaging routine.)

Looking at the logs:

First

[INFO][05/10/2023 01:41:12.789Z][Thread 0014][Cluster (akka://test)] Cluster Node [15.0.0] - Node [akka.tcp://[email protected]:64654] is JOINING itself (with roles [], version [15.0.0]) and forming a new cluster

Last

[INFO][05/10/2023 01:41:18.679Z][Thread 0079][akka.tcp://[email protected]:64654/system/sharding/TestConsumer-1Coordinator] Singleton manager stopping singleton actor [akka://test/system/sharding/TestConsumer-1Coordinator/singleton]

That's a grand total of 6 seconds - and there's several other asserts that happen between the 25 second one and the one prior.

[Aaronontheweb]

This is the figure - if the EndCondition is satisfied before we've processed any messages, we're probably being recreated via the ProducerController<T> + ShardRegion, therefore we want to ignore without sending anymore Collected responses.