Akka.Persistence.Query.Sql: PersistenceIds hangs after about 2 hours

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.4.48
Which Akka.NET Modules? Akka.Persistence.Query.Sql and Akka.Persistence.SqlServer

Describe the bug

When running the following query:

https://github.com/petabridge/akkadotnet-code-samples/blob/ed35dfd4ee862e68afe6abdd825a9f3029d382ed/src/clustering/sharding-sqlserver/SqlSharding.Host/Actors/ProductIndexActor.cs#L80-L89

/*
 * Kicks off an Akka.Persistence.Query instance that will continuously
 */
var query = Context.System.ReadJournalFor<SqlReadJournal>(SqlReadJournal.Identifier);
query.PersistenceIds()
    .Where(c => c.StartsWith(ProductTotalsActor.TotalsEntityNameConstant))
    .Select(c =>
    {
        var splitPivot = c.IndexOf("-", StringComparison.Ordinal);
        return new ProductFound(c[(splitPivot + 1)..]);
    })
    .To(Sink.ActorRef<ProductFound>(Self, Done.Instance))
    .Run(Context.Materializer());

This query will cease to produce new Akka.Stream items after approximately 2 hours (don't have an exact measurement) even though new PersistenceIds are being committed to the event journal. Restarting the query will reproduce all of the missing events.

To Reproduce
Steps to reproduce the behavior:

1. Clone https://github.com/petabridge/akkadotnet-code-samples/tree/master/src/clustering/sharding-sqlserver
2. Follow the instructions to run the project in the README.md
3. Go to http://localhost:5001
4. Add some entries
5. Look at the index page
6. Wait 2 hours
7. Add additional entries
8. Look at index page again (new entries added in step 7 will not be present)

Expected behavior

Query should continue producing new events indefinitely.

Actual behavior

Query stopped, raised no errors, and stopped producing

Environment

SQL Server latest running on Linux, .NET 6 running on Linux - although I don't think it matters.

[Arkatufus]

Its caused by the computer going to sleep.
I've tried to run the sample all night long without letting the computer to go to sleep and everything works just fine, but the stream died as soon as the computer were brought to sleep.
Reproduction is by running the sample project and then bringing the computer to sleep for more than 5 seconds.
My first theory is that when the computer wakes up, all of Akka.NET internal clocks are de-synchronized from the computer clock, and this somehow kills the stream.

[Aaronontheweb]

Are we thinking that this is an IScheduler bug then?

[Aaronontheweb]

And let me check with the original user if their app ever went to sleep

[Arkatufus]

Found the real cause, its the way non-blocking async "Ask" is being set up inside CurrentPersistenceIdsPublisher, it doesn't account for failure response and its stuck waiting for a reply from the journal

[Aaronontheweb]

Found the real cause, its the way non-blocking async "Ask" is being set up inside CurrentPersistenceIdsPublisher, it doesn't account for failure response and its stuck waiting for a reply from the journal

that's fantastic detective work - nice job @Arkatufus