Possible regression: System.NotImplementedException in Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal

[object]

Version Information
Version of Akka.NET? 1.4.31
Which Akka.NET Modules? 1.4.31

Describe the bug
After we upgraded Akka packages from version 1.4.29 to 1.4.31, we observe frequent persistence failures. Here's an example:

[19:30:07 ERR] An error occurred during event batch processing. ChunkId: [63], batched requests: [1]
System.NotImplementedException: The method or operation is not implemented.
at Phobos.Actor.Remote.Impl.PhobosRemoteActorRef.get_PhobosContext()
at Phobos.Actor.Instrumentation.PhobosLocalActorRef.TellInternal(Object message, IActorRef sender)
at Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal2.WriteMessagesResult.FinalizeSuccess(BatchingSqlJournal2 journal)
at Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal2.ExecuteChunk(RequestChunk chunk, IActorContext context) at Akka.Util.Internal.AtomicState.CallThrough[T](Func1 task)
at Akka.Util.Internal.AtomicState.CallThrough[T](Func1 task) at Akka.Actor.PipeToSupport.PipeTo[T](Task1 taskToPipe, ICanTell recipient, IActorRef sender, Func2 success, Func2 failure)
[19:30:07 ERR] Rejected to persist event type [Akka.Persistence.Reminders.Reminder+Completed] with sequence number [203] for persistenceId [rem:playbackbump] due to [The method or operation is not implemented.].
System.NotImplementedException: The method or operation is not implemented.
at Phobos.Actor.Remote.Impl.PhobosRemoteActorRef.get_PhobosContext()
at Phobos.Actor.Instrumentation.PhobosLocalActorRef.TellInternal(Object message, IActorRef sender)
at Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal2.WriteMessagesResult.FinalizeSuccess(BatchingSqlJournal2 journal)
at Akka.Persistence.Sql.Common.Journal.BatchingSqlJournal2.ExecuteChunk(RequestChunk chunk, IActorContext context) at Akka.Util.Internal.AtomicState.CallThrough[T](Func1 task)
at Akka.Util.Internal.AtomicState.CallThrough[T](Func1 task) at Akka.Actor.PipeToSupport.PipeTo[T](Task1 taskToPipe, ICanTell recipient, IActorRef sender, Func2 success, Func2 failure)

To Reproduce
Steps to reproduce the behavior:

1. Update Akka packages to 1.4.31 (in our case it was update from 1.4.29)

Environment
We run our services on Windows servers, .NET 5.0

Additional context
Rolling back the libraries to the version 1.4.29 eliminated these errors.

[Aaronontheweb]

@object you might need to upgrade the version of Phobos that you're on as well - v1.3.2 takes a dependency on v1.4.31.

I've been somewhat more liberal recently about the API changes we've allowed lately and my decisions there created some issues with binary compatibility in recent releases. Lesson learned: we're going to tighten that up and I'm almost done formalizing our policies on those subjects here: #5461

[Aaronontheweb]

Also, which version of Akka.Persistence.SqlServer are you using?

[object]

Also, which version of Akka.Persistence.SqlServer are you using?

I first used Akka.Persistence.SqlServer 1.4.25, then upgraded it to 1.4.31, still failed.
And I used latest Phobos packages 1.3.2.

[object]

@Aaronontheweb BTW would you recommend switching to Akka.Persistence.Linq2Db to eliminate this error? I noticed that while Akka.Persistence.SqlServer was updated to 1.4.31, Akka.Persistence.Linq2Db is still an older version, so I was not sure if I should give it a try.

[Aaronontheweb]

Sounds like this is still an issue then if everything is failing with v1.4.31.

BTW would you recommend switching to Akka.Persistence.Linq2Db to eliminate this error? I noticed that while Akka.Persistence.SqlServer was updated to 1.4.31, Akka.Persistence.Linq2Db is still an older version, so I was not sure if I should give it a try.

As for why Linq2Db has not been upgraded, that's a @to11mtm question. But generally, yes I think we're going to see better performance out of Linq2Db over the long haul.

[object]

@Aaronontheweb ... and after downgrading it to 1.4.29 I have been running services for almost a day with no such error. So this sounds like 1.4.31 error.
Can't get more information from our logs about which call exactly causes NotImplementedException but apparently it only occurs when we use Akka.Persistence.Reminders, an add-on written by @Horusiath. This extension uses standard Akka event journal API so there should not be anything special with it, but AFAIK among other things it deletes old journal entries, and this is the only place where event journal entries are deleted in our code. Can this be something?

[Aaronontheweb]

This is probably related to some of the API changes that occurred in 1.4.30 - we had to roll back some of those in v1.4.31 already, but there might be others.

[Aaronontheweb]

Also, this should be relatively easy for us to reproduce.

[to11mtm]

Sounds like this is still an issue then if everything is failing with v1.4.31.

BTW would you recommend switching to Akka.Persistence.Linq2Db to eliminate this error? I noticed that while Akka.Persistence.SqlServer was updated to 1.4.31, Akka.Persistence.Linq2Db is still an older version, so I was not sure if I should give it a try.

As for why Linq2Db has not been upgraded, that's a @to11mtm question. But generally, yes I think we're going to see better performance out of Linq2Db over the long haul.

Generally speaking, an older version of the DB Plugins -should- be fine with newer versions of Akka.Net unless there is a breaking change elsewhere in the API. The last major one that comes to mind was #4680 but maybe there was another.

That being said while switching is probably a good idea, I don't know whether it would really help in this case =/.

[object]

@to11mtm I tried both older (1.4.25) and newest (1.4.31) versions of Akka.Persistence.SqlServer with the latest (1.4.31) version of Akka and Phobos packages, and they both failed with NotImplementedException.

[Arkatufus]

A few things stood out to me when I read the stack trace:

1. Phobos is routing persistence messages through the PhobosLocalActorRef wrapper
2. It is calling the TellInternal method
3. TellInternal is calling PhobosRemoteActorRef.PhobosContext which is not implemented.

This looks like a compounded bug between Akka.Persistence.Sql.Common 1.4.31 and Phobos 1.3.2. I will need to try and make a reproduction test case for this.

[Aaronontheweb]

Looks like this was a Phobos issue all along. We're releasing Phobos v1.3.3 now which should fix this issue. It was introduced by some of the correlation changes we introduced in order to make PipeTo traceable in v1.3.2.

[Aaronontheweb]

Phobos 1.3.3 is now released and should fix this issue. Please let us know, after you upgrade, if this resolves your issue with Akka.NET v1.4.31.

[object]

Great! Will try tomorrow.

[object]

Build a new version, looks good so far!

[Aaronontheweb]

Excellent, thanks for letting us know!