perf: Try to load previous seqNr if missing, in validation

[patriknw]

• makes it more safe to evict from memory, but keep more in database
• a step towards lazy loading of offsets, but full lazy loading would require many more changes due to the concurrency model used in the R2dbcOffsetStore
• otherwise it will use timestampOf, which are be more costly for gRPC projections
• it still falls back to timestampOf as last resort

[patriknw]

I'm a little worried about the index for this since timestamp_offset is before persistence_id in:

PRIMARY KEY(slice, projection_name, timestamp_offset, persistence_id, seq_nr)

but might be fine?

 Limit  (cost=4.31..4.32 rows=1 width=16) (actual time=0.137..0.138 rows=1 loops=1)
   ->  Sort  (cost=4.31..4.32 rows=1 width=16) (actual time=0.135..0.136 rows=1 loops=1)
         Sort Key: seq_nr DESC
         Sort Method: quicksort  Memory: 25kB
         ->  Index Only Scan using akka_projection_timestamp_offset_store_pkey on akka_projection_timestamp_offset_store  (cost=0.28..4.30 rows=1 width=16) (actual time=0.085..0.087 rows=2 loops=1)
               Index Cond: ((slice = 449) AND (projection_name = 'b267e71d-7ed9-4f02-8726-b96b0783ae7a'::text) AND (persistence_id = 'p1'::text))
               Heap Fetches: 0

The reason timestamp_offset is before is for the deletes.

[pvlugter]

Could also have an optional extra index for this.

[patriknw]

One thing that this could enable as a next step would be to decouple the db deletes from how much it holds in memory. I'd like to keep things in db for much longer than the in-memory window.

[pvlugter]

LGTM