Fix for mediator load testing race condition when scaling horizontally

[ianco]

Pause and retry if a routing record is not found

[codecov-commenter]

Codecov Report

Merging #2009 (ab6c64e) into main (f857f8c) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #2009   +/-   ##
=======================================
  Coverage   93.51%   93.51%           
=======================================
  Files         539      539           
  Lines       34608    34625   +17     
=======================================
+ Hits        32363    32380   +17     
  Misses       2245     2245           

[shaangill025]

@ianco We can also use the Redis cache across scaled-up mediators to track if RoutingRecord for a recip_key is being created? Mediator_A before committing a new RoutingRecord to wallet adds routing_record::{recip_key} to cache. When mediator_B calls retrieve_by_recipient_key and StorageNotFoundError gets raised then we check the cache for routing_record::{recip_key} and retry only if it exists.

[ianco]

@ianco We can also use the Redis cache across scaled-up mediators to track if RoutingRecord for a recip_key is being created? Mediator_A before committing a new RoutingRecord to wallet adds routing_record::{recip_key} to cache. When mediator_B calls retrieve_by_recipient_key and StorageNotFoundError gets raised then we check the cache for routing_record::{recip_key} and retry only if it exists.

@shaangill025 I think using the redis (or kafka) queue is a better approach than the hack-y solution I put in. However unless we make queues a requirement (as @swcurran suggested) we also have to have a fall-back solution for when there are no queues deployed. (I think this problem can occur even if there is just a single instance of the mediator, although much less likely.)

For a redis/kafka solution, I think all that aca-py need to do - if there is no routing record found - is just leave the message on the queue with a "retry later" flag, I don't think there's a reason to do any additional checking. (Plus I'm not sure of the timing - is it possible that Mediator_B will try to find the routing record before Mediator_A even gets the request to add the routing record? I'm not sure of the root cause of this problem so we would need to assume this is a possibility.)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[swcurran]

Confirmed and we agree this should be merged. A subsequent update for adding queuing support in this area may be needed, but that can follow.