Ideas to handle AskTimeoutException

[ZviMints]

I have the following HTTP-based application that routes every request to an Akka Actor which uses a long chain of Akka Actors to process the request.

   path("process-request") {
      post {
        val startedAtAsNano = System.nanoTime()
        NonFunctionalMetrics.requestsCounter.inc()
        NonFunctionalMetrics.requestsGauge.inc()
        entity(as[Request]) { request =>
          onComplete(distributor ? [Response](replyTo => Request(request, replyTo))) {
            case Success(response) =>
              NonFunctionalMetrics.requestsGauge.dec()
              NonFunctionalMetrics.responseHistogram.labels(HttpResponseStatus.OK.getCode.toString).observeAsMicroseconds(startedAtAsNano, System.nanoTime())
              complete(response)
            case Failure(ex) =>
              NonFunctionalMetrics.requestsGauge.dec()
              NonFunctionalMetrics.responseHistogram.labels(HttpResponseStatus.INTERNAL_SERVER_ERROR.getCode.toString).observeAsMicroseconds(startedAtAsNano, System.nanoTime())
              logger.warn(s"A general error occurred for request: $request, ex: ${ex.getMessage}")
              complete(InternalServerError, s"A general error occurred: ${ex.getMessage}")
          }
        }
      }
    }

As you can see, I'm sending the distributor an ask request for response.

The problem is that on really high RPS, sometimes, the distributor fails with the following exception:

2022-04-16 00:36:26.498 WARN c.d.p.b.http.AkkaHttpServer - A general error occurred for request: Request(None,0,None,Some(EntitiesDataRequest(10606082,0,-1,818052,false))) with ex: Ask timed out on [Actor[akka://MyApp/user/response-aggregator-pool#1374579366]] after [5000 ms]. Message of type [com.dv.phoenix.common.pool.WorkerPool$Request]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.

This is a typical non-informative Exception, the normal processing time is about 700 micros, 5 seconds its must be stuck somewhere at the pipeline since it cannot be that high.

I want to monitor this, I thought about adding Kamon integration which provides Akka Actors module with mailboxes, etc.

I tried to add the following configurations but its not working for me:
https://kamon.io/docs/latest/instrumentation/akka/ask-pattern-timeout-warning/ (didn't show any effect)

Is there other suggestions to understand the cause for this issue on high RPS system?

Thanks!