chore: Implement DEX rate limiter retry mechanism

[squadgazzz]

Background

The DEX rate limiter skips the rate-limited order, which could be avoided with a retry mechanism.

Details

The suggestion is to implement a configurable retry mechanism for the rate-limited orders instead of just skipping them. A config with only the max_retries param should be sufficient.

Acceptance criteria

• A rate-limited order is to be processed multiple times before skipping it.
• A thread with the retry action can be canceled by the auction deadline.

Depends on #2071

[fleupold]

This only affects the first order in the auction (as others will already wait for the back_off interval before attempting to execute), right?
Also orders are "automatically" retried in the next auction. I'd argue that for the sophistication we are aiming to provide to the baseline dex solvers that behavior is sufficient.

[MartinquaXD]

This only affects the first order in the auction (as others will already wait for the back_off interval before attempting to execute), right?

Theoretically this can happen to any order in the auction. The issue is that the backoff interval configured in the rate limiting strategy might not be in sync with whatever rate limiting strategy the external API actually enforces.
Let's say we are optimistic about backing off and start out with a few ms and double it every successive 429. Since our nginx enforces rate limits every second AFAIK we would run into multiple 429 until our backoff time is big enough to actually succeed the nginx 1 second limit.
If we then use so many requests that we trigger another rate limit we'll again skip a handful of orders in the same auction.

One easier way out would be to configure our min_back_off = 1s since we know all our requests go through the nginx that enforces that.

However, over all I think having the retry would be nice but I would not implement it under all circumstances. If the implementation ends up verbose and hard to understand I would probably prefer not to introduce the complexity since you already mentioned that orders get retried in the next auction.

[squadgazzz]

One easier way out would be to configure our min_back_off = 1s since we know all our requests go through the nginx that enforces that.

That's the current default value:

services/crates/solvers/src/infra/config/dex/file.rs

Lines 84 to 86 in 07a966e

	fn default_min_back_off() -> u64 {
	1
	}

Closing the issue since orders will be retried in the next auctions. Adding complexity here doesn't seem to be profitable.