Hopefully fix noisy submission error alert on eden

[MartinquaXD]

Currently eden's submission error alert is one of our noisiest. Also it's not really actionable so let's try to fix it.

My current working hypothesis is that eden sometimes submits a transaction into the mempool but reports an error anyway. This would lead our submission logic to think the gas price which would require a replacement tx would be the price of tx_1 * 1.25 (MIN_GAS_BUMP). But since the mempool already knows about tx_2 which must have at least a gas price of tx_1 * 1.25 the actual replacement tx would have to have a gas price of at least tx_2 * (1.25^2).

This hypothesis came from logs like this where at some point eden returned some generic internal server error and afterwards reports many replacement transaction underpriced errors in a row.

As a side note this change could also lead to faster submission times when a solver run loop encounters that issue.

I don't really know how to test this idea without deploying it so please let me know if it seems reasonable to you.
We could also enable the gas price factor bumping with a CLI switch to test the fix temporarily without redeploying the solver.

Test Plan

CI

[nlordell]

One concern I have here is that it is possible for gas prices to "escalate" quite quickly since we are making a positive "increase gas cost" feedback loop.

We should keep a close eye on this change (maybe add a log when we set the tx_consecutively_underpriced so we can easily track what its value is).

As for testing - I'm not sure what the right way to test this is. Maybe we can create a mock submitter and get it to return "tx underprised" errors?

[nlordell]

Maybe we can reach out to the Eden team and let them know that transactions are getting added to the tx pool even if their RPC endpoint is returning an error?

[sunce86]

I am worried if we start going this path (trying to handle these kind of errors) our code will eventually become maintenance nightmare.

[MartinquaXD]

I am worried if we start going this path (trying to handle these kind of errors) our code will eventually become maintenance nightmare.

While I agree that we ideally shouldn't have to worry about that, I think it makes sense to address this bug because it seems to cause our noisiest alert.
I hope I found a way to make this behavior configurable with the CLI in a reasonable manner.

[sunce86]

Lg.
What would be interesting is to add some kind of metric to know how often this happens, so we can be alerted when Eden fixes this error internally (if ever). Or we can set a manual reminder somewhere to check for the logs :) -> all so we can revert this change once the error is cleared.

[MartinquaXD]

What would be interesting is to add some kind of metric to know how often this happens, so we can be alerted when Eden fixes this error internally (if ever). Or we can set a manual reminder somewhere to check for the logs :) -> all so we can revert this change once the error is cleared.

Agree on all fronts. Since I don't know yet, whether this PR will actually fix the error (just an idea) I will deploy the change and monitor the impact. If it actually fixes the problem I will add the metrics otherwise I will revert the PR again.