broadcast multiple tx per block from the same wallet

[agouin]

Currently, a transaction error when there is a batch of multiple messages means we don't always know exactly which specific message caused the error. This also means some messages could be given up on due to being grouped with messages that have errors.

The current retry logic for broadcasting messages in SendMessages does not pair well with the retry logic in the PathProcessor. It is redundant retrying, and The new SendMessagesToMempool does not have retry logic, allowing the PathProcessors to manage retries in tune with block production.

This also means that PathProcessors will not be blocked waiting for transactions to be included in blocks. They will only be blocked until the transaction enters the mempool (or fails making it there).

This splits up messages to be two per transaction (update client plus specific IBC tx) and overrides broadcast logic to pack multiple transactions into a single block signed by a single account.

This ensures that messages are error handled independently by wrapping them each in their own transactions. It copies in logic from lens to use sync broadcast mode, but not blocking on block inclusion. It uses the nextAccountSeq to make assumptions on the account sequence numbers that should be used for transactions in the same block.

Trade-off: this provides a much higher guarantee that valid messages will be delivered in the midst of error scenarios, but costs more in gas fees as the relayer is not batching messages.

Question - for broadcast-mode: batch, why not simulate txs with messages individually first to identify valid messages, then batch the valid ones?
This means potentially making many simulate API calls before batching messages. We can assume that for the high majority, all messages within the batch will be valid. By simulating the batched message first, if it errors, it will break out to individual simulate calls after that. This reduces overall API calls for typical operation under broadcast-mode: batch, and just means one extra API call in the case the batch has an error.

TODO:

• Make this behavior configurable
• [] Wire in with fee-grant work (handle as follow up)
• Hybrid mode where messages can be retried individually in the case the batched transaction fails
• [] Cleanup by pulling the new broadcast logic into lens (or vise versa?) (handle as follow up)

Example showing multiple txs for the same relayer wallets in the Stride ICQ interchaintest case during ICA channel handshake (4 at once):

[jtieri]

Everything here looks correct!

I had a few comments, most of which could probably be addressed in follow up PRs but I'll block on approving for now until you are back in office and respond

[jtieri]

This looks great! You did a great job on the refactor.

I had the one comment here but I don't think it's worth blocking approval on. It could be argued that this is a premature optimization anyways as it seems like we spend more time writing than reading so I will leave it up to you whether to address my feedback or not.

[danwt]

Why this

if ibcHeaderCurrent, ok := dst.ibcHeaderCache[src.clientState.ConsensusHeight.RevisionHeight]; ok && 
 	dst.clientTrustedState.IBCHeader != nil && 
 	bytes.Equal(dst.clientTrustedState.IBCHeader.NextValidatorsHash(),

?

Thanks