Account sequence errors from connecting to partially synced chains

[faddat]

Crate

relayer-cli

Summary of Bug

If hermes is started while a chain is syncing, it picks up old acccount sequence values, and then attempts to use them repetitively.

The temporary solution is "don't start hermes against a partially synced chain"

The permanent solution is probably to actively check the account sequence

Version

0.7.3 + the compatiblity pr #1461

Steps to Reproduce

Heheh, little bit of a hard one here, but I guess it's approximately:

Using a wallet that has already been used for relaying, start Hermes against a partially synced Juno or Sifchain. What is likely to occur here is that Hermes will keep trying to make a tx with an invalid sequence number.

Another solution to this may be to disable hermes from interacting with partially-synced chains.

Acceptance Criteria

It should no longer be possible for Account sequence issues to crop up because it was connected to partially-synced chains from a previously used wallet.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Hey Jacob,

This is a tough one! In general, Hermes cannot trust account sequence numbers from a full node unless that node (a) is synced with the network and (b) has a clean mempool (ref). For (b) we don't have a solution yet, but we're working with tm-go team to find one. The problem you're highlighting here is in reference to case (a).

With respect to (a) it's not clear how can Hermes reliably infer whether the node is or isn't synced with the network. For SDK-based chains there is the option to query the node's status endpoint and read the catching_up = true field; if false then the node is clearly not synced; if true then it's actually not clear what the status is, because the node may be out of sync and be unaware (the true flag is unreliable).

Beside the catching_up flag, is there any API that Hermes can use to infer if the full node is synced or not? Any insights appreciated!

[MasterPi-2124]

Hi @adizere, thank you for your comment! We checked our chains, in this case, Juno, and it's unsynced with catching_up=false, but other chains are unsynced too, and they are still healthy.

[mircea-c]

catching_up is not really reliable for chain sync. You can have a chain that shows catching_up=false but it's really still behind the head of the chain. I've experienced this with multiple chains.

The only way, currently, to accurately gauge this is to compare block heights from multiple nodes.

[adizere]

The only way, currently, to accurately gauge this is to compare block heights from multiple nodes.

Good to know!!

[adizere]

Marking as blocked until we get more insights. We'll perhaps arrive at a solution together with the tendermint-go team!

[adizere]

This problem has been partly solved in PR #1349, which closed a related issue #1264.

Another solution to this may be to disable hermes from interacting with partially-synced chains.
Currently Hermes either

It should no longer be possible for Account sequence issues to crop up because it was connected to partially-synced chains from a previously used wallet.

Currently, Hermes either:

• detects that a chain is out of sync via the health checkup (namely, the parameter catching_up), and in that case it emits a clear warning that some functionality is impacted.
• does not detect the out of sync problem, but encounters an account sequence mismatch error and has an built-in mechanism to recover (by retrying) from that, thanks to the work in Retry send_tx on account sequence mismatch #1349.
  • operators no longer need to restart Hermes, but they may need to restart their full node if Hermes is unable to make progress due to the account sequence mismatch error.

I will therefore close this issue, and we may consider reopening if we still see problem in the future.