Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #444.
This PR replaces maintenance error alerts (i.e.
tracing::error!
log messages that cause alert notifications) with metrics tracking the last seen block, as well as the last successfully updated block.This allows us to have more complex alerts (such as >3 successive maintenance failures, or no new block in past minute) and reduce noise.
EDIT: The context for this change is that we run connected to load-balanced nodes that have non-consistent views of the blockchain - where one may return
eth_blockNumber(latest)
asN
but subsequent queries toeth_call(..., N)
gets served by another node and may cause an error if the other node hasn't seen blockN
yet.Test Plan
Check the metrics are there: