You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Context and scope
On startup, the relayer back-processes missed blocks since it last started up. It first opens a subscription to the source blockchain before back-processing in order to guarantee full block coverage. We currently write both live incoming messages as well as back-processed messages to the same processing queue. This implies that messages are not processed in order while back-processing is active. As a result, we currently have to be very careful about when we update the checkpoint in the database. Specifically, we must NOT write to the database if 1) we are actively back-processing and 2) the message is a live incoming message. This requires us to differentiate between live incoming messages and back-processed messages at the message level, which becomes unused metadata as soon as we are done catching up.
Discussion and alternatives
If instead we buffer live incoming messages until after the catch up mechanism is complete, then flush them to the main processing queue, we can guarantee ordering in the processing queue. When #31 is implemented, ordering will not be guaranteed after message processing is initiated, but being able to guarantee ordering ahead of message processing will
make the overall application easier to reason about, and
Note that ordering is guaranteed at the subscriber level, but providing those same guarantees in the main relayer processing loop would provide better logical separation among components, as both the subscriber and the database interfaces would interact only indirectly via the Relayer
The text was updated successfully, but these errors were encountered:
Context and scope
On startup, the relayer back-processes missed blocks since it last started up. It first opens a subscription to the source blockchain before back-processing in order to guarantee full block coverage. We currently write both live incoming messages as well as back-processed messages to the same processing queue. This implies that messages are not processed in order while back-processing is active. As a result, we currently have to be very careful about when we update the checkpoint in the database. Specifically, we must NOT write to the database if 1) we are actively back-processing and 2) the message is a live incoming message. This requires us to differentiate between live incoming messages and back-processed messages at the message level, which becomes unused metadata as soon as we are done catching up.
Discussion and alternatives
If instead we buffer live incoming messages until after the catch up mechanism is complete, then flush them to the main processing queue, we can guarantee ordering in the processing queue. When #31 is implemented, ordering will not be guaranteed after message processing is initiated, but being able to guarantee ordering ahead of message processing will
Note that ordering is guaranteed at the subscriber level, but providing those same guarantees in the main relayer processing loop would provide better logical separation among components, as both the subscriber and the database interfaces would interact only indirectly via the
Relayer
The text was updated successfully, but these errors were encountered: