Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] improve(relayer): Improve resilience against non-updated SpokePoolClient #1837

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from
+26 −8
Draft

[WIP] improve(relayer): Improve resilience against non-updated SpokePoolClient #1837

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1bffacf
improve(relayer): Permit up to 60s for SpokePoolClient readiness
pxrl Sep 24, 2024
4ae703d
Permit up to 2 minutes before bailing
pxrl Sep 24, 2024
0bb83ce
Update
pxrl Sep 24, 2024
bdc2563
improve(relayer): Improve resilience against non-updated SpokePoolClient
pxrl Sep 24, 2024
630adc2
Revert unnecessary
pxrl Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 26 additions & 8 deletions src/relayer/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ export async function runRelayer(_logger: winston.Logger, baseSigner: Signer): P
const relayer = new Relayer(await baseSigner.getAddress(), logger, relayerClients, config);
await relayer.init();

const { spokePoolClients } = relayerClients;
const simulate = !sendingRelaysEnabled;
let { spokePoolClients } = relayerClients;
let txnReceipts: { [chainId: number]: Promise<string[]> } = {};

try {
Expand All @@ -61,14 +61,32 @@ export async function runRelayer(_logger: winston.Logger, baseSigner: Signer): P
const ready = await relayer.update();
const activeRelayer = redis ? await redis.get(botIdentifier) : undefined;

// If there is another active relayer, allow up to 10 update cycles for this instance to be ready,
// If there is another active relayer, allow up to 120 seconds for this instance to be ready,
// then proceed unconditionally to protect against any RPC outages blocking the relayer.
if (!ready && activeRelayer && run < 10) {
const runTime = Math.round((performance.now() - tLoopStart) / 1000);
const delta = pollingDelay - runTime;
logger.debug({ at: "Relayer#run", message: `Not ready to relay, waiting ${delta} seconds.` });
await delay(delta);
continue;
if (!ready && activeRelayer) {
if (run * pollingDelay < 120) {
const runTime = Math.round((performance.now() - tLoopStart) / 1000);
const delta = pollingDelay - runTime;
logger.debug({ at: "Relayer#run", message: `Not ready to relay, waiting ${delta} seconds.` });
await delay(delta);
continue;
}

// Some SpokePoolClients have not updated. Filter out the ones that are not updated to avoid upsetting the
// BundleDataClient.
const droppedSpokePools: string[] = [];
spokePoolClients = relayerClients.spokePoolClients = Object.fromEntries(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nb. There are a few scenarios to test with this and I haven't made it that far, but this may not be the only change required. We should also consider whether these chain IDs should be removed from configured origin/destination chains, and possibly other configurables.

Object.values(relayerClients.spokePoolClients)
.filter(({ chainId, isUpdated }) => {
if (!isUpdated) {
droppedSpokePools.push(getNetworkName(chainId));
}

return isUpdated;
})
.map((spokePoolClient) => [spokePoolClient.chainId, spokePoolClient])
);
logger.warn({ at: "Relayer#run", message: "Proceeding without some SpokePools", droppedSpokePools });
}

// Signal to any existing relayer that a handover is underway, or alternatively
Expand Down