Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store-sync): wait for idle after each chunk of logs in a block #2254

Merged
merged 6 commits into from
Feb 16, 2024
Merged

feat(store-sync): wait for idle after each chunk of logs in a block #2254

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
704d5a3
fix: send hydration updates to main thread and change loading % calcu…
Kooshaba Feb 12, 2024
18d0043
fix: PR feedback
Kooshaba Feb 13, 2024
b00037c
Create nasty-rice-pull.md
holic Feb 13, 2024
2d81b02
Update packages/store-sync/src/createStoreSync.ts
holic Feb 13, 2024
ad891f2
Update packages/store-sync/src/createStoreSync.ts
holic Feb 13, 2024
a0cd05c
test(e2e): test sync percentage (#2264)
yonadaa Feb 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/nasty-rice-pull.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@latticexyz/store-sync": minor
---

`createStoreSync` now [waits for idle](https://developer.mozilla.org/en-US/docs/Web/API/Window/requestIdleCallback) between each chunk of logs in a block to allow for downstream render cycles to trigger. This means that hydrating logs from an indexer will no longer block until hydration completes, but rather allow for `onProgress` callbacks to trigger.
9 changes: 7 additions & 2 deletions packages/store-sync/src/createStoreSync.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ import {
} from "rxjs";
import { debug as parentDebug } from "./debug";
import { SyncStep } from "./SyncStep";
import { bigIntMax, chunk, isDefined } from "@latticexyz/common/utils";
import { bigIntMax, chunk, isDefined, waitForIdle } from "@latticexyz/common/utils";
import { getSnapshot } from "./getSnapshot";
import { fetchAndStoreLogs } from "./fetchAndStoreLogs";

Expand Down Expand Up @@ -145,11 +145,16 @@ export async function createStoreSync<TConfig extends StoreConfig = StoreConfig>
await storageAdapter({ blockNumber, logs: chunk });
onProgress?.({
step: SyncStep.SNAPSHOT,
percentage: ((i + chunk.length) / chunks.length) * 100,
percentage: (i / chunks.length) * 100,
Copy link
Member

@holic holic Feb 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think both of these code snippets might be wrong. The original intent of this was to have the percentage represent the completed state of the chunk (since we called storageAdapter above this).

I think the deleted snippet was meant to be ((i * chunkSize + chunk.length) / logs.length) * 100

And the new snippet is changing this to the "start" of the chunk (rather than "end"), so this probably should be ((i + 1) / chunks.length) * 100

holic marked this conversation as resolved.
Show resolved Hide resolved
latestBlockNumber: 0n,
lastBlockNumberProcessed: blockNumber,
message: "Hydrating from snapshot",
});

// RECS is a synchronous API so hydrating in a loop like this blocks downstream render cycles
// that would display the percentage climbing up to 100.
// We wait for idle callback here to give rendering a chance to complete.
await waitForIdle();
}

onProgress?.({
Expand Down
Loading