fix(cli): improve performance of linked library resolution during deployment #3197

alvrs · 2024-09-18T22:34:57Z

Significantly improved the deployment performance for large projects with public libraries by implementing a more efficient algorithm to resolve public libraries during deployment.
The local deployment time on a large reference project was reduced from over 10 minutes to 4 seconds.

changeset-bot · 2024-09-18T22:35:01Z

🦋 Changeset detected

Latest commit: 0bce659

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 26 packages

Name	Type
@latticexyz/cli	Patch
@latticexyz/world-modules	Patch
mock-game-contracts	Patch
@latticexyz/abi-ts	Patch
@latticexyz/block-logs-stream	Patch
@latticexyz/common	Patch
@latticexyz/config	Patch
create-mud	Patch
@latticexyz/dev-tools	Patch
@latticexyz/explorer	Patch
@latticexyz/faucet	Patch
@latticexyz/gas-report	Patch
@latticexyz/protocol-parser	Patch
@latticexyz/react	Patch
@latticexyz/recs	Patch
@latticexyz/schema-type	Patch
solhint-config-mud	Patch
solhint-plugin-mud	Patch
@latticexyz/stash	Patch
@latticexyz/store-indexer	Patch
@latticexyz/store-sync	Patch
@latticexyz/store	Patch
@latticexyz/utils	Patch
@latticexyz/world-module-metadata	Patch
@latticexyz/world	Patch
ts-benchmarks	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

alvrs · 2024-09-18T22:37:17Z

packages/cli/src/deploy/createPrepareDeploy.ts

-      const library = libraries.find((lib) => lib.path === placeholder.path && lib.name === placeholder.name);
+      const libraryKey = getLibraryKey(placeholder);
+      const library = libraryMap[libraryKey];


Previously we were doing a linear search to find the matching library every time is function was called, now we're just looking up the matching library in constant time. Doesn't make a big difference with small numbers of libraries but compounds with many libraries.

alvrs · 2024-09-18T22:42:55Z

packages/cli/src/deploy/createPrepareDeploy.ts

-      bytecode = spliceHex(
-        bytecode,
-        placeholder.start,
-        placeholder.length,
-        library.prepareDeploy(deployer, libraries).address,
-      );
+
+      // Store the prepared address in the library map to avoid preparing the same library twice
+      library.address ??= library.prepareDeploy(deployer, libraryMap).address;
+
+      bytecode = spliceHex(bytecode, placeholder.start, placeholder.length, library.address);


Previously we were resolving each library multiple times if it was used in multiple contracts/libraries. The time complexity of this quickly explodes if there are many libraries that use other libraries. Now we're caching a library's resolved address, so each library is only resolved once.

alvrs · 2024-09-18T22:44:45Z

packages/cli/src/runDeploy.ts

@@ -133,6 +133,7 @@ export async function runDeploy(opts: DeployOptions): Promise<WorldDeploy> {
        : undefined,
    }),
    account,
+    pollingInterval: 100,


Reducing polling time for a short time during deployment should be safe and makes a significant impact on deployment time (with the project i was testing with this change brought the time from 16s to 4s after the previous improvements were implemented)

(moved this to waitForTransactions instead to only increase the polling rate where it matters)

Moved this change to #3198 since it's unrelated to library linking, and CI wasn't happy with it yet

…loyment

holic · 2024-09-19T09:15:26Z

packages/cli/src/deploy/getLibraryMap.ts

+}
+
+export function getLibraryKey({ path, name }: { path: string; name: string }): string {
+  return `${path}:${name}`;


should we include the deployer address in the key since it changes the bytecode and thus the address?

the deployer address is passed in later (in prepareDeploy) while the map is created before, but we can change address in the map to { [deployer: Address]: Address } to cache per deployer address

holic · 2024-09-19T09:16:52Z

packages/cli/src/deploy/deployWorld.ts

-    throw new Error("world deploy failed");
-  }
-
+  const [receipt] = await waitForTransactions({ client, hashes: [tx], debugLabel: "world deploy" });


iirc I intentionally didn't replace this one with waitForTransactions because I wanted to throw a more specific error, but this is fine too

reason i changed this was bc i changed polling settings in waitForTransactions and would have had to duplicate the changes here, but moved that to #3198 so could also move this change there

holic · 2024-09-19T09:24:24Z

packages/cli/src/verify.ts

@@ -105,7 +105,7 @@ export async function verify({
    );

    modules.map(({ name, prepareDeploy }) => {
-      const { address } = prepareDeploy(deployerAddress, []);
+      const { address } = prepareDeploy(deployerAddress, {});


thoughts on defaulting the second arg in prepareDeploy so callers don't need to know the shape of the thing to pass in

holic · 2024-09-19T09:26:17Z

packages/cli/src/deploy/getLibraryMap.ts

+export type LibraryMap = { [key: string]: Library & { address?: Hex } };
+
+export function getLibraryMap(libraries: readonly Library[]): LibraryMap {
+  return Object.fromEntries(libraries.map((library) => [getLibraryKey(library), library]));


not blocking but I wonder if we should just return some interface with get and set and leave the internal representation and key shape as implementation details, then you can use this almost like a regular map but with 2-3 args for the key (path + name + maybe deployer address)

holic

lgtm!

main question is whether or not we need to key by deployer since we cache the address and the address is based on the deployer

holic

nice! I dig the new LibraryMap

vercel bot deployed to Preview September 18, 2024 22:36 View deployment

alvrs commented Sep 18, 2024

View reviewed changes

alvrs marked this pull request as ready for review September 18, 2024 22:48

alvrs requested a review from holic as a code owner September 18, 2024 22:48

vercel bot deployed to Preview September 18, 2024 22:50 View deployment

vercel bot deployed to Preview September 18, 2024 23:11 View deployment

vercel bot deployed to Preview September 18, 2024 23:49 View deployment

alvrs added 5 commits September 19, 2024 09:21

fix(cli): improve performance of linked library resolution during dep…

6b782bd

…loyment

Create few-olives-judge.md

10ea71c

move polling interval to waitForTransactions, add better retry logic

e5d7b58

increase retry count

6247103

temporarily revert polling changes

c816de9

alvrs force-pushed the alvrs/prepare-deploy branch from 88028b7 to c816de9 Compare September 19, 2024 08:35

vercel bot deployed to Preview September 19, 2024 08:38 View deployment

alvrs mentioned this pull request Sep 19, 2024

feat(cli): speed up deployment by decreasing polling interval when waiting for transactions to confirm #3198

Draft

holic reviewed Sep 19, 2024

View reviewed changes

holic previously approved these changes Sep 19, 2024

View reviewed changes

change library map interface

701b78a

alvrs dismissed holic’s stale review via 701b78a September 19, 2024 12:11

remove unrelated change

50d4b9c

vercel bot deployed to Preview September 19, 2024 12:14 View deployment

vercel bot deployed to Preview September 19, 2024 12:19 View deployment

default prepareDeploy param

0bce659

vercel bot deployed to Preview September 19, 2024 12:26 View deployment

holic approved these changes Sep 19, 2024

View reviewed changes

holic merged commit 22c37c3 into main Sep 19, 2024
12 checks passed

holic deleted the alvrs/prepare-deploy branch September 19, 2024 12:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(cli): improve performance of linked library resolution during deployment #3197

fix(cli): improve performance of linked library resolution during deployment #3197

alvrs commented Sep 18, 2024 •

edited

Loading

changeset-bot bot commented Sep 18, 2024 •

edited

Loading

alvrs Sep 18, 2024

alvrs Sep 18, 2024

alvrs Sep 18, 2024

alvrs Sep 19, 2024

alvrs Sep 19, 2024

holic Sep 19, 2024 •

edited

Loading

alvrs Sep 19, 2024

holic Sep 19, 2024 •

edited

Loading

alvrs Sep 19, 2024

holic Sep 19, 2024

holic Sep 19, 2024

holic left a comment

holic left a comment

fix(cli): improve performance of linked library resolution during deployment #3197

fix(cli): improve performance of linked library resolution during deployment #3197

Conversation

alvrs commented Sep 18, 2024 • edited Loading

changeset-bot bot commented Sep 18, 2024 • edited Loading

🦋 Changeset detected

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic left a comment

Choose a reason for hiding this comment

holic left a comment

Choose a reason for hiding this comment

alvrs commented Sep 18, 2024 •

edited

Loading

changeset-bot bot commented Sep 18, 2024 •

edited

Loading

holic Sep 19, 2024 •

edited

Loading

holic Sep 19, 2024 •

edited

Loading