[cli] New alg for tree shaking #21356

stefan-mysten · 2025-02-26T01:27:19Z

Description

This PR updates the tree shaking algorithm to use linkage tables on-chain. We need to use that because implicit dependencies feature that is being worked in #21204 would not work with this algorithm. This is due to the way transitive dependencies are computed in the previous tree shaking algorithm. Implicit dependencies will add those system dependencies to the transitive dependencies, thus system packages will get included in the final list of dependencies in the package that is to be published/upgraded after tree shaking is complete.

Test plan

Existing tests. Updated a test for system packages.

cd crates/sui
cargo nextest run -- cli_tests

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2025-02-26T01:27:28Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 6, 2025 0:44am

2 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview		Mar 6, 2025 0:44am
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview		Mar 6, 2025 0:44am

crates/sui-move/src/build.rs

amnn

The changes around sui move build look good to me, but the new algorithm does not seem quite right. I have suggested a fix and a test that would highlight the issue.

crates/sui/src/client_commands.rs

amnn · 2025-02-27T13:31:18Z

crates/sui/src/client_commands.rs

+    compiled_package
+        .dependency_ids
+        .published
+        .retain(|pkg_name, id| {
+            linkage_table_ids.contains(id)
+                || pkgs.contains_key(pkg_name)
+                || pkg_name_to_orig_id
+                    .get(pkg_name)
+                    .is_some_and(|orig_id| pkg_ids.contains(orig_id))
+        });


This does not seem right:

compiled_package.dependency_ids.published is a list of storage IDs (they point to the actual version of the package that you want to use)

similarly, pkg_ids is also a list of storage IDs, because it was generated by filtering the same list.

linkage_table_ids is a list of original IDs (they point to the first version of the package you want to use, regardless of its version), because that is what the key in the linkage table is.

values in pkg_name_to_orig_id are also original IDs.

In this code, we are freely comparing original IDs and storage IDs -- this should only work for packages that have never been upgraded -- have you tested this approach in cases involving upgraded packages? The following example should surface an issue if there is one:

B is published depending on C v1

A is published depending on B and C v2, but does not directly refer to anything in C v2 (the dependency just serves to upgrade B's dependency).

In that example, A's dependency list should include both B, which it does use, and C v2, which it doesn't use but is an upgrade of one of its immediate dependency's transitive dependencies.

The retention test goes something like this, I think:

compiled_package .dependency_ids .published .retain(|pkg_name, id| { pkgs.contains_key(pkg_name) || pkg_name_to_orig_id .get(pkg_name) .is_some_and(|orig_id| linkage_table_original_ids.contains(orig_id)) })

I added this test and fixed this part, I think, but I cannot get the test to work properly. It's working when following the same steps with a CLI build. I think the issue is around publishing without tree shaking using old test-transaction-builder, which does not create a Move.lock file.

Need to think a bit what to do.

Ok, so here's the new test for it
K, K_v2, L_depends_on_K (with code reference to K), and M_depends_on_L_and_K_v2, with code calling function from L_depends_on_K package, but no code reference to K_v2.

All should work, but I have an intuition that all this breaks if there are no Move.lock files and only Move.toml with published-at field. I need to understand how things worked before automated address mgmt.

Added the test that uses the "old" way of publishing/upgrading, without automated address management, just for my sanity check.

crates/sui/src/client_commands.rs

packages to keep for deps.

@stefan-mysten

…#21383) ## Description This PR adds support for implicit dependencies to the package management at the move layer. ## PR stack #21204 Add implicit Sui dependencies #21384 Refactors to BuildConfig #21383 Add support for implicits to move package management This also depends on @stefan-mysten 's ongoing tree-shaking work in #21356 ## Test plan See `test_sources/implicits/README.md` for a description of the new snapshot tests --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

@stefan-mysten

## Description This removes `sui_move_build::BuildConfig::default` and makes other small changes ## PR stack #21204 Add implicit Sui dependencies #21384 Refactors to BuildConfig #21383 Add support for implicits to move package management This also depends on @stefan-mysten 's ongoing tree-shaking work in #21356 ## Test plan This disables two broken bytecode verification tests. See DVX-786 --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK:

amnn

There are still issues with how this is written, PTAL at the inline comments.

crates/sui/src/client_commands.rs

amnn · 2025-03-10T20:16:05Z

crates/sui/src/client_commands.rs

+                        pkg_id_to_name.get(&object_id).unwrap()
+                    )
+                }
+                _ => anyhow!("Cannot convert data into an object: {e}"),


This looks like a call to context in disguise.

amnn · 2025-03-10T20:16:52Z

crates/sui/src/client_commands.rs

+            p.to_move_package(u64::MAX /* safe as this pkg comes from the network */)
+                .map_err(|e| anyhow!(e))


Would this work?

Suggested change

p.to_move_package(u64::MAX /* safe as this pkg comes from the network */)

.map_err(|e| anyhow!(e))

p.to_move_package(u64::MAX /* safe as this pkg comes from the network */)?

Unfortunately no, because p_to_move_package returns a sui_types::ExecutionError and here we need to return an anyhow::Error.

amnn · 2025-03-10T20:25:07Z

crates/sui/src/client_commands.rs

+        // orig id
+        for (name, id) in &pkg_name_to_orig_id {
+            if *linkage_orig_id == id {
+                pkgs_to_keep.insert(name);
+            }
+        }


This is accidentally quadratic -- create a reverse mapping once to make this look-up efficient.

amnn · 2025-03-10T20:26:51Z

crates/sui/src/upgrade_compatibility/mod.rs

Spurious change?

Not really, it's an intended change. I modified before the code to pass SuiClient rather than ReadApi, and now I reverted that to use ReadApi everywhere where's possible, so it comes from that refactoring. Sorry for the confusion.

amnn · 2025-03-10T20:41:20Z

crates/sui/src/client_commands.rs

+        .map(|(name, id)| (id, name))
+        .collect();
+    let immediate_dep_move_pkgs =
+        fetch_move_packages(read_api, &immediate_dep_pkgs_ids, &pkg_id_to_name).await?;


There seems to be a fair bit of massaging of data coming out of find_immediate_deps_pkgs_to_keep to make it work for fetch_move_packages. Does that need to happen here?

Could the data come out of find_immediate_deps_pkgs_to_keep in a more appropriate form for what it's about to be used for here?

Could any other processing happen inside fetch_move_packages?

crates/sui/src/client_commands.rs

amnn · 2025-03-10T20:49:17Z

crates/sui/src/client_commands.rs

+    let immediate_dep_pkgs_linkage_tables: BTreeMap<_, _> = immediate_dep_move_pkgs
+        .iter()
+        .flat_map(|pkg| pkg.linkage_table())
+        .collect();
+
+    // for every immediate dep package, we need to use its linkage table to determine its
+    // transitive dependencies and ensure that we keep the required packages
+    let mut pkgs_to_keep: BTreeSet<&Symbol> = BTreeSet::new();
+    let published_pkgs = compiled_package.dependency_ids.published.clone();
+    for (linkage_orig_id, upgrade_info) in &immediate_dep_pkgs_linkage_tables {
+        // orig id
+        for (name, id) in &pkg_name_to_orig_id {
+            if *linkage_orig_id == id {
+                pkgs_to_keep.insert(name);
+            }
+        }
+
+        // for dependencies on pkgs that are upgrades, find the name of the published
+        // package by matching the upgraded package id with the published package id
+        for (name, id) in &published_pkgs {
+            if &upgrade_info.upgraded_id == id {
+                pkgs_to_keep.insert(name);
+            }
+        }
+    }


This seems suspicious for a couple of reasons:

immediate_dep_pkgs_linkage_tables saves all upgrade info, but if two immediate deps depend on the same transitive dep at different versions, then you will arbitrarily pick one.

In the upgraded package case, you are checking that the dependency exactly matches the upgraded ID of one of your dependency's dependencies.

These two details alone should set off alarm bells because what if you overwrote the version of the dependency that you meant to match with here? But the situation is worse than that, because the package being published is allowed to upgrade any of its dependency's dependencies as well, and this would not allow for that.

The correct check is to keep every published ID whose corresponding original ID shows up as a key in the linkage table of one of the immediate dependencies.

stefan-mysten self-assigned this Feb 26, 2025

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 26, 2025 01:27 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 26, 2025 01:29 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 7b3494c to 3fd4ec1 Compare February 26, 2025 02:12

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 26, 2025 02:12 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 26, 2025 02:13 View deployment

stefan-mysten commented Feb 26, 2025

View reviewed changes

crates/sui-move/src/build.rs Show resolved Hide resolved

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 3fd4ec1 to f1af2fb Compare February 26, 2025 02:56

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 26, 2025 02:56 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 26, 2025 02:58 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from f1af2fb to ec0080a Compare February 26, 2025 03:52

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 26, 2025 03:52 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 26, 2025 03:53 View deployment

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 26, 2025 22:17 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 26, 2025 22:19 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 8b24a12 to bde8329 Compare February 26, 2025 22:22

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 26, 2025 22:22 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 26, 2025 22:23 View deployment

stefan-mysten requested review from amnn and mdgeorge4153 February 26, 2025 22:27

amnn reviewed Feb 27, 2025

View reviewed changes

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from bde8329 to 89e3602 Compare February 27, 2025 19:52

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 27, 2025 19:52 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 27, 2025 19:54 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 89e3602 to 2a77055 Compare February 27, 2025 19:59

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env February 27, 2025 19:59 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 27, 2025 20:00 View deployment

This was referenced Feb 28, 2025

[Package management] add implicit system package dependencies (dvx-197) #21204

Open

[package management] Add implicit dependency support to move compiler #21383

Merged

mdgeorge4153 mentioned this pull request Feb 28, 2025

Refactors to BuildConfig #21384

Merged

7 tasks

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env March 1, 2025 00:12 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 1, 2025 00:14 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from b28b84f to 020d05a Compare March 1, 2025 04:00

stefan-mysten requested a review from ronny-mysten as a code owner March 1, 2025 04:00

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env March 1, 2025 04:00 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 1, 2025 04:01 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 020d05a to 814ab75 Compare March 4, 2025 01:00

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env March 4, 2025 01:00 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 4, 2025 01:02 View deployment

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 814ab75 to 07c4799 Compare March 4, 2025 01:31

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env March 4, 2025 01:32 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 4, 2025 01:33 View deployment

stefan-mysten requested a review from amnn March 4, 2025 05:03

New alg for tree shaking that uses linkage tables for determining which

ef77a66

packages to keep for deps.

stefan-mysten force-pushed the tree_shaking_with_linkage_table branch from 07c4799 to ef77a66 Compare March 6, 2025 00:43

stefan-mysten temporarily deployed to sui-typescript-aws-kms-test-env March 6, 2025 00:43 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 6, 2025 00:44 View deployment

amnn reviewed Mar 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cli] New alg for tree shaking #21356

[cli] New alg for tree shaking #21356

stefan-mysten commented Feb 26, 2025 •

edited

Loading

vercel bot commented Feb 26, 2025 •

edited

Loading

amnn left a comment

amnn Feb 27, 2025

stefan-mysten Mar 1, 2025

stefan-mysten Mar 1, 2025 •

edited

Loading

stefan-mysten Mar 6, 2025

amnn left a comment

amnn Mar 10, 2025

amnn Mar 10, 2025

stefan-mysten Mar 10, 2025

amnn Mar 10, 2025

amnn Mar 10, 2025

stefan-mysten Mar 10, 2025

amnn Mar 10, 2025

amnn Mar 10, 2025

		p.to_move_package(u64::MAX /* safe as this pkg comes from the network */)
		.map_err(\|e\| anyhow!(e))

[cli] New alg for tree shaking #21356

Are you sure you want to change the base?

[cli] New alg for tree shaking #21356

Conversation

stefan-mysten commented Feb 26, 2025 • edited Loading

Description

Test plan

Release notes

vercel bot commented Feb 26, 2025 • edited Loading

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefan-mysten Mar 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefan-mysten commented Feb 26, 2025 •

edited

Loading

vercel bot commented Feb 26, 2025 •

edited

Loading

stefan-mysten Mar 1, 2025 •

edited

Loading