Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: create valid Standard JSON to verify for projects with symlinks #35

Merged
merged 6 commits into from
Dec 28, 2023

Conversation

tash-2s
Copy link
Contributor

@tash-2s tash-2s commented Dec 24, 2023

Currently, when a project includes symbolic links to Solidity files, Project::standard_json_input returns a Standard JSON Input struct that is unusable for contract verification on block explorers. solc cannot compile these contracts based on the json, leading to verification failures. This pull request addresses this issue.

I encountered this problem when using the forge verify-contract command, which internally uses this function. While forge build completes successfully, the verification command fails, indicating that the block explorer cannot find a source file.

This problem occurs in projects that contain symlinked Solidity files and when these files are imported from other files. My project, structured as a monorepo, uses external libraries installed via pnpm. These libraries, accessible from the ./node_modules/ directory, are set up with remappings in my Foundry project. However, directories under ./node_modules/ are symlinks pointing to other locations, leading to this issue.

Reproduction

I added a test named can_create_standard_json_input_with_symlink to demonstrate the issue within this repository. Also, the error can be reproduced using the forge verify-contract command and steps below:

Environment: macOS (Apple silicon), forge 0.2.0 (c312c0d 2023-12-22T00:20:29.297186000Z)

$ mkdir repro && cd repro

$ mkdir -p project/src project/node_modules dependency

$ cat << EOF > project/src/A.sol
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.23;
import "@dependency/B.sol";
contract A is B {}
EOF

$ cat << EOF > dependency/B.sol
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.23;
import "./C.sol";
contract B is C {}
EOF

$ cat << EOF > dependency/C.sol
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.23;
contract C {}
EOF

$ cat << EOF > project/foundry.toml
[profile.default]
remappings = ["@dependency/=node_modules/dependency/"]
allow_paths = ["../dependency/"]
EOF

# Create a symbolic link
$ cd project/node_modules
$ ln -s ../../dependency dependency
$ cd ../..

# Display the file structure
$ tree
.
├── dependency
│   ├── B.sol
│   └── C.sol
└── project
    ├── foundry.toml
    ├── node_modules
    │   └── dependency -> ../../dependency
    └── src
        └── A.sol

# `build` succeeds
$ cd project
$ forge build

# `verify-contract` generates an unintended json (C.sol has a host absolute path name)
$ forge verify-contract --show-standard-json-input 0x0000000000000000000000000000000000000000 A | jq . > A.json
$ cat A.json
{
  "language": "Solidity",
  "sources": {
    "src/A.sol": {
      "content": "// SPDX-License-Identifier: UNLICENSED\npragma solidity ^0.8.23;\nimport \"@dependency/B.sol\";\ncontract A is B {}\n"
    },
    "node_modules/dependency/B.sol": {
      "content": "// SPDX-License-Identifier: UNLICENSED\npragma solidity ^0.8.23;\nimport \"./C.sol\";\ncontract B is C {}\n"
    },
    "/Users/t/bench/repro/dependency/C.sol": {
      "content": "// SPDX-License-Identifier: UNLICENSED\npragma solidity ^0.8.23;\ncontract C {}\n"
    }
  },
  "settings": {
    "remappings": [
      "@dependency/=node_modules/dependency/",
      "dependency/=node_modules/dependency/"
    ],
    "optimizer": {
      "enabled": true,
      "runs": 200
    },
    "metadata": {
      "useLiteralContent": false,
      "bytecodeHash": "ipfs",
      "appendCBOR": true
    },
    "outputSelection": {
      "*": {
        "": [
          "ast"
        ],
        "*": [
          "abi",
          "evm.bytecode",
          "evm.deployedBytecode",
          "evm.methodIdentifiers",
          "metadata"
        ]
      }
    },
    "evmVersion": "paris",
    "libraries": {}
  }
}

# `solc` cannot compile using the json
$ solc --standard-json --no-import-callback A.json | jq .errors
[
  {
    "component": "general",
    "errorCode": "6275",
    "formattedMessage": "ParserError: Source \"node_modules/dependency/C.sol\" not found: No import callback.\n --> node_modules/dependency/B.sol:3:1:\n  |\n3 | import \"./C.sol\";\n  | ^^^^^^^^^^^^^^^^^\n\n",
    "message": "Source \"node_modules/dependency/C.sol\" not found: No import callback.",
    "severity": "error",
    "sourceLocation": {
      "end": 81,
      "file": "node_modules/dependency/B.sol",
      "start": 64
    },
    "type": "ParserError"
  }
]

Manually editing the host filesystem's absolute path as shown below allows the solc command to succeed.

   "node_modules/dependency/B.sol": {
     "content": "// SPDX-License-Identifier: UNLICENSED\npragma solidity ^0.8.23;\nimport \"./C.sol\";\ncontract B is C {}\n"
   },
-  "/Users/t/bench/repro/dependency/C.sol": {
+  "node_modules/dependency/C.sol": {
     "content": "// SPDX-License-Identifier: UNLICENSED\npragma solidity ^0.8.23;\ncontract C {}\n"
   }
 },

Cause

The issue arises because import paths in Solidity files are processed by the std::fs::canonicalize function. This function resolves symlinks and normalizes paths. When symlinks are resolved, it results in solc being unable to locate the corresponding source paths in the json, as it relies on Solidity import statements. Therefore, symlinks should not be resolved here. The paths should be maintained as specified in the Solidity files, except for basic normalization.

Solution

To address this, I implemented an import path normalization function and replaced the canonicalization function where necessary. Based on the Solidity documentation page, this function resolves . and .. segments/components without resolving symlinks.

The Standard JSON's source paths, for verification purposes, should be based on the project root. This allows the compiler to find sources within the json. The conversion of normalized paths to project root-based paths occurs after all sources are processed. That conversion is already implemented, so this PR doesn't need to address it. (It seems like the path conversion needs improvement, but it is a separate issue and should be handled in another PR.)

compilers/src/lib.rs

Lines 514 to 525 in b1561d8

let root = self.root();
let sources = sources
.into_iter()
.map(|(path, source)| {
let path: PathBuf = if let Ok(stripped) = path.strip_prefix(root) {
stripped.to_slash_lossy().into_owned().into()
} else {
path.to_slash_lossy().into_owned().into()
};
(path, source.clone())
})
.collect();

With the changes proposed in this PR, I confirmed the fix of the issue by building forge with this patch and testing both the reproduction case and my project. The resulting json matches the manually edited json diff mentioned earlier.

A potential downside of this approach is that the same file could be represented differently in the json if accessed via both symlinked and real paths. However, I see no issues with this, and it aligns with the behavior of the compiler's virtual filesystem.

fs::metadata(&normalized).map(|_| normalized).map_err(|err| SolcIoError::new(err, original))
}

fn clean_solidity_path(original_path: impl AsRef<Path>) -> PathBuf {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rust doesn't have this normalization function (resolves . and .. but doesn't resolve symlinks), so I wrote the function myself. The function is simple, but I found several crates that perform similar tasks, so I can switch to one of them if preferred.
e.g.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha—I'll defer to @mattsse here, but the function is so small I don't mind keeping it here instead of pulling in an additional dep, as we also have tests.

Copy link
Member

@Evalir Evalir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is awesome, thank you so much!

I haven't reviewed this deeply yet, just some nits:

fs::metadata(&normalized).map(|_| normalized).map_err(|err| SolcIoError::new(err, original))
}

fn clean_solidity_path(original_path: impl AsRef<Path>) -> PathBuf {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add some docs here? I know it's an internal function, but having docs is nice and useful to revisit this later whenever needed

fs::metadata(&normalized).map(|_| normalized).map_err(|err| SolcIoError::new(err, original))
}

fn clean_solidity_path(original_path: impl AsRef<Path>) -> PathBuf {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha—I'll defer to @mattsse here, but the function is so small I don't mind keeping it here instead of pulling in an additional dep, as we also have tests.

@tash-2s
Copy link
Contributor Author

tash-2s commented Dec 25, 2023

Thank you for your review! I've added the doc to the function.

Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is awesome, ty!

symlinks man...

before we merge this, could you please open a foundry PR with the foundry-compilers crate patched to this PR to see if there are any sideffects?

@tash-2s
Copy link
Contributor Author

tash-2s commented Dec 27, 2023

@mattsse Thank you for taking the time to review this pull request.

I've created a companion PR in foundry with this patch. Also, I have confirmed that the tests in foundry passed successfully.

@tash-2s tash-2s requested a review from mattsse December 27, 2023 19:24
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits

src/utils.rs Outdated
use path_slash::PathExt;
let normalized = PathBuf::from(dunce::simplified(&cleaned).to_slash_lossy().as_ref());

fs::metadata(&normalized).map(|_| normalized).map_err(|err| SolcIoError::new(err, original))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this call for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function normalize_solidity_import_path checks if the returned path exists in the file system. The use of fs::metadata(&normalized) is to obtain an io::Error for non-existing files without reading their entire contents. To return the same type of error as utils::canonicalize, an io::Error is necessary for SolcIoError.

While there is a specific function, Path::is_file(), that confirms if a path exists in the file system, it only returns a bool.

https://doc.rust-lang.org/std/path/struct.Path.html#method.is_file

This is a convenience function that coerces errors to false. If you want to check errors, call fs::metadata and handle its Result.

However, I have now realized that there's an alias Path::metadata() for paths. Therefore, I will switch to using this function and add a comment.

https://doc.rust-lang.org/std/path/struct.Path.html#method.metadata

src/utils.rs Outdated
Comment on lines 241 to 247
Component::ParentDir => match new_path.last() {
Some(Component::Normal(..)) => {
new_path.pop();
}
_ => {
new_path.push(component);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can use if let Some(Component::Normal) = new_path.last() {pop} else {...} here

which is a bit nicer imo

@tash-2s
Copy link
Contributor Author

tash-2s commented Dec 27, 2023

I've addressed these comments.

@tash-2s tash-2s requested a review from mattsse December 27, 2023 20:51
Copy link
Member

@mattsse mattsse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, tysm for these fixes! 🙏

@mattsse mattsse merged commit ba50be3 into foundry-rs:main Dec 28, 2023
20 checks passed
@tash-2s
Copy link
Contributor Author

tash-2s commented Dec 28, 2023

Appreciate your review on the PR. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants