Provide binary macOS releases

[ulope]

Please think about providing precompiled macOS binaries. Compiling a new solc version is excruciatingly slow locally and almost impossible on Travis CI due to maximum job runtime.

[axic]

What kind of macOS version are you using? On a early-2011 Macbook Pro with 8GB ram it doesn't take longer than 10 minutes to build.

[tbocek]

I would also like to see a precompiled macOS binary, as this would easy the maintenance for e.g., https://github.com/ether-camp/solcJ.

[axic]

Now that we have macos CI in place we could very well do this.

[michaelsbradleyjr]

Now that we have macos CI in place we could very well do this.

Please do this.

[chriseth]

Does someone know how to do this properly?

[jcaracciolo]

Hey, i am Juan working for @nomiclabs , was told to take a look at this. The project was cloned and built in a MacOS Catalina machine, with no issue. This was tested on a clean machine, only installing XCode previously, as explained in the documentation. The result was the expected build folder in which the solc binary can be found. This binary was tested on other machines with success.

With the flag correctly set here Solidity declares it is supporting MacOS of 10.14 and greater (Mojave and Catalina), which roughly correspond to 70% user market share and probably more for developers. Architecture-wise, there is no issues since all Mac machines are x86-64 processors since 2007.

It appears that the Solidity Team already has everything set up for MacOS distribution. Only thing that is left is to add this to the release process, building it on a MacOS machine CI.

I can tell the tasks in .travis.yml regarding MacOS are commented. along with some comments regarding older MacOS versions (10.9-10.12) which Solidity should not support, due to the requirement of the function std::visit. What would be the issue you guys are having with these tasks?

[chriseth]

@jcaracciolo we are "in the process" of moving away from travis. Would you be able to create a circleci config file to create MacOS artifacts that work on most MacOS machines? We already have a job that compiles a binary on CircleCI, but our main problem is that we have no experience in packaging for MacOS, which I assume is especially difficult for commandline applications like solc.

[axic]

There is no packaging we can do for macOS, just release the binary.

Packaging is achieved by homebrew or someone creating a GUI for solc (hope not).

[chriseth]

Oh right - does it make sense to provide a binary homebrew release?

[jcaracciolo]

Although a binary release in homebrew would be an improvement, most people use solc thru their enviroment tool (aka Buidler, Truffle, etc). These tools allow you pick specific version of solidity needed for your project, and they manage the download and verification. They are currently using the releases available in https://solc-bin.ethereum.org/. If the binary releases would be available there, the tools could provide the option to decide whether to use solc-js or use solc binary.

As a starting point, i would generate binary releases on github, the same way you guys are releasing linux and windows versions. Unlike windows and Linux, MacOS binaries all have the same architecture and, as i explained earlier, you guys already set the flag correctly for supporting all 10.13 and 10,14 MacOS, this means that by compiling the binary on any 10.13 or greater MacOS machine, the resulting binary is ready for distribution as it will be compatible on any other 10.13 and 10.14 MacOS

[chriseth]

Sounds great, thanks!

[shoenseiwaso]

Although a binary release in homebrew would be an improvement, most people use solc thru their enviroment tool (aka Buidler, Truffle, etc). These tools allow you pick specific version of solidity needed for your project, and they manage the download and verification.

A note that these tools use solcjs not solc, so actually the Mac binary release doesn't apply to them. I suspect 95% of Mac devs will ultimately use Homebrew to consume these binaries, for purposes other than Truffle, etc., so building both the binary, and setting up a Homebrew pipeline would be much appreciated!

[shoenseiwaso]

For example, here's Truffle:

$ truffle compile

Compiling your contracts...
===========================
> Compiling ./contracts/ConvertLib.sol
> Compiling ./contracts/MetaCoin.sol
> Compiling ./contracts/Migrations.sol
> Artifacts written to /tmp/sample/build/contracts
> Compiled successfully using:
   - solc: 0.5.16+commit.9c3226ce.Emscripten.clang   <== solcjs, not solc

[fzeoli]

The tools use solcjs because that's the only option as binaries are not distributed for Mac. We (Nomic Labs) are going to make Buidler download the native binary if it's available and use it instead of solcjs, as well as push for other tools (mainly Truffle) to do the same. Devs on Mac will end up using the binary through their existing tooling

[shoenseiwaso]

@fzeoli that's great!

Still, a Homebrew option (with, ideally, easy past version switching) would be very welcome for those of us who live mostly outside of the NPM ecosystem. For example, go-ethereum just calls solc directly, which is fine.

[alcuadrado]

(with, ideally, easy past version switching) would be very welcome for those of us who live mostly outside of the NPM ecosystem

If native solc versions were available in https://solc-bin.ethereum.org/, as @jcaracciolo mentioned, creating an nvm/rvm-like tool for solidity would be really straightforward. This kind of tool is, in general, better suited than Hombrew for managing multiple versions.

[jcaracciolo]

As far as i have seen, you guys already have a CircleCI MacOS build task that generates a solc binary as an artifact for MacOS. I tried out the resulting file and it works great. However, the file seems to be larger than expected (150MB). Is there a reason for this? Or what do you think could be the cause of the file size?

[chriseth]

It is built in debug mode: https://github.com/ethereum/solidity/blob/develop/.circleci/config.yml#L534

I actually think that this is not necessary. You can just replace it by Release.

[alcuadrado]

Hey @chriseth,

How can we help publish these build in https://solc-bin.ethereum.org/ ?

[chriseth]

First, we have to verify that they work, you can help with that :)
Then, I would prefer to publish them on the github release page, we can test that for the next release (probably happening in the next 14 days).

Would you like to also publish binaries for older packages?

What about homebrew, can we use binaries there?

[alcuadrado]

I just downloaded the binaries from the CI run of this commit 61568fd and tested them in MacOS Mojave and Catalina. Worked in both cases.

@jcaracciolo can you give more info about why Solc won't work on MacOS version < Mojave, please?

[alcuadrado]

Would you like to also publish binaries for older packages?

Yes, that would be ideal.

What about homebrew, can we use binaries there?

I think so. We'll dig deeper into this and report back.

[jcaracciolo]

@alcuadrado It is explained here. Solidity makes use of std::visit all throughout the code. In order to use that function, the target has to be set to 10.14 or higher or else an error will be prompted: Call to unavailable function 'visit': introduced in macOS 10.14. Even though it may sometimes work on older versions, one can't be sure since the deployment target is raised.

[alcuadrado]

Then, I would prefer to publish them on the github release page, we can test that for the next release (probably happening in the next 14 days).

The 0.6.9 release page doesn't include a macos binary. Will it be added? or should we wait for 0.6.10?

Is this blocking adding binary builds to solc-bin?

[chriseth]

Oh I'm sorry! I just added the binary - can you try if it works? I named it solc-macos - is that a name that works in the macos naming scheme?

[alcuadrado]

Thanks, @chriseth !

I tested it on Catalina and High Sierra, and it works.

The namesolc-macos makes sense.

[fzeoli]

@chriseth next step for this is to get the binaries built for all the previous versions and published in https://solc-bin.ethereum.org/, that's when we can get tools to use it. Is there anything we can do to help do this?

[chriseth]

We need an architecture about how to organize and publish the binaries. github pages is not a solution because it does not scale to our needs. One problem we encountered was that some ways to host the binaries does not provide the correct mime type for the solcjs binaries.

[Elyx0]

Did 0.8.16 reintroduce this bug?

forge test
[⠊] Compiling...
[⠒] installing solc version "0.8.16"
[⠆] Successfully installed solc 0.8.16
Error:
Solc Error: dyld[74016]: Library not loaded: /opt/homebrew/opt/z3/lib/libz3.dylib
  Referenced from: /Users/me/.svm/0.8.16/solc-0.8.16
  Reason: tried: '/opt/homebrew/opt/z3/lib/libz3.dylib' (no such file), '/usr/local/lib/libz3.dylib' (no such file), '/usr/lib/libz3.dylib' (no such file)

Going 8.14 -> 8.15 had no issues.

[cameel]

I think you just don't have Z3 installed. Looks like forge must have switched to linking to Z3 dynamically on macOS. We don't provide binaries for M1 right now but the ones for Intel macs have Z3 linked statically.

[leonardoalt]

I think SVM (binaries that Foundry uses) already fixed that.