Adding solc binaries to solc-bin

[alcuadrado]

I'm creating this issue to discuss/track adding Windows, Linux, and MacOS binaries of solc to solc-bin.

We've already talked about this in multiple places, but there was no issue to keep track of its progress. The closes thing was #3168, but that's ready so that issue is no longer relevant.

Abstract & Motivation

The reasons to do this are:

1. It can be used by JS tools like truffle and buidler to improve the compilation speed.

2. Tools that are not written in JS need this to simplified their required setup. Now they only work with docker or a github API key. @iamdefinitelyahuman can provide more info about this if needed.

Specification

Just like with asm.js and wasm, binary versions for each of the major OS should be added.

Backwards Compatibility

This doesn't introduce any change to the language.

To make the effort effective all the versions in solc-bin should be compiled and published, except maybe nightlies.

[chriseth]

It turns out that in the meantime, we hit the size limit of github-pages, so we need to find a new solution on where to put released binaries anyway. It is a good idea to combine different architectures.

Maybe to clarify this a bit more: This repository would contain statically-linked binaries that are as portable as possible. Distributing other binaries should be done through the respective package manager of the OS.

[iamdefinitelyahuman]

Very excited about this!

With py-solc-x I'm pulling the binaries from Github, and can confirm that it creates a headache when used in the CI. Without an access token it almost always fails from rate limiting. Having binaries served elsewhere will be a huge help to me.

Re OSX: I wrote a wiki article to field the repeated questions, but I think more often than not I end up losing potential new users who hit the issue. Especially when working on the mainnet Brownie installs many solc versions to locally compile source codes from Etherscan. The delay in compiling solc, combined with uncertainty that it won't fail at 90%, is just too much.

[cameel]

A little update on the progress here:

• Linux, MacOS and Windows binaries are now available in solc-bin repo. You can see the matrix of available versions in my comment in #42
  • We currently only have MacOS builds starting from v0.6.9 on but we're planning to build older versions and add them to solc-bin too.
• Due to size constraints of GH pages the new binaries are not accessible via solc-bin.ethereum.org yet. They can't be included until we start hosting them elsewhere and switch over the domain. We're in the process of setting that up.
  • By the way, please don't use the solc-bin.github.io domain directly. We don't control that domain so we can't transparently switch that over. We'll keep the soljson.js files available there not to break tools that have it hard-coded but we won't be able to make binaries for other platforms available that way.
• We set up a new directory structure for the binaries, segregated by platform. The old soljson.js binaries are still available in bin/ and wasm/ but they're also symlinked under new names. Please use the new ones whenever possible.
• We'd like to make the content of solc-bin available via IPFS as well, though not necessarily as a part of this task.

[alcuadrado]

This is great news, @cameel! Thanks for the update.

[cameel]

If anyone wants to try the new setup, binaries are already available via https://solc-bin.s3.eu-central-1.amazonaws.com with no rate limiting. The paths are exactly as in solc-bin repo, e.g. https://solc-bin.s3.eu-central-1.amazonaws.com/macosx-amd64/list.json.

I have just finished setting up mirroring of solc-bin to this S3 bucket and it contains all the releases (including the latest one). My mirroring script still has to go through review and the official domain has not been switched over to point at S3 so please don't treat it as production-ready yet but I expect the transition to be smooth.

Now I need to deal with some smaller left-over tasks related to solc-bin and solc-js and when I'm done, the Mac OS X builds are next on my todo list.

[cameel]

The mirroring is now up and running. I have also just asked the devops team to switch over the domain (https://github.com/ethereum/devops/issues/499) so you can expect solc-bin.ethereum.org to start pointing at the S3 bucket very soon.

[cameel]

I got info from the devops team that the domain has been switched over today in the morning.
Changes are not visible in DNS yet but once the cached old DNS entries expire, https://solc-bin.ethereum.org should start serving from S3.
Please let me know if you notice any breakage.

[cameel]

We had some unexpected problems with the switchover (DNS was updated but Cloudfront kept serving the data from GH pages) but it's solved and the new binaries are finally available at solc-bin.ethereum.org.

[cameel]

Good news. I managed to rebuild all solc versions down to 0.3.6 on MacOS. For now they're available in my fork of solc-bin: https://github.com/cameel/solc-bin/tree/macosx-static-binaries/macosx-amd64 (ethereum/solc-bin#54). Please try them out and let me know if you notice any issues (especially with the older ones).

The script for building them is currently being reviewed (ethereum/solc-bin#53).

[cameel]

Also, another piece of news related to the binaries:

We recently updated our CI to produce completely static Windows builds (#9811) so packing them in a .zip is no longer necessary. In one of the upcoming releases we're going to start putting uncompressed .exe files in solc-bin.

[alcuadrado]

Awesome news, @cameel ! Thanks

[jcaracciolo]

Great news! Will definitely improve and accelerate tooling development

[cameel]

The PR has been merged. MacOS binaries are now available at https://solc-bin.ethereum.org

[alcuadrado]

Thanks @cameel! We are already integrating this into Buidler 🥳

[fzeoli]

This is amazing news

…

On Tue, Sep 29, 2020 at 5:15 PM Patricio Palladino ***@***.***> wrote: Thanks @cameel <https://github.com/cameel>! We are already integrating this into Buidler 🥳 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9226 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABYV3RRSPWGLEPG4GPMVULSII54TANCNFSM4OA26DSQ> .

-- Franco Zeoli CEO @zfran <https://twitter.com/zfran> | @nomiclabs <https://twitter.com/nomiclabs>

[cameel]

I think we're done here so I'm closing the issue. There are a few minor things related to the release process covered by #9258 (see #9258 (comment)) but I think that the core issue is solved.