-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tools for deterministic builds of standalone binaries and distribution archives #121
Conversation
Concept ACK. Will review after 120 is merged. I'm thinking of adding a (mandatory?) |
3521cf1
to
fd79eb0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not really well-equipped to review this.
What's the suggested review process?
docs/release-process.md
Outdated
@@ -0,0 +1,8 @@ | |||
# Release Process | |||
|
|||
1. Bump version number in `pyproject.toml`, generate the setup.py file, and tag release |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
git tag? what tag?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Git tag.
Clarified.
fd79eb0
to
ade217c
Compare
ade217c
to
2b5f16f
Compare
The best way to review this is probably to just have everyone do the build and check that their hashes match. All of the builds can be done with These are what I got on commit 2b5f16f:
Once I get access to a Mac, I'll do the MacOS build too. MacOS build:
|
I had to jump through some hoops on macOS, including For macOS maybe point to The wine part doesn't work for me, see logs. I'm get the same checksum for
|
For macOS, I had issues getting faketime to work. I have a fix for poetry that removes the need to use faketime but is waiting to be merged. The difference for I think the reason the resulting binaries are different are because the installed python binaries are different since we are using pyenv which compiles locally. I'll have to investigate this. |
d54402d
to
88ce6ab
Compare
docs/release-process.md
Outdated
|
||
Now install [Poetry](https://github.com/sdispater/poetry) with `pip install --user poetry` | ||
|
||
You will also need to install `faketime`. On Ubuntu/Debian: `sudo apt install faketime`. On Mac: `brew install faketime` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
brew install libfaketime
Nice. Getting setup was easy (on macOS Building with 88ce6ab, I'm getting these hashes:
When I try and do the windows build
|
pyproject.toml contains everything that was in the setup.py. The setup.py file is replaced with the one that poetry automatically generates.
00f818f
to
c47eb22
Compare
I've investigated the non-determinism and I think I've finally fixed it. Since for linux and macOS we are using pyenv which self compiles python, the python installation that each person was using was non-deterministic and thus introducing non-determinism. There are now instructions in I've also added more to the Lastly I've added another travis job that builds the binary and checks that it works. Below are the hashes of the result of the complete build as of 464c51d .
I haven't done the macOS build yet. I will do so once I get to a mac. I will also investigate the wine issue on macOS. |
c47eb22
to
464c51d
Compare
Tested building at 464c51d: zipped
|
It looks like faketime isn't working on mac which is why dist/hwi-0.0.5.tar.gz have different hashes. |
2bf186c
to
44269e3
Compare
I've updated the dockerfile to use debian stretch-slim as the base image. There's also a travis job that runs the determinstic build, tests the linux binary, and outputs the hashes of the result. Note that it doesn't do the Windows build because that (for some reason) has a tendency to hang. Another travis job does the macOS build but does not run any tests. The hashes I got were:
|
I'm seeing the same hashes as you:
As mentioned, sometimes the Windows build hangs (using |
56674ab
to
346f624
Compare
Adds builds scripts that are used to build releases deterministically. Also adds documentation that explains the release process and what the build scripts do.
346f624
to
b3f750d
Compare
b3f750d
to
8931ae0
Compare
I've made a few changes to Can someone with a mac test that this binary works and actually communicates with devices? It can be downloaded from https://s3.amazonaws.com/hwi-travis-builds/achow101/hardware-wallet-interface/114/114.4/hwi. This was built in a travis job on my personal fork of this repo: https://travis-ci.com/achow101/hardware-wallet-interface/jobs/183298042. |
wget https://s3.amazonaws.com/hwi-travis-builds/achow101/hardware-wallet-interface/114/114.4/hwi
sha256sum hwi
d9acc15b507ce2079134b372e68278f71780ff0c762249c906778aefdf47f5f4 hwi
chmod +x hoi
|
Merging as there is agreement on the produced binary hashes and that they all work. |
… distribution archives 8931ae0 Run tests using the binary distribution (Andrew Chow) 39a6fc9 Fixes for Windows (Andrew Chow) a229de0 Update .travis.yml to use poetry for build (Andrew Chow) 77257a1 Add build scripts and documentation for building releases (Andrew Chow) 9e04d1a Add a hwi.spec file for pyinstaller to build standalone binaries (Andrew Chow) d6b24b8 Add pyproject.toml and poetry.lock for poetry dependency manager (Andrew Chow) Pull request description: This PR adds several scripts and tools for making standalone binaries of the `hwi.py` script and for creating distribution archives that will go on pypi.org. To achieve deterministic builds, the dependencies used must be locked to specific versions and hashes. To do this, I have added configuration files for using the [Poetry dependency manager](https://github.com/sdispater/poetry). Because Poetry uses a `pyproject.toml` file instead of `setup.py`, I have created a helper script which will automatically generate the proper `setup.py` file from `pyproject.toml`. The reason I chose Poetry instead of Pipenv for this task is because it has the ability to do deterministic builds of the distribution archives (python wheel and source tar for pypi.org) which Pipenv does not have. Additionally scripts have been added to the newly created `contrib/` folder which will perform the deterministic builds of the binaries and distribution archives. The builds of the binaries are done using [pyinstaller](http://www.pyinstaller.org/). In order to build for different platforms, the `contrib/build_bin.sh` script needs to be run on each of the platforms we wish to release for. It can also be run in wine to do windows builds (see `contrib/build_wine.sh`). The configuration file that pyinstaller needs has also been added. Lastly the pyenv version for this project has been bumped to 3.6.8 since using python 3.5.x produced standalone binaries that did not work. This PR is built on #120 as reducing the number of dependencies fixed several issues with the standalone binary builds. Tree-SHA512: abc1a6ac06d663b1316cde254980b0b1e8c392a6ffe478710df7c8e48a344cd57105e83555ec8fdcdc30e2b7d6d9cd6464367afda80653cb3cbc3acaf6119f48
This PR adds several scripts and tools for making standalone binaries of the
hwi.py
script and for creating distribution archives that will go on pypi.org.To achieve deterministic builds, the dependencies used must be locked to specific versions and hashes. To do this, I have added configuration files for using the Poetry dependency manager. Because Poetry uses a
pyproject.toml
file instead ofsetup.py
, I have created a helper script which will automatically generate the propersetup.py
file frompyproject.toml
. The reason I chose Poetry instead of Pipenv for this task is because it has the ability to do deterministic builds of the distribution archives (python wheel and source tar for pypi.org) which Pipenv does not have.Additionally scripts have been added to the newly created
contrib/
folder which will perform the deterministic builds of the binaries and distribution archives. The builds of the binaries are done using pyinstaller. In order to build for different platforms, thecontrib/build_bin.sh
script needs to be run on each of the platforms we wish to release for. It can also be run in wine to do windows builds (seecontrib/build_wine.sh
). The configuration file that pyinstaller needs has also been added.Lastly the pyenv version for this project has been bumped to 3.6.8 since using python 3.5.x produced standalone binaries that did not work.
This PR is built on #120 as reducing the number of dependencies fixed several issues with the standalone binary builds.