Deterministic Builds #3
Labels
Milestone
Comments
maltfield
added a commit
that referenced
this issue
Jul 3, 2020
* #3 I confirmed that our linux build is *not* deterministc because: 1. Our AppImage is shipping with a ton of __pycache__/.../*.pyc files that differ from the GitHub build and our local build 2. The contents of buskill_version.py breaks determinism. I can manually set GITUB_SHA and GITHUB_REF on local builds (attempted in this commit), but GITHUB_RUN breaks determinism, so it's been removed
|
I'm starting with this task starting with Linux. I had two issues with my linux build script that was preventing it from being deterministic:
I fixed those in this commit 0a98242 Unfortunately, I found the build from GitHub and my local build on a Debian 10 DispVM produced distinct checksums. I extracted the AppImage and found the dirs identical. I built a new AppImage from one of the build's extracted It looks like the issue is related to In the meantime, it looks like some other projects such as the electrum crypto wallet have hacked |
maltfield
added a commit
that referenced
this issue
Jul 4, 2020
* #3 Unfortunately the latest stable version of appimagetool (v12 from 2019-05) uses a very old version of mksquashfs (v4.3 from 2014-05) which does not support building reproducable builds due to some parallelization optimizations * https://github.com/AppImage/AppImageKit/releases * https://sourceforge.net/p/squashfs/code/ci/master/tree/CHANGES * AppImage/AppImageKit#929 (comment) Other projects like the electurm crypto wallet got around this issue by hacking the latest stable appimagetool to use a fork of squashfs-tools = squashfskit * https://github.com/squashfskit/squashfskit * https://github.com/SomberNight/electrum/blob/ae714772c38410a0169f2c76a14a64a62c0daff0/contrib/build-linux/appimage/build.sh#L212-L225 But in 2019-08, squashfs-tools released an updated version (v4.4) which added support for reproducable builds, as well as fixed a couple CVEs and other changes * https://lore.kernel.org/lkml/CAB3wodcL=gnQOmHGGNukWK3OUbU2p=OHzLmzPi7ns_WNTGBEwg@mail.gmail.com/ Therefore, in this commit, I've added the code to make changes to appimagetool's latest release to use the updated version of squashfs-tools. The asked the maintainer of AppImageKit about this, and they said there's no future releases expected for the existing appimagetools repo, as development has been moved to a Go version, which is currently still a work-in-progress with no eta on it becoming stableo * https://github.com/SomberNight/electrum/blob/ae714772c38410a0169f2c76a14a64a62c0daff0/contrib/build-linux/appimage/build.sh#L212-L225 So, for now, this hack is as good as it gets.
|
Linux deterministic builds are complete. For a diff of the changes (and associated commit messages) needed to achieve this, see: After the changes listed in the comment above, this is what was required:
|
|
I hit a wall trying to get I created a repo specific to this task here: And I filed a bug report with Per my last comment, it might be that the order of the dependencies written to the binary are out-of-order and |
|
I added a windows build CI process to the workflow in my pyinstaller-deterministic-test repo above to see if PyInstaller could successfully produce reproducible builds in Windows of a simple 1-line hello world app. Sadly, it couldn't. I added this info to a new comment in the bug report I created yesterday with the PyInstaller team, but this actually might require a distinct patch from the MacOS issues found above. Sadly, I'm not sure there's much I can do to make BusKill's builds deterministic in MacOS/Windows (other than abandoning PyInstaller?) until this is fixed upstream. |
maltfield
added
enhancement
maltfield
added a commit
that referenced
this issue
Jul 12, 2020
This is the first commit where I can say that this app is finally functional on all 3 platforms: Linux, Windows, and MacOS. Further testing will be done, but I won't be implmeneting new features until our first release of v0.1.0 is out Before I can make a release, we need to address some prerequisite tasks around documentation and security: 1. I need a proper documentation solution. I'll probably use Sphinx / Read The Docs for this 2. I need to sign all my commits on github with gpg 3. I need to update the build scripts to verify the integrity of any depends being downloaded 4. I need to make sure that the builds are deterministic. Any files produced by the build from the GitHub's CI process should match exactly the files produced by the bulild from my local machine. 5. I need to find out if it's possible for me to upload a (distinct, purpose-created) pgp private key to GitHub somehow such that it's secure/not-public and can be used by the CI process to cryptographically sign build artifacts 6. I need to setup TUF to sign releases. This will also be leveraged in a later release when I implement a self-updater feature into this app. I hope that this commit solves #2 above, as I've added a signing subkey to my primary, master gpg key and copied it to my development environment, uploaded my updated public key to my GitHub account, and enabled commit signing by default in this repo's remote sandbox on my machine. Hopefully this commit will be signed and GitHub will give it a green checkbox. As part of addressing #3, my build process currently requires downloading two dependencies from pip. Pip does not provide cryptographic authentication or integrity checks, but it does have a system for project maintainers to sign their releases with gpg and upload that to PyPI using twine. I've opened tickets requesting that the developers for the projects that we depend on begin to sign their releases to PyPI so that our build process can be more secure. In the meantime, I should 3TOFU a set of checksums and update the build scripts to fail if the integrity doesn't match the hard-coded checkums. * vpelletier/python-libusb1#54 * kivy/kivy#6973
|
I made a few final changes to the linux build script so our linux builds are reproducible across the github shared runners and my laptop Note that, as of now, our first (alpha) release is out! |
maltfield
added a commit
that referenced
this issue
Sep 8, 2020
For some reason the .app built from GitHub spits this out and exits immediately where it *should* give me a pdb shell.
10:44:35,169 root DEBUG ========================================================
=======================
10:44:35,169 root DEBUG BUSKILL_VERSION|{'VERSION': '1599550992', 'GITHUB_REF':
'refs/heads/dev', 'GITHUB_SHA': 'ba2300aee40f587dbc652b284088f6f561cc7908', 'SOU
RCE_DATE_EPOCH': '1599550992'}|
10:44:35,169 root DEBUG os.environ|environ({'PWD': '/Users/maltfield/Downloads',
'TERMCAP': 'SC|screen|VT 100/ANSI X3.64 virtual terminal:\\\n\t:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:\\\n\t:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:\\\n\t:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:\\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:\\\n\t:li#47:co#80:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:\\\n\t:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:\\\n\t:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:\\\n\t:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:\\\n\t:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:\\\n\t:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mr=\\E[7m:me=\\E[m:ms:\\\n\t:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:\\\n\t:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:\\\n\t:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\\\n\t:po=\\E[5i:pf=\\E[4i:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:\\\n\t:k4=\\EOS:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:\\\n\t:k9=\\E[20~:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:F3=\\E[1;2P:\\\n\t:F4=\\E[1;2Q:F5=\\E[1;2R:F6=\\E[1;2S:F7=\\E[15;2~:\\\n\t:F8=\\E[17;2~:F9=\\E[18;2~:FA=\\E[19;2~:kb=^H:K2=\\EOE:\\\n\t:kB=\\E[Z:kF=\\E[1;2B:kR=\\E[1;2A:*4=\\E[3;2~:*7=\\E[1;2F:\\\n\t:#2=\\E[1;2H:#3=\\E[2;2~:#4=\\E[1;2D:%c=\\E[6;2~:%e=\\E[5;2~:\\\n\t:%i=\\E[1;2C:kh=\\E[1~:@1=\\E[1~:kH=\\E[4~:@7=\\E[4~:\\\n\t:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:ku=\\EOA:kd=\\EOB:\\\n\t:kr=\\EOC:kl=\\EOD:km:', 'USER': 'maltfield', 'SSH_TTY': '/dev/ttys001', 'LANG': 'en_US.UTF-8', 'WINDOW': '3', 'TERM': 'screen', 'LOGNAME': 'maltfield', 'STY': '1277.buskill', 'PATH': '/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin', 'SHLVL': '3', 'OLDPWD': '/Users/maltfield/Downloads/buskill-1599545602.app', 'SSH_CLIENT': '110.44.115.190 17388 4518', 'SHELL': '/bin/zsh', '_': '/usr/bin/open', 'HOME': '/Users/maltfield', '__CF_USER_TEXT_ENCODING': '0x1F6:0:2', 'TMPDIR': '/var/folders/kx/2fp3kfgj4dj7rx9s5mlb52640000gp/T/', 'SSH_CONNECTION': '110.44.115.190 17388 207.254.29.173 4518', 'XPC_SERVICE_NAME': 'buskill.9148', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.Bx0RjQIl42/Listeners', 'XPC_FLAGS': '0x0', 'KIVY_DATA_DIR': '/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/kivy_install/data', 'KIVY_MODULES_DIR': '/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/kivy_install/modules', 'GST_REGISTRY_FORK': 'no', 'GST_PLUGIN_PATH': '/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS:/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/gst-plugins', 'GST_REGISTRY': '/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/registry.bin', 'GST_PLUGIN_SYSTEM_PATH': ''})|
10:44:35,169 root DEBUG sys.argv|['/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/buskill']|
10:44:35,169 root DEBUG sys.builtin_modules_names|('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype', 'zipimport')|
10:44:35,169 root DEBUG sys.executable|/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/buskill|
10:44:35,170 root DEBUG sys.path|['/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/base_library.zip', '/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS']|
10:44:35,170 root DEBUG sys.platform|darwin|
10:44:35,170 root DEBUG sys.prefix|/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS|
10:44:35,170 root DEBUG sys.version|3.7.8 (default, Jul 4 2020, 10:17:17)
[Clang 11.0.3 (clang-1103.0.32.62)]|
10:44:35,170 root DEBUG sys.api_version|1013|
10:44:35,170 root DEBUG sys.version_info|sys.version_info(major=3, minor=7, micro=8, releaselevel='final', serial=0)|
10:44:35,170 root INFO buskill version {'VERSION': '1599550992', 'GITHUB_REF': 'refs/heads/dev', 'GITHUB_SHA': 'ba2300aee40f587dbc652b284088f6f561cc7908', 'SOURCE_DATE_EPOCH': '1599550992'}
10:44:35,225 kivy INFO [Logger ] Record log in /Users/maltfield/.kivy/logs/kivy_20-09-08_11.txt
10:44:35,225 kivy INFO [Kivy ] v1.11.1
10:44:35,226 kivy INFO [Kivy ] Installed at "/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/kivy/__init__.pyc"
10:44:35,226 kivy INFO [Python ] v3.7.8 (default, Jul 4 2020, 10:17:17)
[Clang 11.0.3 (clang-1103.0.32.62)]
10:44:35,226 kivy INFO [Python ] Interpreter at "/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/buskill"
10:44:35,260 kivy INFO [Factory ] 184 symbols loaded
10:44:35,345 kivy INFO [Image ] Providers: img_tex, img_imageio, img_dds, img_sdl2, img_gif (img_pil, img_ffpyplayer ignored)
10:44:35,560 kivy INFO [Window ] Provider: sdl2
10:44:35,761 kivy INFO [GL ] Using the "OpenGL ES 2" graphics system
10:44:35,761 kivy INFO [GL ] Backend used <sdl2>
10:44:35,762 kivy INFO [GL ] OpenGL version <b'2.1 INTEL-14.7.8'>
10:44:35,762 kivy INFO [GL ] OpenGL vendor <b'Intel Inc.'>
10:44:35,762 kivy INFO [GL ] OpenGL renderer <b'Intel HD Graphics 4000 OpenGL Engine'>
10:44:35,762 kivy INFO [GL ] OpenGL parsed version: 2, 1
10:44:35,762 kivy INFO [GL ] Shading version <b'1.20'>
10:44:35,762 kivy INFO [GL ] Texture max size <16384>
10:44:35,763 kivy INFO [GL ] Texture max units <16>
10:44:35,781 kivy INFO [Window ] auto add sdl2 input provider
10:44:35,806 kivy INFO [Window ] virtual keyboard not allowed, single mode, not docked
10:44:35,828 kivy INFO [Text ] Provider: sdl2
10:44:35,834 kivy INFO [INFO ] using DATA_DIR:|/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/.buskill|
10:44:35,889 kivy INFO [GL ] NPOT texture support is available
10:44:35,905 kivy INFO [Base ] Start application main loop
10:46:11,686 kivy INFO [Base ] Leaving application in progress...
10:46:11,687 kivy WARNING stderr: Traceback (most recent call last):
10:46:11,687 kivy WARNING stderr: File "main.py", line 97, in <module>
10:46:11,687 kivy WARNING stderr: BusKillApp().run()
10:46:11,687 kivy WARNING stderr: File "kivy/app.py", line 855, in run
10:46:11,688 kivy WARNING stderr: File "kivy/base.py", line 504, in runTouchApp
10:46:11,688 kivy WARNING stderr: File "kivy/core/window/window_sdl2.py", line 747, in mainloop
10:46:11,688 kivy WARNING stderr: File "kivy/core/window/window_sdl2.py", line 479, in _mainloop
10:46:11,688 kivy WARNING stderr: File "kivy/base.py", line 342, in idle
10:46:11,688 kivy WARNING stderr: File "kivy/base.py", line 327, in dispatch_input
10:46:11,689 kivy WARNING stderr: File "kivy/base.py", line 293, in post_dispatch_input
10:46:11,689 kivy WARNING stderr: File "kivy/_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
10:46:11,689 kivy WARNING stderr: File "kivy/uix/behaviors/button.py", line 179, in on_touch_up
10:46:11,689 kivy WARNING stderr: File "kivy/_event.pyx", line 703, in kivy._event.EventDispatcher.dispatch
10:46:11,689 kivy WARNING stderr: File "kivy/_event.pyx", line 1214, in kivy._event.EventObservers.dispatch
10:46:11,689 kivy WARNING stderr: File "kivy/_event.pyx", line 1098, in kivy._event.EventObservers._dispatch
10:46:11,690 kivy WARNING stderr: File "kivy/lang/builder.py", line 64, in custom_callback
10:46:11,690 kivy WARNING stderr: File "/Users/maltfield/Downloads/buskill-1599550992.app/Contents/MacOS/buskill.kv", line 190, in <module>
10:46:11,690 kivy WARNING stderr: on_release: root.continue_function()
10:46:11,690 kivy WARNING stderr: File "buskill_gui.py", line 154, in update2
10:46:11,691 kivy WARNING stderr: self.bk.upgrade_bg()
10:46:11,691 kivy WARNING stderr: File "packages/buskill/__init__.py", line 639, in upgrade_bg
10:46:11,691 kivy WARNING stderr: File "packages/buskill/__init__.py", line 639, in upgrade_bg
10:46:11,692 kivy WARNING stderr: File "bdb.py", line 88, in trace_dispatch
10:46:11,692 kivy WARNING stderr: File "bdb.py", line 113, in dispatch_line
10:46:11,692 kivy WARNING stderr: bdb.BdbQuit
10:46:11,692 kivy WARNING stderr: [INFO/MainProcess] process shutting down
10:46:11,692 kivy WARNING stderr: [DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
10:46:11,693 kivy WARNING stderr: [DEBUG/MainProcess] running the remaining "atexit" finalizers
|
update: newer versions of PyInstaller appear to have fixed reproducibility or MacOS TODO: see if it works for our buskill MacOS builds, too. |
|
TOFU 1/3 via VPN in Netherlands |
|
Hmm..I ran this on my actual buskill dev VM, and I got a diff. It's actually a totally distinct file. In the TOFU above, I get a file named: But on my buskill VM, I get: Not that both are running Debian 10 with the same kernel version, but the former is using pip v18.1 and the latter pip v20.1.1. EDIT: Ah, it looks like I pinned pip to v20.1.1 in 2020-07 as it was a dependency for kivy: So in order for this 3TOFU to work, I should first make sure I'm using pip v20.1.1. I'll update the TOFU commands to be: |
|
Re-doing TOFU (via Netherlands VPN) 1/3 w/ pip v20.1.1 |
maltfield
added a commit
that referenced
this issue
Dec 1, 2021
The current version v3.6 has some bugs preventing reproducable builds in MacOS. * #3 * pyinstaller/pyinstaller#4972 The changelog says these were fixed in PyInstaller v4.2. I'm going ahead and upgrading past that to the latest at the time of writing, which is v4.7.
|
Maybe I have to go back to the tarball somehow because the I ran this from our MacMini, and confirmed it got the tarball The hashes of the files here match the first TOFU above -- except there's an additional file for |
|
It's going to be a PIA to do 3TOFU from my one mac mini server, so I hacked-up a better set of commands. Note the grepping hackery of the PyPI simple API because PIP has no way to say |
|
So here's the third attempt of TOFU 1/3 (also VPN via Netherlands) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This task is for me to confirm that all three of our build scripts produce deterministic artifacts.
Files produced by our build script run via GitHub's CI workflows should have identical checksums to files produced by our build script run on our developer's local machines.
See also:
The text was updated successfully, but these errors were encountered: