pip-21.0.1 is breaking reproducible wheels

[kushaldas]

pip version

21.0.1

Python version

3.7.3 and 3.9.1

OS

Debian Buster, Fedora 33

Additional information

No response

Description

We were using --build flag to build reproducible wheels. Even if we mention different TMP* options as mentioned in #8333 (comment) , pip will create a temporary directory inside of the temp path, and that will be used to build. This is causing the reproducible build failures.

Here is the script https://github.com/freedomofpress/securedrop-debian-packaging/blob/main/scripts/build-sync-wheels#L99-L100 we are using for the reproducible builds.

Expected behavior

pip should create reproducible wheels.

How to Reproduce

How to test?

• checkout https://github.com/freedomofpress/securedrop-debian-packaging or copy the shell script locally
• Build any platform specific wheel by pointing the script to a directory with a requirements file which contains hashes. Please add any environment variable to a fixed temporary directory, and then: ./scripts/build-sync-wheels -p /home/user/code/testproject I will also attach the requirements.txt file I used in the issue.
• This is the raw pip command used pip3 wheel --no-binary :all: --find-links /tmp/tmppqis0hcw --progress-bar pretty --wheel-dir /tmp/tmppqis0hcw --build /tmp/pip-wheel-build --requirement /home/kdas/code/ladkhorecrypto/requirements.txt feel free to remove the --build if required.
• copy the wheel to a temporary directory, in my example I copied to ./part1.
• build the wheels again
• verify the sha256sums or run Debian's diffoscope tool

Output

✦ ❯ sha256sum localwheels/cryptography-3.4.4-cp39-cp39-linux_x86_64.whl part1/cryptography-3.4.4-cp39-cp39-linux_x86_64.whl 
b882d10b1c36118b18a49fa6c95cdff7d19184ba9cce077bd1bcf5d4e5d779d1  localwheels/cryptography-3.4.4-cp39-cp39-linux_x86_64.whl
6413ad572666600657a7630c3e3d1cf24bf9ad6f73590d518c257ce47520029c  part1/cryptography-3.4.4-cp39-cp39-linux_x86_64.whl

Code of Conduct

• I agree to follow the PSF Code of Conduct

[kushaldas]

requirements.txt
I used this file for test. I can build reproducible wheels using older versions of pip.

[pfmoore]

pip should create reproducible wheels.

I'm not sure this should be something pip guarantees. Pip is an installer, not a dedicated wheel builder (even though we do have features for creating wheels). Maybe this would be better addressed by https://github.com/pypa/build?

[kushaldas]

pip should create reproducible wheels.

I'm not sure this should be something pip guarantees. Pip is an installer, not a dedicated wheel builder (even though we do have features for creating wheels). Maybe this would be better addressed by https://github.com/pypa/build?

I filed the issue as we used pip to both build and install the dependencies in a proper reproducible way till now. I guess we can have some defined points for future which pip will guaranty and also hopefully pypa can come up with some solutions for reproducible wheels which we all can then use. If it is a different tool than pip, that is also very much okay.

[uranusjr]

At some point in the future pip will probably switch to pypa/build for building wheels. But we need more people to help make it mature.

[abitrolly]

@kushaldas can you paste the output of diffoscope to see where non-determinism creeps in? Recording the location of a build dir into the wheel sounds unlikely to be a feature to me, so there must be something else.

[kushaldas]

@kushaldas can you paste the output of diffoscope to see where non-determinism creeps in? Recording the location of a build dir into the wheel sounds unlikely to be a feature to me, so there must be something else.

https://kushaldas.in/volatile/diffoscope_output.txt 9MB file.

[abitrolly]

@kushaldas for the start I see the problem in compiled .so files. The proper way is to pass -fdebug-prefix-map to C compiler to make absolute build path relative as described here https://interrupt.memfault.com/blog/reproducible-firmware-builds#fdebug-prefix-map I haven't researched how to do that with pip though.

[kushaldas]

@kushaldas for the start I see the problem in compiled .so files. The proper way is to pass -fdebug-prefix-map to C compiler to make absolute build path relative as described here https://interrupt.memfault.com/blog/reproducible-firmware-builds#fdebug-prefix-map I haven't researched how to do that with pip though.

Okay, still now providing a fixed build location in pip fixed that for us, because of no easy way to pass things to compiler during pip building the wheels.

[kushaldas]

Oh, there is also the rust part of the code.

[abitrolly]

I guess somebody needs to report to https://reproduciblewheels.com/ that --build no longer works. Maybe switching to flit can solve the problem. At least its docs mention that feature - https://flit.readthedocs.io/en/latest/reproducible.html

[kushaldas]

I guess somebody needs to report to https://reproduciblewheels.com/ that --build no longer works. Maybe switching to flit can solve the problem. At least its docs mention that feature - https://flit.readthedocs.io/en/latest/reproducible.html

I will give it a try, though I wonder how that can be done for every available dependency. We can easily do this for our code, but the problem is for dependencies.

[kushaldas]

I guess somebody needs to report to https://reproduciblewheels.com/ that --build no longer works. Maybe switching to flit can solve the problem. At least its docs mention that feature - https://flit.readthedocs.io/en/latest/reproducible.html

@redshiftzero ^^^