Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vendor setuptools / wheel. #607

Closed
jsirois opened this issue Oct 23, 2018 · 5 comments · Fixed by #624
Closed

Vendor setuptools / wheel. #607

jsirois opened this issue Oct 23, 2018 · 5 comments · Fixed by #624
Assignees
@jsirois
Labels
enhancement

Comments

@jsirois
Copy link
Member

jsirois commented Oct 23, 2018

We rely on wheel and setuptools (pkg_resources) APIs heavily. Supporting changes in these APIs leads to complex and brittle code - we should vendor what we need and simplify associated code and interpreter setup in the pex build phase.
@jsirois jsirois mentioned this issue Oct 23, 2018
Closed
@jsirois
Copy link
Member Author

jsirois commented Oct 23, 2018

There was some discussion of this on slack (link will die eventually: https://pantsbuild.slack.com/archives/C087V4P1T/p1539287221000100). @kwlzn and I concurred vendoring would make sense with the only real downside being a need to pick up new pex releases when new features in those support libs were needed. This downside being one we mainly suffer right now though anyhow with an upper-bound on the versions we're willing to resolve that has traditionally been capped under the next major version.

@jsirois
Copy link
Member Author

jsirois commented Oct 23, 2018

This will allow removing demote_bootstrap and its uses since we'll be importing our vendored deps under a vendored top-level package name:
https://github.com/pantsbuild/pex/blob/eac023361fb5e524f2f31916dd186621cec46d6d/pex/pex.py#L399-L431

@jsirois jsirois self-assigned this Oct 23, 2018
@jsirois
Copy link
Member Author

jsirois commented Oct 23, 2018 •
edited
Loading

Noting that the total distribution size will increase ~6x from ~115KB to ~706KB.

Vendored code ~591KB:

$ pip install -t /tmp/vendor --no-compile --no-binary :all: setuptools==40.4.3 wheel==0.31.1
$ rm -rf /tmp/vendor/{bin,easy_install.py,*.egg-info,*.dist-info}
$ zip -r vendored.zip /tmp/vendor
$ ls -l vendored.zip 
-rw-r--r-- 1 jsirois jsirois 605166 Oct 23 14:43 vendored.zip

vs current code ~115KB:

$ ls -l dist/pex-1.5.1-py2.py3-none-any.whl 
-rw-r--r-- 1 jsirois jsirois 117115 Oct 23 14:06 dist/pex-1.5.1-py2.py3-none-any.whl

@jsirois
Copy link
Member Author

jsirois commented Oct 23, 2018

I believe the full extent of the third party API surface area is constrained to pkg_resources.Distribution which is shared between pex.resolver as a producer and pex.pex_builder as a consumer. PEX uses this api via duck-typing but a concern could be API clients of resolve that type-test.

jsirois added a commit to jsirois/pex that referenced this issue Oct 26, 2018
@jsirois
Vendor setuptools and wheel.
f829ac8 
Fixes pex-tool#607
@jsirois jsirois mentioned this issue Oct 26, 2018
Closed
jsirois added a commit to jsirois/pex that referenced this issue Oct 26, 2018
@jsirois
Vendor setuptools and wheel.
5ef7084 
Fixes pex-tool#607
jsirois added a commit to jsirois/pex that referenced this issue Oct 28, 2018
@jsirois
Vendor setuptools and wheel.
888105a 
Fixes pex-tool#607
jsirois added a commit to jsirois/pex that referenced this issue Oct 28, 2018
@jsirois
Vendor setuptools and wheel.
b84c8db 
Fixes pex-tool#607
@jsirois
Copy link
Member Author

jsirois commented Oct 28, 2018

In the end, the vendoring did not use renaming since it presented challenges executing requirement setup.py when building pexes. As such, demote_bootstrap remains, but much code was still removed and un-special-cased with knowledge of a fixed setuptools and wheel.

jsirois added a commit to jsirois/pex that referenced this issue Oct 28, 2018
@jsirois
Vendor setuptools and wheel.
ff24049 
Fixes pex-tool#607
jsirois added a commit to jsirois/pex that referenced this issue Oct 28, 2018
@jsirois
Vendor setuptools and wheel.
187a224 
Fixes pex-tool#607
jsirois added a commit to jsirois/pex that referenced this issue Oct 28, 2018
@jsirois
Vendor setuptools and wheel.
bec879d 
Fixes pex-tool#607
jsirois added a commit to jsirois/pex that referenced this issue Oct 28, 2018
@jsirois
Vendor setuptools and wheel.
681a668 
Fixes pex-tool#607
@jsirois jsirois mentioned this issue Nov 3, 2018
Closed
@jsirois jsirois mentioned this issue Nov 13, 2018
Merged
jsirois added a commit to jsirois/pex that referenced this issue Nov 17, 2018
@jsirois
Vendor setuptools and wheel.
31e1362 
This change removes pex dependence on `setuptools` and `wheel`
resolution at build time in favor of vendoring pinned versions.
Switching to a vendoring model allows us to eliminate code that dealt
with the vagaries of supporting a ~5 year span of setuptools organic
growth and API drift and it also allows us to fully isolate pex
build-time interpreters. Going forward more libraries could be vendored
easily at the cost of pex distribution size.

Three changes make this possible:
1. The `pex.vendor` module is introduced to centralize vendored
   distribution management including establishing vendored distribution
   requirements and setting up interpreters with vendored distribution
   extras.
2. A `pex.vendor.__main__` is introduced and run via `tox -e vendor` to
   actually perform vendoring of third-party distributions inside pex.
   This includes installing the distributions in a the
   `pex/vendor/_vendored` subtree and re-writing their imports
   (shading).
3. The `pex.third_party` module is introduced to provide runtime import
   support for the vendored code using a
   [PEP-302](https://www.python.org/dev/peps/pep-0302/) `sys.meta_path`
   importer.

Important consequences of this change include:
1. The pex runtime that used to be vendored under `.bootstrap/_pex` is
   now vendored under `.bootstrap/pex` in order to support a fixed
   `pex.third_party` vendor shading import prefix. The PEX runtime
   bootstrap demotion code is made more aggressive, correct and complete
   as a result to support and old pex vendored runtime handing off to a
   newer pex distribution to support pexed pex. In addtion, the
   `pex.third_party` vendoring importer is re-used to expose `pex` as
   `_pex` for users taking advantage of `_pex.util.DistributionHelper`
   and its `access_zipped_assets` for a deprecation period up to pex
   2.0.0.
2. Since we no longer re-name `pex` to `_pex` in the runtime, all
   imports are now absolute.
3. The pex build-time interpreter cache is eliminated. Since we vendor
   the extras needed by interpreters, setup is now cheap, fast and
   hermetic and needs no cache. As a result the
   `--interpreter-cache-dir` flag now no-ops and is deprecated for
   removal in pex 2.0.0.

Additional changes of note include:
1. The `bdist_pex` distutils command now dogfoods the pex cli for better
   fidelity with standard pex tool usage.
2. The pex `setup.py` is now self-hosting and can be run by a python
   interpreter lacking `setuptools` and/or `wheel`; in fact, pex can be
   pexed via `python -sSE setup.py bdist_pex`.

Replaces pex-tool#612
Fixes pex-tool#607
@jsirois jsirois mentioned this issue Nov 17, 2018
Merged
jsirois added a commit to jsirois/pex that referenced this issue Nov 17, 2018
@jsirois
Vendor setuptools and wheel.
3e85c66 
This change removes pex dependence on `setuptools` and `wheel`
resolution at build time in favor of vendoring pinned versions.
Switching to a vendoring model allows us to eliminate code that dealt
with the vagaries of supporting a ~5 year span of setuptools organic
growth and API drift and it also allows us to fully isolate pex
build-time interpreters. Going forward more libraries could be vendored
easily at the cost of pex distribution size.

Three changes make this possible:
1. The `pex.vendor` module is introduced to centralize vendored
   distribution management including establishing vendored distribution
   requirements and setting up interpreters with vendored distribution
   extras.
2. A `pex.vendor.__main__` is introduced and run via `tox -e vendor` to
   actually perform vendoring of third-party distributions inside pex.
   This includes installing the distributions in a the
   `pex/vendor/_vendored` subtree and re-writing their imports
   (shading).
3. The `pex.third_party` module is introduced to provide runtime import
   support for the vendored code using a
   [PEP-302](https://www.python.org/dev/peps/pep-0302/) `sys.meta_path`
   importer.

Important consequences of this change include:
1. The pex runtime that used to be vendored under `.bootstrap/_pex` is
   now vendored under `.bootstrap/pex` in order to support a fixed
   `pex.third_party` vendor shading import prefix. The PEX runtime
   bootstrap demotion code is made more aggressive, correct and complete
   as a result to support and old pex vendored runtime handing off to a
   newer pex distribution to support pexed pex. In addtion, the
   `pex.third_party` vendoring importer is re-used to expose `pex` as
   `_pex` for users taking advantage of `_pex.util.DistributionHelper`
   and its `access_zipped_assets` for a deprecation period up to pex
   2.0.0.
2. Since we no longer re-name `pex` to `_pex` in the runtime, all
   imports are now absolute.
3. The pex build-time interpreter cache is eliminated. Since we vendor
   the extras needed by interpreters, setup is now cheap, fast and
   hermetic and needs no cache. As a result the
   `--interpreter-cache-dir` flag now no-ops and is deprecated for
   removal in pex 2.0.0.

Additional changes of note include:
1. The `bdist_pex` distutils command now dogfoods the pex cli for better
   fidelity with standard pex tool usage.
2. The pex `setup.py` is now self-hosting and can be run by a python
   interpreter lacking `setuptools` and/or `wheel`; in fact, pex can be
   pexed via `python -sSE setup.py bdist_pex`.

Replaces pex-tool#612
Fixes pex-tool#607
jsirois added a commit that referenced this issue Nov 26, 2018
@jsirois
Vendor setuptools and wheel. (#624)
8fd5e1b 
This change removes pex dependence on `setuptools` and `wheel`
resolution at build time in favor of vendoring pinned versions.
Switching to a vendoring model allows us to eliminate code that dealt
with the vagaries of supporting a ~5 year span of setuptools organic
growth and API drift and it also allows us to fully isolate pex
build-time interpreters. Going forward more libraries could be vendored
easily at the cost of pex distribution size.

Three changes make this possible:
1. The `pex.vendor` module is introduced to centralize vendored
   distribution management including establishing vendored distribution
   requirements and setting up interpreters with vendored distribution
   extras.
2. A `pex.vendor.__main__` is introduced and run via `tox -e vendor` to
   actually perform vendoring of third-party distributions inside pex.
   This includes installing the distributions in a the
   `pex/vendor/_vendored` subtree and re-writing their imports
   (shading).
3. The `pex.third_party` module is introduced to provide runtime import
   support for the vendored code using a
   [PEP-302](https://www.python.org/dev/peps/pep-0302/) `sys.meta_path`
   importer.

Important consequences of this change include:
1. The pex runtime that used to be vendored under `.bootstrap/_pex` is
   now vendored under `.bootstrap/pex` in order to support a fixed
   `pex.third_party` vendor shading import prefix. The PEX runtime
   bootstrap demotion code is made more aggressive, correct and complete
   as a result to support and old pex vendored runtime handing off to a
   newer pex distribution to support pexed pex. In addtion, the
   `pex.third_party` vendoring importer is re-used to expose `pex` as
   `_pex` for users taking advantage of `_pex.util.DistributionHelper`
   and its `access_zipped_assets` for a deprecation period up to pex
   2.0.0.
2. Since we no longer re-name `pex` to `_pex` in the runtime, all
   imports are now absolute.
3. The pex build-time interpreter cache is eliminated. Since we vendor
   the extras needed by interpreters, setup is now cheap, fast and
   hermetic and needs no cache. As a result the
   `--interpreter-cache-dir` flag now no-ops and is deprecated for
   removal in pex 2.0.0.

Additional changes of note include:
1. The `bdist_pex` distutils command now dogfoods the pex cli for better
   fidelity with standard pex tool usage.
2. The pex `setup.py` is now self-hosting and can be run by a python
   interpreter lacking `setuptools` and/or `wheel`; in fact, pex can be
   pexed via `python -sSE setup.py bdist_pex`.

Replaces #612
Fixes #607
@jsirois jsirois mentioned this issue Sep 9, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsirois jsirois

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Vendor setuptools and wheel. jsirois/pex
1 participant
@jsirois