Can not install any poetry based project with pip from source distribution #760
Comments
|
Version 19 of pip coming January 2019 will support this. See #321. |
|
Will this also solve the problem of cyclic build dependencies? Like it is the case with poetry <> tomlkit. |
|
@cs01 PEP 517 was already implemented for pip and also merged (pypa/pip#5743) and I gave current pip from master a try and I get the same error. The difference between #321 and using As much as I dislike vendoring I think that this is the only solution. |
|
I am the maintainer of backoff and I am trying figure out if I need to do a workaround on my side or if this is something that I can expect to be fixed upstream. I believe the issue is that there is a circular dependency: poetry and tomlkit depend on each other at build-time. It is not clear to me if this is a pip bug, or if circular dependencies just aren't supported when building from source. I think @garbas is correct that vendoring (probably tomlkit in poetry?) would work around the problem. Any thoughts on how to address this or if I should be looking for a workaround downstream? |
|
I tried using the latest pip on a poetry-based project with no circular dependencies and it did not work. It looks like a bug in pip (or could at least be a nicer error message) but I suppose it's possible poetry's backend doesn't work. I filed an issue with pip pypa/pip#6100. |
|
@cs01 i think that issue you reference to is not really related to this one, except that both show that there are issues with pip and poetry. The problem is that there is a circular dependency in poetry dependencies (poetry <-> tomlkit) which is usually not a problem since pip (or any other python package management tool) can handle this. This becomes a problem when you are required to use source distributions (no wheels for me aka So this is what happens when you run
|
|
@garbas I agree that I don't have circular dependencies. For some reason I assumed we were both hitting the same |
|
Any news on this? I can put time into solution once we decide what a solution is. The error I get is: $ ./env/bin/pip install backoff --no-binary :all:
Collecting backoff
Using cached https://files.pythonhosted.org/packages/37/0b/e6d0e14cf95039fbc3b0b0e8370e1cec7bd6294d05dcdb669d0edbfde102/backoff-1.8.0.tar.gz
Installing build dependencies ... error
Complete output from command /tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wowed2bl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
Collecting poetry>=0.12
Downloading https://files.pythonhosted.org/packages/75/44/824cee51be9ee1f2c28042a58591a36cef0c85f688f7ddc5b088af26d50a/poetry-0.12.11.tar.gz (150kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting cleo<0.7.0,>=0.6.7 (from poetry>=0.12)
Using cached https://files.pythonhosted.org/packages/cb/9f/8dd57785485b2a2746edbe598a31b0852740a1ff91a2739c104b628c8520/cleo-0.6.8.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting shellingham<2.0,>=1.1 (from poetry>=0.12)
Using cached https://files.pythonhosted.org/packages/04/47/c3f0b5fce44827d011f7f4b8831dd20984a228d7a7f46d4d45f1aaa6fd90/shellingham-1.2.8.tar.gz
Collecting pkginfo<2.0,>=1.4 (from poetry>=0.12)
Downloading https://files.pythonhosted.org/packages/6c/04/fd6683d24581894be8b25bc8c68ac7a0a73bf0c4d74b888ac5fe9a28e77f/pkginfo-1.5.0.1.tar.gz
Collecting cachy<0.3,>=0.2 (from poetry>=0.12)
Using cached https://files.pythonhosted.org/packages/68/ef/eabaf6c36bb87e4c1a74790b19e44064df25e09a55f191eb71ccd2b512ad/cachy-0.2.0.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting jsonschema<4.0,>=3.0a3 (from poetry>=0.12)
Downloading https://files.pythonhosted.org/packages/b6/1f/b9941309e4b2b65b2ef34a8b36d1cbc53d2de6bd168d946042b5dc208ff6/jsonschema-3.0.0a5.tar.gz (119kB)
Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12)
Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-uppwnvo8/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3:
Collecting poetry>=0.12a3
Downloading https://files.pythonhosted.org/packages/93/8a/270aa699a07405df9b260f9192eec043cf486b4b49fafd7ebcee09cb2cc5/poetry-1.0.0a2.tar.gz (154kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Collecting tomlkit<0.6.0,>=0.5.1 (from poetry>=0.12a3)
Using cached https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz
Exception:
Traceback (most recent call last):
File "/tmp/pip/src/pip/_internal/cli/base_command.py", line 154, in main
status = self.run(options, args)
File "/tmp/pip/src/pip/_internal/commands/install.py", line 315, in run
resolver.resolve(requirement_set)
File "/tmp/pip/src/pip/_internal/resolve.py", line 131, in resolve
self._resolve_one(requirement_set, req)
File "/tmp/pip/src/pip/_internal/resolve.py", line 294, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/tmp/pip/src/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
self.require_hashes
File "/tmp/pip/src/pip/_internal/operations/prepare.py", line 355, in prepare_linked_requirement
with self.req_tracker.track(req):
File "/nix/store/9vcy8i7mcwmcp486krc22v7851spcdqj-python3-3.7.1/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/tmp/pip/src/pip/_internal/req/req_tracker.py", line 86, in track
self.add(req)
File "/tmp/pip/src/pip/_internal/req/req_tracker.py", line 55, in add
% (link, fp.read()))
LookupError: https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from https://pypi.org/simple/tomlkit/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*) is already being built: tomlkit<0.6.0,>=0.5.1 from https://files.pythonhosted.org/packages/f7/f7/bbd9213bfe76cb7821c897f9ed74877fd74993b4ca2fe9513eb5a31030f9/tomlkit-0.5.3.tar.gz#sha256=d6506342615d051bc961f70bfcfa3d29b6616cc08a3ddfd4bc24196f16fd4ec2 (from poetry>=0.12)
----------------------------------------
Command "/tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-uppwnvo8/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12a3" failed with error code 2 in None
----------------------------------------
Command "/tmp/pip/env/bin/python3.7m /tmp/pip/src/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-wowed2bl/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in None |
|
The same happens when trying to build Dockerfile: FROM python:3.7.2-alpine
RUN pip3 install -U pip
RUN pip3 install --no-binary :all: pytzdataOutput: |
|
Also adding |
|
Any progress ? |
|
It's even simpler than any of the other commands given. |
|
It seems like, in lieu of a fix from pip, poetry needs to do something to solve this issue. The only real solution to me seems like vendoring any of poetry's dependencies which, in turn, use poetry for their builds. This likely also applies to any transitive dependencies of poetry too, which makes the problem not only more complicated but prone to changing as more packages switch to poetry. I have found a workaround which works for my use-case, at least for now. I'm building wheels for everything in the poetry.lock during a build. If I pass each of the packages to Here's an example of what the (insanely long) command looks like in my case (built by a script): |
|
I have also been commenting on the pip issue pypa/pip#5739 and thinking more on this and it seems to me like a mis-feature of pip causing this problem (although I could be convinced otherwise if there is documentation saying otherwise) that is requiring source-based installs of build-time dependencies, especially when those dependencies are already installed and available. In the case of the command I gave above I'm not only running the pip command via poetry, in a poetry created virtualenv, but poetry is itself already installed in that virtualenv, so poetry should be already-available to pip when it comes to building packages that require poetry at build-time. |
|
Like @reversefold I posted on the other thread as well, after just having this issue brought to my attention when I switched two of my projects over to poetry. I think that trying to selectively vendor in dependancies would be hard to manage and more error prone than simply vendoring in all the dependancies. While I say that, I have no idea how difficult vendored dependancy management is. Or what the feeling of the maintainers are on doing that. I'm curious to hear thoughts from someone on the team. |
|
This is still not resolved on pip side as of 20.0.1. |
|
Any plan to fix this? |
|
i believe this is what's preventing me from installing apache-airwatch using puppet-python |
|
Any update here? |
|
The upcoming release should provide a solution for this. With the next release diff --git a/pyproject.toml b/pyproject.toml
index 3469bf2..a42652d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,5 +38,5 @@ pytest-cov = "^2.6"
pytest-asyncio = {version = "^0.10.0",python = "^3.5"}
[build-system]
-requires = ["poetry>=0.12"]
-build-backend = "poetry.masonry.api"
+requires = ["poetry-core>=1.0.0a5"]
+build-backend = "poetry.core.masonry.api"Note that this uses an alpha release of |
|
That's awesome, anything I can do to help this move forward? |
|
@abn This is fantastic. Are there any news on when poetry-core will be available, or if it can already be used somehow? This worked on my Windows (git bash) machine but crashed on my Linux RHEL 7 machine with Python 3.7: |
|
I found out why it was crashing: #2626 |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (
-vvvoption).OS version and name: NixOS 18.09
Poetry version: --
Link of a Gist with the contents of your pyproject.toml file: --
Issue
Hi,
this is not really a bug about how poetry works, but how poetry interacts with other python packaging ecosystem.
While working on a project of mine which uses pip inside (for now) I noticed that we can not install a package called
backoffwhich in its recent version switched to poetry.To reproduce the error you simply need to invoke the following command:
$ pip install backoff --no-binary :all:Above command tries to install packages from source distribution and not from wheel (which is a requirement of my project).
Digging deeper I noticed that problem is that poetry depends on tomlkit which depends on poetry as a build tool.
I would bet that this is a bug of pip and looking at their issue tracker I found the issue being reported there: pypa/pip#5739. If you look closed you will see that also
flitwas affected but they solved it by moving their dependencies around a bit.I'm not sure what a possible solution might be but maybe vendoring of the dependencies might be a solution (eg. like pip does it). Or maybe there is a better way...
This kinda puts poetry (and actually all projects using it) as a blocker for me and prevents me from investigating further into poetry, which I really hope becomes more used in python ecosystem. This is also a reason why I'm posting this issue here, to try to bring attention to this kinds of bugs and to make poetry adoption easier.
In any case, thank you for you work.
The text was updated successfully, but these errors were encountered: