-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove Dependency Links - Needs Discussion #1519
Conversation
One note, when removing this a grand total of 2 tests failed. So this was a very poorly tested feature :( |
correct me if I'm wrong, but I don't think the case brought up on distutils the other day can conveniently be handled with requirements files his was a private scenario, where all of his projects were only in vcs (and not published as packages on local indexes) with vcs dependency links in all the projects, he could install his top-level app (with dependencies) with a pip call that only had one argument (the top-level project vcs url) the alternative with requirements files involves maintaining pinned vcs urls recursively for all dependencies, and there's no easy way that I could tell to manage that file. e.g. supposing he knows that he wants to upgrade one child dependency, sure he can change the vcs url for that one entry, but how to easily do it recursively? |
I don't understand the question. If you're pinning your versions exactly (either via == or an @hash) then an upgrade will install the same version recursively or not dependency links or not. If you're not pinning and you're just pointing directly at the master branch or what have you then pip will already do the right thing when you reinstall the requirements.txt file. |
suppose:
(pinned versioning isn't critical to the example) first, let's make this concrete with vcs dependency links: first case: A's dep links:
B's dep links:
second case: A's dep links:
B's dep links:
you install A==1 with: pip install git://a@1#egg=A now, let's make it concrete with a requirements file for A first case:
2nd case:
you install A==1 with: pip install -r requirements.txt (from A==1 src) The concern here when using requirements files was that the author of A has to manage changing the C url. In the dep's link scenario, that was managed by the developer of B. |
Hmm, well OK so that particular case can become somewhat more complicated than without dependency links. However AFAIK this is no different than other languages besides Go which has a horrible packaging story. Not that other languages not having a feature is a reason to get rid of it. So yea, this particular case would be somewhat more work out of the box. However it wouldn't be all that difficult for someone to create a tool outside of pip to automate updating the requirements.txt. The original reporter of that thread seemed to have been OK with an external tool serving this purpose for him. Dependency links has a pretty narrow scope of where it's really reasonable to use them at all, and seeing as how an external tool combined with the requirements file format can completely replicate them in what is more likely to be a saner idea in general I don't see a whole lot of point to leaving this attractive nuisance around. Especially when the "worst" case is that some folks have to manually update a requirements.txt file. And really if you have multiple independent packages owned by several different people you should be using a package server.. |
but, as for dropping in this 1.6, I'm noticing the 1.5 changelog doesn't |
Hmm, you're right the changelog doesn't mention a deprecation, however there was an actual warning raised when you attempted to use the |
I'm using dependency_links to pull a package from GitHub since the version on PyPI breaks with Python 3 while the one on GitHub is more recent and works. What's my alternative? I know the best option would be uploading the new version to PyPI, but that probably won't happen, or at least not soon, considering that the last time the package author made a change to the project was 6 months ago. P.S. I'm re-posting this from pypa-dev group since it seems more active here. |
So you have project A that depends on project B from GitHub. Are you distributing project A on PyPI or is it internal only? |
Not yet but will so on PyPI. |
So if you're uploading A to PyPI then the answer is to rename project B and upload it to PyPI under a different name. Using dependency links in the case where you're distributing things on PyPI is a use case we've actively tried to stop as a random project on PyPI should not be able to force you to get your dependency's from a third party site. |
Thanks. That seems the right way. |
It seems [to me] the main use case people have developed for this is for setting up a tree of package dependencies based on VCS urls. If your situation is complex enough, I think it's warranted to recommend moving to a local / lan pypi caching server, like devpi or mypypi. Sometimes this is made more difficult when one needs to set up verified HTTPS for such a server just so pip doesn't complain. For only local caching, I can appreciate working off VCS urls can make for quite a simple workflow for small-medium teams for their own use. It is often even quite secure when most VCS services run over HTTPS. I think a caching server would be a lot easier to recommend, if pypi could be specifically configured to allow HTTP for some domains / IPs. This means you could allow a development pypi server so that it doesn't need a CA-bought or self-signed-and-installed certificate just to work nicely. |
Pip doesn't prevent you from using HTTP at all. It just warns you that you're using HTTP. |
To be clear, |
I had the impression that people were concerned about 'warning fatigue' from this sort of situation, though? The above is a proposal to alleviate such, but without going overboard with some sort of a |
Ah right. Warning fatigue is a thing for sure, and we actually have an issue here somewhere for allowing you to acknowledge that you're using an insecure transport. |
ok, I see the "will be removed in pip 1.6" ouput when using the flag. |
TBH, I don't use dependency links, and I don't really understand the scenarios people are discussing, so I don't have an opinion on the technical issues. However, like @qwcode I am concerned that we've been forcing the pace a lot recently, and we may want to be cautious about adding too many breakages in one go. Ultimately I'll go with whatever you decide, though. |
@pfmoore did you read my first comment at all? Basically, imagine you are developing a number of packages as part of your team, that have inter depedencies on eachother; you keep the source for these in git repositories that you control and can retrieve from. The depedency links feature lets you really easily link these dependencies up simply with your normal git checkout URLs, no other fiddling about necessary. You can even link them to specific commits in some cases, making it quite easy to change exact versions found. That's the simple workflow I think people are bemoaning being removed. |
@Ivoz yes I did, and I understand what you're saying. I just don't have any experience of that workflow, nor of how dependency-links works, nor of what other workflows might be potential candidates as workarounds (if any). So I'm deferring the decision to people who do understand, that's all. |
If I may chime in. I have been using dependency_links a lot for my job in cases where we develop and/or deploy from our local code repositories. Much of this code is not on pypi, github, bitbucket or any other publicly accessible resource. While some of our packages are on an internal pypi server, the majority either are not or particular versions/tags/branches are not. I'm only mentioning this because many of the comments seem to come from the point of view that most things that they install with pip come from public sources, whereas most things we install are in-house and not public pypi packages. If dependency_links is no longer recommended then does anyone have an alternative? |
I'd suggest automating uploading tagged (or another filter) versions to your internal pypi server. Should be getting easier and easier with the preponderance of CI servers out there these days. |
That, or use a reqirements.txt file to point at the git repositories. |
The requirements.txt would be particularly sticky because we have dozens of little libraries with all kinds of inter-dependencies (think library A depends on library B which depends on library C) plus the fact that some of them use optional "extras_require" (is that going away too?). So to have one or even a few requirements.txt's that satisfies all use cases would probably take a lot of time to do and be error-prone. Getting the systems guys to package each version of each library that's in use and put it on the pypi server would also be challenging for us, which is probably the main reason we went the dependency_links route as it was relatively painless (I know: not your problem). I think in the end, we'll just have to make sure we don't use the newer pips when we deploy until we can figure out a more permanent/stable solution. |
Perhaps some pointers on successful internal setups could be added to the docs |
need something in the docs for this. maybe an orphan page linked from the changelog to explain the justification and alternative. |
Ok @qwcode I've added documentation for this. Waiting for the build to complete and then I'll merge it. |
Remove Dependency Links - Needs Discussion
@@ -2,6 +2,10 @@ | |||
|
|||
* **BACKWARD INCOMPATIBLE** Dropped support for Python 3.1. | |||
|
|||
* Removed the deprecated support for dependency links and the | |||
``--process-dependency-links`` flag that turned them on. For alternatives to | |||
dependency links please see http://www.pip-installer.org/en/latest/installing.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whoops, copy/paste fail.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix here: #1595
4 months passed and do we now have a document to guide "successful internal setups" as requested by @Ivoz ? Particularly, a git based setup without the need to package every pushed commit? |
It was added to this branch before it was merged, see: https://pip.pypa.io/en/develop/dependency_links.html |
The solutions given by that page are: However, for solution A, it does not address the "the author of A has to manage changing the C url" problem mentioned in @qwcode's comment #1519 (comment) . I'd like to see ideas or guide to "create a tool outside of pip to automate updating the requirements.txt" mentioned in your reply, or, since 4 months has passed, does such a tool already exist? For solution B, it needs to run package and upload to the private index for every commit, which is painful, like what @enku complained in #1519 (comment) and I agree with him. So, in my point of view, the document page you gave can not be considered as a successful internal setup guide, yet. Can you give more detail on how to overcome the drawbacks of those two solutions? |
Revert "Merge pull request #1519 from dstufft/remove-dependency-links"
|
So if I understand things correctly, If module |
@reubano if |
The link https://pip.pypa.io/en/develop/dependency_links.html is dead. |
Yup, the supposed documentation page for this removal seems to be gone... |
We're on version I really miss having something like |
With 'dependency_links=' setup.py supports requirements on packages hosted on other reposetories then PyPi, see "Packages Not On PyPI" [1]. The big drawback is, due to security reasons, this feature is not supported by pip [2] [1] http://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi [2] pypa/pip#1519 Signed-off-by: Markus Heiser <[email protected]>
I found this issue a bit misleading. Will |
So when dependency links were deprecated we specified they'd have an accelerated removal and they'd be removed in 1.6. We are now at the point that develop is 1.6.dev1 so I've done the work to remove these. However the point stands of do we want to remove these.
There have been a few requests to keep them:
Overall I think that this is a misfeature of setuptools that pip should no longer support. There doesn't seem to be a massive number of users for this feature, and nobody has thus far pointed out any uses of dependency links that can't also be handled perfectly fine with a requirements.txt file.
What do we think?