Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Dependency Links - Needs Discussion #1519

Merged
merged 1 commit into from
Feb 26, 2014

Conversation

dstufft
Copy link
Member

@dstufft dstufft commented Jan 30, 2014

So when dependency links were deprecated we specified they'd have an accelerated removal and they'd be removed in 1.6. We are now at the point that develop is 1.6.dev1 so I've done the work to remove these. However the point stands of do we want to remove these.

There have been a few requests to keep them:

Overall I think that this is a misfeature of setuptools that pip should no longer support. There doesn't seem to be a massive number of users for this feature, and nobody has thus far pointed out any uses of dependency links that can't also be handled perfectly fine with a requirements.txt file.

What do we think?

@dstufft
Copy link
Member Author

dstufft commented Jan 30, 2014

One note, when removing this a grand total of 2 tests failed. So this was a very poorly tested feature :(

@qwcode
Copy link
Contributor

qwcode commented Jan 30, 2014

nobody has thus far pointed out any uses of dependency links that can't also be handled
perfectly fine with a requirements.txt file

correct me if I'm wrong, but I don't think the case brought up on distutils the other day can conveniently be handled with requirements files
http://code.activestate.com/lists/python-distutils-sig/22434/

his was a private scenario, where all of his projects were only in vcs (and not published as packages on local indexes)

with vcs dependency links in all the projects, he could install his top-level app (with dependencies) with a pip call that only had one argument (the top-level project vcs url)

the alternative with requirements files involves maintaining pinned vcs urls recursively for all dependencies, and there's no easy way that I could tell to manage that file. e.g. supposing he knows that he wants to upgrade one child dependency, sure he can change the vcs url for that one entry, but how to easily do it recursively?

@dstufft
Copy link
Member Author

dstufft commented Jan 30, 2014

I don't understand the question. If you're pinning your versions exactly (either via == or an @hash) then an upgrade will install the same version recursively or not dependency links or not. If you're not pinning and you're just pointing directly at the master branch or what have you then pip will already do the right thing when you reinstall the requirements.txt file.

@qwcode
Copy link
Contributor

qwcode commented Jan 30, 2014

suppose:

  1. A==1 requires B==1 (which requires C==1 and D==1)
  2. A==2 requires B==2 (which requires C==2 and D==1)

(pinned versioning isn't critical to the example)

first, let's make this concrete with vcs dependency links:
(assuming tags are present equivalent to the versions; also assuming install_requires is filled in)

first case:

A's dep links:

  • git://b@1#egg=B

B's dep links:

  • git://c@1#egg=C
  • git://d@1#egg=D

second case:

A's dep links:

  • git://b@2#egg=B

B's dep links:

  • git://c@2#egg=C
  • git://d@1#egg=D

you install A==1 with: pip install git://a@1#egg=A
you install A==2 with: pip install git://a@2#egg=A

now, let's make it concrete with a requirements file for A

first case:

-e .
git://b@1#egg=B
git://c@1#egg=C
git://d@1#egg=D

2nd case:

- e .
git://b@2#egg=B
git://c@2#egg=C
git://d@1#egg=D

you install A==1 with: pip install -r requirements.txt (from A==1 src)
you install A==2 with: pip install -r requirements.txt (from A==2 src)

The concern here when using requirements files was that the author of A has to manage changing the C url. In the dep's link scenario, that was managed by the developer of B.
For many projects, that can become unwieldy.

@dstufft
Copy link
Member Author

dstufft commented Jan 30, 2014

Hmm, well OK so that particular case can become somewhat more complicated than without dependency links. However AFAIK this is no different than other languages besides Go which has a horrible packaging story. Not that other languages not having a feature is a reason to get rid of it.

So yea, this particular case would be somewhat more work out of the box. However it wouldn't be all that difficult for someone to create a tool outside of pip to automate updating the requirements.txt. The original reporter of that thread seemed to have been OK with an external tool serving this purpose for him.

Dependency links has a pretty narrow scope of where it's really reasonable to use them at all, and seeing as how an external tool combined with the requirements file format can completely replicate them in what is more likely to be a saner idea in general I don't see a whole lot of point to leaving this attractive nuisance around. Especially when the "worst" case is that some folks have to manually update a requirements.txt file.

And really if you have multiple independent packages owned by several different people you should be using a package server..

@qwcode
Copy link
Contributor

qwcode commented Jan 30, 2014

Hmm, well OK so that particular case can become somewhat more complicated
than without dependency links. However AFAIK this is no different than
other languages besides Go which has a horrible packaging story. Not that
other languages not having a feature is a reason to get rid of it.

So yea, this particular case would be somewhat more work out of the box.
However it wouldn't be all that difficult for someone to create a tool
outside of pip to automate updating the requirements.txt. The original
reporter of that thread seemed to have been OK with an external tool
serving this purpose for him.

the reporter was more interested in some kind of general mapping scheme
towards the end (vs a parser tool), that I personally thought was pretty
neat, but thinking it's neat, and having pip implement it is another
matter.

but, as for dropping in this 1.6, I'm noticing the 1.5 changelog doesn't
mention a DEPRECATION for it, but rather just mentions the behavior change,
and the need for the new flag.

@dstufft
Copy link
Member Author

dstufft commented Jan 30, 2014

Hmm, you're right the changelog doesn't mention a deprecation, however there was an actual warning raised when you attempted to use the --process-dependency-links flag.

@clee704
Copy link

clee704 commented Jan 30, 2014

I'm using dependency_links to pull a package from GitHub since the version on PyPI breaks with Python 3 while the one on GitHub is more recent and works. What's my alternative? I know the best option would be uploading the new version to PyPI, but that probably won't happen, or at least not soon, considering that the last time the package author made a change to the project was 6 months ago.

P.S. I'm re-posting this from pypa-dev group since it seems more active here.

@dstufft
Copy link
Member Author

dstufft commented Jan 30, 2014

So you have project A that depends on project B from GitHub. Are you distributing project A on PyPI or is it internal only?

@clee704
Copy link

clee704 commented Jan 30, 2014

Not yet but will so on PyPI.

@dstufft
Copy link
Member Author

dstufft commented Jan 30, 2014

So if you're uploading A to PyPI then the answer is to rename project B and upload it to PyPI under a different name. Using dependency links in the case where you're distributing things on PyPI is a use case we've actively tried to stop as a random project on PyPI should not be able to force you to get your dependency's from a third party site.

@clee704
Copy link

clee704 commented Jan 30, 2014

Thanks. That seems the right way.

@Ivoz
Copy link
Contributor

Ivoz commented Jan 31, 2014

It seems [to me] the main use case people have developed for this is for setting up a tree of package dependencies based on VCS urls.

If your situation is complex enough, I think it's warranted to recommend moving to a local / lan pypi caching server, like devpi or mypypi. Sometimes this is made more difficult when one needs to set up verified HTTPS for such a server just so pip doesn't complain.

For only local caching, --no-index --find-links=<dir> is generally simple enough.

I can appreciate working off VCS urls can make for quite a simple workflow for small-medium teams for their own use. It is often even quite secure when most VCS services run over HTTPS.

I think a caching server would be a lot easier to recommend, if pypi could be specifically configured to allow HTTP for some domains / IPs. This means you could allow a development pypi server so that it doesn't need a CA-bought or self-signed-and-installed certificate just to work nicely.

@dstufft
Copy link
Member Author

dstufft commented Jan 31, 2014

Pip doesn't prevent you from using HTTP at all. It just warns you that you're using HTTP.

@dstufft
Copy link
Member Author

dstufft commented Jan 31, 2014

To be clear, pip install -i http://example.com/foo/ bar is something that will always work, even though we don't have a TLS transport. The only way we'd ever make it a requirement to have TLS is if there was always a way to turn off that requirement.

@Ivoz
Copy link
Contributor

Ivoz commented Jan 31, 2014

I had the impression that people were concerned about 'warning fatigue' from this sort of situation, though? The above is a proposal to alleviate such, but without going overboard with some sort of a --no-warn-insecure flag that people might be tempted to chuck everywhere or try to indiscriminately enable.

@dstufft
Copy link
Member Author

dstufft commented Jan 31, 2014

Ah right. Warning fatigue is a thing for sure, and we actually have an issue here somewhere for allowing you to acknowledge that you're using an insecure transport.

@dstufft
Copy link
Member Author

dstufft commented Feb 1, 2014

So what do we think about this, @qwcode @pfmoore ?

@qwcode
Copy link
Contributor

qwcode commented Feb 1, 2014

ok, I see the "will be removed in pip 1.6" ouput when using the flag.
we should continue on with the plan, but I admit to feeling concerned about more complaints.
some specific docs on how to transition would be good.
also, let's backfill the changelog as well for when the deprecation was official.

@pfmoore
Copy link
Member

pfmoore commented Feb 1, 2014

TBH, I don't use dependency links, and I don't really understand the scenarios people are discussing, so I don't have an opinion on the technical issues. However, like @qwcode I am concerned that we've been forcing the pace a lot recently, and we may want to be cautious about adding too many breakages in one go.

Ultimately I'll go with whatever you decide, though.

@Ivoz
Copy link
Contributor

Ivoz commented Feb 2, 2014

@pfmoore did you read my first comment at all?

Basically, imagine you are developing a number of packages as part of your team, that have inter depedencies on eachother; you keep the source for these in git repositories that you control and can retrieve from.

The depedency links feature lets you really easily link these dependencies up simply with your normal git checkout URLs, no other fiddling about necessary. You can even link them to specific commits in some cases, making it quite easy to change exact versions found.

That's the simple workflow I think people are bemoaning being removed.

@pfmoore
Copy link
Member

pfmoore commented Feb 2, 2014

@Ivoz yes I did, and I understand what you're saying. I just don't have any experience of that workflow, nor of how dependency-links works, nor of what other workflows might be potential candidates as workarounds (if any). So I'm deferring the decision to people who do understand, that's all.

@enku
Copy link

enku commented Feb 10, 2014

If I may chime in. I have been using dependency_links a lot for my job in cases where we develop and/or deploy from our local code repositories. Much of this code is not on pypi, github, bitbucket or any other publicly accessible resource. While some of our packages are on an internal pypi server, the majority either are not or particular versions/tags/branches are not.

I'm only mentioning this because many of the comments seem to come from the point of view that most things that they install with pip come from public sources, whereas most things we install are in-house and not public pypi packages. If dependency_links is no longer recommended then does anyone have an alternative?

@Ivoz
Copy link
Contributor

Ivoz commented Feb 10, 2014

I'd suggest automating uploading tagged (or another filter) versions to your internal pypi server. Should be getting easier and easier with the preponderance of CI servers out there these days.

@dstufft
Copy link
Member Author

dstufft commented Feb 10, 2014

That, or use a reqirements.txt file to point at the git repositories.

@enku
Copy link

enku commented Feb 11, 2014

The requirements.txt would be particularly sticky because we have dozens of little libraries with all kinds of inter-dependencies (think library A depends on library B which depends on library C) plus the fact that some of them use optional "extras_require" (is that going away too?). So to have one or even a few requirements.txt's that satisfies all use cases would probably take a lot of time to do and be error-prone.

Getting the systems guys to package each version of each library that's in use and put it on the pypi server would also be challenging for us, which is probably the main reason we went the dependency_links route as it was relatively painless (I know: not your problem). I think in the end, we'll just have to make sure we don't use the newer pips when we deploy until we can figure out a more permanent/stable solution.

@Ivoz
Copy link
Contributor

Ivoz commented Feb 12, 2014

Perhaps some pointers on successful internal setups could be added to the docs

@dstufft
Copy link
Member Author

dstufft commented Feb 25, 2014

I plan on merging this tomorrow unless @qwcode or @pfmoore speak up and ask me not to.

@qwcode
Copy link
Contributor

qwcode commented Feb 25, 2014

need something in the docs for this. maybe an orphan page linked from the changelog to explain the justification and alternative.

@dstufft
Copy link
Member Author

dstufft commented Feb 25, 2014

Ok @qwcode I've added documentation for this. Waiting for the build to complete and then I'll merge it.

dstufft added a commit that referenced this pull request Feb 26, 2014
Remove Dependency Links - Needs Discussion
@dstufft dstufft merged commit da02f07 into pypa:develop Feb 26, 2014
@dstufft dstufft deleted the remove-dependency-links branch February 26, 2014 00:57
@@ -2,6 +2,10 @@

* **BACKWARD INCOMPATIBLE** Dropped support for Python 3.1.

* Removed the deprecated support for dependency links and the
``--process-dependency-links`` flag that turned them on. For alternatives to
dependency links please see http://www.pip-installer.org/en/latest/installing.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, copy/paste fail.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix here: #1595

@hongqn
Copy link
Contributor

hongqn commented Jul 2, 2014

4 months passed and do we now have a document to guide "successful internal setups" as requested by @Ivoz ? Particularly, a git based setup without the need to package every pushed commit?

@dstufft
Copy link
Member Author

dstufft commented Jul 3, 2014

It was added to this branch before it was merged, see: https://pip.pypa.io/en/develop/dependency_links.html

@hongqn
Copy link
Contributor

hongqn commented Jul 3, 2014

The solutions given by that page are:
A) point to vcs in requirements.txt, or
B) setup a private package index.

However, for solution A, it does not address the "the author of A has to manage changing the C url" problem mentioned in @qwcode's comment #1519 (comment) . I'd like to see ideas or guide to "create a tool outside of pip to automate updating the requirements.txt" mentioned in your reply, or, since 4 months has passed, does such a tool already exist?

For solution B, it needs to run package and upload to the private index for every commit, which is painful, like what @enku complained in #1519 (comment) and I agree with him.

So, in my point of view, the document page you gave can not be considered as a successful internal setup guide, yet. Can you give more detail on how to overcome the drawbacks of those two solutions?

dstufft added a commit to dstufft/pip that referenced this pull request Aug 1, 2014
…inks"

This reverts commit da02f07, reversing
changes made to 2ad8888.

Conflicts:
	CHANGES.txt
	pip/commands/freeze.py
	pip/commands/list.py
	pip/commands/wheel.py
	pip/index.py
	pip/req/req_set.py
	pip/vcs/subversion.py
dstufft added a commit that referenced this pull request Aug 4, 2014
Revert "Merge pull request #1519 from dstufft/remove-dependency-links"
@reubano
Copy link

reubano commented Feb 14, 2015

A doesn't work in setup.py because install_requires only accepts strings like package>=version, package==version, package<=version, etc. So -e git+https://github.com/... isn't an option. CR #1428

@reubano
Copy link

reubano commented Feb 16, 2015

So if I understand things correctly, If module foo requires module bar, then there's no way to specify a github repo for any of bar's requirements? This is because requirements.txt only comes into play for foo. CR #1428

@Ivoz
Copy link
Contributor

Ivoz commented Feb 17, 2015

@reubano if foo wants bar with a custom verison of spam, then specify the custom version of spam in foo's own install_requires (before spam), or put it at the top in a requirements.txt you are always going to use to install a particular set of packages.

@mforbes
Copy link

mforbes commented May 16, 2015

@jacobsvante
Copy link

Yup, the supposed documentation page for this removal seems to be gone...

@jacobsvante
Copy link

We're on version 8.0.3 now and this feature was supposed to be removed in 1.6. Are you going to keep the feature? If so perhaps the deprecation warning should be removed.

I really miss having something like npm install github:user/project in pip. It's such a pain to work with deployment of private projects written in Python...

return42 added a commit to return42/linuxdoc that referenced this pull request Jul 23, 2016
With 'dependency_links=' setup.py supports requirements on packages
hosted on other reposetories then PyPi, see "Packages Not On PyPI" [1].

The big drawback is, due to security reasons, this feature is not
supported by pip [2]

[1] http://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi
[2] pypa/pip#1519

Signed-off-by: Markus Heiser <[email protected]>
@jsmedmar
Copy link

jsmedmar commented Aug 24, 2016

I found this issue a bit misleading. Will --process-dependency-links be deprecated or not? This pull request: #1955 seems to revert the deprecation

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 3, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet