-
-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use pip #518
Use pip #518
Conversation
when did this drift happen? If it is use of conda's
Recipes should all depend on the package described by 1 here, and never the metapackage. This is the best way to keep all dependencies explicit. |
@msarahan I don't know the reason behind this decision and I may be saying something that is absurd here, so bare with me. I brought this topic b/c I was inquired recently, at a local Python meeting, regarding the "many" things conda-forge is doing that is in the opposite direction of what the larger Python community. I was able to answer most of them but not the lack of |
That sounds like an interesting discussion. I'd be curious to hear those perceptions, and how many of them are conda-forge vs. the broader conda ecosystem (i.e. us). Maybe email me if you don't want to post them here? |
Indeed most of them were related to the broader
I guess I can post here so others can see. It is not a big deal to go off topic on this PR. So... from my memory:
That one was easy to answer. I explained that conda is older than wheels and how the static linking in wheels are not a good solution for the scientific community.
Actually I ended up agreeing with this part, not that conda should be more like pip but that pip should be used as much as possible when dealing with pure python packages.
I demonstrated my workflow, that uses conda-envs for reproducible research that depends on R and perk packages. Got a few 👍 from the scientists in the room and a few 👎 from the Python purists.
I basically differed them to Jake's post
Well... After explaining the differences between conda packages, the agnostic part of coda, etc all over again... I had to agree that when it comes to pure Python packages he was right. At the end of the day there was nothing in the discussion that we already know, most of the people in the room were already conda users and only a few Python purists that never used it brought those questions. The only question that I did not had a clean answer was the fact that our Python does not include |
Well, the history is that conda added (and currently still adds) pip and setuptools to python automatically. That was/is a nasty hack, and it was done at such a low level that it actually introduces some really awkward hacking around the solver level. We'd really like to get rid of it. We acknowledge that upstream python ships with pip, and that's fine. We need a way of having python without pip or setuptools, so that we can package setuptools and pip independently of python. That's why I mentioned the metapackage approach above. The best thing to tell people, IMHO, is that we do include pip and setuptools with python - but it's as a dependency, not as actual files. If we bundled pip and setuptools with python, we would not be able to update them independently from python. That means old python versions would not get new pip, and we'd also be rebuilding our python packages (very expensive timewise) for the very frequent setuptools releases. Python purists would probably balk at the idea of rebuilding python just to update setuptools, but the packaging model is just fundamentally different. How would they propose updating to the next bugfix revision of python? Maybe virtualenv can do this? Can they update their root installations? Then you go back to why conda-forge doesn't do the automatic dependency addition: because not listing your actual dependencies makes your software break. Depending on the implicit addition of pip and setuptools is fragile. That implicit addition actually breaks our ability to build setuptools or pip packages, and that's the main reason we turn it off, but having the implicit addition on for building is just generally fragile. As for forking pypi for pure python stuff, we all feel this pain. It is a goal to have conda working better with wheels, and with pip in general. That's Peter's top priority for conda this year. I don't know what the final solution might be, but perhaps it will include conda directly installing wheels or other contents from pypi. You'll still have the debate between virtualenv and conda envs, but conda would then totally replace pip for conda users. If pip ever gets a dependency solver and supports other languages, then maybe we'll be in a better place to discuss unification of env concepts. |
This is what I needed 😄 Thanks! |
Just a note here:
Python doesn’t bundle pip or setuptools either, for exactly the same reason.
It does bundle “ensurepip”, which is a complication.
And PyPa is slowly working in disentangling pip and setuptools anyway,
so we should keep it clean.
…-CHB
|
Interesting, do you have any links to that discussion? I heard the opposite in that meeting. |
@ocefpaf should we post your responses (or something like them somewhere) in the docs? (maybe under FAQ) That way we could have some canned answers for this when people ask. |
I guess that those would be better in a blog post, after some polishing though, like what Jake did with the conda vs pip myths post. |
And PyPa is slowly working in disentangling pip and setuptools anyway,
It’s been more or less the primary topic on the distutils SIG for a long
while, and there are a few PEPs trying to lock down the meta-Data standards
and protocols so we can cleanly separate build tools from install tools.
561 and 566 I think.
However, setuptools is a big bundle of building, installing, and run-time
tools, so an ugly mess to untangle.
…-CHB
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's roll with it. 👍
@jakirkham, given your involvement in #528 and conda-forge/staged-recipes#642, would you mind merging? Given the documentation and the clear statement, I propose we also close conda-forge/staged-recipes#642. |
I would like to add that even though this change is being suggested for philosophical/community reasons, there is a strong technical argument to suggest the usage of pip over setup tools. pip, by default will build a wheel instead of an egg. In the case of pure python, a wheel is python version independent (truly universal) and therefore is what 'noarch: python' packages need. Egg builds have an egg-info entry that still contain some python version information. This is only one technical benefit for pure python modules to be wheels instead of eggs. FWIW, setuptools with wheel can build wheels if so commanded:
This is not nearly as clean as the suggest pip one-liner. |
I tried to use pip instead of setuptools for my conda recipes a while back -- in that case, so that I could use the same code to build/install both pure distutils and setuptools based package -- and to get avoid inadvertent use of eggs and eash_install. I gave up though -- what happened to me is that pip had agressive and confusing (to me) caching that could not seem to turn off -- so when I was testing teh builds, pip would cache something ,and then I'd make a change, and ip would use the cached version, and I wouldn't see the change in the new build. This was a very frustrating experience! It's a not-issue when builds are done in a clean CI, but could still be a challenge when developing/debugging build recipes... So we'll see how it goes... |
Sorry @pelson I think I was out the week that you asked about this and haven't had time to circle back. If using the old That said, Also there is a rocky history with some packages being installed with Other than these two caveats, agree that |
On a related point, would encourage people interested in using |
pip has a
pip also has a
Yep. PEP 517 and PEP 518.
That's what we're hoping PEP 517 and PEP 518 would help fix. Any constructive feedback on them would definitely be appreciated. :) As for pip's current dependency handling, there's Aside: all options in pip have a corresponding environment variable that can be used, if needed. Command line options override the value in the environment variable. |
Yeah these are all good options to include. Though it becomes a bit much for maintainers to carry around, refresh, and perhaps even no all the ins-and-outs of. Had looked at making a package in PR ( conda-forge/staged-recipes#642 ) that would automatically set these sorts of things for the user as long as they add it as a build dependency. Other options would be to move this same information into cc @msarahan |
Closes conda-forge/staged-recipes#528
PS: b/c conda-forge drifted away from Python standards and does not ship with
pip
by default we need to specifypip
as a build dependency. Ideally we should review that policy and get closer to the Python community and shippip
by default again.