Ensure the new pipenv tutorial works on Windows

[ncoghlan]

https://packaging.python.org/new-tutorials/installing-and-using-packages/#installing-pipenv covers amending PATH to handle user script installations, but the directory isn't $USER_BASE/bin on Windows, it's $USER_BASE/Scripts (I think)

@pfmoore Would you be able to take a look through https://packaging.python.org/new-tutorials/installing-and-using-packages/ and check for any *nix-specific details?

[theacodes]

When I get back from vacation I'll also run through this on a clean windows machine.

…

On Thu, Aug 31, 2017, 11:53 PM Nick Coghlan ***@***.***> wrote: https://packaging.python.org/new-tutorials/installing-and-using-packages/#installing-pipenv covers amending PATH to handle user script installations, but the directory isn't $USER_BASE/bin on Windows, it's $USER_BASE/Scripts (I think) @pfmoore <https://github.com/pfmoore> Would you be able to take a look through https://packaging.python.org/new-tutorials/installing-and-using-packages/ and check for any *nix-specific details? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#373>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUczfDK49RN5EYJhRYUmB8qzHZY3yLks5sd3_GgaJpZM4PJrTA> .

[pfmoore]

The instructions for adding the user site directory are wrong on Windows:

>py -m site --user-base
C:\Users\xxx\AppData\Roaming\Python
>dir C:\Users\xxx\AppData\Roaming\Python


    Directory: C:\Users\xxx\AppData\Roaming\Python


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       21/06/2017     11:36                Python36

You actually need to add the Python36\Scripts directory to PATH and it's not at all obvious how you'd know it's Python36. I'm not actually aware of a reliable way of getting the Python interpreter to give you that directory (py -m site --user-site, then go up one and into the Scripts directory from there???)

Two issues with pipenv. Sorry I don't have time at the moment to report them to the project or follow up with helping them address the issues:

pipenv install requests says "Warning! If you are running on Windows, you should use the --python option instead." but doesn't say why. I didn't use --python (it was too late by the time I saw the message!!!) but saw no obvious ill effects. From the help, it looks like it's for specifying the Python version (i.e. it gets passed to virtualenv) but there's no need for that - virtualenv finds the default Python from the registry perfectly fine by itself.

The final line of the output of pipenv install requests says "P.S. You have excellent taste! ✨ 🍰 ✨" but the graphics don't display on the Windows console. Chances are they aren't in the font I'm using (I'm on Python 3.6, so displaying Unicode shouldn't be a problem), but I guess the code could also be playing encoding games and getting things wrong... Honestly, I'd suggest simply not including that line - I don't mean to be grumpy but it's clearly only for fun, and by producing garbled output it could have precisely the opposite effect...

[ncoghlan]

Adding @kennethreitz to the thread, as some of @pfmoore's comments above relate to pipenv itself, not just our instructions for using pipenv

[ncoghlan]

I think the suggestion of recommending py -m site --user-site to Windows users is actually a good one, as the editing direction is then just "replace site-packages with Scripts, and add that to PATH".

As part of creating https://github.com/ofek/pybin, @Okef also found this detailed tutorial on updating the System PATH in the Java docs: https://www.java.com/en/download/help/path.xml

However, looking more closely at that, I don't think it's one we want to reference ourselves, since it doesn't cover updating the user PATH (which is what we want people to adjust).

[pfmoore]

Yeah that sounds good (user-site with site-packages replaced with Scripts).

I don't have a good feel for how to advise people on updating PATH, as I know too much to be a good test case for that ;-) What I will say is that I've never had much problem with colleagues needing to update paths - if they don't know they typically either google, or ask someone around them, or muddle through.

[kennethreitz]

@pfmoore can you open issues at https://github.com/kennethreitz/pipenv? we'll be happy to get a patch out asap.

[kennethreitz]

FWIW, the python installation guides here instruct users on how to update their path accordingly.

[kennethreitz]

@pfmoore landed a fix in master that fixes the warning you saw. thanks for the report! Also got basic windows test coverage up and running today, again. Work in progress still, but it's better than nothing :)

[kennethreitz]

Will remove the ✨ 🍰 ✨ on windows next. Unfortunate, because it works in better terminal emulators, (like cmder). but it is what it is.

[kennethreitz]

okay, removed ✨ 🍰 ✨ from windows. will be in the next release.

[kennethreitz]

v5.3.1 released, which includes these changes!

[pfmoore]

@kennethreitz Thanks, and apologies for not being able to follow up with issues as requested - life's been pretty busy lately.

[theacodes]

Okay, it seems the outstanding issues with the guide itself is just the PATH update guidance. I'll try to get my hands on a Window laptop on Friday and run through all of this to make sure it's good.

[kennethreitz]

If I can be of any help, let me know! Have a fresh windows machine sitting right here.

[theacodes]

@kennethreitz If you wanna run through it yourself, PRs are happily accepted. :)

(I'm fresh back from vacation, so I gotta bury myself out of email and peer reviews before I can get to this)

[pfmoore]

What's the expectation on when this guide will go live? I've just tried pipenv "for real" for the first time, and there are some issues I've had that made for what I felt was a bit of a confusing experience once I got past the very basic pipenv install foo stage. I've reported those that are actionable, but for others, I'm struggling to work out how to express "I got confused" in a usable way at the moment 😄.

I'm wondering what's the best way to ensure that when it does go in, users have a good experience with it, in terms of knowing where they go when their needs go beyond what's covered here.

I'm happy to discuss the specifics of my experience, just not sure where the best place to do so would be.

[theacodes]

@pfmoore I was planning on doing the final touches to the pipenv guide and moving it to be the new tutorial during my block of hours on Friday.

I'm wondering what's the best way to ensure that when it does go in, users have a good experience with it, in terms of knowing where they go when their needs go beyond what's covered here.

This is totally valid (and useful). I'd be really interested in hearing your thoughts.

I'm happy to discuss the specifics of my experience, just not sure where the best place to do so would be.

Here is fine. I suppose I could also hop in IRC sometime.

[pfmoore]

@jonparrott I'm happy with here. In the first instance it probably makes sense to document what I did and why, and how it caused me to stumble. That's easier in this format than IRC (which I don't normally use, but can if it helps). We can then work out where to go from there. I assume @kennethreitz will probably be interested too, and if he prefers to take this somewhere else (e.g., specific issues on the pipenv tracker) then I'm OK with that too. I'll do a write-up later this evening.

[theacodes]

Cool - if it's easier for you feel free to write up a "friction log" using google docs or send it over via PR so we can comment/review/discuss.

[pfmoore]

The situation I was in, is that I was setting up for a new piece of work, analyzing data. It's an exploratory project rather than a "build an application" type of thing, typical (as I understand it) of many data science types of project.

I wanted to maintain the project environment in a virtualenv, and it seemed like pipenv would be a good tool to use, for various reasons - as a learning exercise (I've never used it before), because it's going to be the recommended approach in future, because it handles "incremental" installs (just do pipenv install foo and it tracks what you added in the Pipfile), and because it gives a reproducible environment very easily. It's better than a requirements file plus pew, because it manages the process of associating the virtualenv with the project directory for me.

The first package I needed was cassiopeia which gets game data for League of Legends via the Riot API. So I started my project with pipenv install cassiopeia, intending to start up a Python REPL to explore the API. I immediately hit issues, because the project needed pycurl, which isn't available as a wheel from PyPI, and needs a C compiler and external libraries to build. Rather than build it myself I went to Christoph Gohlke's site and downloaded the wheel he provides. This is when I started to struggle. pipenv install doesn't seem to allow me to supply a wheel file as an argument, nor does it support a --find-links option like pip. I found this out by trial and error, but then I needed to work out how to proceed.

What I decided I wanted to do was to store the downloaded wheel in my project under a "wheels" directory, and direct pipenv to install from there. So I was now off to look for how to do that.

I went to look in the pipenv documentation (one minor stumbling block here - the "Further documentation guides" for pipenv are a long way down the main page, after "User testimonials", which I normally expect near the end, so for a while I totally missed the fact that there were more docs than the simple "install, use pipenv install and you're off" introduction - but that's my fault for not reading through properly). I was somewhat confused by what I read, as it wasn't immediately obvious to me if the Pipfile file was intended to be user editable, or managed by pipenv. After some digging, I realised it was OK to edit it myself. But there's very little formal documentation I could find of what was allowed in the file. There are examples, of things like specifying an index or a URL for a github repo, but nothing that covered what I wanted to do. So I'm not sure if this scenario is deliberately not supported, not supported yet, or supported but I missed how to do it.

I managed in the end to get pycurl installed by temporarily setting PIP_FIND_LINKS, but Windows doesn't have the Unix shell feature of being able to set a variable just for one command invocation, so that's less convenient than it is on Unix. It also seemed to confuse pipenv a bit, as there's nothing in the Pipfile to indicate what happened, and so reproducing the environment is a bit flaky.

(Using --find-links in a requirements file fails, by the way - pipenv just ignores that line :-()

Next problem was that I needed to install certifi. That was trivial, because it's available from PyPI. Score one for pipenv here :-)

However, I then hit a problem with a dependency, networkx. The 2.0 release is backward incompatible with 1.x, and cassiopeia hasn't been updated to support it yet (and they haven't done a release with an updated dependency of networkx<2.0). So I need to add networkx<2.0 as an explicit install. I tried pipenv install "networkx<2.0", and got a prompt "did you mean networkx?" My first thought was no, I meant to specify the version, so please don't ignore it. But after trying both options, I gather that the correct thing to say was "yes", but I'm not sure what the point of the question was in that case (what else could I have meant?). So I now had networkx 1.11 installed - but honestly, I'm not confident it'll rebuild the environment with 1.11 rather than with 2.0 (although that may be because I'm too aware of the oddities of pip's dependency resolution and I'm worrying unnecessarily - so I wouldn't hold that against pipenv).

So I finally ended up with something that worked, mostly. I didn't feel that confident in the result, though, and I honestly feel I'd have been better with a simple requirements file and pew mktmpenv -r requirements.txt.

There's a fair amount of the above that's a cosnequence of problems installing cassiopeia. And that's not pipenv's fault. Personally, though I found that confusion over what pipenv was doing made my problems worse not better. I can't say that a complete newcomer would feel the same - it's easy for me to say that making a requirements file is simple enough, but it may well be just as much a problem for a newcomer as pipenv was. And of course, pipenv is still relatively new, and I'm 100% confident that as time goes on, this sort of thing will get ironed out resulting in a much nicer experience.

One final annoyance - pipenv shell on Windows doesn't respect the current shell - it always starts cmd.exe. As a Powershell user, that's a real annoyance for me. I've raised a pipenv issue for that - there's probably code in pew (which gets this right) that pipenv can use. But that's a minor thing, and completely fixable.

OK. So that's my experience with pipenv on a real project. Mostly a result of a fairly problematic package to install, and I doubt any tool I was new to using would have fared that much better. And there's nothing here that can't be dealt with. But it might be worth making sure that if this is our recommended approach going forward, we include some advice on how to troubleshoot situations like this. As I say, there's not a lot actionable in the above, but maybe we could include a FAQ covering points like:

• How to handle projects that aren't available on PyPI.
• How to handle projects that are on PyPI but for which you need wheels that aren't.
• How to handle fixing dependency issues (i.e. manually specifying the correct version of a dependency to install).

Hopefully, some of this is useful. If I can do anything to clarify any of the points here, or help with any documentation improvements that seem worthwhile, I'll be happy to do so.

[ncoghlan]

I think there are two potential paths to go down here, since pipenv really does aim to optimise for the case where wheel files are available from a readily accessible shared index server, without relying on local machine-dependent artifact stores.

Option 1 is to simply say for these cases "Use the --site-packages option, and install whatever automation-unfriendly components you need globally on the machine rather than including it in each virtual environment". That's how i personally handle cases like RPM and DNF bindings, as those currently aren't available through PyPI at all, let alone as wheel files. So for pycurl on Windows, you'd install a regular installation-wide copy of the wheel file, and then allow --site-packages access from the venvs that needed it. This isn't a great long term option, as interpreter-wide installs on Windows are fairly annoying as well (since there isn't anything comparable to "dnf install python3-pycurl"), but potentially OK as an interim measure (and one that continues to encourage publishers to make the components they publish through PyPI natively venv-friendly).

Option 2 is to propose that pipenv gain a --community-wheels option (or similar) which configures the virtualenv to implicitly consider Christoph Gohlke's binaries for possible installation. If we did decide we wanted to do something like that longer term, then I'd suggest that we get the PSF's @MarkMangoba involved, as I wouldn't consider it reasonable for us to go down that path without talking to Christoph and his department at UCI first, and I think it would be better to manage such a discussion via the PSF rather than as a completely informal community level thing. We'd also want to look at mirroring those wheel files on PSF-provided infrastructure, such that they were behind the Fastly CDN and available via a PEP 503 compliant index page (even if UCI continued to handle the process of building and publishing them in the first place). Another reason I think we'd need to have the PSF involved in such a discussion is because at that point, we're moving beyond the mere "conveyor of bits" role that's covered by the PyPI ToS and move on to actually running build system code, which relies on the additional permissions granted by open source licenses.

However, I don't think pipenv makes this problem worse for novices - they wouldn't know how to handle it regardless. It may be worth suggesting a section aimed at folks accustomed to their own DIY tooling setups what pipenv deliberately makes harder (e.g. the fact that it takes the view that any external dependency installation steps should be undertaken once-per-interpreter, not once-per-venv)

[pfmoore]

Hmm, if that's the case, then there's no indication of that fact in either pipenv's documentation or ours. Something like "if you need to use packages that are not hosted on a public index server like PyPI, then you should use (alternative approach)" would be useful - although do we have a recommended alternative approach?

While more and more packages (particularly in the scientific stack) seem to be getting binary wheels on PyPI, people getting directed to Christoph Gohlke's set of wheel builds on Windows is still a very common situation. Having to say to people "grab the wheel from Christoph's site - oh, and change your development workflow to stop using the recommended pipenv approach" isn't exactly ideal :-( (And yes, I do think that discovering or adding a new dependency part way through development is common enough to warrant consideration in a "beginners" tutorial.

Option 1 - I don't consider packages that don't provide their own wheels as "automation unfriendly". It's easy to automate the way I describe (include the wheel in the archive, and install via --find-links). Nor do I see any indication that pipenv is only for "automation" scenarios. Again, if that's how we're framing pipenv then our current docs are misleading as they imply it's a good solution for adhoc exploratory projects like the one I describe.

Option 2 - agreed that if Christoph's archive were available as a PyPI-style index, this would not be an issue. My understanding from what I've heard in the past is that he's unwilling to do this, but I don't know why, so that's something that as you say would have to be handled via non-technical channels. (AIUI, pipenv does support extra indexes, so it wouldn't even need a change to pipenv in that case).

I don't think pipenv makes this problem worse for novices

A couple of points here.

1. There's nothing in our current page that implies that we aren't recommending pipenv for non-novices. The page in effect states that pipenv is the recommended approach, unqualified, and the pipenv webpage reinforces that view ("Pipenv — the officially recommended Python packaging tool from Python.org, free (as in freedom)". From my reading, I got the strong impression that even as someone who's familiar with pip, virtualenv, pew and similar, the recommendation is that I should switch to pipenv.
2. At the current time, pipenv does make the problem worse, as there's no solution within pipenv. The response so far to my report that there's no --find-links equivalent is that it's viewed as a future enhancement. If a novice were using pip + pew, they'd have a problem, but StackOverflow or similar would quickly direct them to --find-links. They may not understand the details of that solution, but they could at least use it blindly and move on. Indeed, that's what I found with the cassiopeia situation - although I needed fewer pointers than a novice would have, the information is available. It's only when I tried to translate that advice to pipenv that I hit problems.

Once again, I'm not arguing against pipenv - it looks like a very cool approach, and I look forward to being able to use it for all my projects. But I do think it may be a little premature to recommend it to the extent that this tutorial currently does. There's a strong argument that recommending it gets more users, hence more feedback, which is a good thing. But my experience was that I ended up sufficiently confused that I didn't feel able to give really constructive feedback, and that's a problem.

[ncoghlan]

And I guess that's the key purpose of the issue: prior PyPA experience with pipenv has been primarily based on Linux and Mac OS X, where the typical answers to getting non-wheel dependencies involving using the system package manager or homebrew, rather than downloading a wheel file via the browser and going from there.

So if there's a showstopper UX problem, there's a showstopper UX problem, and we need to work out how to deal with that, rather than try to pretend it isn't a problem and proceed with the recommendation anyway.

[ncoghlan]

@pfmoore Is there something that can be placed in a pip.ini file to say "Here's an extra directory to get wheel files from"? Given that PIP_FIND_LINKS worked for you, then I'm guessing there may be.

If so, I'm thinking this may be amenable to a config file based approach: https://pip.pypa.io/en/stable/user_guide/#config-file

Essentially, we'd tell folks "If you need access to extra wheel files that aren't published on PyPI, adjust your user level pip config accordingly, and pipenv will be able to find them when it needs them".

[pfmoore]

I think the key thing that needs to be clear is what the core use cases for pipenv are. My naive expectation was that it was intended to handle the various use cases that pip does:

• Installs from PyPI
• Editable installs
• Installing from wheel files/local repositories

If that is the intention, then the problem's simple - not all the functionality is implemented yet. If the expectation is that certain things shouldn't be handled via pipenv, then there's a need to define scope, consider how projects migrate when they need to go beyond the boundaries of pipenv's scope, etc.

Is there something that can be placed in a pip.ini file

Yes, but that has the same problem as the environment variable - it's unknown to pipenv and cannot be stored with the project, so copying the project to another machine (something I do a lot) needs out-of-band information on how to configure things so it'll work.

For me, the huge plus of pipenv is that you just take the project directory, and everything you need is present. Environment management is all handled by a simple pipenv install. Out of band configuration disrupts the smoothness of that process.

[pfmoore]

I think the key thing that needs to be clear is what the core use cases for pipenv are.

I should also say that this isn't really for us to say. Ultimately, it's a question for the @kennethreitz and the pipenv developers. We should only be documenting how to use the tool within its scope, and how to recognise when it's not appropriate for a given situation.

[ncoghlan]

I wouldn't expect pipenv to natively support installing from local files, since that makes it harder to reliably reproduce the environment on another machine. Doing that is a problem for the same reason actually using the --site-packages option is a problem: you have to manage any related dependency installation steps manually, outside pipenv.

So I see two parts to this UX concern:

1. Setting up a machine to provide the external dependencies
2. Setting up an individual virtual environment to access the external dependencies

For non-Windows machines, the answers to those two questions are:

1. Install the component at the interpreter level
2. Enable --site-packages on the affected virtual environments

Beyond that, we don't really worry about it - if the platform package manager isn't considered adequate, folks can always opt for something like conda instead.

Those answers aren't great on Windows, since we can't assume that getting things installed at the interpreter level will be any easier than getting them installed into the virtual environment. The UCI/Gohlke wheel files exist, but aren't currently amenable to automated installation (hence the idea of getting Mark to ask if that's just a bandwidth management problem that the PSF could help with - I'll bring that up with the Packaging Working Group).

As an interim measure though, it would be nice if there was a way to say "Save the wheel files you want to use to a local directory" and "Tell pip[env] where that supplemental wheel directory is" (as you did with PIP_FIND_LINKS).

The UX request for pipenv would then just be "Provide a way to include pip config settings in Pipfile and inject them into the created virtual environment" rather than "Support ad hoc installations directly from local artifacts". The first request seems more in line with the spirit of pipenv than the latter, as it also enables things like pulling down packages from private company repositories, not just from local directories on the developer's machine.

[ncoghlan]

Specifically, the section of Pipfile that I'm think of is this one: https://docs.pipenv.org/advanced.html#specifying-package-indexes

If that permitted file:// URLs, or offered a local_dir option as an alternative to url, then it would cover this case, while also reasonably clearly indicating that the result Pipfile depends on external downloads to be used.

[pfmoore]

Absolutely. That's where I expected to find the information.

And regarding reproducibility, I think you're missing my point (or consider it "advanced" usage) that I planned on holding the required wheels within the project directory and installing them from there. That effectively makes for a 100% reproducible setup. With that in mind, I think that "support installations from artifacts stored within the project directory" is a completely reasonable expectation for pipenv (albeit one they may not wish to address - that's up to them), and far less controversial than "support installations from arbitrary local locations".

Also, I don't personally think that using system-installed libraries is something we should be expecting as a component of our recommended workflow. Isolation of project environments is way too important for that to be a good idea. (That's an opinion borne of years of bitter experience dealing with juggling wininst installers on Windows...)

[ncoghlan]

Oh, I'd misunderstood what you were suggesting - I thought you were expecting to be able to use wheels from arbitrary host directories.

I think that makes even more sense as a pipenv RFE, whereby you can specify something like:

[[source]]
bundled_artifacts = "relative_path_within_repo"
name = "bundled"

[pfmoore]

Yep, precisely that. That seems like a really nice enhancement, and totally in line with how pipenv works. I'll add that specific suggestion to the issue I opened over there, as it expresses what I want far better than I did.

[pfmoore]

Relevant pipenv issue: https://github.com/kennethreitz/pipenv/issues/914

[theacodes]

Okay - so are we okay to move forward with promoting to this to top-level tutorial?

I plan to amend the bottom of the tutorial to link to the pip/virtualenv flow as well as mention conda for scientific packages.

[pfmoore]

Personally, I think it's premature. I found trying to use pipenv on a project fairly confusing as soon as I hit a dependency that wasn't trivially installable from PyPI. At that point, neither this guide, nor the pipenv docs, really offer any useful guidance, and I ended up needing to abandon pipenv in favour of pew plus requirements.txt (which was painless, but that may just be because I'm familiar with pip)..

It is possible this is just me having a bad experience. I think we should get a few more opinions.

[theacodes]

Summoning @ncoghlan and @dstufft to weigh in.

@pfmoore is there anything I can add to this tutorial, pipenv's docs, or to another guide to better address your concerns? Or is it just a fundamentally missing feature in pipenv?

[pfmoore]

I honestly don't know. I've been thinking about it, and while there is a missing feature (some equivalent of pip's --find-links) that's not the key point. The real issue is more a conceptual one, of what pipenv is for, and how to use it in a development workflow. So I guess what I'd see as missing is:

1. Why should you use pipenv? I guess the implicit answer is "because it's the recommended approach". But the key thing for me was that it manages your project dependencies and your development virtualenv as a combined "project environment". I still don't see a huge and obvious advantage over pew plus requirements.txt, though - although some of that is probably because I know pip, and because I tend to use throwaway temporary environments rather than a persistent project env.
2. What's the recommended workflow when developing a project using pipenv? (in particular, how do I start if I don't have any dependencies yet - the docs launch straight into pipenv install).
3. Where do I go if I hit something pipenv can't handle? There's no escape hatch that I can see, to add a particular dependency "manually", but keep using pipenv for everything else.

In workflow, there's also the questions around whether people should live in a pipenv shell, or use pipenv run as needed. I know there's a big element of personal preference to that, but it's something I did feel confused about when trying to work out how to get going (although that wasn't helped by a bug that means pipenv shell on Windows currently forces you into cmd even if your main shell is powershell - but they know about that so it should be fixed soon).

Also, some things in pipenv are just obscure. It took me a long time to find pipenv --rm, for example. Why's it an option rather than a command? A task based section ("how do I do X?") might help here.

[theacodes]

A task based section ("how do I do X?") might help here.

I do want to do this, similar to what we have now for pip and virtualenv. I was planning on doing this post-cut-over but happy to block on that.

[ncoghlan]

For the Why pipenv? aspect, there are two major cross-platform pieces:

1. It's not just replacing pew + requirements.txt, it's replacing pew + pip-tools + requirements.txt.in. Defaulting to dependency pinning matters, since it more clearly separates "These are the dependencies I expect to work" (Pipfile) from "This is the combination of dependencies I've actually tested" (Pipfile.lock), even for application dependencies.
2. It records pipenv install commands by default, rather than having to edit your requirements.txt.in and then do a pip-compile + pip-sync

And then one more platform specific one:

3. When using a Linux distro provided Python binary, pipenv install inherently protects you from attempting to install into the system site-packages, getting a permissions error, then risking breaking your system by doing sudo pip install, without being any harder to use than a plain pip install command

As far as a day-to-day workflow goes, I definitely recommend pipenv shell. I find pipenv run is mainly useful if you want to let folks run a development branch, without necessarily wanting to contribute to it (e.g. that's how I use it in https://pagure.io/modularity/fedmod )

So folks certainly can use pew + pip-tools if they prefer, and even just pew if they're not worried about pinning their dependencies, but I see doing that as the package management equivalent of choosing Flask over Django: while doing so does give you more freedom to make your own choices, that very freedom means there are more things that newcomers will have to learn the hard way.

[pfmoore]

@ncoghlan Thanks for the explanations. I guess I'm still unclear (as a consumer of this guide) why I should care about pinning, and what effect it'll have on me. If I'm developing something like a data analysis project (the actual situation I tried out pipenv for) I just want the current version of the library that gets my data from its source. If I check my project into github, should I check in just Pipfile, or should I also check in Pipfile.lock? What's the implication of whichever I choose?

I understand if I'm building an application for deployment, I need to think about making sure I release exactly what I tested. But in my mind, that's more "advanced" use than simply knocking up a script for personal use, or for sharing with my peers, which is where I, personally need a simple environment management process (and it's where colleagues struggle with "how do I do this?" and end up installing stuff into their system Python "because it's what the installation instructions said to do"). I'd have expected people doing full application deployment to have (or be developing) workflows and standards for this, so while the tutorial would give them a starting point, it's not what I'd expect them to use without understanding the options.

[pfmoore]

Also, I guess the other question (from people familiar with older approaches) is, what is the recommended approach to replace pew (or virtualenv) plus requirements.txt? When did using pip-tools (either directly or via pipenv) become recommended practice for newcomers, and why? That's probably where my confusion stems from - I had never seen any "beginner's tutorial" recommending pip-tools, so it's not something I've ever used or considered...

PS - although I work with pip/virtualenv a lot, I do consider myself a "beginner" when it comes to development workflows. I'm only just starting to go beyond the stdlib for anything I'd be willing to consider as reusable for "real work", so this is genuinely really important to me, and I honestly do feel that I'm the target audience for this guide.

[ncoghlan]

You should check in Pipfile.lock, such that when someone else clones the repository later and runs pipenv install or pipenv install --dev, they'll get the exact same versions you did (including checking the hashes to ensure that nobody has intercepted the downloads and messed with them). pipenv thus takes the view that "I want to upgrade my dependencies now" should be a deliberate decision by the project's developer(s), and should go through the same development, testing, and review workflow as any other change to your code.

By contrast, if you only check in Pipfile (or an unpinned requirements.txt), then whoever clones the repository later will get some arbitrary combination of potentially updated dependency versions which may or may not work (depending on how good the publishers of your dependencies are about avoiding compatibility breaks). As soon as there's more than one person working with a repository, this puts the burden of dealing with dependency version updates on arbitrary people at arbitrary times, rather than it being a conscious choice on the part of whoever decides to update the lock file.

While pip-tools lets you retrofit dependency pinning into a requirements.txt based workflow, we don't recommend it to beginners because the UX can be most appropriately described as "expert friendly" - because it's bolted on to the side of regular pip use, it exposes all the lower level mechanics that pipenv just handles for you behind the scenes :)

While the dependency pinning approach avoids dependency updates randomly breaking your stuff, it does carry some risks in terms of continuing to use old versions with known security flaws. To mitigate that, pipenv includes the pipenv check command, which looks up public CVE databases and lets you know if you're relying on anything that should be updated. Folks only use pipenv for local development and testing don't really need to worry about that aspect though - it's mainly applicable to actual app development and service deployments.

I'll also note that the difference between runtime dependencies and dev dependencies is another distinction that pipenv inherently introduces people to: the fact that the dependencies you need to run the application, and those you need to work on changing it aren't always going to be the same. It's not so applicable to the data analysis use case, since analyses don't tend to get "deployed" as such, but it applies for a lot of other things (and even for data analyses, things like linters are going to be dev dependencies rather than runtime dependencies).

[pfmoore]

OK. I'm not 100% comfortable with that view, but that's not the point here. If the tutorial explained the background and recommended approach, and how pipenv supports that approach, that would help a lot.

Again, task based examples (I want to develop a library, I want to develop a standalone application, I'm writing an adhoc script to be shared with others, I'm working on a data analysis project, ...) would help here, as I don't think the priorities are the same for all of them. The trick is not to overwhelm the user with choices of course, while not accidentally leading them into a workflow that's not appropriate for them.

[pfmoore]

OK, thinking some more about this, the following are actual tasks I needed to do when working on my current project. They were all things that the library I was using needed, and they are pretty typical things to do, from my experience and what I've seen from others.

1. Install a package off PyPI.
   Easy, and already covered: pipenv install foo.
2. Install a wheel from somewhere other than PyPI.
   pipenv install file:///C:/absolute/path/to/foo.whl
   This is the classic case of needing a wheel from Christoph Gohlke's archive for Windows users.
   Specifying a local path on the command line (pipenv install C:\Path\To\Foo.whl) fails due to https://github.com/kennethreitz/pipenv/issues/939 but is fixable by using a file: URL. Problems here are that file: URLs can't (as far as I know) handle relative paths, and the rules for how to convert a path on Windows to a URL are not obvious (prepend file:/// and change backslashes to forward slashes seems to work, but there are out of date variations around on the web). There's no way to use a local wheelhouse with pipenv.
3. Install the development version of a package from github.
   pipenv install git+<url>#egg=<proj>
pipenv install git+<url> doesn't work without a #egg= fragment[*]. The error if you omit it explains what to do, but includes a traceback which is suboptimal. (Reported as https://github.com/kennethreitz/pipenv/issues/974). Installing with a #egg= fragment gives the warning "Warning: You installed a VCS dependency in non–editable mode. This will work fine, but sub-depdendencies will not be resolved by $ pipenv lock. To enable this sub–dependency functionality, specify that this dependency is editable." I'm not clear what this means, but it seems a bit scary. And as this is a 3rd party project, not my own project, I don't want editable mode, so it's not clear what I'm doing wrong (getting a warning gives the message that I am doing something wrong).
4. Find out if anything needs updating.
   pipenv update --dry-run
5. Update everything to the latest versions.
   pipenv update
   I've not been able to check if this works for git+https dependencies yet, but I guess I could set up a test to check. The warning mentioned above makes me think something will go wrong - but that may just be indicative of the fact that the warning is unnecessarily scary ;-)
   It won't work for local wheels, as discussed in (2), due to lack of wheelhouse support. I assume I'd have to update the dependency to point to a new wheel, but (short of uninstalling the old wheel and installing the new one manually, or editing Pipfile by hand) I don't know how I'd do that. Also, I presume that pipenv update --dry-run wouldn't tell me that there's a new version of the project on PyPI that I need to get a newer wheel for...

So that's more or less the sort of thing I'd like to see in the tutorial. I'd be happy to work this up into a PR, but at the moment it's somewhat more questions than answers, plus a scattering of "this is suboptimal" comments, and I'd like a PR to be more positive than this ;-)

[*] Technically, the pip docs don't mention omitting the #egg= fragment as an option, but it's always worked fine for me, and projects don't often include them when saying "to use the version from github, do ...". And typing git+ and then pasting the URL in is super-easy.

I think I've said all I can now from my experiences trying out pipenv, so I'll leave things here, for others to decide how to take this further.

[theacodes]

Thanks @pfmoore for your detailed feedback.

It seems (1) is covered by the tutorial and it seems reasonable to add (4) and (5) to the tutorial directly.

For (2) and (3) I think those are best covered in a separate guide (something like "Using pipenv for advanced use cases").

I will try to address all of this during my next set of hours here. :)

[pfmoore]

Thanks @jonparrott

Personally, I still don't think I'll recommend pipenv to newcomers yet. But hopefully it'll improve as additional feedback is generated from the people introduced to it by this tutorial.

[techalchemy]

Hey @jonparrott I followed you over here from Pipenv (I am a maintainer) where we had support for local files which was regressed somewhere. We actually maintained that support and still have it as long as the local files are in the top level project path, but the requirements parser library we use errors out if you pass in anything like a directory separator. I just pushed a working fix for this regression as well as some tests to catch it going forward, so that pipfiles can house local paths relative to the project directory (see kennethreitz/pipenv#540 for the initial discussion on this).

I would say that, from a maintainer perspective, if having a wheel directory as a source option is critical I assume we can discuss it further. As a user who has deployed on windows (sometimes), however, and is a reasonably advanced user of pip, I have never actually used this flag. I don't think it is something most novice users actually need as part of their normal python usage, since as was mentioned most people will just download the applicable wheels and ship it in a local artifacts directory.

[ncoghlan]

@pfmoore As a couple of examples of reasons I think recommending a higher level tool by default is valuable, look into pipenv graph (showing you your current dependency tree) and pipenv check (checking for known security flaws in your dependency stack).

You can do that kind of thing yourself if you want to, but with pipenv you get it by default (hence my comparison to Django as a full-fledged opinionated application development platform vs Flask as a less opinionated foundation that's a more suitable building block for DIY architectural designs).