Add a warning when run as root (e.g., sudo pip)

[winsonluk]

This change discourages the usage of pip as root by showing a warning when pip install is run with sudo.

$ sudo pip3 install
WARNING: Running pip as root is not recommended. This could result in the corruption of the package directories when a package manager is used. See #6409 for more details.
ERROR: You must give at least one requirement to install (see "pip help install")

Fixes #6409
...and potentially helps helps prevent hundreds of future issues related to sudo pip: https://github.com/pypa/pip/issues?q=is%3Aissue+%22sudo+pip%22

[hroncok]

The Fedora's patch: https://src.fedoraproject.org/rpms/python-pip/blob/master/f/emit-a-warning-when-running-with-root-privileges.patch

[winsonluk]

The Fedora's patch: https://src.fedoraproject.org/rpms/python-pip/blob/master/f/emit-a-warning-when-running-with-root-privileges.patch

that's some good stuff, thanks @hroncok. i added the venv check and updated the warning message with the tip

[pradyunsg]

/cc @dstufft since last I checked he was opposed to just doing this.

[pradyunsg]

There's still one sticking point with this: docker containers give you root by default.

I don't think we should be calling out the use of pip in root shells like that.

[hroncok]

I think we should. Even in docker, installing with pip as root can lead to serious problems.

[pradyunsg]

Perhaps. But that's still more disruptive than I'd like.

I haven't seen a Dockerfile that's using pip that even tries to install as non-root. Is it even possible to use it as non-root from a Dockerfile?

[hroncok]

It is possible, however not that straightforward. I get your point.

[uranusjr]

Is this really that disruptive? It’s “just” a warning, people ignore warnings all the time when they feel like it. Using root in a Docker container is ultimately still not best practice, and it’s arguably still not a bad idea to show the warning in this situation.

If it’s really that important, we can have --disable-root-check for people to set if they think they know what they’re doing.

[hroncok]

--root / is for people who think they know they are doing ;)

[winsonluk]

Yeah, most people don't use Docker, so I think the benefit of preventing mis-installations exceeds whatever disruption a one-line message causes to Docker users.

Plus this is just a logger.info, not a logger.warning.

[xavfernandez]

Perhaps. But that's still more disruptive than I'd like.

I haven't seen a Dockerfile that's using pip that even tries to install as non-root. Is it even possible to use it as non-root from a Dockerfile?

I can confirm that using virtual environments inside docker is definitely a thing

[pradyunsg]

Alrighty. Consider me convinced that this is a good idea overall.

This is the first time I'm looking at the code in this PR and... I have some opinions about how we should implement this. :)

[winsonluk]

Changes requested by @pradyunsg have been implemented.

[uranusjr]

I would try making the message shorter. The longer the warning is, the less likely users would read it.

[hroncok]

This is shorter, yet OTOH the user has no idea what to do instead:

logger.warning("Running pip as root may destroy your system.")`

Do you think providing a link would work? I.e.:

logger.warning("Running pip as root may destroy your system. See https://bit.ly/pip-root for details.")`

[uranusjr]

A link to the documentation feels like a good compromise to me.

[winsonluk]

There's two situations I can think of where someone would run pip with sudo:

1. Unintentionally: it's often easy to forget which commands need sudo and which don't. In this case, the "Running pip as root may break packages" part of the message should be enough to dissuade them; I don't think we'll need to link an essay.
2. Intentionally: A user may run pip without sudo, encounter a permission error, and retry with sudo. We should discourage this by suggesting virtual environments. This is where a link could help imo.

How's this?

Running pip as root may break packages and permissions. Permission errors should be fixed with venv: https://docs.python.org/3/library/venv.html 

[uranusjr]

Either of the followings would be a better link IMO:

• https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments
• https://docs.python.org/3/tutorial/venv.html

The library documentation page covers more on the programmatic interface and isn’t very useful for the audience we target here.

[pradyunsg]

I like @hroncok's link-based suggestion. I'd say use https://pip.pypa.io/warnings/root-usage for that right now. We can setup a redirect in our RTD configuration, to make it point to the right place once this is merged.

It'll also defer the discussion on what the right place is, which I like. 🙃

[winsonluk]

I still think we should explicitly call out permission errors and venv in the warning message, since that's by far the most common misuse of sudo.

So instead of a generic https://pip.pypa.io/warnings/root-usage redirect, how about we specify https://pip.pypa.io/warnings/venv and tentatively redirect it to https://docs.python.org/3/tutorial/venv.html?

Updated warning message:
Running pip as root may break packages and permissions. Permission errors should be fixed with venv: https://pip.pypa.io/warnings/venv

[pfmoore]

How about "Permission errors should be fixed with venv" -> "If you need to install packages not supplied by your system package manager, use a virtual environment: https://pip.pypa.io/warnings/venv"? Focus on what the user wants to do, rather than simply on fixing the error they got.

[pradyunsg]

I think we should call this out for what they're doing here, and provide better guidance on what they should be doing in the link (using a virtualenv, why, what the risks are etc).

[winsonluk]

Shorter message with a focus on the user:

    logger.warning("Running pip as root may break packages and permissions. "                       
                   "You should install packages reliably by using venv: "                                             
                   "https://pip.pypa.io/warnings/venv") 

I agree that technical details belong in the link.

[sbidoul]

I just stumbled upon on this part of the pip code base which warns about permissions and reckons that the user might be using sudo. I thought this might be relevant to the conversation here.

[winsonluk]

I just stumbled upon on this part of the pip code base which warns about permissions and reckons that the user might be using sudo. I thought this might be relevant to the conversation here.

Good call out, this warning often appears when users run as root too. But I think it's useful to keep both cache and root warnings separate, since each case can still exist independently:

• The user may not be running as root, but the cache may still be owned by another user; or,
• The user may be running as root, but the cache and its parent directory are both owned by root.

[winsonluk]

@uranusjr @pradyunsg @pfmoore Rebase is complete. We still need to set up a redirect from https://pip.pypa.io/warnings/venv to https://docs.python.org/3/tutorial/venv.html before merging, but that should be the last step!

[winsonluk]

#9680 configures the redirect. would appreciate a review :)

[pradyunsg]

pip.pypa.io/warnings/venv to docs.python.org/3/tutorial/venv.html

Configured the redirect.

[winsonluk]

Thanks @pradyunsg, this should be good to merge then.

[pradyunsg]

Could you do a rebase of this PR? I'd like to make sure it still passes the current linter setup.

[winsonluk]

Could you do a rebase of this PR? I'd like to make sure it still passes the current linter setup.

@pradyunsg Sure, rebased without issues.

[pradyunsg]

Thanks for this PR! ^.^

[graue70]

I'm adding the current version of the warning message so this PR with some explanations can be found more easily:

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[graue70]

I think we should. Even in docker, installing with pip as root can lead to serious problems.

Using root in a Docker container is ultimately still not best practice, and it’s arguably still not a bad idea to show the warning in this situation.

I can confirm that using virtual environments inside docker is definitely a thing

Both the documentation of the official python docker images and the official docker docs install pip packages with sudo. I can't find any information on the web as to how it should be done instead. Would you please explain how I'm supposed to get rid of the warning when using a Dockerfile? @hroncok @uranusjr @xavfernandez

[uranusjr]

Processes In Containers Should Not Run As Root

Deploying Python Applications with Docker - A Suggestion (It covers a ton of stuff, but your can search for virtualenv to find the relevant parts.)

Maybe we should write a documentation page on this, and point https://pip.pypa.io/warnings/venv to that page instead.

[alexchamberlain]

I think we should. Even in docker, installing with pip as root can lead to serious problems.

@hroncok What issues were you thinking of here? Generally, I recommend people use Docker (well any container) or virtualenv, not both.

[hroncok]

For example, if your docker container image is based on a distribution that uses Python for some critical software (let's say the distribution's package manager is written in Python), using pip with root can essentially turn that package manager unusable and depending on the container image usage, this might break use cases of people building stuff on top of it.

[potiuk]

It is totally legitimate case to use pip with root user. I can give you a very good example

In our Production Dockerflie Image for Apache Airflow we have two segments:

Segment 1) installs all the packages for Apache Airlfow including all the packages that need "build-essentials" - i.e. compilation support. This is all done as "root" user and installed with --user flag, so that all the python packages but also .so libraries are placed in ${HOME}/.local of the user (which happens to be root). Using root user helps us to simplify the build segment "USER" management - we do not have to switch between different users to install debian packages. Hell - even more. That segment has NO OTHER USER than root. There is no need for that. And I would not like to add such user because PIP

Then we have segment 2) which uses "root" user to install all "debian" packages, but then we switch to "airflow" user - and first thing we do - we copy the ".local" folder of the "root" user from segment 1 to segment 2 - changing the ownership of all files to "airflow" while copying (nice feature of COPY command":

COPY --chown=airflow:root --from=airflow-build-image /root/.local "${AIRFLOW_USER_HOME_DIR}/.local"

This results in 25-30% smaller images - ca. 250 MB (no need to keep build-essentials and -dev packages around). Also any kind of secrets you might need in segment 1 (for example .pypirc containing passwords to your registry) are not present in the final "segment 2" even if they needed to be copied to "segment 1" in order to be able to install "private packages".

I wonder if the best solution here is:

• warn users by default - this will fulfil it's need of warning the users who are not fully aware of the consequences of running pip as root.

• provide a flag for "power users" to disable the warning. There are quite a number of people who "know what they do" when running PIP as root. And those people should be able to disable it. It could be difficult. Might require some woodo-magic - whatever. It can be complex to disable the warning. Those people will figure it out. And will be able to install pip using root user without the negative consequences.

I would really love to be able to have "warning-free" docker build of Airflow Image. It's not possible currently (or maybe you need stronger magic).

[hroncok]

provide a flag for "power users" to disable the warning.

So, technically, the Fedora patch did allow you to use --root / if you knew what you were doing. This PR does not contain that and I think it was a legitimate thing to have. Thoughts?