Bazel is using python user site packages by default

[jwnimmer-tri]

Description of the problem / feature request:

See https://docs.python.org/3/library/site.html#site.ENABLE_USER_SITE for background.

When Bazel runs build actions, we want them to be hermetic by default, maybe with opt-out for certain cases or enforced-through-sandbox for others.

Python by default uses PYTHONPATH to find modules, which is easy enough to make hermetic by not including it in the --action_env -- and indeed, its off by default.

Python also has magic to hunt down modules from other places on the filesystem. Some are in /usr/lib, some are like /usr/local/lib/python2.7/dist-packages, etc., but these are par for the course and similar to other languages' rules' level of reliance on /usr being invariant.

However, then we come to Python's "User Site Packages" which are based in $HOME/.local/lib/... on Ubuntu, for example. The python interpreter will use these by default, but for Bazel purposes using undeclared files from $HOME at action runtime seems extremely dangerous, and has caused problems for some of my Bazel users that have a cluttered python environment in their homedir.

Feature requests: what underlying problem are you trying to solve with this feature?

Disable Python User Site Packages by default, at least for actions (and also probably repository rules).

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

jwnimmer@call-cc:~/tmp/8475$ mkdir -p /home/jwnimmer/.local/lib/python2.7/site-packages
jwnimmer@call-cc:~/tmp/8475$ touch WORKSPACE
jwnimmer@call-cc:~/tmp/8475$ cat BUILD
py_binary(
  name = "foo",
  srcs = ["foo.py"]
)
genrule(
  name = "gen",
  outs = ["out"],
  cmd = "$(location :foo) $@",
  tools = [":foo"],
)
jwnimmer@call-cc:~/tmp/8475$ cat foo.py
import sys
with open(sys.argv[1], "w") as out:
    for item in sys.path:
        out.write(item + "\n");
jwnimmer@call-cc:~/tmp/8475$ bazel build out
...
jwnimmer@call-cc:~/tmp/8475$ cat bazel-genfiles/out 
/home/jwnimmer/tmp/8475
/home/jwnimmer/.cache/bazel/_bazel_jwnimmer/b990de88bae8ac9db8d9065922f293cb/bazel-sandbox/7539283830963849603/execroot/__main__/bazel-out/host/bin/foo.runfiles
/home/jwnimmer/.cache/bazel/_bazel_jwnimmer/b990de88bae8ac9db8d9065922f293cb/bazel-sandbox/7539283830963849603/execroot/__main__/bazel-out/host/bin/foo.runfiles/__main__
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/lib-old
/usr/lib/python2.7/lib-dynload
/home/jwnimmer/.local/lib/python2.7/site-packages     <<======= BAD
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/PILcompat
/usr/lib/python2.7/dist-packages/gtk-2.0
/usr/lib/python2.7/dist-packages/wx-3.0-gtk2

What operating system are you running Bazel on?

Ubuntu 16.04.

What's the output of bazel info release?

release 0.10.1

Any other information, logs, or outputs that you want to share?

See RobotLocomotion/drake#8476 for my project-specific fix -- setting --action_env=PYTHONNOUSERSITE=1. I haven't tested whether this fixes repository rules, but seems to work for regular build rules, at least.

Edit: Setting the action_env only repairs genrule()s. Skylark rule()s have an empty env, so each one that is calling Python needs to pass env to actions.run explicitly.

[johnynek]

this is an expanded version of #890

Also note the effort here:
https://groups.google.com/forum/#!forum/bazel-sig-python

[brandjon]

I agree we definitely should opt out of user site packages by default. Perhaps we can add an attribute to py_runtime to optionally opt back in.

[brandjon]

See #7959 for related sys.path issues.

[NathanHowell]

I'm experimenting with using virtual environments in Python 3 to accomplish this, see https://gist.github.com/NathanHowell/5cf4a353a8dd3a1025e682c4707d5bac

[keith]

To solve this so far I've started setting PYTHONNOUSERSITE in our bazel wrapper, as well as passing --test_env=PYTHONNOUSERSITE=nonempty to all test invocations. Without the latter py_test invocations still included the user's site-packages

[neilisaac]

In case this helps anyone, you may be able to disable user site packages using env -S in the shebang of the generated py_binary wrapper scripts:

py_runtime(
    ...
    stub_shebang = "#!/usr/bin/env -S PYTHONNOUSERSITE=1 python",
)

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!

[kjw-enf]

We tripped over this again today. My user's ~/.local/lib/python3.8/site-packages/ dir caused bazel startup to mysteriously fail (processing WORKSPACE). Details and repros for how to demonstrate how the Python (compiled-in) path functions and how the PYTHONNOUSERSITE=notemptyvalue operates can be found in https://stackoverflow.com/questions/41360022/how-to-ignore-python-module-in-local-lib-python2-7-site-packages.

See also bazelbuild/rules_python#2060 which suggests that env variable propagation can be a headache, but I think it's fine in this case. I'm sure somewhere, that someone is abusing the .local and will complain, but I will likely be implementing one of the above workarounds.