build.sh failed in aws-s3-deployment due to the way pip3 is invoked

[hencrice]

Note: for support questions, please first reference our documentation, then use Stackoverflow. This repository's issues are intended for feature requests and bug reports.

• I'm submitting a ...

  • 🪲 bug report
  • 🚀 feature request
  • 📚 construct library gap
  • ☎️ security issue or vulnerability => Please see policy
  • ❓ support request => Please see note at the top of this template.

• What is the current behavior?
  If the current behavior is a 🪲bug🪲: Please provide the steps to reproduce
  ./build.sh under @aws-cdk/aws-s3-deployment exits with 1 due to the following reason:

pip3 install --ignore-installed --prefix /tmp/tmp.VUvxbIE7Ql -r /tmp/tmp.5zArwykbRC/requirements.txt
ERROR: Can not combine '--user' and '--prefix' as they imply different installation locations

• What is the expected behavior (or behavior of feature suggested)?
  the build script should have run successfully

• Please tell us about your environment:

  • CDK CLI Version: ?? (not sure where to look this up)
  • Module Version: ?? (not sure where to look this up)
  • OS: Ubuntu 18.04 launched using the latest EC2 AMI
  • Language: N/A

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc)
  changing the line to: PYTHONUSERBASE=${piptemp} pip3 install --ignore-installed -r ${staging}/requirements.txt fixed the issue on Ubuntu. However, the exact command cause the build script to fail on Mac with:

+ pip3 install --ignore-installed -r /var/folders/v4/nbxvs8tj4ms68w5ghhnzn_f1ckngqg/T/tmp.iSt8znck/requirements.txt
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/lib'
Consider using the `--user` option or check the permissions.

[rix0rrr]

Do you have a system-wide pip configuration that is interfering with the call here?

[clareliguori]

I ran into this too on Ubuntu, using pip3 from apt-get install python3-pip (no virtualenv or anything). From pip3 install --help:

  --user                      Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation for
                              site.USER_BASE for full details.)  On Debian systems, this is the default when running outside of a virtual environment and not as root.
  --system                    Install using the system scheme (overrides --user on Debian systems)

So on Ubuntu, pip3 will enable the --user flag silently. Could possibly be fixed by adding --system?

[clareliguori]

Confirmed that for me, adding the --system flag resolved the issue. No idea how this would affect other types of python installations.

diff --git a/packages/@aws-cdk/aws-s3-deployment/lambda/build.sh b/packages/@aws-cdk/aws-s3-deployment/lambda/build.sh
index fe753e2aa..7bd737113 100755
--- a/packages/@aws-cdk/aws-s3-deployment/lambda/build.sh
+++ b/packages/@aws-cdk/aws-s3-deployment/lambda/build.sh
@@ -27,7 +27,7 @@ cd ${staging}
 # install python requirements
 # Must use --prefix to because --target cannot be used on
 # platforms that have a default --prefix set.
-pip3 install --ignore-installed --prefix ${piptemp} -r ${staging}/requirements.txt
+pip3 install --system --ignore-installed --prefix ${piptemp} -r ${staging}/requirements.txt
 mv ${piptemp}/lib/python*/*-packages/* .
 
 # create archive

[clareliguori]

It looks like the pip community considers the Ubuntu/Debian version to be broken because of this behavior that was only merged into the Debian fork. Probably not worth fixing in the cdk scripts.
pypa/pip#1668 (comment)
pypa/pip#4222 (comment)

Workarounds on Ubuntu are 1) install a virtualenv, 2) install from the official pip distribution, or 3) set env variable PIP_USER=0

[rhboyd]

This also bit me when I tried to build CDK in Cloud9. If any future wanderer stumbles across this Issue, the following command will create the appropriate symlink for a Cloud9 environment.

$ sudo ln -s /usr/bin/pip-3.6 /usr/bin/pip3

[iliapolo]

I believe this has been resolved by #8487.

Please reach out if someone still experiences this.