Pip install breaks on S3 due to package mismatch error

[spence95]

Apache Airflow version:

1.10.12

Environment:

Any environment

What happened:

Since the beginning of this week, everytime I try to pip install airflow with the following command:

pip install git+https://github.com/apache/incubator-airflow.git@${AIRFLOW_VERSION}#egg=apache-airflow[async,crypto,celery,jdbc,password,postgres,s3]

I get this error:

ERROR: Could not find a version that satisfies the requirement docutils<0.16,>=0.10 (from botocore)
--
980 | ERROR: No matching distribution found for docutils<0.16,>=0.10

But if I install without s3 in the apache-airflow[] array it works.

What you expected to happen:

To install successfully with s3 included so that I can use s3 hooks and such (like for remote logging)

How to reproduce it:

Run the pip install command above

Anything else we need to know:

Full stack trace here:

+ pip install git+https://github.com/apache/[email protected]#egg=apache-airflow[async,crypto,celery,jdbc,password,postgres,s3] --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt
--
950 | Collecting apache-airflow[async,celery,crypto,jdbc,password,postgres,s3]
951 | Cloning https://github.com/apache/incubator-airflow.git (to revision 1.10.12) to /tmp/pip-install-7bsvzbk0/apache-airflow_9a5affed6a26418c8c4cefedfd40b0f9
952 | Collecting alembic<2.0,>=1.0
953 | Downloading alembic-1.4.2.tar.gz (1.1 MB)
954 | Installing build dependencies: started
955 | Installing build dependencies: finished with status 'done'
956 | Getting requirements to build wheel: started
957 | Getting requirements to build wheel: finished with status 'done'
958 | Preparing wheel metadata: started
959 | Preparing wheel metadata: finished with status 'done'
960 | Collecting argcomplete~=1.10
961 | Downloading argcomplete-1.12.0-py2.py3-none-any.whl (38 kB)
962 | Collecting attrs~=19.3
963 | Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
964 | Collecting bcrypt>=2.0.0
965 | Downloading bcrypt-3.2.0-cp36-abi3-manylinux2010_x86_64.whl (63 kB)
966 | Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python3.6/site-packages (from bcrypt>=2.0.0->apache-airflow[async,celery,crypto,jdbc,password,postgres,s3]) (1.15.0)
967 | Collecting boto3~=1.10
968 | Downloading boto3-1.14.44-py2.py3-none-any.whl (129 kB)
969 | Collecting botocore<1.18.0,>=1.17.44
970 | Downloading botocore-1.17.44-py2.py3-none-any.whl (6.5 MB)
971 | INFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while.
972 | INFO: pip is looking at multiple versions of bcrypt to determine which version is compatible with other requirements. This could take a while.
973 | INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
974 | INFO: pip is looking at multiple versions of argcomplete to determine which version is compatible with other requirements. This could take a while.
975 | INFO: pip is looking at multiple versions of apache-airflow to determine which version is compatible with other requirements. This could take a while.
976 | INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
977 | INFO: pip is looking at multiple versions of alembic to determine which version is compatible with other requirements. This could take a while.
978 | INFO: pip is looking at multiple versions of apache-airflow[async,celery,crypto,jdbc,password,postgres,s3] to determine which version is compatible with other requirements. This could take a while.
979 | ERROR: Could not find a version that satisfies the requirement docutils<0.16,>=0.10 (from botocore)
980 | ERROR: No matching distribution found for docutils<0.16,>=0.10

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ashb]

Paging @potiuk as our resident "pip resolver expert" (A title which he hates, I'm sure 😁)

[potiuk]

Why no, not really @ashb :).

@spence95 Can you please downgrade PIP to 20.2.4 ? Tjhis most likely the issue with new PIP 20.3 rolled out on Nov 30th.

There are many issues (at least for Airflow ubt also a number of other projects) with the new PIP (including this pypa/pip#9203) and we have updated our docs to point out everyone to downgrade PIP before reporting any issues. The updated docs (for 1.10.14 but should also work for 1.10.12) - https://github.com/apache/airflow/blob/v1-10-stable/docs/installation.rst

Let us know if still does not work after that.

[potiuk]

Pleaase let us know @spence95 if downgrading PIP solves the problem

[potiuk]

I confirmed it myself:

With PIP 20.3:

This fails:
pip install 'git+https://github.com/apache/[email protected]#egg=apache-airflow[async,crypto,celery,jdbc,password,postgres,s3]' --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt

This succeeds:

pip install --use-deprecated legacy-resolver 'git+https://github.com/apache/[email protected]#egg=apache-airflow[async,crypto,celery,jdbc,password,postgres,s3]' --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt

So I can officially confirm it's the new PIP.