How do I resolve moto and boto3 more quickly?

[iainelder]

Recently we deleted the lock file from our project because we wanted to automatically use the latest versions of all the dependencies.

I think now it was a mistake because the project doesn't install properly.

Poetry appears to fail to resolve correctly the dependencies of moto and boto3.

With the lock file, poetry install would complete in a few seconds and of course with the same exact versions.

The lock file before it was deleted specified moto==3.0.4 and boto3==1.26.34.

Without the lock file, poetry install is impossibly slow to complete, and ends up installing ancient dependencies that break the test code.

The latest CI run took nearly 4 hours and ended up installing ancient versions moto==1.3.6 and boto3==1.7.84.

You can show the problem with Poetry's resolver by running the following commands against my project.

cd "$(mktemp --dir)"
git clone https://github.com/connelldave/botocove
cd botocove
poetry debug resolve -vv

And you can produce the same output with this minimal pyproject.toml file.

[tool.poetry]
name = "test"
version = "1"
description = ""
authors = []

[tool.poetry.dependencies]
python = "3.8"
moto = "*"
boto3 = "*"

The complete output of the first solution:

   1: fact: resolver-test is 0.1.0
   1: derived: resolver-test
   1: fact: resolver-test depends on boto3 (*)
   1: fact: resolver-test depends on moto (3.0.4)
   1: selecting resolver-test (0.1.0)
   1: derived: moto[organizations,sts] (==3.0.4)
   1: derived: boto3
   1: fact: moto (3.0.4) depends on moto (3.0.4)
   1: fact: moto (3.0.4) depends on boto3 (>=1.9.201)
   1: fact: moto (3.0.4) depends on botocore (>=1.12.201)
   1: fact: moto (3.0.4) depends on cryptography (>=3.3.1)
   1: fact: moto (3.0.4) depends on requests (>=2.5)
   1: fact: moto (3.0.4) depends on xmltodict (*)
   1: fact: moto (3.0.4) depends on werkzeug (*)
   1: fact: moto (3.0.4) depends on pytz (*)
   1: fact: moto (3.0.4) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: moto (3.0.4) depends on responses (>=0.9.0)
   1: fact: moto (3.0.4) depends on MarkupSafe (!=2.0.0a1)
   1: fact: moto (3.0.4) depends on Jinja2 (>=2.10.1)
   1: selecting moto[organizations,sts] (3.0.4)
   1: derived: Jinja2 (>=2.10.1)
   1: derived: MarkupSafe (!=2.0.0a1)
   1: derived: responses (>=0.9.0)
   1: derived: python-dateutil (>=2.1,<3.0.0)
   1: derived: pytz
   1: derived: werkzeug
   1: derived: xmltodict
   1: derived: requests (>=2.5)
   1: derived: cryptography (>=3.3.1)
   1: derived: botocore (>=1.12.201)
   1: derived: boto3 (>=1.9.201)
   1: derived: moto (==3.0.4)
   1: fact: jinja2 (3.1.2) depends on MarkupSafe (>=2.0)
   1: selecting jinja2 (3.1.2)
   1: derived: MarkupSafe (>=2.0)
   1: selecting markupsafe (2.1.2)
   1: fact: python-dateutil (2.8.2) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.2)
   1: derived: six (>=1.5)
   1: selecting six (1.16.0)
   1: selecting xmltodict (0.13.0)
   1: fact: cryptography (40.0.2) depends on cffi (>=1.12)
   1: selecting cryptography (40.0.2)
   1: derived: cffi (>=1.12)
   1: fact: cffi (1.15.1) depends on pycparser (*)
   1: selecting cffi (1.15.1)
   1: derived: pycparser
   1: selecting pycparser (2.21)
   1: fact: responses (0.23.1) depends on requests (>=2.22.0,<3.0)
   1: fact: responses (0.23.1) depends on urllib3 (>=1.25.10)
   1: fact: responses (0.23.1) depends on pyyaml (*)
   1: fact: responses (0.23.1) depends on types-PyYAML (*)
   1: selecting responses (0.23.1)
   1: derived: types-PyYAML
   1: derived: pyyaml
   1: derived: urllib3 (>=1.25.10)
   1: derived: requests (>=2.22.0,<3.0)
   1: fact: requests (2.30.0) depends on charset-normalizer (>=2,<4)
   1: fact: requests (2.30.0) depends on idna (>=2.5,<4)
   1: fact: requests (2.30.0) depends on urllib3 (>=1.21.1,<3)
   1: fact: requests (2.30.0) depends on certifi (>=2017.4.17)
   1: selecting requests (2.30.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<3)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2,<4)
   1: selecting idna (3.4)
   1: selecting pyyaml (6.0)
   1: selecting charset-normalizer (3.1.0)
   1: selecting urllib3 (2.0.2)
   1: selecting certifi (2023.5.7)
   1: selecting types-pyyaml (6.0.12.9)
   1: selecting pytz (2023.3)
   1: fact: werkzeug (2.3.4) depends on MarkupSafe (>=2.1.1)
   1: selecting werkzeug (2.3.4)
   1: fact: boto3 (1.26.135) depends on botocore (>=1.29.135,<1.30.0)
   1: fact: boto3 (1.26.135) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: boto3 (1.26.135) depends on s3transfer (>=0.6.0,<0.7.0)
   1: selecting boto3 (1.26.135)
   1: derived: s3transfer (>=0.6.0,<0.7.0)
   1: derived: jmespath (>=0.7.1,<2.0.0)
   1: derived: botocore (>=1.29.135,<1.30.0)
   1: fact: botocore (1.29.135) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: botocore (1.29.135) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.29.135) depends on urllib3 (>=1.25.4,<1.27)
   1: derived: not botocore (==1.29.135)
   1: fact: no versions of botocore match >1.29.135,<1.30.0
   1: conflict: no versions of botocore match >1.29.135,<1.30.0
   1: ! botocore (>1.29.135,<1.30.0) is partially satisfied by not botocore (==1.29.135)
   1: ! which is caused by "botocore (1.29.135) depends on urllib3 (>=1.25.4,<1.27)"
   1: ! thus: botocore (>=1.29.135,<1.30.0) requires urllib3 (>=1.25.4,<1.27)
   1: fact: botocore (>=1.29.135,<1.30.0) requires urllib3 (>=1.25.4,<1.27)
   1: derived: not botocore (>=1.29.135,<1.30.0)
   1: derived: not boto3 (==1.26.135)

The errors from the next five solutions:

   2: conflict: no versions of botocore match >1.29.134,<1.29.135
   2: ! botocore (>1.29.134,<1.29.135) is partially satisfied by not botocore (==1.29.134)
   2: ! which is caused by "botocore (1.29.134) depends on urllib3 (>=1.25.4,<1.27)"
   2: ! thus: botocore (>=1.29.134,<1.29.135) requires urllib3 (>=1.25.4,<1.27)
    3: conflict: no versions of botocore match >1.29.133,<1.29.134
   3: ! botocore (>1.29.133,<1.29.134) is partially satisfied by not botocore (==1.29.133)
   3: ! which is caused by "botocore (1.29.133) depends on urllib3 (>=1.25.4,<1.27)"
   3: ! thus: botocore (>=1.29.133,<1.29.134) requires urllib3 (>=1.25.4,<1.27)
   4: conflict: no versions of botocore match >1.29.132,<1.29.133
   4: ! botocore (>1.29.132,<1.29.133) is partially satisfied by not botocore (==1.29.132)
   4: ! which is caused by "botocore (1.29.132) depends on urllib3 (>=1.25.4,<1.27)"
   4: ! thus: botocore (>=1.29.132,<1.29.133) requires urllib3 (>=1.25.4,<1.27)
   5: conflict: no versions of botocore match >1.29.131,<1.29.132
   5: ! botocore (>1.29.131,<1.29.132) is partially satisfied by not botocore (==1.29.131)
   5: ! which is caused by "botocore (1.29.131) depends on urllib3 (>=1.25.4,<1.27)"
   5: ! thus: botocore (>=1.29.131,<1.29.132) requires urllib3 (>=1.25.4,<1.27)

The resolver appears to be counting down through every single version of botocore, of which there are thousands, and discarding each of the for some unresolved dependency on urllib3.

My library uses moto for testing and so I can control its version precisely if I need to.

I don't want to restrict the version of boto3 because users of the library will also use it explicitly and specify a version.

[edgarrmondragon]

@iainelder Can you try pinning urllib3 to <2 and check if the resolution time improves?

Incompatibility is a known issue (boto/botocore#2926) at the moment, so giving a hint to Poetry could make the resolution faster.

[iainelder]

Awesome, @edgarrmondragon ! Thanks for the pointing to the botocore issue.

The new resolution time is 2 seconds and it seems to pick the latest versions of the moto and boto3.

pyproject.toml file that pins urllib3:

[tool.poetry]
name = "test"
version = "1"
description = ""
authors = []

[tool.poetry.dependencies]
python = "3.8"
moto = "*"
boto3 = "*"
urllib3 = "<2"

Full output of the resolver:

Resolving dependencies...
   1: fact: test is 1
   1: derived: test
   1: fact: test depends on moto (*)
   1: fact: test depends on boto3 (*)
   1: fact: test depends on urllib3 (<2)
   1: selecting test (1)
   1: derived: urllib3 (<2)
   1: derived: boto3
   1: derived: moto
   1: selecting urllib3 (1.26.15)
   1: fact: moto (4.1.9) depends on boto3 (>=1.9.201)
   1: fact: moto (4.1.9) depends on botocore (>=1.12.201)
   1: fact: moto (4.1.9) depends on cryptography (>=3.3.1)
   1: fact: moto (4.1.9) depends on requests (>=2.5)
   1: fact: moto (4.1.9) depends on xmltodict (*)
   1: fact: moto (4.1.9) depends on werkzeug (>=0.5,<2.2.0 || >2.2.0,<2.2.1 || >2.2.1)
   1: fact: moto (4.1.9) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: moto (4.1.9) depends on responses (>=0.13.0)
   1: fact: moto (4.1.9) depends on Jinja2 (>=2.10.1)
   1: selecting moto (4.1.9)
   1: derived: Jinja2 (>=2.10.1)
   1: derived: responses (>=0.13.0)
   1: derived: python-dateutil (>=2.1,<3.0.0)
   1: derived: werkzeug (>=0.5,!=2.2.0,!=2.2.1)
   1: derived: xmltodict
   1: derived: requests (>=2.5)
   1: derived: cryptography (>=3.3.1)
   1: derived: botocore (>=1.12.201)
   1: derived: boto3 (>=1.9.201)
   1: fact: jinja2 (3.1.2) depends on MarkupSafe (>=2.0)
   1: selecting jinja2 (3.1.2)
   1: derived: MarkupSafe (>=2.0)
   1: selecting markupsafe (2.1.2)
   1: fact: responses (0.23.1) depends on requests (>=2.22.0,<3.0)
   1: fact: responses (0.23.1) depends on urllib3 (>=1.25.10)
   1: fact: responses (0.23.1) depends on pyyaml (*)
   1: fact: responses (0.23.1) depends on types-PyYAML (*)
   1: selecting responses (0.23.1)
   1: derived: types-PyYAML
   1: derived: pyyaml
   1: derived: requests (>=2.22.0,<3.0)
   1: fact: requests (2.30.0) depends on charset-normalizer (>=2,<4)
   1: fact: requests (2.30.0) depends on idna (>=2.5,<4)
   1: fact: requests (2.30.0) depends on urllib3 (>=1.21.1,<3)
   1: fact: requests (2.30.0) depends on certifi (>=2017.4.17)
   1: selecting requests (2.30.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2,<4)
   1: selecting idna (3.4)
   1: selecting pyyaml (6.0)
   1: selecting charset-normalizer (3.1.0)
   1: fact: python-dateutil (2.8.2) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.2)
   1: derived: six (>=1.5)
   1: selecting six (1.16.0)
   1: selecting xmltodict (0.13.0)
   1: fact: cryptography (40.0.2) depends on cffi (>=1.12)
   1: selecting cryptography (40.0.2)
   1: derived: cffi (>=1.12)
   1: fact: cffi (1.15.1) depends on pycparser (*)
   1: selecting cffi (1.15.1)
   1: derived: pycparser
   1: selecting pycparser (2.21)
   1: selecting certifi (2023.5.7)
   1: selecting types-pyyaml (6.0.12.9)
   1: fact: werkzeug (2.3.4) depends on MarkupSafe (>=2.1.1)
   1: selecting werkzeug (2.3.4)
   1: fact: boto3 (1.26.135) depends on botocore (>=1.29.135,<1.30.0)
   1: fact: boto3 (1.26.135) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: boto3 (1.26.135) depends on s3transfer (>=0.6.0,<0.7.0)
   1: selecting boto3 (1.26.135)
   1: derived: s3transfer (>=0.6.0,<0.7.0)
   1: derived: jmespath (>=0.7.1,<2.0.0)
   1: derived: botocore (>=1.29.135,<1.30.0)
   1: fact: botocore (1.29.135) depends on jmespath (>=0.7.1,<2.0.0)
   1: fact: botocore (1.29.135) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.29.135) depends on urllib3 (>=1.25.4,<1.27)
   1: selecting botocore (1.29.135)
   1: fact: s3transfer (0.6.1) depends on botocore (>=1.12.36,<2.0a.0)
   1: selecting s3transfer (0.6.1)
   1: selecting jmespath (1.0.1)
   1: Version solving took 2.200 seconds.
   1: Tried 1 solutions.

Resolution results:

six                1.16.0  
jmespath           1.0.1   
python-dateutil    2.8.2   
urllib3            1.26.15 
botocore           1.29.135
certifi            2023.5.7
charset-normalizer 3.1.0   
idna               3.4     
pycparser          2.21    
cffi               1.15.1  
markupsafe         2.1.2   
pyyaml             6.0     
requests           2.30.0  
s3transfer         0.6.1   
types-pyyaml       6.0.12.9
boto3              1.26.135
cryptography       40.0.2  
jinja2             3.1.2   
responses          0.23.1  
werkzeug           2.3.4   
xmltodict          0.13.0  
moto               4.1.9