Missing metadata causing dependency resolution slowdown

[l0b0]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the User Guide and the API reference (Not applicable)
• I've searched for previous similar issues and didn't find any solution

Describe the bug

From a Poetry issue:

Apparently the awscli packages are not published with metadata about its dependencies so poetry literally has to download each version to figure out what it depends on. This is amplified by the fact that there are hundreds of releases of awscli on PyPI. The resolution time is greatly reduced if you give poetry a starting point for awscli.

SDK version number N/A

Platform/OS/Hardware/Device
What are you running the cli on? Ubuntu 20.04

To Reproduce (observed behavior)

1. Add awscli = "*" to Poetry dependencies
2. Run poetry update
3. Wait for a so far undetermined amount of time (many minutes at least) for Poetry to resolve dependencies:

   1: fact: awscli (1.18.173) depends on botocore (1.19.13)
   1: fact: awscli (1.18.173) depends on docutils (>=0.10,<0.16)
   1: fact: awscli (1.18.173) depends on s3transfer (>=0.3.0,<0.4.0)
   1: fact: awscli (1.18.173) depends on PyYAML (>=3.10,<5.4)
   1: fact: awscli (1.18.173) depends on colorama (>=0.2.5,<0.4.4)
   1: fact: awscli (1.18.173) depends on rsa (>=3.1.2,<=4.5.0)
   1: derived: not awscli (1.18.173)
   […]
   1: fact: awscli (1.16.194) depends on botocore (1.12.184)
   1: fact: awscli (1.16.194) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.194) depends on docutils (>=0.10)
   1: fact: awscli (1.16.194) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.194) depends on PyYAML (>=3.10,<=5.1)
   1: fact: awscli (1.16.194) depends on s3transfer (>=0.2.0,<0.3.0)
   1: derived: not awscli (1.16.194)
   […]
   

Expected behavior
Dependencies should resolve in at most a couple seconds.

[kdaily]

Hi @l0b0, Thanks for the report. I'm following up on the issue at Poetry.

[kdaily]

Looks like to me that we would have to specify dependencies in a pyproject.toml file specifically for Poetry:

• https://python-poetry.org/docs/basic-usage/#specifying-dependencies
• https://python-poetry.org/docs/faq/#why-is-the-dependency-resolution-process-slow

[lorengordon]

@kdaily FWIW, I did some research into how to expose dependencies via the package metadata to the PyPi JSON API, and best I can tell, the metadata missing is due to using an old client to publish the package to PyPi. I can't find anything indicating it would be necessary to use pyproject.toml.

According to this issue/comment, the dependency info is stored in info.requires_dist, and if that is not present then most likely the client publishing the package is the problem.

❯ curl -s https://pypi.org/pypi/awscli/json | jq ".info.requires_dist"
null
❯ curl -s https://pypi.org/pypi/boto3/json | jq ".info.requires_dist"
null
❯ curl -s https://pypi.org/pypi/botocore/json | jq ".info.requires_dist"
null
❯ curl -s https://pypi.org/pypi/aws-lambda-powertools/json | jq ".info.requires_dist"
[
  "aws-xray-sdk (>=2.8.0,<3.0.0)",
  "fastjsonschema (>=2.14.5,<3.0.0)",
  "boto3 (>=1.12,<2.0)",
  "jmespath (>=0.10.0,<0.11.0)",
  "pydantic (>=1.8.2,<2.0.0); extra == \"pydantic\"",
  "email-validator; extra == \"pydantic\""
]

[nateprewitt]

Hey @l0b0, thanks for the deep dive! We are specifying this info in our setup.cfg and it's available in the wheels we distribute. I think you've nailed the issue though, the upload process appears to be uploading our sdist (.tar.gz) distribution first which for some reason isn't including this metadata. We'll either need to look at flipping the order of which artifact gets uploaded to PyPI first, or get the metadata included correctly in our sdist build process.

[dswalter]

@kdaily FWIW, I did some research into how to expose dependencies via the package metadata to the PyPi JSON API, and best I can tell, the metadata missing is due to using an old client to publish the package to PyPi. I can't find anything indicating it would be necessary to use pyproject.toml.

According to this issue/comment, the dependency info is stored in info.requires_dist, and if that is not present then most likely the client publishing the package is the problem.

❯ curl -s https://pypi.org/pypi/awscli/json | jq ".info.requires_dist"
null
❯ curl -s https://pypi.org/pypi/boto3/json | jq ".info.requires_dist"
null
❯ curl -s https://pypi.org/pypi/botocore/json | jq ".info.requires_dist"
null
❯ curl -s https://pypi.org/pypi/aws-lambda-powertools/json | jq ".info.requires_dist"
[
  "aws-xray-sdk (>=2.8.0,<3.0.0)",
  "fastjsonschema (>=2.14.5,<3.0.0)",
  "boto3 (>=1.12,<2.0)",
  "jmespath (>=0.10.0,<0.11.0)",
  "pydantic (>=1.8.2,<2.0.0); extra == \"pydantic\"",
  "email-validator; extra == \"pydantic\""
]

Bless you for finding this. I tried to wander down this path but didn't get to the definitive pointer you did. Great work!

[nateprewitt]

Good news! We've recently refactored a large portion of our release process which made getting this fix out possible. We've published today's releases using it and the metadata is now available for awscli, boto3, and botocore. s3transfer doesn't release on a regular cadence, but it will also see the updates automatically in its next release.

Please let us know if you're still seeing any issues, otherwise we'll plan to close this soon. Thanks!

[nateprewitt]

Resolving since the metadata has been available for the last week and seems to be working correctly. Please let us know if you encounter any more issues.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.