No module named 'pkg_resources' during sdist build of dependency when mixing Poetry 1.2 and 1.1

[elliotdes]

• [x ] I am on the latest Poetry version.
• [ x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Monterey 12.3, M1
• Poetry version: 1.2.0

Issue

I updated my personal machine to 1.2.0. My CI/CD pipeline still uses 1.13.0 and is running on a python:3.10-slim docker container.

I ran poetry update dparse on my local machine, with version 1.2.0. When running tmy CI/CD (Poetry 1.13.0), poetry run bandit application/ -r -c "pyproject.toml" fails citing:

ModuleNotFoundError: No module named 'pkg_resources'

However, after downgrading my personal machine to poetry 1.13.0 and running poetry update dparse(after reverting the previous changes), the CI/CD passes without error.

Any reasons for this behaviour?

[neersighted]

This is impossible to reproduce without a pyproject.toml, and preferably a container-based reproduction. However, this sounds like your project depended on pkg_resources/setuptools and was relying on 1.1's behavior of implicitly including setuptools.

[roniemartinez]

We had the same issue but it's always the same poetry version in Dockerfile (image is python:3.10).

was relying on 1.1's behavior of implicitly including setuptools

I don't think this behavior was changed or even documented somewhere in 1.2 release. no-setuptools was introduced but it means disabling it so I think this behavior is still "implicit" in 1.2: https://python-poetry.org/blog/announcing-poetry-1.2.0/#virtualenvsoptionsno-setuptools

The weird thing that we observed is changing pip to pip3 when doing pip install -U poetry setuptools fixed it for us. We also think this could be related to Python 3.10.6 image update.

[roniemartinez]

UPDATE: Downgrading python from python:3.10 (which installs 3.10.6) to python:3.10.5 also fixed it for us. Poetry 1.2 version probably has issues with Python 3.10.6

[neersighted]

There's still no reproduction here, and the additional context makes this more confusing...

When I refer to implicit inclusion of setuptools, I mean Poetry would basically ignore it in the environment unless it was explicitly configured in pyproject.toml, whereas 1.2 will record it to the lockfile/not special case it. It's a small line item in the changelog that doesn't affect most projects with well-defined dependencies, but some may get tripped up by implicit coupling they did not realize was there.

Anyway, a Dockerfile or reproduction log is still needed to debug this -- if anyone turns one up please post it!

[mfisco]

I also encountered this after upgrading to 1.2.0 on both my development and production machines. For me, simply running poetry add setuptools fixed the issue - but I don't quite know why the exception wouldn't have been thrown prior to 1.2.0 if Poetry would "ignore it in the environment".

Dependencies in pyproject.toml

[tool.poetry.dependencies]
python = "~3.9"
django = "~4.0.7"
django-crispy-forms = "^1.14.0"
numpy = "^1.23.2"
pandas = "^1.4.4"
django-constance = "^2.9.1"
django-dbbackup = "^4.0.1"
django-extra-views = "^0.14.0"
django-extensions = "^3.1.3"
django-import-export = "^2.5.0"
django-anymail = {extras = ["mailgun"], version = "^8.6"}
django-filter = "^22.1"
django-widget-tweaks = "^1.4.12"
django-simple-history = "^3.0.0"
django-tinymce = "^3.5.0"
django-notifications-hq = "^1.6.0"
django-storages = {extras = ["boto3"], version = "^1.11.1"}
django-picklefield = "^3.1"
weasyprint = "^56.1"
mysqlclient = "^2.1.1"
django-htmx = "^1.12.2"
toml = "^0.10.2"
django-environ = "^0.9.0"
django-entangled = "^0.5.1"
django-admin-rangefilter = "^0.8.8"
setuptools = "^65.3.0"                        # after adding this both machines worked as they did before

[tool.poetry.group.dev.dependencies]
django-debug-toolbar = "^3.6.0"

[tool.poetry.group.test.dependencies]
freezegun = "^1.2.2"
coverage = "^6.4.4"
isort = "^5.9.3"
black = "^22.8.0"
pre-commit = "^2.14.0"
tblib = "^1.7.0"
pytest = "^7.1.3"
pytest-django = "^4.4.0"
pytest-xdist = {extras = ["psutil"], version = "^2.4.0"}
flake8 = "^5.0.4"
djlint = "^1.12.1"

• development machine

  • Poetry: 1.2.0 (previously 1.0.7)
  • Python: 3.9.5
  • OS: macOS Monterey 12.5 | Macbook Pro (16-inch, 2019)

• production machine

  • Poetry: 1.2.0 (previously 1.12.0)
  • Python: 3.9.5
  • OS: Linux - Ubuntu 20.04.2 LTS

[neersighted]

I'm simplifying, but essentially what happens on 1.1 is Poetry will ignore setuptools unless it is in the lockfile. If setuptools is required by a dev-dependency (tox is the main offender/reason for this to happen, incidentally), but not a regular dependency, and is in the lockfile (as written by 1.2), 1.1 will remove setuptools unless installed with --dev. There are many other edge cases you could construct around this, but hopefully that explains a little bit of how this happens.

[totlsota]

any advice on how to deal with this presently? I am on poetry 1.2.2 and having the same pkg_resources issue

it sounds like the issue in br3ndonland/inboard#55 where packages depending on pkg_resources can't locate it

[neersighted]

Declare a dependency on setuptools explicitly.

[totlsota]

Declare a dependency on setuptools explicitly.

@neersighted - thanks, I'd done that and the issue is still there

It looks like:

❯ poetry run python -m XXX
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "XXX/__main__.py", line 6, in <module>
    from web3 import Web3
  File "XXX/Library/Caches/pypoetry/virtualenvs/XXX-neLnbEMt-py3.10/lib/python3.10/site-packages/web3/__init__.py", line 1, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

Also of note, this is on a mac today, but the same issue occurs on my linux machine. There are no open issues to do with this in web3.

[totlsota]

There are at least 2 libraries that have the same error (been on an evil comment/uncomment dig) {web3 and eth-utils}.

As per this link, #6328 (comment), I've moved back from python 3.10.7 to python 3.10.5 and it is sorted, for now, after much weeping and gnashing of teeth!

Posting in case this helps any poor soul.

[linnik]

When setuptools dependency is declared, looks like version 1.1.14 has this error and 1.2.0 does not
I made a repo with Dockerfile to reproduce this issue: linnik/poetrybug
In my case I was using eht16/python-logstash-async which relies on alisaifee/limits with usage of pkg_resources module and setuptools as a dependency.

[dimbleby]

no reproduction, old, and almost certainly the fix was to declare an explicit dependency on setuptools all along

let's close

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.