[2.12.x] false positives: no-name-in-module, import-error

[paride]

Bug description

Pylint 2.12.x emits no-name-in-module and import-error false positives with Python 3.6. This doesn't happen with pylint 2.11.x, or with newer Python versions.

Configuration

No response

Command used

The issue is trivial to reproduce with a 1-line Python file:

$ cat test.py
import distutils.util

$ python3 test.py
$

$ python3 -m pylint test.py

Pylint output

$ python3 -m pylint test.py
************* Module test
test.py:1:0: C0114: Missing module docstring (missing-module-docstring)
test.py:1:0: E0611: No name 'util' in module 'distutils' (no-name-in-module)
test.py:1:0: E0401: Unable to import 'distutils.util' (import-error)
test.py:1:0: W0611: Unused import distutils.util (unused-import)

Expected behavior

No E tags.

Pylint version

$ python3 -m pylint --version
pylint 2.12.2
astroid 2.9.2
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0]

OS / Environment

Ubuntu 18.04.

Additional dependencies

No response

[Pierre-Sassoulas]

Thank you for opening the issue. Sorry but we're not going to fix this unless someone open a MR fixing it before pylint 2.13.0 is released as pylint 2.13.0 is the latest version supporting python 3.6 and as python 3.6 is past end of life now.

[DanielNoord]

@Pierre-Sassoulas Didn't we add a test specifically for this 11eba03 as an regression test against changes in astroid 2.9.2?

@paride Just to be sure, could you try this while pinning astroid to 2.9.0?

[Pierre-Sassoulas]

I thougth so but the version reported is:

pylint 2.12.2
astroid 2.9.2

As we did not change anything in pylint and the astroid used is the latest I don't think the bug is gone away.

[paride]

@DanielNoord no false positive when pinning astroid==2.9.0.

[DanielNoord]

Note that DanielNoord#74 does not report a false positive here...

I'm not sure what is going wrong here. It probably has to do with recent changes in astroid but apparently it does not fall on all setups. @paride Are you using a venv or anything?

[paride]

@DanielNoord now I'm a bit confused: I can reproduce the issue in a virtual environment created using virtualenv(1), but not in a venv.

venv:

jenkins@torkoal:~$ python3 -m venv /tmp/venv
jenkins@torkoal:~$ source /tmp/venv/bin/activate
(venv) jenkins@torkoal:~$ which pip3
/tmp/venv/bin/pip3
(venv) jenkins@torkoal:~$ pip3 install pylint
[...]

(venv) jenkins@torkoal:~$ python3 -m pylint --version
pylint 2.12.2
astroid 2.9.3
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0]
(venv) jenkins@torkoal:~$ python3 -m pylint -E /tmp/test.py

virtualenv:

jenkins@torkoal:~$ virtualenv -p python3 /tmp/virtualenv
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/virtualenv/bin/python3
Also creating executable in /tmp/virtualenv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
jenkins@torkoal:~$ source /tmp/virtualenv/bin/activate
(virtualenv) jenkins@torkoal:~$ pip3 install -q pylint
(virtualenv) jenkins@torkoal:~$ python3 -m pylint --version
pylint 2.12.2
astroid 2.9.3
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0]

(virtualenv) jenkins@torkoal:~$ python3 -m pylint -E /tmp/test.py
************* Module test
/tmp/test.py:1:0: E0611: No name 'util' in module 'distutils' (no-name-in-module)
/tmp/test.py:1:0: E0401: Unable to import 'distutils.util' (import-error)

[DanielNoord]

Hmm this has something to do with virtualenv and they way they patch distutils. See changes in pylint-dev/astroid#1321

Could you check if the issues also exists in 3.7? I'm having trouble compiling 3.6 and 3.7 locally for some weird reason..

[paride]

Also happening with Python 3.8 (from the python3.8 Ubuntu Bionic package):

virtualenv

(p38virtualenv) jenkins@torkoal:~$ python3 -m pylint --version
pylint 2.12.2
astroid 2.9.3
Python 3.8.0 (default, Dec  9 2021, 17:53:27) 
[GCC 8.4.0]

(p38virtualenv) jenkins@torkoal:~$ python3 -m pylint -E /tmp/test.py
************* Module test
/tmp/test.py:1:0: E0611: No name 'util' in module 'distutils' (no-name-in-module)
/tmp/test.py:1:0: E0401: Unable to import 'distutils.util' (import-error)

It works in a venv.

[DanielNoord]

Ah that's too bad. I hoped it would be resolved. Looks like we're going to have to revert pylint-dev/astroid#1321 then, or at least try and come up with a solution.

I hope to get to this somewhere this week. I'd like to fix this issue for real this time 😄

[paride]

@DanielNoord thanks! If you need testing/verification do not hesitate to ping me.

[DanielNoord]

@paride One follow-up question. Which version of virtualenv are you using?

I can't reproduce easily (for now..), using:

❯ cd /tmp
❯ echo "import distutils.util" > test.py
❯ virtualenv -p python3 /tmp/virtualenv
/usr/local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
created virtual environment CPython3.9.9.final.0-64 in 526ms
  creator CPython3Posix(dest=/private/tmp/virtualenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/daniel/Library/Application Support/virtualenv)
    added seed packages: pip==21.3.1, setuptools==59.6.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
❯ source /tmp/virtualenv/bin/activate
❯ pip3 install pylint
Collecting pylint
  Using cached pylint-2.12.2-py3-none-any.whl (414 kB)
Collecting toml>=0.9.2
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting platformdirs>=2.2.0
  Using cached platformdirs-2.4.1-py3-none-any.whl (14 kB)
Collecting typing-extensions>=3.10.0
  Using cached typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Collecting isort<6,>=4.2.5
  Using cached isort-5.10.1-py3-none-any.whl (103 kB)
Collecting mccabe<0.7,>=0.6
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Collecting astroid<2.10,>=2.9.0
  Using cached astroid-2.9.3-py3-none-any.whl (254 kB)
Collecting wrapt<1.14,>=1.11
  Using cached wrapt-1.13.3-cp39-cp39-macosx_10_9_x86_64.whl (33 kB)
Collecting lazy-object-proxy>=1.4.0
  Using cached lazy_object_proxy-1.7.1-cp39-cp39-macosx_10_9_x86_64.whl (22 kB)
Requirement already satisfied: setuptools>=20.0 in ./virtualenv/lib/python3.9/site-packages (from astroid<2.10,>=2.9.0->pylint) (59.6.0)
Installing collected packages: wrapt, typing-extensions, lazy-object-proxy, toml, platformdirs, mccabe, isort, astroid, pylint
Successfully installed astroid-2.9.3 isort-5.10.1 lazy-object-proxy-1.7.1 mccabe-0.6.1 platformdirs-2.4.1 pylint-2.12.2 toml-0.10.2 typing-extensions-4.0.1 wrapt-1.13.3
❯ python3 -m pylint test.py
************* Module test
test.py:1:0: C0114: Missing module docstring (missing-module-docstring)
test.py:1:0: W0611: Unused import distutils.util (unused-import)

----------------------------------------------------------------------
Your code has been rated at -10.00/10 (previous run: -10.00/10, +0.00)

❯ python3 -m pylint --version
pylint 2.12.2
astroid 2.9.3
Python 3.9.9 (main, Nov 21 2021, 03:23:44)
[Clang 13.0.0 (clang-1300.0.29.3)]
❯ virtualenv --version
/usr/local/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
virtualenv 20.7.2 from /usr/local/lib/python3.9/site-packages/virtualenv/__init__.py

[paride]

@DanielNoord an older one:

$ virtualenv --version
15.1.0

[DanielNoord]

Ah that might explain it. I know that something changed in how virtualenv patches distutils, although I'm not sure when that change occurred. Let me retry!

[DanielNoord]

I'm running into issues getting virtualenv==15.1.0 to work on my Mac with a homebrew installation of python.
I guess that's one of the things that was fixed in 20.0.0.

I would really like for this to be solved without adding a new dependency to distutils to astroid... We could also pin virtualenv to 20+ which is what tox seems to be doing. If anybody can get virtualenv to work it would be good to know what list(distutils.__path__) is in this commit https://github.com/PyCQA/astroid/pull/1321/files. We thought it would be the same as [spec.location], but clearly it is not..

[DanielNoord]

@Pierre-Sassoulas Would you be willing to take a look at this? To recap, pylint-dev/astroid#1321 is incompatible with virtualenv==15.1.0 and I believe everything under 20.0.0. We though removing those lines on L169 was possible since list(distutils.__path__) seemed similar to [spec.location] due to the check on L166. Apparently it is not.

It would be very helpful if you could investigate what is different about the two in a virtualenv on version 15.1.0. We can revert the change there, but I really want to try and see if we can fix this without it. I can't do this myself as virtualenv < 20.0 seems to broken on Mac OS both for homebrew and the pre-installed versions of Python.

To reproduce this you can use the following bash commands:

cd /tmp
echo "import distutils.util" > test.py
python3 -m pip install virtualenv==15.1
python3 -m virtualenv -p python3 /tmp/virtualenv
source /tmp/virtualenv/bin/activate
# Just to make sure we're using the correct versions
# Check if python3 here refers to the virtualenv python executable
python3 -m pylint --version
virtualenv --version
python3 -m pip3 install pylint
python3 -m pylint test.py

This should report no-name-in-module. If you then install an editable installation of astroid in the virtualenv it should be relatively straightforward to add a breakpoint() + print() statement at the affected lines.

[Pierre-Sassoulas]

Sorry I did not find the time.

To recap, pylint-dev/astroid#1321 is incompatible with virtualenv==15.1.0 and I believe everything under 20.0.0.

It seems it's affecting very recent virtualenv version then. Should we fix it ? Or is it also only affecting python 3.6 that is past end of life ? We have so much to do that if it's affecting python 3.6 only I don't think it's a wise allocation of our time to fix it.

[DanielNoord]

It's affecting all versions of Python with those older versions of setuptools. The fix is fairly straightforward as we only need to revert that commit, but it would be very helpful if we could get someone to do my steps from 2 comments ago so that we can actually see what that to-be-reverted-commit does. It would be much better if we could come up with a way to replicate it without importing distutils itself.

[DanielNoord]

@paride Sorry for letting this rest so long. I finally got "Windows" to work via Wine on my Mac and I could do some very bare-bones testing.

However, when I revert the commit in pylint-dev/astroid#1321 I actually keep getting the import-error.
I used the same instructions as here #5645 (comment) and then cd into the virtualenv folder to change the relevant file in astroid in site-packages. Could I ask you to check if reverting this commit in astroid does fix the issue for you?
Perhaps the distutils patching virtualenv does is only done on Linux, but I was quite surprised to see that reverting this did not fix the issue immediately on Windows.

[DanielNoord]

I actually got my hands on a windows and can't make the issue go away, even on pylint 2.10... I think util might actually be undefined in distutils in a virtualenv on Windows?

[paride]

@DanielNoord sorry for not getting back to you earlier. Tomorrow I'll try reverting the commit you mentioned on my system and see if the issue goes away, unless you say that's not useful info anymore at this point.

[DanielNoord]

@paride No worries! That would actually be quite useful as my recent dabbling with Windows makes me wonder if it's actually that commit that is causing these issues..

[jacobtylerwalls]

@DanielNoord I saw your call for reinforcements

I just tried reproducing on windows and couldn't. Do I need to pin to a specific version of setuptools? Any tips you have for reproducing would be helpful. Using GitHub Actions is a lot easier than an emulator IMO! :D

EDIT: maybe the test file needs to be in the venv?

[DanielNoord]

No only the virtualenv version seems to matter (it should be <20).

I could use Github Actions a little smarter indeed, but it is difficult to git bisect a change with it though.

[jacobtylerwalls]

Reproduced it now. (There was a flaw with my github actions recipe.)

I can reproduce even on astroid==2.9.0 with Python 3.8, virtualenv==15.1.0 and windows-latest:

$ cat test.py
import distutils.util  # pylint: disable=unused-import
$ pylint --version
pylint 2.12.2
astroid 2.9.0
Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
$ pylint test.py
************* Module test
test.py:1:0: E0611: No name 'util' in module 'distutils' (no-name-in-module)
test.py:1:0: E0401: Unable to import 'distutils.util' (import-error)

I think it's a fool's errand to spend more time trying to bisect: we're talking a major matrix of python versions, platforms, pylint/astroid commits (possibly incompatible with each other), and virtualenv versions.

I'll open a draft PR to discuss an approach for special-casing this without importing distutils again.

---

if we could get someone to do my steps from 2 comments ago so that we can actually see what that to-be-reverted-commit does

File "d:\a\astroid\astroid\astroid\interpreter\_import\spec.py", line 166, in contribute_to_path
    assert False, (spec.name, spec.location, distutils.__path__)
AssertionError: ('distutils', 'D:\\a\\astroid\\astroid\\venv\\lib\\distutils', ['D:\\a\\astroid\\astroid\\venv\\lib\\site-packages\\setuptools\\_distutils'])

[DanielNoord]

Hmm that directly contrasts @paride's comment here: #5645 (comment).

For me the most important question is whether this regression was actually introduced in 2.9.x in astroid and therefore in pylint 2.12.x. That's why I would like to bisect and get the faulty commit.
If it was not introduced in 2.9.x this might just be a regular false positive that just coincidentally popped while we working on distutils changes. I would be fine with releasing A2.10 and P2.13 without it.
If it was introduced in 2.9.x we should see if we can get a fix in before P2.13 is released.

[jacobtylerwalls]

Sure thing. I'm just worried it will turn out to be a different faulty commit on various platforms and python versions.

Anyway, reproduced on astroid 2.8.1. (I'd have to keep doing this to try to get a "good" commit, and I'd have to rollback python and pylint versions, probably.)

$ pylint --version
pylint 2.12.2
astroid 2.8.2-dev0
Python 3.8.10 (tags/v3.8.10:3d[89]
$ pylint test.py
************* Module test
test.py:1:0: E0611: No name 'util' in module 'distutils' (no-name-in-module)
test.py:1:0: E0401: Unable to import 'distutils.util' (import-error)

[DanielNoord]

Hmm, do you know what version of setuptools was present in those two tests?

And @paride do you know which version of setuptools you used?

[jacobtylerwalls]

setuptools=60.8.2

The monkeypatching started in >60, based on discussion

[DanielNoord]

Thanks to the great work from @jacobtylerwalls and reviewers this has been fixed in astroid 2.10 with pylint-dev/astroid#1386. Expect the fix to be available in pylint 2.13. 😄

[DanielNoord]

@paride Just because I'm curious: we just released 2.13. Can you confirm this has been fixed? 😄

[paride]

@DanielNoord I can't reproduce the issue with 2.13.0, so looks like it is. Thanks!

[MrPointer]

Sorry for bumping a closed issue, but I got this error again, and it seems like a false positive 😕

For this piece of code:

from pydantic import BaseModel, DirectoryPath


class LaunchEnvironment(BaseModel):
    working_dir: DirectoryPath


class LocalLaunchEnvironment(LaunchEnvironment):
    pass

pylint fails for all imported pydantic modules, although they're installed properly and are available for the used interpreter.

I'm using poetry to manage the project and its associated venv, so both pylint and pydantic are installed side by side in the same venv, hence using the same interpreter.
The venv's python version is 3.10.8.

Here's my pyproject.toml:

[tool.poetry]
BLA-BLA-BLA

[tool.poetry.scripts]
BLA-BLA-BLA

[tool.poetry.dependencies]
python = "^3.8"
rich = "^12.6.0"
typer = {extras = ["all"], version = "^0.6.1"}
emoji = "^2.1.0"
pydantic = "^1.10.2"

[tool.poetry.group.dev.dependencies]
pylint = "^2.15.3"
black = "^22.10.0"
pytest = "^7.1.3"

[tool.black]
line-length = 120

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I've intentionally hidden any details that are internal to the company I'm working at, please excuse me for that. Nothing meaningful there anyway.

[Pierre-Sassoulas]

Hey @MrPointer, your problem is harder to fix for us as pydantic is doing funky things under the hood (creating attribute of function dynamically, most likely), so pylint don't understand it. This issue was reproducible with a small example. You can search for issue related to pydantic in our issue tracker there's probably already one that is still opened.

[MrPointer]

@Pierre-Sassoulas Thanks for the quick reply, got ya.
I think I'll just disable any pydantic-specific checks for now, hopefully you could get around it one day! 🙏🏻

[cdce8p]

@MrPointer Found this in a pydantic issue: pydantic/pydantic#1961 (comment). Haven't tested it myself though.

[Main]
extension-pkg-whitelist=pydantic

There is also #1524 which might be helpful.

[MrPointer]

@cdce8p Sorry for replying so late, I began replying from my phone and forgot to finish it 😀
I ended up doing just that! Seems like a fair enough compromise for me, honestly.