Improve the situation for no-name-in-module for C extensions and modules using dynamic features

[chrisspen]

I have the ProxyTypes package installed and have a file mymodule.py that imports it like:

from peak.util.proxies import CallbackProxy

However, pylint throws this false error for it:

E:  8, 0: No name 'util' in module 'mymodule.peak' (no-name-in-module)

Pylint should reference the peak package installed into my virtualenv, and not assume it's a local non-existent module.

[PCManticore]

Please provide a minimal working example, I'm not sure I completely understand what is happening, especially since the error you posted indicate something more complex going on than the first import.

[chrisspen]

You can easily reproduce it with the following procedure:

$ mkdir test
$ cd test
$ virtualenv .env
Running virtualenv with interpreter /usr/bin/python2
New python executable in test/.env/bin/python2
Also creating executable in test/.env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
$ .env/bin/pip install pylint ProxyTypes
Collecting pylint
  Using cached pylint-1.7.1-py2.py3-none-any.whl
Collecting ProxyTypes
  Downloading ProxyTypes-0.9.zip
Collecting six (from pylint)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting mccabe (from pylint)
  Using cached mccabe-0.6.1-py2.py3-none-any.whl
Collecting singledispatch; python_version < "3.4" (from pylint)
  Using cached singledispatch-3.4.0.3-py2.py3-none-any.whl
Collecting isort>=4.2.5 (from pylint)
  Using cached isort-4.2.15-py2.py3-none-any.whl
Collecting astroid>=1.5.1 (from pylint)
  Using cached astroid-1.5.3-py2.py3-none-any.whl
Collecting configparser; python_version == "2.7" (from pylint)
Collecting backports.functools-lru-cache; python_version == "2.7" (from pylint)
  Using cached backports.functools_lru_cache-1.4-py2.py3-none-any.whl
Collecting enum34>=1.1.3; python_version < "3.4" (from astroid>=1.5.1->pylint)
  Using cached enum34-1.1.6-py2-none-any.whl
Collecting lazy-object-proxy (from astroid>=1.5.1->pylint)
  Using cached lazy_object_proxy-1.3.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting wrapt (from astroid>=1.5.1->pylint)
Building wheels for collected packages: ProxyTypes
  Running setup.py bdist_wheel for ProxyTypes ... done
  Stored in directory: /home/chris/.cache/pip/wheels/e3/8f/4c/5d7b9f8e251cc9567b730e7f9e2f74a9ea36a4d9189e0d2864
Successfully built ProxyTypes
Installing collected packages: six, mccabe, singledispatch, isort, enum34, lazy-object-proxy, wrapt, backports.functools-lru-cache, astroid, configparser, pylint, ProxyTypes
Successfully installed ProxyTypes-0.9 astroid-1.5.3 backports.functools-lru-cache-1.4 configparser-3.5.0 enum34-1.1.6 isort-4.2.15 lazy-object-proxy-1.3.1 mccabe-0.6.1 pylint-1.7.1 singledispatch-3.4.0.3 six-1.10.0 wrapt-1.10.10
$ echo -e "from peak.util.proxies import CallbackProxy\n" > mymodule.py
$ ls -lah
total 16K
drwxrwxr-x  3 chris chris 4.0K Jun 20 16:54 .
drwxr-xr-x 11 chris chris 4.0K Jun 20 16:52 ..
drwxrwxr-x  7 chris chris 4.0K Jun 20 16:53 .env
-rw-rw-r--  1 chris chris   45 Jun 20 16:54 mymodule.py
$ .env/bin/pylint mymodule.py
No config file found, using default configuration
************* Module mymodule
C:  2, 0: Trailing newlines (trailing-newlines)
C:  1, 0: Missing module docstring (missing-docstring)
E:  1, 0: No name 'util' in module 'mymodule.peak' (no-name-in-module)
E:  1, 0: Unable to import 'peak.util.proxies' (import-error)
W:  1, 0: Unused CallbackProxy imported from peak.util.proxies (unused-import)

------------------------------------------------------------------------
Your code has been rated at -120.00/10 (previous run: -120.00/10, +0.00)


$ .env/bin/python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from peak.util.proxies import CallbackProxy
>>> 

[meniluca]

I see the same behaviour with another library. Any idea how to fix this?

[andrewxiwu]

Same problem here. Any idea/reference to fix this issue? It is really annoying to have these false positives.

[neocogent]

Same problem here with gevent.ssl.
pylint says SSLContext and PROTOCOL_SSLv23 don't exist but they clearly do.

Python 2.7.12. gevent 1.2.2

python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gevent.ssl import SSLContext, PROTOCOL_SSLv23
>>> 

And another very common one also reported as non-existent, in this case as no-member,

[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb as db
>>> db.Error
<class '_mysql_exceptions.Error'>
>>> 

Have to disable no-name-in-module for now.

Maybe it's related error but I also get a warning on this,
from backports.functools_lru_cache import lru_cache
It says 'relative-import' when it's clearly not.

Incidentally I am not using virtualenv here. This is system wide python.

[soundlake]

It seems duplicated. #1138

[soundlake]

@neocogent --unsafe-load-any-extension=y worked for me.

[rawrgulmuffins]

I'm running into a similar issue at my work with a pure python sub-package. --unsafe-load-any-extension has no effect in this case.

[jcezarms]

I'm an year late, but still seeing the exact same behaviour with tensorflow.python.lib.io, even though both the library and its __init__.py file exist.

As I'm using vscode, this issue causes workspace errors to render in a misleading and incorrect way. To quickly reproduce, install tensorflow and type from tensorflow.python.lib.io import file_io in a newly created file:

Running pylint v1.9.2 from the command line on the above file points the same

$ pylint some.py
No config file found, using default configuration
************* Module some
C:  1, 0: Final newline missing (missing-final-newline)
C:  1, 0: Missing module docstring (missing-docstring)
E:  1, 0: No name 'python' in module 'tensorflow' (no-name-in-module)
W:  1, 0: Unused file_io imported from tensorflow.python.lib.io (unused-import)

-----------------------------------------------------------------------
Your code has been rated at -70.00/10 (previous run: -40.00/10, -30.00)

For these specs

$ pylint --version
No config file found, using default configuration
pylint 1.9.2,
astroid 1.6.5
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)]

@PCManticore I'd consider the above a minimal working example for this issue, can you review it to maybe consider reopening this one?

[brycepg]

• peak issue is due to How should I correctly lint an implicit namespace package? #1361 I think
• gevent.ssl is doing copy_globals(_source, globals()) (wontfix / maybe astroid brain fix)
• tensorflow seems to be a valid issue

[rafaelmendy]

Same problem in importing PyQt5 libraries.

The code works but pylint send errors.

[eirikgje]

Another datapoint here:

from netCDF4 import Dataset
leads to
[pylint] E0611: No name 'Dataset' in module 'netCDF4' (no-name-in-module)

[PCManticore]

Folks, before commenting here, do make sure to do the following:

• try running pylint first with pylint --extension-pkg-whitelist=<the package that gives you trouble>. e.g pylint --extension-pkg-whitelist=netCDF4 <your files> This way we can make sure it's not dealing with a C extension package that pylint cannot understand
• if that didn't work, it's most likely something related to how that package is structured, in which case we'll need separate issues per each package, as each own can have a different particularity (e.g some of them are namespace packages, some of them uses globals(), others import from an inner C extension etc). Creating a separate issue in this case means we don't conflate a problem in a package with another one for this issue.

[rawrgulmuffins]

Tested --extension-pkg-whistlist and in my case and it did not solve the problem.

I'll open an issue for mine but it's unfortunately not an open source project causing this issue so I won't be able to point to a specific project link. =(

[gaziqbal]

Is there any guidance on how to bypass the tensorflow.python issue mentioned by @jcezarms above? i'd like to avoid disabling the no-name-in-module error just to get past this false positive.

[chm123]

Is there any guidance on how to bypass the tensorflow.python issue mentioned by @jcezarms above? i'd like to avoid disabling the no-name-in-module error just to get past this false positive.

In VSCode, I had to set

    "python.linting.pylintArgs": [
        "--ignored-modules=tensorflow",
        "--extension-pkg-whitelist=tensorflow"
    ],

to get rid of the false positives.

[danielbraun89]

any updates?

[ynarwal]

It happens with specific names of python modules, not sure why though.
I renamed module factories to model_factories warning went away.

[ghost]

also happens with

from win32api import GetShortPathName

which throws
No name 'GetShortPathName' in module 'win32api' pylint(no-name-in-module)

GetShortPathName is called later in the code and does work correctly.

[PCManticore]

This error usually happens because the given module is either a C extension, so a static analysis tool can't read that source code to figure out the members, or the module tries to be too smart by using dynamic features such as globals().update to update the global of one or more files. pylint can't really help with these kind of modules, and there are some workarounds that you can use:

• use --ignored-modules= to ignore the module that is giving you the error
• use --generated-members to specify runtime specific attributes
• use --extension-pkg-whitelist= to let pylint know that it should import a C extension to build the AST from the live module

[micklat]

A suggestion to others who encounter this: ensure that your LD_LIBRARY_PATH (or similar) contains paths to all the dependencies of the module you're trying to import. I had failures like the above when importing a C extension that depended on other extensions, and fixing LD_LIBRARY_PATH apparently made the problem go away.

[kevlarr]

I'm running into this issue (and several others) but only on our build machine. Both use pylint 2.4.4.

Given...

from pydantic import BaseModel

Locally (MacOS Mojave 10.14.6) I see no errors.

However, on our build machine (Ubuntu 18.04.1 LTS) I see:

E0611: No name 'BaseModel' in module 'pydantic' (no-name-in-module)

[djelekar]

from pydantic import Json

E0611: No name 'Json' in module 'pydantic' (no-name-in-module)

pylint=2.5.3 pydantic=1.6.1

The same thing happens here. Works on Windows, while fails on Ubuntu. What can we do about it?

[lesinigo]

Same happens with setproctitle, at least since dvarrazzo/py-setproctitle#31 (early 2014).

AFAIK it is reproducible on every kind of system, but if you want a simple test scenario you could get one with this:

$ sudo docker run -ti --rm alpine:3.12
... here we are inside the container ...
/ # apk add py3-pip py3-setproctitle
... lots of packages installed ...
/ # pip install pylint
... pylint installed (it's currently not included in Alpine 3.12 packages) ...
/ # pylint --version
pylint 2.5.3
astroid 2.4.2
Python 3.8.5 (default, Jul 20 2020, 23:11:29)
/ # pip list -v | grep setproctitle
setproctitle      1.1.10     /usr/lib/python3.8/site-packages

/ # echo 'from setproctitle import setproctitle' > example.py
/ # pylint -E example.py
************* Module example
/example.py:1:0: E0611: No name 'setproctitle' in module 'setproctitle' (no-name-in-module)

Installing setproctitle through pip instead of apk does not make any difference:

$ sudo docker run -ti --rm alpine:3.12
/ # apk add gcc musl-dev python3-dev py3-pip
/ # pip install pylint setproctitle
/ # pip list -v | grep setproctitle
setproctitle      1.1.10     /usr/lib/python3.8/site-packages
/ # pylint --version
pylint 2.5.3
astroid 2.4.2
Python 3.8.5 (default, Jul 20 2020, 23:11:29)

/ # echo 'from setproctitle import setproctitle' > example.py
/ # pylint -E example.py
************* Module example
/example.py:1:0: E0611: No name 'setproctitle' in module 'setproctitle' (no-name-in-module)

[craigtrim]

@djelekar same situation. If you discover a better solution than just putting
# pylint: disable=no-name-in-module
on each file with this import, I'd like to hear it!

[amandalishus]

This problem also occurs when using Python with Apache Spark via the pyspark module, at least when using pip install pyspark==2.4.6. When we started this project using an earlier version of the module the problem did not exist, but we began seeing it after updating to the latest version to stay in sync with AWS Glue.

To reproduce, just install the module as above and import any functions from pyspark.sql.functions. For instance:
from pyspark.sql.functions import col, lit, size

[dplepage]

from pydantic import Json

E0611: No name 'Json' in module 'pydantic' (no-name-in-module)

pylint=2.5.3 pydantic=1.6.1

The same thing happens here. Works on Windows, while fails on Ubuntu. What can we do about it?

This fails on OSX, as well. If you look at where pydantic is installed, you'll see that it precompiled itself, so you're actually importing .../site_packages/pydantic/__init__.cpython-39-darwin.so (or some other extension, depending on your platform and python version). I assume some of the other modules mentioned here have the same problem.

Is there a way for pylint to detect the __init__.py etc.? In my dev environment I dealt with by just deleting all the .so files pydantic installed.

[adam-grant-hendry]

Adding my two cents here:

From the pylint 2.15.0-a0 docs on the no-member / E1101:

Linting C extension modules is not supported out of the box, especially since pylint has no way to get an AST object out of the extension module.

...Pylint has a flag, called extension-pkg-allow-list (formerly extension-pkg-whitelist), through which you can tell it to import that module and to build an AST from that imported module
...
Be aware though that using this flag means that extensions are loaded into the active Python interpreter and may run arbitrary code, which you may not want. This is the reason why we disable by default loading C extensions. In case you do not want the hassle of passing C extensions module with this flag all the time, you can enable unsafe-load-any-extension in your configuration file, which will build AST objects from all the C extensions that pylint encounters

So it seems extension-pkg-allow-list was made precisely to handle this problem.

Note that, if you are using static type checking, you can still create .pyi stubs for static analysis. (If you are making a package, be sure to include the requisite py.typed marker (file) to the root of the package (per PEP 561).)

Interestingly, when extension-pkg-allow-list isn't set, if stubs for your C extension module use wildcard imports (i.e. from package import *), you will see a no-name-in-module / E0611error, which is created by the pylint variables checker rather than a c-extension-no-member / I1101, which is created by the pylint typechecker checker.

Although mypy allows using wildcard imports to export modules, this last one may surprise you. I've seen this happen occasionally when the name you import has been imported in the target module with a wildcard. For instance, the python extension of the VTK library ships with generate_pyi.py, which will make stubs for you, but they use wildcard imports.

Rest assured, your extension modules are typed (if they have stubs! 😄), but just be sure to extension-pkg-allow-list (and only set it for packages you trust!).

[jace]

So it seems extension-pkg-allow-list was made precisely to handle this problem.

Note that, if you are using static type checking, you can still create .pyi stubs for static analysis. (If you are making a package, be sure to include the requisite py.typed marker (file) to the root of the package (per PEP 561).)

How do you get pylint to read a .pyi stub file instead of the regular .py file? I'm using mkinit to create a lazy-loading __init__.py with a corresponding __init__.pyi stub file. Type checkers are fine with this, but Pylint is not. I have new false positives for isinstance-second-argument-not-valid-type, no-name-in-module and not-callable.

[jacobtylerwalls]

How do you get pylint to read a .pyi stub file instead of the regular .py file?

Support is pending in pylint 3.1, see pylint-dev/astroid#2375

[jacobtylerwalls]

Thanks @adam-grant-hendry for the write up! Closing as resolved. pyi support tracked separately.