pylint 2.6.0->2.6.1-dev1 regression - raises running against [email protected] #3913

doublethefish · 2020-10-21T09:32:52Z

Steps to reproduce

This was identified by, and impacts, PR #3498 .

Running pylint against numpy via tox -e profile_against_external
Ignore the pylint reported errors, if you see any.
Notice the exception thrown AttributeError: 'FunctionDef' object has no attribute '_proxied'

Current behavior

An unexpected exception is thrown running the against numpy v1.18.3. See the full log output bellow.

Expected behavior

We expect the same behavior as running pylint 2.6.0 against [email protected] (numpy@master) where :

the run completes
you get a code quality of around 3.96.

pylint --version output

Latest failing version used was 3d712f2 Date: Thu Aug 6 10:43:25 2020 +0100

pylint 2.6.1-dev1
astroid 2.5.0
Python 3.9.0 (default, Oct  6 2020, 04:17:54) 
[Clang 12.0.0 (clang-1200.0.32.2)]

Passes:

pylint 2.6.0
astroid 2.4.2
Python 3.8.3 (v3.8.3:6f8c8320e9, May 13 2020, 16:29:34) 
[Clang 6.0 (clang-600.0.57)]

full log

ln contributing/pylint (profile/pylint_against_numpy)$ tox -e profile_against_external
GLOB sdist-make: contributing/pylint/setup.py
profile_against_external create: contributing/pylint/.tox/profile_against_external
profile_against_external installdeps: https://github.com/PyCQA/astroid/tarball/master#egg=astroid-master-2.0, gprof2dot, mccabe, pytest, pytest-profiling, pytest-xdist
profile_against_external inst: contributing/pylint/.tox/.tmp/package/1/pylint-2.6.1.dev1.zip
profile_against_external installed: apipkg==1.5,astroid @ https://github.com/PyCQA/astroid/tarball/master,attrs==20.2.0,execnet==1.7.1,gprof2dot==2019.11.30,iniconfig==1.1.1,isort==5.6.4,lazy-object-proxy==1.4.3,mccabe==0.6.1,packaging==20.4,pluggy==0.13.1,py==1.9.0,pylint @ file://contributing/pylint/.tox/.tmp/package/1/pylint-2.6.1.dev1.zip,pyparsing==2.4.7,pytest==6.1.1,pytest-forked==1.3.0,pytest-profiling==1.7.0,pytest-xdist==2.1.0,six==1.15.0,toml==0.10.1,wrapt==1.12.1
profile_against_external run-test-pre: PYTHONHASHSEED='2157072702'
profile_against_external run-test: commands[0] | python -Wi -m pytest --exitfirst --profile-svg contributing/pylint/tests/profile/test_profile_against_externals.py
============================================================================================================================= test session starts ==============================================================================================================================
platform darwin -- Python 3.9.0, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
cachedir: .tox/profile_against_external/.pytest_cache
rootdir: contributing/pylint, configfile: pytest.ini
plugins: xdist-2.1.0, profiling-1.7.0, forked-1.3.0
collected 1 item                                                                                                                                                                                                                                                               

../tests/profile/test_profile_against_externals.py F                                                                                                                                                                                                                     [100%]

=================================================================================================================================== FAILURES ===================================================================================================================================
______________________________________________________________________________________________________________ test_run[numpy-https://github.com/numpy/numpy.git] ______________________________________________________________________________________________________________

func = <function register_builtin_transform.<locals>._transform_wrapper at 0x108043940>, instance = None, args = (<Call l.15 at 0x19754da30>, None), kwargs = {}
_cache = {(functools.partial(<function infer_numpy_member at 0x107fc45e0>, "def array(object, dtype=None, copy=True, order='K',..._numpy_ndarray at 0x1080459d0>, <Attribute.ndarray l.52 at 0x109671a30>): [<ClassDef.ndarray l.2 at 0x10a4061f0>], ...}

    @wrapt.decorator
    def _inference_tip_cached(func, instance, args, kwargs, _cache={}):
        """Cache decorator used for inference tips"""
        node = args[0]
        try:
>           return iter(_cache[func, node])
E           KeyError: (<function register_builtin_transform.<locals>._transform_wrapper at 0x108043940>, <Call l.15 at 0x19754da30>)

profile_against_external/lib/python3.9/site-packages/astroid/__init__.py:93: KeyError

During handling of the above exception, another exception occurred:

tmp_path = PosixPath('/private/var/folders/mm/qst6ghg51594pxf80pg1hl440000gp/T/pytest-of-/pytest-1/test_run_numpy_https___github_0'), name = 'numpy', git_repo = 'https://github.com/numpy/numpy.git'

    @pytest.mark.skipif(
        not os.environ.get("PYTEST_PROFILE_EXTERNAL", False),
        reason="PYTEST_PROFILE_EXTERNAL, not set, assuming not a profile run",
    )
    @pytest.mark.parametrize(
        "name,git_repo", [("numpy", "https://github.com/numpy/numpy.git")]
    )
    def test_run(tmp_path, name, git_repo):
        """ Runs pylint against external sources """
        checkoutdir = tmp_path / name
        checkoutdir.mkdir()
        os.system(
            "git clone --depth=1 {git_repo} {checkoutdir}".format(
                git_repo=git_repo, checkoutdir=str(checkoutdir)
            )
        )
        filepaths = _get_py_files(scanpath=str(checkoutdir))
        print("Have %d files" % len(filepaths))
    
>       runner = Run(filepaths, reporter=Reporter(), do_exit=False)

../tests/profile/test_profile_against_externals.py:55: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
profile_against_external/lib/python3.9/site-packages/pylint/lint/run.py:358: in __init__
    linter.check(args)
profile_against_external/lib/python3.9/site-packages/pylint/lint/pylinter.py:862: in check
    self._check_files(
profile_against_external/lib/python3.9/site-packages/pylint/lint/pylinter.py:896: in _check_files
    self._check_file(get_ast, check_astroid_module, name, filepath, modname)
profile_against_external/lib/python3.9/site-packages/pylint/lint/pylinter.py:922: in _check_file
    check_astroid_module(ast_node)
profile_against_external/lib/python3.9/site-packages/pylint/lint/pylinter.py:1054: in check_astroid_module
    retval = self._check_astroid_module(
profile_against_external/lib/python3.9/site-packages/pylint/lint/pylinter.py:1099: in _check_astroid_module
    walker.walk(ast_node)
profile_against_external/lib/python3.9/site-packages/pylint/utils/ast_walker.py:75: in walk
    self.walk(child)
profile_against_external/lib/python3.9/site-packages/pylint/utils/ast_walker.py:75: in walk
    self.walk(child)
profile_against_external/lib/python3.9/site-packages/pylint/utils/ast_walker.py:75: in walk
    self.walk(child)
profile_against_external/lib/python3.9/site-packages/pylint/utils/ast_walker.py:72: in walk
    callback(astroid)
profile_against_external/lib/python3.9/site-packages/pylint/checkers/classes.py:1878: in visit_functiondef
    inferred = _safe_infer_call_result(node, node)
profile_against_external/lib/python3.9/site-packages/pylint/checkers/classes.py:444: in _safe_infer_call_result
    value = next(inferit)
profile_against_external/lib/python3.9/site-packages/astroid/scoped_nodes.py:1722: in infer_call_result
    yield from returnnode.value.infer(context)
profile_against_external/lib/python3.9/site-packages/astroid/node_classes.py:357: in infer
    yield from self._explicit_inference(self, context, **kwargs)
profile_against_external/lib/python3.9/site-packages/astroid/__init__.py:95: in _inference_tip_cached
    result = func(*args, **kwargs)
profile_against_external/lib/python3.9/site-packages/astroid/brain/brain_builtin_inference.py:173: in _transform_wrapper
    result = transform(node, context=context)
profile_against_external/lib/python3.9/site-packages/astroid/brain/brain_builtin_inference.py:764: in infer_len
    return nodes.Const(helpers.object_len(argument_node, context=context))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

node = <Attribute.array l.15 at 0x19754df10>, context = None

    def object_len(node, context=None):
        """Infer length of given node object
    
        :param Union[nodes.ClassDef, nodes.Instance] node:
        :param node: Node to infer length of
    
        :raises AstroidTypeError: If an invalid node is returned
            from __len__ method or no __len__ method exists
        :raises InferenceError: If the given node cannot be inferred
            or if multiple nodes are inferred or if the code executed in python
            would result in a infinite recursive check for length
        :rtype int: Integer length of node
        """
        # pylint: disable=import-outside-toplevel; circular import
        from astroid.objects import FrozenSet
    
        inferred_node = safe_infer(node, context=context)
    
        # prevent self referential length calls from causing a recursion error
        # see https://github.com/PyCQA/astroid/issues/777
        node_frame = node.frame()
        if (
            isinstance(node_frame, scoped_nodes.FunctionDef)
            and node_frame.name == "__len__"
>           and inferred_node._proxied == node_frame.parent
        ):
E       AttributeError: 'FunctionDef' object has no attribute '_proxied'

profile_against_external/lib/python3.9/site-packages/astroid/helpers.py:255: AttributeError
----------------------------------------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------------------------------------
Have 502 files
----------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------
Cloning into '/private/var/folders/mm/qst6ghg51594pxf80pg1hl440000gp/T/pytest-of-/pytest-1/test_run_numpy_https___github_0/numpy'...
Profiling (from contributing/pylint/.tox/prof/combined.prof):
Wed Oct 21 08:25:36 2020    contributing/pylint/.tox/prof/combined.prof

         409548173 function calls (340388327 primitive calls) in 401.101 seconds

   Ordered by: cumulative time
   List reduced from 2780 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  401.373  401.373 runner.py:107(pytest_runtest_protocol)
    25/12    0.000    0.000  401.373   33.448 hooks.py:272(__call__)
    25/12    0.000    0.000  401.373   33.448 manager.py:90(_hookexec)
    25/12    0.000    0.000  401.373   33.448 manager.py:84(<lambda>)
    25/12    0.000    0.000  401.372   33.448 callers.py:157(_multicall)
        1    0.000    0.000  401.372  401.372 runner.py:115(runtestprotocol)
        3    0.000    0.000  401.372  133.791 runner.py:213(call_and_report)
        3    0.000    0.000  400.816  133.605 runner.py:241(call_runtest_hook)
        3    0.000    0.000  400.816  133.605 runner.py:299(from_call)
        3    0.000    0.000  400.816  133.605 runner.py:256(<lambda>)
        1    0.000    0.000  400.809  400.809 runner.py:154(pytest_runtest_call)
        1    0.000    0.000  400.809  400.809 python.py:1625(runtest)
        1    0.000    0.000  400.809  400.809 python.py:177(pytest_pyfunc_call)
        1    0.000    0.000  400.809  400.809 test_profile_against_externals.py:36(test_run)
        1    0.000    0.000  394.978  394.978 run.py:74(__init__)
        1    0.000    0.000  394.871  394.871 pylinter.py:837(check)
        1    0.000    0.000  394.761  394.761 pylinter.py:885(_check_files)
      197    0.003    0.000  393.821    1.999 pylinter.py:898(_check_file)
      197    0.023    0.000  373.055    1.894 pylinter.py:1047(check_astroid_module)
      197    0.010    0.000  373.032    1.894 pylinter.py:1064(_check_astroid_module)


SVG profile in contributing/pylint/.tox/prof/combined.svg.
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED ../tests/profile/test_profile_against_externals.py::test_run[numpy-https:/github.com/numpy/numpy.git] - AttributeError: 'FunctionDef' object has no attribute '_proxied'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================================================================================================== 1 failed in 402.95s (0:06:42) =========================================================================================================================
ERROR: InvocationError for command contributing/pylint/.tox/profile_against_external/bin/python -Wi -m pytest --exitfirst --profile-svg contributing/pylint/tests/profile/test_profile_against_externals.py (exited with code 1)
___________________________________________________________________________________________________________________________________ summary ____________________________________________________________________________________________________________________________________
ERROR:   profile_against_external: commands failed

Edit: did some more regression testing
Edit: fixed typos in desc

The text was updated successfully, but these errors were encountered:

hippo91 · 2020-10-22T16:57:12Z

@doublethefish thanks for the report. It is indeed not normal.

brycepg · 2020-10-26T17:34:33Z

Oops looks like I made this regression in a bug fix

Looks like numpy is trying to take the length of a function somewhere and apparently this code breaks at that point.

Pierre-Sassoulas · 2021-03-07T12:48:58Z

@brycepg I don't know if this is fixed, I'm trying to close those RecursionError issues because a lot of it has been fixed recently. Is this one different ?

doublethefish · 2021-03-22T10:02:51Z

The error still occurs as of master@ 25e63b0

tox -e profile_against_external (took 311.599s)

bogdandm · 2021-04-09T10:48:19Z

Have this crash too with

Python 3.8.9
pylint==2.7.4
astroid==2.5.2

Traceback (most recent call last):
  File "/tv/lib/python3.8/site-packages/astroid/__init__.py", line 95, in _inference_tip_cached
    return iter(_cache[func, node])
KeyError: (<function register_builtin_transform.<locals>._transform_wrapper at 0x7fb288495b80>, <Call l.106 at 0x7fb277ef61c0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/tv/lib/python3.8/site-packages/pylint/lint/parallel.py", line 69, in _worker_check_single_file
    _worker_linter.check_single_file(name, filepath, modname)
  File "/tv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 895, in check_single_file
    self._check_file(
  File "/tv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 936, in _check_file
    check_astroid_module(ast_node)
  File "/tv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1070, in check_astroid_module
    retval = self._check_astroid_module(
  File "/tv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1115, in _check_astroid_module
    walker.walk(ast_node)
  File "/tv/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/tv/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/tv/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 72, in walk
    callback(astroid)
  File "/tv/lib/python3.8/site-packages/pylint/checkers/classes.py", line 1879, in visit_functiondef
    inferred = _safe_infer_call_result(node, node)
  File "/tv/lib/python3.8/site-packages/pylint/checkers/classes.py", line 442, in _safe_infer_call_result
    value = next(inferit)
  File "/tv/lib/python3.8/site-packages/astroid/scoped_nodes.py", line 1728, in infer_call_result
    yield from returnnode.value.infer(context)
  File "/tv/lib/python3.8/site-packages/astroid/node_classes.py", line 360, in infer
    yield from self._explicit_inference(self, context, **kwargs)
  File "/tv/lib/python3.8/site-packages/astroid/__init__.py", line 97, in _inference_tip_cached
    result = func(*args, **kwargs)
  File "/tv/lib/python3.8/site-packages/astroid/brain/brain_builtin_inference.py", line 175, in _transform_wrapper
    result = transform(node, context=context)
  File "/tv/lib/python3.8/site-packages/astroid/brain/brain_builtin_inference.py", line 764, in infer_len
    return nodes.Const(helpers.object_len(argument_node, context=context))
  File "/tv/lib/python3.8/site-packages/astroid/helpers.py", line 261, in object_len
    and inferred_node._proxied == node_frame.parent
AttributeError: 'ClassDef' object has no attribute '_proxied'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "tools/gitlab_ci/python_code_validity/run.py", line 83, in <module>
    exit_code = gitlab_ci_pychecks()
  File "tools/gitlab_ci/python_code_validity/run.py", line 40, in gitlab_ci_pychecks
    _, reporter, linter = next(run_pylint(
  File "/builds/tv/tradingview/tools/python_code_checking/pylint_wrapper.py", line 89, in run_pylint
    linter = Run(
  File "/tv/lib/python3.8/site-packages/pylint/lint/run.py", line 358, in __init__
    linter.check(args)
  File "/tv/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 880, in check
    check_parallel(
  File "/tv/lib/python3.8/site-packages/pylint/lint/parallel.py", line 138, in check_parallel
    for (
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
AttributeError: 'ClassDef' object has no attribute '_proxied'

DanielNoord · 2021-11-08T20:20:45Z

This might have actually been fixed in pylint-dev/astroid#1234.
I did not check initially but #5244 might have been a duplicate of this report. I could run pylint fine on numpy expect for a bug fixed with pylint-dev/astroid#1234.

This also seems to be somewhat related to the error we found in the numpy brain. We might want to leave it open to record it, although we should then probably rename the issue.

DanielNoord · 2021-11-15T09:40:53Z

Seeing as our draft primer tests in #5173 have been running fine on main of numpy I'm going to close this. I think this was indeed a duplicate.

Please let me know if any issues still persist and I'll reopen!

doublethefish · 2021-11-16T12:58:12Z

I believe there’s an profiling-test in pylint’s tox.ini that I added to profile pylint runs against numpy. Perhaps it’s now worth turning that on in CiCd now?

Pierre-Sassoulas · 2021-11-16T13:02:19Z

I totally forgot about this profiling test, you might bring valuable insight about what we do right now. We're running on performance issue for primer test in #5173. Let's discuss how to integrate this profiling test in what we do now if you want to participate @doublethefish :)

doublethefish changed the title ~~pylint 2.4.4->2.5.0 regression raises again [email protected]~~ pylint 2.4.4->2.5.? regression raises again [email protected] Oct 21, 2020

doublethefish changed the title ~~pylint 2.4.4->2.5.? regression raises again [email protected]~~ pylint 2.6.0->2.6.1-dev1 regression raises again [email protected] Oct 21, 2020

doublethefish changed the title ~~pylint 2.6.0->2.6.1-dev1 regression raises again [email protected]~~ pylint 2.6.0->2.6.1-dev1 regression - raises running against [email protected] Oct 21, 2020

doublethefish mentioned this issue Oct 21, 2020

profile| Runs pylint against external code, generating profile-heatmaps #3498

Merged

1 task

hippo91 added Regression Crash 💥 A bug that makes pylint crash labels Oct 22, 2020

brycepg self-assigned this Oct 26, 2020

Pierre-Sassoulas added the Bug 🪲 label Jan 1, 2021

Pierre-Sassoulas mentioned this issue Jan 1, 2021

Maximum recursion depth crash with pyreverse -S option #3602

Open

Pierre-Sassoulas mentioned this issue Feb 12, 2021

RecursionError: maximum recursion depth exceeded while calling a Python object #4081

Closed

DanielNoord closed this as completed Nov 15, 2021

DanielNoord added this to the 2.12.0 milestone Nov 15, 2021

DanielNoord added the Duplicate 🐫 Duplicate of an already existing issue label Nov 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pylint 2.6.0->2.6.1-dev1 regression - raises running against [email protected] #3913

pylint 2.6.0->2.6.1-dev1 regression - raises running against [email protected] #3913

doublethefish commented Oct 21, 2020 •

edited

Loading

hippo91 commented Oct 22, 2020

brycepg commented Oct 26, 2020 •

edited

Loading

Pierre-Sassoulas commented Mar 7, 2021

doublethefish commented Mar 22, 2021

bogdandm commented Apr 9, 2021

DanielNoord commented Nov 8, 2021

DanielNoord commented Nov 15, 2021

doublethefish commented Nov 16, 2021 •

edited

Loading

Pierre-Sassoulas commented Nov 16, 2021

pylint 2.6.0->2.6.1-dev1 regression - raises running against [email protected] #3913

pylint 2.6.0->2.6.1-dev1 regression - raises running against [email protected] #3913

Comments

doublethefish commented Oct 21, 2020 • edited Loading

Steps to reproduce

Current behavior

Expected behavior

pylint --version output

full log

hippo91 commented Oct 22, 2020

brycepg commented Oct 26, 2020 • edited Loading

Pierre-Sassoulas commented Mar 7, 2021

doublethefish commented Mar 22, 2021

bogdandm commented Apr 9, 2021

DanielNoord commented Nov 8, 2021

DanielNoord commented Nov 15, 2021

doublethefish commented Nov 16, 2021 • edited Loading

Pierre-Sassoulas commented Nov 16, 2021

doublethefish commented Oct 21, 2020 •

edited

Loading

brycepg commented Oct 26, 2020 •

edited

Loading

doublethefish commented Nov 16, 2021 •

edited

Loading