[FIX] duplicate-code: ignored disable=duplicate-code comments

[moylop260]

Fixes / new features

• Fix The duplicate-code (R0801) can't be disabled #214

It's a proposal from last comment #1014 (comment) from 15 days ago.

• It don't depend on PyLinter to do its job
• It was fixed a clean way
• The user can enable/disable the message many times in the same file
• Add docs
• Tested with file duplicate-code-disable-enable-example.py and pylint/test/input/similar1

# wget goo.gl/jsW67L -o duplicate-code-disable-enable-example.py
# pylint  -rn -d all -e duplicate-code duplicate-code-disable-enable-example.py pylint/test/input/similar1
************* Module input.similar1
R:  1, 0: Similar lines in 2 files
==duplicate-code-disable-enable-example:11
==input.similar1:7
eight
nine
''' ten
eleven
twelve '''
thirteen
fourteen
fifteen




sixteen (duplicate-code)
R:  1, 0: Similar lines in 2 files
==duplicate-code-disable-enable-example:3
==input.similar1:0
import one
from two import two
three
four
five
six # comments optionally ignored
seven (duplicate-code)

Note:

• My apologies for the insistence, I promise this is my last attempt to repair this issue.
  Just I could add a reference in the original issue to other people that want apply this patch.
• We can add the tests (where a pylinter is required for now) for 3.0 version (based on [REF] duplicate-code: Check state of the lines to strip them #1014 (comment))

[moylop260]

@PCManticore
Can I help with another matter for this PR?

[bittner]

Any chance this is being merged?

@moylop260 Can you resolve the conflict with ChangeLog? Maybe that's holding the maintainers back from merging.

[moylop260]

Fixed conflicts

[hbto]

@moylop260 There are new conflicts in this PR.

Regards

[moylop260]

@hbto
I just deleted the commit of documentation because add many conflicts in each merge of other PR.

I let the message and the change deleted here:

commit 7862b586fb1443963df7e94032e95f9d3bd861a0
Author: Moisés López <[email protected]>
Date:   Sun Aug 28 20:57:53 2016 -0500

    [REF] doc: Add fix duplicate-code

diff --git a/ChangeLog b/ChangeLog
index f57d6ec..792fdb7 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -183,6 +183,10 @@ Release date: tba
     * Added a new Python 3 warning around implementing '__div__' or '__idiv__'
       as those methods are phased out in Python 3.
 
+    * Fix the duplicate-code (R0801) can't be disabled using
+      pylint enable/disable comments.
+
+      Closes issue #214.
 
 What's new in Pylint 1.6.3?
 ===========================
diff --git a/doc/whatsnew/2.0.rst b/doc/whatsnew/2.0.rst
index a25f303..b988ea1 100644
--- a/doc/whatsnew/2.0.rst
+++ b/doc/whatsnew/2.0.rst
@@ -446,6 +446,10 @@ Bug fixes
          linter.register_checker(DummyPlugin1(linter))
          linter.register_checker(DummyPlugin2(linter))
 
+* Fix the duplicate-code (R0801) can't be disabled using
+  pylint enable/disable comments.
+
+  Closes issue #214
 
 
 Removed Changes

[brianmay]

Is there anything holding this back from getting merged? Anything that can be fixed?

[moylop260]

@brianmay @bittner

Currently I'm applying this patch from our CI system using the following lines:

wget -q https://patch-diff.githubusercontent.com/raw/PyCQA/pylint/pull/1055.patch -O /tmp/pylint_pr1055.patch
patch -f -p0 $(python -c "from pylint.checkers import similar;print similar.__file__.rstrip('c')") -i /tmp/pylint_pr1055.patch

You can use them since that this fix currently is not priority for maintainers

[rogalski]

I am all for merging it, but we need automated tests that confirms new feature works.

[moylop260]

@rogalski
Could you check the changes, please?

The tests of the duplicated code are executed from a run method. It required depends on PyLinter to support disable/enable comments.

[rogalski]

@moylop260 can we please keep pytest-style functional tests instead of introducing a class?

Other than that, it looks great.

[moylop260]

@rogalski Done! Using pytest-style

[rogalski]

IMHO it's ready to be merged.

The only thing to be discussed is creating this fake PyLinter instance just to get state lines. Maybe we can extract this functionality to more manageable form.

[rogalski]

This almost look like we need to extract this functionality from PyLinter class

[moylop260]

I'm not sure of understand it
Do you have a example to clarify it?

[rogalski]

I only mean that information about locally disabled messages should be acquirable without creating an instance of PyLinter. I'm not sure if this is possible right now, I'm not familiar with this part of repository. I'll check it and get back to you. Dnia 14.08.2017 o godz. 23:15 moylop260 <[email protected]> napisał(a):

…

@moylop260 commented on this pull request. In pylint/checkers/similar.py: > @@ -327,6 +336,27 @@ def usage(status=0): [-i|--ignore-comments] [--ignore-docstrings] [--ignore-imports] file1...') sys.exit(status) + +def _get_state_lines(linter, filename): I'm not sure of understand it Do you have a example to clarify it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[PCManticore]

Folks, as I already mentioned in one of the many iterations of this PR (#1014), I'm not going to merge this PR in this state, where the file state is retrieved as is by instantiating a PyLinter class and whatnot. One has to to come up with a better engineered solution for solving this problem, so let's stop for a moment, let's identify how we can approach this in a sane manner and after that we can write the code.

[moylop260]

@PCManticore

The pylinter is not required to do the work.
The first commit is enough.

We are using this patch 1 year ago working without problem.

Could you help us to identify why the first commit/soluction is a bad engineered, please?
In order to know where I'm wrong

[bakert]

I guess this one rotted away? It would be swell to be able to disable duplicate-code for one particular file in my project. I'm probably going to skip all linting of that file to get that effect which is much less desirable. Seems like the patch here avoids the criticisms elsewhere of dependence on a PyLinter instance and not having any tests and could be merged. But I'm no expert on pylint :D Was another solution ever found?

[moylop260]

Seems like the patch here avoids the criticisms elsewhere of dependence on a PyLinter instance and not having any tests and could be merged.

Yes, you are right.
This patch don't depend on PyLinter to work.
But the maintainers request me: remove PyLinter and I removed it
After that, they request me adds test and I added PyLinter
After that, they silent me that I shouldn't attempt to fix it again using PyLinter... ignoring my comments about.

I was banned to contribute with this project.

But you can use the following patch in order to fix it locally or for your CI using the following command:

wget -q https://raw.githubusercontent.com/Vauxoo/pylint-conf/master/conf/pylint_pr1055.patch -O /tmp/pylint_pr1055.patch
patch -f -p0 $(python -c "from __future__ import print_function; from pylint.checkers import similar; print(similar.__file__.rstrip('c'))") -i /tmp/pylint_pr1055.patch

UPDATED 2019-03-07:
it patch now has divergences with new pylint versions
You will need apply it manually

[moylop260]

I mean, the minimal patch to fix this too old issue is (more than 4 years ago):
#214

diff --git a/pylint/checkers/similar.py b/pylint/checkers/similar.py
index aa59bfd..a7c0c0e 100644
--- a/pylint/checkers/similar.py
+++ b/pylint/checkers/similar.py
@@ -31,7 +31,7 @@ class Similar(object):
         self.ignore_imports = ignore_imports
         self.linesets = []
 
-    def append_stream(self, streamid, stream, encoding=None):
+    def append_stream(self, streamid, stream, encoding=None, state_lines=None):
         """append a file to search for similarities"""
         if encoding is None:
             readlines = stream.readlines
@@ -42,7 +42,7 @@ class Similar(object):
                                          readlines(),
                                          self.ignore_comments,
                                          self.ignore_docstrings,
-                                         self.ignore_imports))
+                                         self.ignore_imports, state_lines))
         except UnicodeDecodeError:
             pass
 
@@ -126,14 +126,17 @@ class Similar(object):
                 for sim in self._find_common(lineset, lineset2):
                     yield sim
 
-def stripped_lines(lines, ignore_comments, ignore_docstrings, ignore_imports):
+def stripped_lines(lines, ignore_comments, ignore_docstrings, ignore_imports, state_lines=None):
     """return lines with leading/trailing whitespace and any ignored code
     features removed
     """
 
     strippedlines = []
     docstring = None
-    for line in lines:
+    for lineno, line in enumerate(lines, start=1):
+        if state_lines and lineno in state_lines and not state_lines[lineno]:
+            strippedlines.append('')
+            continue
         line = line.strip()
         if ignore_docstrings:
             if not docstring and \
@@ -158,12 +161,12 @@ def stripped_lines(lines, ignore_comments, ignore_docstrings, ignore_imports):
 class LineSet(object):
     """Holds and indexes all the lines of a single source file"""
     def __init__(self, name, lines, ignore_comments=False,
-                 ignore_docstrings=False, ignore_imports=False):
+                 ignore_docstrings=False, ignore_imports=False, state_lines=None):
         self.name = name
         self._real_lines = lines
         self._stripped_lines = stripped_lines(lines, ignore_comments,
                                               ignore_docstrings,
-                                              ignore_imports)
+                                              ignore_imports, state_lines)
         self._index = self._mk_index()
 
     def __str__(self):
@@ -290,10 +293,11 @@ class SimilarChecker(BaseChecker, Similar):
 
         stream must implement the readlines method
         """
+        state_lines = self.linter.file_state._module_msgs_state.get('R0801')
         with node.stream() as stream:
             self.append_stream(self.linter.current_name,
                                stream,
-                               node.file_encoding)
+                               node.file_encoding, state_lines)
 
     def close(self):
         """compute and display similarities on closing (i.e. end of parsing)"""

[filips123]

@moylop260 Your fix doesn't work. Even if I have # pylint: disable=duplicate-code in each file with duplicate code, I still get errors.

[moylop260]

@filips123
It is helpful if you send me what version of astroid and pylint are you using and what files with duplicated code are you testing?

[filips123]

Here are versions:

pylint 2.3.0
astroid 2.2.0
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516]

And some of my code which is duplicated is:

    @staticmethod
    def __zip_directory(source, destination):
        """Make ZIP archive from directory."""

        zipf = zipfile.ZipFile(destination, 'w', zipfile.ZIP_DEFLATED)

        for root, _, files in os.walk(source):
            for file in files:
                real = os.path.join(root, file)
                name = real[len(source):]
                zipf.write(real, name)

        zipf.close()

    @staticmethod
    def __unzip_directory(source, destination):
        """Extract ZIP archive to directory."""

        zipf = zipfile.ZipFile(source)
        zipf.extractall(destination)
        zipf.close()

I can't use the shared code for this because there are some changes in every function. I fixed this for now with specifying high min-similarity-lines files in the config file.

[moylop260]

@filips123
Did you have a trouble applying the patch using a command?
Maybe a Hunk .. FAILED

If yes, then you need apply it manually since that it patch was created 3 years ago and it could have
divergences

[filips123]

Yes, there are errors:

patching file 'C:\Program Files\Python37\lib\site-packages\pylint\checkers\similar.py'
Hunk #1 FAILED at 31.
Hunk #2 FAILED at 42.
Hunk #3 FAILED at 126.
Hunk #4 FAILED at 158.
Hunk #5 FAILED at 290.
5 out of 5 hunks FAILED -- saving rejects to file 'C:\Program Files\Python37\lib\site-packages\pylint\checkers\similar.py.rej'

And similar.py.rej is:

--- a/pylint/checkers/similar.py
+++ b/pylint/checkers/similar.py
@@ -31,7 +31,7 @@ class Similar(object):
         self.ignore_imports = ignore_imports
         self.linesets = []

-    def append_stream(self, streamid, stream, encoding=None):
+    def append_stream(self, streamid, stream, encoding=None, state_lines=None):
         """append a file to search for similarities"""
         if encoding is None:
             readlines = stream.readlines
@@ -42,7 +42,7 @@ class Similar(object):
                                          readlines(),
                                          self.ignore_comments,
                                          self.ignore_docstrings,
-                                         self.ignore_imports))
+                                         self.ignore_imports, state_lines))
         except UnicodeDecodeError:
             pass

@@ -126,14 +126,17 @@ class Similar(object):
                 for sim in self._find_common(lineset, lineset2):
                     yield sim

-def stripped_lines(lines, ignore_comments, ignore_docstrings, ignore_imports):
+def stripped_lines(lines, ignore_comments, ignore_docstrings, ignore_imports, state_lines=None):
     """return lines with leading/trailing whitespace and any ignored code
     features removed
     """

     strippedlines = []
     docstring = None
-    for line in lines:
+    for lineno, line in enumerate(lines, start=1):
+        if state_lines and lineno in state_lines and not state_lines[lineno]:
+            strippedlines.append('')
+            continue
         line = line.strip()
         if ignore_docstrings:
             if not docstring and \
@@ -158,12 +161,12 @@ def stripped_lines(lines, ignore_comments, ignore_docstrings, ignore_imports):
 class LineSet(object):
     """Holds and indexes all the lines of a single source file"""
     def __init__(self, name, lines, ignore_comments=False,
-                 ignore_docstrings=False, ignore_imports=False):
+                 ignore_docstrings=False, ignore_imports=False, state_lines=None):
         self.name = name
         self._real_lines = lines
         self._stripped_lines = stripped_lines(lines, ignore_comments,
                                               ignore_docstrings,
-                                              ignore_imports)
+                                              ignore_imports, state_lines)
         self._index = self._mk_index()

     def __str__(self):
@@ -290,10 +293,11 @@ class SimilarChecker(BaseChecker, Similar):

         stream must implement the readlines method
         """
+        state_lines = self.linter.file_state._module_msgs_state.get('R0801')
         with node.stream() as stream:
             self.append_stream(self.linter.current_name,
                                stream,
-                               node.file_encoding)
+                               node.file_encoding, state_lines)

     def close(self):
         """compute and display similarities on closing (i.e. end of parsing)"""

[moylop260]

For pylint >= 2.2.0
Specific SHA 76ce7c1

The new patch is:

• https://github.com/Vauxoo/pylint-conf/blob/master/conf/pylint_pr1055_2.patch

wget -q https://raw.githubusercontent.com/Vauxoo/pylint-conf/master/conf/pylint_pr1055_2.patch -O /tmp/pylint_pr1055_2.patch
patch -f -p0 $(python -c "from __future__ import print_function; from pylint.checkers import similar; print(similar.__file__.rstrip('c'))") -i /tmp/pylint_pr1055_2.patch

For older ones use:

wget -q https://raw.githubusercontent.com/Vauxoo/pylint-conf/master/conf/pylint_pr1055.patch -O /tmp/pylint_pr1055.patch
patch -f -p0 $(python -c "from __future__ import print_function; from pylint.checkers import similar; print(similar.__file__.rstrip('c'))") -i /tmp/pylint_pr1055.patch

[mattcanty]

We're hitting this issue with AWS Lambda functions. Nothing shared so duplication is expected. What is the ETA on this being in a release rather than patch?

[moylop260]

Hi @mattcanty

I'm not a maintainer of this project then I can't tell you a ETA to release a fix.
It is just my 5 cents to fix it using a patch.

You can disable this check or increase the min-similarity-lines parameter.

[mitchmieuxplacer]

Here is a way to get around this issue : edit your pylintrc file and add:

# Minimum lines number of a similarity.
min-similarity-lines=100

Setting a high value to min-similarity-lines will disable this warning.