INTERNAL ERROR: Black produced code that is not equivalent to the source

[ghost]

Hello ! I'm facing an issue while trying to use black to lint a file.

When trying to lint my piece of code (either with pre-commit or running black itself), the below error happen. Unfortunately, i can't provide any example as it's not my property.

The resulting error is:

error: cannot format core/nomenclature/tests.py: INTERNAL ERROR: Black produced code that is not equivalent to the source. Please report a bug on https://github.com/psf/black/issues. This diff might be helpful: /tmp/blk_818irkgm.log

You can find the log content here: https://privatebin.net/?dfa3a27759345ea2#CehPbuNtkh7aqgdpn5WWhXdVs2zD1fCQVoAiG2SMp1J5

Environment

• Black's version: 22.3.0
• OS and Python version: Ubuntu 20.04.4 LTS on WSL2 - Python 3.8.13

[felix-hilden]

The "not equivalent to the source" is a safety check we perform to ensure we don't mess up your files, so it being pretty generic makes it pretty much impossible to narrow the problem down. Thank you for the log file though!

Could you take the file and try to simplify it while preserving the error, so that you could share it with us?

[ghost]

@felix-hilden Actually i find out what was going wrong:

I add some # fmt: off for several lines in a first piece of code where i didn't want black to lint. Later, i add another piece of code, and a that moment black start to crash. If i remove that later piece of code, it works, if i add it again, it crash again. I finally suspect the # fmt: off to be the root of the problem, removing them make black fully working again.

An example:

def test_a_random_test(self):
   ...
   # some code
   ...
   # fmt: off
   # Insert here a very long series of slicing over a nested list (a graphql api response)
   # fmt: off
   # Insert here a very long series of slicing over a nested list (a graphql api response)
   # fmt: off
   # Insert here a very long series of slicing over a nested list (a graphql api response)
   ...
   # some code
   ...

def test_later_added(self):
  ...
  # some code
  ...

Removing/adding the test_later_added allow to fully reproduce/stop the black problem. Finally, removing the format off tags solve this definitely.

You can close this issue if you want, as i identify what was going wrong. Thanks !

[dignissimus]

A minimal example

def first(self):
    # fmt: off
    ...

def second(self):
    ...

Black produces

def first(self):
    # fmt: off
    ...

    def second(self):
        ...

This is wrong

[dignissimus]

AST diff

--- src
+++ dst
@@ -29,54 +29,54 @@
                 None,  # NoneType
               value=
                 Ellipsis,  # ellipsis
             )  # /Constant
         )  # /Expr
+        FunctionDef(
+          args=
+            arguments(
+              args=
+                arg(
+                  annotation=
+                    None,  # NoneType
+                  arg=
+                    'self',  # str
+                  type_comment=
+                    None,  # NoneType
+                )  # /arg
+              defaults=
+              kw_defaults=
+              kwarg=
+                None,  # NoneType
+              kwonlyargs=
+              posonlyargs=
+              vararg=
+                None,  # NoneType
+            )  # /arguments
+          body=
+            Expr(
+              value=
+                Constant(
+                  kind=
+                    None,  # NoneType
+                  value=
+                    Ellipsis,  # ellipsis
+                )  # /Constant
+            )  # /Expr
+          decorator_list=
+          name=
+            'second',  # str
+          returns=
+            None,  # NoneType
+          type_comment=
+            None,  # NoneType
+        )  # /FunctionDef
       decorator_list=
       name=
         'first',  # str
       returns=
         None,  # NoneType
       type_comment=
         None,  # NoneType
     )  # /FunctionDef
-    FunctionDef(
-      args=
-        arguments(
-          args=
-            arg(
-              annotation=
-                None,  # NoneType
-              arg=
-                'self',  # str
-              type_comment=
-                None,  # NoneType
-            )  # /arg
-          defaults=
-          kw_defaults=
-          kwarg=
-            None,  # NoneType
-          kwonlyargs=
-          posonlyargs=
-          vararg=
-            None,  # NoneType
-        )  # /arguments
-      body=
-        Expr(
-          value=
-            Constant(
-              kind=
-                None,  # NoneType
-              value=
-                Ellipsis,  # ellipsis
-            )  # /Constant
-        )  # /Expr
-      decorator_list=
-      name=
-        'second',  # str
-      returns=
-        None,  # NoneType
-      type_comment=
-        None,  # NoneType
-    )  # /FunctionDef
   type_ignores=
 )  # /Module
\ No newline at end of file

[dignissimus]

I think it's a problem with block statements in general, the issue also occurs with the following

if True:
    # fmt: off
    ...

print(2)

if True:
    ...
    #fmt: off
    print(4)

print(2)

[felix-hilden]

Thanks! Alright, we assume that a # fmt: off is followed by a # fmt: on. It's not a toggle, but a "scope" that encompasses a block or multiple blocks of code.

A while back a similar issue was discussed in #2985, and less recently in #1817 and #569, which this seems to essentially be a duplicate of.