Compatibility with Python 3.11

[frenzymadness]

Hello.

In Fedora, we are again trying to prepare the new Python 3.11 soon and during the rebuild of our packages, we have discovered that some tests of attrs package fail:

We are building attrs 21.4.0 together with Python 3.11.0~a3 (changelog)

=================================== FAILURES ===================================
_______________________ test_init_subclass_vanilla[True] _______________________

slots = True

    @pytest.mark.parametrize("slots", [True, False])
    def test_init_subclass_vanilla(slots):
        """
        `super().__init_subclass__` can be used if the subclass is not an attrs
        class both with dict and slotted classes.
        """
    
        @attr.s(slots=slots)
        class Base:
            def __init_subclass__(cls, param, **kw):
                super().__init_subclass__(**kw)
                cls.param = param
    
>       class Vanilla(Base, param="foo"):

tests/test_init_subclass.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'tests.test_init_subclass.test_init_subclass_vanilla.<locals>.Vanilla'>
param = 'foo', kw = {}

    def __init_subclass__(cls, param, **kw):
>       super().__init_subclass__(**kw)
E       TypeError: super(type, obj): obj must be an instance or subtype of type

tests/test_init_subclass.py:24: TypeError
_____________ TestClosureCellRewriting.test_closure_cell_rewriting _____________

self = <tests.test_slots.TestClosureCellRewriting object at 0x7fb02ec6da50>

    def test_closure_cell_rewriting(self):
        """
        Slotted classes support proper closure cell rewriting.
    
        This affects features like `__class__` and the no-arg super().
        """
        non_slot_instance = C1(x=1, y="test")
        slot_instance = C1Slots(x=1, y="test")
    
        assert non_slot_instance.my_class() is C1
>       assert slot_instance.my_class() is C1Slots
E       AssertionError: assert <class 'tests.test_slots.C1Slots'> is C1Slots
E        +  where <class 'tests.test_slots.C1Slots'> = <bound method C1Slots.my_class of C1Slots(x=1, y='test')>()
E        +    where <bound method C1Slots.my_class of C1Slots(x=1, y='test')> = C1Slots(x=1, y='test').my_class

tests/test_slots.py:424: AssertionError
__________________ TestClosureCellRewriting.test_inheritance ___________________

self = <tests.test_slots.TestClosureCellRewriting object at 0x7fb02ec658d0>

    def test_inheritance(self):
        """
        Slotted classes support proper closure cell rewriting when inheriting.
    
        This affects features like `__class__` and the no-arg super().
        """
    
        @attr.s
        class C2(C1):
            def my_subclass(self):
                return __class__
    
        @attr.s
        class C2Slots(C1Slots):
            def my_subclass(self):
                return __class__
    
        non_slot_instance = C2(x=1, y="test")
        slot_instance = C2Slots(x=1, y="test")
    
        assert non_slot_instance.my_class() is C1
>       assert slot_instance.my_class() is C1Slots
E       AssertionError: assert <class 'tests.test_slots.C1Slots'> is C1Slots
E        +  where <class 'tests.test_slots.C1Slots'> = <bound method C1Slots.my_class of C2Slots(x=1, y='test')>()
E        +    where <bound method C1Slots.my_class of C2Slots(x=1, y='test')> = C2Slots(x=1, y='test').my_class

tests/test_slots.py:451: AssertionError
________________ TestClosureCellRewriting.test_cls_static[True] ________________

self = <tests.test_slots.TestClosureCellRewriting object at 0x7fb02ec71c90>
slots = True

    @pytest.mark.parametrize("slots", [True, False])
    def test_cls_static(self, slots):
        """
        Slotted classes support proper closure cell rewriting for class- and
        static methods.
        """
        # Python can reuse closure cells, so we create new classes just for
        # this test.
    
        @attr.s(slots=slots)
        class C:
            @classmethod
            def clsmethod(cls):
                return __class__
    
>       assert C.clsmethod() is C
    
        @attr.s(slots=slots)
E       AssertionError: assert <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'> is <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>
E        +  where <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'> = <bound method TestClosureCellRewriting.test_cls_static.<locals>.C.clsmethod of <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>>()
E        +    where <bound method TestClosureCellRewriting.test_cls_static.<locals>.C.clsmethod of <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>> = <class 'tests.test_slots.TestClosureCellRewriting.test_cls_static.<locals>.C'>.clsmethod

tests/test_slots.py:475: AssertionError
____________________ test_slots_super_property_get_shurtcut ____________________

    @pytest.mark.skipif(PY2, reason="shortcut super() is PY3-only.")
    def test_slots_super_property_get_shurtcut():
        """
        On Python 3, the `super()` shortcut is allowed.
        """
    
        @attr.s(slots=True)
        class A(object):
            x = attr.ib()
    
            @property
            def f(self):
                return self.x
    
        @attr.s(slots=True)
        class B(A):
            @property
            def f(self):
                return super().f ** 2
    
>       assert B(11).f == 121

tests/test_slots.py:739: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = B(x=11)

    @property
    def f(self):
>       return super().f ** 2
E       TypeError: super(type, obj): obj must be an instance or subtype of type

tests/test_slots.py:737: TypeError
=============================== warnings summary ===============================
../../BUILDROOT/python-attrs-21.4.0-1.fc36.x86_64/usr/lib/python3.11/site-packages/attr/_make.py:918
tests/test_init_subclass.py::test_init_subclass_vanilla[True]
tests/test_slots.py::TestClosureCellRewriting::test_cls_static[True]
tests/test_slots.py::test_slots_empty_cell
tests/test_slots.py::test_slots_super_property_get
tests/test_slots.py::test_slots_super_property_get_shurtcut
  /builddir/build/BUILDROOT/python-attrs-21.4.0-1.fc36.x86_64/usr/lib/python3.11/site-packages/attr/_make.py:918: RuntimeWarning: Running interpreter doesn't sufficiently support code object introspection.  Some features like bare super() or accessing __class__ will not work with slotted classes.
    set_closure_cell(cell, cls)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/test_3rd_party.py:14: could not import 'cloudpickle': No module named 'cloudpickle'
SKIPPED [1] tests/test_make.py:469: No old-style classes in Py3
SKIPPED [1] tests/test_make.py:1002: PY2-specific keyword-only error behavior
SKIPPED [1] tests/test_make.py:1967: Needs to be only caught on Python 2.
SKIPPED [1] tests/test_make.py:2333: Pre-3.10 only.
SKIPPED [1] tests/test_pyright.py:26: Requires pyright.
SKIPPED [1] tests/test_slots.py:485: can't break CodeType.replace() via monkeypatch
SKIPPED [1] tests/test_slots.py:532: slots without weakref_slot should only work on PyPy
XFAIL tests/test_setattr.py::TestSetAttr::test_slotted_confused
FAILED tests/test_init_subclass.py::test_init_subclass_vanilla[True] - TypeEr...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_closure_cell_rewriting
FAILED tests/test_slots.py::TestClosureCellRewriting::test_inheritance - Asse...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_cls_static[True]
FAILED tests/test_slots.py::test_slots_super_property_get_shurtcut - TypeErro...
====== 5 failed, 1166 passed, 8 skipped, 1 xfailed, 6 warnings in 19.65s =======

[Tinche]

@frenzymadness should we raise this on the Python side? They're breaking code that works on previous versions. The code is very black magic though.

[frenzymadness]

I haven't had time to investigate it more. If you think those are bugs in Python, then yes, open issues for them. From a quick look, I don't see anything related to this in the changelog.

[Tinche]

Link to BPO: https://bugs.python.org/issue46404

[frenzymadness]

Thank you very much for the upstream report. It's interesting reading.

[frenzymadness]

As I mentioned in the bug in Python upstream, I'd like to make this situation more stable which means to implement the fix you have here for dataclasses and test it in Python so every future change in behavior of __closure__ cells is detected and investigated.

I'm able to reproduce the problem with property:

@dataclasses.dataclass(slots=True)
class A:
    @property
    def prop(self):
        return __class__

a = A()
print(a.prop)

and also with inheritance and arg-less call to super:

@dataclasses.dataclass(slots=True)
class A:
    pass

@dataclasses.dataclass(slots=True)
class B(A):
    def test(self):
        super()

b = B()
b.test()

but the same problem should appear in classmethod and staticmethod but I'm not able to reproduce those. I'm able to find the __closure__ cell of the new class but it already points to the new class instead of the old one. Is it possible that this has changed in Python 3.11? I'm testing this code:

import dataclasses

@dataclasses.dataclass(slots=True)
class A:
    @classmethod
    def class_method(self):
        return __class__

a = A()
a.class_method()

And it works fine in both 3.11 and 3.10 without any fixes. slots have been added to dataclasses in 3.10 so I cannot check this with older versions.

You seem to understand the problem very well so I thought you might know how to reproduce the third use case.

[frenzymadness]

@vstinner could you please help me here. Are classmethods and staticmethods something we should care about here or not?

[vstinner]

I created https://bugs.python.org/issue47143 "Add functools.copy_class() which updates closures".

[Tinche]

@vstinner That would be great!

[frenzymadness]

Unfortunately, it seems that it's not easily doable to implement such general-purpose function in Python, see the reasons mentioned in the last comment: python/cpython#91299 (comment)

We'll probably have to keep the code in attrs with all the branches for older and newer Pythons.

[mgorny]

With 3.11.0b1, I see even more test failures:

FAILED tests/test_3rd_party.py::TestCloudpickleCompat::test_repr - IndexError: tuple index out of range
FAILED tests/test_annotations.py::TestAnnotations::test_auto_attribs[True] - AssertionError: assert {'a': <class ...ist[int], ...} =...
FAILED tests/test_annotations.py::TestAnnotations::test_auto_attribs[False] - AssertionError: assert {'a': <class ...ist[int], ...} ...
FAILED tests/test_annotations.py::TestAnnotations::test_annotations_strings[True] - AssertionError: assert {'a': <class ...ist[int],...
FAILED tests/test_annotations.py::TestAnnotations::test_annotations_strings[False] - AssertionError: assert {'a': <class ...ist[int]...
FAILED tests/test_init_subclass.py::test_init_subclass_vanilla[True] - TypeError: super(type, obj): obj must be an instance or subty...
FAILED tests/test_make.py::TestAutoDetect::test_detects_setstate_getstate[True] - AssertionError: assert None is <built-in method __...
FAILED tests/test_make.py::TestAutoDetect::test_detects_setstate_getstate[False] - AssertionError: assert None is <built-in method _...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_closure_cell_rewriting - AssertionError: assert <class 'tests.test_slots....
FAILED tests/test_slots.py::TestClosureCellRewriting::test_inheritance - AssertionError: assert <class 'tests.test_slots.C1Slots'> i...
FAILED tests/test_slots.py::TestClosureCellRewriting::test_cls_static[True] - AssertionError: assert <class 'tests.test_slots.TestCl...
FAILED tests/test_slots.py::TestPickle::test_no_getstate_setstate_for_dict_classes - AssertionError: assert None is <built-in method...
FAILED tests/test_slots.py::TestPickle::test_no_getstate_setstate_if_option_false - AssertionError: assert None is <built-in method ...
FAILED tests/test_slots.py::test_slots_super_property_get - TypeError: super(type, obj): obj must be an instance or subtype of type
FAILED tests/test_slots.py::test_slots_super_property_get_shortcut - TypeError: super(type, obj): obj must be an instance or subtype...

Full log:
py311.txt

[frenzymadness]

Just linking another PR: #969

[hynek]

fixed by #969

[vstinner]

Great, thanks!