abstract class instantiable when subclassing built-in types

[thet]

BPO	5996
Nosy	@aleaxit, @rhettinger, @terryjreedy, @pitrou, @benjaminp, @jwilk, @merwok, @bitdancer, @thet, @durban, @eltoder, @ericsnowcurrently, @soltysh, @MojoVampire, @poleto, @zhangyangyu, @nathanielmanistaatgoogle, @DimitrisJim, @remilapeyre, @2xB
Files	abc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2009-05-11.14:06:01.619>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'abstract class instantiable when subclassing built-in types'
updated_at = <Date 2020-10-30.23:49:27.541>
user = 'https://github.com/thet'

bugs.python.org fields:

activity = <Date 2020-10-30.23:49:27.541>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2009-05-11.14:06:01.619>
creator = 'thet'
dependencies = []
files = ['21600']
hgrepos = []
issue_num = 5996
keywords = ['patch']
message_count = 16.0
messages = ['87574', '87576', '87854', '113698', '133440', '138411', '140108', '140114', '266704', '279479', '279481', '279483', '279484', '279499', '299815', '335337']
nosy_count = 24.0
nosy_names = ['aleax', 'rhettinger', 'terry.reedy', 'pitrou', 'nadeem.vawda', 'benjamin.peterson', 'jwilk', 'eric.araujo', 'r.david.murray', 'cvrebert', 'thet', 'daniel.urban', 'eltoder', 'eric.snow', 'maciej.szulik', 'josh.r', 'luiz.poleto', 'xiang.zhang', 'Kevin Shweh', 'Nathaniel Manista', 'Jim Fasarakis-Hilliard', 'remi.lapeyre', 'Jon McMahon', '2xB']
pr_nums = []
priority = 'low'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5996'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[thet]

when declaring a abstract base class with an abstract property or method
and subclassing from dict, the class is instantiable (instanceable?).

>>> import abc
>>> class A(object):
...     __metaclass__ = abc.ABCMeta
...     @abc.abstractproperty
...     def abstract(self): return True
... 
>>> a = A()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class A with abstract methods abstract
>>> 
>>> class A2(dict):
...     __metaclass__ = abc.ABCMeta
...     @abc.abstractproperty
...     def abstract(self): return True
... 
>>> a2 = A2()
>>> 

although, when using the dict definition from __builtin__.pi directly,
the abc behaves like expected. but this may be a bug in the
c-implementation from dict.

platform:
Python 2.6.2 (r262:71600, Apr 25 2009, 21:56:41)
[GCC 4.3.2] on linux2

[pitrou]

Also happens with other builtin types such as tuple. It's probably
related to the way subclasses of those types are instantiated, bypassing
the normal metaclass mechanism.

[terryjreedy]

I presume you claim the dict example to be a bug in relation to "A class
that has a metaclass derived from ABCMeta cannot be instantiated unless
all of its abstract methods and properties are overridden."

There is the same difference with @abstractproperty
Windows, 3.0.1

class C(metaclass=abc.ABCMeta):
	@abc.abstractmethod
	def f(self): return True

class C2(dict,metaclass=abc.ABCMeta):
	@abc.abstractmethod
	def f(self): return True

c2=C2()
print(c2.f())

c=C()
# prints
True
...
TypeError: Can't instantiate abstract class C with abstract methods f

[durban]

I think, that the reason is that, object.__new__ checks, if the class is instantiable (object_new in Objects/typeobject.c ). dict.__new__ (and tuple.__new__, and I guess the __new__ method of other built-in types) doesn't call object.__new__, but user defined types typically either doesn't have a __new__, or call object.__new__ from it (directly or with super).

[eltoder]

This patch fixes the problem by moving the check from object_new to PyType_GenericAlloc. The check is very cheap, so this should not be an issue.

[eltoder]

Anyone has any thoughts on this?

[rhettinger]

IIRC, classes weren't supposed to be able inherit from two classes that had different metaclasses (since differing metaclasses don't necessarily play well with one another).

[eltoder]

They are, when there's a most specific metaclass -- the one which is a subclass of all others (described here http://www.python.org/download/releases/2.2.3/descrintro/#metaclasses, implemented here http://hg.python.org/cpython/file/ab162f925761/Objects/typeobject.c#l1956). Since ABCMeta is a subclass of type this holds.

Also, in the original example there's no multiple inheritance at all.

[zhangyangyu]

I think subclassing builtin types and making it abstract is rare. And when there is a need, we can mimic this in application level (this may also apply to types having custom __new__):

In [2]: class CustomDict(dict, metaclass=abc.ABCMeta):
...: def __new__(cls, *args, **kwargs):
...: if getattr(cls, '__abstractmethods__', None):
...: raise TypeError
...: return super().__new__(cls, *args, **kwargs)
...: @abc.abstractmethod
...: def f(self):
...: pass

Adding the abstract class checking in tp_alloc or builtin types' tp_new maybe degrade performance.

Is it necessary to add this support?

[gvanrossum]

Honestly let's just forget about this.

[nathanielmanistaatgoogle]

Wait, really? My report came out of a real bug that I had in my system and shipped to my users; it wasn't academic or contrived at all.

[gvanrossum]

Where did you report that? I don't see your name on this bug -- it has a patch that's been unapplied for 5 years, so I doubt it's very important.

[gvanrossum]

Oh sorry. I received the emails in a strange order. I guess it can stay open.

[bitdancer]

My apologies, I added Nathaniel to nosy here when I closed the duplicate, but forgot to add a link to the closed issue: bpo-28537.

[bitdancer]

Closed bpo-31127 as a duplicate of this one.

[MojoVampire]

Closed bpo-35958 as a duplicate of this issue (and updated the title, since clearly the problem is not specific to dict).

Patch probably needs to be rebased/rewritten against latest trunk (given it dates from Mercurial days).

[Yc7521]

The reason for this bug is that the creation of the subclass avoids the invocation of the 'object_new' method (e.g. for built-in types or when __new__ is overridden without calling super().__new__).
It seems that considering adding an abstract flag check in tp_new_wrapper could solve this issue.

some links:
object_new
tp_new_wrapper

A simple code for enforcing ABC check to correct the behavior. (only for fixing inheriting built-in types)

class MyABC(ABC):
    def __new__(cls, *args, **kwargs):
        if len(cls.__abstractmethods__) > 0:
            raise TypeError("Can't instantiate abstract class MyABC with abstract methods test")
        return super().__new__(cls, *args, **kwargs)


ABC = MyABC

[gvanrossum]

I got as far as a repro, but this morning my Pooh brain doesn't follow the root cause. I'll come back to it.

import abc
class A(object, metaclass=abc.ABCMeta):
    @abc.abstractproperty
    def pooh(self): return True

class A2(dict, metaclass=abc.ABCMeta):
    @abc.abstractproperty
    def pooh(self): return True

a2 = A2()  # Passes
a = A()  # Can't instantiate abstract class A without an implementation for abstract method 'pooh'

[gvanrossum]

The reason for this bug is that the creation of the subclass avoids the invocation of the 'object_new' method (e.g. for built-in types or when __new__ is overridden without calling super().__new__).

You're right that it is because object_new isn't called when the base class is a builtin.

I'm skeptical of Python-level code whose __new__ doesn't call super().__new__, because how would it come up with an object to return at all?

I'm also skeptical of putting anything in tp_new_wrapper, since that isn't called by e.g. dict_new nor by most other builtin types' tp_new callback.

One of the duplicates reported the same problem when inheriting from Exception, which doesn't even override tp_new, so there's something more going on there that I haven't tracked down yet.

My gut feeling still says that the exception you get for instantiating an abstract class is intended to be helpful, but not a feature you can depend on, but that's at least in part because I don't know how to solve this yet without slowing down e.g. dict creation, which is pretty time sensitive.

[gvanrossum]

I'm tempted to close this as won't fix, unless there is further discussion. (A PR would also be helpful to focus the discussion; I'm not considering old patch files from bpo.)

[Yc7521]

Maybe someday someone will come up with a relatively good solution. At least this is indeed a bug.

[gvanrossum]

Thanks, it is indeed a bug, but we won't fix it (yet).