Kill __subclasscheck__

[gvanrossum]

Mark Shannon wants me to drop __subclasscheck__ from all type objects (Any, Union etc.). This is somewhat major surgery since it is used by Union simplification and for various type checks. See also #133 and #135.

[ilevkivskyi]

This is a bit sad. People would want to do some interactive experimentation to see what is issubclass(int, Union[int, str]), but probably there are some good reasons behind this decision.

[1st1]

Maybe it is a good thing, at least for 3.5. This will effectively force type objects to be used only in annotations, which gives us more room to tweak their behaviour before 3.6.

[ilevkivskyi]

Maybe one could introduce a helper function acceptable(obj, hint) in typing.py that tells whether obj is OK for type hint. Like acceptable(42, Union[int, str]) == True.

[gvanrossum]

Here's what Mark wrote about the subject:

 My remaining concern is the isinstance and issubclass support.
There is a real different difference between whether a value is a member of a type and whether it is an instance of a class.

Support for isinstance and issubclass applied to types just breeds confusion.

As you said, for most users, the distinction between types and classes is subtle. But I think it is important to make people aware that there is a distinction, even if they fully understand it.

For example,
List[int] and List[str] and mutually incompatible types, yet
isinstance([], List[int]) and isinstance([], List[str))
both return true.

Just delete all the __instancecheck__ and __subclasscheck__ methods and I'll be happy.

[1st1]

Maybe one could introduce a helper function acceptable(obj, hint) in typing.py that tells whether obj is OK for type hint. Like acceptable(42, Union[int, str]) == True.

I think that we should leave the task of validating types to external tools (and compilers), not user code. So I'd be -1 on this.

[gvanrossum]

The equivalent of issubclass() should probably be called is_consistent_with() (or some variation of that) to match the terminology used by Jeremy Siek in his Gradual Typing blog post (http://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typing/). I'm not sure what to call the isinstance() equivalent, perhaps is_acceptable_for(). But I plan to ask others to contribute such functions.

[ilevkivskyi]

The intended use would be not to validate types in code but just do some quick interactive tests in REPL. But OK, maybe you are right people anyway will start using it for validation.

[ilevkivskyi]

@gvanrossum, Indeed, to have two such functions would be very cool. Even their presence would underline the distinction between the classes and types.

[NYKevin]

If we're drawing this type/class distinction, PEP 483 should be updated:

We use the terms type and class interchangeably, and we assume type(x) is x.class.

Should I report that as a separate bug?

[gvanrossum]

Here is fine. I've made some updates to PEP 483; it really does use the two words interchangeably, and I don't want to have to update the text too much, but I elaborated that particular sentence (and removed the thing about class, which is irrelevant).

[agronholm]

I'm confused -- as Python 3.5 is now in RC, does this mean the typing types were really left with only __subclasscheck__ working and not __isinstance__ ? I had a working runtime type checker (https://gist.github.com/agronholm/00311c169527442ddd5b) which relied on isinstance() but it broke at some point when __instancecheck__ was disabled. Should I not rely on even __subclasscheck__ then?

[gvanrossum]

You should not rely on subclass checking either. I ran out of time, but PEP 484 is accepted provisionally, which means we can still change the API in later 3.5.x releases or in 3.6.

[agronholm]

Ok, so with both __instancecheck__ and __subclasscheck__ out of commission, how am I to implement a runtime type checker? I would need to be able to somehow detect that a type, say Dict[str, int] is derived from plain Dict, but I can't find any way to do that since issubclass(Dict[str, int], Dict) returns False. Likewise, I'd like to make use of the __extra__ argument to automatically check against the appropriate ABC when one is available, but I'm hesistant to just do getattr(obj, '__extra__', None) since __extra__ could be anything outside the typing module.

[gvanrossum]

The idea is that you would have to write your own functions similar to isinstance() and issubclass() to introspect the objects created at run-time by type annotations. You are right however that the introspection interface has not yet been specified -- I think it would have to be a separate PEP. I would not rely on the __extra__ attribute unless you're willing to see it disappear at any point in the future -- possibly as soon 3.5.1. (However, the reasoning about __extra__ possibly being something else is somewhat faulty -- non-stdlib code should never use dunder names for anything other than their documented meaning, as the stdlib can at any point define a new dunder name without any concern for existing occurrences of that dunder name in user code. So non-stdlib code that uses __extra__ is already broken.)

[agronholm]

Alright, so I guess this means runtime type checking has to wait until that introspection interface has been specified. FWIW, here's what I had before I ran into this problem: https://gist.github.com/agronholm/b06bdba54c73d9b2f1f1

[gvanrossum]

I do encourage you to continue along this path, assuming the introspection API won't be terribly different from what is currently implemented. (In fact I think extra is probably the only thing that might significantly change, as it isn't very principled.) Instead of issubclass() and isinstance() you'll have to code your own, but looking at parameters etc. sounds about right. Have you thought about what to do about requiring that type variables conform across parameters and return values yet? E.g. for def find(x: T, xs: Iterable[T]) -> int: ...; you should allow find(42, [1, 2, 3]) and even find(0, []) but you should reject find(42, ['a', 'b', 'c']) and find(0, [1, 2, 'x']).

[agronholm]

I had thought about type variable checking but I hadn't gotten that far yet. The problem I was stuck with is that if I encounter a type hint like List[int], how am I supposed to programmatically determine that it should be treated as a List? The original class (plain List) does not show up on the MRO so I can't use isinstance() or issubclass() to determine the necessary checking semantics. The only solution I could find was using issubclass() on the undocumented __origin__ attribute, but I am again hesitant to use something that is undocumented.

[gvanrossum]

The best I can recommend is using __origin__ -- if we were to change this attribute there would still have to be some other way to access the same information, and it would be easy to grep your code for occurrences of __origin__. (I'd be less worried about changes to __origin__ than to __extra__.) You may also look at the internal functions _gorg() and _geqv() (these names will not be part of any public API, obviously, but their implementations are very simple and conceptually useful).

[agronholm]

I'll look at them -- thanks!

[agronholm]

OK, delivered: https://pypi.python.org/pypi/typeguard

[gvanrossum]

See also http://bugs.python.org/issue26075 (Union[str, Iterable[int]] returns Iterable[int], which is wrong!)

[prechelt]

I have just uploaded the new version of my runtime type checker, typecheck-decorator 1.3, to PyPI, now with 'typing' support:
https://pypi.python.org/pypi/typecheck-decorator/

Compared to typeguard 1.1.1, it has broader support (except for Callable, which is yet missing) and a more precise documentation of its semantics and limitations.

I struggled enormously with the semantics of TypeVars, because there are several strange issues in the current typing.py. For instance, tg.ItemsView has three generic parameters (tg.T_co, tg.KT, tg.VT_co), the first of which represents the tg.Tuple[tg.KT, tg.VT_co] returned by each call to the iterator -- but how, in general (that is, for user-defined types), is a poor type checker to know this?
Should I report this?
(I found other bugs/oddities, but those have to do with __subclasscheck__...)

[gvanrossum]

I think I'll wait until 3.5.2 is released to get back to this.

[bintoro]

@ilevkivskyi It would be pretty easy to incorporate this in #207, I think.

I'll add one more note here. Assuming parameterized generics now become types (non-classes), then something that needs discussion is what to do about instantiating them.

There are examples of this in the PEP:

IntNode = Node[int]
StrNode = Node[str]
p = IntNode()
q = StrNode()

If Node[int] isn't a class, and instance checking against it is deemed nonsensical, then instantiating it should presumably be an error as well. Subclassing becomes a required step:

class IntNode(Node[int]):
    pass

p = IntNode()

[gvanrossum]

Those examples should work.

However, even though this would be equivalent at run time:

p = Node[int]()

that's disallowed by the PEP and indeed mypy reports it as an error (but
allows the examples from the PEP).

That's all intentional, and the implementation somehow has to deal with it.

[ilevkivskyi]

@bintoro This is clearly the case where practicality beats purity. I think it is not a good idea to force people to write

class IntNode(Node[int]):
    pass

instead of just IntNode = Node[int] only from the point of view of purity. Moreover the type erasure is a widespread behavior in many languages, as said in PEP. So that here I agree with

That's all intentional, and the implementation somehow has to deal with it.

[bintoro]

I think it is not a good idea to force people to write [...]

The point was that the validity of isinstance(x, IntNode) arguably shouldn't depend on the syntax by which IntNode was defined.

However, upon further reflection, I think I have to withdraw this complaint, because the two syntaxes should not be construed as substitutes for each other.

Consider a case where a library provides a class named MessageList. Initially I felt that, ideally, the consumer of the library shouldn't have to care if the class was internally defined as

MessageList = List[Message]

or

class MessageList(List[Message]): ...

But inevitably they do have to care. These things are not the same at all: one is an alias of List and the other is a subclass of List.

Simply preserving isinstance would solve nothing, since type erasure would bring about the seemingly bizarre outcome isinstance([], MessageList) == True in the alias case.

I suppose the correct answer is that a library shouldn't advertise MessageList as a class if it's really just a type alias. In general, exposing type aliases requires care because then the consumer must be aware of how they work.

[ilevkivskyi]

I suppose the correct answer is that a library shouldn't advertise MessageList as a class if it's really just a type alias. In general, exposing type aliases requires care because then the consumer must be aware of how they work.

This is a good point, If you think that the language in the PEP is too loose, then propose a modification to tighten it, so that this is clear.

[AdamWill]

I just got here via https://stackoverflow.com/questions/49171189/whats-the-correct-way-to-check-if-an-object-is-a-typing-generic and https://bugzilla.redhat.com/show_bug.cgi?id=1598574 . The SO thread notes this quote from @gvanrossum above:

"The best I can recommend is using __origin__ -- if we were to change this attribute there would still have to be some other way to access the same information, and it would be easy to grep your code for occurrences of __origin__."

Well, it seems you did change it. As related in the RH bug, in Python 3.7, typing.Tuple[int, str].__origin__ is no longer the class typing.Tuple, but the class tuple. Per the quote above, there ought now to be "some other way to access the same information", but I'm buggered if I can find it - any hints? So far the best I came up with is a hideous string comparison (see https://bugzilla.redhat.com/show_bug.cgi?id=1598574#c1 ). Thanks!

[JelleZijlstra]

You should probably use https://github.com/ilevkivskyi/typing_inspect.

[AdamWill]

@JelleZijlstra thanks, we'd have to package it and add it to anaconda's deps...

edit: also, the get_origin docstring seems to suggest it doesn't work for several types we use, like List and Dict.

[ilevkivskyi]

You can get the actual typing object using something like this assert typing.__dict__[Tuple[int, str]._name] is typing.Tuple. Just be sure to check ._name is not None. It is None for most things, except List, Tuple, and few others.

But in long term perspective, I would not recommend depending on private internal APIs. If you are interested in inspection of typing objects, you can submit a feature request for this, it looks like more people than expected are interested in this. Maybe we can provide something in Python 3.8.

[AdamWill]

@ilevkivskyi thanks for the suggestion!

I mean, it's possible this code is doing something 'silly' in the first place; I didn't write it and I haven't yet looked at whether what it's doing actually makes a lot of sense or if there'd be some better alternative. For now I was just focusing on the direct issue as it's an urgent problem for us (Fedora).

[ilevkivskyi]

For now I was just focusing on the direct issue as it's an urgent problem for us (Fedora).

OK, I see. Let me know if you need more help with this.