Self types in generic classes

[JukkaL]

Do something reasonable with self types (e.g., self: T) in generic classes.

This is continuation of work in #2193.

[jboning]

A motivating example is typeshed's AbstractSet:

class AbstractSet(Sized, Iterable[_T_co], Container[_T_co], Generic[_T_co]):
    def __and__(self, s: AbstractSet[Any]) -> AbstractSet[_T_co]: ...
    def __or__(self, s: AbstractSet[_T]) -> AbstractSet[Union[_T_co, _T]]: ...

With #2193 we can annotate __and__, and it works beautifully:

    def __and__(self: SelfT, s: AbstractSet[Any]) -> SelfT: ...

def f(s, fs):
    # type: (MutableSet[str], FrozenSet[str]) -> None
    reveal_type(s.__and__)  # Revealed type is 'def (typing.AbstractSet[Any]) -> typing.MutableSet[builtins.str]'
    reveal_type(fs.__and__)  # Revealed type is 'def (typing.AbstractSet[Any]) -> typing.FrozenSet[builtins.str]'

But, we don't have the tools to give __or__ a similarly specific type. To do this, we need some way to talk about the "generic" part of the self type.

[elazarg]

I think this issue is resolved - we can talk about the generic part if the self type. Or maybe I don't completely understand the issue.

[jboning]

What would the annotated version of AbstractSet's __or__ method be using self types? I think that's a good illustrative example.

[elazarg]

I admit all these underscores confuse me, but maybe you wanted something like this?
(I am writing AS instead of AbstractSet to make it easier for me to read :) )

class AS(Generic[_T_co]):
    def __and__(self: AS[T1], s: AS[T2]) -> AS[T1]: ...
    def __or__ (self: AS[T1], s: AS[T2]) -> AS[Union[T1, T2]]: ...

[elazarg]

(actually I think __and__ should return AS[Intersect[T1, T2]] if we had something like that)

[jboning]

Those annotations aren't as specific as I'd like--I want to be able to use self types to select the right return type depending on the subclass of AbstractSet. So FrozenSet.__and__ and FrozenSet.__or__ should return some kind of FrozenSet.

I'm imagining something like this, except that I don't think it's possible to write SelfT[T1]:

class AS(Generic[_T_co]):
    def __and__(self: SelfT[T1], s: AS[T2]) -> SelfT[T1]: ...
    def __or__ (self: SelfT[T1], s: AS[T2]) -> SelfT[Union[T1, T2]]: ...

[ilevkivskyi]

SelfT[T1] is this higher order kind? :-)

[JukkaL]

@jboning Yes, that is what this issue was originally about.

However, it's unclear if we'd want that annotation for AbstractSet. For example, consider this example:

class C(set): pass
print(type(C([1]) | C([2]))  # <class 'set'> (not C!)

Arguably AbstractSet[T] already has the correct return type, and if some subclasses of AbstractSet return more specific types, they should be indicated in the stubs/annotations for those classes, not in AbstractSet. Maybe MutableSet should override methods such as __or__ to return MutableSet instead.

[jboning]

Huh. I agree that then it makes sense for MutableSet and FrozenSet to override with their own, more specific types for these methods. I'll go post a typeshed change for that.

Is it still a useful example of the sort of annotation one might want to be able to write?

[ilevkivskyi]

Raising priority since there were quite a few requests for this recently.

[JukkaL]

Decreasing priority back to normal since this seems like a somewhat complicated feature and requires an addition to mypy_extensions (and/or PEP 484).