Introduce an Intersection

[ilevkivskyi]

This question has already been discussed in #18 long time ago, but now I stumble on this in a practical question: How to annotate something that subclasses two ABC's. Currently, a workaround is to introduce a "mix" class:

from typing import Iterable, Container

class IterableContainer(Iterable[int], Container[int]):
    ...

def f(x: IterableContainer) -> None: ...

class Test(IterableContainer):
    def __iter__(self): ...
    def __contains__(self, item: int) -> bool: ...

f(Test())

but mypy complains about this

error: Argument 1 of "__contains__" incompatible with supertype "Container"

But then I have found this code snippet in #18

def assertIn(item: T, thing: Intersection[Iterable[T], Container[T]]) -> None:
    if item not in thing:
        # Debug output
        for it in thing:
            print(it)

Which is exactly what I want, and it is also much cleaner than introducing an auxiliary "mix" class. Maybe then introducing Intersection is a good idea, @JukkaL is it easy to implement it in mypy?

[JukkaL]

Mypy complains about your code because __contains__ should accept an argument of type object. It's debatable whether this is the right thing to do, but that's how it's specified in typeshed, and it allows Container to be covariant.

I'm worried that intersection types would be tricky to implement in mypy, though conceptually it should be feasible. I'd prefer supporting structural subtyping / protocols -- they would support your use case, as IterableContainer could be defined as a protocol (the final syntax might be different):

from typing import Iterable, Container

class IterableContainer(Iterable[int], Container[int], Protocol):
    ...

def f(x: IterableContainer) -> None: ...

class Test:
    def __iter__(self): ...
    def __contains__(self, item: int) -> bool: ...

f(Test())  # should be fine (except for the __contains__ argument type bit)

[ilevkivskyi]

It would be really cool to implement protocols. Still, in this case intersection could be added as a "syntactic sugar", since there would be a certain asymmetry: Assume you want a type alias for something that implements either protocol, then you write:
IterableOrContainer = Union[Iterable[int], Container[int]]
But if you want a type alias for something that implements both, you would write:
class IterableContainer(Iterable[int], Container[int], Protocol): ...
I imagine such asymmetry could confuse a novice. Intersection could then be added (very roughly) as:

class _Intersection:
    def __getitem__(self, bases):
        full_bases = bases+(Protocol,)
        class Inter(*full_bases): ...
        return Inter

Intersection = _Intersection()

then one could write:
IterableContainer = Intersection[Iterable[int], Container[int]]

[JukkaL]

Intersection[...] gets tricky once you consider type variables, callable types and all the other more special types as items. An intersection type that only supports protocols would be too special purpose to include, as it's not even clear how useful protocols would be.

[ilevkivskyi]

I understand what you mean. That could be indeed tricky in general case.

Concerning protocols, I think structural subtyping would be quite natural for Python users, but only practice could show whether it will be useful. I think it will be useful.

[gvanrossum]

This keeps coming up, in particular when people have code that they want to support both sets and sequences -- there is no good common type, and many people believe Iterable is the solution, but it isn't (it doesn't support __len__).

[ilevkivskyi]

I think Intersection is a very natural thing (at least if one thinks about types as sets, as I usually do). Also, it naturally appears when one wants to support several ABCs/interfaces/protocols.

I don't think that one needs to choose between protocols and Intersection, on the contrary they will work very well in combination. For example, if one wants to have something that supports either "old-style" reversible protocol (i.e. has __len__ and __iter__ methods) or "new-style" (3.6+) reversible protocol (i.e. has __reversed__ method), then the corresponding type is Union[Reversible, Intersection[Sized, Iterable]].

It is easy to add Intersection to PEP 484 (it is already mentioned in PEP 483) and to typing.py, the more difficult part is to implement it in mypy (although @JukkaL mentioned this is feasible).

[gvanrossum]

For cross-reference from #2702, this would be useful for type variables, e.g. T = TypeVar('T', bound=Intersection[t1, t2]).

[jeffkaufman]

Intersection[FooClass, BarMixin] is something I found myself missing today

[matthiaskramm]

If we had an intersection class in typing.py, what would we call it?

Intersection is linguistically symmetric with Union, but it's also rather long.
Intersect is shorter, but it's a verb. Meet is the type-theoretic version and also nice and short, but, again, you'd expect Union to be called Join if you call Intersection Meet.

[jeffkaufman]

As a data point, I first looked for Intersection in the docs.

[ilevkivskyi]

Just as a random idea I was thinking about All (it would be more clear if Union would be called Any, but that name is already taken). In general, I don't think long name is a big issue, I have seen people writing from typing import Optional as Opt or even Optional as O depending on their taste. Also generic aliases help in such cases:

T = TypeVar('T')
CBack = Optional[Callable[[T], None]]

def process(data: bytes, on_error: CBack[bytes]) -> None:
    ...

[mitar]

I just opened #483 hoping for exactly the same thing. I literally named it the same. I would be all for Intersection or All to allow to require a list of base classes.

[ilevkivskyi]

Requests for Intersection appear here and there, maybe we should go ahead and support it in mypy? It can be first put in mypy_extensions or typing_extensions. It is a large piece of work, but should not be too hard. @JukkaL @gvanrossum what do you think?

[gvanrossum]

I think we should note the use cases but not act on it immediately -- there are other tasks that IMO are more important.

[ilevkivskyi]

@gvanrossum

I think we should note the use cases but not act on it immediately -- there are other tasks that IMO are more important.

OK, I will focus now on PEP 560 plus related changes to typing. Then later we can get back to Intersection, this can be a separate (mini-) PEP if necessary.

Btw, looking at the milestone "PEP 484 finalization", there are two important issues that need to be fixed soon: #208 (str/bytes/unicode) and #253 (semantics of @overload). The second will probably require some help from @JukkaL.

[JukkaL]

I agree that now's not the right time to add intersection types, but it may make sense later.

[Kentzo]

(been redirected here from the mailing list)

I think the Not type needs to be added in addition to Intersection:

Intersection[Any, Not[None]]

Would mean anything but None.

[rinarakaki]

How about the expression Type1 | Type2 and Type1 & Type2 alternative to Union and Intersection respectively.

example:

x: int & Const = 42

[emmatyping]

@rnarkk these have already been proposed many times, but have not been accepted.

[JelleZijlstra]

The Not special form hasn't been proposed before to my knowledge. I suppose you could equivalently propose a Difference[Any, None] type.

What's the use case for that? It's not something I've ever missed in a medium-sized typed codebase at work and in lots of work on typeshed.

[nineteendo]

The specification was moved: https://github.com/CarliJoy/intersection_examples/blob/main/docs/specification.rst

[ViktorSky]

Can I ask why this has not been implemented yet? What is the problem with the Intersection concept?

[superbobry]

@ViktorSky if your question is about a specific type checker, you might want to open a bug on their bug tracker instead.

Otherwise, the short answer is that there is no PEP/specification ready for type checkers to adopt.

[CarliJoy]

@ViktorSky well the devil is in the details. Currently nobody was able to create PEP with a sound definition of Intersections within the existing python typing system. Especially handling Intersections with Any is a problem as it is ill defined and even if you fully type your code, the type checker could still internally find an Any to intersect with.

If you want to dig into details see CarliJoy/intersection_examples#38 and the other still open issues.

A current draft is discussed here. You are invited to contribute there.

BTW: It turned out that github is a rather poor system to discuss / keep track of issue as it misses quite a lot of basic features like answering in threads, adding related issues/block etc.

[mark-todd]

Thanks @CarliJoy - yeah it turns out there's a lot of considerations to developing a fully fledged PEP - at the moment the discussion has been quite focused on whether to go for an ordered or unordered variant, as there are pros and cons to each. Please anyone feel free to have a review of the PR - there's been some discussion about whether we should go back to the unordered approach, but we don't have a PEP for this yet.

[ViktorSky]

@ViktorSky well the devil is in the details. Currently nobody was able to create PEP with a sound definition of Intersections within the existing python typing system. Especially handling Intersections with Any is a problem as it is ill defined and even if you fully type your code, the type checker could still internally find an Any to intersect with.

If you want to dig into details see CarliJoy/intersection_examples#38 and the other still open issues.

A current draft is discussed here. You are invited to contribute there.

BTW: It turned out that github is a rather poor system to discuss / keep track of issue as it misses quite a lot of basic features like answering in threads, adding related issues/block etc.

Thank you very much, I will be looking for what could be done to solve intersections with Any

[tylerlaprade]

@mark-todd , ordered sounds far too easy to make mistakes with. What benefits does it bring?

[CarliJoy]

@tylerlaprade pls follow the discussions in this repo. This issue is not suitable for further discussions. Don't post here unless there is general progress in the intersection.

In short the ordered nature was thought about while handling the Any case. See this issue.