Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-102433: Use inspect.getattr_static in typing._ProtocolMeta.__instancecheck__ #103034

Merged
merged 11 commits into from
Apr 2, 2023

Conversation

AlexWaygood
Copy link
Member

@AlexWaygood AlexWaygood commented Mar 25, 2023

Use getattr_static rather than hasattr in _ProtocolMeta.__instancecheck__, in order to avoid unexpectedly evaluating descriptors and __getattr__ methods in the process of calling isinstance() against runtime-checkable protocols.

This slows down isinstance() checks against runtime-checkable protocols by around 46% (on my machine), but the consensus in python/typing#1363 is that the performance penalty is worth it because:

  • Any isinstance() check causing an expensive (or otherwise badly behaved) property or __getattr__ method to be evaluated is pretty surprising behaviour.
  • Bringing the runtime's understanding of runtime-checkable protocols closer to the way static type checkers understand these classes is highly desirable.
  • Runtime-checkable protocols should not be used for performance-critical code anyway (this is now clearly documented following Typing: Document performance pitfalls of protocols decorated with @runtime_checkable #102936.)

In various subtle edge cases, this change will mean that some classes that used to be considered structural "subclasses" of a certain runtime-checkable protocol will no longer be considered subclasses of that protocol (and vice versa). As such, this can't be backported. (Perhaps we should consider documenting the existing behaviour for 3.11 and 3.10.)

Note that most discussion has actually taken place in a python/typing issue rather than a CPython issue:

@AlexWaygood AlexWaygood requested a review from carljm March 25, 2023 17:08
@AlexWaygood AlexWaygood changed the title gh-102344: Use inspect.getattr_static in typing._ProtocolMeta.__instancecheck__ gh-102433: Use inspect.getattr_static in typing._ProtocolMeta.__instancecheck__ Mar 25, 2023
@AlexWaygood AlexWaygood added topic-typing 3.12 bugs and security fixes stdlib Python modules in the Lib dir labels Mar 25, 2023
@AlexWaygood
Copy link
Member Author

AlexWaygood commented Mar 25, 2023

I'd be very interested to know if any maintainers of libraries that use type annotations at runtime have any concerns about this proposed change. Off the top of my head, I can think of:

@gvanrossum
Copy link
Member

I’m on vacation, please find another reviewer.

@gvanrossum gvanrossum removed their request for review March 25, 2023 21:09
@samuelcolvin
Copy link
Contributor

Thanks for tagging me, I'll discuss with the pydantic team and respond in a few days.

@leycec
Copy link

leycec commented Mar 26, 2023

Thanks so much for the apropos ping, @AlexWaygood. You do the good typing work! Sadly, this...

This slows down isinstance() checks against runtime-checkable protocols by around 46% (on my machine)...

...is a non-starter. That's measurably unacceptable degradation. The existing Protocol API works. There's no demonstrable justification for both breaking that implementation in backwards-incompatible edge cases and imposing additional runtime costs while doing so.

Python ≥ 3.11's renewed focus on runtime performance is germane here. Efficiency is no longer orthogonal to our goals in the Python ecosystem. Efficiency is increasingly a goal in and of itself. Do not sacrifice quantifiable speed at the golden altar of well-intended but ultimately futile attempts to force parity between static and runtime type-checkers.

Static and runtime type-checkers will inevitably disagree. These are profoundly different understandings of what type-checking even means. Reducing the scope of that disagreement is useful – but preserving (or ideally improving) runtime performance is substantially more useful.

Static type-checking serves Python's runtime needs by improving the robustness and reliability of Python at runtime. Static analysis is subordinate to runtime. Runtime always comes first. Not last. We're all here for what Python accomplishes at runtime. To paraphrase President Truman:

The buck stops at runtime.

@Tinche
Copy link
Contributor

Tinche commented Mar 26, 2023

Should be fine for cattrs, we have separate build and run stages for the de/serializers.

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Mar 26, 2023

Thanks for the feedback @leycec! To be clear, I fully agree with your sentiment here, and have no wish to make the CPython runtime subservient to the needs of static typing:

Static type-checking serves Python's runtime needs by improving the robustness and reliability of Python at runtime. Static analysis is subordinate to runtime. Runtime always comes first. Not last. We're all here for what Python accomplishes at runtime.

I realise that in my summary of this PR at the top, I maybe should have been clearer about the bug that this PR is trying to fix. Ultimately this PR is first and foremost about fixing unexpected behaviour at runtime. Improving consistency between how static type checkers and the runtime see certain classes is a "nice to have" that comes along with the fix.

Here is a demonstration of the user-visible bug that this PR is trying to fix (which has been reported to the CPython repo multiple times, so is clearly surprising behaviour for users). Properties, other descriptors, and __getattr__ methods are all currently "called" if you compare an object against a runtime-checkable protocol using isinstance(). This is the issue that this PR is trying to fix.

REPL session demonstrating the bug as it exists on CPython `main`
>>> import time
>>> from typing import runtime_checkable, Protocol
>>> @runtime_checkable
... class HasX(Protocol):
...     x: int
...
>>> class A:
...     @property
...     def x(self):
...         raise RuntimeError('no')
...
>>> class B:
...     @property
...     def x(self):
...         time.sleep(3600)
...         return 42
...
>>> class C:
...     def __getattr__(self, name):
...         raise RuntimeError('no')
...
>>> isinstance(A(), HasX)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1968, in __instancecheck__
    if all(hasattr(instance, attr) and
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1968, in <genexpr>
    if all(hasattr(instance, attr) and
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "<stdin>", line 4, in x
RuntimeError: no
>>> isinstance(C(), HasX)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1968, in __instancecheck__
    if all(hasattr(instance, attr) and
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python311\Lib\typing.py", line 1968, in <genexpr>
    if all(hasattr(instance, attr) and
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "<stdin>", line 3, in __getattr__
RuntimeError: no
>>> isinstance(B(), HasX)  # this will take an hour to evaluate

I don't like (at all) that this results in a 46% performance degradation for simple isinstance() checks against runtime-checkable protocols, and I suggested a few workarounds in python/typing#1363 that would still fix the bug for most users of runtime-checkable protocols, while having a much smaller performance degradation (e.g. python/typing#1363 (comment)). These were criticized, however, on the grounds that:

  • they weren't particularly "principled" solutions, which might have bugs in subtle edge cases
  • they felt like premature optimizations, since isinstance() checks against runtime-checkable protocols are abysmally slow anyway, so users should probably not be using runtime-checkable protocols for performance-critical code.

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Mar 26, 2023

There are three options here that I can see:

  1. Go with this solution (take the performance hit for now, but see if we can optimise _ProtocolMeta.__instancecheck__ and/or inspect.getattr_static in followup PRs).
  2. Go with a "less principled" solution that doesn't use inspect.getattr_static, and therefore has a smaller performance degradation, but might have bugs in subtle edge cases.
  3. Declare the bug as "wontfix"; document clearly in the typing documentation that you should be careful about comparing instances of a class X against a runtime-checkable protocol if class X has properties with side effects, which would be triggered during the course of an isinstance() check against a runtime-checkable protocol.

@agronholm
Copy link
Contributor

agronholm commented Mar 26, 2023

IMHO, correct behavior should always come first, then maybe good performance.

As far as typeguard is concerned, I don't see a problem here.

@samuelcolvin
Copy link
Contributor

I'm pretty sure this shouldn't affect pydantic since we tend to avoid isinstance() checks unless they're unavoidable, we also don't have a validator for Protocol, and if we do, I guess we would need our own logic.

I'm therefore neutral on it.

FWIW (and it's not worth much since it's probably too late), my personal opinion is that torturing isinstance to provide partial runtime type checking is a mistake - it would be better if isinstance just checked that the first arg's type inherits from the second arg - no magic, predicable behaviour, predictably fast. Then python could provide a second method for more powerful type checking with worse performance and more magic. I would argue that the decision to contort isinstance is the route cause headaches like this, and many more.

@AlexWaygood
Copy link
Member Author

FWIW (and it's not worth much since it's probably too late), my personal opinion is that torturing isinstance to provide partial runtime type checking is a mistake - it would be better if isinstance just checked that the first arg's type inherits from the second arg - no magic, predicable behaviour, predictably fast. Then python could provide a second method for more powerful type checking with worse performance and more magic. I would argue that the decision to contort isinstance is the route cause headaches like this, and many more.

FWIW I agree fully with this, and I'm quite glad that protocols are by default not runtime-checkable. But, as you say, it's almost certainly too late to change this now without going through a painful deprecation period that (in my opinion) would cause more trouble than it's worth.

@agronholm
Copy link
Contributor

I agree with you both. The checking of run-time protocols would probably be best left to run-time type checkers, but getting rid of run-time checkable protocols would be a PITA at this point.

typeguard is lazy and uses instance() on run-time protocols, and doesn't currently provide support for static protocols. That could be changed with enough motivation and free time though.

@carljm
Copy link
Member

carljm commented Mar 27, 2023

Note the recent comment on the issue reinforcing that the current behavior of @runtime_checkable executing arbitrary properties causes (and hides) real bugs.

There are three options here that I can see:

  1. Go with this solution (take the performance hit for now, but see if we can optimise _ProtocolMeta.__instancecheck__ and/or inspect.getattr_static in followup PRs).
  2. Go with a "less principled" solution that doesn't use inspect.getattr_static, and therefore has a smaller performance degradation, but might have bugs in subtle edge cases.
  3. Declare the bug as "wontfix"; document clearly in the typing documentation that you should be careful about comparing instances of a class X against a runtime-checkable protocol if class X has properties with side effects, which would be triggered during the course of an isinstance() check against a runtime-checkable protocol.

I agree that these are the options.

I think the advice we would have to document in the case of (3) is basically un-followable. If you are doing isinstance(x, SomeProtocol) in your code, it's because you don't already know the type of x, so there's no way you can "be careful" that x isn't of a type with dangerous or costly executable properties.

I agree with @agronholm that we should prioritize correct behavior first, and then seek to optimize it. IMO correct behavior (purely from a runtime perspective, ignoring static typing) is that we should not execute arbitrary descriptor code in an isinstance check, and inspect.getattr_static is how to reliably do that. So I favor (1).

@AlexWaygood
Copy link
Member Author

I'm going to merge this once the CI passes.

We've already merged two optimisation PRs into main, which, taken together, basically make up for the performance hit that this PR gives:

There are two more PRs (which are slightly riskier) that have yet to be decided on:

If anybody has any opinions on either of those, feedback is very welcome! Once decisions have been taken on the two open PRs, I'll create another PR adding a separate NEWS entry summarising all of the performance work that's been done (and the internal changes to typing.py that were needed for the performance optimisations).

I'd also like to look into optimising inspect.getattr_static itself, as well, since the core reason for the performance hit here is obviously that inspect.getattr_static is very slow compared to hasattr.

@AlexWaygood AlexWaygood merged commit 6d59c9e into python:main Apr 2, 2023
@AlexWaygood AlexWaygood deleted the runtime-protocols-getattr_static branch April 2, 2023 13:22
@AlexWaygood
Copy link
Member Author

Many thanks to everybody who took their time to provide feedback on this PR, I really appreciate it!

@tiangolo
Copy link
Contributor

tiangolo commented Apr 2, 2023

Sorry for my delay replying here. 🙈 Thanks a lot for pinging me. 🙏

As FastAPI and my other tools heavily depend and interact with Pydantic (or copy its ideas), if this doesn't affect Pydantic (as Samuel confirmed above), it doesn't affect FastAPI and friends. 🚀

gaogaotiantian pushed a commit to gaogaotiantian/cpython that referenced this pull request Apr 8, 2023
warsaw pushed a commit to warsaw/cpython that referenced this pull request Apr 11, 2023
kraj pushed a commit to YoeDistro/poky that referenced this pull request Jun 1, 2023
Changelog:
============
-  Fix use of @deprecated on classes with __new__ but no __init__.
-  Fix regression in version 4.6.1 where comparing a generic class against a
   runtime-checkable protocol using isinstance() would cause AttributeError to
   be raised if using Python 3.7.
-  Change deprecated @runtime to formal API @runtime_checkable in the error message.
-  Fix regression in 4.6.0 where attempting to define a Protocol that was generic
   over a ParamSpec or a TypeVarTuple would cause TypeError to be raised.
-  typing_extensions is now documented at https://typing-extensions.readthedocs.io/en/latest/.
-  Add typing_extensions.Buffer, a marker class for buffer types, as proposed
   by PEP 688. Equivalent to collections.abc.Buffer in Python 3.12.
-  Backport two CPython PRs fixing various issues with typing.Literal:
   python/cpython#23294 and python/cpython#23383. Both CPython PRs were originally,
   and both were backported to Python >=3.9.1, but no earlier.
-  A side effect of one of the changes is that equality comparisons of Literal
   objects will now raise a TypeError if one of the Literal objects being compared
   has a mutable parameter. (Using mutable parameters with Literal is not
   supported by PEP 586 or by any major static type checkers.)
-  Literal is now reimplemented on all Python versions <= 3.10.0.
-  Backport CPython PR 26067, ensuring that isinstance() calls on protocols raise
   TypeError when the protocol is not decorated with @runtime_checkable.
-  Backport several significant performance improvements to runtime-checkable protocols
   that have been made in Python 3.12 (see python/cpython#74690 for details).
-  A side effect of one of the performance improvements is that the members of a
   runtime-checkable protocol are now considered "frozen" at runtime as soon as the
   class has been created. Monkey-patching attributes onto a runtime-checkable
   protocol will still work, but will have no impact on isinstance() checks comparing
   objects to the protocol. See "What's New in Python 3.12" for more details.
-  isinstance() checks against runtime-checkable protocols now use inspect.getattr_static()
   rather than hasattr() to lookup whether attributes exist (backporting python/cpython#103034).
-  Backport the ability to define __init__ methods on Protocol classes, a change
   made in Python 3.11 (originally implemented in python/cpython#31628
-  Speedup isinstance(3, typing_extensions.SupportsIndex) by >10x on Python <3.12.
-  Add typing_extensions versions of SupportsInt, SupportsFloat, SupportsComplex,
   SupportsBytes, SupportsAbs and SupportsRound. These have the same semantics as
   the versions from the typing module, but isinstance() checks against the
   typing_extensions versions are >10x faster at runtime on Python <3.12.
-  Add __orig_bases__ to non-generic TypedDicts, call-based TypedDicts, and call-based NamedTuples.
-  Add typing_extensions.get_original_bases, a backport of types.get_original_bases,
   introduced in Python 3.12 (CPython PR python/cpython#101827, originally
-  This function should always produce correct results when called on classes
   constructed using features from typing_extensions.
-  Constructing a call-based TypedDict using keyword arguments for the fields
   now causes a DeprecationWarning to be emitted. This matches the behaviour
   of typing.TypedDict on 3.11 and 3.12.
-  Backport the implementation of NewType from 3.10 (where it is implemented as
   a class rather than a function). This allows user-defined NewTypes to be pickled.
-  Fix tests and import on Python 3.12, where typing.TypeVar can no longer be subclassed.
-  Add typing_extensions.TypeAliasType, a backport of typing.TypeAliasType from PEP 695.
-  Backport changes to the repr of typing.Unpack that were made in order
   to implement PEP 692 (backport of python/cpython#104048).

(From OE-Core rev: 2b1d07c7deb4f0247765bc737fb11a1747143edf)

Signed-off-by: Wang Mingyu <[email protected]>
Signed-off-by: Alexandre Belloni <[email protected]>
rpurdie pushed a commit to yoctoproject/poky that referenced this pull request Jun 2, 2023
Changelog:
============
-  Fix use of @deprecated on classes with __new__ but no __init__.
-  Fix regression in version 4.6.1 where comparing a generic class against a
   runtime-checkable protocol using isinstance() would cause AttributeError to
   be raised if using Python 3.7.
-  Change deprecated @runtime to formal API @runtime_checkable in the error message.
-  Fix regression in 4.6.0 where attempting to define a Protocol that was generic
   over a ParamSpec or a TypeVarTuple would cause TypeError to be raised.
-  typing_extensions is now documented at https://typing-extensions.readthedocs.io/en/latest/.
-  Add typing_extensions.Buffer, a marker class for buffer types, as proposed
   by PEP 688. Equivalent to collections.abc.Buffer in Python 3.12.
-  Backport two CPython PRs fixing various issues with typing.Literal:
   python/cpython#23294 and python/cpython#23383. Both CPython PRs were originally,
   and both were backported to Python >=3.9.1, but no earlier.
-  A side effect of one of the changes is that equality comparisons of Literal
   objects will now raise a TypeError if one of the Literal objects being compared
   has a mutable parameter. (Using mutable parameters with Literal is not
   supported by PEP 586 or by any major static type checkers.)
-  Literal is now reimplemented on all Python versions <= 3.10.0.
-  Backport CPython PR 26067, ensuring that isinstance() calls on protocols raise
   TypeError when the protocol is not decorated with @runtime_checkable.
-  Backport several significant performance improvements to runtime-checkable protocols
   that have been made in Python 3.12 (see python/cpython#74690 for details).
-  A side effect of one of the performance improvements is that the members of a
   runtime-checkable protocol are now considered "frozen" at runtime as soon as the
   class has been created. Monkey-patching attributes onto a runtime-checkable
   protocol will still work, but will have no impact on isinstance() checks comparing
   objects to the protocol. See "What's New in Python 3.12" for more details.
-  isinstance() checks against runtime-checkable protocols now use inspect.getattr_static()
   rather than hasattr() to lookup whether attributes exist (backporting python/cpython#103034).
-  Backport the ability to define __init__ methods on Protocol classes, a change
   made in Python 3.11 (originally implemented in python/cpython#31628
-  Speedup isinstance(3, typing_extensions.SupportsIndex) by >10x on Python <3.12.
-  Add typing_extensions versions of SupportsInt, SupportsFloat, SupportsComplex,
   SupportsBytes, SupportsAbs and SupportsRound. These have the same semantics as
   the versions from the typing module, but isinstance() checks against the
   typing_extensions versions are >10x faster at runtime on Python <3.12.
-  Add __orig_bases__ to non-generic TypedDicts, call-based TypedDicts, and call-based NamedTuples.
-  Add typing_extensions.get_original_bases, a backport of types.get_original_bases,
   introduced in Python 3.12 (CPython PR python/cpython#101827, originally
-  This function should always produce correct results when called on classes
   constructed using features from typing_extensions.
-  Constructing a call-based TypedDict using keyword arguments for the fields
   now causes a DeprecationWarning to be emitted. This matches the behaviour
   of typing.TypedDict on 3.11 and 3.12.
-  Backport the implementation of NewType from 3.10 (where it is implemented as
   a class rather than a function). This allows user-defined NewTypes to be pickled.
-  Fix tests and import on Python 3.12, where typing.TypeVar can no longer be subclassed.
-  Add typing_extensions.TypeAliasType, a backport of typing.TypeAliasType from PEP 695.
-  Backport changes to the repr of typing.Unpack that were made in order
   to implement PEP 692 (backport of python/cpython#104048).

(From OE-Core rev: a37154b9166323d05cca970ebb37bee0d5250893)

Signed-off-by: Wang Mingyu <[email protected]>
Signed-off-by: Alexandre Belloni <[email protected]>
Signed-off-by: Richard Purdie <[email protected]>
halstead pushed a commit to openembedded/openembedded-core that referenced this pull request Jun 2, 2023
Changelog:
============
-  Fix use of @deprecated on classes with __new__ but no __init__.
-  Fix regression in version 4.6.1 where comparing a generic class against a
   runtime-checkable protocol using isinstance() would cause AttributeError to
   be raised if using Python 3.7.
-  Change deprecated @runtime to formal API @runtime_checkable in the error message.
-  Fix regression in 4.6.0 where attempting to define a Protocol that was generic
   over a ParamSpec or a TypeVarTuple would cause TypeError to be raised.
-  typing_extensions is now documented at https://typing-extensions.readthedocs.io/en/latest/.
-  Add typing_extensions.Buffer, a marker class for buffer types, as proposed
   by PEP 688. Equivalent to collections.abc.Buffer in Python 3.12.
-  Backport two CPython PRs fixing various issues with typing.Literal:
   python/cpython#23294 and python/cpython#23383. Both CPython PRs were originally,
   and both were backported to Python >=3.9.1, but no earlier.
-  A side effect of one of the changes is that equality comparisons of Literal
   objects will now raise a TypeError if one of the Literal objects being compared
   has a mutable parameter. (Using mutable parameters with Literal is not
   supported by PEP 586 or by any major static type checkers.)
-  Literal is now reimplemented on all Python versions <= 3.10.0.
-  Backport CPython PR 26067, ensuring that isinstance() calls on protocols raise
   TypeError when the protocol is not decorated with @runtime_checkable.
-  Backport several significant performance improvements to runtime-checkable protocols
   that have been made in Python 3.12 (see python/cpython#74690 for details).
-  A side effect of one of the performance improvements is that the members of a
   runtime-checkable protocol are now considered "frozen" at runtime as soon as the
   class has been created. Monkey-patching attributes onto a runtime-checkable
   protocol will still work, but will have no impact on isinstance() checks comparing
   objects to the protocol. See "What's New in Python 3.12" for more details.
-  isinstance() checks against runtime-checkable protocols now use inspect.getattr_static()
   rather than hasattr() to lookup whether attributes exist (backporting python/cpython#103034).
-  Backport the ability to define __init__ methods on Protocol classes, a change
   made in Python 3.11 (originally implemented in python/cpython#31628
-  Speedup isinstance(3, typing_extensions.SupportsIndex) by >10x on Python <3.12.
-  Add typing_extensions versions of SupportsInt, SupportsFloat, SupportsComplex,
   SupportsBytes, SupportsAbs and SupportsRound. These have the same semantics as
   the versions from the typing module, but isinstance() checks against the
   typing_extensions versions are >10x faster at runtime on Python <3.12.
-  Add __orig_bases__ to non-generic TypedDicts, call-based TypedDicts, and call-based NamedTuples.
-  Add typing_extensions.get_original_bases, a backport of types.get_original_bases,
   introduced in Python 3.12 (CPython PR python/cpython#101827, originally
-  This function should always produce correct results when called on classes
   constructed using features from typing_extensions.
-  Constructing a call-based TypedDict using keyword arguments for the fields
   now causes a DeprecationWarning to be emitted. This matches the behaviour
   of typing.TypedDict on 3.11 and 3.12.
-  Backport the implementation of NewType from 3.10 (where it is implemented as
   a class rather than a function). This allows user-defined NewTypes to be pickled.
-  Fix tests and import on Python 3.12, where typing.TypeVar can no longer be subclassed.
-  Add typing_extensions.TypeAliasType, a backport of typing.TypeAliasType from PEP 695.
-  Backport changes to the repr of typing.Unpack that were made in order
   to implement PEP 692 (backport of python/cpython#104048).

Signed-off-by: Wang Mingyu <[email protected]>
Signed-off-by: Alexandre Belloni <[email protected]>
renovate bot referenced this pull request in allenporter/flux-local Jun 3, 2023
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [typing-extensions](https://togithub.com/python/typing_extensions)
([changelog](https://togithub.com/python/typing_extensions/blob/main/CHANGELOG.md))
| `==4.5.0` -> `==4.6.3` |
[![age](https://badges.renovateapi.com/packages/pypi/typing-extensions/4.6.3/age-slim)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://badges.renovateapi.com/packages/pypi/typing-extensions/4.6.3/adoption-slim)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://badges.renovateapi.com/packages/pypi/typing-extensions/4.6.3/compatibility-slim/4.5.0)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://badges.renovateapi.com/packages/pypi/typing-extensions/4.6.3/confidence-slim/4.5.0)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>python/typing_extensions</summary>

###
[`v4.6.3`](https://togithub.com/python/typing_extensions/blob/HEAD/CHANGELOG.md#Release-463-June-1-2023)

[Compare
Source](https://togithub.com/python/typing_extensions/compare/4.6.2...4.6.3)

-   Fix a regression introduced in v4.6.0 in the implementation of
    runtime-checkable protocols. The regression meant
that doing `class Foo(X, typing_extensions.Protocol)`, where `X` was a
class that
    had `abc.ABCMeta` as its metaclass, would then cause subsequent
    `isinstance(1, X)` calls to erroneously raise `TypeError`. Patch by
Alex Waygood (backporting the
CPython[https://github.com/python/cpython/pull/105152](https://togithub.com/python/cpython/pull/105152)l/105152).
-   Sync the repository's LICENSE file with that of CPython.
    `typing_extensions` is distributed under the same license as
    CPython itself.
- Skip a problematic test on Python 3.12.0b1. The test fails on 3.12.0b1
due to
    a bug in CPython, which will be fixed in 3.12.0b2. The
    `typing_extensions` test suite now passes on 3.12.0b1.

###
[`v4.6.2`](https://togithub.com/python/typing_extensions/blob/HEAD/CHANGELOG.md#Release-462-May-25-2023)

[Compare
Source](https://togithub.com/python/typing_extensions/compare/4.6.1...4.6.2)

- Fix use of `@deprecated` on classes with `__new__` but no `__init__`.
    Patch by Jelle Zijlstra.
- Fix regression in version 4.6.1 where comparing a generic class
against a
runtime-checkable protocol using `isinstance()` would cause
`AttributeError`
    to be raised if using Python 3.7.

###
[`v4.6.1`](https://togithub.com/python/typing_extensions/blob/HEAD/CHANGELOG.md#Release-461-May-23-2023)

[Compare
Source](https://togithub.com/python/typing_extensions/compare/4.6.0...4.6.1)

- Change deprecated `@runtime` to formal API `@runtime_checkable` in the
error
    message. Patch by Xuehai Pan.
- Fix regression in 4.6.0 where attempting to define a `Protocol` that
was
generic over a `ParamSpec` or a `TypeVarTuple` would cause `TypeError`
to be
    raised. Patch by Alex Waygood.

###
[`v4.6.0`](https://togithub.com/python/typing_extensions/blob/HEAD/CHANGELOG.md#Release-460-May-22-2023)

[Compare
Source](https://togithub.com/python/typing_extensions/compare/4.5.0...4.6.0)

-   `typing_extensions` is now documented at
https://typing-extensions.readthedocs.io/en/latest/. Patch by Jelle
Zijlstra.
- Add `typing_extensions.Buffer`, a marker class for buffer types, as
proposed
by PEP 688. Equivalent to `collections.abc.Buffer` in Python 3.12. Patch
by
    Jelle Zijlstra.
- Backport two CPython PRs fixing various issues with `typing.Literal`:

[https://github.com/python/cpython/pull/23294](https://togithub.com/python/cpython/pull/23294)3294
[https://github.com/python/cpython/pull/23383](https://togithub.com/python/cpython/pull/23383)ll/23383.
Both CPython PRs were
originally by Yurii Karabas, and both were backported to Python >=3.9.1,
but
    no earlier. Patch by Alex Waygood.

A side effect of one of the changes is that equality comparisons of
`Literal`
objects will now raise a `TypeError` if one of the `Literal` objects
being
compared has a mutable parameter. (Using mutable parameters with
`Literal` is
    not supported by PEP 586 or by any major static type checkers.)
-   `Literal` is now reimplemented on all Python versions <= 3.10.0. The
`typing_extensions` version does not suffer from the bug that was fixed
in

[https://github.com/python/cpython/pull/29334](https://togithub.com/python/cpython/pull/29334)9334.
(The CPython bugfix was
    backported to CPython 3.10.1 and 3.9.8, but no earlier.)
- Backport [CPython PR
26067](https://togithub.com/python/cpython/pull/26067)
    (originally by Yurii Karabas), ensuring that `isinstance()` calls on
    protocols raise `TypeError` when the protocol is not decorated with
    `@runtime_checkable`. Patch by Alex Waygood.
- Backport several significant performance improvements to
runtime-checkable
protocols that have been made in Python 3.12
([https://github.com/python/cpython/issues/74690](https://togithub.com/python/cpython/issues/74690)es/74690
for details). Patch by Alex
    Waygood.

A side effect of one of the performance improvements is that the members
of
a runtime-checkable protocol are now considered “frozen” at runtime as
soon
    as the class has been created. Monkey-patching attributes onto a
runtime-checkable protocol will still work, but will have no impact on
    `isinstance()` checks comparing objects to the protocol. See
["What's New in Python
3.12"](https://docs.python.org/3.12/whatsnew/3.12.html#typing)
    for more details.
-   `isinstance()` checks against runtime-checkable protocols now use
    `inspect.getattr_static()` rather than `hasattr()` to lookup whether
attributes exist
(backport[https://github.com/python/cpython/pull/103034](https://togithub.com/python/cpython/pull/103034)3034).
    This means that descriptors and `__getattr__` methods are no longer
unexpectedly evaluated during `isinstance()` checks against
runtime-checkable
protocols. However, it may also mean that some objects which used to be
considered instances of a runtime-checkable protocol on older versions
of
`typing_extensions` may no longer be considered instances of that
protocol
using the new release, and vice versa. Most users are unlikely to be
affected
    by this change. Patch by Alex Waygood.
- Backport the ability to define `__init__` methods on Protocol classes,
a
change made in Python 3.11 (originally
implemented[https://github.com/python/cpython/pull/31628](https://togithub.com/python/cpython/pull/31628)ll/31628
by Adrian Garcia Badaracco).
    Patch by Alex Waygood.
- Speedup `isinstance(3, typing_extensions.SupportsIndex)` by >10x on
Python
    <3.12. Patch by Alex Waygood.
-   Add `typing_extensions` versions of `SupportsInt`, `SupportsFloat`,
`SupportsComplex`, `SupportsBytes`, `SupportsAbs` and `SupportsRound`.
These
have the same semantics as the versions from the `typing` module, but
`isinstance()` checks against the `typing_extensions` versions are >10x
faster
    at runtime on Python <3.12. Patch by Alex Waygood.
- Add `__orig_bases__` to non-generic TypedDicts, call-based TypedDicts,
and
call-based NamedTuples. Other TypedDicts and NamedTuples already had the
attribute.
    Patch by Adrian Garcia Badaracco.
-   Add `typing_extensions.get_original_bases`, a backport of

[`types.get_original_bases`](https://docs.python.org/3.12/library/types.html#types.get_original_bases),
introduced in Python 3.12
(CPython[https://github.com/python/cpython/pull/101827](https://togithub.com/python/cpython/pull/101827)l/101827,
originally by James
    Hilton-Balfe). Patch by Alex Waygood.

This function should always produce correct results when called on
classes
    constructed using features from `typing_extensions`. However, it may
produce incorrect results when called on some `NamedTuple` or
`TypedDict`
    classes that use `typing.{NamedTuple,TypedDict}` on Python <=3.11.
- Constructing a call-based `TypedDict` using keyword arguments for the
fields
now causes a `DeprecationWarning` to be emitted. This matches the
behaviour
    of `typing.TypedDict` on 3.11 and 3.12.
- Backport the implementation of `NewType` from 3.10 (where it is
implemented
as a class rather than a function). This allows user-defined `NewType`s
to be
    pickled. Patch by Alex Waygood.
- Fix tests and import on Python 3.12, where `typing.TypeVar` can no
longer be
    subclassed. Patch by Jelle Zijlstra.
- Add `typing_extensions.TypeAliasType`, a backport of
`typing.TypeAliasType`
    from PEP 695. Patch by Jelle Zijlstra.
- Backport changes to the repr of `typing.Unpack` that were made in
order to
    implement [PEP 692](https://peps.python.org/pep-0692/) (backport of

[https://github.com/python/cpython/pull/104048](https://togithub.com/python/cpython/pull/104048)4048).
Patch by Alex Waygood.

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://app.renovatebot.com/dashboard#github/allenporter/flux-local).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNS4xMDUuMiIsInVwZGF0ZWRJblZlciI6IjM1LjEwNS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
pmhahn pushed a commit to univention/univention-corporate-server that referenced this pull request Jun 10, 2023
It was caused by the new [4.6 version of typing-extensions](https://github.com/python/typing_extensions/blob/main/CHANGELOG.md#release-460-may-22-2023) used by jsonschema-spec to validate protocol.

>    isinstance() checks against runtime-checkable protocols now use inspect.getattr_static() rather than hasattr() to lookup whether attributes exist (backporting python/cpython#103034). This means that descriptors and getattr methods are no longer unexpectedly evaluated during isinstance() checks against runtime-checkable protocols. However, it may also mean that some objects which used to be considered instances of a runtime-checkable protocol on older versions of typing_extensions may no longer be considered instances of that protocol using the new release, and vice versa. Most users are unlikely to be affected by this change. Patch by Alex Waygood.

Temporary workaround will be downgrading typing-extensions to version below 4.6:

    pip install "typing-extensions<4.6"

Bug #50048
daregit pushed a commit to daregit/yocto-combined that referenced this pull request May 22, 2024
Changelog:
============
-  Fix use of @deprecated on classes with __new__ but no __init__.
-  Fix regression in version 4.6.1 where comparing a generic class against a
   runtime-checkable protocol using isinstance() would cause AttributeError to
   be raised if using Python 3.7.
-  Change deprecated @runtime to formal API @runtime_checkable in the error message.
-  Fix regression in 4.6.0 where attempting to define a Protocol that was generic
   over a ParamSpec or a TypeVarTuple would cause TypeError to be raised.
-  typing_extensions is now documented at https://typing-extensions.readthedocs.io/en/latest/.
-  Add typing_extensions.Buffer, a marker class for buffer types, as proposed
   by PEP 688. Equivalent to collections.abc.Buffer in Python 3.12.
-  Backport two CPython PRs fixing various issues with typing.Literal:
   python/cpython#23294 and python/cpython#23383. Both CPython PRs were originally,
   and both were backported to Python >=3.9.1, but no earlier.
-  A side effect of one of the changes is that equality comparisons of Literal
   objects will now raise a TypeError if one of the Literal objects being compared
   has a mutable parameter. (Using mutable parameters with Literal is not
   supported by PEP 586 or by any major static type checkers.)
-  Literal is now reimplemented on all Python versions <= 3.10.0.
-  Backport CPython PR 26067, ensuring that isinstance() calls on protocols raise
   TypeError when the protocol is not decorated with @runtime_checkable.
-  Backport several significant performance improvements to runtime-checkable protocols
   that have been made in Python 3.12 (see python/cpython#74690 for details).
-  A side effect of one of the performance improvements is that the members of a
   runtime-checkable protocol are now considered "frozen" at runtime as soon as the
   class has been created. Monkey-patching attributes onto a runtime-checkable
   protocol will still work, but will have no impact on isinstance() checks comparing
   objects to the protocol. See "What's New in Python 3.12" for more details.
-  isinstance() checks against runtime-checkable protocols now use inspect.getattr_static()
   rather than hasattr() to lookup whether attributes exist (backporting python/cpython#103034).
-  Backport the ability to define __init__ methods on Protocol classes, a change
   made in Python 3.11 (originally implemented in python/cpython#31628
-  Speedup isinstance(3, typing_extensions.SupportsIndex) by >10x on Python <3.12.
-  Add typing_extensions versions of SupportsInt, SupportsFloat, SupportsComplex,
   SupportsBytes, SupportsAbs and SupportsRound. These have the same semantics as
   the versions from the typing module, but isinstance() checks against the
   typing_extensions versions are >10x faster at runtime on Python <3.12.
-  Add __orig_bases__ to non-generic TypedDicts, call-based TypedDicts, and call-based NamedTuples.
-  Add typing_extensions.get_original_bases, a backport of types.get_original_bases,
   introduced in Python 3.12 (CPython PR python/cpython#101827, originally
-  This function should always produce correct results when called on classes
   constructed using features from typing_extensions.
-  Constructing a call-based TypedDict using keyword arguments for the fields
   now causes a DeprecationWarning to be emitted. This matches the behaviour
   of typing.TypedDict on 3.11 and 3.12.
-  Backport the implementation of NewType from 3.10 (where it is implemented as
   a class rather than a function). This allows user-defined NewTypes to be pickled.
-  Fix tests and import on Python 3.12, where typing.TypeVar can no longer be subclassed.
-  Add typing_extensions.TypeAliasType, a backport of typing.TypeAliasType from PEP 695.
-  Backport changes to the repr of typing.Unpack that were made in order
   to implement PEP 692 (backport of python/cpython#104048).

(From OE-Core rev: a37154b9166323d05cca970ebb37bee0d5250893)

Signed-off-by: Wang Mingyu <[email protected]>
Signed-off-by: Alexandre Belloni <[email protected]>
Signed-off-by: Richard Purdie <[email protected]>
dangotbanned added a commit to dangotbanned/altair that referenced this pull request May 31, 2024
…python>=3.12.0`

Any objects using `__getattr__`, e.g. `Parameter` break on older versions [see PR](python/cpython#103034)
@GDYendell
Copy link

GDYendell commented Nov 20, 2024

inspect.getattr_static uses inspect._check_instance which uses object.__getattribute__(obj, "__dict__"). This fails for bound instance methods (although obj.__dict__ does work), so they cannot fulfill runtime_checkable Protocols anymore.

from typing import Protocol, runtime_checkable


@runtime_checkable
class Labelled(Protocol):
    label: str


class A:
    def f(self):
        pass


A.f.label = "f"

a = A()
print(a.f.__dict__)  # {'label': 'f'}
print(object.__getattribute__(a.f, "__dict__"))  # AttributeError

assert isinstance(a.f, Labelled)  # AssertionError

Is this an intended/known change? Could it be noted in the changelog here as an example of case that will no longer work?

@AlexWaygood
Copy link
Member Author

@GDYendell, I will admit that I did indeed fail to consider how this would change the behaviour of runtime-checkable protocols when used in conjunction with bound instance methods that have extra attributes monkeypatched onto them.

Can you possibly open a new issue about this? I haven't got the time to look at it right now, and comment threads on long-merged PRs are unfortunately likely to get forgotten about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.12 bugs and security fixes stdlib Python modules in the Lib dir topic-typing
Projects
None yet
Development

Successfully merging this pull request may close these issues.