-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MRO: Behavior change from 3.10 to 3.11: multiple inheritance issue with C-API (pybind11) #92678
Comments
Do you have a minimal example that reproduces the issue, or failing that an example of a piece of pybind11 code demonstrating the problem? You say that the issue first appeared in 3.11a3. Are you able to bisect further to a specific commit? |
@JelleZijlstra The failing test can be found here: pybind/pybind11#3923 |
Do you get an exception? What is the exception? |
Ah, the "Upstream / 🐍 3.11 dev • ubuntu-latest • x64 (pull_request)" failed with:
This error message comes from the mro_check() function of typeobject.c:
with:
|
extra_ivars() used by solid_base() is different in Python 3.11, the following code path was removed in 3.11:
Python 3.11 changes: I don't understand well the purpose of the extra_ivars(type, base) function. |
pybind11 code:
It sets |
@vstinner Hmm, any idea how to ensure the ivars is happy then? |
I have no idea. It would help to have a simpler reproducer than pybind11 test suite. |
@JelleZijlstra I have a failing portion of PyBind11 test suite now. Any ideas how to debug this further? |
Sorry, I am not familiar with this area of the code so I don't think I can be of more help than Victor. As Victor said, it would be helpful to have a self-contained example that doesn't rely on pybind11. Also, would be good to know how big the impact is on pybind11: is this something a lot of users are likely to run into, or just an obscure edge case that your test suite happens to cover? |
Here's a pybind11 reproducer: #include <pybind11/pybind11.h>
namespace py = pybind11;
struct Vanilla {};
struct WithDict {};
struct VanillaDictMix2 : WithDict, Vanilla {};
struct VanillaDictMix1 : Vanilla, WithDict {};
PYBIND11_MODULE(example, m) {
py::class_<Vanilla>(m, "Vanilla").def(py::init<>());
py::class_<WithDict>(m, "WithDict", py::dynamic_attr()).def(py::init<>());
py::class_<VanillaDictMix2, WithDict, Vanilla>(m, "VanillaDictMix2").def(py::init<>()); // OK
py::class_<VanillaDictMix1, Vanilla, WithDict>(m, "VanillaDictMix1").def(py::init<>()); // Broken in 3.11
} (Names chosen to match the test suite, which is why 2 comes before 1 - 2 works, 1 doesn't) Compile and run (fish syntax): $ clang++ -Wall -shared -std=c++14 -undefined dynamic_lookup (pipx run --python=~/.pyenv/versions/3.11.0b1/bin/python pybind11 --includes | string split " ") -g example.cpp -o example(~/.pyenv/versions/3.11.0b1/bin/python3-config --extension-suffix)
$ ~/.pyenv/versions/3.11.0b1/bin/python -c "import example"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: VanillaDictMix1: PyType_Ready failed (TypeError: mro() returned base with unsuitable layout ('example.WithDict'))! Note that WithDict has a |
@markshannon: You made the two commits changing extra_ivars(). Do you have any idea why Python 3.11 behaves differently? Is it a deliberate choice? |
Strictly speaking, this is a pybind11 bug, not a CPython bug. Happy to help, though 🙂 First all, why is pybind11 doing its own class creation instead of calling the C-API? Regarding If you want to add a In 3.12 we might do something similar with the Hope that helps. |
@Skylion007 does that help? |
I tried this locally and it seems to work (it passes the test above, anyway)! We are a little stuck in CI because
|
I got this same error while trying to port mypyc to 3.11. See python/mypy#13206. # cat rep.py
from typing import Protocol
class Proto(Protocol):
pass
class A:
pass
class B(A, Proto):
pass # mypyc rep.py
>>> import rep
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "rep.py", line 9, in <module>
class B(A, Proto):
File "/home/ken/Documents/GitHub/cpython/Lib/abc.py", line 106, in __new__
cls = super().__new__(mcls, name, bases, namespace, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: multiple bases have instance lay-out conflict
>>> Snippets of the generated C API code:
Is mypyc doing something wrong by setting |
@Fidget-Spinner Your B_traverse function needs to also visit the type of the self object after 3.9. Doubt that's the problem though. https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_traverse |
@Skylion007 thanks for the catch. I didn't notice that. I see that Pybind11 uses In mypyc, we manually called |
@pablogsal can I set this as deferred blocker please? This issue affects breaks I have a fix at #95242. |
Yeah, go ahead 👍 Bear in mind that if we mark it as deferred blocker there will be no beta to test this if the fix goes after today's release. |
Also, seems that @markshannon disagrees that this is a CPython bug. Please check with him before merging anything. I will try to review the issue and the bug more carefully soon, but today is going to be complicated as I have to do the last beta release. |
Yes it's not a CPython bug in 3.11 per-se. It relies on certain assumptions (see #95242 (comment)). It can also be solved by setting I had a working fix for mypyc which set the flag properly, but it had no GC traversal for the The fix is a compatibility hack, and it should just stay in place till we get better class creation API in 3.12. |
Rather than modify the (difficult to understand) internals of typeobject.c, wouldn't we be better exposing functions to visit and clear the managed dictionary? It seems like a safer alternative.
|
* Add test for inheriting explicit __dict__ and weakref. * Restore 3.10 behavior for multiple inheritance of C extension classes that store their dictionary at the end of the struct.
@Skylion007 could you confirm that #95596 is enough for pybind11 to work with 3.11 before the better API is provided in 3.12? |
We have a GHA workflow that uses https://github.com/actions/setup-python, but that's only pulling the current beta:
I searched but didn't get lucky: is there a GHA recipe for git cloning cpython from a branch (3.11), configure, make install? I'll try that interactively for now. |
Given that we merged this today I don't think there will be anything available, unfortunately. As we are going to release the RC tomorrow, it would be great if you could validate this for us today even if its done manually :) |
TL;DR: Yes, we're good (pybind11). Details: I built Python 3.11 from scratch (configure; make install) for two versions:
I also backed out the With that:
For completeness:
diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h
index db7cd8ef..65e223a3 100644
--- a/include/pybind11/attr.h
+++ b/include/pybind11/attr.h
@@ -345,11 +345,7 @@ struct type_record {
bases.append((PyObject *) base_info->type);
-#if PY_VERSION_HEX < 0x030B0000
dynamic_attr |= base_info->type->tp_dictoffset != 0;
-#else
- dynamic_attr |= (base_info->type->tp_flags & Py_TPFLAGS_MANAGED_DICT) != 0;
-#endif
if (caster) {
base_info->implicit_casts.emplace_back(type, caster);
diff --git a/include/pybind11/detail/class.h b/include/pybind11/detail/class.h
index 42720f84..3db8429b 100644
--- a/include/pybind11/detail/class.h
+++ b/include/pybind11/detail/class.h
@@ -549,12 +549,8 @@ extern "C" inline int pybind11_clear(PyObject *self) {
inline void enable_dynamic_attributes(PyHeapTypeObject *heap_type) {
auto *type = &heap_type->ht_type;
type->tp_flags |= Py_TPFLAGS_HAVE_GC;
-#if PY_VERSION_HEX < 0x030B0000
type->tp_dictoffset = type->tp_basicsize; // place dict at the end
type->tp_basicsize += (ssize_t) sizeof(PyObject *); // and allocate enough space for it
-#else
- type->tp_flags |= Py_TPFLAGS_MANAGED_DICT;
-#endif
type->tp_traverse = pybind11_traverse;
type->tp_clear = pybind11_clear;
With "back" version of 3.11:
|
Thanks a lot for checking! I'm closing this issue, let's reopen if we discover that we missed anything |
Yeah! That's cool! Thanks for fixing pybind11 ;-) |
To clarify (especially for people looking here later):
|
…ary offset calculations. (pythonGH-95598) Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Stanley <[email protected]> (cherry picked from commit 8d37c62) Co-authored-by: Mark Shannon <[email protected]>
…fset calculations. (GH-95598) Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Stanley <[email protected]>
…fset calculations. (GH-95598) (GH-95821) Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Stanley <[email protected]> Co-authored-by: Mark Shannon <[email protected]>
…ary offset calculations. (pythonGH-95598) Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Stanley <[email protected]>
See also issue #96046 which might be related. |
Bug report
We have started testing the 3.11 beta branch in pybind11 and have found an issue where a class constructed using the C-API that inherits from two base classes that have different tpget_set implementations. Interestingly, this issue occurs when the first base does not have dynamic attributes enabled, but another base class does. I tried making the child class also have dynamic attributes (and therefore a larger size to store the dict etc.), but I still get a PyTypeReady failed. We've been encountering this issue since alpha 3, but I have not found any issues on BPO of other people having similar issues. I am wondering what changed and how we can fix our API usage to allow for full support of Python 3.11 as it enters beta.
I suspect it's something very subtle with how we are constructing our Python types, but there was nothing in the 3.11 migration guide that flags this issue. Any thoughts on how to fix issue? Is this known behavior or a bug? Or is it something that should be added to the migration guide?
@vstinner I know you are very familiar with the C-API and helped us deal with some of the other API changes, any thoughts?
Here is the failing test: pybind/pybind11#3923
The text was updated successfully, but these errors were encountered: