Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Python 3.11 #7

Merged
merged 1 commit into from
Feb 24, 2024

Conversation

radoering
Copy link
Contributor

@radoering radoering commented Feb 19, 2024

Resolves: #5

Notable changes:

  • new class PyInterpreterFrame because PyFrameObject objects may not be available (in contrast to PyFrameObject PyInterpreterFrame is not a PyObject!)
  • new command !pyinterpreterframe that is used for the [Frame] link (We cannot use !pyobj anymore because PyInterpreterFrame is not a subclass of PyObject)
  • format of mapping between instructions and line numbers changed significantly (again), see PyCodeObject
  • the __dict__ of Python objects is not stored as PyDictObject anymore but as a "managed dict", see PyObject and PyManagedDict
  • new class PyDictKeysObject because it is used in PyDictObject and PyManagedDict
  • the symbol PyMemberDef is not available (will be available again in 3.12), see PyTypeObject, PyMemberDefAuto and PyMemberDefManual

All tests are passing with Python 3.11 on my machine now.

- new class `PyInterpreterFrame` because `PyFrameObject` objects may not be available (in contrast to `PyFrameObject` `PyInterpreterFrame` is not a `PyObject`!)
- new command `!pyinterpreterframe` that is used for the `[Frame]` link (We cannot use `!pyobj` anymore because `PyInterpreterFrame` is not a subclass of `PyObject`)
- format of mapping between instructions and line numbers changed significantly (again), see `PyCodeObject`
- the `__dict__` of Python objects is not stored as `PyDictObject` anymore but as a "managed dict", see `PyObject` and `PyManagedDict`
- the symbol `PyMemberDef` is not available (will be available again in 3.12), see `PyTypeObject`, `PyMemberDefAuto` and `PyMemberDefManual`
- new class `PyDictKeysObject` because it is used in `PyDictObject` and `PyManagedDict`
@SeanCline SeanCline merged commit ff468c2 into SeanCline:master Feb 24, 2024
@SeanCline
Copy link
Owner

SeanCline commented Feb 24, 2024

Wow!

That was a much bigger change than I was expecting. Thanks for getting 3.11 working!

One regression I noticed is that 3.10 is quite a bit slower on the Fibonacci test (generating and walking more than 500 stack frames). I think 3.11 support is more important than a performance regression on older versions, so I'm good with merging this and optimizing later if it becomes a problem.

3.12 looks like a smaller change (luckily), but I noticed they've done more of the same. Similar to PyInterpreterFrame, more data has been moved outside of PyObjects.

@radoering
Copy link
Contributor Author

One regression I noticed is that 3.10 is quite a bit slower on the Fibonacci test (generating and walking more than 500 stack frames).

I don't know if it's relevant but that's a part in the code I do not like very much:

PyExt/src/pystack.cpp

Lines 68 to 74 in ff468c2

try {
auto& interpreterFrame = dynamic_cast<const PyInterpreterFrame&>(frame);
oss << utils::link("[Frame]", "!pyinterpreterframe 0n"s + to_string(interpreterFrame.offset()), "Inspect interpreter frame (including localsplus).") << " ";
} catch (bad_cast&) {
auto& frameObject = dynamic_cast<const PyFrameObject&>(frame);
oss << utils::link("[Frame]", "!pyobj 0n"s + to_string(frameObject.offset()), "Inspect frame object (including localsplus).") << " ";
}

With Python 3.10, the exception is thrown for each frame. I deliberately chose to make the Python 3.11+ branch the faster one. Rethinking about it, we could somehow remember what type of object frame will be and replace the try-catch with an if. (You could check if swapping the try and catch makes 3.10 faster and 3.11 slower before to check if it's worth to optimize it.)

3.12 looks like a smaller change (luckily), but I noticed they've done more of the same. Similar to PyInterpreterFrame, more data has been moved outside of PyObjects.

In case, it's relevant for your release planning: I probably won't be able to work on Python 3.12 support within the next months. If you don't beat me to it, I'll probably take a look at the end of this year. 😃

@SeanCline
Copy link
Owner

I don't know if it's relevant but that's a part in the code I do not like very much

There are plenty of places in this codebase where I've done something similar. 😬
In such cases, I do exactly as you have -- optimise for the newer version as eventually older versions will stop being used so much.

While exceptions are slow, they aren't in the order of seconds (more like microseconds to unwind a stack and catch) so this isn't the cause, but while we're talking about it, I'll switched it over to the pointer version of dynamic_cast so it doesn't show up as a "first chance exception" while debugging PyExt. It's also slightly faster as it still uses RTTI, but doesn't have to do any stack unwinding.

I probably won't be able to work on Python 3.12 support within the next months.

I have a small fix for 3.12 PyStringObjects ready to push but I don't know if I'll get around to fully 3.12 support in the next few months either. I've been spending a lot of time on home-improvement projects, and I'm getting married this year, which has taken my focus away from hobby programming projects. But I suspect you have similar things going on. 😃

Your contributions are greatly appreciated when you do have the time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python 3.11 compatibility
2 participants