Python 3.11 Support #43
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Just to fix compilation. It doesn't really work yet. For #41
No relate yet. Dir and getattr lacks support for localsplus. Honestly I'm ot sure how to even do relate for localsplus. It needs both the frame name and the local variable name, which isn't something path supports. Though, we can add it. But then, ByVia classifier would be much less useful if a bunch of local variables share the same name. (Is it possible for, maybe, a pycapsule to represent a _PyInterpreterFrame?) For #41
It's not accessible from Python.
And only use the public APIs with PyFrameObjects. The trace won't be as accurate anymore (because the profile will create all these frame objects rather than just observing them) but at least there won't be a regression (Py < 3.11 always creates these objects). For #41
Python built-in only traverses when FRAME_OWNED_BY_FRAME_OBJECT. :( For #41
The interpreter sometimes sets stacktop to bad values. Not exactly
sure why, but it causes assertion in test_Classifiers.test_classification
=====================================================================
ERROR: test_classification (guppy.heapy.test.test_Classifiers.ClassificationCase.test_classification)
----------------------------------------------------------------------
Traceback (most recent call last):
File "guppy3/guppy/heapy/View.py", line 423, in obj_at
return self.immnodeset(self.hv.heap()).obj_at(addr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: No object found at address 0x7f8d7aa1b540
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "guppy3/guppy/heapy/test/test_Classifiers.py", line 906, in test_classification
self.aseq(iso(o).byid.kind, Use.Id(id(o)))
^^^^^^^^^^^^^
File "guppy3/guppy/heapy/UniSet.py", line 25, in __call__
def __call__(self, *args, **kwds): return self.fam.c_call(self, args, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "guppy3/guppy/heapy/UniSet.py", line 1938, in c_call
return a.classifier.get_userkind(*args, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "guppy3/guppy/heapy/Classifiers.py", line 237, in get_userkind
return self.get_kind(self.mod.View.obj_at(address))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "guppy3/guppy/heapy/View.py", line 425, in obj_at
raise ValueError('No object found at address %s' % hex(addr))
ValueError: No object found at address 0x7f8d7aa1b540
For some reason in Python 3.11 sometimes Share can directly reference
its members without going through the dict, causing assertion:
test_6 (guppy.heapy.test.test_ER.FirstCase.test_6)
Test of .refdby on all others ... <guppy.etc.Glue.Share object at 0x7fa056c66710>
aseq: Expected: b = Partition of a set of 6 objects. Total size = 2216 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 17 1688 76 1688 76 type
1 1 17 296 13 1984 90 dict of guppy.heapy.test.test_ER.C
2 1 17 72 3 2056 93 module
3 1 17 64 3 2120 96 dict (no owner)
4 1 17 56 3 2176 98 guppy.heapy.test.test_ER.C
5 1 17 40 2 2216 100 types.MappingProxyType
Got actually : a = Partition of a set of 5 objects. Total size = 2144 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 1 20 1688 79 1688 79 type
1 1 20 296 14 1984 93 dict of guppy.heapy.test.test_ER.C
2 1 20 64 3 2048 96 dict (no owner)
3 1 20 56 3 2104 98 guppy.heapy.test.test_ER.C
4 1 20 40 2 2144 100 types.MappingProxyType
The difference in the test was that the kind shows sys module was
directly referenced by the Share while running it again shows it
was referenced indirectly through the Share's dict.
I think it's better to reference directly anyways though slots and
likely get a speed boost.
Python added _PyType_PreHeaderSize for the preheader and it's no longer just sizeof(PyGC_Head) See python/cpython#101430 For #41
The paths are sorted in an unexpected way. Test both cases (though only one case happens per Python version), and document what is going on behind the test case. In Python 3.11 both the object and the list have the same size. The object grew by 8 bytes compared to Python 3.10. For #41
Some referenced attributes are no longer slotted. I'm not sure how to get a fallback to default relate for slotted attributes so I'm just going to make relate do the entire thing. For #41
Traverse from the top (newest) of stack to oldest, as a linked reference, instead of linking every frame against rootstate. This behavior is expected by the tests. Also fix f_back processing because Py 3.11 only tracks f_back when FRAME_OWNED_BY_FRAME_OBJECT, i.e. the frame finished executing. We don't always have this luxary. For #41
There seems to be no way to distinguish managed dict references vs slots references. I'm not sure how to better do this but I'll just materialize all the dicts so the reference chain can be consistent. For #41
Refpat seems also sorted by size. Though I'm not sure which comes first when both entries are at the same size. For #41
The Github Actions CI doesn't seem to support 3.6 anymore on latest Ubuntu. It's EOL-ed anyways so there's not much point in more support.
No more Python 3.6, added support for Python 3.11
sys seems to be too chaotic. Most of the flakiness comes from 3.11 where we have crazy materialization going on that could change the reference any time. But I also got it occur on 3.8 too [1]. Let's just change to something more stable like 'inspect'. [1] https://github.com/zhuyifei1999/guppy3/actions/runs/4966465890/jobs/8887920211
I have no idea how to debug this anymore.
It's no longer all native
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Creating a PR so I can run github workflows as CI.
I'm not sure I'm perfectly happy with the way the code turned out. Looking at the heap is no longer a sort of "fishbowl", but it actively creates all the frame objects and materializes the managed dicts of objects. They will over-report the amount of memory consumption, but I can't think of a better way around this. See #41 (comment) for context around the frame objects.
Fixes #41