Fix trivial inference tests for Python 3.12 support #31170
Conversation
|
R: @tvalentyn |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
| @@ -143,6 +143,8 @@ def copy(self): | |||
| return FrameState(self.f, self.vars, self.stack, self.kw_names) | |||
|
|
|||
| def const_type(self, i): | |||
| print(self.co.co_consts) | |||
There was a problem hiding this comment.
Yeah, leftover from some checking. This weirdly broke a test in the snippets directory that relies on stdout, which is maybe bad test design but did a good job catching these
| """Defines the actions intrinsic bytecodes have on the frame. | ||
|
|
||
| Each function here corresponds to a bytecode documented in | ||
| https://docs.python.org/2/library/dis.html or |
There was a problem hiding this comment.
Py2 docs shouldn't be relevant anymore.
| pass | ||
|
|
||
|
|
||
| int_one_ops = [ |
There was a problem hiding this comment.
| int_one_ops = [ | |
| INT_ONE_OPS = tuple([ |
(to make it immutable and and to show it's a constant.
| intrinsic_typevartuple, | ||
| intrinsic_subscript_generic, | ||
| intrinsic_typealias | ||
| ] |
| pass | ||
|
|
||
|
|
||
| int_one_ops = [ |
There was a problem hiding this comment.
Could you add a comment what defines the order of ops in this sequence? How can we make sure that this list stays up to date with future Python versions potentially adding more intrinsic ops? I worry that the order might become skewed but tests we won't notice the skew.
Is it possible to add a unit test that runs on Python 3.12+ which would fail if INT_ONE_OPS doesn't match the list of intrinsic ops known to dis? If that's not easy, maybe a unit test that would just fail on Python 3.13 with an instruction to: revisit intrinsic_one_ops.INT_ONE_OPS, make necessary changes, and change the test to fail starting next Python minor version. Then, it will make use manually look at this list once in a while.
There was a problem hiding this comment.
I'll see if the dis package provides a utility to get that list easily so we can check against it. Otherwise documenting why the order is what it is and adding some check for python versions might be the best way to gate it off.
There was a problem hiding this comment.
Okay I can pull the table out from the module in a slightly-janky way but it does become testable. That way if ops change orders or new ones are added we can be aware of it.
|
|
||
| Each function here corresponds to a bytecode documented in | ||
| https://docs.python.org/2/library/dis.html or | ||
| https://docs.python.org/3/library/dis.html. The first argument is a (mutable) |
There was a problem hiding this comment.
| https://docs.python.org/3/library/dis.html. The first argument is a (mutable) | |
| https://docs.python.org/3/library/dis.html . The first argument is a (mutable) |
(to make it clickable)
There was a problem hiding this comment.
done (did it on my branch instead of taking the suggestion)
| elif isinstance(base, typehints.IndexableTypeConstraint): | ||
| out = base | ||
| else: | ||
| out = element_type(base) |
There was a problem hiding this comment.
For my understanding, what is the interpretation of
out = element_type(base)?
does this say: take a container, and return the type of individual element that lives in the container? Wouldn't a slice always return a container as opposed to individual elements?
There was a problem hiding this comment.
Hmm that's actually an implementation interpretation I hadn't thought of. The documentation for the op is here - https://docs.python.org/3/library/dis.html#opcode-BINARY_SLICE and I was adapting the BINARY_SUBSCR code (which is doing similar work with a single index arg instead of a slice.) I think most cases will be caught by the indexable type constraint (so the type of the container is what we return) but that else case might not be handling things correctly. I'll see if I can get a bit more of an understanding on this op code specifically.
| @@ -432,6 +433,8 @@ def infer_return_type_func(f, input_types, debug=False, depth=0): | |||
| elif op in dis.haslocal: | |||
| print('(' + co.co_varnames[arg] + ')', end=' ') | |||
| elif op in dis.hascompare: | |||
| if (sys.version_info.major, sys.version_info.minor) >= (3, 12): | |||
| arg = arg >> 4 | |||
There was a problem hiding this comment.
for my knowledge, where does this come from? Perhaps there is a link you could add in a comment.
There was a problem hiding this comment.
This one is funny in that it was actually an undocumented change as best as I could tell. The comparison operations are similar to the intrinsic functions here in that the arg for a comparison op is an integer index into a list of operations that you index into. The output is always a boolean here anyway so we don't necessarily care what the operation itself is, but for debugging purposes we get the operation to print anyway. The problem was that these indices got bit shifted (a trick this package really likes to do for operations in different versions), so our print statement here was failing with an out-of-range index.
I'll try to add some comments around some of these changes since they're a little arbitrary, but for the most part it really just breaks down to "this argument's representation was changed in the package so we have to change it here for this version"
| if debug: | ||
| print("Executing intrinsic one op", int_op.__name__.upper()) | ||
| int_op(state, arg) | ||
| elif opname == 'CALL_INTRINSIC_2': |
There was a problem hiding this comment.
Should we add a comment sth like: all currently available binary intrinsic ops are not relevant for type-checking, and as i mentioned above, also have some safeguard to revisit this logic if we can detect that the list of intrinsic ops has changed.
There was a problem hiding this comment.
I'm inclined to think I can remove the intrinsic 2 call block here for now so if there's a failure later we will get the clear "cannot handle operation x" failure. I think this was a holdover from when I was anticipating having to implement most of the new calls to get tests green.
| @@ -30,7 +30,7 @@ class IntrinsicOneOpsTest(unittest.TestCase): | |||
| def testOperationsOrder(self): | |||
| if sys.version_info >= (3, 12): | |||
| dis_order = dis.__dict__['_intrinsic_1_descs'] | |||
| beam_order = list() | |||
| beam_order = [] | |||
There was a problem hiding this comment.
nit: use list comprehensions for a more Pythonic syntax. e.g.
beam_ops = [fn.__name_upper() for fn in intrinsic_one_ops.INT_ONE_OPS]
|
|
||
|
|
||
| class IntrinsicOneOpsTest(unittest.TestCase): | ||
| def testOperationsOrder(self): |
There was a problem hiding this comment.
| def testOperationsOrder(self): | |
| def test_unary_intrinsic_ops_are_in_the_same_order_as_in_cpython(self): | |
| (self): |
Implements new operations for trivial inference ahead of Python 3.12 support to fix existing unit tests. These operations can be seen listed at https://docs.python.org/3/library/dis.html
Not every operation has been implemented, but not every operation comes up in the context of us doing type introspection on functions. A few of these operations were actually introduced in 3.11 and weren't used in our test cases until 3.12!
fixes: #31047
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.