Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix SystemError raised from PyUnicodeDecodeError_Create on PyPy 3.10 #3297

Merged
merged 1 commit into from
Jul 5, 2023

Conversation

davidhewitt
Copy link
Member

In pydantic/pydantic-core#659 a SystemError is reported in the Pydantic test suite for PyPy 3.10:

_____________________________________________________ test_bytes_invalid_cpython ______________________________________________________

    def test_bytes_invalid_cpython():
        # PyO3/pyo3#2770 is now fixed
        s = SchemaSerializer(core_schema.bytes_schema())
    
        with pytest.raises(UnicodeDecodeError, match="'utf-8' codec can't decode byte 0x81 in position 0: invalid utf-8"):
>           s.to_python(b'\x81', mode='json')
E           SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

Turns out that this is down to our PyPy implementation of PyUnicodeDecodeError_Create. Looks like there's a behavioural change in Python 3.10 to error when calling PyObject_CallFunction with a #.

There's a PY_SSIZE_T_CLEAN macro which must be defined to enable these formats, which has the effect in the C header of redirecting PyObject_CallFunction to _PyObject_CallFunction_SizeT.

I don't think we can do this automatic redirection, because it's a breaking change that leads to UB as PyObject_CallFunction on Python 3.9 and older would accept # formats but treat the length as c_int instead of Py_ssize_t. So I've opted to just expose _PyObject_CallFunction_SizeT and change our implementation to use that. I verified by hand this fixes the Pydantic test suite on PyPy 3.10, and continues to work on PyPy 3.9.

A further note - it looks like on Python 3.13 the differences between PyObject_CallFunction and _PyObject_CallFunction_SizeT will be removed (and the latter symbol removed from the header, but kept in the ABI). That's a problem for the future though 😄

@mgorny
Copy link

mgorny commented Jul 5, 2023

Thank you! I can confirm that with this patch the tests on pydantic-core pass for me.

@davidhewitt davidhewitt added this pull request to the merge queue Jul 5, 2023
Merged via the queue into PyO3:main with commit 8fcf444 Jul 5, 2023
@davidhewitt davidhewitt deleted the pypy-ssize-t branch July 5, 2023 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants