Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

COMPAT: __Pyx_PyNumber_IntOrLong raises ValueError on PyPy #16336

Closed
mattip opened this issue May 12, 2017 · 5 comments · Fixed by #16563
Closed

COMPAT: __Pyx_PyNumber_IntOrLong raises ValueError on PyPy #16336

mattip opened this issue May 12, 2017 · 5 comments · Fixed by #16563
Labels
Compat pandas objects compatability with Numpy or Python functions
Milestone

Comments

@mattip
Copy link
Contributor

mattip commented May 12, 2017

the function __Pyx_PyNumber_IntOrLong from cython raises an exception when the conversion fails. On CPython, this will always be a TypeError from line 314, but on other implementations i.e. PyPy the call to PyNumber_Int on line 298 raises a ValueError upon failure.

Pandas catches only a TypeError. The easiest fix would be for pandas to catch both TypeError and ValueError when conversion fails, a harder fix would be to change cython to raise a ValueError on CPython as well, and then fix all the downstream libraries that use it (Or to scope the call to PyNumber_Int and convert the exception, and then require a minimum cython version for pandas-on-PyPy).

I already have the fix for catching multiple exceptions, should I issue a pull request for that?

@jreback
Copy link
Contributor

jreback commented May 12, 2017

can you show an example of where this happens?

@jreback jreback added the Compat pandas objects compatability with Numpy or Python functions label May 12, 2017
@mattip
Copy link
Contributor Author

mattip commented May 12, 2017

here is a minimal test case that raises in indexes/base.py, Index.__contains__(), line 1592 or so, note __contains__ only catches TypeError

# build extensions with 
# CFLAGS='-g -01' python setup.py --inplace --force
# then run
# PYTHONPATH=. gdb --args python /path/to/this/file.py
#
import pandas as pd

r = pd.RangeIndex(4)
print('')
print('xxxxxx test starts here')
print('break into the debuger with <Ctrl-C> and set a breakpoint')
print('b __Pyx_PyNumber_IntOrLong')
print('then "c" to continue from gdb')
print('and "c" again to continue from pdb')
import pdb;pdb.set_trace()
result = 'some string' in r
print result

here is the complete changeset to fix this in pandas. I filed an issue with cython as well, not sure where the best place for a fix is

@jreback
Copy link
Contributor

jreback commented May 12, 2017

this is fine as a PR here to get this working. But I agree the underlying is the problem.

@mattip
Copy link
Contributor Author

mattip commented May 28, 2017

Unfortunately cython is reluctant to change their code cython/cython#1704

@jreback could you weigh in on where the fix should be?

@jreback
Copy link
Contributor

jreback commented May 31, 2017

@mattip your changeset looks fine here. assuming it passes everything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Compat pandas objects compatability with Numpy or Python functions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants