BUG: NumbaPendingDeprecationWarning with numba 0.58.0

[kartiksubbarao]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

df = pd.DataFrame({'col1': [1, 2, 3, 4, 5]})
df.col1.rolling(2).apply(np.prod, raw=True, engine='numba')

Running this code with the latest version of numba (0.58.0) results in the following warning:

/srv/temp/p311/lib64/python3.11/site-packages/pandas/core/window/numba_.py:72: NumbaPendingDeprecationWarning: Code using Numba extension API maybe depending on 'old_style' error-capturing, which is deprecated and will be replaced by 'new_style' in a future release. See details at https://numba.readthedocs.io/en/latest/reference/deprecation.html#deprecation-of-old-style-numba-captured-errors
Exception origin:
  File "/srv/temp/p311/lib64/python3.11/site-packages/numba/core/typing/templates.py", line 807, in _build_impl
    raise AssertionError(msg.format(pyfunc))

  result[i] = numba_func(window, *args)

Issue Description

It looks like Numba is deprecating an old style of error capturing and is printing the deprecation warning to alert developers to migrate to the new style. Here are the details:

https://numba.readthedocs.io/en/latest/reference/deprecation.html#deprecation-of-old-style-numba-captured-errors

Expected Behavior

Expected behavior is for the code to run without printing the deprecation warning.

Installed Versions

/srv/temp/p311/lib64/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : e86ed37
python : 3.11.4.final.0
python-bits : 64
OS : Linux
OS-release : 6.4.14-200.fc38.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Sat Sep 2 16:36:06 UTC 2023
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.25.2
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 65.5.1
pip : 22.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : 0.10.0
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.0
numba : 0.58.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.2
sqlalchemy : None
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[lithomas1]

Hm, at a first glance, it looks like the warning is coming from inside numba itself.

I don't have 0.58.0 available to me yet, since I'm using numba out of conda-forge, so I won't be able to verify if the issue is from pandas/numba, until that is out.

cc @esc

[kartiksubbarao]

Thanks for the response @lithomas1. I have posted a question to the numba forum to see if someone there might be able to clarify the source of the warning:
https://numba.discourse.group/t/pandas-source-of-old-style-error-capturing-warning/2169

[stuartarchibald]

Thanks for reporting this @kartiksubbarao. I'm pretty sure that this is the issue I described in #30151 (comment), i.e. that the functionality of rolling.apply would be altered such that the previously accepted case of a raw NumPy function (cf. np.prod in the OP above) would not work any more. I think this is probably a policy decision for Pandas to make, it's either a regression that needs fixing or it needs guarding more strongly and documenting that only user defined functions that are a types.FunctionType (or a jitted version of one) will be accepted (or of course some other suitable decision!). The warning that Numba is raising now will end up as an assertion error in the future, it's occurring because the implementation of @register_jitable uses @overload internally and @overload decorated functions need to return a types.FunctionType which it's not in the case of returning np.prod as it is doing now. I'm somewhat surprised that Numba manages to compile this at all!

[kartiksubbarao]

Thanks very much for the detailed reply @stuartarchibald! I see above that @lithomas1 has created a pull request #55327 to fix the issue, if you want to take a look at that please feel free as well.

[stuartarchibald]

Thanks very much for the detailed reply @stuartarchibald! I see above that @lithomas1 has created a pull request #55327 to fix the issue, if you want to take a look at that please feel free as well.

No problem. I've taken a look at the patch and left a short review. I think it will fix the issue described here, thanks for writing the patch @lithomas1.

[kartiksubbarao]

@lithomas1 When you get a chance, can you take a look at @stuartarchibald's review? I noticed that 2.1.2 was released today, so was just hoping to see if this fix might be able to be included in the next release.

[kartiksubbarao]

@lithomas1 It looks like this may have stalled, do you need some more time to review @stuartarchibald's comments and update the patch? If you can share some comments on your current thinking that'd be great, just so we have a sense of the work involved. Thanks again for your efforts on this.

[lithomas1]

@lithomas1 It looks like this may have stalled, do you need some more time to review @stuartarchibald's comments and update the patch? If you can share some comments on your current thinking that'd be great, just so we have a sense of the work involved. Thanks again for your efforts on this.

Sorry for the very slow response - I got sidetracked by some other pandas maintenance work.

I'll try to get this in for pandas 2.2 (the next minor release) - milestoning this issue as such so I don't forget.

[kartiksubbarao]

Thanks @lithomas1, much appreciated!