DEPR: add stacklevel to FutureWarnings (GH9584)

[jorisvandenbossche]

Closes #9584

[jreback]

should revamp this....its pretty useful :)

[jorisvandenbossche]

yes, on my todo list!
The problem I encountered is that for certain Warnings that are raised deeper into the pandas internals, the needed stacklevel is not always the same for different end user use cases. So it will not be possible to fix all of them. But for many (all warnings of deprecated functions or keywords, which are raised rather top-level), it is also simple to fix.

I now patched the assert_produced_warning to check for this, but should make it option so you can turn if off when it is not possible to get it right.

[jreback]

@jorisvandenbossche sounds good. Well at least the top-level ones would be a step up.

[jorisvandenbossche]

@jreback updated this. Added a check to assert_produces_warning that the stacklevel is set correctly, but for now only check it for FutureWarning/DeprecationWarning (all other warnings could be checked as well, but it was a bit too much work to get these all right for now).

[jreback]

whoosh deep nesting here!

[jreback]

lgtm.

what we actually need is a warning that then traverses the stack and finds the user level code as opposed to using an explict stack level, but I suspect thats a bit tricky.

[jorisvandenbossche]

what we actually need is a warning that then traverses the stack and finds the user level code as opposed to using an explict stack level, but I suspect thats a bit tricky.

Hmm, that goes beyond my python skills, so for somebody else :-) (doubt if this is even possible)

Will merge tomorrow if no further comments

[jreback]

hahha

[jorisvandenbossche]

cc @njsmith since you reported this. This should solve almost all FutureWarnings we have in pandas.

[njsmith]

Doing that stack traversal thing is totally doable. Really it should be added as a feature upstream, like warn(..., not_package="pandas.*") to warn at the first level where the code isn't inside pandas (and there's some somewhat-relevant discussion in http://bugs.python.org/file40211). But in the mean time, woot, awesome.

[jorisvandenbossche]

Inferring the stacklevel is indeed not that difficult :-)

Something like this did it:

import warnings
import inspect

frame = inspect.currentframe()
stacklevel = 1

while '\\pandas\\' in frame.f_code.co_filename:
    frame = frame.f_back
    stacklevel += 1

warnings.warn('test!', stacklevel=stacklevel)

(the '\\pandas\\' is because I was testing on Windows)
But don't how robust something like this would be.

[jreback]

FYI I think u have to do a

del frame at the end to break the gc cycles

[jreback]

@jorisvandenbossche let's merge this. I think I already changed a few as I thought you had merged this.

[jreback]

@jorisvandenbossche pls rebase and merge