Failed to detect AST nodes in Windows ipython for verb calls

[GitHunter0]

Feature Type

• Adding new functionality to datar

• Changing existing functionality in datar

• Removing existing functionality in datar

Problem Description

The code below

    import datar.all as d
    import pandas as pd
    new_name = "y"
    pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'})

generates the following warning message:

c:\Users\me\miniconda3\envs\cenv\lib\site-packages\pipda\utils.py:89: PipeableCallCheckWarning:
Failed to detect AST node calling `rename`, assuming a piping call.

Feature Description

Being able to suppress the warning message above.

Additional Context

No response

[pwwang]

Could you also indicate the environment where you were running the code? i.e. python script, jupyter, python REPL, etc

[GitHunter0]

For sure @pwwang , I'm using in a .py file in VScode (via its Interactive Window, which uses jupyter kernel).

[pwwang]

I can run it with no problem:

here is the version info:

Version: 1.74.3 (system setup)
Commit: 97dec172d3256f8ca4bfb2143f3f76b503ca0534
Date: 2023-01-09T16:59:02.252Z
Electron: 19.1.8
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.25276
Sandboxed: No

vscode-jupyter: v2022.11.1003412109

[GitHunter0]

The problem was indeed with VScode, I updated it to the last version and that warning message disappeared, thank you @pwwang .

[GitHunter0]

@pwwang , I checked that the issue was solved only for python 3.8 (after I updated datar and restarted VScode).
All conda environments I tested with python 3.9 or 3.10 that message appears, even directly in the command prompt (without VScode and ipython), it's very weird...

[GitHunter0]

Steps to reproduce the issue.
Go to Windows OS powershell and type:

conda create --name py3.9 python=3.9 
conda activate py3.9 
pip install datar[pandas] 
python

Then run:

    import datar.all as d
    import pandas as pd
    new_name = "y"
    pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'})

It will display this message:

C:\Users\me\miniconda3\envs\py3.9\lib\site-packages\pipda\utils.py:89: PipeableCallCheckWarning: Failed to detect AST node calling `rename`, assuming a piping call.
  warnings.warn(

[pwwang]

This is expected.
There is some description here: https://pwwang.github.io/pipda/piping/#fallbacks-when-ast-node-detection-fails

Tracing executing node is not available in the raw python REPL.
How about ipython or bpython?

[GitHunter0]

I could not install bpython to test, but with ipython the same issue occurs and since VScode relies on ipython for its Interactive Window, it inherits the same behavior.

[GitHunter0]

Maybe an option to temporarily silent that message would suffice as a workaround (in the context of a function, we could turn off in the beginning and turn on in the end).

[pwwang]

You can silence it by:

pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'}, __ast_fallback="piping")

It's kinda wired that ipython shows the same issue.

Could you provide datar.get_versions() and also the version of your python?

[GitHunter0]

Awesome, __ast_fallback="piping" works.

Could you provide datar.get_versions() and also the version of your python?

For sure:

python      : 3.9.16 (main, Jan 11 2023, 16:16:36) [MSC v.1916 64 bit (AMD64)]
datar       : 0.11.1
simplug     : 0.2.2
executing   : 1.2.0
pipda       : 0.11.0
datar-numpy : 0.1.0
numpy       : 1.24.1
datar-pandas: 0.2.0
pandas      : 1.5.2

The python version installed on my Windows OS is 3.9.4.

[pwwang]

The python version installed on my Windows OS is 3.9.4.

Sorry, I meant ipython.

[pwwang]

pipda v0.11.1 was just released. With it, you can change the default fallback of a verb:

# importing ...
d.rename.ast_fallback  = "piping"
# So you don't need to pass "__ast_fallback" to every call
new_name1 = 'y'
new_name2 = 'z'
pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name1: 'x'})
pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name2: 'x'})

[GitHunter0]

pipda v0.11.1 was just released. With it, you can change the default fallback of a verb:

Very nice to know.

[GitHunter0]

Sorry, I meant ipython.

The ipython version is 8.8.0

[pwwang]

I don't see a critical difference between your environment and mine below:

Could you also take a similar snapshot?

[GitHunter0]

For sure:

[pwwang]

Looks like the only difference is the platform. I can reproduce the issue on Windows. The snapshot I took was on Linux (WSL).
I'll dig deeper to see if there is a fix.

[pwwang]

Looks like it is due to a python bug on Windows. AST nodes cannot be detected when we are trying to deconstruct dicts as keyword-arguments.

new_name = 'y'
# Warns
pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'})
# But this is Okay
pd.DataFrame(dict(x=[2])) >> d.rename(y='x')

Related:

alexmojaki/executing#27
samuelcolvin/python-devtools#93
samuelcolvin/python-devtools#93 (comment)
python/cpython#89271

[GitHunter0]

Great, at least now we know the root cause.

[pwwang]

Close it for now since this problem can not be solved in this repo.

A temporarily solution is to pass __ast_fallback to bypass the AST node detection:

pd.DataFrame(dict(x=[2])) >> d.rename(**{new_name: 'x'}, __ast_fallback="piping")

See more details here: https://pwwang.github.io/pipda/piping/#fallbacks-when-ast-node-detection-fails