Roadmap for PyCall 2.0

[stevengj]

Now that #517 is merged, there are a few things remaining to do before we try to move all PyCall users to the new interface.

First, we need to deprecate some more things (probably in a "1.9" release or similar) to help with the transition:

• deprecate @pyimport and pywrap — just do pyimport now. deprecate pywrap #633
• deprecate getindex(::PyObject, ::Integer) and similar — these will stop subtracting 1 in PyCall 2.0, so they will have to be deprecated for now. deprecate o[integer] so that we can transition to 0-based access #632
• Make sure major PyCall-using packages (PyPlot, SymPy, etcetera) continue to work with PyCall master, albeit with lots of deprecation warnings.

Some other things I would like to change/fix during the big transition:

• Make automatic conversion more conservative about producing native Julia objects. Convert Python lists to PyVector, dictionaries to PyDict, and NumPy arrays to PyArray by default, but don't convert subclasses of these? (see RFC: Lazier and more interface-based conversion #617)
• Audit calls toPyPtr to see if they need GC.@protect for gc safety. be more careful about potential gc of pyptrs #630

[marius311]

Are we slated to lose @pyimport numpy as np type syntax? If so, I'd cast my vote against that, I find it quite logical to be able to mirror exactly the Python importing syntax, and I'd bet it makes things for anyone coming from Python simpler too.

[tkf]

I think #599 is the ultimate way to make PyCall Pythonista-friendly. If you know Python syntax, just eval/exec it. If py"..." has a proper namespacing, it's PEP 20-compliant so no Pythonista would complain :)

[marius311]

Interesting, so eventually it would just be py"import numpy as np" and then you have access to np from Julia? That's pretty cool, I see no problem with that solution from a user perspective (all I'd ask is that @pyimport stays around until that's implemented).

[stevengj]

Yes, @pyimport numpy as np is deprecated and will disappear. You now do np = pyimport("numpy").

And py"import numpy as np" seems unlikely to ever define a variable np in Julia. @tkf is just talking about enhancing the usage of Python snippets via py"..." so that it is better namespaced. I don't think it's feasible to have py"..." blocks use the same namespace as surrounding Julia code, however. It seems like there's no way to do that in a local scope, for example, and you wouldn't want py"..." code that executes in a local scope to suddenly start defining a bunch of global Julia variables with eval.

The goal of PyCall is not to allow you to use Python syntax in Julia (except in py"..." blocks).

[tkf]

Also:

• Switch to Compat.hasproperty when Add hasfield and hasproperty JuliaLang/Compat.jl#650 is merged and released.

[stevengj]

What I'd like to do now is to test some of the main PyCall-using packages like SymPy and PyPlot on PyCall master — we should make sure they should continue to work, albeit with lots of deprecation messages, before making a release with getproperty. Help with this would be appreciated.

[tkf]

By testing, you mean reverse-dependency tests like this JuliaPy/PyPlot.jl#425, JuliaPy/SymPy.jl#251, and JuliaPy/Pandas.jl#47 (note: all of them worked)? Or just trying PyCall master in daily use?

[stevengj]

Reverse-dependency tests are fine. PyPlot's test suite isn't very good, though, so it requires some manual playing around. In particular, would be good to fix #636

[tkf]

Now that #636 is fixed, are there other blockers for v1.19?

[stevengj]

No, I think we are ready for a new 1.x release.

[ararslan]

The getindex(::PyObject, ::AbstractString) deprecation introduced two years ago means that accessing values in Python dicts requires d.__getitem__(k) instead of d[k]. I'm assuming that the removal of that deprecation would permit using d[k] directly, which would be nice. Is there an expected timeline now for PyCall 2.0? Seems all but one of Steven's items in the issue body have been implemented.

[tkf]

Another thing we can do is to change the default build to use libpython shipped via the artifact system, even in Linux. This might improve the experience of end-users who happen to use Python libraries as transitive dependencies.