Proposal: support dynamically setting Python distribution

[malmaud]

Problem

A large number of Github issues on my Julia packages that wrap Python packages is related to which Python distribution PyCall.jl is using.

One issue is that the Julia package is tested to work with a certain version of Python and a certain version of the Python package being wrapped, which the user might not have (especially problematic if they have versions more recent than my Julia wrapper supports).

Or the user might have multiple Python distributions installed and PyCall picks up on the one the user isn't expected - I know this case can be fixed by setting environment variables and rebuilding PyCall.jl, but this problem still introduces frustrating friction to users (and maintainers...)

Proposed solution

Change the PyCall.jl API to allow for clients to dynamically choose the Python distribution, instead of it being a (strictly) build-time configuration. Then my packages could programmatically ensure that a Miniconda distribution with the right version of the underlying Python package is being used by PyCall.jl without requiring any work on the part of the end users or interfering with their existing Python distribution.

An issue is what if two packages are loaded in the same Julia session that both want PyCall to use a different Python distribution - maybe use an API like the below to handle that situation:

py1 = load_python_distribution("/miniconda1/libpython.so")
py2 = load_python_distribution("/miniconda2/libpython.so")

set_python_distribution(py1)
x=pyimport("x") # Loads package x from miniconda1 

set_python_distribution(py2)
y = pyimport("y") # Loads package y from minconda2

---

Thoughts? I can work on adding this functionality myself, but wanted some buy-in before committing time.

[stevengj]

Loading from multiple Python distros simultaneously at runtime is a non-starter; I don't see any way to make that happen without actually linking and initializing multiple libpython runtimes simultaneously, which would be a nightmare. Of course, you can add paths to Python's sys.path dynamically, but in general it won't be possible to mix distros.

As for determining which libpython to link at runtime, this is in fact how PyCall used to work (based on the PYTHON environment variable). In practice, however, this really interfered with static compilation and led to very slow load times (see #167), and configuring Python at build time greatly simplified the code.

Moreover, determining the python distro at runtime wouldn't really solve the problem of packages depending on a particular distro, since as you point out you might get conflicting requirements from different packages.

So, I'm not really in favor of this.

[stevengj]

Nothing seems likely to happen here.