Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use find_libpython.py in deps/build.jl #556

Merged
merged 9 commits into from
Sep 23, 2018

Conversation

tkf
Copy link
Member

@tkf tkf commented Sep 6, 2018

as I suggested in JuliaPy/pyjulia#190 (comment)

I think we can also remove this part (untouched ATM):

PyCall.jl/deps/build.jl

Lines 95 to 134 in 49029e0

# We do this *last* because the libpython in the system
# library path might be the wrong one if multiple python
# versions are installed (we prefer the one in LIBDIR):
for lib in libs
lib = splitext(lib)[1]
try
return (Libdl.dlopen(lib, Libdl.RTLD_LAZY|Libdl.RTLD_DEEPBIND|Libdl.RTLD_GLOBAL),
lib)
catch e
push!(error_strings, string("dlopen($lib) ==> ", e))
end
end
if "yes" == get(ENV, "PYCALL_DEBUG_BUILD", "no") # print out extra info to help with remote debugging
println(stderr, "------------------------------------- exceptions -----------------------------------------")
for s in error_strings
print(s, "\n\n")
end
println(stderr, "---------------------------------- get_config_vars ---------------------------------------")
print(stderr, read(`python -c "import distutils.sysconfig; print(distutils.sysconfig.get_config_vars())"`, String))
println(stderr, "--------------------------------- directory contents -------------------------------------")
for libpath in libpaths
if isdir(libpath)
print(libpath, ":\n")
for file in readdir(libpath)
if occursin("pyth", file)
println(" ", file)
end
end
end
end
println(stderr, "------------------------------------------------------------------------------------------")
end
error("""
Couldn't find libpython; check your PYTHON environment variable.
The python executable we tried was $python (= version $v);
the library names we tried were $libs
and the library paths we tried were $libpaths""")

ctypes.util.find_library used in find_libpython.py is probably good enough. What do you think?

(Edit: the fallback should stay: https://travis-ci.org/JuliaPy/PyCall.jl/builds/425108395)

deps/build.jl Outdated
else
pipe = process = open(cmd)
end
libpy_name = read(pipe.out, String)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ATM find_libpython.py is run in the default "print the best one" mode. I can also run it with --list-all option, loop over all the candidate paths and use the first one that Libdl.dlopen succeeds. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best to be defensive here and try multiple paths, I suppose

@stevengj
Copy link
Member

stevengj commented Sep 10, 2018

I find it very frustrating that the code to find libpython from python has to be this convoluted. Not sure what can be done about it, though.

@tkf
Copy link
Member Author

tkf commented Sep 10, 2018

Yeah, I can't agree enough... The upside is that we wrote (or about to finish writing) a portable script so that other people can just grab it if they need it.

Locate libpython associated with this Python executable.
"""

# License
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I inlined the license so that it's easier to grab this file and put it in other projects.


# LIBPL seems to be the right config_var to use. It is the one
# used in python-config when shared library is not enabled:
# https://github.com/python/cpython/blob/v3.7.0/Misc/python-config.in#L55-L57
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it seems adding LIBPL was the key (it was not in the PyJulia version; I'll back-port it later). This script finds libpython in case the old julia function couldn't: #565 (comment)

@tkf tkf changed the title WIP: Use find_libpython.py in deps/build.jl Use find_libpython.py in deps/build.jl Sep 15, 2018
@tkf
Copy link
Member Author

tkf commented Sep 15, 2018

I think it's good to go. CI on appveyor is failing even on master at:

PyCall.jl/test/runtests.jl

Lines 327 to 330 in 0f685cf

# import pyzmq to test PR #294
let already_installed = "pyzmq" Conda._installed_packages()
@test !ispynull(pyimport_conda("zmq", "pyzmq"))
@test "pyzmq" Conda._installed_packages()

which is irrelevant to this PR (the build is already succeeded at this stage).

@tkf
Copy link
Member Author

tkf commented Sep 18, 2018

@andreasnoack
Copy link

It would be great to get this one merged since it fixes #565 which currently causes many failures on CIBot.

@tkf
Copy link
Member Author

tkf commented Sep 23, 2018

Backported to JuliaPy/pyjulia#203 and the tests pass there too.

lib_basenames = list(candidate_names(suffix=suffix))

for directory in lib_dirs:
for basename in lib_basenames:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might want to guard this with if directory: in case get_config_var returned None

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's done with append_truthy utility function defined above. If my usage of utility function is confusing I can remove it and add explicit ifs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I see.

@stevengj
Copy link
Member

I still wish we had a simpler solution, but I will merge this since it seems like it handles cases that our current code doesn't.

@stevengj stevengj merged commit fb88f4d into JuliaPy:master Sep 23, 2018
@tkf tkf mentioned this pull request Oct 17, 2018
1 task
@tkf tkf deleted the find_libpython.py branch November 4, 2018 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants