Vim plugin doesn't work with statically compiled Python

[zsol]

Operating system: Arch
Python version: 3.6.4
Black version: master
Does also happen on master: yep
vim version: 8.0 - https://gist.github.com/zsol/9c3d853bebc66128f611ae5a4d9d4c69

Error detected while processing /home/zsol/.vim/plugin/black.vim:
line  101:
Traceback (most recent call last):
  File "<string>", line 43, in <module>
  File "/home/zsol/.vim/black/lib/python3.6/site-packages/black.py", line 32, in <module>
    from attr import dataclass, Factory
  File "/home/zsol/.vim/black/lib/python3.6/site-packages/attr/__init__.py", line 5, in <module>
    from . import converters, exceptions, filters, validators
  File "/home/zsol/.vim/black/lib/python3.6/site-packages/attr/filters.py", line 7, in <module>
    from ._compat import isclass
  File "/home/zsol/.vim/black/lib/python3.6/site-packages/attr/_compat.py", line 139, in <module>
    set_closure_cell = make_set_closure_cell()
  File "/home/zsol/.vim/black/lib/python3.6/site-packages/attr/_compat.py", line 131, in make_set_closure_cell
    set_closure_cell = ctypes.pythonapi.PyCell_Set
  File "/usr/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: vim: undefined symbol: PyCell_Set

full trace: https://gist.github.com/zsol/3b800e69ed8995252e8be368b962790e

[ambv]

I will need more info as I am using the plugin right now with no issues. If you do this in your Vim, what does it say:

:py3 import sys; print(sys.version)

[ambv]

This will likely match 3.6 since Vim could read the syntax of black.py. So the next thing we need to see is why is pythonapi empty. Try:

:py3 import ctypes; print(repr(ctypes.pythonapi))

This will likely tell us something strange. The thing is, PyCell_Set is a function that's been in the Python API for ages (including Python 2) so the only explanation here is that you have your Python compiled into the Vim binary (and not using libpython).

Did you compile your own Python and your own Vim? If so, compile Python with --enable-shared and recompile Vim. If you're using default versions from Arch, they should be using the shared library so we'll have to dig further.

[zsol]

That's right, vim is using

3.6.4 (default, Jan  5 2018, 02:35:40)
[GCC 7.2.1 20171224]

Here's the output of that print

<PyDLL 'None', handle 7f2c4fc1b100 at 0x7f2c49de4898>

Yep I'm using the default Arch versions for vim and python: https://gist.github.com/zsol/f211d2359f7cb9ecc830bff02b85a6c6

And yeah it looks like vim is not using libpython: https://gist.github.com/zsol/1c93e08858fd17ddfa9ab65909416f5a

[ambv]

You could ask [email protected] why Python is statically linked. Maybe it's not deliberate?

In the mean time, it looks like attrs has a workaround for this anyway: python-attrs/attrs@dc6040f#diff-a05bc174beb6d89fc8e65f812a3e9360

This isn't released yet so activate your .vim/black venv, uninstall attrs and install it from GitHub:

pip install git+https://github.com/python-attrs/attrs.git

Let me know if attrs from master work around the issue. If not, I'll have to find a different workaround for people with statically compiled Python :(

[zsol]

Using attrs from master worked, thanks!

re: why python is statically linked into vim - According to its PKGBUILD this is not explicit, so maybe it's by accident? cc @anthraxx

[ambv]

cc @hynek, when is the next attrs release coming along?

[hynek]

We could use your help on python-attrs/attrs#361 which is the last blocker.

[ambv]

This will be fixed by attrs 18.1.

[ambv]

Or rather, "worked around".

[anthraxx]

No bug and no accident, this is how vim works when both runtimes were enabled at the same time. There is nothing python related statically compiled into vim. Vim does not use the regular dynamic system linker to load both libpython on application start (hence no trace via ldd or readelf) but rather loads them fully dynamical from within the vim runtime by searching for libpython3.6m.so.1.0 and libpython2.7.so.1.0
http://vimdoc.sourceforge.net/htmldoc/if_pyth.html#python-dynamic