bdist_wheel creates a wheel that installs incorrectly publicsuffix (data file issue)

[agronholm]

Originally reported by: pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne)

---

Using the latest setuptools, pip, virtualenv and wheel 0.24 on win7 74bits, Python 2.7.8 32bits.

When installed with
pip install publicsuffix the publicsuffix.txt data files goes into site-packages side by side with the .py file

If I create a wheel from publicsuffix, and then install this wheel then the publicsuffix.txt data file gets installed in my current directory and not site-packages and the package becomes unusable

See https://pypi.python.org/pypi/publicsuffix/1.0.5
http://www.tablix.org/~avian/git/publicsuffix.git/
and a clone at:
https://bitbucket.org/pombredanne/publicsuffix2/src?at=master

---

• Bitbucket: https://bitbucket.org/pypa/wheel/issue/120

[agronholm]

Original comment by Giles Brown (Bitbucket: giles_brown, GitHub: Unknown):

---

Same here.

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

Actually my conclusion is that any package that uses files that are not "package data" is likely a lost cause for wheels...
There are several of these which are rather popular and the issue is that both packaging AND the code using these files need to be changed for wheels to work in these cases.
I am rather puzzled by this change of behaviour when compared to running a setup.py...

For the case of publicsuffix which is just one among many package with similar issues, I am actually forking it and repackaging it but this is not a solution IMHO

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

I forked publicsuffix as publicsuffix2 https://github.com/pombredanne/python-publicsuffix2 and pushed it to Pypi

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

@pombredanne would your issue be a duplicate of #92 ?

[agronholm]

Original comment by Giles Brown (Bitbucket: giles_brown, GitHub: Unknown):

---

benjaoming yes I think this is the same as that issue:

The setup.py for https://pypi.python.org/pypi/publicsuffix/1.0.5 uses distutils and specifies:

▷⋅⋅⋅data_files = [('', ['publicsuffix.txt'])],

This gets install directly into sys.prefix when install from a wheel, but the code expects it alongside it's _ file _.

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

FYI, since pip 7.0.0 all packages are wheeled before install, meaning that this bug and #92 are getting prime exposure in several packages.
See pypa/pip#2874

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

@benjaoming yes this sounds like the same as #92

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

@dholth would you know where to look for a fix to this?

[agronholm]

Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth):

---

I would look in the source code for bdist_egg. Egg does something different with "package data" compared to bdist_wheel that may be more desirable; try making an egg of the offending packages and see what you think.

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

@dholth

This is not a matter of changing the build method, but an issue of pip forcing a call to bdist_wheel which creates unexpected results for data_files.

[agronholm]

Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth):

---

If the way egg works is better, then we will copy that technique into bdist_wheel

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

Ah that way, yes, agreed!

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

@dholth in earnest I find the data_files directive confusing at best and a tad wharty...
I might rather prefer the current wheel behaviour somehow and we should IMHO encourage package_data at all times.
In fact the behaviour of eggs to ignore sys.prefix might be the bug rather than wheel being buggy by respecting it?

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

You cannot encourage package data if say a python application wants to put some files in /usr/share/application-data or /etc/application.conf -- which would be a totally fair case IMO :)

[agronholm]

Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth):

---

There was a discussion on distutils-sig a month or so ago about fixing data_files, a feature that has been broken at least since the introduction of virtualenv and probably longer than that. The big problem is that if you are inside a virtualenv then it is not OK to let you install files outside of $VIRTUAL_ENV. Second, how do you find $VIRTUAL_ENV/etc/ when you asked to be installed into /etc/?

The proposal was to have a lot of aliases e.g. $confidir instead of /etc/ and provide APIs for installing into and locating those directories at runtime.

Next step is to define the list of aliases.

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

AFAIK if you're in a virtualenv, then using sys.prefix is safe, where as I'm not sure what the defined behavior of '/' is. The idea is that sys.prefix resolves to the same, weather accessing it from setup.py or post-installation from the application itself.

Regarding '/', you always have the option of leaving this behavior to the setup.py implementation... I mean, there might not be a fits-all behavior for an application expecting a file in '/etc'.. maybe it shares it with a non-python app.

For instance an application that wants to create '/etc/init.d/my-service', this would be completely meaningless in a virtualenv.

I don't think that the virtualenv part for '/' was solved in distutils, so maybe it's better to focus on the immediate regressions and just implement something that matches previous behavior?

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

I have tried to sum up my personal take here: pypa/pip#2874 (comment)
Sorry for the cross posting...

Net net for me : I think that the setuptools egg behaviour of installing data_files side-by-side with the code is preferable.
The distutil behaviour should be deprecated in favor of the setuptools behaviour.

Having Python packages install more than Python related thing outside of the target directory does not make sense, is fraught with possible problems including possible security issues and is better left to native OS package installation approaches. It would not work consistently anyway across OSes in most cases. Think about what /etc/init.d/ means for Windows...

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

@pombredanne
I commented on Github, too.

For your point about platform universal behavior, please consider that setup.py is a python application in itself, and it's a very powerful tool for resolving platform-dependent installation procedures, not a reason to entirely rule out the possibility that I made a conditional concatenation to data_files in order to have a specific file placed on *nix.

[agronholm]

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

---

I think we should probably just ditch data_files all together and focus on getting the more expressive path stuff done. I think making data_files a poor man's clone of package_data that also (sometimes) supports absolute files is silly.

[agronholm]

Original comment by Domen Kožar (Bitbucket: iElectric, GitHub: iElectric):

---

Somehow relevant is the problem in setuptools https://bitbucket.org/pypa/setuptools/issues/130/install_data-doesnt-respect-prefix

[agronholm]

Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):

---

Since a while has gone, let's contemplate some more: I've been working on a project with data_files for a while now. The reason is legacy and an illusion that pip can handle platform independent installation processes AND virtualenv. Seeing things in retrospect: If the ill-defined data_files directive wasn't an option anymore, we would be forced to handle things elsewhere. Which would be good. I wouldn't have gone through lots of trouble to realize inconsistencies between distutils/setuptools/wheel. My life would actually have been easier had I been forced to look at other solutions, and we would probably have been more abrupt about removing bad architecture.

Summary: I'm +1 for ditching data_files

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

@benjaoming I am all +1 for ditching datafiles. Package files are good enough. Doing system wide install beyond the confines of a Python install does not make sense and should not be in the goals or missions of a Python package installer IMHO.

[agronholm]

Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth):

---

If you want people to write applications, and not just libraries, in
Python then it needs to be able to install files to different locations.
The right way to do this is to allow the package to ask to be installed
anywhere while still giving the installer control of where it is
actually installed. We've worked on some proposals to allow additional
named categories of files inside wheel that would all have well-defined
destinations. Even with this proposed feature pip would usually confine
installations to remain entirely inside a virtualenv.

[agronholm]

Original comment by Domen Kožar (Bitbucket: iElectric, GitHub: iElectric):

---

This is a mostly an issue with $PREFIX. data_files should support setting that to whatever and then installing share/bla.txt would be installed into $PREFIX, be it /usr/ or /opt/ or something else.

[agronholm]

Original comment by Benno Fünfstück (Bitbucket: bennofs, GitHub: bennofs):

---

Can this be closed as a duplicate of #92?

[agronholm]

Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne):

---

@bennofs As far as I am concerned I am fine to close this if you think this is a dupe of #92 ... FWIW I do consider that any data file that is not package_data is evil and should be eradicated; and that dealing with the application usecase (vs. libraries) should not be the realm of Python packaging and wheels entirely ;) but something handled elsewhere.

[agronholm]

Closed as duplicate of #92.