Spaces in file names make it impossible to package for Solaris 9 and 10

[ghost]

Originally reported by: automatthias (Bitbucket: automatthias, GitHub: automatthias)

---

The script template.py file name causes packaging of setuptools to fail, because the Solaris 8, 9, and 10 package manager does not support file names with spaces: spaces are the column delimiter in its internal data files.

References

1. http://docs.oracle.com/cd/E23824_01/html/821-1473/pkgmap-4.html
2. https://mail.python.org/pipermail/distutils-sig/2013-October/022803.html
3. https://gar.svn.sourceforge.net/svnroot/gar/csw/mgar/pkg/lang-python/pysetuptools/trunk/Makefile
4. https://gar.svn.sourceforge.net/svnroot/gar/csw/mgar/pkg/lang-python/pysetuptools/trunk/files/0001-Solaris-package-mge-does-not-handle-spaces.patch

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/134

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Maciej, thanks for reporting this issue. I sympathize with your concerns. It appears you're reporting a defect/limitation in the Solaris packaging tools and seeking to extend that limitation to setuptools.

Your report states that the spaces make it "impossible" to package for Solaris 9, but your references suggest that there is already a workaround implemented.

In my opinion, that's the optimal solution. While it would be trivial for setuptools omit the use of spaces in the filename, it would also remove the demonstration that setuptools supports the use of spaces in package resources. Using spaces in the filename serves as a signal that this use-case is supported and works. It furthermore provides for a better aesthetic and is more congruent with the canonical word delimiter in English.

Consider viewing the files in Windows Explorer, for example. Without spaces, the file names are clumsily displayed:

But if spaces are employed, they appear much more naturally:

More than just aesthetics, there are cases where allowing spaces means that resources can be presented to users without transformation (whereas if the names are encoded with a spaces as underscores encoding, it must be decoded prior to user consumption).

Therefore, I believe there are tangible benefits to allowing resources with spaces in the names and having that use-case proven in the setuptools package has value. Part of this value is the exhibition of the edge cases such as the limitation in the Solaris package manager.

In my opinion, it's embarrassing that a first-class package manager could exist without having support for such file names, especially considering that most file systems have had support for these names for most of two decades.

Nevertheless, I don't want to press the issue if there is a serious impact to portability. Can you share more detail about the impact of this limitation? Is the workaround that you linked not adequate? Are there possibly other more elegant solutions? Could the packager instead use a zip-egg distribution, which would contain only that one file and thus wouldn't be subject to the disallowed filenames limitation?

[ghost]

Original comment by automatthias (Bitbucket: automatthias, GitHub: automatthias):

---

Hi Jason,

I'm a package maintainer at OpenCSW, a community project dedicated to working closely with upstream projects to improve portability of open source software. I hope you and I are on the same page with the aim of setuptools being portable, meaning that you can build, install and package unmodified setuptools code on all platforms users care about.

I'm trying to push the fix upstream so that I don't need to maintain a downstream set of patches. It'll also help other Solaris users who won't run into this problem in the future.

The best place for checking if things work as required are unit tests. A unit test can prove that resources can be presented to users without transformation, without installing any troublesome files in the filesystem.

I'm not sure about a zip-egg distribution. As long as files in the package file map have whitespace-free names, it should be fine.

The simplest test to verify if the file names are OK is this:

#!bash

cd <setuptools-install-image>
if [[ $(find . -name '* *' | wc -l) -gt 0 ]]; then echo BAD; else echo GOOD; fi

Otherwise I'm getting this:

#!text

## Building pkgmap from package prototype file.
ERROR in /home/maciej/src/opencsw/pkg/lang-python/pysetuptools/trunk/work/solaris9-sparc/build-global/CSWpy-setuptools.prototype-sparc:
    garbled entry
    - pathname: /opt/csw/lib/python2.6/site-packages/setuptools/script
    - problem: mode is not numeric.
pkgmk: ERROR: unable to build pkgmap from prototype file
## Packaging was not successful.

I agree that it's a silly limitation and you're right from software design point of view. In general, SVR4 package manager is a popular target of complaints and ridicule, for many reasons. I'm sending the bug report from a pragmatic standpoint, trying to save users and packagers headache.

Maciej

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Maciej, how does the package manager convert the setuptools tarfile into the files built for a particular target OS/Python version? I don't see anything in that Makefile that invokes setup.py to build or install setuptools. Does that happen implicitly due to the presence of a setup.py file in the source distribution?

[ghost]

Original comment by automatthias (Bitbucket: automatthias, GitHub: automatthias):

---

Hi Jason, the CATEGORIES variable is set to "python", so the build system runs python setup.py build and python setup.py --root=/path/to/install/image install, then runs the find utility to collect the files. It boils down to:

$ touch example.txt  # the file must exist
$ find . -print | pkgproto
f none example.txt 0644 maciej other

The resulting line is then used to create the package. A similar line is put into the package itself, in the pkgmap file.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Thanks for that information. Thanks also for pull request 33.

I was hoping there might be a way to use a zip egg, but given that's the way the packaging routine handles Python packages, I don't see a straightforward way to produce a zip egg (with about as much customization as is already in the existing patch).

If the issue were inherent in Solaris, I'd say that Setuptools should adjust to account for the issue. However, since the issue is in a packaging tool for Solaris, I believe the workaround in place is the best place for it. The patch serves as a signal to the limitation in the packaging system. If Setuptools were to adjust instead, there would be no good place to document the limitation in the system. That is, one couldn't provide hyperlinks to resources as you've done in this ticket to describe the impact of the limitation, because it would be subtly suppressed.

That, along with my previously-mentioned benefits to providing the resource with spaces, I'll elect not to apply the patch. I make this choice with pained reservation because the tradeoffs don't suggest an obvious choice.

I have added a comment to the code to reference this issue, which should bring attention to this issue if the script names are ever changed (so that the SVR4 patches can be considered).

[ghost]

Original comment by renstrom (Bitbucket: renstrom, GitHub: renstrom):

---

Just thought I'd chime in here that we're seeing this problem as well when trying to package up a virtualenv inside an RPM for RedHat.

[ghost]

Original comment by mmokrejs (Bitbucket: mmokrejs, GitHub: mmokrejs):

---

I just came here because the spaces and round brackets break simple searches on the filesystem.

Conside a command like:

$ find /usr -name *blah* | while read f; do ls -latr $f; done

This breaks on files containing spaces.

Honestly, I am shocked \what you say here. Your intentionally invented this commit: https://bitbucket.org/pypa/setuptools/commits/90d6e15a0f5601f1f086d456bd01c835a9a10d65
Seriously, do you want to create troubles to all of us to show setuptools can handle spaces and maybe brackets in filenames and dirnames? Why isn't the first criterion whether can the file can be placed on a filesystem and whether it is searchable? Anyway, please generate least troubles to the rest of the world. sorry to say that.

BTW, I do not mind this would have worked around in the particular example:

$ find /usr -name *blah* | while read f; do ls -latr "$f"; done

[ghost]

Original comment by arfrever (Bitbucket: arfrever, GitHub: arfrever):

---

Not using quotes for filenames in shell commands is flawed.

[ghost]

Original comment by philip_brown (Bitbucket: philip_brown, GitHub: Unknown):

---

The rational for keeping in this behaviour is outrageous.
To keep "the demonstration that setuptools supports the use of spaces in package resources" ???
No-one looks for that stuff in the language tree itself.
Nor is "a demonstration" even neccessary, or useful. Either it supports it, or it doesnt.
You now have complaints from TWO separate packaging systems that this is harmful to crossplatform use.
The considerate thing to do, would be to just rename the two files with spaces in their names.

[ghost]

Original comment by ThurnerRupert (Bitbucket: ThurnerRupert, GitHub: Unknown):

---

jason, python is for multiple platforms. every platform or operating system has legacy code built in which cannot handle spaces in filenames properly, windows included: http://www.sevenforums.com/browsers-mail/272850-downloading-file-ie8-replaces-spaces-underscores.html. setuptools purpose is not to demonstrate others deficiencies. but i find it a nice idea to have a "deficiency demonstrator" if it only does that :)

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I agree; the purpose of Setuptools is not to demonstrate deficiencies. So I'll reconsider.

There are many documented cases where Setuptools goes out of its way to support all platforms and their idiosyncracies (running unittests with LC_ALL=C, facilitating de-vendoring of bundled dependencies, supplying launcher executables for ARM-based Windows machines).

The primary reason the files are named thus is because that's the best name choice for the file or files (in my opinion), and while that choice does cause some consternation for some programmers, I believe the consternation reveals more about the programmer than the systems.

The Windows issue you cite is a perfect case in point. The user is surprised that files appear to be downloaded with space as underscore encoding, which was probably named that way by a programmer or other geek who felt obliged to omit spaces from names. Yet, the first response to the post goes into a long-winded rant that only late explains the underlying cause (the source file was named that way). What appears at first blush as a bug in IE is probably just poor name choice for the filename (at least from the perspective of the user).

And while it's not Setuptools' place to bully package maintainers into any particular behavior, it also should not be over-accommodating to support the lowest-common-denominator.

By having this behavior in place, if a developer asks herself the question, "can I use spaces in my package data filenames," she can cite Setuptools, as a foundational package of the ecosystem, as an indication that it will work where setuptools works. Without this behavior in place, the developer is left without a good indication and for pragmatic reasons may choose to avoid the use of spaces, perpetuating the traditional limitation.

Setuptools does violate two positive principles here, the principle of least astonishment and the robustness principle, but it does so with fair consideration for the implications of this choice. Were there revealed a concrete example where a suitable workaround was not feasible, I'd be compelled to rename the files.

For the platforms cited, that is not the case. Both Solaris and Redhat support spaces in filenames. The OP has indicated that there is a suitable (an appropriately-situated) workaround for the limitation in SVR4. The issue with Redhat was non-specific, and I guess it was probably fixed promptly by adding support for spaces. If there are subsequent, emergent issues with the filenames with spaces, please file a separate ticket.

Spaces in filenames have been supported by all major operating systems for almost as long as Python has been around and for twice the lifetime of Setuptools. This usage has been in place for most of four years (9c014a80) with minimal harm. It seems reasonable that Setuptools should be able to use this feature and to reveal its effects. Setuptools seeks not to be complicit in the repression of spaces in filenames.

[jayeye]

The purpose of any tool is to make life easier for the tool user, not to be a vehicle for the ego of its writer.

There is a tremendous number of Unix tools that do not handle spaces in filenames well. It would be far easier to remove the space from that file (which, btw, is the only filename with a space in the entire package).

[jaraco]

There is a tremendous number of Unix tools that do not handle spaces in filenames well.

I acknowledge that many would just prefer to admit defeat and not support spaces in filenames. I think we can rise to the challenge and do better.

Thusfar, there have been two concrete reports of easily surmountable issues over the several years that this naming has been in place.

[aixtools]

Hello Jaraco,

I see this is labeled as closed - but you pointed me here (quite a while ago, my apologies. The impact is that I have ignored that project (packaging Python3) - in part because of this issue).

Anyway, ...

Finally getting around to replying to your comment from pypa/pip#3989 (comment)

I scanned above - and shall give as short a reply as possible.

While an OS filesystem may support names with spaces it does not make them suddenly better than names without spaces.
Wonderful that setuptools supports installations that want to use that feature. But forcing it on all is dogma. It becomes your opinion of what is best for all becoming the law.

In - what I would call 'UserSpace' these may be useful (e.g., what the python developer is developing as a User application) - but for normal 'binary' packaging (what platform packagers are doing to make a language such as python more easily available) they may be undesired - aka a hindrance for normal systems administration.

In short, you forcing packagers and system admins to support spaces in all cases - meaning they must review and/or rewrite maintenance 'scripts'. Leaving the solution as a task of an individual packager only promotes non-uniformity (at best). In my case I just stopped trying. If you follow the path - I first contacted python who said, not my problem, try talking with 'pip' - and pip said you are the real owner.

IMHO - if you want to support portability and 'freedom' have 'spaces' as a default, but also demonstrate an option to either replace with an underscore, a dash, or remove and FirstLetterCapitalize. Let the 'user' of setuptools decide what is best for them.

For now:
a) there is not a simple solution
b) the easiest hack I can come up with is scan for files with 'unfortunate' characters (for the installer template format) and rename them. End result is that it never looks as documented.
c) Or, just do not package at all. setuptools becomes the show-stopper rather than as an asset to packaging and installing.

IMHO - it is about choice. I hope you will consider supporting a choice here.

[jaraco]

End result is that it never looks as documented.

I agree. It's a bit ugly that on Solaris, one must get an install of Setuptools that appears "broken" with respect to the reference implementation.

In my opinion, though, that's the price a user of Solaris pays for using a package manager that's inherently hostile to otherwise valid characters. I'm pretty sure one could download/install pip or setuptools from source without any issues. There's no excuse in 2017 for a package manager not supporting all file names valid in the file system.

if you want to support portability, demonstrate [another] option...

My understanding is that the maintenance scripts for Solaris packages already demonstrate this option and that this approach is viable and stable. What's the harm of the current approach? What would you suggest for this project to better support these package manager limitations?

[varder]

It is very bad the is space in files. It results multiple issue in Yocto build system.

[jaraco]

It is very bad the is space in files. It results multiple issue in Yocto build system.

Condolences. I hope they fix the issues soon.

[kludgeaudio]

Wonderful. Sarcasm. We have systems that have backup issues because some Windows fool thinks that spaces in filenames are a good idea, and attempts to get this remedied result only in sarcasm.

[jaraco]

Sarcasm

I wasn't being sarcastic. Maybe a bit flippant. I'm genuinely sorry that people experiencing issues. I've been using spaces in filenames since 1993 on Unix systems and since 1995 on Windows, so it seems reasonable to me that systems dealing with files (such as a build system) should have such support, and I genuinely hope they do that sooner than later.

some Windows fool

Please be mindful of the Code of Conduct. Attacks and aspersions on individuals and groups aren't tolerated here.

attempts to get this remedied result only in sarcasm

To the contrary, this issue has been adjudicated and revisited over the years, and it's still my position that we shouldn't bully projects into restricting the files they can use because a handful of tools don't support the full range of commonly supported and demonstrably useful characters in the file systems.