PyPI does not accept wheel file name with `.` replaced with `_` #10030

Jackenmen · 2021-09-11T14:54:02Z

Describe the bug
Trying to upload a wheel with file name that has . replaced with _ in the dist name, results in an error:

λ twine upload dist/* --verbose
Uploading distributions to https://upload.pypi.org/legacy/
  dist\simplepackage_py-1.0.0-py2.py3-none-any.whl (1.1 KB)
  dist\simplepackage.py-1.0.0.tar.gz (0.5 KB)
username set from keyring
password set from keyring
username: jack1142
password: <hidden>
Uploading simplepackage_py-1.0.0-py2.py3-none-any.whl
100%|█████████████████████████████████████████████████████████████████████████████| 3.70k/3.70k [00:01<00:00, 3.18kB/s]
Content received from server:
<html>
 <head>
  <title>400 Start filename for 'simplepackage.py' with 'simplepackage.py'.</title>
 </head>
 <body>
  <h1>400 Start filename for 'simplepackage.py' with 'simplepackage.py'.</h1>
  The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/>
Start filename for &#x27;simplepackage.py&#x27; with &#x27;simplepackage.py&#x27;.


 </body>
</html>
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Start filename for 'simplepackage.py' with 'simplepackage.py'.

Expected behavior
I expected PyPI to accept a wheel file named simplepackage_py-1.0.0-py2.py3-none-any.whl for a package named simplepackage.py per the wheel spec

To Reproduce

git clone https://github.com/jack1142/simplepackage.py
cd simplepackage.py
python -m pip install -U build twine
python -m build
twine upload dist/*

My Platform

build==0.6.0.post1
twine==3.4.2
flit==3.3.0

Additional context
I first thought this is an issue with flit but then I was pointed out to the wheel spec and told that this should be fixed in warehouse:
pypa/flit#442

The text was updated successfully, but these errors were encountered:

di · 2021-09-11T15:28:40Z

I think this is a valid bug. Here's where the validation happens:

https://github.com/pypa/warehouse/blob/3802138b4eed60a1b8218168919e6de3792e9105/warehouse/forklift/legacy.py#L1190-L1197

And here's a test that can be used to write a failing test for this: https://github.com/pypa/warehouse/blob/3802138b4eed60a1b8218168919e6de3792e9105/tests/unit/forklift/test_legacy.py#L2391-L2428

Additionally it seems like pkg_resources.safe_name might not be the right thing to use here:

>>> pkg_resources.safe_name("simplepackage_py-1.0.0-py2.py3-none-any.whl")
'simplepackage-py-1.0.0-py2.py3-none-any.whl'

takluyver · 2021-09-13T09:18:44Z

I'm pretty sure using safe_name on an existing filename is wrong, and I suspect using it on the project name isn't exactly right either. The docs say:

Return a “safe” form of a project’s name, suitable for use in a Requirement string, as a distribution name, or a PyPI project name. All non-alphanumeric runs are condensed to single “-” characters, such that a name like “The $$$ Tree” becomes “The-Tree”. Note that if you are generating a filename from this value you should combine it with a call to to_filename() so all dashes (“-”) are replaced by underscores (“_”).

There was an attempt to make it follow normalisation defined in PEP 503, but that was rejected:

It's not obvious from this PR what safe_name is used for, so it's not clear that PEP 503 even applies. PEP 503 is about normalizing names in a package index...

I think the right thing to do is to normalise the project name with the trivial normalisation function in PEP 503, except replace - with _ to get the expected prefix for the filename.

The project name should also be verified (metadata spec), but hopefully that's already done somewhere else.

uranusjr · 2021-09-22T12:21:43Z

I created a PR to remove the safe_name usage in the filename check logic.

takluyver · 2021-11-26T09:24:28Z

Could I gently encourage someone with merge rights to look at @uranusjr's PR #10072 ?

I added support for namespace packages in Flit recently, and the obvious way to use it will give you a . in the project name (the project name and the import name are the same by default), which means that uploads fail.

dstufft · 2022-07-09T23:22:50Z

The situation about what file names are acceptable is extremely messy right now, see this discuss thread.

I would prefer if we didn't change Warehouse here until someone does the work to actually fix the specs as to what the requirements on file names are, since any change we make here has risks to make the situation even messier.

mgorny · 2023-02-24T05:43:27Z

Can we please move forward? From my quick testing, only pdm-pep517 and setuptools still use old normalization rules. All other PEP517 build systems use new rules (while my original test didn't involve dots in filenames, I've just retested and they're normalizing dots as well).

As you can see, people are repeatedly hitting problems because of this. My personal stake is that we're seeing different build systems follow different rules, and we really can't fix this without suddenly breaking a lot of projects using setuptools right now (because pypi would start rejecting their wheels).

dstufft · 2023-02-24T18:35:37Z

I don't think the status quo has changed from the discourse thread I linked last year. The current spec mandates something that isn't being followed in practice, and I'm -1 on changing anything in Warehouse until we clarify the exact rules we want these filenames to follow, with a plan on how to deal with the mess we're in currently. That should be done through a PEP, but nobody has spent the time to do that work.

mgorny · 2023-02-24T19:35:58Z

The current spec mandates something that isn't being followed in practice

This simply isn't true. The only build systems "not following the practice" are pdm-pep517 and setuptools. pdm-pep517 is barely used by a handful of packages (and already replaced by pdm-backend, sigh).

As for setuptools, we're facing a hen & egg problem. Setuptools can't follow the new standards because pypi won't accept the resulting wheels. PyPI refuses to accept the resulting wheels because "standards aren't being followed".

[…] we clarify the exact rules we want these filenames to follow, with a plan on how to deal with the mess we're in currently. That should be done through a PEP, but nobody has spent the time to do that work.

Could you be more specific on what needs to be "clarified"? There are the PyPA standards, as well as PEPs referencing them or deferring to them. Yes, there was some discussion that boiled down to "I would like to see the standard say something else" but IMO no progress on that only proves that further changes aren't justified — and I'm really happy that they aren't because the last thing we need right now is a third standard that will fracture the build systems even more.

dstufft · 2023-02-25T16:48:30Z

This simply isn't true. The only build systems "not following the practice" are pdm-pep517 and setuptools. pdm-pep517 is barely used by a handful of packages (and already replaced by pdm-backend, sigh).

It's also Flit, if you test a project with a . in it's name:

Build System	sdist filename	wheel filename
flit-core	Test.Project-1.tar.gz	test_project-1-py2.py3-none-any.whl
hatchling	test_project-1.tar.gz	test_project-1-py2.py3-none-any.whl
meson-python	test_project-1.tar.gz	test_project-1-py3-none-any.whl
pdm-pep517	Test.Project-1.tar.gz	Test.Project-1-py3-none-any.whl
poetry-core	test_project-1.tar.gz	test_project-1-py2.py3-none-any.whl
setuptools	Test.Project-1.tar.gz	Test.Project-1-py3-none-any.whl

As for setuptools, we're facing a hen & egg problem. Setuptools can't follow the new standards because pypi won't accept the resulting wheels. PyPI refuses to accept the resulting wheels because "standards aren't being followed".

Setuptools is also the most widely used build backend afaik, particularly in existing projects where they've already named themselves. I suspect that the vast bulk of people who have a name with a . in it are either still on setuptools or have renamed themselves.

Could you be more specific on what needs to be "clarified"? There are the PyPA standards, as well as PEPs referencing them or deferring to them. Yes, there was some discussion that boiled down to "I would like to see the standard say something else" but IMO no progress on that only proves that further changes aren't justified — and I'm really happy that they aren't because the last thing we need right now is a third standard that will fracture the build systems even more.

That isn't an accurate characterization of that discussion. It may have started out with someone asking to see the "spec" change, but once we started to investigate the change, something far messier was discovered.

Originally there was just PEP 427, which was inconsistent in it's text in how names should be normalizes, with the text saying one thing but the example code implementing something differently. The text of the PEP didn't actually make sense if read strictly, and everyone who had implemented it at the time had implemented what the code said.

Then it was discovered that due to a bad interaction between PEP 427 and PEP 440, some more esoteric version numbers weren't possible to represent, because the PEP 427 rules were too strict, so it was asked to relax those rules to allow a few characters that PEP 440 needed. In that discussion the PEP was ported over to packaging.python.org, and was subsequently updated to fix the version number issue, and was inadvertently, and without much discussion updated to also change the normalization rules for names as well.

Unfortunately, that change to names was to making what was valid stricter, which would mean that all existing tools were suddenly emitting "invalid", according to the spec, wheel names. Since this was done without any real discussion, nobody picked up on that and there was no plan for how to handle that backwards incompatibility.

Fast forward a bit and several other backends have implemented the wheel spec and they used the backwards incompatible text at packaging.python.org to guide their implementation. Which because there was overlap between the two, mostly worked fine until someone stumbled upon one of the edge cases that the backwards incompatible change broke. They opened up this issue and it's been sitting in limbo ever since.

Because the current situation is so messy, I am -1 on changing anything in Warehouse until someone writes a PEP and clarifies (A) What the actual rules are intended to be now and (B) what the migration path is supposed to look like.

I don't actually care what the answer to (A) is, it's bikeshedding to me and it doesn't really matter.

(B) is important though, because right now if Warehouse just suddenly switched to implementing the PEP as it exists today with the undiscussed backwards incompatibility, it would break everyone who was using a . in their name who were using setuptools as their build backend. Of course the current situation is that everyone who wants to use a . with those other build backends are currently broken, but that broken is the status quo so they've already figured out a path around it.

I suspect the answer is that the PEP needs updated to allow the names to either be normalized or not, but potentially recommend that they be normalized, because that's the only way I can see for implementing this in Warehouse that isn't going to hard break everyone using a . in their name with setuptools. But I refuse to make the situation even messier by just implementing that in Warehouse without getting some actual answers codified.

tl;dr Original spec said two things, one of which was broken and one of which everyone implemented, the spec was later updated with little to no discussion in a backwards incompatible way and others started to implement the backwards incompatible spec. Warehouse will not change until someone puts in the effort to clarify both the rules, and what the intended migration from the previous to the new rules is intended to be.

CAM-Gerlach · 2023-06-18T10:51:18Z

Just for others' reference, see this Discourse thread for some recent discussion on this topic.

A couple updates to the table above since it was first published:

Flit >=3.9 (released) now normalizes sdist names following PEP 625 (i.e. same as the wheel spec) per flit_core: Normalize sdist names according to PEP 625 pypa/flit#628
pdm-pep517 is deprecated and no longer developed; pdm-backend is the current PDM backend, and does normalize both sdist and wheel names.
Setuptools has now indicated their willingness to do the same per [BUG] Wheel naming is not following PEP 491 convention pypa/setuptools#3777, given all other backends do so and following the general consensus in the Packaging Discourse thread

Therefore, the current state of the backends listed above is now only Setuptools does not normalize, and that may be changing soon-ish per the aforementioned issue.

warsaw · 2023-07-17T20:06:27Z

I'm wondering if there's any progress on this bug, as I am blocked from uploading any new versions of my flufl packages, now that PDM does the right thing and preserves the project name in the metadata and upload request?

di · 2023-07-17T20:15:22Z

I think there's two steps here:

Start accepting all filenames that are valid per PEP 427 (and the updates to it)
Start rejecting filenames that aren't compliant with the spec

The first will resolve this issue and end @warsaw's woes, and can happen ~immediately. The second is what @dstufft's primary concern is IIUC, which I think can be mitigated by a long deprecation period with sufficient warning, as well as coordination amongst the primary build tools for wheels. We probably need a separate issue for that effort.

warsaw · 2023-07-18T01:35:26Z

Thanks @di - I'd love to get an ETA on resolving the first issue. I've got uploads backed up waiting for it. Is this issue (#10030) the right one to follow for progress?

Jackenmen added the bug 🐛 label Sep 11, 2021

uranusjr mentioned this issue Sep 22, 2021

Use PEP 503 rules to validate upload filename #10072

Closed

takluyver mentioned this issue Nov 26, 2021

Invalid wheel filename generated for namespace package pypa/flit#473

Closed

This was referenced Nov 30, 2021

Add equality operator to EnvironCredential jaraco/keyring#549

Merged

New backend for getting credentials from environment variables jaraco/keyring#539

Open

vincentsarago mentioned this issue May 11, 2022

0.6.0a1 breaks namespace packages developmentseed/titiler#471

Closed

mkniewallner mentioned this issue Jun 6, 2022

poetry publish fails for package with . in its name. python-poetry/poetry#5782

Closed

2 tasks

dimbleby mentioned this issue Sep 25, 2022

Normalize sdist filenames python-poetry/poetry-core#484

Merged

woodruffw mentioned this issue Oct 4, 2022

Meta: Warehouse's handling and validation of distribution filenames #12316

Open

This was referenced Oct 5, 2022

Case sensitivity of project names when publishing to PyPI python-poetry/poetry#1202

Closed

Regression in namespace package metadata on poetry-core 1.1.0rc1 python-poetry/poetry#6198

Closed

PeterJCLaw mentioned this issue Mar 5, 2023

Warn about potentially incompatible project name canonicalisations python-poetry/poetry-core#560

Closed

2 tasks

FFY00 mentioned this issue Mar 17, 2023

Move to setuptools FFY00/ffy00.packaging#1

Closed

ofek mentioned this issue Apr 23, 2023

Problem publishing at pypi Client error '400 Start filename for 'github3.py' with 'github3.py'.' pypa/hatch#833

Closed

staticdev mentioned this issue Apr 23, 2023

Remove hatch strict naming sigmavirus24/github3.py#1144

Merged

ofek mentioned this issue Apr 24, 2023

Modernize the build: switch to hatchling sigmavirus24/github3.py#1142

Merged

di mentioned this issue Jun 22, 2023

Replace deprecated usage of pkg_resources #13991

Closed

di mentioned this issue Jul 18, 2023

Accept file names with . replaced with _ #14155

Merged

di closed this as completed in #14155 Jul 19, 2023

wwuck mentioned this issue Jul 23, 2023

Please add support for credentials via environment variables pypa/pip#4789

Open

This was referenced Aug 2, 2023

put pretty names in metadata python-poetry/poetry-core#620

Merged

Clarify dots canonicalisation change is breaking python-poetry/poetry#7607

Closed

krismolendyke mentioned this issue Oct 3, 2023

Remove deprecated urllib3 secure extra krismolendyke/tank-utility#2

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PyPI does not accept wheel file name with `.` replaced with `_` #10030

PyPI does not accept wheel file name with `.` replaced with `_` #10030

Jackenmen commented Sep 11, 2021

di commented Sep 11, 2021

takluyver commented Sep 13, 2021

uranusjr commented Sep 22, 2021

takluyver commented Nov 26, 2021

dstufft commented Jul 9, 2022

mgorny commented Feb 24, 2023

dstufft commented Feb 24, 2023

mgorny commented Feb 24, 2023

dstufft commented Feb 25, 2023

CAM-Gerlach commented Jun 18, 2023

warsaw commented Jul 17, 2023

di commented Jul 17, 2023

warsaw commented Jul 18, 2023

PyPI does not accept wheel file name with . replaced with _ #10030

PyPI does not accept wheel file name with . replaced with _ #10030

Comments

Jackenmen commented Sep 11, 2021

di commented Sep 11, 2021

takluyver commented Sep 13, 2021

uranusjr commented Sep 22, 2021

takluyver commented Nov 26, 2021

dstufft commented Jul 9, 2022

mgorny commented Feb 24, 2023

dstufft commented Feb 24, 2023

mgorny commented Feb 24, 2023

dstufft commented Feb 25, 2023

CAM-Gerlach commented Jun 18, 2023

warsaw commented Jul 17, 2023

di commented Jul 17, 2023

warsaw commented Jul 18, 2023

PyPI does not accept wheel file name with `.` replaced with `_` #10030

PyPI does not accept wheel file name with `.` replaced with `_` #10030