Retain valid names with underscores in egg_info. #4159

jaraco · 2023-12-14T15:27:00Z

Closes #2522.

Summary of changes

Closes

Pull Request Checklist

Changes have tests
News fragment added in newsfragments/.
(See documentation for details)

Closes #2522.

jaraco · 2023-12-14T15:45:55Z

I'm pleased to see that tests are passing (ignoring diffcov, which is obviously wrong), so making the fix doesn't violate any tested expectations. Therefore, I'm inclined to make the change in the hopes that it doesn't break any untested assumptions. If it causes undue disruption, we can roll it back and revisit a transitional approach.

A recent change in pypa/setuptools#4159 may have caused our `dask-cudf` wheels to be published as `dask_cudf-cu12` instead of `dask-cudf-cu12`. Additionally, `cudf_kafka` wheels would have this issue, but 1) we do not publish wheels for `cudf_kafka` and 2) the conda packages are published as `cudf_kafka` (with underscore), so be a larger refactor later on. Authors: - Ray Douglass (https://github.com/raydouglass) Approvers: - Bradley Dice (https://github.com/bdice) - Charles Blackmon-Luca (https://github.com/charlesbluca) - Jake Awe (https://github.com/AyodeAwe)

When creating a PyPi package, Python setuptools prior to v69 would record the name of a package in egg_info by replacing underscores with dashes. So after installing `pulumi-random==4.15.0`, `pip list` would report the package name as `pulumi-random`. However, setuptools >= 69 now retains the underscore in egg_info (pypa/setuptools#4159). So after installing the latest version of `pulumi-random==4.15.1`, `pip list` now reports the package name as `pulumi_random` instead of `pulumi-random`. We use pip to list installed packages so we can determine which ones are Pulumi packages for `GetRequiredPlugins`. It turns out that this change is largely benign for us. We don't really care if the reported name is `pulumi-random` or `pulumi_random`, because we will replace dashes with underscores in the returned name anyway, because the location the package is installed on disk is going to have the underscore, and it's in this location where we look for `pulumi-plugin.json`. https://github.com/pulumi/pulumi/blob/8bcef51fb8a781fcb9b9b630b4cc96fecc859640/sdk/python/cmd/pulumi-language-python/main.go#L393-L398 All that's required is updating the test's expected value, which is now `pulumi_random` as of 4.15.1 of that package. Fixes #15192

mattip · 2024-02-06T06:11:19Z

I think this might have changed wheel building for projects with a - in the project name (from the project's pyproject.toml). Now the wheel filename for a project like scipy-openblas64 will become scipy_openblas64-0.3.26-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl. When using twine to upload to PyPI the name is converted back to scipy-openblas64, but the anaconda uploader will respect the _ and try to upload to scipy_openblas64. See scientific-python/upload-nightly-action#61

jaraco · 2024-02-11T17:48:20Z

In #4167, I believe the consensus is that the name should be retained in metadata but that the filenames should not have changed. I'll be working on it there.

Updated pre-commit deps

- Updated pyproject.toml to fix the changes made in pypa/setuptools#4159 (#6) - Updated pre-commit deps

#3)

#4) * Updated pyproject.toml to fix the changes made in pypa/setuptools#4159 * Added more pre-commit hooks

* Updated pyproject.toml to fix the changes made in pypa/setuptools#4159 * Added more pre-commit hooks

#3) * Drop Python 3.7 support * Added more pre-commit hooks * Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

Pulumi automatically determines required plugins for a program. For Python programs, it essentially does this by running `python -m pip list --format json`, and any packages prefixed with `pulumi-` or `pulumi_` are assumed to have associated plugins, unless the package includes a `pulumi-plugin.json` file that has indicated there is no plugin via `{ "resource": false }`. When the `GetRequiredPlugins` support was originally added to the Python language host, it [hardcoded](https://github.com/pulumi/pulumi/blob/e6d20d26f7f64a624238f86204d862642ff27e16/sdk/python/cmd/pulumi-language-python/main.go#L428-L430) that `pulumi-policy` did not have an associated plugin. The hardcode mainly for older versions of `pulumi-policy` that did not contain a `pulumi-plugin.json` file. `pulumi-plugin.json` was actually originally named `pulumiplugin.json` (no dash). This file wasn't used anywhere, aside from in `pulumi-policy`, but the hardcod prevented it from ever being loaded in that case. It turned out that there was a bug parsing `pulumiplugin.json` that caused the CLI to error when that file existed in other packages. Since we were planning to expand the use of the file to other languages, and make it generated by default by SDKgen, we changed the name from `pulumiplugin.json` to `pulumi-plugin.json` to avoid breaking older CLIs with the bug (see pulumi/pulumi#8593). After making that change, we never followed up to rename the `pulumiplugin.json` file in `pulumi-policy` to `pulumi-plugin.json`, largely because it didn't matter since we had the hardcode. However, this hardcode no longer works with the latest version of `pulumi-policy` (v1.11.0). This version was built with a newer version of `setuptools` which has a behavior change where the package name in the metadata will now allow underscores, instead of having underscores replaced with dashes (pypa/setuptools#4159). This means that the package name reported from `pip list` is now `pulumi_policy` instead of `pulumi-policy`, which doesn't match the hardcoded list. And since there is no `pulumi-plugin.json` file in the package (it's still the old `pulumiplugin.json` name), the Pulumi Python language host thinks this package needs a plugin and tries to retrieve one that doesn't exist. This change addresses the issue by renaming `pulumiplugin.json` to `pulumi-plugin.json` in the package.

In the Python language host, we hardcode that `pulumi-policy` doesn't have an associated resource (provider) plugin, because we know it doesn't have one. However, this hardcode no longer works with the latest version of `pulumi-policy` (v1.11.0) because it was built with a newer version of `setuptools` which has a behavior change where the package name in the metadata will now allow underscores, instead of having underscores replaced with hyphens (pypa/setuptools#4159). This means that the package name reported from `pip list` is now `pulumi_policy` instead of `pulumi-policy`, which doesn't match the hardcoded list. Note that this change is really only to help with `pulumi-policy` v1.11.0. Future versions of `pulumi-policy` will have a `pulumi-plugin.json` file in the package, which properly indicates that it doesn't have an associated plugin.

In the Python language host, we hardcode that `pulumi-policy` doesn't have an associated resource (provider) plugin, because we know it doesn't have one. However, this hardcode no longer works with the latest version of `pulumi-policy` (v1.11.0) because it was built with a newer version of `setuptools` which has a behavior change where the package name in the metadata will now allow underscores, instead of having underscores replaced with hyphens (pypa/setuptools#4159). This means that the package name reported from `pip list` is now `pulumi_policy` instead of `pulumi-policy`, which doesn't match the hardcoded list. Note that this change is really only to help with `pulumi-policy` v1.11.0. Future versions of `pulumi-policy` will have a `pulumi-plugin.json` file in the package, which properly indicates that it doesn't have an associated plugin. Related: pulumi/pulumi-policy#358 Part of addressing: pulumi/pulumi-policy#356

Pulumi automatically determines required plugins for a program. For Python programs, it essentially does this by running `python -m pip list --format json`, and any packages prefixed with `pulumi-` or `pulumi_` are assumed to have associated plugins, unless the package includes a `pulumi-plugin.json` file that has indicated there is no plugin via `{ "resource": false }`. When the `GetRequiredPlugins` support was originally added to the Python language host, it [hardcoded](https://github.com/pulumi/pulumi/blob/e6d20d26f7f64a624238f86204d862642ff27e16/sdk/python/cmd/pulumi-language-python/main.go#L428-L430) that `pulumi-policy` did not have an associated plugin. The hardcode mainly for older versions of `pulumi-policy` that did not contain a `pulumi-plugin.json` file. `pulumi-plugin.json` was actually originally named `pulumiplugin.json` (no dash). This file wasn't used anywhere, aside from in `pulumi-policy`, but the hardcod prevented it from ever being loaded in that case. It turned out that there was a bug parsing `pulumiplugin.json` that caused the CLI to error when that file existed in other packages. Since we were planning to expand the use of the file to other languages, and make it generated by default by SDKgen, we changed the name from `pulumiplugin.json` to `pulumi-plugin.json` to avoid breaking older CLIs with the bug (see pulumi/pulumi#8593). After making that change, we never followed up to rename the `pulumiplugin.json` file in `pulumi-policy` to `pulumi-plugin.json`, largely because it didn't matter since we had the hardcode. However, this hardcode no longer works with the latest version of `pulumi-policy` (v1.11.0). This version was built with a newer version of `setuptools` which has a behavior change where the package name in the metadata will now allow underscores, instead of having underscores replaced with dashes (pypa/setuptools#4159). This means that the package name reported from `pip list` is now `pulumi_policy` instead of `pulumi-policy`, which doesn't match the hardcoded list. And since there is no `pulumi-plugin.json` file in the package (it's still the old `pulumiplugin.json` name), the Pulumi Python language host thinks this package needs a plugin and tries to retrieve one that doesn't exist. This change addresses the issue by renaming `pulumiplugin.json` to `pulumi-plugin.json` in the package.

Retain valid names with underscores in egg_info.

d0b0a4d

Closes #2522.

jaraco mentioned this pull request Dec 14, 2023

setuptools changes underscore to dash for package name #2522

Closed

jaraco merged commit 5a2eb0c into main Dec 14, 2023
38 of 42 checks passed

jaraco deleted the bugfix/2522-retain-valid-name branch December 14, 2023 15:46

abravalheri mentioned this pull request Jan 4, 2024

[BUG] Setuptools 69.0.3 setup.py develop generates a .egg-link file with underscores in the name component #4167

Open

raydouglass mentioned this pull request Jan 5, 2024

Update dask-cudf wheel name rapidsai/cudf#14713

Merged

This was referenced Jan 20, 2024

TestDeterminePulumiPackages/non-empty is failing in CI pulumi/pulumi#15192

Closed

Update python langhost test for pulumi-random change pulumi/pulumi#15193

Merged

mattip mentioned this pull request Feb 5, 2024

scipy-openblas32 and scipy-openblas64 have wrong permissions for their tokens scientific-python/upload-nightly-action#61

Closed

jaraco assigned jaraco and unassigned jaraco Feb 11, 2024

jaraco mentioned this pull request Feb 11, 2024

Building wheels with a - produces wheels with _. #4214

Closed

justinvp mentioned this pull request Mar 2, 2024

Receiving error GetLatestVersion is not supported for plugins from http sources when using the CloudInit Provider with Python pulumi/pulumi#15536

Closed

PatrickBaus added a commit to PatrickBaus/tinkerforge_async that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

a87de45

Updated pre-commit deps

PatrickBaus mentioned this pull request Jun 22, 2024

Fix module name PatrickBaus/tinkerforge_async#6

Merged

PatrickBaus added a commit to PatrickBaus/tinkerforge_async that referenced this pull request Jun 22, 2024

Fix module name

48cd856

- Updated pyproject.toml to fix the changes made in pypa/setuptools#4159 (#6) - Updated pre-commit deps

PatrickBaus added a commit to PatrickBaus/labnode_async that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

145f8b4

PatrickBaus mentioned this pull request Jun 22, 2024

Fix module name PatrickBaus/labnode_async#3

Merged

PatrickBaus added a commit to PatrickBaus/labnode_async that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

f70cc37

PatrickBaus added a commit to PatrickBaus/labnode_async that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159 (

c261df1

#3)

PatrickBaus added a commit to PatrickBaus/pyAsyncPrologixGpib that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

2fdca24

PatrickBaus added a commit to PatrickBaus/pyAsyncFluke5440B that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

9fa2d5f

PatrickBaus mentioned this pull request Jun 22, 2024

Fix package name PatrickBaus/pyAsyncFluke5440B#3

Merged

PatrickBaus added a commit to PatrickBaus/pyAsyncFluke5440B that referenced this pull request Jun 22, 2024

Fix package name (#3)

c2ddc54

* Updated pyproject.toml to fix the changes made in pypa/setuptools#4159 * Added more pre-commit hooks

PatrickBaus added a commit to PatrickBaus/pyAsyncHP3478A that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

3e8422b

PatrickBaus added a commit to PatrickBaus/pyUgGpib that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

659709b

PatrickBaus mentioned this pull request Jun 22, 2024

Fix module name haata/ugsimple-usb-gpib#8

Closed

PatrickBaus added a commit to PatrickBaus/pyAsyncHP3478A that referenced this pull request Jun 22, 2024

Updated pyproject.toml to fix the changes made in pypa/setuptools#4159

f25979e

corneliusroemer mentioned this pull request Jul 4, 2024

Why did the tarball filename change from nextstrain-cli-VERSION.tar.gz to nextstrain_cli-VERSION.tar.gz from 8.2.0 to 8.3.0 nextstrain/cli#378

Closed

justinvp mentioned this pull request Jul 20, 2024

Indicate the PyPi package doesn't have a plugin pulumi/pulumi-policy#358

Merged

justinvp mentioned this pull request Jul 20, 2024

Exclude pulumi_policy from plugin determination pulumi/pulumi#16727

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retain valid names with underscores in egg_info. #4159

Retain valid names with underscores in egg_info. #4159

jaraco commented Dec 14, 2023 •

edited

Loading

jaraco commented Dec 14, 2023

mattip commented Feb 6, 2024

jaraco commented Feb 11, 2024

Retain valid names with underscores in egg_info. #4159

Retain valid names with underscores in egg_info. #4159

Conversation

jaraco commented Dec 14, 2023 • edited Loading

Summary of changes

Pull Request Checklist

jaraco commented Dec 14, 2023

mattip commented Feb 6, 2024

jaraco commented Feb 11, 2024

jaraco commented Dec 14, 2023 •

edited

Loading