Remove most of `setuptools._distutils` #9795

Avasam · 2023-02-22T00:32:51Z

Just like regular implementation details, private or vendored modules, we only keep what's used in our stubs. Because it's easier to reference and find the source if the structure is the same.
I could have imported from distutils to remove setuptools._distutils completely. But that's not a good idea for two reasons imo:

Eventually stdlib/distutils won't exist anymore and that'll cause issues.
May as well use the version setuptools vendors, with the correct types, that we can test with stubtest.

Closes #9520
Also resolves overuse of Any in the leftover setuptools._distutils classes for #9550

I added an extra_description which I believe can be very valuable to users of types-setuptools who may not be aware of distutils's situation. Feel free to wordsmith.
Preview:

stubs/setuptools/METADATA.toml

Co-authored-by: Alex Waygood <[email protected]>

Avasam · 2023-02-22T01:16:43Z

apprise is a types-babel issue. babel and mypyc are the same issue: They could be importing from either distutils or setuptools.
It used to be fine because types-setuptools classes used to (incorrectly) subclass distutils classes. But now it's correctly showing that the types have no direct relation.

mypyc is fixable, it's voluntarily importing setuptools for its monkey patching. I think it can just change the return type of get_extension https://github.com/python/mypy/blob/master/mypyc/build.py#L56

As for apprise, well, babel doesn't directly depend on setuptools, so we can't just update the type there. https://github.com/python-babel/babel/blob/master/babel/messages/frontend.py#L41
Unless you consider that having installed babel means you've had to have setuptools installed because of https://github.com/python-babel/babel/blob/master/setup.py#L4

AlexWaygood

LGTM (other than my one comment). Will leave open for a bit in case any other maintainers object, though.

AlexWaygood · 2023-02-22T16:08:45Z

stubs/setuptools/METADATA.toml

+extra_description = """\
+Having [setuptools](https://pypi.org/project/setuptools/) installed results in incorrect \
+[distutils](https://docs.python.org/3/distutils/introduction.html) typing. \
+`types-setuptools` makes no attempt at correcting this issue. As of \
+`setuptools>=60.0.0`, you [should not be using `distutils` directly](\
+https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html). \
+`distutils` is deprecated as of Python 3.10 and will be fully removed in Python 3.12. \
+(see [PEP 632](https://peps.python.org/pep-0632/))
+"""


I'm not sure that this is necessary -- it sort of feels like it's setuptools's problem rather than ours. If we do want some kind of a note here, I'd prefer it if it were slightly more concise.

Does this look better to you? Same links, and all the same info can be found at those locations.

Having setuptools installed results in incorrect distutils typing. types-setuptools makes no attempt at correcting this issue. See Porting from Distutils and PEP 632 – Deprecate distutils module.

Hmm, the more I think about it, the more I lean towards not including this note at all. It feels like unnecessary finger-pointing at setuptools from us. The distutils situation is very frustrating for me, a typeshed maintainer, but as far as I know we haven't received any complaints from users of our stubs about incorrect distutils typing, so it sort of feels like this note is attempting to solve something that may not actually be much of an issue for users. If we do get complaints or queries from users, we can maybe rethink this, but for now I think we can probably do without this :)

I can see what you mean. I also expect the amount of users who'll actually land on the PyPI page to be taught about the issue of mixing setuptools and distutils together to be really small. (pyright ships stubs, and the average mypy user probably installs stubs from mypy's CLI suggestion).
I'll omit the message for now. Can be revised later if needed or if other maintainers feel differently about it.

Avasam · 2023-02-22T18:50:37Z

Nice timing https://github.com/python/typeshed/actions/runs/4245809415/jobs/7381814113#step:5:185

Fixes #9799. Stubtest started failing on our `setuptools` stubs because of a default value added to our stdlib `distutils` stubs. The reason for this is because our `setuptools` stubs erroneously claim here that the `setuptools._distutils.core.Distribution` class is the same as the stdlib `distutils.dist.Distribution` class: https://github.com/python/typeshed/blob/06755e10ba0d39e7e4c18fcc2663d9da564a71ad/stubs/setuptools/setuptools/_distutils/core.pyi#L3 In actual fact, they're not, and they have different default values for this parameter. But reverting the addition of the default value is the simplest short-term fix for now. (An alternative fix would be to just merge #9795.)

AlexWaygood · 2023-02-25T18:51:24Z

mypy (https://github.com/python/mypy)
+ mypyc/build.py:64: error: Incompatible import of "Extension" (imported name has type "Type[setuptools.extension.Extension]", local name has type "Type[distutils.extension.Extension]")  [assignment]

I think you're correct that this one is easy to fix, but it is going to cause mypy's CI to suddenly and unexpectedly start failing once this PR is merged, since they don't pin types-setuptools in CI. Either you or I should probably go fix it as soon as this PR is merged :)

…p-_distutils

AlexWaygood

Two more nits (sorry!)

stubs/setuptools/setuptools/_distutils/command/register.pyi

stubs/setuptools/setuptools/_distutils/dist.pyi

Avasam · 2023-02-25T21:16:02Z

stubs/setuptools/setuptools/command/upload_docs.pyi

@@ -8,7 +10,8 @@ class upload_docs(upload):
    user_options: ClassVar[list[tuple[str, str | None, str]]]
    boolean_options: ClassVar[list[str]]
    def has_sphinx(self): ...
-    sub_commands: Any
+    # The callable parameter is self: Self, but using Self still trips up mypy
+    sub_commands: ClassVar[list[tuple[str, Callable[[Self], bool] | None]]]  # type: ignore[misc, assignment]


I didn't need to modify this file (leaving it as Incomplete or Any would be enough for this PR), but I wanted to show what I think may be a mypy issue with Self in Callable. I spotted. pyright looks correct to me.

Example with sub_commands removed and inferred from superclass:

Interesting. I think using Self in ClassVars might be underspecified/I'm not sure exactly what the semantics should be really. But this looks fine for now imo.

github-actions · 2023-02-26T10:48:39Z

Diff from mypy_primer, showing the effect of this PR on open source code:

mypy (https://github.com/python/mypy)
+ mypyc/build.py:64: error: Incompatible import of "Extension" (imported name has type "Type[setuptools.extension.Extension]", local name has type "Type[distutils.extension.Extension]")  [assignment]

apprise (https://github.com/caronc/apprise)
+ setup.py:74: error: Argument "cmdclass" to "setup" has incompatible type "Dict[str, Type[babel.messages.frontend.Command]]"; expected "Mapping[str, Type[setuptools._distutils.cmd.Command]]"  [arg-type]

AlexWaygood

Thank you!

A simple type-only change since mypyc explicitely uses setuptools' monkeypatching. And `setup.py` imports from setuptools. Tests should stay the same. This should fix a sudden failure of tests once python/typeshed#9795 is merged.

Following python/typeshed#9795, stubtest now passes for typeshed's `setuptools` stubs even if the correct default parameter is added to the stdlib stub for `distutils.dist.Distribution.announce`

A simple type-only change since mypyc explicitely uses setuptools' monkeypatching. And `setup.py` imports from setuptools. Tests should stay the same. This should fix a sudden failure of tests once python/typeshed#9795 is merged.

Linking python#10255 There are cases of these being used, including in mypy This is reverts a small part of python#9795

Avasam added 3 commits February 21, 2023 18:33

Remove most _distutils

b663057

Update refs

f95f804

Fix Liskov substitution principle violation

ba7bbb0

AlexWaygood reviewed Feb 22, 2023

View reviewed changes

stubs/setuptools/METADATA.toml Outdated Show resolved Hide resolved

Update stubs/setuptools/METADATA.toml

4be1d84

Co-authored-by: Alex Waygood <[email protected]>

This comment has been minimized.

Sign in to view

Merge branch 'main' into cleanup-_distutils

533acd4

This comment has been minimized.

Sign in to view

AlexWaygood approved these changes Feb 22, 2023

View reviewed changes

AlexWaygood mentioned this pull request Feb 23, 2023

Daily tests failed on Thu Feb 23 2023 #9799

Closed

AlexWaygood mentioned this pull request Feb 23, 2023

Remove a default value in distutils #9800

Merged

Avasam added 2 commits February 25, 2023 14:48

Merge branch 'main' of https://github.com/python/typeshed into cleanu…

abe4a12

…p-_distutils

Remove extra_description

402833a

This comment has been minimized.

Sign in to view

AlexWaygood reviewed Feb 25, 2023

View reviewed changes

stubs/setuptools/setuptools/_distutils/command/register.pyi Outdated Show resolved Hide resolved

stubs/setuptools/setuptools/_distutils/dist.pyi Outdated Show resolved Hide resolved

Avasam mentioned this pull request Feb 25, 2023

Support next types-setuptools update python/mypy#14781

Merged

sub_commands is a ClassVar and fix overloads

eea4b4e

Avasam commented Feb 25, 2023

View reviewed changes

This comment has been minimized.

Sign in to view

Avasam requested a review from AlexWaygood February 25, 2023 21:30

Merge branch 'main' into cleanup-_distutils

06b57f2

AlexWaygood approved these changes Feb 26, 2023

View reviewed changes

AlexWaygood merged commit fee5f1b into python:main Feb 26, 2023

AlexWaygood mentioned this pull request Feb 26, 2023

Remove distutils.dist.Distribution.announce from the blacklist JelleZijlstra/stubdefaulter#63

Merged

FFY00 mentioned this pull request Mar 4, 2023

Should we delete the _distutils directory from our setuptools stubs ? #9520

Closed

maresb mentioned this pull request Apr 17, 2023

Add missing exception classes in setuptools.errors #10057

Merged

AlexWaygood mentioned this pull request Jun 4, 2023

distutils and setuptools._distutils stubs are different #10255

Closed

hauntsaninja added a commit to hauntsaninja/typeshed that referenced this pull request Jul 3, 2023

Bring back a few setuptools._distutils files

1ece6e6

Linking python#10255 There are cases of these being used, including in mypy This is reverts a small part of python#9795

hauntsaninja mentioned this pull request Jul 3, 2023

Bring back a few setuptools._distutils files #10401

Merged

Avasam deleted the cleanup-_distutils branch October 20, 2023 21:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove most of `setuptools._distutils` #9795

Remove most of `setuptools._distutils` #9795

Avasam commented Feb 22, 2023 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

Avasam commented Feb 22, 2023 •

edited

Loading

AlexWaygood left a comment

AlexWaygood Feb 22, 2023

Avasam Feb 25, 2023 •

edited

Loading

AlexWaygood Feb 25, 2023

Avasam Feb 25, 2023 •

edited

Loading

Avasam commented Feb 22, 2023 •

edited

Loading

AlexWaygood commented Feb 25, 2023

This comment has been minimized.

AlexWaygood left a comment

Avasam Feb 25, 2023 •

edited

Loading

AlexWaygood Feb 26, 2023

This comment has been minimized.

github-actions bot commented Feb 26, 2023

AlexWaygood left a comment

Remove most of setuptools._distutils #9795

Remove most of setuptools._distutils #9795

Conversation

Avasam commented Feb 22, 2023 • edited Loading

This comment has been minimized.

This comment has been minimized.

Avasam commented Feb 22, 2023 • edited Loading

AlexWaygood left a comment

Choose a reason for hiding this comment

AlexWaygood Feb 22, 2023

Choose a reason for hiding this comment

Avasam Feb 25, 2023 • edited Loading

Choose a reason for hiding this comment

AlexWaygood Feb 25, 2023

Choose a reason for hiding this comment

Avasam Feb 25, 2023 • edited Loading

Choose a reason for hiding this comment

Avasam commented Feb 22, 2023 • edited Loading

AlexWaygood commented Feb 25, 2023

This comment has been minimized.

AlexWaygood left a comment

Choose a reason for hiding this comment

Avasam Feb 25, 2023 • edited Loading

Choose a reason for hiding this comment

AlexWaygood Feb 26, 2023

Choose a reason for hiding this comment

This comment has been minimized.

github-actions bot commented Feb 26, 2023

AlexWaygood left a comment

Choose a reason for hiding this comment

Remove most of `setuptools._distutils` #9795

Remove most of `setuptools._distutils` #9795

Avasam commented Feb 22, 2023 •

edited

Loading

Avasam commented Feb 22, 2023 •

edited

Loading

Avasam Feb 25, 2023 •

edited

Loading

Avasam Feb 25, 2023 •

edited

Loading

Avasam commented Feb 22, 2023 •

edited

Loading

Avasam Feb 25, 2023 •

edited

Loading