Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade mypy #12293

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ repos:
- id: ruff

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v0.961
rev: v1.5.1
hooks:
- id: mypy
exclude: tests/data
Expand Down
Empty file.
1 change: 1 addition & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ allow_subclassing_any = True
allow_untyped_calls = True
warn_return_any = False
ignore_missing_imports = True
explicit_package_bases = True

[mypy-pip._internal.utils._jaraco_text]
ignore_errors = True
Expand Down
3 changes: 1 addition & 2 deletions src/pip/_internal/locations/_distutils.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,7 @@ def distutils_scheme(
try:
d.parse_config_files()
except UnicodeDecodeError:
# Typeshed does not include find_config_files() for some reason.
paths = d.find_config_files() # type: ignore
paths = d.find_config_files()
logger.warning(
"Ignore distutils configs in %s due to encoding errors.",
", ".join(os.path.basename(p) for p in paths),
Expand Down
5 changes: 3 additions & 2 deletions src/pip/_internal/metadata/_json.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,11 @@ def sanitise_header(h: Union[Header, str]) -> str:
key = json_name(field)
if multi:
value: Union[str, List[str]] = [
sanitise_header(v) for v in msg.get_all(field)
sanitise_header(v)
for v in msg.get_all(field) # type: ignore[union-attr]
]
else:
value = sanitise_header(msg.get(field))
value = sanitise_header(msg.get(field)) # type: ignore[arg-type]
Comment on lines -70 to +71
Copy link
Member

@uranusjr uranusjr Sep 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What changed here? This seems to be ignored in a lot of places.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing something now thinks get can return None, because it can't infer from the earlier if field not in msg that the field is guaranteed to be present.

I think this sort of ignore needs a more extensive comment, as it feels very much like a workaround for a limitation in mypy's ability to infer types correctly, or a lack of expressiveness in the type system to annotate get in such a way that this sort of LBYL idiom is supported.

We could of course add an assert stating that the return from get is not None, but I think that sort of assert is counterproductive, and obscures the actual logic here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, I mentioned this in the body of the PR: python/typeshed#9620
I can add a comment, should I do so in all affected places or only here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't honestly know. This is the sort of "typing is frustrating" issue I've been going on about on Discourse. I dislike #type: ignore because it normalises switching off typing without documenting why. After all, someone could remove the check and then we're ignoring a genuine error.

I guess we should add a comment like "field is guaranteed to be present (see check on line XXX)". Unfortunately, the #type: ignore syntax doesn't appear to allow explanatory trailing text, which would be preferable.

At the end of the day, I'm just expressing my frustration. I don't really mind how we address this, it's not like we're worse off than we would be without type annotations at all.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this particular one I’d just add an explanation comment here. Other occurrences are in tests and those are less problematic especially a reader can more easily find the comment by tracing the test code.

if key == "keywords":
# Accept both comma-separated and space-separated
# forms, for better compatibility with old data.
Expand Down
4 changes: 3 additions & 1 deletion src/pip/_internal/network/xmlrpc.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@ def request(
self,
host: "_HostType",
handler: str,
request_body: bytes,
# Should ideally be typed with SizedBuffer,
# but that's not easily usable till Python 3.11
request_body: bytes, # type: ignore[override]
verbose: bool = False,
) -> Tuple["_Marshallable", ...]:
assert isinstance(host, str)
Expand Down
2 changes: 1 addition & 1 deletion src/pip/_internal/utils/misc.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
VersionInfo = Tuple[int, int, int]
NetlocTuple = Tuple[str, Tuple[Optional[str], Optional[str]]]
OnExc = Callable[[FunctionType, Path, BaseException], Any]
OnErr = Callable[[FunctionType, Path, ExcInfo], Any]
OnErr = Callable[[Callable[..., Any], str, ExcInfo], Any]


def get_pip_version() -> str:
Expand Down
2 changes: 1 addition & 1 deletion tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -1031,7 +1031,7 @@ def html_index_with_onetime_server(
class InDirectoryServer(http.server.ThreadingHTTPServer):
def finish_request(self, request: Any, client_address: Any) -> None:
self.RequestHandlerClass(
request, client_address, self, directory=str(html_index_for_packages) # type: ignore[call-arg] # noqa: E501
request, client_address, self, directory=str(html_index_for_packages) # type: ignore[call-arg, arg-type] # noqa: E501
)

class Handler(OneTimeDownloadHandler):
Expand Down
5 changes: 3 additions & 2 deletions tests/lib/configuration_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,9 @@ def overridden() -> None:
)
old()

# https://github.com/python/mypy/issues/2427
self.configuration._load_config_files = overridden # type: ignore[assignment]
self.configuration._load_config_files = ( # type: ignore[method-assign]
overridden
)
Comment on lines +40 to +42
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we just global-ignore method-assign in all files in tests.


@contextlib.contextmanager
def tmpfile(self, contents: str) -> Iterator[str]:
Expand Down
13 changes: 8 additions & 5 deletions tests/lib/test_wheel.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@

def test_message_from_dict_one_value() -> None:
message = message_from_dict({"a": "1"})
assert set(message.get_all("a")) == {"1"}
assert set(message.get_all("a")) == {"1"} # type: ignore[arg-type]


def test_message_from_dict_multiple_values() -> None:
message = message_from_dict({"a": ["1", "2"]})
assert set(message.get_all("a")) == {"1", "2"}
assert set(message.get_all("a")) == {"1", "2"} # type: ignore[arg-type]


def message_from_bytes(contents: bytes) -> Message:
Expand Down Expand Up @@ -67,7 +67,7 @@ def test_make_metadata_file_custom_value_list() -> None:
f = default_make_metadata(updates={"a": ["1", "2"]})
assert f is not None
message = default_metadata_checks(f)
assert set(message.get_all("a")) == {"1", "2"}
assert set(message.get_all("a")) == {"1", "2"} # type: ignore[arg-type]


def test_make_metadata_file_custom_value_overrides() -> None:
Expand Down Expand Up @@ -101,7 +101,10 @@ def default_wheel_metadata_checks(f: File) -> Message:
assert message.get_all("Wheel-Version") == ["1.0"]
assert message.get_all("Generator") == ["pip-test-suite"]
assert message.get_all("Root-Is-Purelib") == ["true"]
assert set(message.get_all("Tag")) == {"py2-none-any", "py3-none-any"}
assert set(message.get_all("Tag")) == { # type: ignore[arg-type]
"py2-none-any",
"py3-none-any",
}
return message


Expand All @@ -122,7 +125,7 @@ def test_make_wheel_metadata_file_custom_value_list() -> None:
f = default_make_wheel_metadata(updates={"a": ["1", "2"]})
assert f is not None
message = default_wheel_metadata_checks(f)
assert set(message.get_all("a")) == {"1", "2"}
assert set(message.get_all("a")) == {"1", "2"} # type: ignore[arg-type]


def test_make_wheel_metadata_file_custom_value_override() -> None:
Expand Down
6 changes: 3 additions & 3 deletions tests/unit/metadata/test_metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def test_dist_get_direct_url_no_metadata(mock_read_text: mock.Mock) -> None:
class FakeDistribution(BaseDistribution):
pass

dist = FakeDistribution()
dist = FakeDistribution() # type: ignore[abstract]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should make FakeDistribution not abstract instead? We can just raise in all methods that are not supposed to be called.

assert dist.direct_url is None
mock_read_text.assert_called_once_with(DIRECT_URL_METADATA_NAME)

Expand All @@ -35,7 +35,7 @@ def test_dist_get_direct_url_invalid_json(
class FakeDistribution(BaseDistribution):
canonical_name = cast(NormalizedName, "whatever") # Needed for error logging.

dist = FakeDistribution()
dist = FakeDistribution() # type: ignore[abstract]
with caplog.at_level(logging.WARNING):
assert dist.direct_url is None

Expand Down Expand Up @@ -84,7 +84,7 @@ def test_dist_get_direct_url_valid_metadata(mock_read_text: mock.Mock) -> None:
class FakeDistribution(BaseDistribution):
pass

dist = FakeDistribution()
dist = FakeDistribution() # type: ignore[abstract]
direct_url = dist.direct_url
assert direct_url is not None
mock_read_text.assert_called_once_with(DIRECT_URL_METADATA_NAME)
Expand Down
9 changes: 4 additions & 5 deletions tests/unit/test_base_command.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# mypy: disable-error-code="method-assign"
import logging
import os
import time
Expand Down Expand Up @@ -150,8 +151,7 @@ def assert_helpers_set(options: Values, args: List[str]) -> int:
return SUCCESS

c = Command("fake", "fake")
# https://github.com/python/mypy/issues/2427
c.run = Mock(side_effect=assert_helpers_set) # type: ignore[assignment]
c.run = Mock(side_effect=assert_helpers_set)
assert c.main(["fake"]) == SUCCESS
c.run.assert_called_once()

Expand All @@ -175,8 +175,7 @@ def create_temp_dirs(options: Values, args: List[str]) -> int:
return SUCCESS

c = Command("fake", "fake")
# https://github.com/python/mypy/issues/2427
c.run = Mock(side_effect=create_temp_dirs) # type: ignore[assignment]
c.run = Mock(side_effect=create_temp_dirs)
assert c.main(["fake"]) == SUCCESS
c.run.assert_called_once()
assert os.path.exists(Holder.value) == exists
Expand All @@ -200,6 +199,6 @@ def create_temp_dirs(options: Values, args: List[str]) -> int:

c = Command("fake", "fake")
# https://github.com/python/mypy/issues/2427
c.run = Mock(side_effect=create_temp_dirs) # type: ignore[assignment]
c.run = Mock(side_effect=create_temp_dirs)
assert c.main(["fake"]) == SUCCESS
c.run.assert_called_once()
10 changes: 4 additions & 6 deletions tests/unit/test_configuration.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Tests for all things related to the configuration
"""

# mypy: disable-error-code="method-assign"
import re
from unittest.mock import MagicMock

Expand Down Expand Up @@ -214,8 +214,7 @@ def test_site_modification(self) -> None:

# Mock out the method
mymock = MagicMock(spec=self.configuration._mark_as_modified)
# https://github.com/python/mypy/issues/2427
self.configuration._mark_as_modified = mymock # type: ignore[assignment]
self.configuration._mark_as_modified = mymock

self.configuration.set_value("test.hello", "10")

Expand All @@ -231,7 +230,7 @@ def test_user_modification(self) -> None:
# Mock out the method
mymock = MagicMock(spec=self.configuration._mark_as_modified)
# https://github.com/python/mypy/issues/2427
self.configuration._mark_as_modified = mymock # type: ignore[assignment]
self.configuration._mark_as_modified = mymock

self.configuration.set_value("test.hello", "10")

Expand All @@ -249,8 +248,7 @@ def test_global_modification(self) -> None:

# Mock out the method
mymock = MagicMock(spec=self.configuration._mark_as_modified)
# https://github.com/python/mypy/issues/2427
self.configuration._mark_as_modified = mymock # type: ignore[assignment]
self.configuration._mark_as_modified = mymock

self.configuration.set_value("test.hello", "10")

Expand Down
2 changes: 1 addition & 1 deletion tests/unit/test_resolution_legacy_resolver.py
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,7 @@ class NotWorkingFakeDist(FakeDist):
def metadata(self) -> email.message.Message:
raise FileNotFoundError(metadata_name)

dist = make_fake_dist(klass=NotWorkingFakeDist)
dist = make_fake_dist(klass=NotWorkingFakeDist) # type: ignore[type-abstract]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, why does this warn about the abstract class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the above, NotWorkingFakeDist doesn't implement BaseDistribution. So make_fake_dist gives you an object that doesn't do what it says it should

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding, is there an actual runtime problem here, or are we simply not able to express what we're doing in terms that the type system can understand?

It feels like we shouldn't be trying to type check mock objects like this as if they are full-fledged versions. After all, that's the whole point of mock objects, surely? What do we hope to gain from the type annotations in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mypy is correct to complain here. If you do dist.from_directory or many of the other methods defined by the BaseDistribution protocol, you'll see that dist does not implement the Protocol it claims to implement. You could use a narrower Protocol to type _check_dist_requires_python, but I think you'd find that not worth it.

So instead we just say "get out of my way, I know I'm doing a thing that could be unsafe, but I don't care because it's a test". # type: ignore is as good a way to say this as any other, such as casting to Any. In general, type checking tests is lower value than type checking other code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

… Continuing from the comment above, it should be OK to add a comment here (and one for FakeDistribution) to say we’re kind ot intentionally not fleshing out the implementation since an unimplemented abstract method would correctly raise and make a test fail if it’s accidentally called.


with pytest.raises(NoneMetadataError) as exc:
_check_dist_requires_python(
Expand Down