-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Return Union type for List.__add__
#3183
Conversation
My opinion: while this indeed describes the dynamic behavior more precisely, such occurrences in real code are often bugs, and I therefore prefer the current behavior that rejects it. For the The from typing import List, TypeVar, Union
S = TypeVar('S')
T = TypeVar('T')
def add_lists(a: List[S], b: List[T]) -> List[Union[S, T]]:
return [*a, *b]
This does work as you expect already - |
Please note last time this was attempted (see #2404) it caused severe regressions elsewhere and was ultimately reverted. |
@ilevkivskyi Sorry for not seeing this. Following #2404 and its linked issues, there aren't any good, easy to grok examples of the regressions. I considered the following before the PR. Given
Situation in master
After PR
So still, after this PR, there are a lot of confusing statements, but I don't see where I flip from 'OK' in master to 'FAIL' in PR. 🤷♂ I also don't see/have a workaround for 'a', 'c' or 'k' in master to get '+' (concat) to actually work. @bluetech Apple/bananas examples like |
Have you tried a: Sequence[object]
b1: List[Any]
b2: List[int]
a = b1 + b2 ? |
I actually have. And yes it flips. But then I could not make any sense of it, so I did not consider it. Probably nobody writes something like this, but mypy could infer such a situation. But then, it is not type-safe t.i. in master |
When we implement
from your testcase now pass. |
@ilevkivskyi @JukkaL As the ones handling the fallout from #2404 do you think we should try this approach? Is there a good test case for making sure this doesn't cause regressions? |
I don't think that we have good test cases for checking that this doesn't cause any regressions, other than type checking some big enough codebase with the change. I can try this against Dropbox internal codebases (not sure exactly when though). |
FWIW, I just ran this against a code-base with ~2000 files without error. |
That's promising! |
Has there been any progress on this ? I was hit by this bug today while testing a code that looked like : for current, previous in zip(l, [None] + l): ... I would then have expected |
With the newer mypy this "hacky" approach is probably outdated because we can now annotate/give a hint about self itself. |
Closing per last comment by @kaste. Thank you for the effort! |
Is there another PR? T.i. I couldn't get my idea to work actually. Should we just try harder? |
Since the issue is still there, it should probably be tracked somewhere |
Sorry, I think I misunderstood. We still need to try this against other large codebases. |
@JukkaL would you mind trying this PR on your codebase? |
Also stubtest is now complaining because |
I've been a bit distracted but managed finally to check this against our big internal codebase. It generated about 50 new errors, mostly of form |
It would be nice if you could add |
This has been open for a while (and now has a merge conflict). Do people think we should merge this? The errors @JukkaL found seem concerning. |
52d959f
to
e147bce
Compare
I fixed the merge conflict, although given what Jukka reports I wouldn't merge until the mypy kinks are worked out (hopefully we can remove the |
Diff from mypy_primer, showing the effect of this PR on open source code: + optuna/integration/__init__.py:32: error: Cannot infer type argument 1 of "__add__" of "list"
- ibis/client.py:468: error: Incompatible return value type (got "List[str]", expected "Set[Any]")
+ ibis/client.py:468: error: Incompatible return value type (got "List[Union[str, Any]]", expected "Set[Any]")
+ paasta_tools/config_utils.py:72: error: Unsupported operand types for + ("List[str]" and "List[str]")
- poetry/utils/setup_reader.py:174: error: Unsupported left operand type for + ("None")
+ poetry/utils/setup_reader.py:174: error: Unsupported operand types for + ("None" and "List[Any]")
+ mypy/fastparse.py:1071: error: Unsupported operand types for + ("List[expr]" and "List[expr]") [operator]
+ mypy/fastparse2.py:917: error: Unsupported operand types for + ("List[expr]" and "List[expr]") [operator]
+ mypy/test/testcmdline.py:59: error: Unsupported operand types for + ("List[str]" and "List[str]") [operator]
+ mypyc/test/test_commandline.py:51: error: Unsupported operand types for + ("List[str]" and "List[str]") [operator]
+ src/schemathesis/utils.py: note: In function "import_app":
+ src/schemathesis/utils.py:193: error: Argument 2 to "getattr" has incompatible type "Optional[str]"; expected "str"
+ pathod/language/websockets.py:245: error: List item 0 has incompatible type "Type[NestedFrame]"; expected "Type[_Component]"
+ src/graphql/type/validate.py:387: error: Unsupported operand types for + ("List[NamedTypeNode]" and "List[NamedTypeNode]")
+ src/graphql/validation/rules/overlapping_fields_can_be_merged.py:81: error: Unsupported operand types for + ("List[FieldNode]" and "List[FieldNode]")
+ src/pip/_internal/vcs/subversion.py:109: error: Result type of + incompatible in assignment
- pandas/io/common.py:452: error: Unsupported operand types for + ("List[Optional[str]]" and "List[str]") [operator]
+ pandas/io/stata.py:892: error: unused 'type: ignore' comment
+ pandas/io/pytables.py:3387: error: unused 'type: ignore' comment
+ tornado/autoreload.py:238: error: Unsupported operand types for + ("List[str]" and "List[str]")
+ pydantic/color.py:199: error: unused 'type: ignore' comment
- scipy/ndimage/tests/__init__.py:11: error: Unsupported operand types for + ("List[Type[integer]]" and "List[Type[floating]]") [operator]
|
This will run mypy_primer on mypy PRs. Changes on the open source corpus are reported as comments on the PR. We integrated this into typeshed CI; you can see examples here: python/typeshed#3183 python/typeshed#4734 This might be a little slow. On typeshed this runs in 10 minutes, but it's using a mypyc compiled wheel. It looks like it takes three minutes to compile a MYPYC_OPT_LEVEL=0 wheel in our CI currently (for a ~2x speedup), so that's probably worth it.
This will run mypy_primer on mypy PRs. Changes on the open source corpus are reported as comments on the PR. We integrated this into typeshed CI; you can see examples here: python/typeshed#3183 python/typeshed#4734 This might be a little slow. On typeshed this runs in 10 minutes, but it's using a mypyc compiled wheel. It looks like it takes three minutes to compile a MYPYC_OPT_LEVEL=0 wheel in our CI currently (for a ~2x speedup), so that's probably worth it. Co-authored-by: hauntsaninja <>
I am going to close this for now due to the upcoming reshuffle. I still think this is generally worthwhile, though, although it obviously needs more investigation. |
Is there any interest in reopening this? |
Reopening directly won't work because of the change in file structure. It still seems worth fixing but tricky to get right. |
In Python, lists of union types are just common. We want to support easy
construction of lists e.g.
We adjust
__iadd__
accordingly. Although returning a union here too looksstrange, it has the effect that
correctly fails.