Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{Core} Make parser compatible with Python 3.11's argparse #24497

Merged
merged 1 commit into from
Dec 15, 2022

Conversation

jiasli
Copy link
Member

@jiasli jiasli commented Oct 31, 2022

Description

Fix #23015
A rework of #24109

Azure CLI fails on Python 3.11:

> az version
...
argparse.ArgumentError: argument _command_package: conflicting subparser: version

Python 3.11 updated argparse's behavior:

https://docs.python.org/3.11/whatsnew/changelog.html#id50

bpo-39716: Raise an ArgumentError when the same subparser name is added twice to an argparse.ArgumentParser. This is consistent with the (default) behavior when the same option string is added twice to an ArgumentParser.

The PR for this change is python/cpython#18605

Per Git blame,

  • subparser.choices[command_verb] = command_verb was renamed from subparser.choices[command_name] = command_name by 517872a
  • subparser.choices[command_name] = command_name was introduced as early as 2a6f545

Running this line makes subparser.choices a str:str mapping.

In argparse._SubParsersAction, self.choices is the same dict object as self._name_parser_map:

https://github.com/python/cpython/blob/v3.10.8/Lib/argparse.py#L1159

class _SubParsersAction(Action):
    ...
    def __init__(self,
        ....
        super(_SubParsersAction, self).__init__(
            ...
            choices=self._name_parser_map,

https://github.com/python/cpython/blob/v3.10.8/Lib/argparse.py#L769

class Action(_AttributeHolder):
    ...
    def __init__(self,
        ...
        self.choices = choices

Later, in argparse._SubParsersAction.add_parser, self._name_parser_map's command_verb key is overwritten as a str:parser mapping:

command_parser = subparser.add_parser(command_verb,

https://github.com/python/cpython/blob/v3.10.8/Lib/argparse.py#L1179

    def add_parser(self, name, **kwargs):
        ...
        self._name_parser_map[name] = parser

Before running this line:

image

After running this line:

image

So subparser.choices[command_verb] = command_verb is basically a no-op, but it starts to cause failure in Python 3.11 due to the new duplication check.

I am not sure why subparser.choices[command_verb] = command_verb was introduced by 2a6f545 6 years ago, possibly due to some Python 2 argparse's limitation?

I checked http://bugs.python.org/issue9253 mentioned in the comment, but didn't see how it is related to this workaround.

We can think it in another way. Since Python 3.11's argparse has no functionality change (only an additional check), if this line should be removed for Python 3.11, it should also be removed for Python 3.10-.

@ghost ghost added Auto-Assign Auto assign by bot Core CLI core infrastructure labels Oct 31, 2022
@ghost ghost requested a review from yonzhan October 31, 2022 05:55
@ghost ghost assigned jiasli Oct 31, 2022
@ghost ghost added this to the Nov 2022 (2022-12-06) milestone Oct 31, 2022
@yonzhan
Copy link
Collaborator

yonzhan commented Oct 31, 2022

Core

@jiasli jiasli changed the title {Core} {Core} Make parser compatible with Python 3.11 Oct 31, 2022
@jiasli jiasli mentioned this pull request Oct 31, 2022
4 tasks
@jiasli jiasli changed the title {Core} Make parser compatible with Python 3.11 {Core} Make parser compatible with Python 3.11's argparse Nov 1, 2022
@jiasli jiasli marked this pull request as ready for review December 15, 2022 03:28
@jiasli jiasli merged commit 37053a3 into Azure:dev Dec 15, 2022
@jiasli jiasli deleted the python311-argparse branch December 15, 2022 03:52
avgale pushed a commit to avgale/azure-cli that referenced this pull request Aug 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot Core CLI core infrastructure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

azure-cli-core: Test failures with conflicting subparsers in Python 3.11.0b3
4 participants