Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make inspect.signature expression evaluation more powerful #68155

Open
larryhastings opened this issue Apr 15, 2015 · 12 comments
Open

Make inspect.signature expression evaluation more powerful #68155

larryhastings opened this issue Apr 15, 2015 · 12 comments
Assignees
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@larryhastings
Copy link
Contributor

BPO 23967
Nosy @ncoghlan, @larryhastings, @zware, @serhiy-storchaka, @1st1, @pdmccormick
Files
  • larry.improved.signature.expressions.1.txt
  • pdm-argument_clinic-mixed_py_and_c_defaults-v1.patch: Argument Clinic patch simplifying the use of the improved signatures
  • larry.improved.signature.expressions.2.txt
  • larry.improved.signature.expressions.3.txt
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/larryhastings'
    closed_at = None
    created_at = <Date 2015-04-15.18:52:37.516>
    labels = ['type-feature', 'library']
    title = 'Make inspect.signature expression evaluation more powerful'
    updated_at = <Date 2020-05-29.17:47:00.140>
    user = 'https://github.com/larryhastings'

    bugs.python.org fields:

    activity = <Date 2020-05-29.17:47:00.140>
    actor = 'brett.cannon'
    assignee = 'larry'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2015-04-15.18:52:37.516>
    creator = 'larry'
    dependencies = []
    files = ['39047', '39066', '39123', '39181']
    hgrepos = []
    issue_num = 23967
    keywords = ['patch']
    message_count = 11.0
    messages = ['241140', '241204', '241205', '241478', '241533', '241534', '241850', '241853', '241855', '242006', '365315']
    nosy_count = 7.0
    nosy_names = ['ncoghlan', 'larry', 'zach.ware', 'serhiy.storchaka', 'yselivanov', 'pdmccormick', 'Eric Wieser']
    pr_nums = []
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue23967'
    versions = ['Python 3.5']

    @larryhastings
    Copy link
    Contributor Author

    Peter's working on converting socket to use Argument Clinic. He had a default that really should look like this:

        min(SOME_SOCKET_MODULE_CONSTANT, 128)

    "min" wasn't something we'd needed before. I thought about it and realized we could do a much better job of simulating the evaluation context of a shared module.

    Initially I thought, all I needed was to bolster the environment we used for eval() to add the builtins. (Which I've done.) But this wasn't sufficient because we deliberately used ast.literal_eval(), which doesn't support function calls by design for superior security. Or subscripting, or attribute access. We already worked around those I think.

    But how concerned are we about security? What is the attack vector here? If the user is able to construct an object that has a villainous __text_signature__ on it... surely they could already do as they like?

    So here's a first draft at modifying the __text_signature__ evaluation environment so it can handle much more sophisticated expressions. It can use anything from builtins, or anything in sys.modules, or anything in the current module; it can call functions, and subscript, and access attributes, and everything.

    To make this work I had to write an ast printer that produces evaluatable Python code. Note that it's not complete, I know it's not complete, it's missing loads of operators. Assume that if this is a good idea I will add all the missing operators.

    Nick was worried that *in the future* we might expose a "turn this string into a signature" function. That might make an easier attack vector. So he asked that the "trusted=" keyword flag be added, and the full-on eval only happen if the string is trusted.

    @larryhastings larryhastings self-assigned this Apr 15, 2015
    @larryhastings larryhastings added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Apr 15, 2015
    @pdmccormick
    Copy link
    Mannequin

    pdmccormick mannequin commented Apr 16, 2015

    This definitely works for the _socket.listen use case!

    In terms of generating such a signature using Argument Clinic, currently this is required:

    backlog: int(py_default="builtins.min(SOMAXCONN, 128)", c_default="Py_MIN(SOMAXCONN, 128)") = 000
    

    The attached patch lets Tools/clinic/clinic.py make an exception when both C and Python defaults are specified, simplifying the above to:

    backlog: int(py_default="builtins.min(SOMAXCONN, 128)", c_default="Py_MIN(SOMAXCONN, 128)")
    

    @pdmccormick
    Copy link
    Mannequin

    pdmccormick mannequin commented Apr 16, 2015

    I missed the fact that Larry's patch obviates the need for the builtins. prefix, shortening the Argument Clinic parameter specification into:

    backlog: int(py_default="min(SOMAXCONN, 128)", c_default="Py_MIN(SOMAXCONN, 128)")
    

    @larryhastings
    Copy link
    Contributor Author

    I should mention that evalify_node() is pretty hacked up here, and is not ready to be checked in. (I'm proposing separately that we simply add something like this directly into the standard library, see issue bpo-24002.)

    @larryhastings
    Copy link
    Contributor Author

    Thanks to bpo-24002 I now know how to write evalify_node properly. This patch is now much better.

    Note that I deliberately made the new function _eval_ast_expr() as a "private" module-level routine. I need that same functionality in Argument Clinic too, so if both patches are accepted I'll have Clinic switch to calling this version.

    @larryhastings
    Copy link
    Contributor Author

    Whoops. Here's the revised patch.

    @larryhastings
    Copy link
    Contributor Author

    Cleaned up the patch some more--the code was stupid in a couple places. I think it's ready to go in.

    @serhiy-storchaka
    Copy link
    Member

    Using complex expressions is deceitful. In Python functions the default value is evaluated only once, at function creation time, but inspect.signature will evaluate it every time. For example foo(x={}) and foo(x=dict()) means the same in function declaration, but different in signature.

    It could also affect security, because allow arbitrary code execution at the place where it was not allowed before.

    I think this issue should be discussed on Python-Dev. I'm not sure that it is pythonic.

    @larryhastings
    Copy link
    Contributor Author

    It's only used for signatures in builtins. Any possible security hole here is uninteresting because the evil hacker already got to run arbitrary C code in the module init.

    Because it's only used for signatures in builtins, we shouldn't encounter a function with a mutable default value like {} or [] which gets mutated later. Builtins don't have those.

    In case you're wondering about the "trusted" parameter, that was suggested by Nick Coghlan at the PyCon sprints. He's thinking that other callers may use _signature_fromstr() in the future, and he wanted the API to make it clear that future uses may be on non-trustworthy sources.

    And, finally, consider that the original version already calls eval(). Admittedly it uses eval() in a way that should be much harder to exploit. But it's not an enormous difference between the two calls.

    I don't really think we need to post to python-dev about this.

    @ncoghlan
    Copy link
    Contributor

    Right, Larry and I had a fairly long discussion about this idea at the sprints, and I was satisfied that all the cases where he's proposing to use this are safe: in order to exploit them you need to be able to set __text_signature__ on arbitrary objects, and if an attacker can do that, you've already lost control of the process.

    However, a natural future extension is to expose this as a public alternative constructor for Signature objects, and for that, the fact that it ultimately calls eval() under the hood presents more of a security risk. The "trusted=False" default on _signature_fromstr allows the function to be used safely on untrusted data, while allowing additional flexibility when you *do* trust the data you're evaluating.

    @EricWieser
    Copy link
    Mannequin

    EricWieser mannequin commented Mar 30, 2020

    To make this work I had to write an ast printer that produces evaluatable Python code. Note that it's not complete, I know it's not complete, it's missing loads of operators. Assume that if this is a good idea I will add all the missing operators.

    Now that ast.unparse is in (bpo-38870), can this patch be simplified?

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    hauntsaninja added a commit to hauntsaninja/cpython that referenced this issue Oct 28, 2022
    …handling
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing python#83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in python#85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in python#85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in python#68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while python#85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`.
    JelleZijlstra pushed a commit that referenced this issue Dec 21, 2022
    …ng (#98796)
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing #83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in #85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in #85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in #68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while #85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`.
    hauntsaninja added a commit to hauntsaninja/cpython that referenced this issue Dec 21, 2022
    …ture__ handling (pythonGH-98796)
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing pythonGH-83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in pythonGH-85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in pythonGH-85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in pythonGH-68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while pythonGH-85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`..
    (cherry picked from commit 79311cb)
    
    Co-authored-by: Shantanu <[email protected]>
    hauntsaninja added a commit to hauntsaninja/cpython that referenced this issue Dec 21, 2022
    …ture__ handling (pythonGH-98796)
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing pythonGH-83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in pythonGH-85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in pythonGH-85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in pythonGH-68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while pythonGH-85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`..
    (cherry picked from commit 79311cb)
    
    Co-authored-by: Shantanu <[email protected]>
    JelleZijlstra pushed a commit that referenced this issue Dec 21, 2022
    … handling (GH-98796) (#100392)
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing GH-83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in GH-85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in GH-85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in GH-68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while GH-85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`..
    (cherry picked from commit 79311cb)
    
    Co-authored-by: Shantanu <[email protected]>
    JelleZijlstra pushed a commit that referenced this issue Dec 21, 2022
    … handling (GH-98796) (#100393)
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing GH-83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in GH-85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in GH-85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in GH-68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while GH-85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`..
    (cherry picked from commit 79311cb)
    
    Co-authored-by: Shantanu <[email protected]>
    iritkatriel added a commit to iritkatriel/cpython that referenced this issue Dec 28, 2022
    * Correct CVE-2020-10735 documentation (python#100306)
    
    * pythongh-94912: Added marker for non-standard coroutine function detection (python#99247)
    
    This introduces a new decorator `@inspect.markcoroutinefunction`,
    which, applied to a sync function, makes it appear async to
    `inspect.iscoroutinefunction()`.
    
    * Docs: Don't upload CI artifacts (python#100330)
    
    * pythongh-89727: Fix os.walk RecursionError on deep trees (python#99803)
    
    Use a stack to implement os.walk iteratively instead of recursively to
    avoid hitting recursion limits on deeply nested trees.
    
    * pythongh-69929: re docs: Add more specific definition of \w (python#92015)
    
    Co-authored-by: Jelle Zijlstra <[email protected]>
    
    * pythongh-89051: Add ssl.OP_LEGACY_SERVER_CONNECT (python#93927)
    
    Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com>
    Co-authored-by: Christian Heimes <[email protected]>
    Co-authored-by: Hugo van Kemenade <[email protected]>
    Fixes python#89051
    
    * pythongh-88211: Change lower-case and upper-case to match recommendations in imaplib docs (python#99625)
    
    * pythongh-100348: Fix ref cycle in `asyncio._SelectorSocketTransport` with `_read_ready_cb` (python#100349)
    
    * pythongh-99925: Fix inconsistency in `json.dumps()` error messages (pythonGH-99926)
    
    * Clarify that every thread has its own default context in contextvars (python#99246)
    
    * pythongh-99576: Fix cookiejar file that was not truncated for some classes (pythonGH-99616)
    
    Co-authored-by: Łukasz Langa <[email protected]>
    
    * pythongh-100188: Reduce misses in BINARY_SUBSCR_(LIST/TUPLE)_INT (python#100189)
    
    Don't specialize if the index is negative.
    
    * pythongh-99991: improve docs on str.encode and bytes.decode (python#100198)
    
    Co-authored-by: C.A.M. Gerlach <[email protected]>
    
    * pythongh-91081: Add note on WeakKeyDictionary behavior when deleting a replaced entry (python#91499)
    
    Co-authored-by: Pieter Eendebak <[email protected]>
    Co-authored-by: Jelle Zijlstra <[email protected]>
    
    * pythongh-85267: Improvements to inspect.signature __text_signature__ handling (python#98796)
    
    This makes a couple related changes to inspect.signature's behaviour
    when parsing a signature from `__text_signature__`.
    
    First, `inspect.signature` is documented as only raising ValueError or
    TypeError. However, in some cases, we could raise RuntimeError.  This PR
    changes that, thereby fixing python#83685.
    
    (Note that the new ValueErrors in RewriteSymbolics are caught and then
    reraised with a message)
    
    Second, `inspect.signature` could randomly drop parameters that it
    didn't understand (corresponding to `return None` in the `p` function).
    This is the core issue in python#85267. I think this is very surprising
    behaviour and it seems better to fail outright.
    
    Third, adding this new failure broke a couple tests. To fix them (and to
    e.g. allow `inspect.signature(select.epoll.register)` as in python#85267), I
    add constant folding of a couple binary operations to RewriteSymbolics.
    
    (There's some discussion of making signature expression evaluation
    arbitrary powerful in python#68155. I think that's out of scope. The
    additional constant folding here is pretty straightforward, useful, and
    not much of a slippery slope)
    
    Fourth, while python#85267 is incorrect about the cause of the issue, it turns
    out if you had consecutive newlines in __text_signature__, you'd get
    `tokenize.TokenError`.
    
    Finally, the `if name is invalid:` code path was dead, since
    `parse_name` never returned `invalid`.
    
    * pythonGH-100363: Speed up `asyncio.get_running_loop` (python#100364)
    
    * pythonGH-100133: fix `asyncio` subprocess losing `stderr` and `stdout` output (python#100154)
    
    * pythongh-100374: Fixed a bug in socket.getfqdn() (pythongh-100375)
    
    * pythongh-100129: Add tests for pickling all builtin types and functions (pythonGH-100142)
    
    * Remove unused variable from `dis._find_imports` (python#100396)
    
    * pythongh-78878: Fix crash when creating an instance of `_ctypes.CField` (python#14837)
    
    * pythonGH-69564: Clarify use of octal format of mode argument in help(os.chmod) (python#20621)
    
    Co-authored-by: Kumar Aditya <[email protected]>
    
    * pythonGH-99554: Pack location tables more effectively (pythonGH-99556)
    
    * Correct typo in typing.py (python#100423)
    
    In the docstring of `ParamSpec`, the name of `P = ParamSpec('P')` was
    mistakenly written as `'T'`.
    
    * pythongh-99761: Add `_PyLong_IsPositiveSingleDigit` function to check for single digit integers  (python#100064)
    
    * pythonGH-99770: Make the correct call specialization fail kind show up in the stats (pythonGH-99771)
    
    * pythongh-78997: fix bad rebase of moved test file (python#100424)
    
    * pythongh-100344: Add C implementation for `asyncio.current_task` (python#100345)
    
    Co-authored-by: pranavtbhat
    
    * pythonGH-99554: Trim trailing whitespace (pythonGH-100435)
    
    
    
    Automerge-Triggered-By: GH:brandtbucher
    
    * pythongh-85432: Harmonise parameter names between C and pure-Python implementations of `datetime.time.strftime`, `datetime.datetime.fromtimestamp` (python#99993)
    
    * pythongh-57762: fix misleading tkinter.Tk docstring (python#98837)
    
    Mentioned as a desired change by terryjreedy on the corresponding issue,
    since Tk is not a subclass of Toplevel.
    
    * pythongh-48496: Added example and link to faq for UnboundLocalError in reference (python#93068)
    
    * Fix typo in 3.12 What's New (python#100449)
    
    * pythongh-76963: PEP3118 itemsize of an empty ctypes array should not be 0 (pythonGH-5576)
    
    The itemsize returned in a memoryview of a ctypes array is now computed from the item type, instead of dividing the total size by the length and assuming that the length is not zero.
    
    * pythonGH-100459: fix copy-paste errors in specialization stats (pythonGH-100460)
    
    * pythongh-99110: Initialize `frame->previous` in init_frame to fix segmentation fault when accessing `frame.f_back` (python#100182)
    
    * pythongh-98712: Clarify "readonly bytes-like object" semantics in C arg-parsing docs (python#98710)
    
    * pythongh-92216: improve performance of `hasattr` for type objects (pythonGH-99979)
    
    * pythongh-100288: Specialise LOAD_ATTR_METHOD for managed dictionaries (pythonGH-100289)
    
    * Revert "pythongh-100288: Specialise LOAD_ATTR_METHOD for managed dictionaries (pythonGH-100289)" (python#100468)
    
    This reverts commit c3c7848.
    
    * pythongh-94155: Reduce hash collisions for code objects (python#100183)
    
    * Uses a better hashing algorithm to get better dispersion and remove commutativity.
    
    * Incorporates `co_firstlineno`, `Py_SIZE(co)`, and bytecode instructions.
    
    * This is now the entire set of criteria used in `code_richcompare`, except for `_PyCode_ConstantKey` (which would incorporate the types of `co_consts` rather than just their values).
    
    * pythongh-83076: 3.8x speed improvement in (Async)Mock instantiation (python#100252)
    
    * pythongh-99482: remove `jython` compatibility parts from stdlib and tests (python#99484)
    
    * bpo-40447: accept all path-like objects in compileall.compile_file (python#19883)
    
    Signed-off-by: Filipe Laíns <[email protected]>
    Signed-off-by: Filipe Laíns <[email protected]>
    Co-authored-by: Irit Katriel <[email protected]>
    Co-authored-by: Shantanu <[email protected]>
    
    * pythonGH-100425: Improve accuracy of builtin sum() for float inputs (pythonGH-100426)
    
    * pythongh-68320, pythongh-88302 - Allow for private `pathlib.Path` subclassing (pythonGH-31691)
    
    Users may wish to define subclasses of `pathlib.Path` to add or modify
    existing methods. Before this change, attempting to instantiate a subclass
    raised an exception like:
    
        AttributeError: type object 'PPath' has no attribute '_flavour'
    
    Previously the `_flavour` attribute was assigned as follows:
    
        PurePath._flavour        = xxx not set!! xxx
        PurePosixPath._flavour   = _PosixFlavour()
        PureWindowsPath._flavour = _WindowsFlavour()
    
    This change replaces it with a `_pathmod` attribute, set as follows:
    
        PurePath._pathmod        = os.path
        PurePosixPath._pathmod   = posixpath
        PureWindowsPath._pathmod = ntpath
    
    Functionality from `_PosixFlavour` and `_WindowsFlavour` is moved into
    `PurePath` as underscored-prefixed classmethods. Flavours are removed.
    
    Co-authored-by: Alex Waygood <[email protected]>
    Co-authored-by: Brett Cannon <[email protected]>
    Co-authored-by: Adam Turner <[email protected]>
    Co-authored-by: Eryk Sun <[email protected]>
    
    * pythongh-99947: Ensure unreported errors are chained for SystemError during import (pythonGH-99946)
    
    * Add "strict" to dotproduct(). Add docstring. Factor-out common code. (pythonGH-100480)
    
    * pythongh-94808: improve test coverage of number formatting (python#99472)
    
    * pythongh-100454: Start running SSL tests with OpenSSL 3.1.0-beta1 (python#100456)
    
    * pythongh-100268: Add is_integer method to int (python#100439)
    
    This improves the lives of type annotation users of `float` - which type checkers implicitly treat as `int|float` because that is what most code actually wants. Before this change a `.is_integer()` method could not be assumed to exist on things annotated as `: float` due to the method not existing on both types.
    
    * pythongh-77771: Add enterabs example in sched (python#92716)
    
    Co-authored-by: Shantanu <[email protected]>
    
    * pythonGH-91166: Implement zero copy writes for `SelectorSocketTransport` in asyncio (python#31871)
    
    Co-authored-by: Guido van Rossum <[email protected]>
    
    * pythonGH-91166: Implement zero copy writes for `SelectorSocketTransport` in asyncio (python#31871)
    
    Co-authored-by: Guido van Rossum <[email protected]>
    
    * Misc Itertools recipe tweaks (pythonGH-100493)
    
    * pythongh-100357: Convert several functions in `bltinsmodule` to AC (python#100358)
    
    * Remove wrong comment about `repr` in `test_unicode` (python#100495)
    
    * pythongh-99908: Tutorial: Modernize the 'data-record class' example (python#100499)
    
    Co-authored-by: Alex Waygood <[email protected]>
    
    * pythongh-100474: Fix handling of dirs named index.html in http.server (pythonGH-100475)
    
    
    
    If you had a directory called index.html or index.htm within a directory, it would cause http.server to return a 404 Not Found error instead of the directory listing. This came about due to not checking that the index was a regular file.
    
    I have also added a test case for this situation.
    
    Automerge-Triggered-By: GH:merwok
    
    * pythongh-100287: Fix unittest.mock.seal with AsyncMock (python#100496)
    
    * pythongh-99535: Add test for inheritance of annotations and update documentation (python#99990)
    
    * pythongh-100428: Make float documentation more accurate (python#100437)
    
    Previously, the grammar did not accept `float("10")`.
    Also implement mdickinson's suggestion of removing the indirection.
    
    * [Minor PR] Quotes in documentation changed into code blocks (python#99536)
    
    Minor formatting fix in documentation
    
    Co-authored-by: Shantanu <[email protected]>
    
    * pythongh-100472: Fix docs claim that compileall parameters could be bytes (python#100473)
    
    * pythongh-100519: simplification to `eff_request_host` in cookiejar.py (python#99588)
    
    `IPV4_RE` includes a `.`, and the `.find(".") == -1` included here is already testing to make sure there's no dot, so this part of the expression is tautological. Instead use more modern `in` syntax to make it clear what the check is doing here. The simplified implementation more clearly matches the wording in RFC 2965.
    
    Co-authored-by: hauntsaninja <[email protected]>
    
    * pythongh-99308: Clarify re docs for byte pattern group names (python#99311)
    
    * pythongh-92446: Improve argparse choices docs; revert bad change to lzma docs (python#94627)
    
    Based on the definition of the collections.abc classes, it is more accurate to use "sequence" instead of "container" when describing argparse choices.
    
    A previous attempt at fixing this in python#92450 was mistaken; this PR reverts that change.
    
    Co-authored-by: Shantanu <[email protected]>
    
    * Fix name of removed `inspect.Signature.from_builtin` method in 3.11.0a2 changelog (python#100525)
    
    * pythongh-100520: Fix `rst` markup in `configparser`  docstrings (python#100524)
    
    * pythongh-99509: Add `__class_getitem__` to `multiprocessing.queues.Queue` (python#99511)
    
    * pythongh-94603: micro optimize list.pop (pythongh-94604)
    
    * Remove `NoneType` redefinition from `clinic.py` (python#100551)
    
    * pythongh-100553: Improve accuracy of sqlite3.Row iter test (python#100555)
    
    * pythonGH-98831: Modernize a ton of simpler instructions (python#100545)
    
    * load_const and load_fast aren't families for now
    * Don't decref unmoved names
    * Modernize GET_ANEXT
    * Modernize GET_AWAITABLE
    * Modernize ASYNC_GEN_WRAP
    * Modernize YIELD_VALUE
    * Modernize POP_EXCEPT (in more than one way)
    * Modernize PREP_RERAISE_STAR
    * Modernize LOAD_ASSERTION_ERROR
    * Modernize LOAD_BUILD_CLASS
    * Modernize STORE_NAME
    * Modernize LOAD_NAME
    * Modernize LOAD_CLASSDEREF
    * Modernize LOAD_DEREF
    * Modernize STORE_DEREF
    * Modernize COPY_FREE_VARS (mark it as done)
    * Modernize LIST_TO_TUPLE
    * Modernize LIST_EXTEND
    * Modernize SET_UPDATE
    * Modernize SETUP_ANNOTATIONS
    * Modernize DICT_UPDATE
    * Modernize DICT_MERGE
    * Modernize MAP_ADD
    * Modernize IS_OP
    * Modernize CONTAINS_OP
    * Modernize CHECK_EXC_MATCH
    * Modernize IMPORT_NAME
    * Modernize IMPORT_STAR
    * Modernize IMPORT_FROM
    * Modernize JUMP_FORWARD (mark it as done)
    * Modernize JUMP_BACKWARD (mark it as done)
    
    Signed-off-by: Filipe Laíns <[email protected]>
    Signed-off-by: Filipe Laíns <[email protected]>
    Co-authored-by: Jeremy Paige <[email protected]>
    Co-authored-by: Carlton Gibson <[email protected]>
    Co-authored-by: Hugo van Kemenade <[email protected]>
    Co-authored-by: Jon Burdo <[email protected]>
    Co-authored-by: Stanley <[email protected]>
    Co-authored-by: Jelle Zijlstra <[email protected]>
    Co-authored-by: Thomas Grainger <[email protected]>
    Co-authored-by: Brad Wolfe <[email protected]>
    Co-authored-by: Richard Kojedzinszky <[email protected]>
    Co-authored-by: František Nesveda <[email protected]>
    Co-authored-by: Pablo Galindo Salgado <[email protected]>
    Co-authored-by: Nikita Sobolev <[email protected]>
    Co-authored-by: Łukasz Langa <[email protected]>
    Co-authored-by: Dennis Sweeney <[email protected]>
    Co-authored-by: Bisola Olasehinde <[email protected]>
    Co-authored-by: C.A.M. Gerlach <[email protected]>
    Co-authored-by: Pieter Eendebak <[email protected]>
    Co-authored-by: Shantanu <[email protected]>
    Co-authored-by: Kumar Aditya <[email protected]>
    Co-authored-by: Dominic Socular <[email protected]>
    Co-authored-by: Serhiy Storchaka <[email protected]>
    Co-authored-by: Hai Shi <[email protected]>
    Co-authored-by: amaajemyfren <[email protected]>
    Co-authored-by: Brandt Bucher <[email protected]>
    Co-authored-by: david-why <[email protected]>
    Co-authored-by: Pieter Eendebak <[email protected]>
    Co-authored-by: penguin_wwy <[email protected]>
    Co-authored-by: Eli Schwartz <[email protected]>
    Co-authored-by: Itamar Ostricher <[email protected]>
    Co-authored-by: Alex Waygood <[email protected]>
    Co-authored-by: Eric Wieser <[email protected]>
    Co-authored-by: Irit Katriel <[email protected]>
    Co-authored-by: Bill Fisher <[email protected]>
    Co-authored-by: Petr Viktorin <[email protected]>
    Co-authored-by: Ken Jin <[email protected]>
    Co-authored-by: Carl Meyer <[email protected]>
    Co-authored-by: Filipe Laíns <[email protected]>
    Co-authored-by: Raymond Hettinger <[email protected]>
    Co-authored-by: Barney Gale <[email protected]>
    Co-authored-by: Brett Cannon <[email protected]>
    Co-authored-by: Adam Turner <[email protected]>
    Co-authored-by: Eryk Sun <[email protected]>
    Co-authored-by: Sebastian Berg <[email protected]>
    Co-authored-by: Illia Volochii <[email protected]>
    Co-authored-by: JosephSBoyle <[email protected]>
    Co-authored-by: James Frost <[email protected]>
    Co-authored-by: MonadChains <[email protected]>
    Co-authored-by: Bart Broere <[email protected]>
    Co-authored-by: Glyph <[email protected]>
    Co-authored-by: hauntsaninja <[email protected]>
    Co-authored-by: Ilya Kulakov <[email protected]>
    Co-authored-by: Guy Yagev <[email protected]>
    Co-authored-by: Jakub Kuczys <[email protected]>
    @serhiy-storchaka
    Copy link
    Member

    I myself have faced the need to use a complex expression ащк for the default value several times. For example os.name != 'nt'. Perhaps the only workaround is to define a module variable and refer to it. But I did not use it, because now pydoc falls back to display literal __text_signature__ if inspect.signature() fails to parse it.

    But using eval() may have bigger impact. There is an open signature for exposing signature parsing from text (#81678). __text_signature__ is now taken into account for pure Python functions too (initially it was used to provide more readable signature when the actual code uses *args, **kwargs to handle positional-only parameters before implementing the syntax for it). Now Argument Clinic allows to override the generated signature by @text_signature. And it is essential in some other cases. There is an open issue for exposing the signature in the XML-RPC (#57613), it will create a precedence for parsing signatures taken from the Net.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants