-
-
Notifications
You must be signed in to change notification settings - Fork 30.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bpo-40447: accept all path-like objects in compileall.compile_file #19883
Conversation
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA). CLA MissingOur records indicate the following people have not signed the CLA: For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. If you have recently signed the CLA, please wait at least one business day You can check yourself to see if the CLA has been received. Thanks again for the contribution, we look forward to reviewing it! |
99f62ad
to
453d882
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason to not to use os.fspath
for supporting all path like objects?
Nop, I just totally forgot about that. Good catch :D |
924a7fd
to
7b48ce8
Compare
4a1312b
to
6cafeac
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Probably worth adding to the tests too :-)
Lib/compileall.py
Outdated
@@ -157,8 +157,8 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, | |||
dfile = os.path.join(ddir, name) | |||
|
|||
if stripdir is not None: | |||
fullname_parts = fullname.split(os.path.sep) | |||
stripdir_parts = stripdir.split(os.path.sep) | |||
fullname_parts = os.fspath(fullname).split(os.path.sep) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has redundancy with the logic on line 151. Not sure why that's gated by the condition on quiet
, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, yes. It is odd.
@berkerpeksag sorry for bothering, do you remember why the quiet check is there? It looks like quiet
is being used across the code to provide some set some kind of failing level here. The docstring is also a bit ambiguous.
full output with False or 0, errors only with 1, no output with 2
I am not sure how to parse the behavior for 1
.
Should this behavior be changed? If not, the docstring should definitely be touched up.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Anyway, I am moving the path-like object translation up there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also added the other path arguments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this PR!
This seems like a good fix, but needs a bit more work:
- ISTM that this should also fix the handling of the same parameters in
compile_dir
. - Tests?
- Clarify the NEWS entry (see inline comment.)
Misc/NEWS.d/next/Library/2020-05-03-12-55-55.bpo-40447.oKR0Lj.rst
Outdated
Show resolved
Hide resolved
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
d63b4e5
to
31251fb
Compare
requested changes; please review again |
Thanks @FFY00, that was quick! The NEWS entry mentions fixing |
Lib/compileall.py
Outdated
@@ -152,8 +152,14 @@ def compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, | |||
"in combination with stripdir or prependdir")) | |||
|
|||
success = True | |||
if quiet < 2 and isinstance(fullname, os.PathLike): | |||
if isinstance(fullname, os.PathLike): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: os.fspath is happy to take str
and bytes
, so you don't need this check for fullname (for the other three, you do need something to guard against None)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right! The last threeos.fspath
calls can simply be moved below into each of the blocks handling the three directory arguments, without the isinstance
checks. The first one shouldn't need the isinstance
guard at all; letting os.fspath
error on invalid types seems like precisely what we want here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delay! This fell off my working pool. Made the changes 😊
a6a1788
to
599bfd9
Compare
Lib/compileall.py
Outdated
name = os.path.basename(fullname) | ||
|
||
dfile = None | ||
|
||
if ddir is not None: | ||
dfile = os.path.join(ddir, name) | ||
dfile = os.path.join(os.fspath(ddir), name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One more nit: I think the call to os.fspath
is only needed on line 165 with stripdir; os.path.join is happy to take PathLike objects
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although actually neither split works when stripdir
is bytes, which the documentation says it's allowed to be, since os.path.sep
is a str. Same for fullname
although the documentation doesn't mention that bytes is allowed there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made the changes, let me know if there's anything else.
@taleinat pushed the requested changes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, though I haven't reviewed in detail. Let me know if there's something in particular you'd like me to critique.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this looks correct to me.
There's still one issue that I pointed out in #19883 (comment) , but that is separate enough that we can deal with it later.
I'm not sure whether to backport. While the code itself is "new feature" it is a feature that the docs in previous versions promise exists. We should either a) backport this change, or b) amend the docs on old branches. I lean towards a) for simplicity.
Misc/NEWS.d/next/Library/2020-05-03-12-55-55.bpo-40447.oKR0Lj.rst
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with backporting since this was specifically documented to work.
Thanks @FFY00 for the PR, and @hauntsaninja for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11. |
GH-100470 is a backport of this pull request to the 3.11 branch. |
GH-100471 is a backport of this pull request to the 3.10 branch. |
…ythonGH-19883) (cherry picked from commit 1ecfd1e) Co-authored-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: Shantanu <[email protected]>
…ythonGH-19883) (cherry picked from commit 1ecfd1e) Co-authored-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: Shantanu <[email protected]>
…H-19883) (cherry picked from commit 1ecfd1e) Co-authored-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: Shantanu <[email protected]>
…H-19883) (cherry picked from commit 1ecfd1e) Co-authored-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: Shantanu <[email protected]>
* Correct CVE-2020-10735 documentation (python#100306) * pythongh-94912: Added marker for non-standard coroutine function detection (python#99247) This introduces a new decorator `@inspect.markcoroutinefunction`, which, applied to a sync function, makes it appear async to `inspect.iscoroutinefunction()`. * Docs: Don't upload CI artifacts (python#100330) * pythongh-89727: Fix os.walk RecursionError on deep trees (python#99803) Use a stack to implement os.walk iteratively instead of recursively to avoid hitting recursion limits on deeply nested trees. * pythongh-69929: re docs: Add more specific definition of \w (python#92015) Co-authored-by: Jelle Zijlstra <[email protected]> * pythongh-89051: Add ssl.OP_LEGACY_SERVER_CONNECT (python#93927) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Co-authored-by: Christian Heimes <[email protected]> Co-authored-by: Hugo van Kemenade <[email protected]> Fixes python#89051 * pythongh-88211: Change lower-case and upper-case to match recommendations in imaplib docs (python#99625) * pythongh-100348: Fix ref cycle in `asyncio._SelectorSocketTransport` with `_read_ready_cb` (python#100349) * pythongh-99925: Fix inconsistency in `json.dumps()` error messages (pythonGH-99926) * Clarify that every thread has its own default context in contextvars (python#99246) * pythongh-99576: Fix cookiejar file that was not truncated for some classes (pythonGH-99616) Co-authored-by: Łukasz Langa <[email protected]> * pythongh-100188: Reduce misses in BINARY_SUBSCR_(LIST/TUPLE)_INT (python#100189) Don't specialize if the index is negative. * pythongh-99991: improve docs on str.encode and bytes.decode (python#100198) Co-authored-by: C.A.M. Gerlach <[email protected]> * pythongh-91081: Add note on WeakKeyDictionary behavior when deleting a replaced entry (python#91499) Co-authored-by: Pieter Eendebak <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]> * pythongh-85267: Improvements to inspect.signature __text_signature__ handling (python#98796) This makes a couple related changes to inspect.signature's behaviour when parsing a signature from `__text_signature__`. First, `inspect.signature` is documented as only raising ValueError or TypeError. However, in some cases, we could raise RuntimeError. This PR changes that, thereby fixing python#83685. (Note that the new ValueErrors in RewriteSymbolics are caught and then reraised with a message) Second, `inspect.signature` could randomly drop parameters that it didn't understand (corresponding to `return None` in the `p` function). This is the core issue in python#85267. I think this is very surprising behaviour and it seems better to fail outright. Third, adding this new failure broke a couple tests. To fix them (and to e.g. allow `inspect.signature(select.epoll.register)` as in python#85267), I add constant folding of a couple binary operations to RewriteSymbolics. (There's some discussion of making signature expression evaluation arbitrary powerful in python#68155. I think that's out of scope. The additional constant folding here is pretty straightforward, useful, and not much of a slippery slope) Fourth, while python#85267 is incorrect about the cause of the issue, it turns out if you had consecutive newlines in __text_signature__, you'd get `tokenize.TokenError`. Finally, the `if name is invalid:` code path was dead, since `parse_name` never returned `invalid`. * pythonGH-100363: Speed up `asyncio.get_running_loop` (python#100364) * pythonGH-100133: fix `asyncio` subprocess losing `stderr` and `stdout` output (python#100154) * pythongh-100374: Fixed a bug in socket.getfqdn() (pythongh-100375) * pythongh-100129: Add tests for pickling all builtin types and functions (pythonGH-100142) * Remove unused variable from `dis._find_imports` (python#100396) * pythongh-78878: Fix crash when creating an instance of `_ctypes.CField` (python#14837) * pythonGH-69564: Clarify use of octal format of mode argument in help(os.chmod) (python#20621) Co-authored-by: Kumar Aditya <[email protected]> * pythonGH-99554: Pack location tables more effectively (pythonGH-99556) * Correct typo in typing.py (python#100423) In the docstring of `ParamSpec`, the name of `P = ParamSpec('P')` was mistakenly written as `'T'`. * pythongh-99761: Add `_PyLong_IsPositiveSingleDigit` function to check for single digit integers (python#100064) * pythonGH-99770: Make the correct call specialization fail kind show up in the stats (pythonGH-99771) * pythongh-78997: fix bad rebase of moved test file (python#100424) * pythongh-100344: Add C implementation for `asyncio.current_task` (python#100345) Co-authored-by: pranavtbhat * pythonGH-99554: Trim trailing whitespace (pythonGH-100435) Automerge-Triggered-By: GH:brandtbucher * pythongh-85432: Harmonise parameter names between C and pure-Python implementations of `datetime.time.strftime`, `datetime.datetime.fromtimestamp` (python#99993) * pythongh-57762: fix misleading tkinter.Tk docstring (python#98837) Mentioned as a desired change by terryjreedy on the corresponding issue, since Tk is not a subclass of Toplevel. * pythongh-48496: Added example and link to faq for UnboundLocalError in reference (python#93068) * Fix typo in 3.12 What's New (python#100449) * pythongh-76963: PEP3118 itemsize of an empty ctypes array should not be 0 (pythonGH-5576) The itemsize returned in a memoryview of a ctypes array is now computed from the item type, instead of dividing the total size by the length and assuming that the length is not zero. * pythonGH-100459: fix copy-paste errors in specialization stats (pythonGH-100460) * pythongh-99110: Initialize `frame->previous` in init_frame to fix segmentation fault when accessing `frame.f_back` (python#100182) * pythongh-98712: Clarify "readonly bytes-like object" semantics in C arg-parsing docs (python#98710) * pythongh-92216: improve performance of `hasattr` for type objects (pythonGH-99979) * pythongh-100288: Specialise LOAD_ATTR_METHOD for managed dictionaries (pythonGH-100289) * Revert "pythongh-100288: Specialise LOAD_ATTR_METHOD for managed dictionaries (pythonGH-100289)" (python#100468) This reverts commit c3c7848. * pythongh-94155: Reduce hash collisions for code objects (python#100183) * Uses a better hashing algorithm to get better dispersion and remove commutativity. * Incorporates `co_firstlineno`, `Py_SIZE(co)`, and bytecode instructions. * This is now the entire set of criteria used in `code_richcompare`, except for `_PyCode_ConstantKey` (which would incorporate the types of `co_consts` rather than just their values). * pythongh-83076: 3.8x speed improvement in (Async)Mock instantiation (python#100252) * pythongh-99482: remove `jython` compatibility parts from stdlib and tests (python#99484) * bpo-40447: accept all path-like objects in compileall.compile_file (python#19883) Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: Shantanu <[email protected]> * pythonGH-100425: Improve accuracy of builtin sum() for float inputs (pythonGH-100426) * pythongh-68320, pythongh-88302 - Allow for private `pathlib.Path` subclassing (pythonGH-31691) Users may wish to define subclasses of `pathlib.Path` to add or modify existing methods. Before this change, attempting to instantiate a subclass raised an exception like: AttributeError: type object 'PPath' has no attribute '_flavour' Previously the `_flavour` attribute was assigned as follows: PurePath._flavour = xxx not set!! xxx PurePosixPath._flavour = _PosixFlavour() PureWindowsPath._flavour = _WindowsFlavour() This change replaces it with a `_pathmod` attribute, set as follows: PurePath._pathmod = os.path PurePosixPath._pathmod = posixpath PureWindowsPath._pathmod = ntpath Functionality from `_PosixFlavour` and `_WindowsFlavour` is moved into `PurePath` as underscored-prefixed classmethods. Flavours are removed. Co-authored-by: Alex Waygood <[email protected]> Co-authored-by: Brett Cannon <[email protected]> Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Eryk Sun <[email protected]> * pythongh-99947: Ensure unreported errors are chained for SystemError during import (pythonGH-99946) * Add "strict" to dotproduct(). Add docstring. Factor-out common code. (pythonGH-100480) * pythongh-94808: improve test coverage of number formatting (python#99472) * pythongh-100454: Start running SSL tests with OpenSSL 3.1.0-beta1 (python#100456) * pythongh-100268: Add is_integer method to int (python#100439) This improves the lives of type annotation users of `float` - which type checkers implicitly treat as `int|float` because that is what most code actually wants. Before this change a `.is_integer()` method could not be assumed to exist on things annotated as `: float` due to the method not existing on both types. * pythongh-77771: Add enterabs example in sched (python#92716) Co-authored-by: Shantanu <[email protected]> * pythonGH-91166: Implement zero copy writes for `SelectorSocketTransport` in asyncio (python#31871) Co-authored-by: Guido van Rossum <[email protected]> * pythonGH-91166: Implement zero copy writes for `SelectorSocketTransport` in asyncio (python#31871) Co-authored-by: Guido van Rossum <[email protected]> * Misc Itertools recipe tweaks (pythonGH-100493) * pythongh-100357: Convert several functions in `bltinsmodule` to AC (python#100358) * Remove wrong comment about `repr` in `test_unicode` (python#100495) * pythongh-99908: Tutorial: Modernize the 'data-record class' example (python#100499) Co-authored-by: Alex Waygood <[email protected]> * pythongh-100474: Fix handling of dirs named index.html in http.server (pythonGH-100475) If you had a directory called index.html or index.htm within a directory, it would cause http.server to return a 404 Not Found error instead of the directory listing. This came about due to not checking that the index was a regular file. I have also added a test case for this situation. Automerge-Triggered-By: GH:merwok * pythongh-100287: Fix unittest.mock.seal with AsyncMock (python#100496) * pythongh-99535: Add test for inheritance of annotations and update documentation (python#99990) * pythongh-100428: Make float documentation more accurate (python#100437) Previously, the grammar did not accept `float("10")`. Also implement mdickinson's suggestion of removing the indirection. * [Minor PR] Quotes in documentation changed into code blocks (python#99536) Minor formatting fix in documentation Co-authored-by: Shantanu <[email protected]> * pythongh-100472: Fix docs claim that compileall parameters could be bytes (python#100473) * pythongh-100519: simplification to `eff_request_host` in cookiejar.py (python#99588) `IPV4_RE` includes a `.`, and the `.find(".") == -1` included here is already testing to make sure there's no dot, so this part of the expression is tautological. Instead use more modern `in` syntax to make it clear what the check is doing here. The simplified implementation more clearly matches the wording in RFC 2965. Co-authored-by: hauntsaninja <[email protected]> * pythongh-99308: Clarify re docs for byte pattern group names (python#99311) * pythongh-92446: Improve argparse choices docs; revert bad change to lzma docs (python#94627) Based on the definition of the collections.abc classes, it is more accurate to use "sequence" instead of "container" when describing argparse choices. A previous attempt at fixing this in python#92450 was mistaken; this PR reverts that change. Co-authored-by: Shantanu <[email protected]> * Fix name of removed `inspect.Signature.from_builtin` method in 3.11.0a2 changelog (python#100525) * pythongh-100520: Fix `rst` markup in `configparser` docstrings (python#100524) * pythongh-99509: Add `__class_getitem__` to `multiprocessing.queues.Queue` (python#99511) * pythongh-94603: micro optimize list.pop (pythongh-94604) * Remove `NoneType` redefinition from `clinic.py` (python#100551) * pythongh-100553: Improve accuracy of sqlite3.Row iter test (python#100555) * pythonGH-98831: Modernize a ton of simpler instructions (python#100545) * load_const and load_fast aren't families for now * Don't decref unmoved names * Modernize GET_ANEXT * Modernize GET_AWAITABLE * Modernize ASYNC_GEN_WRAP * Modernize YIELD_VALUE * Modernize POP_EXCEPT (in more than one way) * Modernize PREP_RERAISE_STAR * Modernize LOAD_ASSERTION_ERROR * Modernize LOAD_BUILD_CLASS * Modernize STORE_NAME * Modernize LOAD_NAME * Modernize LOAD_CLASSDEREF * Modernize LOAD_DEREF * Modernize STORE_DEREF * Modernize COPY_FREE_VARS (mark it as done) * Modernize LIST_TO_TUPLE * Modernize LIST_EXTEND * Modernize SET_UPDATE * Modernize SETUP_ANNOTATIONS * Modernize DICT_UPDATE * Modernize DICT_MERGE * Modernize MAP_ADD * Modernize IS_OP * Modernize CONTAINS_OP * Modernize CHECK_EXC_MATCH * Modernize IMPORT_NAME * Modernize IMPORT_STAR * Modernize IMPORT_FROM * Modernize JUMP_FORWARD (mark it as done) * Modernize JUMP_BACKWARD (mark it as done) Signed-off-by: Filipe Laíns <[email protected]> Signed-off-by: Filipe Laíns <[email protected]> Co-authored-by: Jeremy Paige <[email protected]> Co-authored-by: Carlton Gibson <[email protected]> Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Jon Burdo <[email protected]> Co-authored-by: Stanley <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]> Co-authored-by: Thomas Grainger <[email protected]> Co-authored-by: Brad Wolfe <[email protected]> Co-authored-by: Richard Kojedzinszky <[email protected]> Co-authored-by: František Nesveda <[email protected]> Co-authored-by: Pablo Galindo Salgado <[email protected]> Co-authored-by: Nikita Sobolev <[email protected]> Co-authored-by: Łukasz Langa <[email protected]> Co-authored-by: Dennis Sweeney <[email protected]> Co-authored-by: Bisola Olasehinde <[email protected]> Co-authored-by: C.A.M. Gerlach <[email protected]> Co-authored-by: Pieter Eendebak <[email protected]> Co-authored-by: Shantanu <[email protected]> Co-authored-by: Kumar Aditya <[email protected]> Co-authored-by: Dominic Socular <[email protected]> Co-authored-by: Serhiy Storchaka <[email protected]> Co-authored-by: Hai Shi <[email protected]> Co-authored-by: amaajemyfren <[email protected]> Co-authored-by: Brandt Bucher <[email protected]> Co-authored-by: david-why <[email protected]> Co-authored-by: Pieter Eendebak <[email protected]> Co-authored-by: penguin_wwy <[email protected]> Co-authored-by: Eli Schwartz <[email protected]> Co-authored-by: Itamar Ostricher <[email protected]> Co-authored-by: Alex Waygood <[email protected]> Co-authored-by: Eric Wieser <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: Bill Fisher <[email protected]> Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Ken Jin <[email protected]> Co-authored-by: Carl Meyer <[email protected]> Co-authored-by: Filipe Laíns <[email protected]> Co-authored-by: Raymond Hettinger <[email protected]> Co-authored-by: Barney Gale <[email protected]> Co-authored-by: Brett Cannon <[email protected]> Co-authored-by: Adam Turner <[email protected]> Co-authored-by: Eryk Sun <[email protected]> Co-authored-by: Sebastian Berg <[email protected]> Co-authored-by: Illia Volochii <[email protected]> Co-authored-by: JosephSBoyle <[email protected]> Co-authored-by: James Frost <[email protected]> Co-authored-by: MonadChains <[email protected]> Co-authored-by: Bart Broere <[email protected]> Co-authored-by: Glyph <[email protected]> Co-authored-by: hauntsaninja <[email protected]> Co-authored-by: Ilya Kulakov <[email protected]> Co-authored-by: Guy Yagev <[email protected]> Co-authored-by: Jakub Kuczys <[email protected]>
https://bugs.python.org/issue40447