From 39e60aeb3837f1f23d8b7f30d3b8d9faf805ef88 Mon Sep 17 00:00:00 2001 From: Rafael Fontenelle Date: Fri, 22 Nov 2024 21:02:51 -0300 Subject: [PATCH 01/24] Fix a few typos found in the docs (GH-127126) --- Doc/library/importlib.metadata.rst | 2 +- Doc/library/multiprocessing.rst | 4 ++-- Doc/library/socket.rst | 2 +- Misc/NEWS.d/3.13.0a6.rst | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index 37cd237357aa4b..d80255f5313061 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -133,7 +133,7 @@ Entry points Details of a collection of installed entry points. - Also provides a ``.groups`` attribute that reports all identifed entry + Also provides a ``.groups`` attribute that reports all identified entry point groups, and a ``.names`` attribute that reports all identified entry point names. diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 036b8f44b9ff3b..783cb025826483 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -291,7 +291,7 @@ processes: of corruption from processes using different ends of the pipe at the same time. - The :meth:`~Connection.send` method serializes the the object and + The :meth:`~Connection.send` method serializes the object and :meth:`~Connection.recv` re-creates the object. Synchronization between processes @@ -828,7 +828,7 @@ For an example of the usage of queues for interprocess communication see used for receiving messages and ``conn2`` can only be used for sending messages. - The :meth:`~multiprocessing.Connection.send` method serializes the the object using + The :meth:`~multiprocessing.Connection.send` method serializes the object using :mod:`pickle` and the :meth:`~multiprocessing.Connection.recv` re-creates the object. .. class:: Queue([maxsize]) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 6358d140484c78..73d495c055ff6e 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -979,7 +979,7 @@ The :mod:`socket` module also offers various network-related services: These addresses should generally be tried in order until a connection succeeds (possibly tried in parallel, for example, using a `Happy Eyeballs`_ algorithm). In these cases, limiting the *type* and/or *proto* can help eliminate - unsuccessful or unusable connecton attempts. + unsuccessful or unusable connection attempts. Some systems will, however, only return a single address. (For example, this was reported on Solaris and AIX configurations.) diff --git a/Misc/NEWS.d/3.13.0a6.rst b/Misc/NEWS.d/3.13.0a6.rst index b9cdbc4e146d5a..2740b4f0d967ba 100644 --- a/Misc/NEWS.d/3.13.0a6.rst +++ b/Misc/NEWS.d/3.13.0a6.rst @@ -642,7 +642,7 @@ Also in the corresponding :class:`ipaddress.IPv4Network` and .. nonce: OToJnG .. section: Library -In :mod:`encodings.idna`, any capitalization of the the ACE prefix +In :mod:`encodings.idna`, any capitalization of the ACE prefix (``xn--``) is now acceptable. Patch by Pepijn de Vos and Zackery Spytz. .. From a13e94d84bff334da3da2cab523ba75b57e0787f Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Fri, 22 Nov 2024 19:18:18 -0800 Subject: [PATCH 02/24] GH-127134: Add note about forward compatibility for suggest_on_error (#127137) --- Doc/library/argparse.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index e574511436d16e..410b6e11af0dc0 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -589,6 +589,14 @@ are strings:: >>> parser.parse_args(['--action', 'sumn', 1, 2, 3]) tester.py: error: argument --action: invalid choice: 'sumn', maybe you meant 'sum'? (choose from 'sum', 'max') +If you're writing code that needs to be compatible with older Python versions +and want to opportunistically use ``suggest_on_error`` when it's available, you +can set it as an attribute after initializing the parser instead of using the +keyword argument:: + + >>> parser = argparse.ArgumentParser(description='Process some integers.') + >>> parser.suggest_on_error = True + .. versionadded:: 3.14 From cc813e10ff190af38b8429d0d49fb9249493d504 Mon Sep 17 00:00:00 2001 From: Barney Gale Date: Sat, 23 Nov 2024 10:41:39 +0000 Subject: [PATCH 03/24] GH-125866: Preserve Windows drive letter case in file URIs (#127138) Stop converting Windows drive letters to uppercase in `urllib.request.pathname2url()` and `url2pathname()`. This behaviour is unnecessary and inconsistent with pathlib's file URI implementation. --- Doc/library/urllib.request.rst | 7 +++++++ Lib/nturl2path.py | 4 ++-- Lib/test/test_urllib.py | 2 ++ .../Library/2024-11-22-04-49-31.gh-issue-125866.TUtvPK.rst | 2 ++ 4 files changed, 13 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2024-11-22-04-49-31.gh-issue-125866.TUtvPK.rst diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index e0831bf7e65ad2..a093a5083e037b 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -152,6 +152,9 @@ The :mod:`urllib.request` module defines the following functions: the path component of a URL. This does not produce a complete URL. The return value will already be quoted using the :func:`~urllib.parse.quote` function. + .. versionchanged:: 3.14 + Windows drive letters are no longer converted to uppercase. + .. versionchanged:: 3.14 On Windows, ``:`` characters not following a drive letter are quoted. In previous versions, :exc:`OSError` was raised if a colon character was @@ -164,6 +167,10 @@ The :mod:`urllib.request` module defines the following functions: path. This does not accept a complete URL. This function uses :func:`~urllib.parse.unquote` to decode *path*. + .. versionchanged:: 3.14 + Windows drive letters are no longer converted to uppercase. + + .. function:: getproxies() This helper function returns a dictionary of scheme to proxy server URL diff --git a/Lib/nturl2path.py b/Lib/nturl2path.py index 66092e4821a0ec..01135d1b7683b2 100644 --- a/Lib/nturl2path.py +++ b/Lib/nturl2path.py @@ -35,7 +35,7 @@ def url2pathname(url): if len(comp) != 2 or comp[0][-1] not in string.ascii_letters: error = 'Bad URL: ' + url raise OSError(error) - drive = comp[0][-1].upper() + drive = comp[0][-1] tail = urllib.parse.unquote(comp[1].replace('/', '\\')) return drive + ':' + tail @@ -60,7 +60,7 @@ def pathname2url(p): # DOS drive specified. Add three slashes to the start, producing # an authority section with a zero-length authority, and a path # section starting with a single slash. - drive = f'///{drive.upper()}' + drive = f'///{drive}' drive = urllib.parse.quote(drive, safe='/:') tail = urllib.parse.quote(tail) diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py index a204ef41c3ce90..22ef3c648e271d 100644 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -1423,6 +1423,7 @@ def test_pathname2url_win(self): self.assertEqual(fn('\\\\?\\unc\\server\\share\\dir'), '//server/share/dir') self.assertEqual(fn("C:"), '///C:') self.assertEqual(fn("C:\\"), '///C:/') + self.assertEqual(fn('c:\\a\\b.c'), '///c:/a/b.c') self.assertEqual(fn('C:\\a\\b.c'), '///C:/a/b.c') self.assertEqual(fn('C:\\a\\b.c\\'), '///C:/a/b.c/') self.assertEqual(fn('C:\\a\\\\b.c'), '///C:/a//b.c') @@ -1480,6 +1481,7 @@ def test_url2pathname_win(self): self.assertEqual(fn("///C/test/"), '\\C\\test\\') self.assertEqual(fn("////C/test/"), '\\\\C\\test\\') # DOS drive paths + self.assertEqual(fn('c:/path/to/file'), 'c:\\path\\to\\file') self.assertEqual(fn('C:/path/to/file'), 'C:\\path\\to\\file') self.assertEqual(fn('C:/path/to/file/'), 'C:\\path\\to\\file\\') self.assertEqual(fn('C:/path/to//file'), 'C:\\path\\to\\\\file') diff --git a/Misc/NEWS.d/next/Library/2024-11-22-04-49-31.gh-issue-125866.TUtvPK.rst b/Misc/NEWS.d/next/Library/2024-11-22-04-49-31.gh-issue-125866.TUtvPK.rst new file mode 100644 index 00000000000000..682e061747689b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-22-04-49-31.gh-issue-125866.TUtvPK.rst @@ -0,0 +1,2 @@ +:func:`urllib.request.pathname2url` and :func:`~urllib.request.url2pathname` +no longer convert Windows drive letters to uppercase. From e3038e976b25a58f512d8c7083a752c89436eb0d Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 23 Nov 2024 14:47:08 -0500 Subject: [PATCH 04/24] Doc: C API: Fix `Py_NewInterpreterFromConfig` example code (#126667) --- Doc/c-api/init.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 24d876d1f35506..970084ce91ab69 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1738,7 +1738,11 @@ function. You can create and destroy them using the following functions: .check_multi_interp_extensions = 1, .gil = PyInterpreterConfig_OWN_GIL, }; - PyThreadState *tstate = Py_NewInterpreterFromConfig(&config); + PyThreadState *tstate = NULL; + PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config); + if (PyStatus_Exception(status)) { + Py_ExitStatusException(status); + } Note that the config is used only briefly and does not get modified. During initialization the config's values are converted into various From dbd23790dbd662169905be6300259992639d4e69 Mon Sep 17 00:00:00 2001 From: "Stan U." <89152624+StanFromIreland@users.noreply.github.com> Date: Sat, 23 Nov 2024 21:41:01 +0000 Subject: [PATCH 05/24] Fix "useable" typo in docs (#127200) Fix typo in docs --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 29db15b41d2c72..394cdc3638485d 100644 --- a/README.rst +++ b/README.rst @@ -64,7 +64,7 @@ the executable is called ``python.exe``; elsewhere it's just ``python``. Building a complete Python installation requires the use of various additional third-party libraries, depending on your build platform and configure options. Not all standard library modules are buildable or -useable on all platforms. Refer to the +usable on all platforms. Refer to the `Install dependencies `_ section of the `Developer Guide`_ for current detailed information on dependencies for various Linux distributions and macOS. From 4ea71278caedb3d12a45f87c757418b8684ba7dd Mon Sep 17 00:00:00 2001 From: Barney Gale Date: Sun, 24 Nov 2024 02:31:00 +0000 Subject: [PATCH 06/24] pathlib tests: move `walk()` tests into their own classes (GH-126651) Move tests for Path.walk() into a new PathWalkTest class, and apply a similar change in tests for the ABCs. This allows us to properly tear down the walk test hierarchy in tearDown(), rather than leaving it to os_helper.rmtree(). --- Lib/test/test_pathlib/test_pathlib.py | 166 ++++++++++++---------- Lib/test/test_pathlib/test_pathlib_abc.py | 25 +++- 2 files changed, 107 insertions(+), 84 deletions(-) diff --git a/Lib/test/test_pathlib/test_pathlib.py b/Lib/test/test_pathlib/test_pathlib.py index 46966b6df2d7b0..6a994f890da616 100644 --- a/Lib/test/test_pathlib/test_pathlib.py +++ b/Lib/test/test_pathlib/test_pathlib.py @@ -1425,84 +1425,6 @@ def test_passing_kwargs_errors(self): with self.assertRaises(TypeError): self.cls(foo="bar") - def setUpWalk(self): - super().setUpWalk() - sub21_path= self.sub2_path / "SUB21" - tmp5_path = sub21_path / "tmp3" - broken_link3_path = self.sub2_path / "broken_link3" - - os.makedirs(sub21_path) - tmp5_path.write_text("I am tmp5, blame test_pathlib.") - if self.can_symlink: - os.symlink(tmp5_path, broken_link3_path) - self.sub2_tree[2].append('broken_link3') - self.sub2_tree[2].sort() - if not is_emscripten: - # Emscripten fails with inaccessible directories. - os.chmod(sub21_path, 0) - try: - os.listdir(sub21_path) - except PermissionError: - self.sub2_tree[1].append('SUB21') - else: - os.chmod(sub21_path, stat.S_IRWXU) - os.unlink(tmp5_path) - os.rmdir(sub21_path) - - def test_walk_bad_dir(self): - self.setUpWalk() - errors = [] - walk_it = self.walk_path.walk(on_error=errors.append) - root, dirs, files = next(walk_it) - self.assertEqual(errors, []) - dir1 = 'SUB1' - path1 = root / dir1 - path1new = (root / dir1).with_suffix(".new") - path1.rename(path1new) - try: - roots = [r for r, _, _ in walk_it] - self.assertTrue(errors) - self.assertNotIn(path1, roots) - self.assertNotIn(path1new, roots) - for dir2 in dirs: - if dir2 != dir1: - self.assertIn(root / dir2, roots) - finally: - path1new.rename(path1) - - def test_walk_many_open_files(self): - depth = 30 - base = self.cls(self.base, 'deep') - path = self.cls(base, *(['d']*depth)) - path.mkdir(parents=True) - - iters = [base.walk(top_down=False) for _ in range(100)] - for i in range(depth + 1): - expected = (path, ['d'] if i else [], []) - for it in iters: - self.assertEqual(next(it), expected) - path = path.parent - - iters = [base.walk(top_down=True) for _ in range(100)] - path = base - for i in range(depth + 1): - expected = (path, ['d'] if i < depth else [], []) - for it in iters: - self.assertEqual(next(it), expected) - path = path / 'd' - - def test_walk_above_recursion_limit(self): - recursion_limit = 40 - # directory_depth > recursion_limit - directory_depth = recursion_limit + 10 - base = self.cls(self.base, 'deep') - path = base.joinpath(*(['d'] * directory_depth)) - path.mkdir(parents=True) - - with infinite_recursion(recursion_limit): - list(base.walk()) - list(base.walk(top_down=False)) - def test_glob_empty_pattern(self): p = self.cls('') with self.assertRaisesRegex(ValueError, 'Unacceptable pattern'): @@ -1886,6 +1808,94 @@ def test_group_windows(self): P('c:/').group() +class PathWalkTest(test_pathlib_abc.DummyPathWalkTest): + cls = pathlib.Path + base = PathTest.base + can_symlink = PathTest.can_symlink + + def setUp(self): + super().setUp() + sub21_path= self.sub2_path / "SUB21" + tmp5_path = sub21_path / "tmp3" + broken_link3_path = self.sub2_path / "broken_link3" + + os.makedirs(sub21_path) + tmp5_path.write_text("I am tmp5, blame test_pathlib.") + if self.can_symlink: + os.symlink(tmp5_path, broken_link3_path) + self.sub2_tree[2].append('broken_link3') + self.sub2_tree[2].sort() + if not is_emscripten: + # Emscripten fails with inaccessible directories. + os.chmod(sub21_path, 0) + try: + os.listdir(sub21_path) + except PermissionError: + self.sub2_tree[1].append('SUB21') + else: + os.chmod(sub21_path, stat.S_IRWXU) + os.unlink(tmp5_path) + os.rmdir(sub21_path) + + def tearDown(self): + if 'SUB21' in self.sub2_tree[1]: + os.chmod(self.sub2_path / "SUB21", stat.S_IRWXU) + super().tearDown() + + def test_walk_bad_dir(self): + errors = [] + walk_it = self.walk_path.walk(on_error=errors.append) + root, dirs, files = next(walk_it) + self.assertEqual(errors, []) + dir1 = 'SUB1' + path1 = root / dir1 + path1new = (root / dir1).with_suffix(".new") + path1.rename(path1new) + try: + roots = [r for r, _, _ in walk_it] + self.assertTrue(errors) + self.assertNotIn(path1, roots) + self.assertNotIn(path1new, roots) + for dir2 in dirs: + if dir2 != dir1: + self.assertIn(root / dir2, roots) + finally: + path1new.rename(path1) + + def test_walk_many_open_files(self): + depth = 30 + base = self.cls(self.base, 'deep') + path = self.cls(base, *(['d']*depth)) + path.mkdir(parents=True) + + iters = [base.walk(top_down=False) for _ in range(100)] + for i in range(depth + 1): + expected = (path, ['d'] if i else [], []) + for it in iters: + self.assertEqual(next(it), expected) + path = path.parent + + iters = [base.walk(top_down=True) for _ in range(100)] + path = base + for i in range(depth + 1): + expected = (path, ['d'] if i < depth else [], []) + for it in iters: + self.assertEqual(next(it), expected) + path = path / 'd' + + def test_walk_above_recursion_limit(self): + recursion_limit = 40 + # directory_depth > recursion_limit + directory_depth = recursion_limit + 10 + base = self.cls(self.base, 'deep') + path = base.joinpath(*(['d'] * directory_depth)) + path.mkdir(parents=True) + + with infinite_recursion(recursion_limit): + list(base.walk()) + list(base.walk(top_down=False)) + + @unittest.skipIf(os.name == 'nt', 'test requires a POSIX-compatible system') class PosixPathTest(PathTest, PurePosixPathTest): cls = pathlib.PosixPath diff --git a/Lib/test/test_pathlib/test_pathlib_abc.py b/Lib/test/test_pathlib/test_pathlib_abc.py index b69d674e1cf1ed..aaa30a17f2af14 100644 --- a/Lib/test/test_pathlib/test_pathlib_abc.py +++ b/Lib/test/test_pathlib/test_pathlib_abc.py @@ -2922,7 +2922,16 @@ def test_delete_missing(self): filename = tmp / 'foo' self.assertRaises(FileNotFoundError, filename._delete) - def setUpWalk(self): + +class DummyPathWalkTest(unittest.TestCase): + cls = DummyPath + base = DummyPathTest.base + can_symlink = False + + def setUp(self): + name = self.id().split('.')[-1] + if name in _tests_needing_symlinks and not self.can_symlink: + self.skipTest('requires symlinks') # Build: # TESTFN/ # TEST1/ a file kid and two directory kids @@ -2966,8 +2975,11 @@ def setUpWalk(self): else: self.sub2_tree = (self.sub2_path, [], ["tmp3"]) + def tearDown(self): + base = self.cls(self.base) + base._rmtree() + def test_walk_topdown(self): - self.setUpWalk() walker = self.walk_path.walk() entry = next(walker) entry[1].sort() # Ensure we visit SUB1 before SUB2 @@ -2984,7 +2996,6 @@ def test_walk_topdown(self): next(walker) def test_walk_prune(self): - self.setUpWalk() # Prune the search. all = [] for root, dirs, files in self.walk_path.walk(): @@ -3001,7 +3012,6 @@ def test_walk_prune(self): self.assertEqual(all[1], self.sub2_tree) def test_walk_bottom_up(self): - self.setUpWalk() seen_testfn = seen_sub1 = seen_sub11 = seen_sub2 = False for path, dirnames, filenames in self.walk_path.walk(top_down=False): if path == self.walk_path: @@ -3036,7 +3046,6 @@ def test_walk_bottom_up(self): @needs_symlinks def test_walk_follow_symlinks(self): - self.setUpWalk() walk_it = self.walk_path.walk(follow_symlinks=True) for root, dirs, files in walk_it: if root == self.link_path: @@ -3048,7 +3057,6 @@ def test_walk_follow_symlinks(self): @needs_symlinks def test_walk_symlink_location(self): - self.setUpWalk() # Tests whether symlinks end up in filenames or dirnames depending # on the `follow_symlinks` argument. walk_it = self.walk_path.walk(follow_symlinks=False) @@ -3097,5 +3105,10 @@ class DummyPathWithSymlinksTest(DummyPathTest): can_symlink = True +class DummyPathWithSymlinksWalkTest(DummyPathWalkTest): + cls = DummyPathWithSymlinks + can_symlink = True + + if __name__ == "__main__": unittest.main() From a4d4c1ede21f9fa72280f4fc0f50212eecfac9ae Mon Sep 17 00:00:00 2001 From: Stephen Morton Date: Sat, 23 Nov 2024 18:36:48 -0800 Subject: [PATCH 07/24] gh-126662: harmonize naming for three namedtuple base classes in urllib.parse (GH-126663) harmonize naming for three namedtuple base classes in urllib.parse --- Lib/urllib/parse.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index 8d7631d5693ece..c412c729852272 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -247,11 +247,11 @@ def _hostinfo(self): return hostname, port -_DefragResultBase = namedtuple('DefragResult', 'url fragment') +_DefragResultBase = namedtuple('_DefragResultBase', 'url fragment') _SplitResultBase = namedtuple( - 'SplitResult', 'scheme netloc path query fragment') + '_SplitResultBase', 'scheme netloc path query fragment') _ParseResultBase = namedtuple( - 'ParseResult', 'scheme netloc path params query fragment') + '_ParseResultBase', 'scheme netloc path params query fragment') _DefragResultBase.__doc__ = """ DefragResult(url, fragment) From f7bb658124aba74be4c13f498bf46cfded710ef9 Mon Sep 17 00:00:00 2001 From: Sergey B Kirpichev Date: Sun, 24 Nov 2024 10:37:37 +0300 Subject: [PATCH 08/24] gh-113841: fix possible undefined division by 0 in _Py_c_pow() (GH-127211) `x**y == 1/x**-y ` thus changing `/=` to `*=` by negating the exponent. --- Lib/test/test_complex.py | 5 +++++ .../2024-11-24-07-01-28.gh-issue-113841.WFg-Bu.rst | 2 ++ Objects/complexobject.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2024-11-24-07-01-28.gh-issue-113841.WFg-Bu.rst diff --git a/Lib/test/test_complex.py b/Lib/test/test_complex.py index ecc97315e50d31..c51327c7f33a0a 100644 --- a/Lib/test/test_complex.py +++ b/Lib/test/test_complex.py @@ -338,6 +338,11 @@ def test_pow(self): except OverflowError: pass + # gh-113841: possible undefined division by 0 in _Py_c_pow() + x, y = 9j, 33j**3 + with self.assertRaises(OverflowError): + x**y + def test_pow_with_small_integer_exponents(self): # Check that small integer exponents are handled identically # regardless of their type. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-11-24-07-01-28.gh-issue-113841.WFg-Bu.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-11-24-07-01-28.gh-issue-113841.WFg-Bu.rst new file mode 100644 index 00000000000000..2b07fdfcc6b527 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-11-24-07-01-28.gh-issue-113841.WFg-Bu.rst @@ -0,0 +1,2 @@ +Fix possible undefined behavior division by zero in :class:`complex`'s +:c:func:`_Py_c_pow`. diff --git a/Objects/complexobject.c b/Objects/complexobject.c index 7b4948fc8ebe3d..9faa57519a424b 100644 --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -168,7 +168,7 @@ _Py_c_pow(Py_complex a, Py_complex b) at = atan2(a.imag, a.real); phase = at*b.real; if (b.imag != 0.0) { - len /= exp(at*b.imag); + len *= exp(-at*b.imag); phase += b.imag*log(vabs); } r.real = len*cos(phase); From 2104bde572aa60f547274fd529312c5708599001 Mon Sep 17 00:00:00 2001 From: Savannah Ostrowski Date: Sun, 24 Nov 2024 07:20:37 -0800 Subject: [PATCH 09/24] GH-127133: Remove ability to nest argument groups & mutually exclusive groups (#127186) --- Doc/library/argparse.rst | 19 ++--- Doc/whatsnew/3.14.rst | 8 ++ Lib/argparse.py | 20 +---- Lib/test/test_argparse.py | 83 ++++--------------- ...-11-23-04-54-42.gh-issue-127133.WMoJjF.rst | 6 ++ 5 files changed, 41 insertions(+), 95 deletions(-) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2024-11-23-04-54-42.gh-issue-127133.WMoJjF.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 410b6e11af0dc0..da4071dee34b8c 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1926,11 +1926,10 @@ Argument groups Note that any arguments not in your user-defined groups will end up back in the usual "positional arguments" and "optional arguments" sections. - .. versionchanged:: 3.11 - Calling :meth:`add_argument_group` on an argument group is deprecated. - This feature was never supported and does not always work correctly. - The function exists on the API by accident through inheritance and - will be removed in the future. + .. deprecated-removed:: 3.11 3.14 + Calling :meth:`add_argument_group` on an argument group now raises an + exception. This nesting was never supported, often failed to work + correctly, and was unintentionally exposed through inheritance. .. deprecated:: 3.14 Passing prefix_chars_ to :meth:`add_argument_group` @@ -1993,11 +1992,11 @@ Mutual exclusion --foo FOO foo help --bar BAR bar help - .. versionchanged:: 3.11 - Calling :meth:`add_argument_group` or :meth:`add_mutually_exclusive_group` - on a mutually exclusive group is deprecated. These features were never - supported and do not always work correctly. The functions exist on the - API by accident through inheritance and will be removed in the future. + .. deprecated-removed:: 3.11 3.14 + Calling :meth:`add_argument_group` or :meth:`add_mutually_exclusive_group` + on a mutually exclusive group now raises an exception. This nesting was + never supported, often failed to work correctly, and was unintentionally + exposed through inheritance. Parser defaults diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index bc4ab6789e1676..2c1acb832080fc 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -641,6 +641,14 @@ argparse of :class:`!argparse.BooleanOptionalAction`. They were deprecated since 3.12. +* Calling :meth:`~argparse.ArgumentParser.add_argument_group` on an argument + group, and calling :meth:`~argparse.ArgumentParser.add_argument_group` or + :meth:`~argparse.ArgumentParser.add_mutually_exclusive_group` on a mutually + exclusive group now raise exceptions. This nesting was never supported, + often failed to work correctly, and was unintentionally exposed through + inheritance. This functionality has been deprecated since Python 3.11. + (Contributed by Savannah Ostrowski in :gh:`127186`.) + ast --- diff --git a/Lib/argparse.py b/Lib/argparse.py index f5a7342c2fc355..d24fa72e573d4f 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -1709,14 +1709,7 @@ def _remove_action(self, action): self._group_actions.remove(action) def add_argument_group(self, *args, **kwargs): - import warnings - warnings.warn( - "Nesting argument groups is deprecated.", - category=DeprecationWarning, - stacklevel=2 - ) - return super().add_argument_group(*args, **kwargs) - + raise ValueError('argument groups cannot be nested') class _MutuallyExclusiveGroup(_ArgumentGroup): @@ -1737,15 +1730,8 @@ def _remove_action(self, action): self._container._remove_action(action) self._group_actions.remove(action) - def add_mutually_exclusive_group(self, *args, **kwargs): - import warnings - warnings.warn( - "Nesting mutually exclusive groups is deprecated.", - category=DeprecationWarning, - stacklevel=2 - ) - return super().add_mutually_exclusive_group(*args, **kwargs) - + def add_mutually_exclusive_group(self, **kwargs): + raise ValueError('mutually exclusive groups cannot be nested') def _prog_name(prog=None): if prog is not None: diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 69243fde5f0f98..488a3a4ed20fac 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -2997,6 +2997,13 @@ def test_group_prefix_chars_default(self): self.assertEqual(msg, str(cm.warning)) self.assertEqual(cm.filename, __file__) + def test_nested_argument_group(self): + parser = argparse.ArgumentParser() + g = parser.add_argument_group() + self.assertRaisesRegex(ValueError, + 'argument groups cannot be nested', + g.add_argument_group) + # =================== # Parent parser tests # =================== @@ -3297,6 +3304,14 @@ def test_empty_group(self): with self.assertRaises(ValueError): parser.parse_args(['-h']) + def test_nested_mutex_groups(self): + parser = argparse.ArgumentParser(prog='PROG') + g = parser.add_mutually_exclusive_group() + g.add_argument("--spam") + self.assertRaisesRegex(ValueError, + 'mutually exclusive groups cannot be nested', + g.add_mutually_exclusive_group) + class MEMixin(object): def test_failures_when_not_required(self): @@ -3664,55 +3679,6 @@ def get_parser(self, required): -c c help ''' -class TestMutuallyExclusiveNested(MEMixin, TestCase): - - # Nesting mutually exclusive groups is an undocumented feature - # that came about by accident through inheritance and has been - # the source of many bugs. It is deprecated and this test should - # eventually be removed along with it. - - def get_parser(self, required): - parser = ErrorRaisingArgumentParser(prog='PROG') - group = parser.add_mutually_exclusive_group(required=required) - group.add_argument('-a') - group.add_argument('-b') - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - group2 = group.add_mutually_exclusive_group(required=required) - group2.add_argument('-c') - group2.add_argument('-d') - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - group3 = group2.add_mutually_exclusive_group(required=required) - group3.add_argument('-e') - group3.add_argument('-f') - return parser - - usage_when_not_required = '''\ - usage: PROG [-h] [-a A | -b B | [-c C | -d D | [-e E | -f F]]] - ''' - usage_when_required = '''\ - usage: PROG [-h] (-a A | -b B | (-c C | -d D | (-e E | -f F))) - ''' - - help = '''\ - - options: - -h, --help show this help message and exit - -a A - -b B - -c C - -d D - -e E - -f F - ''' - - # We are only interested in testing the behavior of format_usage(). - test_failures_when_not_required = None - test_failures_when_required = None - test_successes_when_not_required = None - test_successes_when_required = None - class TestMutuallyExclusiveOptionalOptional(MEMixin, TestCase): def get_parser(self, required=None): @@ -4883,25 +4849,6 @@ def test_all_suppressed_mutex_with_optional_nargs(self): usage = 'usage: PROG [-h]\n' self.assertEqual(parser.format_usage(), usage) - def test_nested_mutex_groups(self): - parser = argparse.ArgumentParser(prog='PROG') - g = parser.add_mutually_exclusive_group() - g.add_argument("--spam") - with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) - gg = g.add_mutually_exclusive_group() - gg.add_argument("--hax") - gg.add_argument("--hox", help=argparse.SUPPRESS) - gg.add_argument("--hex") - g.add_argument("--eggs") - parser.add_argument("--num") - - usage = textwrap.dedent('''\ - usage: PROG [-h] [--spam SPAM | [--hax HAX | --hex HEX] | --eggs EGGS] - [--num NUM] - ''') - self.assertEqual(parser.format_usage(), usage) - def test_long_mutex_groups_wrap(self): parser = argparse.ArgumentParser(prog='PROG') g = parser.add_mutually_exclusive_group() diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-11-23-04-54-42.gh-issue-127133.WMoJjF.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-11-23-04-54-42.gh-issue-127133.WMoJjF.rst new file mode 100644 index 00000000000000..56b496bdf1e310 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-11-23-04-54-42.gh-issue-127133.WMoJjF.rst @@ -0,0 +1,6 @@ +Calling :meth:`argparse.ArgumentParser.add_argument_group` on an argument group, +and calling :meth:`argparse.ArgumentParser.add_argument_group` or +:meth:`argparse.ArgumentParser.add_mutually_exclusive_group` on a mutually +exclusive group now raise exceptions. This nesting was never supported, often +failed to work correctly, and was unintentionally exposed through inheritance. +This functionality has been deprecated since Python 3.11. From 3d8ac48aed6e27c43bf5d037018edcc31d9e66d8 Mon Sep 17 00:00:00 2001 From: Yuki Kobayashi Date: Mon, 25 Nov 2024 00:43:25 +0900 Subject: [PATCH 10/24] gh-101100: Fix sphinx warnings of removed opcodes (#127222) --- Doc/whatsnew/3.11.rst | 2 +- Doc/whatsnew/3.4.rst | 2 +- Doc/whatsnew/3.6.rst | 12 ++++++------ Doc/whatsnew/3.7.rst | 4 ++-- Doc/whatsnew/3.8.rst | 10 +++++----- Doc/whatsnew/3.9.rst | 2 +- 6 files changed, 16 insertions(+), 16 deletions(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index e5c6d7cd308504..ed41ecd50b0011 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1670,7 +1670,7 @@ Replaced opcodes | | | for each direction | | | | | +------------------------------------+------------------------------------+-----------------------------------------+ -| | :opcode:`!SETUP_WITH` | :opcode:`BEFORE_WITH` | :keyword:`with` block setup | +| | :opcode:`!SETUP_WITH` | :opcode:`!BEFORE_WITH` | :keyword:`with` block setup | | | :opcode:`!SETUP_ASYNC_WITH` | | | +------------------------------------+------------------------------------+-----------------------------------------+ diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst index 9d746b378995c3..71b186aeed7359 100644 --- a/Doc/whatsnew/3.4.rst +++ b/Doc/whatsnew/3.4.rst @@ -1979,7 +1979,7 @@ Other Improvements now works correctly (previously it silently returned the first python module in the file). (Contributed by Václav Šmilauer in :issue:`16421`.) -* A new opcode, :opcode:`LOAD_CLASSDEREF`, has been added to fix a bug in the +* A new opcode, :opcode:`!LOAD_CLASSDEREF`, has been added to fix a bug in the loading of free variables in class bodies that could be triggered by certain uses of :ref:`__prepare__ `. (Contributed by Benjamin Peterson in :issue:`17853`.) diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 2276fed60c8db3..1fcc5d7cbfb387 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -2366,27 +2366,27 @@ There have been several major changes to the :term:`bytecode` in Python 3.6. (Contributed by Demur Rumed with input and reviews from Serhiy Storchaka and Victor Stinner in :issue:`26647` and :issue:`28050`.) -* The new :opcode:`FORMAT_VALUE` and :opcode:`BUILD_STRING` opcodes as part +* The new :opcode:`!FORMAT_VALUE` and :opcode:`BUILD_STRING` opcodes as part of the :ref:`formatted string literal ` implementation. (Contributed by Eric Smith in :issue:`25483` and Serhiy Storchaka in :issue:`27078`.) -* The new :opcode:`BUILD_CONST_KEY_MAP` opcode to optimize the creation +* The new :opcode:`!BUILD_CONST_KEY_MAP` opcode to optimize the creation of dictionaries with constant keys. (Contributed by Serhiy Storchaka in :issue:`27140`.) * The function call opcodes have been heavily reworked for better performance and simpler implementation. - The :opcode:`MAKE_FUNCTION`, :opcode:`CALL_FUNCTION`, - :opcode:`CALL_FUNCTION_KW` and :opcode:`BUILD_MAP_UNPACK_WITH_CALL` opcodes + The :opcode:`MAKE_FUNCTION`, :opcode:`!CALL_FUNCTION`, + :opcode:`!CALL_FUNCTION_KW` and :opcode:`!BUILD_MAP_UNPACK_WITH_CALL` opcodes have been modified, the new :opcode:`CALL_FUNCTION_EX` and - :opcode:`BUILD_TUPLE_UNPACK_WITH_CALL` have been added, and + :opcode:`!BUILD_TUPLE_UNPACK_WITH_CALL` have been added, and ``CALL_FUNCTION_VAR``, ``CALL_FUNCTION_VAR_KW`` and ``MAKE_CLOSURE`` opcodes have been removed. (Contributed by Demur Rumed in :issue:`27095`, and Serhiy Storchaka in :issue:`27213`, :issue:`28257`.) -* The new :opcode:`SETUP_ANNOTATIONS` and :opcode:`STORE_ANNOTATION` opcodes +* The new :opcode:`SETUP_ANNOTATIONS` and :opcode:`!STORE_ANNOTATION` opcodes have been added to support the new :term:`variable annotation` syntax. (Contributed by Ivan Levkivskyi in :issue:`27985`.) diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 2d433ef4759d52..f420fa5c04479b 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -2476,10 +2476,10 @@ avoiding possible problems use new functions :c:func:`PySlice_Unpack` and CPython bytecode changes ------------------------ -There are two new opcodes: :opcode:`LOAD_METHOD` and :opcode:`CALL_METHOD`. +There are two new opcodes: :opcode:`LOAD_METHOD` and :opcode:`!CALL_METHOD`. (Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) -The :opcode:`STORE_ANNOTATION` opcode has been removed. +The :opcode:`!STORE_ANNOTATION` opcode has been removed. (Contributed by Mark Shannon in :issue:`32550`.) diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst index bdc4ca5cab5245..7aca35b2959cd2 100644 --- a/Doc/whatsnew/3.8.rst +++ b/Doc/whatsnew/3.8.rst @@ -2152,11 +2152,11 @@ CPython bytecode changes cleaning-up code for :keyword:`break`, :keyword:`continue` and :keyword:`return`. - Removed opcodes :opcode:`BREAK_LOOP`, :opcode:`CONTINUE_LOOP`, - :opcode:`SETUP_LOOP` and :opcode:`SETUP_EXCEPT`. Added new opcodes - :opcode:`ROT_FOUR`, :opcode:`BEGIN_FINALLY`, :opcode:`CALL_FINALLY` and - :opcode:`POP_FINALLY`. Changed the behavior of :opcode:`END_FINALLY` - and :opcode:`WITH_CLEANUP_START`. + Removed opcodes :opcode:`!BREAK_LOOP`, :opcode:`!CONTINUE_LOOP`, + :opcode:`!SETUP_LOOP` and :opcode:`!SETUP_EXCEPT`. Added new opcodes + :opcode:`!ROT_FOUR`, :opcode:`!BEGIN_FINALLY`, :opcode:`!CALL_FINALLY` and + :opcode:`!POP_FINALLY`. Changed the behavior of :opcode:`!END_FINALLY` + and :opcode:`!WITH_CLEANUP_START`. (Contributed by Mark Shannon, Antoine Pitrou and Serhiy Storchaka in :issue:`17611`.) diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst index 6118b02dd9bd48..b062e6b4c9bca0 100644 --- a/Doc/whatsnew/3.9.rst +++ b/Doc/whatsnew/3.9.rst @@ -1203,7 +1203,7 @@ Changes in the C API CPython bytecode changes ------------------------ -* The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the +* The :opcode:`!LOAD_ASSERTION_ERROR` opcode was added for handling the :keyword:`assert` statement. Previously, the assert statement would not work correctly if the :exc:`AssertionError` exception was being shadowed. (Contributed by Zackery Spytz in :issue:`34880`.) From e0ef08f5b444950ad9e900b27f5b5dbc706f4459 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?B=C3=A9n=C3=A9dikt=20Tran?= <10796600+picnixz@users.noreply.github.com> Date: Sun, 24 Nov 2024 17:36:15 +0100 Subject: [PATCH 11/24] gh-122356: restore the position of a file-like object after `zipfile.is_zipfile` (#122397) --- Lib/test/test_zipfile/test_core.py | 10 ++++++++-- Lib/zipfile/__init__.py | 2 ++ .../2024-07-29-15-20-30.gh-issue-122356.wKCmFx.rst | 3 +++ 3 files changed, 13 insertions(+), 2 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2024-07-29-15-20-30.gh-issue-122356.wKCmFx.rst diff --git a/Lib/test/test_zipfile/test_core.py b/Lib/test/test_zipfile/test_core.py index 36f7f542872897..c36228c033a414 100644 --- a/Lib/test/test_zipfile/test_core.py +++ b/Lib/test/test_zipfile/test_core.py @@ -1969,10 +1969,16 @@ def test_is_zip_valid_file(self): zip_contents = fp.read() # - passing a file-like object fp = io.BytesIO() - fp.write(zip_contents) + end = fp.write(zip_contents) + self.assertEqual(fp.tell(), end) + mid = end // 2 + fp.seek(mid, 0) self.assertTrue(zipfile.is_zipfile(fp)) - fp.seek(0, 0) + # check that the position is left unchanged after the call + # see: https://github.com/python/cpython/issues/122356 + self.assertEqual(fp.tell(), mid) self.assertTrue(zipfile.is_zipfile(fp)) + self.assertEqual(fp.tell(), mid) def test_non_existent_file_raises_OSError(self): # make sure we don't raise an AttributeError when a partially-constructed diff --git a/Lib/zipfile/__init__.py b/Lib/zipfile/__init__.py index 08c83cfb760250..6907ae6d5b7464 100644 --- a/Lib/zipfile/__init__.py +++ b/Lib/zipfile/__init__.py @@ -241,7 +241,9 @@ def is_zipfile(filename): result = False try: if hasattr(filename, "read"): + pos = filename.tell() result = _check_zipfile(fp=filename) + filename.seek(pos) else: with open(filename, "rb") as fp: result = _check_zipfile(fp) diff --git a/Misc/NEWS.d/next/Library/2024-07-29-15-20-30.gh-issue-122356.wKCmFx.rst b/Misc/NEWS.d/next/Library/2024-07-29-15-20-30.gh-issue-122356.wKCmFx.rst new file mode 100644 index 00000000000000..0a4632ca975f6b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-07-29-15-20-30.gh-issue-122356.wKCmFx.rst @@ -0,0 +1,3 @@ +Guarantee that the position of a file-like object passed to +:func:`zipfile.is_zipfile` is left untouched after the call. +Patch by Bénédikt Tran. From 2bb7846cacb342246aada5ed92d323e54c946063 Mon Sep 17 00:00:00 2001 From: da-woods Date: Sun, 24 Nov 2024 17:21:02 +0000 Subject: [PATCH 12/24] Fix macro expansions in critical section docs (#127226) --- Doc/c-api/init.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 970084ce91ab69..ba1c2852f0bd53 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -2470,7 +2470,7 @@ code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`. { PyCriticalSection2 _py_cs2; - PyCriticalSection_Begin2(&_py_cs2, (PyObject*)(a), (PyObject*)(b)) + PyCriticalSection2_Begin(&_py_cs2, (PyObject*)(a), (PyObject*)(b)) In the default build, this macro expands to ``{``. @@ -2482,7 +2482,7 @@ code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`. In the free-threaded build, this macro expands to:: - PyCriticalSection_End2(&_py_cs2); + PyCriticalSection2_End(&_py_cs2); } In the default build, this macro expands to ``}``. From 97b2ceaaaf88a73a45254912a0e972412879ccbf Mon Sep 17 00:00:00 2001 From: Serhiy Storchaka Date: Sun, 24 Nov 2024 19:30:29 +0200 Subject: [PATCH 13/24] gh-127217: Fix pathname2url() for paths starting with multiple slashes on Posix (GH-127218) --- Lib/test/test_urllib.py | 3 +++ Lib/urllib/request.py | 4 ++++ .../Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst | 2 ++ 3 files changed, 9 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py index 22ef3c648e271d..fe16badc5bc77d 100644 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -1458,6 +1458,9 @@ def test_pathname2url_posix(self): fn = urllib.request.pathname2url self.assertEqual(fn('/'), '/') self.assertEqual(fn('/a/b.c'), '/a/b.c') + self.assertEqual(fn('//a/b.c'), '////a/b.c') + self.assertEqual(fn('///a/b.c'), '/////a/b.c') + self.assertEqual(fn('////a/b.c'), '//////a/b.c') self.assertEqual(fn('/a/b%#c'), '/a/b%25%23c') @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index 80be65c613e971..9e555432688a5b 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -1667,6 +1667,10 @@ def url2pathname(pathname): def pathname2url(pathname): """OS-specific conversion from a file system path to a relative URL of the 'file' scheme; not recommended for general use.""" + if pathname[:2] == '//': + # Add explicitly empty authority to avoid interpreting the path + # as authority. + pathname = '//' + pathname encoding = sys.getfilesystemencoding() errors = sys.getfilesystemencodeerrors() return quote(pathname, encoding=encoding, errors=errors) diff --git a/Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst b/Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst new file mode 100644 index 00000000000000..3139e33302f378 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-24-12-41-31.gh-issue-127217.UAXGFr.rst @@ -0,0 +1,2 @@ +Fix :func:`urllib.request.pathname2url` for paths starting with multiple +slashes on Posix. From 307c63358681d669ae39e5ecd814bded4a93443a Mon Sep 17 00:00:00 2001 From: Barney Gale Date: Sun, 24 Nov 2024 17:33:46 +0000 Subject: [PATCH 14/24] Improve `pathname2url()` and `url2pathname()` docs (#127125) These functions have long sown confusion among Python developers. The existing documentation says they deal with URL path components, but that doesn't fit the evidence on Windows: >>> pathname2url(r'C:\foo') '///C:/foo' >>> pathname2url(r'\\server\share') '////server/share' # or '//server/share' as of quite recently If these were URL path components, they would imply complete URLs like `file://///C:/foo` and `file://////server/share`. Clearly this isn't right. Yet the implementation in `nturl2path` is deliberate, and the `url2pathname()` function correctly inverts it. On non-Windows platforms, the behaviour until quite recently is to simply quote/unquote the path without adding or removing any leading slashes. This behaviour is compatible with *both* interpretations -- 1) the value is a URL path component (existing docs), and 2) the value is everything following `file:` (this commit) The conclusion I draw is that these functions operate on everything after the `file:` prefix, which may include an authority section. This is the only explanation that fits both the Windows and non-Windows behaviour. It's also a better match for the function names. --- Doc/library/urllib.request.rst | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index a093a5083e037b..9055556a3703bb 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -148,9 +148,15 @@ The :mod:`urllib.request` module defines the following functions: .. function:: pathname2url(path) - Convert the pathname *path* from the local syntax for a path to the form used in - the path component of a URL. This does not produce a complete URL. The return - value will already be quoted using the :func:`~urllib.parse.quote` function. + Convert the given local path to a ``file:`` URL. This function uses + :func:`~urllib.parse.quote` function to encode the path. For historical + reasons, the return value omits the ``file:`` scheme prefix. This example + shows the function being used on Windows:: + + >>> from urllib.request import pathname2url + >>> path = 'C:\\Program Files' + >>> 'file:' + pathname2url(path) + 'file:///C:/Program%20Files' .. versionchanged:: 3.14 Windows drive letters are no longer converted to uppercase. @@ -161,11 +167,17 @@ The :mod:`urllib.request` module defines the following functions: found in any position other than the second character. -.. function:: url2pathname(path) +.. function:: url2pathname(url) + + Convert the given ``file:`` URL to a local path. This function uses + :func:`~urllib.parse.unquote` to decode the URL. For historical reasons, + the given value *must* omit the ``file:`` scheme prefix. This example shows + the function being used on Windows:: - Convert the path component *path* from a percent-encoded URL to the local syntax for a - path. This does not accept a complete URL. This function uses - :func:`~urllib.parse.unquote` to decode *path*. + >>> from urllib.request import url2pathname + >>> url = 'file:///C:/Program%20Files' + >>> url2pathname(url.removeprefix('file:')) + 'C:\\Program Files' .. versionchanged:: 3.14 Windows drive letters are no longer converted to uppercase. From 17c16aea66b606d66f71ae9af381bc34d0ef3f5f Mon Sep 17 00:00:00 2001 From: Brandt Bucher Date: Sun, 24 Nov 2024 14:42:50 -0800 Subject: [PATCH 15/24] GH-115869: Make jit_stencils.h reproducible (GH-127166) --- ...-11-22-08-46-46.gh-issue-115869.UVLSKd.rst | 1 + Tools/jit/_stencils.py | 3 ++- Tools/jit/_targets.py | 19 +++++++++++++------ Tools/jit/_writer.py | 2 +- Tools/jit/build.py | 2 +- 5 files changed, 18 insertions(+), 9 deletions(-) create mode 100644 Misc/NEWS.d/next/Build/2024-11-22-08-46-46.gh-issue-115869.UVLSKd.rst diff --git a/Misc/NEWS.d/next/Build/2024-11-22-08-46-46.gh-issue-115869.UVLSKd.rst b/Misc/NEWS.d/next/Build/2024-11-22-08-46-46.gh-issue-115869.UVLSKd.rst new file mode 100644 index 00000000000000..9e8a078983f20b --- /dev/null +++ b/Misc/NEWS.d/next/Build/2024-11-22-08-46-46.gh-issue-115869.UVLSKd.rst @@ -0,0 +1 @@ +Make ``jit_stencils.h`` (which is produced during JIT builds) reproducible. diff --git a/Tools/jit/_stencils.py b/Tools/jit/_stencils.py index 61be8fd3bbdf55..ee761a73fa808a 100644 --- a/Tools/jit/_stencils.py +++ b/Tools/jit/_stencils.py @@ -202,7 +202,8 @@ def pad(self, alignment: int) -> None: """Pad the stencil to the given alignment.""" offset = len(self.body) padding = -offset % alignment - self.disassembly.append(f"{offset:x}: {' '.join(['00'] * padding)}") + if padding: + self.disassembly.append(f"{offset:x}: {' '.join(['00'] * padding)}") self.body.extend([0] * padding) def remove_jump(self, *, alignment: int = 1) -> None: diff --git a/Tools/jit/_targets.py b/Tools/jit/_targets.py index d8dce0a905c0f8..d23ced19842347 100644 --- a/Tools/jit/_targets.py +++ b/Tools/jit/_targets.py @@ -61,10 +61,11 @@ async def _parse(self, path: pathlib.Path) -> _stencils.StencilGroup: args = ["--disassemble", "--reloc", f"{path}"] output = await _llvm.maybe_run("llvm-objdump", args, echo=self.verbose) if output is not None: + # Make sure that full paths don't leak out (for reproducibility): + long, short = str(path), str(path.name) group.code.disassembly.extend( - line.expandtabs().strip() + line.expandtabs().strip().replace(long, short) for line in output.splitlines() - if not line.isspace() ) args = [ "--elf-output-style=JSON", @@ -90,9 +91,6 @@ async def _parse(self, path: pathlib.Path) -> _stencils.StencilGroup: if group.data.body: line = f"0: {str(bytes(group.data.body)).removeprefix('b')}" group.data.disassembly.append(line) - group.process_relocations( - known_symbols=self.known_symbols, alignment=self.alignment - ) return group def _handle_section(self, section: _S, group: _stencils.StencilGroup) -> None: @@ -122,6 +120,10 @@ async def _compile( f"-I{CPYTHON / 'Tools' / 'jit'}", "-O3", "-c", + # Shorten full absolute file paths in the generated code (like the + # __FILE__ macro and assert failure messages) for reproducibility: + f"-ffile-prefix-map={CPYTHON}=.", + f"-ffile-prefix-map={tempdir}=.", # This debug info isn't necessary, and bloats out the JIT'ed code. # We *may* be able to re-enable this, process it, and JIT it for a # nicer debugging experience... but that needs a lot more research: @@ -167,7 +169,12 @@ async def _build_stencils(self) -> dict[str, _stencils.StencilGroup]: c.write_text(template.replace("CASE", case)) coro = self._compile(opname, c, work) tasks.append(group.create_task(coro, name=opname)) - return {task.get_name(): task.result() for task in tasks} + stencil_groups = {task.get_name(): task.result() for task in tasks} + for stencil_group in stencil_groups.values(): + stencil_group.process_relocations( + known_symbols=self.known_symbols, alignment=self.alignment + ) + return stencil_groups def build( self, out: pathlib.Path, *, comment: str = "", force: bool = False diff --git a/Tools/jit/_writer.py b/Tools/jit/_writer.py index 81a9f08db31703..5588784544ee00 100644 --- a/Tools/jit/_writer.py +++ b/Tools/jit/_writer.py @@ -77,6 +77,6 @@ def dump( groups: dict[str, _stencils.StencilGroup], symbols: dict[str, int] ) -> typing.Iterator[str]: """Yield a JIT compiler line-by-line as a C header file.""" - for opname, group in sorted(groups.items()): + for opname, group in groups.items(): yield from _dump_stencil(opname, group) yield from _dump_footer(groups, symbols) diff --git a/Tools/jit/build.py b/Tools/jit/build.py index 4a23c6f0afa74a..a8cb0f67c36363 100644 --- a/Tools/jit/build.py +++ b/Tools/jit/build.py @@ -8,7 +8,7 @@ import _targets if __name__ == "__main__": - comment = f"$ {shlex.join([sys.executable] + sys.argv)}" + comment = f"$ {shlex.join([pathlib.Path(sys.executable).name] + sys.argv)}" parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "target", type=_targets.get_target, help="a PEP 11 target triple to compile for" From c595eae84c7f665eced09ce67a1ad83b7574d6e5 Mon Sep 17 00:00:00 2001 From: Sergey B Kirpichev Date: Mon, 25 Nov 2024 08:29:55 +0300 Subject: [PATCH 16/24] gh-127238: adjust error message for sys.set_int_max_str_digits() (#127241) Now it's correct and closer to Python/initconfig.c --- .../2024-11-25-05-15-21.gh-issue-127238.O8wkH-.rst | 1 + Python/sysmodule.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2024-11-25-05-15-21.gh-issue-127238.O8wkH-.rst diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-11-25-05-15-21.gh-issue-127238.O8wkH-.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-11-25-05-15-21.gh-issue-127238.O8wkH-.rst new file mode 100644 index 00000000000000..e8a274fcd31f26 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-11-25-05-15-21.gh-issue-127238.O8wkH-.rst @@ -0,0 +1 @@ +Correct error message for :func:`sys.set_int_max_str_digits`. diff --git a/Python/sysmodule.c b/Python/sysmodule.c index aaef5aa532412b..6df297f364c5d3 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -4104,7 +4104,7 @@ _PySys_SetIntMaxStrDigits(int maxdigits) { if (maxdigits != 0 && maxdigits < _PY_LONG_MAX_STR_DIGITS_THRESHOLD) { PyErr_Format( - PyExc_ValueError, "maxdigits must be 0 or larger than %d", + PyExc_ValueError, "maxdigits must be >= %d or 0 for unlimited", _PY_LONG_MAX_STR_DIGITS_THRESHOLD); return -1; } From f4f075b3d5a5b0bc1b13cc27ef2a7de8c103fd04 Mon Sep 17 00:00:00 2001 From: Jun Komoda <45822440+junkmd@users.noreply.github.com> Date: Mon, 25 Nov 2024 21:53:17 +0900 Subject: [PATCH 17/24] Replace `:platform:` with `.. availability::` in `socket.ioctl` doc. (GH-127122) In `socket.ioctl`, `:platform:` -> `.. availability::` --- Doc/library/socket.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 73d495c055ff6e..58323ba6514eac 100644 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -1596,8 +1596,6 @@ to sockets. .. method:: socket.ioctl(control, option) - :platform: Windows - The :meth:`ioctl` method is a limited interface to the WSAIoctl system interface. Please refer to the `Win32 documentation `_ for more @@ -1609,9 +1607,12 @@ to sockets. Currently only the following control codes are supported: ``SIO_RCVALL``, ``SIO_KEEPALIVE_VALS``, and ``SIO_LOOPBACK_FAST_PATH``. + .. availability:: Windows + .. versionchanged:: 3.6 ``SIO_LOOPBACK_FAST_PATH`` was added. + .. method:: socket.listen([backlog]) Enable a server to accept connections. If *backlog* is specified, it must From c7f1e3e150ca181f4b4bd1e5b59d492749f00be6 Mon Sep 17 00:00:00 2001 From: Jun Komoda <45822440+junkmd@users.noreply.github.com> Date: Mon, 25 Nov 2024 22:55:07 +0900 Subject: [PATCH 18/24] gh-127183: Add `_ctypes.CopyComPointer` tests (GH-127184) * Make `create_shelllink_persist` top level function. * Add `CopyComPointerTests`. * Add more tests. * Update tests. * Add assertions for `Release`'s return value. --- .../test_win32_com_foreign_func.py | 132 +++++++++++++++--- 1 file changed, 115 insertions(+), 17 deletions(-) diff --git a/Lib/test/test_ctypes/test_win32_com_foreign_func.py b/Lib/test/test_ctypes/test_win32_com_foreign_func.py index 651c9277d59af9..8d217fc17efa02 100644 --- a/Lib/test/test_ctypes/test_win32_com_foreign_func.py +++ b/Lib/test/test_ctypes/test_win32_com_foreign_func.py @@ -9,7 +9,7 @@ raise unittest.SkipTest("Windows-specific test") -from _ctypes import COMError +from _ctypes import COMError, CopyComPointer from ctypes import HRESULT @@ -78,6 +78,19 @@ def is_equal_guid(guid1, guid2): ) +def create_shelllink_persist(typ): + ppst = typ() + # https://learn.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-cocreateinstance + ole32.CoCreateInstance( + byref(CLSID_ShellLink), + None, + CLSCTX_SERVER, + byref(IID_IPersist), + byref(ppst), + ) + return ppst + + class ForeignFunctionsThatWillCallComMethodsTests(unittest.TestCase): def setUp(self): # https://learn.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-coinitializeex @@ -88,19 +101,6 @@ def tearDown(self): ole32.CoUninitialize() gc.collect() - @staticmethod - def create_shelllink_persist(typ): - ppst = typ() - # https://learn.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-cocreateinstance - ole32.CoCreateInstance( - byref(CLSID_ShellLink), - None, - CLSCTX_SERVER, - byref(IID_IPersist), - byref(ppst), - ) - return ppst - def test_without_paramflags_and_iid(self): class IUnknown(c_void_p): QueryInterface = proto_query_interface() @@ -110,7 +110,7 @@ class IUnknown(c_void_p): class IPersist(IUnknown): GetClassID = proto_get_class_id() - ppst = self.create_shelllink_persist(IPersist) + ppst = create_shelllink_persist(IPersist) clsid = GUID() hr_getclsid = ppst.GetClassID(byref(clsid)) @@ -142,7 +142,7 @@ class IUnknown(c_void_p): class IPersist(IUnknown): GetClassID = proto_get_class_id(((OUT, "pClassID"),)) - ppst = self.create_shelllink_persist(IPersist) + ppst = create_shelllink_persist(IPersist) clsid = ppst.GetClassID() self.assertEqual(TRUE, is_equal_guid(CLSID_ShellLink, clsid)) @@ -167,7 +167,7 @@ class IUnknown(c_void_p): class IPersist(IUnknown): GetClassID = proto_get_class_id(((OUT, "pClassID"),), IID_IPersist) - ppst = self.create_shelllink_persist(IPersist) + ppst = create_shelllink_persist(IPersist) clsid = ppst.GetClassID() self.assertEqual(TRUE, is_equal_guid(CLSID_ShellLink, clsid)) @@ -184,5 +184,103 @@ class IPersist(IUnknown): self.assertEqual(0, ppst.Release()) +class CopyComPointerTests(unittest.TestCase): + def setUp(self): + ole32.CoInitializeEx(None, COINIT_APARTMENTTHREADED) + + class IUnknown(c_void_p): + QueryInterface = proto_query_interface(None, IID_IUnknown) + AddRef = proto_add_ref() + Release = proto_release() + + class IPersist(IUnknown): + GetClassID = proto_get_class_id(((OUT, "pClassID"),), IID_IPersist) + + self.IUnknown = IUnknown + self.IPersist = IPersist + + def tearDown(self): + ole32.CoUninitialize() + gc.collect() + + def test_both_are_null(self): + src = self.IPersist() + dst = self.IPersist() + + hr = CopyComPointer(src, byref(dst)) + + self.assertEqual(S_OK, hr) + + self.assertIsNone(src.value) + self.assertIsNone(dst.value) + + def test_src_is_nonnull_and_dest_is_null(self): + # The reference count of the COM pointer created by `CoCreateInstance` + # is initially 1. + src = create_shelllink_persist(self.IPersist) + dst = self.IPersist() + + # `CopyComPointer` calls `AddRef` explicitly in the C implementation. + # The refcount of `src` is incremented from 1 to 2 here. + hr = CopyComPointer(src, byref(dst)) + + self.assertEqual(S_OK, hr) + self.assertEqual(src.value, dst.value) + + # This indicates that the refcount was 2 before the `Release` call. + self.assertEqual(1, src.Release()) + + clsid = dst.GetClassID() + self.assertEqual(TRUE, is_equal_guid(CLSID_ShellLink, clsid)) + + self.assertEqual(0, dst.Release()) + + def test_src_is_null_and_dest_is_nonnull(self): + src = self.IPersist() + dst_orig = create_shelllink_persist(self.IPersist) + dst = self.IPersist() + CopyComPointer(dst_orig, byref(dst)) + self.assertEqual(1, dst_orig.Release()) + + clsid = dst.GetClassID() + self.assertEqual(TRUE, is_equal_guid(CLSID_ShellLink, clsid)) + + # This does NOT affects the refcount of `dst_orig`. + hr = CopyComPointer(src, byref(dst)) + + self.assertEqual(S_OK, hr) + self.assertIsNone(dst.value) + + with self.assertRaises(ValueError): + dst.GetClassID() # NULL COM pointer access + + # This indicates that the refcount was 1 before the `Release` call. + self.assertEqual(0, dst_orig.Release()) + + def test_both_are_nonnull(self): + src = create_shelllink_persist(self.IPersist) + dst_orig = create_shelllink_persist(self.IPersist) + dst = self.IPersist() + CopyComPointer(dst_orig, byref(dst)) + self.assertEqual(1, dst_orig.Release()) + + self.assertEqual(dst.value, dst_orig.value) + self.assertNotEqual(src.value, dst.value) + + hr = CopyComPointer(src, byref(dst)) + + self.assertEqual(S_OK, hr) + self.assertEqual(src.value, dst.value) + self.assertNotEqual(dst.value, dst_orig.value) + + self.assertEqual(1, src.Release()) + + clsid = dst.GetClassID() + self.assertEqual(TRUE, is_equal_guid(CLSID_ShellLink, clsid)) + + self.assertEqual(0, dst.Release()) + self.assertEqual(0, dst_orig.Release()) + + if __name__ == '__main__': unittest.main() From d9331f1b16033796a3958f6e0b12626ed7d3e199 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Mon, 25 Nov 2024 15:24:33 +0100 Subject: [PATCH 19/24] gh-107954: Document PEP 741 in What's New 3.14 (#127056) --- Doc/c-api/init_config.rst | 4 ++++ Doc/whatsnew/3.14.rst | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 6194d7446c73e4..e621545cffcee3 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -6,6 +6,8 @@ Python Initialization Configuration *********************************** +.. _pyconfig_api: + PyConfig C API ============== @@ -1592,6 +1594,8 @@ The ``__PYVENV_LAUNCHER__`` environment variable is used to set :c:member:`PyConfig.base_executable`. +.. _pyinitconfig_api: + PyInitConfig C API ================== diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index 2c1acb832080fc..0e4b9eb0cf0b9c 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -65,6 +65,8 @@ Summary -- release highlights .. PEP-sized items next. +* :ref:`PEP 649: deferred evaluation of annotations ` +* :ref:`PEP 741: Python Configuration C API ` New features @@ -172,6 +174,40 @@ Improved error messages ValueError: too many values to unpack (expected 3, got 4) +.. _whatsnew314-pep741: + +PEP 741: Python Configuration C API +----------------------------------- + +Add a :ref:`PyInitConfig C API ` to configure the Python +initialization without relying on C structures and the ability to make +ABI-compatible changes in the future. + +Complete the :pep:`587` :ref:`PyConfig C API ` by adding +:c:func:`PyInitConfig_AddModule` which can be used to add a built-in extension +module; feature previously referred to as the “inittab”. + +Add :c:func:`PyConfig_Get` and :c:func:`PyConfig_Set` functions to get and set +the current runtime configuration. + +PEP 587 “Python Initialization Configuration” unified all the ways to configure +the Python initialization. This PEP unifies also the configuration of the +Python preinitialization and the Python initialization in a single API. +Moreover, this PEP only provides a single choice to embed Python, instead of +having two “Python” and “Isolated” choices (PEP 587), to simplify the API +further. + +The lower level PEP 587 PyConfig API remains available for use cases with an +intentionally higher level of coupling to CPython implementation details (such +as emulating the full functionality of CPython’s CLI, including its +configuration mechanisms). + +(Contributed by Victor Stinner in :gh:`107954`.) + +.. seealso:: + :pep:`741`. + + Other language changes ====================== From 3e7ce6e9aed8616c8ce53eaef4279402d3ee38ec Mon Sep 17 00:00:00 2001 From: Sergey Muraviov Date: Mon, 25 Nov 2024 17:56:46 +0300 Subject: [PATCH 20/24] gh-127248: Fix several removed or misnamed files in pythoncore.vcxproj (GH-127249) --- PCbuild/pythoncore.vcxproj | 7 +------ PCbuild/pythoncore.vcxproj.filters | 14 +------------- 2 files changed, 2 insertions(+), 19 deletions(-) diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 95552cade52b75..9ebf58ae8a9bc4 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -169,9 +169,7 @@ - - @@ -220,7 +218,6 @@ - @@ -366,14 +363,12 @@ - - @@ -415,7 +410,7 @@ - + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 1708cf6e0b3a52..6c76a6ab592a84 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -231,9 +231,6 @@ Include - - Include - Include @@ -342,9 +339,6 @@ Include - - Modules - Parser @@ -447,15 +441,9 @@ Include\cpython - - Include - Include - - Include - Include @@ -903,7 +891,7 @@ Modules\zlib - + Modules\zlib From d3da04bfc91ec065fe587451409102213af0e57c Mon Sep 17 00:00:00 2001 From: Sam Gross Date: Mon, 25 Nov 2024 17:24:37 +0000 Subject: [PATCH 21/24] gh-127022: Remove `_PyEvalFramePushAndInit_UnTagged` (gh-127168) The interpreter now handles `_PyStackRef`s pointing to immortal objects without the deferred bit set, so `_PyEvalFramePushAndInit_UnTagged` is no longer necessary. --- Python/ceval.c | 35 ++++------------------------------- 1 file changed, 4 insertions(+), 31 deletions(-) diff --git a/Python/ceval.c b/Python/ceval.c index 2a3938572c1569..eba0f233a81ef3 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1800,33 +1800,6 @@ _PyEvalFramePushAndInit(PyThreadState *tstate, _PyStackRef func, return NULL; } -static _PyInterpreterFrame * -_PyEvalFramePushAndInit_UnTagged(PyThreadState *tstate, _PyStackRef func, - PyObject *locals, PyObject *const* args, - size_t argcount, PyObject *kwnames, _PyInterpreterFrame *previous) -{ -#if defined(Py_GIL_DISABLED) - size_t kw_count = kwnames == NULL ? 0 : PyTuple_GET_SIZE(kwnames); - size_t total_argcount = argcount + kw_count; - _PyStackRef *tagged_args_buffer = PyMem_Malloc(sizeof(_PyStackRef) * total_argcount); - if (tagged_args_buffer == NULL) { - PyErr_NoMemory(); - return NULL; - } - for (size_t i = 0; i < argcount; i++) { - tagged_args_buffer[i] = PyStackRef_FromPyObjectSteal(args[i]); - } - for (size_t i = 0; i < kw_count; i++) { - tagged_args_buffer[argcount + i] = PyStackRef_FromPyObjectSteal(args[argcount + i]); - } - _PyInterpreterFrame *res = _PyEvalFramePushAndInit(tstate, func, locals, (_PyStackRef const *)tagged_args_buffer, argcount, kwnames, previous); - PyMem_Free(tagged_args_buffer); - return res; -#else - return _PyEvalFramePushAndInit(tstate, func, locals, (_PyStackRef const *)args, argcount, kwnames, previous); -#endif -} - /* Same as _PyEvalFramePushAndInit but takes an args tuple and kwargs dict. Steals references to func, callargs and kwargs. */ @@ -1851,9 +1824,9 @@ _PyEvalFramePushAndInit_Ex(PyThreadState *tstate, _PyStackRef func, Py_INCREF(PyTuple_GET_ITEM(callargs, i)); } } - _PyInterpreterFrame *new_frame = _PyEvalFramePushAndInit_UnTagged( + _PyInterpreterFrame *new_frame = _PyEvalFramePushAndInit( tstate, func, locals, - newargs, nargs, kwnames, previous + (_PyStackRef const *)newargs, nargs, kwnames, previous ); if (has_dict) { _PyStack_UnpackDict_FreeNoDecRef(newargs, kwnames); @@ -1888,9 +1861,9 @@ _PyEval_Vector(PyThreadState *tstate, PyFunctionObject *func, Py_INCREF(args[i+argcount]); } } - _PyInterpreterFrame *frame = _PyEvalFramePushAndInit_UnTagged( + _PyInterpreterFrame *frame = _PyEvalFramePushAndInit( tstate, PyStackRef_FromPyObjectNew(func), locals, - args, argcount, kwnames, NULL); + (_PyStackRef const *)args, argcount, kwnames, NULL); if (frame == NULL) { return NULL; } From a2ee89968299fc4f0da4b5a4165025b941213ba5 Mon Sep 17 00:00:00 2001 From: sobolevn Date: Mon, 25 Nov 2024 20:32:02 +0300 Subject: [PATCH 22/24] gh-127182: Fix `io.StringIO.__setstate__` crash when `None` is the first value (#127219) Co-authored-by: Victor Stinner --- Lib/test/test_io.py | 15 ++++++++++ ...-11-24-14-20-17.gh-issue-127182.WmfY2g.rst | 2 ++ Modules/_io/stringio.c | 30 ++++++++++--------- 3 files changed, 33 insertions(+), 14 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2024-11-24-14-20-17.gh-issue-127182.WmfY2g.rst diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index aa1b8268592ff7..f1f8ce57668f3b 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -1148,6 +1148,21 @@ def test_disallow_instantiation(self): _io = self._io support.check_disallow_instantiation(self, _io._BytesIOBuffer) + def test_stringio_setstate(self): + # gh-127182: Calling __setstate__() with invalid arguments must not crash + obj = self._io.StringIO() + with self.assertRaisesRegex( + TypeError, + 'initial_value must be str or None, not int', + ): + obj.__setstate__((1, '', 0, {})) + + obj.__setstate__((None, '', 0, {})) # should not crash + self.assertEqual(obj.getvalue(), '') + + obj.__setstate__(('', '', 0, {})) + self.assertEqual(obj.getvalue(), '') + class PyIOTest(IOTest): pass diff --git a/Misc/NEWS.d/next/Library/2024-11-24-14-20-17.gh-issue-127182.WmfY2g.rst b/Misc/NEWS.d/next/Library/2024-11-24-14-20-17.gh-issue-127182.WmfY2g.rst new file mode 100644 index 00000000000000..2cc46ca3d33977 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-24-14-20-17.gh-issue-127182.WmfY2g.rst @@ -0,0 +1,2 @@ +Fix :meth:`!io.StringIO.__setstate__` crash, when :const:`None` was passed as +the first value. diff --git a/Modules/_io/stringio.c b/Modules/_io/stringio.c index f558613dc6233c..65e8d97aa8ac19 100644 --- a/Modules/_io/stringio.c +++ b/Modules/_io/stringio.c @@ -908,23 +908,25 @@ _io_StringIO___setstate___impl(stringio *self, PyObject *state) once by __init__. So we do not take any chance and replace object's buffer completely. */ { - PyObject *item; - Py_UCS4 *buf; - Py_ssize_t bufsize; - - item = PyTuple_GET_ITEM(state, 0); - buf = PyUnicode_AsUCS4Copy(item); - if (buf == NULL) - return NULL; - bufsize = PyUnicode_GET_LENGTH(item); + PyObject *item = PyTuple_GET_ITEM(state, 0); + if (PyUnicode_Check(item)) { + Py_UCS4 *buf = PyUnicode_AsUCS4Copy(item); + if (buf == NULL) + return NULL; + Py_ssize_t bufsize = PyUnicode_GET_LENGTH(item); - if (resize_buffer(self, bufsize) < 0) { + if (resize_buffer(self, bufsize) < 0) { + PyMem_Free(buf); + return NULL; + } + memcpy(self->buf, buf, bufsize * sizeof(Py_UCS4)); PyMem_Free(buf); - return NULL; + self->string_size = bufsize; + } + else { + assert(item == Py_None); + self->string_size = 0; } - memcpy(self->buf, buf, bufsize * sizeof(Py_UCS4)); - PyMem_Free(buf); - self->string_size = bufsize; } /* Set carefully the position value. Alternatively, we could use the seek From 5bb059fe606983814a445e4dcf9e96fd7cb4951a Mon Sep 17 00:00:00 2001 From: Barney Gale Date: Mon, 25 Nov 2024 19:59:20 +0000 Subject: [PATCH 23/24] GH-127236: `pathname2url()`: generate RFC 1738 URL for absolute POSIX path (#127194) When handed an absolute Windows path such as `C:\foo` or `//server/share`, the `urllib.request.pathname2url()` function returns a URL with an authority section, such as `///C:/foo` or `//server/share` (or before GH-126205, `////server/share`). Only the `file:` prefix is omitted. But when handed an absolute POSIX path such as `/etc/hosts`, or a Windows path of the same form (rooted but lacking a drive), the function returns a URL without an authority section, such as `/etc/hosts`. This patch corrects the discrepancy by adding a `//` prefix before drive-less, rooted paths when generating URLs. --- Doc/library/urllib.request.rst | 10 ++++++---- Lib/nturl2path.py | 20 +++++++++++-------- Lib/test/test_urllib.py | 10 +++++----- Lib/urllib/request.py | 8 +++++--- ...-11-23-12-25-06.gh-issue-125866.wEOP66.rst | 5 +++++ 5 files changed, 33 insertions(+), 20 deletions(-) create mode 100644 Misc/NEWS.d/next/Library/2024-11-23-12-25-06.gh-issue-125866.wEOP66.rst diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 9055556a3703bb..3c07dc4adf434a 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -159,12 +159,14 @@ The :mod:`urllib.request` module defines the following functions: 'file:///C:/Program%20Files' .. versionchanged:: 3.14 - Windows drive letters are no longer converted to uppercase. + Paths beginning with a slash are converted to URLs with authority + sections. For example, the path ``/etc/hosts`` is converted to + the URL ``///etc/hosts``. .. versionchanged:: 3.14 - On Windows, ``:`` characters not following a drive letter are quoted. In - previous versions, :exc:`OSError` was raised if a colon character was - found in any position other than the second character. + Windows drive letters are no longer converted to uppercase, and ``:`` + characters not following a drive letter no longer cause an + :exc:`OSError` exception to be raised on Windows. .. function:: url2pathname(url) diff --git a/Lib/nturl2path.py b/Lib/nturl2path.py index 01135d1b7683b2..7e13ae3128333d 100644 --- a/Lib/nturl2path.py +++ b/Lib/nturl2path.py @@ -55,13 +55,17 @@ def pathname2url(p): p = p[4:] if p[:4].upper() == 'UNC/': p = '//' + p[4:] - drive, tail = ntpath.splitdrive(p) - if drive[1:] == ':': - # DOS drive specified. Add three slashes to the start, producing - # an authority section with a zero-length authority, and a path - # section starting with a single slash. - drive = f'///{drive}' + drive, root, tail = ntpath.splitroot(p) + if drive: + if drive[1:] == ':': + # DOS drive specified. Add three slashes to the start, producing + # an authority section with a zero-length authority, and a path + # section starting with a single slash. + drive = f'///{drive}' + drive = urllib.parse.quote(drive, safe='/:') + elif root: + # Add explicitly empty authority to path beginning with one slash. + root = f'//{root}' - drive = urllib.parse.quote(drive, safe='/:') tail = urllib.parse.quote(tail) - return drive + tail + return drive + root + tail diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py index fe16badc5bc77d..00e46990c406ac 100644 --- a/Lib/test/test_urllib.py +++ b/Lib/test/test_urllib.py @@ -1434,7 +1434,7 @@ def test_pathname2url_win(self): self.assertEqual(fn('C:\\foo:bar'), '///C:/foo%3Abar') self.assertEqual(fn('foo:bar'), 'foo%3Abar') # No drive letter - self.assertEqual(fn("\\folder\\test\\"), '/folder/test/') + self.assertEqual(fn("\\folder\\test\\"), '///folder/test/') self.assertEqual(fn("\\\\folder\\test\\"), '//folder/test/') self.assertEqual(fn("\\\\\\folder\\test\\"), '///folder/test/') self.assertEqual(fn('\\\\some\\share\\'), '//some/share/') @@ -1447,7 +1447,7 @@ def test_pathname2url_win(self): self.assertEqual(fn('//?/unc/server/share/dir'), '//server/share/dir') # Round-tripping urls = ['///C:', - '/folder/test/', + '///folder/test/', '///C:/foo/bar/spam.foo'] for url in urls: self.assertEqual(fn(urllib.request.url2pathname(url)), url) @@ -1456,12 +1456,12 @@ def test_pathname2url_win(self): 'test specific to POSIX pathnames') def test_pathname2url_posix(self): fn = urllib.request.pathname2url - self.assertEqual(fn('/'), '/') - self.assertEqual(fn('/a/b.c'), '/a/b.c') + self.assertEqual(fn('/'), '///') + self.assertEqual(fn('/a/b.c'), '///a/b.c') self.assertEqual(fn('//a/b.c'), '////a/b.c') self.assertEqual(fn('///a/b.c'), '/////a/b.c') self.assertEqual(fn('////a/b.c'), '//////a/b.c') - self.assertEqual(fn('/a/b%#c'), '/a/b%25%23c') + self.assertEqual(fn('/a/b%#c'), '///a/b%25%23c') @unittest.skipUnless(os_helper.FS_NONASCII, 'need os_helper.FS_NONASCII') def test_pathname2url_nonascii(self): diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index 9e555432688a5b..1fcaa89188188d 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -1667,9 +1667,11 @@ def url2pathname(pathname): def pathname2url(pathname): """OS-specific conversion from a file system path to a relative URL of the 'file' scheme; not recommended for general use.""" - if pathname[:2] == '//': - # Add explicitly empty authority to avoid interpreting the path - # as authority. + if pathname[:1] == '/': + # Add explicitly empty authority to absolute path. If the path + # starts with exactly one slash then this change is mostly + # cosmetic, but if it begins with two or more slashes then this + # avoids interpreting the path as a URL authority. pathname = '//' + pathname encoding = sys.getfilesystemencoding() errors = sys.getfilesystemencodeerrors() diff --git a/Misc/NEWS.d/next/Library/2024-11-23-12-25-06.gh-issue-125866.wEOP66.rst b/Misc/NEWS.d/next/Library/2024-11-23-12-25-06.gh-issue-125866.wEOP66.rst new file mode 100644 index 00000000000000..0b8ffdb3901db3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-11-23-12-25-06.gh-issue-125866.wEOP66.rst @@ -0,0 +1,5 @@ +:func:`urllib.request.pathname2url` now adds an empty authority when +generating a URL for a path that begins with exactly one slash. For example, +the path ``/etc/hosts`` is converted to the scheme-less URL ``///etc/hosts``. +As a result of this change, URLs without authorities are only generated for +relative paths. From 26ff32b30553e1f7b0cc822835ad2da8890c180c Mon Sep 17 00:00:00 2001 From: funkyrailroad Date: Mon, 25 Nov 2024 16:34:01 -0500 Subject: [PATCH 24/24] gh-127265: Remove single quotes from 'arrow's in tutorial/errors.rst (GH-127267) --- Doc/tutorial/errors.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 4c61cbb2b5bc3d..c01cb8c14a0360 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -23,7 +23,7 @@ complaint you get while you are still learning Python:: ^^^^^ SyntaxError: invalid syntax -The parser repeats the offending line and displays little 'arrow's pointing +The parser repeats the offending line and displays little arrows pointing at the token in the line where the error was detected. The error may be caused by the absence of a token *before* the indicated token. In the example, the error is detected at the function :func:`print`, since a colon