Avoid hangs on many-core Windows machines (#7035)

Creating a multiprocessing Pool with too many processes can hit ValueError exceptions or hangs or both. The number that counts as "too many" depends on the Python version so this change uses 56 as a guaranteed safe limit. Details in #6965 Note that this only avoids the issue if an explicit jobs count is not passed in. Co-authored-by: Daniël van Noord <[email protected]>
pylint-dev · Jun 29, 2022 · bee24cd · bee24cd
1 parent b379ef3
commit bee24cd
Show file tree

Hide file tree

Showing 3 changed files with 10 additions and 1 deletion.
diff --git a/doc/whatsnew/2/2.14/full.rst b/doc/whatsnew/2/2.14/full.rst
@@ -13,6 +13,11 @@ Release date: TBA
 
   Closes #7006
 
+* Fixed an issue where many-core Windows machines (>~60 logical processors) would hang when
+  using the default jobs count.
+
+  Closes #6965
+
 What's New in Pylint 2.14.3?
 ----------------------------
 Release date: 2022-06-18

diff --git a/pylint/lint/base_options.py b/pylint/lint/base_options.py
@@ -244,7 +244,8 @@ def _make_linter_options(linter: PyLinter) -> Options:
                 "short": "j",
                 "default": 1,
                 "help": "Use multiple processes to speed up Pylint. Specifying 0 will "
-                "auto-detect the number of processors available to use.",
+                "auto-detect the number of processors available to use, and will cap "
+                "the count on Windows to avoid hangs.",
             },
         ),
         (

diff --git a/pylint/lint/run.py b/pylint/lint/run.py
@@ -82,6 +82,9 @@ def _cpu_count() -> int:
         cpu_count = multiprocessing.cpu_count()
     else:
         cpu_count = 1
+    if sys.platform == "win32":
+        # See also https://github.com/python/cpython/issues/94242
+        cpu_count = min(cpu_count, 56)  # pragma: no cover
     if cpu_share is not None:
         return min(cpu_share, cpu_count)
     return cpu_count