Calculate linter.config.jobs in cgroupsv2 environments #10089
Conversation
DominicLavery
force-pushed
the
cgroupsv2-cpu-count
branch
from
9ff146e
to
ff64dd8
Compare
This comment has been minimized.
This comment has been minimized.
DominicLavery
force-pushed
the
cgroupsv2-cpu-count
branch
from
ff64dd8
to
ab7b61b
Compare
Codecov ReportAll modified and coverable lines are covered by tests β
Additional details and impacted files@@ Coverage Diff @@
## main #10089 +/- ##
=======================================
Coverage 95.80% 95.80%
=======================================
Files 174 174
Lines 18962 18972 +10
=======================================
+ Hits 18166 18176 +10
Misses 796 796
|
|
π€ According to the primer, this change has no effect on the checked open source code. π€π This comment was generated for commit ab7b61b |
pylint/lint/run.py
Outdated
| @@ -65,6 +65,18 @@ def _query_cpu() -> int | None: | |||
| cpu_shares = int(file.read().rstrip()) | |||
| # For AWS, gives correct value * 1024. | |||
| avail_cpu = int(cpu_shares / 1024) | |||
| elif Path("/sys/fs/cgroup/cpu.max").is_file(): | |||
There was a problem hiding this comment.
Shouldn't this take precedence over cpu.shares? If the new file is present it should probably be preferred over the old one?
There was a problem hiding this comment.
I wouldn't expect them to co-exist so just went with a quick grouping around v1 vs v2 but it probably makes sense just in case.
I've pushed a new commit which prefers the v2 files and I think also makes the logic a bit clearer
tests/lint/test_run.py
Outdated
| from pylint.testutils.utils import _test_cwd | ||
|
|
||
|
|
||
| @pytest.mark.parametrize( |
There was a problem hiding this comment.
Why isn't this grouped with the other tests? I don't think we need a separate file as we don't follow the "file + test_file" structure
There was a problem hiding this comment.
I had a bit of confusion around using protected members in tests python. Figured it out and merged the tests back into the other file
DominicLavery
force-pushed
the
cgroupsv2-cpu-count
branch
from
ab7b61b
to
ad9f26c
Compare
|
Hey @DanielNoord! Thanks so much for the review. I've replied to your comments and pushed some fixes |
Type of Changes
Description
In containers running on cgroupv2 systems
_query_cpucurrently returnsNone. This results insched_getaffinitybeing used, which will normally return all installed CPUs of the host. This can result in crashes with the error:The changes here use the CPU quota in v2 systems.
max 100000is the default value, and will continue to result in the hosts CPU count being used.cpu.weight(the replacement of cpu shares from v1) could also be used, but as it's impact on CPU scheduling is relative to the rest of the cgroup hierarchy it isn't possible to get an accurate value on all systems where pylint may be run. Whereas this method is reliable on any container with a CPU quota