Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce noise from GC-related logging #8804

Merged
merged 6 commits into from
Jul 30, 2024
Merged

Reduce noise from GC-related logging #8804

merged 6 commits into from
Jul 30, 2024

Conversation

hendrikmakait
Copy link
Member

@hendrikmakait hendrikmakait commented Jul 26, 2024 •
edited
Loading

(Partially?) addresses #8543

  • Tests added / passed
  • Passes pre-commit run --all-files

@hendrikmakait hendrikmakait changed the title Reduce nose from GC-related logging Reduce noise from GC-related logging Jul 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 26, 2024 •
edited
Loading

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    29 files  ±0      29 suites  ±0   11h 55m 10s ⏱️ +55s
 4 092 tests ±0   3 978 ✅ +3    111 💤 ±0  3 ❌  - 3 
55 353 runs  ±0  52 920 ✅ +3  2 430 💤 ±0  3 ❌  - 3 

For more details on these failures, see this check.

Results for commit 1da9819. ± Comparison against base commit 40fcd65.

This pull request removes 3 and adds 3 tests. Note that renamed tests count towards both. 
distributed.tests.test_utils_perf ‑ test_fractional_timer
distributed.tests.test_utils_perf ‑ test_gc_diagnosis_cpu_time
distributed.tests.test_utils_perf ‑ test_gc_diagnosis_rss_win

distributed.tests.test_gc ‑ test_fractional_timer
distributed.tests.test_gc ‑ test_gc_diagnosis_cpu_time
distributed.tests.test_gc ‑ test_gc_diagnosis_rss_win
This pull request removes 1 skipped test and adds 1 skipped test. Note that renamed tests count towards both. 
distributed.tests.test_utils_perf ‑ test_gc_diagnosis_rss_win

distributed.tests.test_gc ‑ test_gc_diagnosis_rss_win

♻️ This comment has been updated with latest results.

hendrikmakait
hendrikmakait commented Jul 26, 2024
View reviewed changes
distributed/gc.py

logger = _logger = logging.getLogger(__name__)
logger.addFilter(RateLimiterFilter("full garbage collections took", rate="60s"))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This limits the amount of logs if they are captured.

@hendrikmakait hendrikmakait marked this pull request as ready for review July 26, 2024 13:04
@hendrikmakait hendrikmakait requested a review from fjetter as a code owner July 26, 2024 13:04
hendrikmakait
hendrikmakait commented Jul 26, 2024
View reviewed changes
distributed/config.py
@@ -97,6 +97,7 @@ def _initialize_logging_old_style(config: dict[Any, Any]) -> None:
loggers: dict[str, str | int] = { # default values
"distributed": "info",
"distributed.client": "warning",
"distributed.gc": "warning",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR renames utils_perf to gc, both the file and the logger. This is breaking, but I don't think too many people use this. If anything, I can perceive people importing disable_gc_diagnosis. If worthwhile, we can run a deprecation cycle for these imports.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can tell, https://github.com/AlexeyPechnikov/pygmtsar/blob/b5c58d61f00350cc4115d63985af9caa2c7cd9de/pygmtsar/pygmtsar/IO.py#L455 is the only package with many users that does this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any real benefit of renaming? Otherwise I'd keep as is if there are actual users out there. Not sure if this package counts as actual users?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any real benefit of renaming? Otherwise I'd keep as is if there are actual users out there. Not sure if this package counts as actual users?

Mostly, I really hate the name as it's completely non-descriptive.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once again, we could add a deprecation cycle, but I'd like to get rid of the name sooner or later.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah lets change it now

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you open an issue there just to be nice?

@hendrikmakait hendrikmakait mentioned this pull request Jul 26, 2024
Merged
2 tasks
@hendrikmakait hendrikmakait merged commit c44ad22 into dask:main Jul 30, 2024
30 of 34 checks passed
@hendrikmakait hendrikmakait deleted the adjust-gc-logging branch December 3, 2024 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@phofl phofl phofl approved these changes

@fjetter fjetter Awaiting requested review from fjetter fjetter is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hendrikmakait @phofl