Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix errors related with distributed.worker.memory.terminate #900

Merged
merged 1 commit into from
Apr 26, 2022

Conversation

pentschev
Copy link
Member

None is not a valid value for distributed.worker.memory.terminate anymore, we should use False instead.

Errors such as the one below are raised otherwise:

2022-04-25 13:42:48,943 - tornado.application - ERROR - Exception in callback functools.partial(<bound method NannyMemoryManager.memory_monitor of <distributed.worker_memory.NannyMemoryManager object at 0x7fd3de281b50>>, <Nanny: tcp://127.0.0.1:39989, threads: 1>)
Traceback (most recent call last):
  File "/datasets/pentschev/miniconda3/envs/tmp/lib/python3.8/site-packages/tornado/ioloop.py", line 905, in _run
    return self.callback()
  File "/datasets/pentschev/miniconda3/envs/tmp/lib/python3.8/site-packages/distributed/worker_memory.py", line 326, in memory_monitor
    if memory / self.memory_limit > self.memory_terminate_fraction:
TypeError: '>' not supported between instances of 'float' and 'NoneType'

`None` is not a valid value for `distributed.worker.memory.terminate`
anymore, we should use `False` instead.
@pentschev pentschev requested a review from a team as a code owner April 25, 2022 20:43
@github-actions github-actions bot added the python python code needed label Apr 25, 2022
@quasiben quasiben added 5 - Ready to Merge Testing and reviews complete, ready to merge non-breaking Non-breaking change labels Apr 25, 2022
@quasiben
Copy link
Member

@gpucibot merge

@quasiben quasiben added the bug Something isn't working label Apr 25, 2022
@codecov-commenter
Copy link

codecov-commenter commented Apr 25, 2022

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.06@23c6e2d). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 697dda1 differs from pull request most recent head 09690f7. Consider uploading reports for the commit 09690f7 to get more accurate results

@@               Coverage Diff               @@
##             branch-22.06     #900   +/-   ##
===============================================
  Coverage                ?   89.49%           
===============================================
  Files                   ?       16           
  Lines                   ?     2094           
  Branches                ?        0           
===============================================
  Hits                    ?     1874           
  Misses                  ?      220           
  Partials                ?        0           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23c6e2d...09690f7. Read the comment docs.

@jakirkham
Copy link
Member

rerun tests

@rapids-bot rapids-bot bot merged commit fa97c0f into rapidsai:branch-22.06 Apr 26, 2022
@pentschev
Copy link
Member Author

Thanks @quasiben and @jakirkham !

@pentschev pentschev deleted the fix-memory-terminate branch April 26, 2022 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Ready to Merge Testing and reviews complete, ready to merge bug Something isn't working non-breaking Non-breaking change python python code needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants