Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve UCX documentation and examples #545

Merged
merged 12 commits into from
Mar 10, 2021
Merged

Improve UCX documentation and examples #545

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
cd77d1b
Add requirements file for docs
charlesbluca Mar 5, 2021
ce9c3c9
Reorganize introduction, add requirements section
charlesbluca Mar 5, 2021
da5a8ac
Reorganize configuration options
charlesbluca Mar 5, 2021
cb90d3b
Reorganize usage examples
charlesbluca Mar 8, 2021
37bf39e
Change dask.distributed to distributed
charlesbluca Mar 8, 2021
1980416
Fix docstring for dask_cuda.initialize.initialize
charlesbluca Mar 8, 2021
ee2e46c
Clean up docstring for LocalCUDACluster
charlesbluca Mar 8, 2021
de83d87
Clarify
charlesbluca Mar 8, 2021
9703376
Clarify that CUDA clusters/workers are UCX configured
charlesbluca Mar 9, 2021
301582a
Address review suggestions
charlesbluca Mar 9, 2021
d68810d
Scheduler is not affected by initialize
charlesbluca Mar 9, 2021
444df77
Fixed for flake8
charlesbluca Mar 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions dask_cuda/initialize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,13 @@ def initialize(
starting up Dask workers which create a variety of threads.

To ensure UCX works correctly, it is important to ensure it is initialized with
the correct options. This is important for scheduler, workers, and client. This
function will ensure that based on the flags and options passed by the user.
the correct options. This is especially important for the scheduler and client,
which cannot be configured to use UCX with arguments like ``LocalCUDACluster``
and ``dask-cuda-worker``. This function will ensure that they are provided a UCX
configuration based on the flags and options passed by the user.
pentschev marked this conversation as resolved.
Show resolved Hide resolved

This function can also be used within a Worker preload script.
This function can also be used within a worker preload script for UCX configuration
of mainline Dask/Distributed.
https://docs.dask.org/en/latest/setup/custom-startup.html

You can add it to your global config with the following YAML:
Expand Down