Fix NVML index usage in CUDAWorker/LocalCUDACluster #671
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pentschev
added
3 - Ready for Review
|
This is similar to issues we had not long ago in Distributed, as @charlesbluca would remember. This came up when discussing MIG support with @akaanirban , thanks Anirban for bringing that to my attention. |
|
@beckernick @VibhuJawa this may have affected past performance on DGX A100s, where we had previously to manually launch |
Codecov Report
@@ Coverage Diff @@
## branch-21.08 #671 +/- ##
=================================================
+ Coverage 60.19% 90.31% +30.12%
=================================================
Files 21 15 -6
Lines 2605 1652 -953
=================================================
- Hits 1568 1492 -76
+ Misses 1037 160 -877
Continue to review full report at Codecov.
|
|
rerun tests |
akaanirban
added a commit
to akaanirban/dask-cuda
that referenced
this pull request
Jul 16, 2021
… devices if available
quasiben
approved these changes
Jul 20, 2021
|
@gpucibot merge |
rapids-bot bot
pushed a commit
that referenced
this pull request
Aug 2, 2021
Adds support to start LocalCUDACluster and cuda workers on MIG instances by passing in uuids of the mig instances. Builds off of existing PR #671 More specifically this PR does the following: 1. Allows starting `LocalCUDACluster` as the following: `cluster = LocalCUDACluster(CUDA_VISIBLE_DEVICES=["MIG-uuid1","MIG-uuid2",...])` or by passing them as `,` separated strings. Needs Discussion: 0. Apart from manually testing on a MIG instance on the cloud, how would we test this? 1. What if the user does not pass in any argument to `LocalCUDACluster` while using MIG instances? By default `LocalCUDACluster` will try to use all the parent GPUs and run into error. 2. What if we have a deployment with MIG-enabled and non-MIG-enabled GPUs? 3. `dask.distributed` diagnostics will also fail if we run on MIG enabled GPUs since it uses `pynvml` APIS for non-MIG-enabled GPUs only at the moment. Authors: - Anirban Das (https://github.com/akaanirban) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #674
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix an issue where device index
0would be used in NVML functions. For CUDA runtime calls, we expect that the GPU being targeted is always on index0as its relative to theCUDA_VISIBLE_DEVICESordering. However, NVML relies on absolute indices, thus we have to always use the actual GPU index being targeted, rather than the first one inCUDA_VISIBLE_DEVICES.This is normally not an issue if no
CUDA_VISIBLE_DEVICESis set, or is just set aslist(",".join(list(str(i) for i in range(get_n_gpus()))), but it may be an issue when targeting a different list of GPUs. For example on a DGX-1, CPU affinity for GPUs0-3is0-19,40-59, and for GPUs4-7it is20-39,60-79, but when the user would setCUDA_VISIBLE_DEVICES=4the CPU affinity for the targeted GPU would be the same as for device index0, or0-19,40-59. This could result in lower performance, as well as wrong computation of total GPU memory for non-homogenous GPU systems.