[README] Tips on using less than 8 MI300 GPUs #1270

nileshnegi · 2024-07-23T21:12:48Z

Details

Work item:
Internal

What were the changes?
Update README on using RCCL with less than 8 MI300 GPUs and how to improve performance

Approval Checklist

Do not approve until these items are satisfied.

Verify the CHANGELOG has been updated, if
- there are any NCCL API version changes,
- any changes impact library users, and/or
- any changes impact any other ROCm library.

Signed-off-by: nileshnegi <[email protected]>

spolifroni-amd · 2024-09-10T16:08:53Z

README.md

@@ -146,6 +146,17 @@ pip3 install -r sphinx/requirements.txt
 python3 -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html
 ```

+### Improving performance on MI300 when using less than 8 GPUs
+
+On a system with 8\*MI300X GPUs, each pair of GPUs are connected with dedicated XGMI links in a fully-connected topology. So, for collective operations, one can achieve good performance when all 8 GPUs (and all XGMI links) are used. When using less than 8 GPUs, one can only achieve a fraction of the potential bandwidth on the system.


Is it "good" performance, or "the best possible performance" or "optimal"? "Good" is very vague. A more precise term would be helpful (but not necessary).

On a system with 8*MI300X GPUs, each pair of GPUs is connected to dedicated XGMI links in a fully-connected topology. For collective operations, good performance is achieved when all 8 GPUs and all XGMI links are used. When using fewer than 8 GPUs, only a fraction of the potential bandwidth on the system can be achieved.

spolifroni-amd · 2024-09-10T16:08:56Z

README.md

+
+On a system with 8\*MI300X GPUs, each pair of GPUs are connected with dedicated XGMI links in a fully-connected topology. So, for collective operations, one can achieve good performance when all 8 GPUs (and all XGMI links) are used. When using less than 8 GPUs, one can only achieve a fraction of the potential bandwidth on the system.
+
+But, if your workload warrants using less than 8 MI300 GPUs on a system, you can set the run-time variable `NCCL_MIN_NCHANNELS` to increase the number of channels.\


If the workload warrants using fewer than 8 MI300 GPUs, the run-time variable NCCL_MIN_NCHANNELS can be used to increase the number of channels.\

spolifroni-amd · 2024-09-10T16:08:57Z

README.md

+But, if your workload warrants using less than 8 MI300 GPUs on a system, you can set the run-time variable `NCCL_MIN_NCHANNELS` to increase the number of channels.\
+E.g.: `export NCCL_MIN_NCHANNELS=32`
+
+Increasing the number of channels can be beneficial to performance, but it also increases GPU utilization for collective operations.


Increasing the number of channels can help with performance, but it can also increase GPU usage for collective operations.

spolifroni-amd · 2024-09-10T16:08:59Z

README.md

+
+Increasing the number of channels can be beneficial to performance, but it also increases GPU utilization for collective operations.
+
+Additionally, we have pre-defined higher number of channels when using only 2 GPUs or 4 GPUs on a 8\*MI300 system. Here, RCCL will use **32 channels** for the 2 MI300 GPUs scenario and **24 channels** for the 4 MI300 GPUs scenario.


Additionally, there are a pre-defined higher number of channels used when using only 2 or 4 of the GPUs on a 8*MI300 system. RCCL will use 32 channels with 2 MI300 GPUs and use 24 channels with 4 MI300 GPUs.

I can't tell if it's that RCCL sets or uses a pre-defined number of channels when using 2 or 4 GPUs. I put in "use", but if its sets (or something else), just put that in. If it's something else, please let me know so I understand for next time.

Rework core for NVIDIA Trusted Computing * Compress work structs so that they are shared between channels * Utilize the full amount of kernel argument space permitted (4k) before resorting to work fifo. * Rework the task preprocessing phase. * Use a separate abortDevFlag which is kept in sync with abortFlag using cudaMemcpy operations. * Rename src/include/align.h to src/include/bitops.h Add lazy connection establishment for collective operations * Move buffer allocation and connection establishment to the first collective operation using that algorithm. * Accelerate init time and reduce memory usage. * Avoid allocating NVLS buffers if all calls are registered. * Compute algo/proto in ncclLaunchCollTasksInfo early on. * Connect peers in ncclCollPreconnectFunc if not connected already. * Also move shared buffer creation to the first send/recv call. Accelerate intra-node NVLink detection * Make each rank only detect NVLinks attached to its GPU. * Fuse XMLs to reconstruct the full NVLink topology Add init profiling to report time spend in different init phases. * Report timings of bootstrap, allgather, search, connect, etc. * Add new "PROFILE" category for NCCL_DEBUG_SUBSYS. Add support for PCI p2p on split PCI switches * Detect split PCI switches through a kernel module exposing switch information. * Update the topology XML and graph to add those inter-switch connections. Add cost estimation API * Add a new ncclGroupEndSimulate primitive to return the estimated time a group would take. Net/IB: Add separate traffic class for fifo messages * Add NCCL_IB_FIFO_TC to control the traffic class of fifo messages independently from NCCL_IB_TC. Merges PR #1194 Net/IB: Add support for IB router * Use flid instead of lid if subnets do not match * Warn if flid is 0 Optimizations and fixes for device network offload (unpack) * Double the default number of channels * Cache netDeviceType * Fix save/increment head logic to enable Tree support. Support ncclGroupStart/End for ncclCommAbort/Destroy * Allow Abort/Destroy to be called within a group when managing multiple GPUs with a single process. Improve Tuner API * Provide to the plugin the original cost table so that the plugin can leave unknown or disabled algo/proto combinations untouched. * Remove nvlsSupport and collnetSupport. Do not print version to stdout when using a debug file * Also print version from all processes with INFO debug level. Fixes issue #1271 Fix clang warnings in NVTX headers * Update NVTX headers to the latest version Fixes issue #1270 Disable port fusion in heterogeneous systems * Do not fuse ports if a mix of multi-port and single port are detected. Fix NVLS graphs search for dual NICs. * Fix NVLS graph search when we have more than one NIC per GPU. Fix crash with collnetDirect * Add separate graph search for collnetDirect, testing alltoall paths and working similarly to the NVLS search. Fix hang when nodes have different CPU types * Add the CPU type to the rank peer info. * Align all ranks on the CPU type after the first allgather. * Only use the aligned CPU type for all tuning operations. Fixes issue #1136 Fixes issue #1184 Fix performance of registered send/recv operations * Allow for single full size operations * Add INFO to confirm the registration of send/recv buffers. Move all sync ops to finalize stage * Ensure ncclCommDestroy is non-blocking if ncclCommFinalize has been called. Improve error reporting during SHM segment creation Improve support of various compilers Merges PR #1177 Merges PR #1228 Allow net and tuner plugins to be statically linked * Search for ncclNet or ncclTuner symbols in the main binary. Merges PR #979 Plugin examples includes cleanup * Harmonize err.h and common.h usage. * Add mixed plugin with both net and tuner.

[README] Tips on using less than 8 MI300 GPUs

888a467

Signed-off-by: nileshnegi <[email protected]>

nileshnegi added the noCI Disable Jenkins for this PR. label Jul 23, 2024

nileshnegi self-assigned this Jul 23, 2024

nileshnegi marked this pull request as ready for review July 23, 2024 21:12

nileshnegi requested a review from a team as a code owner July 23, 2024 21:12

randyh62 approved these changes Aug 6, 2024

View reviewed changes

nileshnegi merged commit a247484 into ROCm:develop Aug 6, 2024
1 of 2 checks passed

spolifroni-amd reviewed Sep 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[README] Tips on using less than 8 MI300 GPUs #1270

[README] Tips on using less than 8 MI300 GPUs #1270

nileshnegi commented Jul 23, 2024

spolifroni-amd Sep 10, 2024

spolifroni-amd Sep 10, 2024

spolifroni-amd Sep 10, 2024

spolifroni-amd Sep 10, 2024


		On a system with 8\*MI300X GPUs, each pair of GPUs are connected with dedicated XGMI links in a fully-connected topology. So, for collective operations, one can achieve good performance when all 8 GPUs (and all XGMI links) are used. When using less than 8 GPUs, one can only achieve a fraction of the potential bandwidth on the system.

		But, if your workload warrants using less than 8 MI300 GPUs on a system, you can set the run-time variable `NCCL_MIN_NCHANNELS` to increase the number of channels.\


		Increasing the number of channels can be beneficial to performance, but it also increases GPU utilization for collective operations.

		Additionally, we have pre-defined higher number of channels when using only 2 GPUs or 4 GPUs on a 8\MI300 system. Here, RCCL will use 32 channels* for the 2 MI300 GPUs scenario and 24 channels for the 4 MI300 GPUs scenario.

[README] Tips on using less than 8 MI300 GPUs #1270

[README] Tips on using less than 8 MI300 GPUs #1270

Conversation

nileshnegi commented Jul 23, 2024

Details

Approval Checklist

spolifroni-amd Sep 10, 2024

Choose a reason for hiding this comment

spolifroni-amd Sep 10, 2024

Choose a reason for hiding this comment

spolifroni-amd Sep 10, 2024

Choose a reason for hiding this comment

spolifroni-amd Sep 10, 2024

Choose a reason for hiding this comment