Distributed Training

Prerequisites: NCCL

• Read https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html

Good to know: MPI

• Good read: https://mpitutorial.com/tutorials/

Data Parallelism

• Understand NCCLAllReduce
• Get familiar with DDP usage/setup

Megatron

• Read https://arxiv.org/abs/1909.08053

Zero

• Read https://arxiv.org/abs/1910.02054

• Zero-1

  • Understand ReduceScatter/AllGather
  • Understand how optimizer state is partitioned

• Zero-2

• Zero-3

Mix of Experts

• Understand All2All

Pipeline Parallelism