Popular repositories Loading

1. Finetune_llama2 Finetune_llama2 Public

   Build a llama fine-tuning script from scratch using PyTorch and transformers API. It needs to support 4 optional features: gradient checkpointing, mixed precision, data parallelism, tensor parallel…

   Python 2

2. Finetune_llama2_Megatron Finetune_llama2_Megatron Public

   Using megatron style to do TP training.

   Python 2

3. ColossalAI ColossalAI Public

   Forked from hpcaitech/ColossalAI

   Making large AI models cheaper, faster and more accessible

   Python 1

4. BandWidth_Test BandWidth_Test Public

   Test the GPU bandwidth of collectives operators like all-reduce, all-gather, broadcast and all-to-all primitives on single-node multi-GPU (2, 4, 8 cards) and multi-node multi-GPU (16 cards) setups,…

   Python 1

5. Pytorch-profile Pytorch-profile Public

   Use pytorch profile api to further analysis the training detailed information, like heaps and stacks, time consuming.

   Python

6. Backend Backend Public