Mixture of experts pretraining benchmark #780
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
ZhiyuLi-goog
force-pushed
the
lizhiyu/moe
branch
from
dcd532b to
1cc20e7
Compare
ZhiyuLi-goog
force-pushed
the
lizhiyu/moe
branch
from
1cc20e7 to
6f0f836
Compare
* fix(moe): Added weight decay parameter * fix(moe): Added proper handling of device count per node * refactor(moe): Data preprocessing cleanup * fix(moe): This container has more stable convergence * fix(gpu): data preprocessing * build(gpu): Fix container image to specific version of NeMo and Megatron
|
Thank you @ShriyaPalsamudram for the reviewing. Could you help me merge the PR when you think it is in good shape since I don't have authorization. Not yet covered nemo 2.0 GPU guides and probably we can add it later in a separate PR:
|
ZhiyuLi-goog
changed the title
|
For sure, I will give the current GPU guide a try over the weekend!
|
* docs(moe): GPU running and slurm docs * docs: Fixed markup
| --output_dir <path to save checkpoint> --hf_token <your token to HF repository> | ||
| ``` | ||
|
|
||
| This script will download specified checkpoint from huggingface repository, preprocess it and save |
There was a problem hiding this comment.
Is there a step to verify checksums of the converted checkpoint to ensure correctness?
Is this converted checkpoint available for download directly from mlcommons drive? If yes, can those instructions be shared here as well?
There was a problem hiding this comment.
There is no checkpoint in any valid form. The one inside the mlcommon is neither in raw HF (maybe thats a good point, we should mirror HF checkpoint inside S3 bucket?) nor Nemo compatible
There was a problem hiding this comment.
We should have both the raw HF checkpoint as well as the NeMo compatible version in the S3 bucket so it lives on until we need it to and to make it easier for submitters to access all artifacts in the same place.
| This script will download specified checkpoint from huggingface repository, preprocess it and save | ||
| into specified directory | ||
|
|
||
| To preprocess dataset, use dataset_preprocessing.py script |
There was a problem hiding this comment.
Same as checkpoint
There was a problem hiding this comment.
Preprocessed dataset can be downloaded from mlcommons bucket. I added annotation while keeping manual preprocessing step in the documentation
|
The
|
Description
Add MoE benchmark to mlcommons repo.
todo list
TPU
GPU
General
cc @suexu1025 @ShriyaPalsamudram