From c94d66d939a2f67101f80296b8ddf5efb7be8f64 Mon Sep 17 00:00:00 2001 From: bachvudinh <89349141+bachvudinh@users.noreply.github.com> Date: Mon, 25 Nov 2024 12:52:30 +0700 Subject: [PATCH] Update README.md --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0928d7d..f93777e 100755 --- a/README.md +++ b/README.md @@ -144,9 +144,7 @@ accelerate launch --config_file ./accelerate_config.yaml train.py ```bash tune download homebrewltd/llama3.1-s-whispervq-init --output-dir ../model_zoo/llama3.1-s-whispervq-init --ignore-patterns "original/consolidated*" ``` - -2. **Pretraining Multi GPU (1-8GPUs Supported)** - [NOTE] : Before start training, make sure you uploaded the resized embedding model to Hugging Face Hub: + [NOTE] : In case you want to use different base model, you can uploaded your own resized embedding model to Hugging Face Hub: ```python # folder containing the checkpoint files model_name = "meta-llama/Llama-3.2-3B-Instruct" @@ -160,6 +158,8 @@ accelerate launch --config_file ./accelerate_config.yaml train.py model.push_to_hub("/Llama3.1-s-whispervq-init") tokenizer.push_to_hub("/Llama3.1-s-whispervq-init") ``` + +2. **Pretraining Multi GPU (1-8GPUs Supported)** ``` tune run --nproc_per_node full_finetune_fsdp2 --config recipes/configs/jan-llama3-1-s/pretrain/8B_full.yaml ```