My multi gpu is not getting detected? #1935
-
|
I have 4xA6000 and wanted to fine tune the model. and this is my config: base_model: unsloth/Qwen2.5-1.5B-bnb-4bit
datasets:
- path: itztheking/worthynews3
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.05
output_dir: ./outputs/out
sequence_len: 100000
# Set to use 4-bit quantization for lower memory usage
load_in_4bit: true
bf16: auto # Automatically detects bf16 support for Ampere and newer GPUs.
adapter: lora # LoRA for efficient fine-tuning
lora_r: 8
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
# Adjusted for memory efficiency
gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 4
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.0002
# Gradient checkpointing to reduce memory load
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
# Set DeepSpeed to Zero2 for better memory management
deepspeed: deepspeed_configs/zero2.json
# Logging and training configuration
logging_steps: 1
xformers_attention: true
flash_attention: true
warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
# Set up the Liger plugin
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true
# Optional: Enable mixed precision for memory efficiency
mixed_precision: bf16
I get the following: root@f24ac015d3f9:/workspace/axolotl# accelerate launch -m axolotl.cli.train examples/qwen2.5/config.yaml --deepspeed deepspeed_configs/zero1.json
More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in `--num_processes=1`.
`--num_machines` was set to a value of `1`
`--mixed_precision` was set to a value of `'no'`
`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
WARNING: BNB_CUDA_VERSION=121 environment variable detected; loading libbitsandbytes_cuda121.so.
This can be used to load a bitsandbytes version that is different from the PyTorch CUDA version.
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
If this was unintended set the BNB_CUDA_VERSION variable to an empty string: export BNB_CUDA_VERSION=
If you use the manual override make sure the right libcudart.so is in your LD_LIBRARY_PATH
For example by adding the following to your .bashrc: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<path_to_cuda_dir/lib64
[2024-09-29 08:58:53,047] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-29 08:58:53,092] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-09-29 08:58:53,453] [INFO] [root.spawn:38] [PID:3059] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -i
system /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmp3x9ihpo8/test.c -o /tmp/tmp3x9ihpo8/test.o
[2024-09-29 08:58:53,472] [INFO] [root.spawn:38] [PID:3059] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmp3x9ihpo8/test.o -laio -o /tmp/tmp3x9ihpo8/a.out
system /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmpiv3r9m76/test.c -o /tmp/tmpiv3r9m76/test.o
[2024-09-29 08:58:53,512] [INFO] [root.spawn:38] [PID:3057] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmpiv3r9m76/test.o -laio -o /tmp/tmpiv3r9m76/a.out
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[2024-09-29 08:58:53,555] [INFO] [root.spawn:38] [PID:3058] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/py3.11/include -fPIC -O2 -i
system /root/miniconda3/envs/py3.11/include -fPIC -c /tmp/tmpjv2_0zgs/test.c -o /tmp/tmpjv2_0zgs/test.o
[2024-09-29 08:58:53,572] [INFO] [root.spawn:38] [PID:3058] gcc -pthread -B /root/miniconda3/envs/py3.11/compiler_compat /tmp/tmpjv2_0zgs/test.o -laio -o /tmp/tmpjv2_0zgs/a.out
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
[WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/pydantic/_internal/_fields.py:151: UserWarning: Field "model_kwargs" has conflict with protected namespace "model_".
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
warnings.warn(
[2024-09-29 08:58:55,465] [INFO] [axolotl.normalize_config:186] [PID:3058] [RANK:2] GPU memory usage baseline: 0.000GB (+0.463GB misc)
[2024-09-29 08:58:55,468] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-09-29 08:58:55,694] [WARNING] [axolotl.scripts.check_user_token:518] [PID:3058] [RANK:2] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` and get your access token from https:
//huggingface.co/settings/tokens if you want to use gated models or datasets.
[2024-09-29 08:58:55,747] [WARNING] [axolotl.utils.config.models.input.hint_lora_8bit:1087] [PID:3057] [RANK:1] We recommend setting `load_in_8bit: true` for LORA finetuning
[2024-09-29 08:58:55,750] [DEBUG] [axolotl.normalize_config:83] [PID:3057] [RANK:1] bf16 support detected, enabling for this configuration.
[2024-09-29 08:58:55,802] [WARNING] [axolotl.utils.config.models.input.hint_lora_8bit:1087] [PID:3059] [RANK:3] We recommend setting `load_in_8bit: true` for LORA finetuning
[2024-09-29 08:58:55,805] [DEBUG] [axolotl.normalize_config:83] [PID:3059] [RANK:3] bf16 support detected, enabling for this configuration.
[2024-09-29 08:58:55,892] [INFO] [axolotl.normalize_config:186] [PID:3057] [RANK:1] GPU memory usage baseline: 0.000GB (+0.463GB misc)
[2024-09-29 08:58:55,894] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-09-29 08:58:55,894] [INFO] [axolotl.normalize_config:186] [PID:3059] [RANK:3] GPU memory usage baseline: 0.000GB (+0.463GB misc)
[2024-09-29 08:58:55,896] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-09-29 08:58:55,945] [WARNING] [axolotl.utils.config.models.input.hint_lora_8bit:1087] [PID:3056] [RANK:0] We recommend setting `load_in_8bit: true` for LORA finetuning
[2024-09-29 08:58:55,947] [DEBUG] [axolotl.normalize_config:83] [PID:3056] [RANK:0] bf16 support detected, enabling for this configuration.
[2024-09-29 08:58:56,049] [INFO] [axolotl.normalize_config:186] [PID:3056] [RANK:0] GPU memory usage baseline: 0.000GB (+0.463GB misc)
dP dP dP
88 88 88
88. .88 .d88b. 88. .88 88 88. .88 88 88
`88888P8 dP' `dP `88888P' dP `88888P' dP dP
trl: 0.9.6
torch: 2.3.1+cu121
bitsandbytes: 0.44.0
****************************************
[2024-09-29 08:58:56,102] [WARNING] [axolotl.scripts.check_user_token:518] [PID:3056] [RANK:0] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` and get your access token from https:
//huggingface.co/settings/tokens if you want to use gated models or datasets.
[2024-09-29 08:58:56,116] [DEBUG] [axolotl.load_tokenizer:282] [PID:3058] [RANK:2] EOS: 151643 / <|endoftext|>
[2024-09-29 08:58:56,116] [DEBUG] [axolotl.load_tokenizer:283] [PID:3058] [RANK:2] BOS: None / None
[2024-09-29 08:58:56,149] [WARNING] [axolotl.scripts.check_user_token:518] [PID:3057] [RANK:1] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` and get your access token from https:
//huggingface.co/settings/tokens if you want to use gated models or datasets.
[2024-09-29 08:58:56,168] [WARNING] [axolotl.scripts.check_user_token:518] [PID:3059] [RANK:3] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` and get your access token from https:
//huggingface.co/settings/tokens if you want to use gated models or datasets.
[2024-09-29 08:58:56,550] [DEBUG] [axolotl.load_tokenizer:282] [PID:3056] [RANK:0] EOS: 151643 / <|endoftext|>
[2024-09-29 08:58:56,550] [DEBUG] [axolotl.load_tokenizer:283] [PID:3056] [RANK:0] BOS: None / None
[2024-09-29 08:58:56,558] [DEBUG] [axolotl.load_tokenizer:282] [PID:3057] [RANK:1] EOS: 151643 / <|endoftext|>
[2024-09-29 08:58:56,558] [DEBUG] [axolotl.load_tokenizer:283] [PID:3057] [RANK:1] BOS: None / None
[2024-09-29 08:58:56,558] [DEBUG] [axolotl.load_tokenizer:284] [PID:3057] [RANK:1] PAD: 151665 / <|PAD_TOKEN|>
[2024-09-29 08:58:56,558] [DEBUG] [axolotl.load_tokenizer:285] [PID:3057] [RANK:1] UNK: None / None
[2024-09-29 08:58:56,558] [INFO] [axolotl.load_tokenizer:296] [PID:3057] [RANK:1] No Chat template selected. Consider adding a chat template for easier inference.
[2024-09-29 08:58:56,575] [DEBUG] [axolotl.load_tokenizer:285] [PID:3059] [RANK:3] UNK: None / None
[2024-09-29 08:58:56,575] [INFO] [axolotl.load_tokenizer:296] [PID:3059] [RANK:3] No Chat template selected. Consider adding a chat template for easier inference.
[2024-09-29 08:58:57,027] [DEBUG] [axolotl.calculate_total_num_steps:316] [PID:3056] [RANK:0] total_num_tokens: 2_232_206
[2024-09-29 08:58:57,030] [INFO] [axolotl.load_tokenized_prepared_datasets:186] [PID:3059] [RANK:3] Prepared dataset loaded from disk...
[2024-09-29 08:58:57,030] [INFO] [axolotl.load_tokenized_prepared_datasets:186] [PID:3058] [RANK:2] Prepared dataset loaded from disk...
[2024-09-29 08:58:57,031] [DEBUG] [axolotl.calculate_total_num_steps:329] [PID:3056] [RANK:0] `total_supervised_tokens: 53_293`
[2024-09-29 08:58:57,031] [INFO] [axolotl.load_tokenized_prepared_datasets:186] [PID:3057] [RANK:1] Prepared dataset loaded from disk...
[2024-09-29 08:58:57,031] [DEBUG] [axolotl.calculate_total_num_steps:407] [PID:3056] [RANK:0] total_num_steps: 24
[2024-09-29 08:58:57,054] [DEBUG] [axolotl.train.train:67] [PID:3056] [RANK:0] loading tokenizer... unsloth/Qwen2.5-1.5B-bnb-4bit
[2024-09-29 08:58:57,437] [DEBUG] [axolotl.load_tokenizer:282] [PID:3056] [RANK:0] EOS: 151643 / <|endoftext|>
[2024-09-29 08:58:57,437] [DEBUG] [axolotl.load_tokenizer:283] [PID:3056] [RANK:0] BOS: None / None
[2024-09-29 08:58:57,437] [DEBUG] [axolotl.load_tokenizer:284] [PID:3056] [RANK:0] PAD: 151665 / <|PAD_TOKEN|>
[2024-09-29 08:58:57,437] [DEBUG] [axolotl.load_tokenizer:285] [PID:3056] [RANK:0] UNK: None / None
[2024-09-29 08:58:57,437] [INFO] [axolotl.load_tokenizer:296] [PID:3056] [RANK:0] No Chat template selected. Consider adding a chat template for easier inference.
[2024-09-29 08:58:57,437] [DEBUG] [axolotl.train.train:96] [PID:3056] [RANK:0] loading model and peft_config...
[2024-09-29 08:58:57,443] [DEBUG] [axolotl.load_tokenizer:282] [PID:3059] [RANK:3] EOS: 151643 / <|endoftext|>
[2024-09-29 08:58:57,443] [DEBUG] [axolotl.load_tokenizer:283] [PID:3059] [RANK:3] BOS: None / None
[2024-09-29 08:58:57,443] [DEBUG] [axolotl.load_tokenizer:284] [PID:3059] [RANK:3] PAD: 151665 / <|PAD_TOKEN|>
[2024-09-29 08:58:57,444] [DEBUG] [axolotl.load_tokenizer:285] [PID:3059] [RANK:3] UNK: None / None
[2024-09-29 08:58:57,444] [INFO] [axolotl.load_tokenizer:296] [PID:3059] [RANK:3] No Chat template selected. Consider adding a chat template for easier inference.
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/quantizers/auto.py:182: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading
[2024-09-29 08:58:57,568] [DEBUG] [axolotl.load_tokenizer:283] [PID:3057] [RANK:1] BOS: None / None
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/quantizers/auto.py:182: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading
[rank3]: File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 4279, in prediction_step
[rank3]: loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/workspace/axolotl/src/axolotl/core/trainer_builder.py", line 665, in compute_loss
[rank3]: File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 3532, in compute_loss
[rank3]: return forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: ^^^^^^^^^^^^^^^^
[rank3]: return forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/nn/modules/loss.py", line 1185, in forward
[rank3]: return F.cross_entropy(input, target, weight=self.weight,
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/nn/functional.py", line 3086, in cross_entropy
[rank3]: return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 28.47 GiB. GPU has a total capacity of 47.53 GiB of which 17.14 GiB is free. Process 3598542 has 30.38 GiB memory in use. Of the allocated
memory 29.75 GiB is allocated by PyTorch, and 119.21 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragm
entation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
W0929 08:59:36.413000 127306780133184 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3056 closing signal SIGTERM
W0929 08:59:36.413000 127306780133184 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3057 closing signal SIGTERM
W0929 08:59:36.414000 127306780133184 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3058 closing signal SIGTERM
E0929 08:59:36.980000 127306780133184 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 3 (pid: 3059) of binary: /root/miniconda3/envs/py3.11/bin/python3
Traceback (most recent call last):
File "/root/miniconda3/envs/py3.11/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1165, in launch_command
multi_gpu_launcher(args)
File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/commands/launch.py", line 799, in multi_gpu_launcher
distrib_run.run(args)
File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/run.py", line 870, in run
elastic_launch(
File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
axolotl.cli.train FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-09-29_08:59:36
host : f24ac015d3f9
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 3059)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.htmlIt does say at first it has detected multi gpu but at the end it feels like it's only allocating to only one of them. Additionally this is the stats: Would appreciate any help. |
Beta Was this translation helpful? Give feedback.
Answered by
NanoCode012
Oct 10, 2024
Replies: 2 comments 5 replies
-
|
Hey, I did a test with your config. I'm not sure if this was a typo, but your |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Indeed, I do want the context window to be 100K |
Beta Was this translation helpful? Give feedback.
-
|
Hey, the model should already support unto 128k tokens* from the model card. In the case of axolotl yaml, the max you should go is 32k. The model learns to scale from rope from the Does that make sense? |
Beta Was this translation helpful? Give feedback.
-
Hey, Thank you again. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, my bad there. I fixed my message a bit on the numbers as well. I mean, i.e. set the below and your model would train fine and support 100k seq during inference :) sequence_len: 32768 |
Beta Was this translation helpful? Give feedback.
Answer selected by
ItzAmirreza
-
|
Hi, I’m currently studying how to increase the context size of a model. Let me clarify my understanding: Meanwhile, the model’s maximum context size is determined by the RoPE (Rotary Position Embedding) configuration at the time of training. For example, the Qwen/Qwen2.5-1.5B model was trained with Thanks for helping me clarify! |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Hey
and If you follow qwen's setup, you should be able to support upto 128k tokens from my understanding! |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey, my bad there. I fixed my message a bit on the numbers as well.
I mean,
sequence_len: 32768is the one you should set to use the model's full context listed above. Therope_thetaparam within the model's config (linked above) is what allows it to scale from 32k to its long context.i.e. set the below and your model would train fine and support 100k seq during inference :)