You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/jovyan/simon-data/home/miniconda3/envs/alpaca39 did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//simon-gpt-0'), PosixPath('8888/notebook/prdsafe/simon-gpt'), PosixPath('http')}
warn(msg)
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/home/jovyan/.local/share/autojump/errors.log')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 111
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Loading binary /home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so...
Overriding torch_dtype=None with torch_dtype=torch.float16 due to requirements of bitsandbytes to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [03:23<00:00, 6.16s/it]
Found cached dataset json (/home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.68s/it]
Loading cached split indices for dataset at /home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-514db5b55ef8a533.arrow and /home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-a2d8efccb09d7efe.arrow
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set no_deprecation_warning=True to disable this warning
warnings.warn(
0%| | 0/585 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/jovyan/simon-data/research/AI/gpt/facebook/japanese-alpaca-lora/finetune.py", line 118, in
trainer.train()
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 1628, in train
return inner_training_loop(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 1895, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 2637, in training_step
loss = self.compute_loss(model, inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 2669, in compute_loss
outputs = model(**inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
output.reraise()
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
output = module(*input, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/peft/peft_model.py", line 529, in forward
return self.base_model(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 770, in forward
outputs = self.model(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 619, in forward
layer_outputs = decoder_layer(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 316, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 217, in forward
key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 242, in forward
out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul
return MatMul8bitLt.apply(A, B, out, bias, state)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x5 and 4x4096)
how to solve
The text was updated successfully, but these errors were encountered:
╰─$ python finetune.py
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /home/jovyan/simon-data/home/miniconda3/envs/alpaca39 did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//simon-gpt-0'), PosixPath('8888/notebook/prdsafe/simon-gpt'), PosixPath('http')}
warn(msg)
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/home/jovyan/.local/share/autojump/errors.log')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 111
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Loading binary /home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so...
Overriding torch_dtype=None with
torch_dtype=torch.float16
due to requirements ofbitsandbytes
to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don't pass this argument at all to remove this warning.Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33/33 [03:23<00:00, 6.16s/it]
Found cached dataset json (/home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.68s/it]
Loading cached split indices for dataset at /home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-514db5b55ef8a533.arrow and /home/jovyan/.cache/huggingface/datasets/json/default-b21b7af1932b77a9/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51/cache-a2d8efccb09d7efe.arrow
/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set
no_deprecation_warning=True
to disable this warningwarnings.warn(
0%| | 0/585 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/jovyan/simon-data/research/AI/gpt/facebook/japanese-alpaca-lora/finetune.py", line 118, in
trainer.train()
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 1628, in train
return inner_training_loop(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 1895, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 2637, in training_step
loss = self.compute_loss(model, inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/trainer.py", line 2669, in compute_loss
outputs = model(**inputs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 171, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 181, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 89, in parallel_apply
output.reraise()
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 64, in _worker
output = module(*input, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/peft/peft_model.py", line 529, in forward
return self.base_model(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 770, in forward
outputs = self.model(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 619, in forward
layer_outputs = decoder_layer(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 316, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 217, in forward
key_states = self.k_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/nn/modules.py", line 242, in forward
out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 488, in matmul
return MatMul8bitLt.apply(A, B, out, bias, state)
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/jovyan/simon-data/home/miniconda3/envs/alpaca39/lib/python3.9/site-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
output += torch.matmul(subA, state.subB)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1024x5 and 4x4096)
how to solve
The text was updated successfully, but these errors were encountered: