RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)

[YuzhouPeng]

Traceback (most recent call last):
File "test.py", line 68, in
num_query)
File "/home/pengyuzhou/workspace/TransReID/processor/processor.py", line 162, in do_inference
feat = model(img, cam_label=camids, view_label=target_view)
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pengyuzhou/workspace/TransReID/model/make_model.py", line 310, in forward
features = self.base(x, cam_label=cam_label, view_label=view_label)
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pengyuzhou/workspace/TransReID/model/backbones/vit_pytorch.py", line 414, in forward
x = self.forward_features(x, cam_label, view_label)
File "/home/pengyuzhou/workspace/TransReID/model/backbones/vit_pytorch.py", line 402, in forward_features
x = blk(x)
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pengyuzhou/workspace/TransReID/model/backbones/vit_pytorch.py", line 189, in forward
x = x + self.drop_path(self.attn(self.norm1(x)))
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pengyuzhou/workspace/TransReID/model/backbones/vit_pytorch.py", line 161, in forward
qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/home/pengyuzhou/miniconda3/envs/transreid/lib/python3.7/site-packages/torch/nn/functional.py", line 1692, in linear
output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)

I use custom data to train the model and try to test it, but got following error, can you tell me how to solve it?

[YuzhouPeng]

I meet this error during inference, but everything is ok during training, can you figure out how to solve it? thanks

[hfutzzw]

Same with me, I use custom data to train the model and try to test it, but got following error, can you tell me how to solve it?

[Hocason]

@YuzhouPeng @hfutzzw
Try to modify the pid and camid in the dataloader starting from 0.
Just like market1501 dataloader (trainset needs to relabel, and testset doesn't).