Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeEror: cublas runtie error : the pu progran failed to execute at /opt/conda/conda-bld/pytorch 1556653099582/work/aten/src/THC/THCBlas.cu:259 #394

Open
RhythmOfTheRain-Byte opened this issue Jun 28, 2023 · 0 comments

Comments

@RhythmOfTheRain-Byte
Copy link

Hello, I ran centerpoint using Docker Image zhiminc/centerpoint_new and
encountered the following issues:

File "/root/CenterPoint/det3d/torchie/trainer/trainer.py", line 543, in runepoch runner(data loaders[i],self.epoch,**kwargs)File "/root/CenterPoint/det3d/torchie/trainer/trainer.py", line 410, in traindata batch train mode=True. **kwargsself .model.File "/root/CenterPoint/det3d/torchie/trainer/trainer.py", line 368, in batch processor inlinemodel(example.return loss=True)losses =File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in callresult = self.forward(*input,**kwargs)File "/root/CenterPoint/det3d/models/detectors/voxelnet.py", line 49, in forward= self.extract feat(data)File "/root/centerpoint/det3d/models/detectors/yoxelnet,py"i line 28pein extract_featinput features ,data["coors"],data["batch size"]File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in callresult = self.forward(*input ,**kwarqsFile "/root/CenterPoint/det3d/models/backbones/scn.py"line 156,in forwardx = self.conv input(ret]File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in _callresult = self.forward(*input,**kwargs )File "/opt/conda/lib/python3.6/site-packages/spconv/modules.py", line 123, in forwardinput = module(input)File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in _callresult = self.forward(*input,**kwargs )File "/opt/conda/lib/python3.6/site-packages/spconv/conv.py", line 161, in forwardoutids .shape[o])File "/opt/conda/lib/python3.6/site-packages/spconv/functional.py", line 83, in forwardreturn opsindice convlfeatures, filters, indice pairs. indice pair num, nun activate out, False, True)File "/opt/conda/lib/python3.6/site-packages/spconv/ops.py", line 112, in indice convint(inverse) , int(subm) )
RuntimeEror: cublas runtie error : the pu progran failed to execute at /opt/conda/conda-bld/pytorch 1556653099582/work/aten/src/THC/THCBlas.cu:259

"/opt/conda/lib/python3.6/site-packages/spconv/ops.py", line 112, in indice convint(inverse) , int(subm) ) coda is as fellows:
in spconv/ops.py
if filters.dtype == torch .float32:
return torch.ops.spconv.indice conv_fp32features, filters, indice_pairs,
indice pair num, num activate out,
int(inverse),int(subm))

env info:
pytorch 1.1.0 py3.6_cuda10.0.130_cudnn7.5.1_0
torchvision 0.2.2
spconv 1.0
with RTX A6000 on my machine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant