Meet RuntimeError in mmdet3d/ops/spconv/src/indice_cuda.cu #21

liuzili97 · 2020-07-15T03:12:27Z

Describe the bug
I follow the install.md and run pip install -v -e . without errors.
However, when I train those models which use spconv, I get an error. It seems like there is something wrong with my spconv.

Reproduction

Did you make any modifications on the code or config? Did you understand what you have modified?
No.

Environment

Please run python mmdet/utils/collect_env.py to collect necessary environment infomation and paste it here.
`Python: 3.6.10 |Anaconda, Inc.| (default, May 8 2020, 02:54:21) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda-10.2
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GPU 0,1,2,3: GeForce RTX 2080 Ti
GCC: gcc (GCC) 5.4.0
PyTorch: 1.5.0
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.5
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.6.0a0+82fd1c8
OpenCV: 4.3.0
MMCV: 1.0.2
MMDetection: 2.3.0rc0+af33f11
MMDetection Compiler: GCC 7.3
MMDetection CUDA Compiler: 10.2`

Error traceback

 File "/home/xx/anaconda3/envs/jio2pt1.5/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    input = module(input)
  File "/home/xx/anaconda3/envs/jio2pt1.5/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/conv.py", line 168, in forward
    result = self.forward(*input, **kwargs)
    result = self.forward(*input, **kwargs)
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/conv.py", line 168, in forward
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/conv.py", line 168, in forward
    result = self.forward(*input, **kwargs)
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/conv.py", line 168, in forward
    grid=input.grid)
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/ops.py", line 94, in get_indice_pairs
    grid=input.grid)
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/ops.py", line 94, in get_indice_pairs
    grid=input.grid)
    grid=input.grid)
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/ops.py", line 94, in get_indice_pairs
  File "/home/xx/fei2_workspace/mmdetection3d/tools/../mmdet3d/ops/spconv/ops.py", line 94, in get_indice_pairs
    int(transpose))
    int(transpose))
RuntimeError: /home/xx/fei2_workspace/mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu 124
cuda execution failed with error 2
    int(transpose))
RuntimeError: /home/xx/fei2_workspace/mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu 124
cuda execution failed with error 2
RuntimeError: /home/xx/fei2_workspace/mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu 124
cuda execution failed with error 2
    int(transpose))
RuntimeError: /home/xx/fei2_workspace/mmdetection3d/mmdet3d/ops/spconv/src/indice_cuda.cu 124
cuda execution failed with error 2

The text was updated successfully, but these errors were encountered:

ZwwWayne · 2020-07-15T14:24:26Z

Hi @liuzili97 ,
Thanks for your bug report. There are several things we need to know to accelerate the debugging process.

Did you modify the code of spconv?
What config are you using?
Can you check whether your PyTorch could run CUDA operation successfully on your GPU?
We never met this error before. From previous experience, it is possible that 1) the indices are not correct due to some reason; 2) the compilation arch is not correct; 3) some unknown things happen.

liuzili97 · 2020-07-16T01:44:37Z

Hi @liuzili97 ,
Thanks for your bug report. There are several things we need to know to accelerate the debugging process.

Did you modify the code of spconv?

What config are you using?

Can you check whether your PyTorch could run CUDA operation successfully on your GPU?
We never met this error before. From previous experience, it is possible that 1) the indices are not correct due to some reason; 2) the compilation arch is not correct; 3) some unknown things happen.

Thanks for your reply!

I didn't.
Pointpillar works well, but all other models which use spconv don't work and raise the error.
For example, I can train a pointpillar using configs/pointpillars/hv_pointpillars_secfpn_6x8_160ex2_kitti-3d-3class.py on GPU, but I'll meet the error when training other models like parta2 using configs/parta2/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-3class.py
I think CUDA works well since the pointpillar can be trained on my GPU.

I have tried deleting build and mmdet3d.egg-info and re-run pip install -v -e .. Still, no error is raised in compiling. However, the cuda problem still exists.

Tai-Wang · 2020-07-16T01:55:57Z

I briefly searched for some information about "cuda runtime error 2". It seems that it is a simple "out of memory" error, but I don't know why the error traceback did not elaborate it. I think you can try to reduce the number of "samples_per_gpu" in the config first.
I will have a try on my 2080Ti soon.

liuzili97 · 2020-07-16T02:12:25Z

I briefly searched for some information about "cuda runtime error 2". It seems that it is a simple "out of memory" error, but I don't know why the error traceback did not elaborate it. I think you can try to reduce the number of "samples_per_gpu" in the config first.
I will have a try on my 2080Ti soon.

Thanks! I have searched and tried to reduce the "samples_per_gpu", but still get the error.
By the way, I'll meet which: no hipcc in (/home/xx/fei2_workspace/gcc-5.4.0/install/gcc/bin:/home/xx/... right after I have allocated resources using the slurm. I don't know if it is related.

ZwwWayne · 2020-07-16T06:38:36Z

I briefly searched for some information about "cuda runtime error 2". It seems that it is a simple "out of memory" error, but I don't know why the error traceback did not elaborate it. I think you can try to reduce the number of "samples_per_gpu" in the config first.
I will have a try on my 2080Ti soon.

Thanks! I have searched and tried to reduce the "samples_per_gpu", but still get the error.
By the way, I'll meet which: no hipcc in (/home/xx/fei2_workspace/gcc-5.4.0/install/gcc/bin:/home/xx/... right after I have allocated resources using the slurm. I don't know if it is related.

which: no hipcc in (/home/xx/fei2_workspace/gcc-5.4.0/install/gcc/bin:/home/xx/.. is just a warning of PyTorch 1.5, it is not related to this issue.

Tai-Wang · 2020-07-16T13:05:34Z

I briefly searched for some information about "cuda runtime error 2". It seems that it is a simple "out of memory" error, but I don't know why the error traceback did not elaborate it. I think you can try to reduce the number of "samples_per_gpu" in the config first.
I will have a try on my 2080Ti soon.

Thanks! I have searched and tried to reduce the "samples_per_gpu", but still get the error.
By the way, I'll meet which: no hipcc in (/home/xx/fei2_workspace/gcc-5.4.0/install/gcc/bin:/home/xx/... right after I have allocated resources using the slurm. I don't know if it is related.

I reconfigured my environment and tried to reproduce your bugs but unfortunately failed. My environment is as follows:

Python: 3.6.10 |Anaconda, Inc.| (default, May 8 2020, 02:54:21) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GPU 0,1,2,3,4,5,6,7: GeForce RTX 2080 Ti
GCC: gcc (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010
PyTorch: 1.5.0
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.5
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.6.0a0+82fd1c8
OpenCV: 4.3.0
MMCV: 1.0.2
MMDetection: 2.3.0rc0+d613f21
MMDetection Compiler: GCC 7.3
MMDetection CUDA Compiler: 10.2

The only differences include different commits of mmdetection from the same version and the version of gcc. Actually different commits of mmdetection should not produce this bug and I also tried to use gcc-5.4 to build before (in another machine).
So my current suggestions are:

You can upgrade the gcc to gcc-5.5 and have a try even though it does not seem to be the most likely reason. But it should be the only difference in the environment that can make sense.
Please run python mmdet3d/utils/collect_env.py in the mmdetection3d to see whether there is any problems in the mmdet3d environments. You can take my output as a reference.

Python: 3.6.10 |Anaconda, Inc.| (default, May 8 2020, 02:54:21) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GPU 0,1,2,3,4,5,6,7: GeForce RTX 2080 Ti
GCC: gcc (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010
PyTorch: 1.5.0
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.2
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.5
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.6.0a0+82fd1c8
OpenCV: 4.3.0
MMCV: 1.0.2
MMDetection: 2.3.0rc0+d613f21
MMDetection3D: 0.5.0+8733947
MMDetection3D Compiler: GCC 5.5
MMDetection3D CUDA Compiler: 10.2

At last maybe you can go through the installation tutorial and check whether there is any tricky things lost and reinstall it.

Hope suggestions can help you. If others come across this bug, welcome further discussion here.

liuzili97 · 2020-07-16T13:55:41Z

@Tai-Wang @ZwwWayne
Thanks, I find that it is just because of the runtime problem (out-of-memory).
I use the following code to allocate the gpu memory before initializing the model to prevent multiple people using a single gpu.
The code works well on many toolboxes(e.g. mmdetection). However, I don't know why it causes the out-of-memory problem when using spconv.

def occupy_mem():
    this_device = torch.cuda.current_device()
    _, free = check_mem(this_device)
    block_mem = int(free * 0.95)
    x = torch.cuda.FloatTensor(256, 1024, block_mem)
    del x

ZwwWayne · 2020-07-16T14:43:56Z

This might simply because spconv uses more memory. However, this is beyond what we can help.

The issue is closed because the bug is find and unrelated to the codebase itself. Feel free to reopen it if you have any further questions.

happinesslz · 2020-07-22T07:32:37Z

@ZwwWayne I also meet the same RuntimeError with @liuzili97 . I can only run the pointpillar. Whether the spconv causes the error. I also try to reduce the number of "samples_per_gpu" to 1. But still get the same error.

My environment is :

`sys.platform: linux
Python: 3.7.6 (default, Jan 8 2020, 19:59:22) [GCC 7.3.0]
CUDA available: True
CUDA_HOME: /usr/local/cuda-10.1
NVCC: Cuda compilation tools, release 10.1, V10.1.168
GPU 0,1: TITAN V
GCC: gcc (GCC) 5.4.0
PyTorch: 1.5.1
PyTorch compiling details: PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.1 Product Build 20200208 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.3
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

TorchVision: 0.6.0a0+35d732a
OpenCV: 4.3.0
MMCV: 1.0.2
MMDetection: 2.3.0rc0+3c21dd0
MMDetection3D: 0.5.0+unknown
MMDetection3D Compiler: GCC 5.4
MMDetection3D CUDA Compiler: 10.1
`

Liaoqing-up · 2021-04-22T10:26:51Z

Hi @liuzili97 ,
Thanks for your bug report. There are several things we need to know to accelerate the debugging process.

Did you modify the code of spconv?

What config are you using?

Can you check whether your PyTorch could run CUDA operation successfully on your GPU?
We never met this error before. From previous experience, it is possible that 1) the indices are not correct due to some reason; 2) the compilation arch is not correct; 3) some unknown things happen.

Thanks for your reply!

I didn't.

Pointpillar works well, but all other models which use spconv don't work and raise the error.
For example, I can train a pointpillar using configs/pointpillars/hv_pointpillars_secfpn_6x8_160ex2_kitti-3d-3class.py on GPU, but I'll meet the error when training other models like parta2 using configs/parta2/hv_PartA2_secfpn_2x8_cyclic_80e_kitti-3d-3class.py

I think CUDA works well since the pointpillar can be trained on my GPU.

I have tried deleting build and mmdet3d.egg-info and re-run pip install -v -e .. Still, no error is raised in compiling. However, the cuda problem still exists.

hello, i want to use pointpillars, but i get the error: ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' (/opt/sdatmp/lq/project/gitproject/mmdetection3d/mmdet3d/ops/ball_query/init.py).
It seems come from these error when i run pip install -v -e . to install MMDetection3D:
warning: no files found matching 'mmdet3d/ops//.cpp'
warning: no files found matching 'mmdet3d/ops//.cu'
warning: no files found matching 'mmdet3d/ops//.h'
warning: no files found matching 'mmdet3d/ops//.cc'

Have you encounter these errors before successfully running pointpillars? I need your help, thankyou

rockywind · 2021-07-29T12:14:53Z

@liuzili97 What's the implementation of the function "check_mem"?

* add register for ort custom op * ort_mmcv_util->ort_util * ortApi->kOrtApi

* modified device selection device cannot sucessfully control through argments "device" * update with_sync * Update ORTWrapper change the way to create ort session, previous work would load same model twice. * Update wrapper.py fixed for lint * Update wrapper.py * Update wrapper.py remove the backslash * formating using yapf to format the file Co-authored-by: AllentDan <[email protected]>

GODBENGAY · 2024-04-08T08:07:55Z

@Tai-Wang @ZwwWayne Thanks, I find that it is just because of the runtime problem (out-of-memory). I use the following code to allocate the gpu memory before initializing the model to prevent multiple people using a single gpu. The code works well on many toolboxes(e.g. mmdetection). However, I don't know why it causes the out-of-memory problem when using spconv.
def occupy_mem():
    this_device = torch.cuda.current_device()
    _, free = check_mem(this_device)
    block_mem = int(free * 0.95)
    x = torch.cuda.FloatTensor(256, 1024, block_mem)
    del x

thanks for your code. But i want to know where to put it

maxiuw · 2024-07-30T12:54:39Z

Similar issue
return get_indice_pairs_func(indices, batch_size, out_shape, RuntimeError: /tmp/mmcv/mmcv/ops/csrc/pytorch/cuda/sparse_indice.cu 126

reduced batch size eliminated the issue

ZwwWayne assigned Tai-Wang Jul 15, 2020

ZwwWayne closed this as completed Jul 16, 2020

happinesslz mentioned this issue Jul 22, 2020

Meet RuntimeError in mmdet3d/ops/spconv/src/indice_cuda.cu #37

Closed

Tai-Wang mentioned this issue Aug 26, 2020

RuntimeError:mmdet3d/ops/spconv/src/indice_cuda.cu 124 #80

Closed

lji72 mentioned this issue Apr 15, 2021

ImvoteNet 2-stage model reproduce error #448

Closed

yezhen17 mentioned this issue Sep 3, 2021

cuda execution failed with error 2 in SECOND #909

Closed

Tai-Wang mentioned this issue Sep 12, 2021

RuntimeError: mmdet3d/ops/spconv/src/indice_cuda.cu 124 #929

Closed

AlmoonYsl mentioned this issue Oct 27, 2021

Docker file for rtx3090 #923

Closed

tpoisonooo pushed a commit to tpoisonooo/mmdetection3d that referenced this issue Sep 5, 2022

[Feature] add register for ort custom op (open-mmlab#21)

04a8046

* add register for ort custom op * ort_mmcv_util->ort_util * ortApi->kOrtApi

mendoza-G mentioned this issue Mar 13, 2023

[Bug] CUDA error: an illegal memory access was encountered when training MVX-NET kitti #2209

Open

3 tasks

jumptiger66 mentioned this issue Apr 25, 2023

Setting --gpu-id 3 when i train ScanNet on Votenet ，i got error #2471

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meet RuntimeError in mmdet3d/ops/spconv/src/indice_cuda.cu #21

Meet RuntimeError in mmdet3d/ops/spconv/src/indice_cuda.cu #21

liuzili97 commented Jul 15, 2020

ZwwWayne commented Jul 15, 2020

liuzili97 commented Jul 16, 2020

Tai-Wang commented Jul 16, 2020

liuzili97 commented Jul 16, 2020

ZwwWayne commented Jul 16, 2020

Tai-Wang commented Jul 16, 2020 •

edited

Loading

liuzili97 commented Jul 16, 2020 •

edited

Loading

ZwwWayne commented Jul 16, 2020

happinesslz commented Jul 22, 2020

Liaoqing-up commented Apr 22, 2021

rockywind commented Jul 29, 2021

GODBENGAY commented Apr 8, 2024

maxiuw commented Jul 30, 2024

Meet RuntimeError in mmdet3d/ops/spconv/src/indice_cuda.cu #21

Meet RuntimeError in mmdet3d/ops/spconv/src/indice_cuda.cu #21

Comments

liuzili97 commented Jul 15, 2020

ZwwWayne commented Jul 15, 2020

liuzili97 commented Jul 16, 2020

Tai-Wang commented Jul 16, 2020

liuzili97 commented Jul 16, 2020

ZwwWayne commented Jul 16, 2020

Tai-Wang commented Jul 16, 2020 • edited Loading

liuzili97 commented Jul 16, 2020 • edited Loading

ZwwWayne commented Jul 16, 2020

happinesslz commented Jul 22, 2020

Liaoqing-up commented Apr 22, 2021

rockywind commented Jul 29, 2021

GODBENGAY commented Apr 8, 2024

maxiuw commented Jul 30, 2024

Tai-Wang commented Jul 16, 2020 •

edited

Loading

liuzili97 commented Jul 16, 2020 •

edited

Loading