[Bug] FCOS3D train on kitti dataset

[Zzzouzou]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I have modified the scripts/configs, or I'm working on my own tasks/models/datasets.

Branch

1.x branch https://github.com/open-mmlab/mmdetection3d/tree/dev-1.x

Environment

win11
CUDA11.5 torch1.11

Reproduces the problem - code sample

python tools/train.py /mmdetection3d/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_kitti-mono3d.py

When I use this command, I find that the model training is not calling the gpu.

Reproduces the problem - command or script

Refer to this document for detailed configuration

#865

Reproduces the problem - error message

Additional information

I want to know if it's because windows doesn't support it ......If it is supported, please help me, I want to know how to enable gpu.

[Lemonade24510]

I'm facing the same issue. What should I do? @mickeyouyou @lbin @atinfinity @Zzzouzou

[atinfinity]

@Zzzouzou @Lemonade24510 Please check the information of your environment.

python mmdet3d/utils/collect_env.py

[Zzzouzou]

ok,It looks like this. @atinfinity

[atinfinity]

@Zzzouzou You use GeForce RTX 4080. The GPU Compute Capability of this GPU is 8.9(https://developer.nvidia.com/cuda-gpus). On the other hands, I found the following information in your log.

• PyTorch was build with CUDA 11.5
• There is no compute_89, sm_89 in NVCC architecture flag

PyTorch needs to be built with CUDA 11.8+ for your GPU.

NVIDIA Hopper and NVIDIA Ada architecture support

https://developer.nvidia.com/blog/cuda-toolkit-11-8-new-features-revealed/

I found Docker image from Docker Hub. But, I'm not sure if mmdetection3d supports PyTorch 2.x.
https://hub.docker.com/r/pytorch/pytorch/tags?name=11.8

[Zzzouzou]

Thanks for the reply！I tried the same thing on another computer（NVIDIA GeForce RTX 2060）, this is my environment configuration, and it seems to have the same gpu unused problem. @atinfinity

[atinfinity]

@Zzzouzou

• It seems that MMDetection3D version is a little old. Did you try the latest version?
• How do you check the usage of GPU while training? You can use nvidia-smi.
• Is there any error messages while training?

And, I found the following message in https://mmdetection3d.readthedocs.io/en/v1.3.0/get_started.html.

MMDetection3D works on Linux, Windows (experimental support) and macOS.

So, you may try the following approaches.

1. use WSL2(and NVIDIA Container Toolkit)
2. use Linux

[Zzzouzou]

ok,I see ,thank you again,there aren’t any error messages.emmm,maybe this vision is not work on windows（c+g）.

@atinfinity

[atinfinity]

@Zzzouzou It seems that training script use NVIDIA GPU. The "C" means "Compute".

C = Compute, which defines the processes that use the compute mode of Nvidia GPUs which use CUDA libraries, used in deep learning training and inferencing using Tensorflow-GPU, Pytorch, etc

https://stackoverflow.com/a/59375300