RBGNet

This is the official implementation of paper:

RBGNet: Ray-based Grouping for 3D Object Detection

PaperLink

NEWS

• [22-8-25] Official implementation based on OpenPCDet is released.
• 🔥 RBGNet is accepted at CVPR 2022.

Introduciton

This paper proposes the RBGNet framework, a voting-based 3D detector for accurate 3D object detection from point clouds. In order to learn better representations of object shape to enhance cluster features for predicting 3D boxes, we propose a ray-based feature grouping module, which aggregates the point-wise features on object surfaces using a group of determined rays uniformly emitted from cluster centers. Considering the fact that foreground points are more meaningful for box estimation, we design a novel foreground biased sampling strategy in downsample process to sample more points on object surfaces and further boost the detection performance.

Installation

Our implementation is based on OpenPCDet, so just follow their Installation and Getting_Started.

Requirements

All the codes are tested in the following environment:

• Linux (tested on Ubuntu 14.04/16.04/18.04/20.04/21.04)
• Python 3.6+
• PyTorch 1.1 or higher (tested on PyTorch 1.1, 1,3, 1,5~1.10)
• CUDA 9.0 or higher (PyTorch 1.3+ needs CUDA 9.2+)

Install pcdet v0.5

NOTE: Please re-install pcdet v0.5 by running python setup.py develop even if you have already installed previous version.

a. Clone this repository.

git clone https://github.com/Haiyang-W/RBGNet.git

b. Install the dependent libraries as follows:

• Install the SparseConv library, we use the implementation from [spconv].
  • If you use PyTorch 1.1, then make sure you install the spconv v1.0 with (commit 8da6f96) instead of the latest one.
  • If you use PyTorch 1.3+, then you need to install the spconv v1.2. As mentioned by the author of spconv, you need to use their docker if you use PyTorch 1.4+.
  • You could also install latest spconv v2.x with pip, see the official documents of spconv.

c. Install this pcdet library and its dependent libraries by running the following command:

python setup.py develop

Data Preparation

We haven't achieved compatibility with the generated data of OpenPCDet yet and use the same data format as mmdeteciton3d for now. We will try to implement indoor data pre-processing based on OpenPCDet as soon as possible. Note this is based on mmdet3d 0.15, mmdet3d >=1.0 will change the coordinate system. You can download the pre-processed data directly here.

ScanNet V2

Please install mmdeteciton3d first and follow the data preparation ScanNet V2. Then link the generated data as follows:

ln -s ${mmdet3d_scannet_dir} ./RBGNet/data/scannet

SUN RGB-D

Please install mmdeteciton3d first and follow the data preparation Sun RGB-D. Then link the generated data as follows:

ln -s ${mmdet3d_sunrgbd_dir} ./RBGNet/data/sunrgbd

Training and Tesing

All the models are trained and evalutation on 4 GPUs.

ScanNet V2

Training with Eval

CUDA_VISIBLE_DEVICES=0,1,2,3 sh scripts/dist_train.sh 4 --cfg_file ./cfgs/scannet_models/RBGNet.yaml --ckpt_save_interval 1 --num_epochs_to_eval 15

Testing

Only support batch size = 1

CUDA_VISIBLE_DEVICES=0,1,2,3 sh scripts/dist_test.sh 4 --cfg_file ./cfgs/scannet_models/RBGNet.yaml --batch_size 4 --ckpt ../output/cfgs/scannet_models/RBGNet/default/ckpt/checkpoint_epoch_${epochid}.pth

Sun RGB-D

Training with Eval

CUDA_VISIBLE_DEVICES=0,1,2,3 sh scripts/dist_train.sh 4 --cfg_file ./cfgs/sunrgbd_models/RBGNet.yaml --ckpt_save_interval 1 --num_epochs_to_eval 15

Testing

Only support batch size = 1

CUDA_VISIBLE_DEVICES=0,1,2,3 sh scripts/dist_test.sh 4 --cfg_file ./cfgs/sunrgbd_models/RBGNet.yaml --batch_size 4 --ckpt ../output/cfgs/sunrgbd_models/RBGNet/default/ckpt/checkpoint_epoch_${epochid}.pth

Main Results

All models are trained with 4 3090 GPUs and the pretrained models will be released soon.

Dataset	[email protected]	mAP0.50	Pretrain Model (will soon)
ScanNet (Ray-66)	70.2(69.6)	54.2(53.6)	model
Sun RGB-D (Ray-66)	64.1(63.6)	47.2(46.3)	model

Citation

Please consider citing our work as follows if it is helpful.

@inproceedings{wang2022rbgnet,
  title={RBGNet: Ray-based Grouping for 3D Object Detection},
  author={Wang, Haiyang and Shi, Shaoshuai and Yang, Ze and Fang, Rongyao and Qian, Qi and Li, Hongsheng and Schiele, Bernt and Wang, Liwei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={1110--1119},
  year={2022}
}

Acknowledgments

This project is based on the following codebases.

• OpenPCDet
• VoteNet