2DPASS

This repository is for 2DPASS introduced in the following paper

Xu Yan*, Jiantao Gao*, Chaoda Zheng*, Chao Zheng, Ruimao Zhang, Shuguang Cui, Zhen Li*, "2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds", ECCV 2022 [arxiv].

If you find our work useful in your research, please consider citing:

@inproceedings{yan20222dpass,
  title={2dpass: 2d priors assisted semantic segmentation on lidar point clouds},
  author={Yan, Xu and Gao, Jiantao and Zheng, Chaoda and Zheng, Chao and Zhang, Ruimao and Cui, Shuguang and Li, Zhen},
  booktitle={European Conference on Computer Vision},
  pages={677--695},
  year={2022},
  organization={Springer}
}

@InProceedings{yan2022let,
      title={Let Images Give You More: Point Cloud Cross-Modal Training for Shape Analysis}, 
      author={Xu Yan and Heshen Zhan and Chaoda Zheng and Jiantao Gao and Ruimao Zhang and Shuguang Cui and Zhen Li},
      year={2022},
      booktitle={NeurIPS}
}

@article{yan2023benchmarking,
  title={Benchmarking the Robustness of LiDAR Semantic Segmentation Models},
  author={Yan, Xu and Zheng, Chaoda and Li, Zhen and Cui, Shuguang and Dai, Dengxin},
  journal={arXiv preprint arXiv:2301.00970},
  year={2023}
}

News

• 2023-04-01 We merge MinkowskiNet and official SPVCNN models from SPVNAS in our codebase. You can check these models in config/. We rename our baseline model from spvcnn.py to baseline.py.
• 2023-03-31 We provide codes for the robustness evaluation on SemanticKITTI-C.
• 2023-03-27 We release a model with higher performance on SemanticKITTI and codes for naive instance augmentation.
• 2023-02-25 We release a new robustness benchmark for LiDAR semantic segmentation at SemanticKITTI-C. Welcome to test your models!

• 2022-10-11 Our new work for cross-modal knowledge distillation is accepted at NeurIPS 2022:smiley: paper / code.
• 2022-09-20 We release codes for SemanticKITTI single-scan and NuScenes 🚀!
• 2022-07-03 2DPASS is accepted at ECCV 2022 🔥!
• 2022-03-08 We achieve 1st place in both single and multi-scans of SemanticKITTI and 3rd place on NuScenes-lidarseg 🔥!

Installation

Requirements

• pytorch >= 1.8
• yaml
• easydict
• pyquaternion
• lightning (tested with pytorch_lightning==1.3.8 and torchmetrics==0.5)
• torch-scatter (pip install torch-scatter -f https://data.pyg.org/whl/torch-1.9.0+${CUDA}.html)
• nuScenes-devkit (optional for nuScenes)
• spconv (tested with spconv==2.1.16 and cuda==11.1, pip install spconv-cu111==2.1.16)
• torchsparse (optional for MinkowskiNet and SPVCNN. sudo apt-get install libsparsehash-dev, pip install --upgrade git+https://github.com/mit-han-lab/[email protected])

Data Preparation

SemanticKITTI

Please download the files from the SemanticKITTI website and additionally the color data from the Kitti Odometry website. Extract everything into the same folder.

./dataset/
├── 
├── ...
└── SemanticKitti/
    ├──sequences
        ├── 00/           
        │   ├── velodyne/	
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 
        |   |   ├── 000000.label
        |   |   ├── 000001.label
        |   |   └── ...
        |   └── image_2/ 
        |   |   ├── 000000.png
        |   |   ├── 000001.png
        |   |   └── ...
        |   calib.txt
        ├── 08/ # for validation
        ├── 11/ # 11-21 for testing
        └── 21/
	    └── ...

NuScenes

Please download the Full dataset (v1.0) from the NuScenes website with lidarseg and extract it.

./dataset/
├── 
├── ...
└── nuscenes/
    ├──v1.0-trainval
    ├──v1.0-test
    ├──samples
    ├──sweeps
    ├──maps
    ├──lidarseg

Training

SemanticKITTI

You can run the training with

cd <root dir of this repo>
python main.py --log_dir 2DPASS_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0

The output will be written to logs/SemanticKITTI/2DPASS_semkitti by default.

NuScenes

cd <root dir of this repo>
python main.py --log_dir 2DPASS_nusc --config config/2DPASS-nuscenese.yaml --gpu 0 1 2 3

Vanilla Training without 2DPASS

We take SemanticKITTI as an example.

cd <root dir of this repo>
python main.py --log_dir baseline_semkitti --config config/2DPASS-semantickitti.yaml --gpu 0 --baseline_only

Testing

You can run the testing with

cd <root dir of this repo>
python main.py --config config/2DPASS-semantickitti.yaml --gpu 0 --test --num_vote 12 --checkpoint <dir for the pytorch checkpoint>

Here, num_vote is the number of views for the test-time-augmentation (TTA). We set this value to 12 as default (on a Tesla-V100 GPU), and if you use other GPUs with smaller memory, you can choose a smaller value. num_vote=1 denotes there is no TTA used, and will cause about ~2% performance drop.

Robustness Evaluation

Please download all subsets of SemanticKITTI-C from this link and extract them.

./dataset/
├── 
├── ...
└── SemanticKitti/
    ├──sequences
    ├──SemanticKITTI-C
        ├── clean_data/           
        ├── dense_16beam/           
        │   ├── velodyne/	
        |   |	├── 000000.bin
        |   |	├── 000001.bin
        |   |	└── ...
        │   └── labels/ 
        |   |   ├── 000000.label
        |   |   ├── 000001.label
        |   |   └── ...
	    ...

You can run the robustness evaluation with

cd <root dir of this repo>
python robust_test.py --config config/2DPASS-semantickitti.yaml --gpu 0  --num_vote 12 --checkpoint <dir for the pytorch checkpoint>

Model Zoo

You can download the models with the scores below from this Google drive folder.

SemanticKITTI

Model (validation)	mIoU (vanilla)	mIoU (TTA)	Parameters
MinkowskiNet	65.1%	67.1%	21.7M
SPVCNN	65.9%	67.8%	21.8M
2DPASS (4scale-64dimension)	68.7%	70.0%	1.9M
2DPASS (6scale-256dimension)	70.7%	72.0%	45.6M

Here, we fine-tune 2DPASS models on SemanticKITTI with more epochs and thus gain the higher mIoU. If you train with 64 epochs, it should be gained about 66%/69% for vanilla and 69%/71% after TTA.

NuScenes

Model (validation)	mIoU (vanilla)	mIoU (TTA)	Parameters
MinkowskiNet	74.3%	76.0%	21.7M
SPVCNN	74.9%	76.9%	21.8M
2DPASS (6scale-128dimension)	76.7%	79.6%	11.5M
2DPASS (6scale-256dimension)	78.0%	80.5%	45.6M

Note that the results on benchmarks are gained by training with additional validation set and using instance-level augmentation.

Acknowledgements

Code is built based on SPVNAS, Cylinder3D, xMUDA and SPCONV.

License

This repository is released under MIT License (see LICENSE file for details).