Add fast indoor models (#29)

* add fast configs for 3 x sunrgbd and scannet

* add new model links toreadme

* remove extra prints

README.md

@@ -4,7 +4,7 @@
 # ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection
 
 **News**:
- * :fire: October, 2021. Our paper is accepted at [WACV 2022](https://wacv2022.thecvf.com). Stay tuned for 5 times faster and more accurate models for indoor datasets.
+ * :fire: October, 2021. Our paper is accepted at [WACV 2022](https://wacv2022.thecvf.com). We simplify 3d neck to make indoor models much faster and accurate. For example, this improves `ScanNet` `mAP` by more than 2%. Please find updated configs in [configs/imvoxelnet/*_fast.py](https://github.com/saic-vul/imvoxelnet/tree/master/configs/imvoxelnet) and [models](https://github.com/saic-vul/imvoxelnet/releases/tag/v1.2).
  * :fire: August, 2021. We adapt center sampling for indoor detection. For example, this improves `ScanNet` `mAP` by more than 5%. Please find updated configs in [configs/imvoxelnet/*_top27.py](https://github.com/saic-vul/imvoxelnet/tree/master/configs/imvoxelnet) and [models](https://github.com/saic-vul/imvoxelnet/releases/tag/v1.1).
  * :fire: July, 2021. We update `ScanNet` image preprocessing both [here](https://github.com/saic-vul/imvoxelnet/pull/21) and in [mmdetection3d](https://github.com/open-mmlab/mmdetection3d/pull/696).
  * :fire: June, 2021. `ImVoxelNet` for `KITTI` is now [supported](https://github.com/open-mmlab/mmdetection3d/tree/master/configs/imvoxelnet) in [mmdetection3d](https://github.com/open-mmlab/mmdetection3d).
@@ -87,14 +87,16 @@ python tools/test.py configs/imvoxelnet/imvoxelnet_kitti.py \
 
 ### Models
 
-| Dataset   | Object Classes | Center Sampling | Download |
-|:---------:|:--------------:|:---------------:|:--------:|
-| SUN RGB-D | 37 from Total3dUnderstanding | &#x2718; <br> &#x2714; | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210525_091810.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210525_091810_atlas_total_sunrgbd.log) &#124; [config](configs/imvoxelnet/imvoxelnet_total_sunrgbd.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_005013.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_005013_imvoxelnet_total_sunrgbd_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_total_sunrgbd_top27.py)|
-| SUN RGB-D | 30 from PerspectiveNet | &#x2718; <br> &#x2714; | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210526_072029.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210526_072029_atlas_perspective_sunrgbd.log) &#124; [config](configs/imvoxelnet/imvoxelnet_perspective_sunrgbd.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_114832.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_114832_imvoxelnet_perspective_sunrgbd_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_perspective_sunrgbd_top27.py)|
-| SUN RGB-D | 10 from VoteNet | &#x2718; <br> &#x2714; | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210428_124351.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210428_124351_atlas_sunrgbd.log) &#124; [config](configs/imvoxelnet/imvoxelnet_sunrgbd.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_112435.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_112435_imvoxelnet_sunrgbd_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_sunrgbd_top27.py)|
-| ScanNet   | 18 from VoteNet | &#x2718; <br> &#x2714; | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210520_223109.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210520_223109_atlas_scannet.log) &#124; [config](configs/imvoxelnet/imvoxelnet_scannet.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_070616.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_070616_imvoxelnet_scannet_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_scannet_top27.py)|
-| KITTI     | Car | &#x2718; | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210503_214214.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210503_214214_atlas_kitti.log) &#124; [config](configs/imvoxelnet/imvoxelnet_kitti.py) |
-| nuScenes  | Car | &#x2718; | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210505_131108.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210505_131108_atlas_nuscenes.log) &#124; [config](configs/imvoxelnet/imvoxelnet_nuscenes.py) |
+`v2` adds center sampling for indoor scenario. `v3` simplifies 3d neck for indoor scenario. Differences are discussed in [v2](https://arxiv.org/abs/2106.01178v2) and [v3](https://arxiv.org/abs/2106.01178v3) preprints.
+
+| Dataset   | Object Classes | Version | Download |
+|:---------:|:--------------:|:-------:|:--------:|
+| SUN RGB-D | 37 from <br> Total3dUnderstanding | v1 &#124; [email protected]: 41.5 <br> v2 &#124; [email protected]: 42.7 <br> v3 &#124; [email protected]: 43.7 | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210525_091810.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210525_091810_atlas_total_sunrgbd.log) &#124; [config](configs/imvoxelnet/imvoxelnet_total_sunrgbd.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_005013.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_005013_imvoxelnet_total_sunrgbd_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_total_sunrgbd_top27.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_105247.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_105247_imvoxelnet_total_sunrgbd_fast.log) &#124; [config](configs/imvoxelnet/imvoxelnet_total_sunrgbd_fast.py)|
+| SUN RGB-D | 30 from <br> PerspectiveNet | v1 &#124; [email protected]: 44.9 <br> v2 &#124;  [email protected]: 47.2 <br> v3 &#124; [email protected]: 48.7 | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210526_072029.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210526_072029_atlas_perspective_sunrgbd.log) &#124; [config](configs/imvoxelnet/imvoxelnet_perspective_sunrgbd.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_114832.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_114832_imvoxelnet_perspective_sunrgbd_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_perspective_sunrgbd_top27.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_105254.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_105254_imvoxelnet_perspective_sunrgbd_fast.log) &#124; [config](configs/imvoxelnet/imvoxelnet_perspective_sunrgbd_fast.py)|
+| SUN RGB-D | 10 from VoteNet | v1 &#124; [email protected]: 38.8 <br> v2 &#124;  [email protected]: 39.4 <br> v3 &#124; [email protected]: 40.7 | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210428_124351.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210428_124351_atlas_sunrgbd.log) &#124; [config](configs/imvoxelnet/imvoxelnet_sunrgbd.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_112435.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210809_112435_imvoxelnet_sunrgbd_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_sunrgbd_top27.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_105255.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_105255_imvoxelnet_sunrgbd_fast.log) &#124; [config](configs/imvoxelnet/imvoxelnet_sunrgbd_fast.py)|
+| ScanNet   | 18 from VoteNet | v1 &#124; [email protected]: 40.6 <br> v2 &#124;  [email protected]: 45.7 <br> v3 &#124; [email protected]: 48.1 | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210520_223109.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210520_223109_atlas_scannet.log) &#124; [config](configs/imvoxelnet/imvoxelnet_scannet.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_070616.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.1/20210808_070616_imvoxelnet_scannet_top27.log) &#124; [config](configs/imvoxelnet/imvoxelnet_scannet_top27.py) <br> [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_113826.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.2/20211007_113826_imvoxelnet_scannet_fast.log) &#124; [config](configs/imvoxelnet/imvoxelnet_scannet_fast.py)|
+| KITTI     | Car | v1 &#124; [email protected]: 17.8 | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210503_214214.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210503_214214_atlas_kitti.log) &#124; [config](configs/imvoxelnet/imvoxelnet_kitti.py) |
+| nuScenes  | Car | v1 &#124; AP: 51.8 | [model](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210505_131108.pth) &#124; [log](https://github.com/saic-vul/imvoxelnet/releases/download/v1.0/20210505_131108_atlas_nuscenes.log) &#124; [config](configs/imvoxelnet/imvoxelnet_nuscenes.py) |
 
 ### Example Detections
 

configs/imvoxelnet/imvoxelnet_perspective_sunrgbd_fast.py

@@ -0,0 +1,127 @@
+model = dict(
+    type='ImVoxelNet',
+    pretrained='torchvision://resnet50',
+    backbone=dict(
+        type='ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=1,
+        norm_cfg=dict(type='BN', requires_grad=False),
+        norm_eval=True,
+        style='pytorch'),
+    neck=dict(
+        type='FPN',
+        in_channels=[256, 512, 1024, 2048],
+        out_channels=256,
+        num_outs=4),
+    neck_3d=dict(
+        type='FastIndoorImVoxelNeck',
+        in_channels=256,
+        out_channels=128,
+        n_blocks=[1, 1, 1]),
+    bbox_head=dict(
+        type='SunRgbdImVoxelHeadV2',
+        n_classes=30,
+        n_channels=128,
+        n_reg_outs=7,
+        n_scales=3,
+        limit=27,
+        centerness_topk=18),
+    n_voxels=(40, 40, 16),
+    voxel_size=(.16, .16, .16))
+train_cfg = dict()
+test_cfg = dict(
+    nms_pre=1000,
+    nms_thr=.15,
+    use_rotate_nms=True,
+    score_thr=.01)
+img_norm_cfg = dict(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
+
+dataset_type = 'SunRgbdPerspectiveMultiViewDataset'
+data_root = 'data/sunrgbd/'
+class_names = ('recycle_bin', 'cpu', 'paper', 'toilet', 'stool', 'whiteboard', 'coffee_table', 'picture',
+               'keyboard', 'dresser', 'painting', 'bookshelf', 'night_stand', 'endtable', 'drawer', 'sink',
+               'monitor', 'computer', 'cabinet', 'shelf', 'lamp', 'garbage_bin', 'box', 'bed', 'sofa',
+               'sofa_chair', 'pillow', 'desk', 'table', 'chair')
+
+train_pipeline = [
+    dict(type='LoadAnnotations3D'),
+    dict(
+        type='MultiViewPipeline',
+        n_images=1,
+        transforms=[
+            dict(type='LoadImageFromFile'),
+            dict(type='RandomFlip', flip_ratio=0.5),
+            dict(type='Resize', img_scale=[(512, 384), (768, 576)], multiscale_mode='range', keep_ratio=True),
+            dict(type='Normalize', **img_norm_cfg),
+            dict(type='Pad', size_divisor=32)]),
+    dict(type='SunRgbdRandomFlip'),
+    dict(type='DefaultFormatBundle3D', class_names=class_names),
+    dict(type='Collect3D', keys=['img', 'gt_bboxes_3d', 'gt_labels_3d'])]
+test_pipeline = [
+    dict(
+        type='MultiViewPipeline',
+        n_images=1,
+        transforms=[
+            dict(type='LoadImageFromFile'),
+            dict(type='Resize', img_scale=(640, 480), keep_ratio=True),
+            dict(type='Normalize', **img_norm_cfg),
+            dict(type='Pad', size_divisor=32)]),
+    dict(type='DefaultFormatBundle3D', class_names=class_names, with_label=False),
+    dict(type='Collect3D', keys=['img'])]
+data = dict(
+    samples_per_gpu=4,
+    workers_per_gpu=4,
+    train=dict(
+        type='RepeatDataset',
+        times=2,
+        dataset=dict(
+            type=dataset_type,
+            data_root=data_root,
+            ann_file=data_root + 'sunrgbd_perspective_infos_train.pkl',
+            pipeline=train_pipeline,
+            classes=class_names,
+            filter_empty_gt=True,
+            box_type_3d='Depth')),
+    val=dict(
+        type=dataset_type,
+        data_root=data_root,
+        ann_file=data_root + 'sunrgbd_perspective_infos_val.pkl',
+        pipeline=test_pipeline,
+        classes=class_names,
+        test_mode=True,
+        box_type_3d='Depth'),
+    test=dict(
+        type=dataset_type,
+        data_root=data_root,
+        ann_file=data_root + 'sunrgbd_perspective_infos_val.pkl',
+        pipeline=test_pipeline,
+        classes=class_names,
+        test_mode=True,
+        box_type_3d='Depth'))
+
+optimizer = dict(
+    type='AdamW',
+    lr=0.0001,
+    weight_decay=0.0001,
+    paramwise_cfg=dict(
+        custom_keys={'backbone': dict(lr_mult=0.1, decay_mult=1.0)}))
+optimizer_config = dict(grad_clip=dict(max_norm=35., norm_type=2))
+lr_config = dict(policy='step', step=[8, 11])
+total_epochs = 12
+
+checkpoint_config = dict(interval=1, max_keep_ckpts=1)
+log_config = dict(
+    interval=50,
+    hooks=[
+        dict(type='TextLoggerHook'),
+        dict(type='TensorboardLoggerHook')
+    ])
+evaluation = dict(interval=1)
+dist_params = dict(backend='nccl')
+find_unused_parameters = True  # todo: fix number of FPN outputs
+log_level = 'INFO'
+load_from = None
+resume_from = None
+workflow = [('train', 1)]

configs/imvoxelnet/imvoxelnet_scannet_fast.py

@@ -0,0 +1,131 @@
+model = dict(
+    type='ImVoxelNet',
+    pretrained='torchvision://resnet50',
+    backbone=dict(
+        type='ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=1,
+        norm_cfg=dict(type='BN', requires_grad=False),
+        norm_eval=True,
+        style='pytorch'),
+    neck=dict(
+        type='FPN',
+        in_channels=[256, 512, 1024, 2048],
+        out_channels=256,
+        num_outs=4),
+    neck_3d=dict(
+        type='FastIndoorImVoxelNeck',
+        in_channels=256,
+        out_channels=128,
+        n_blocks=[1, 1, 1]),
+    bbox_head=dict(
+        type='ScanNetImVoxelHeadV2',
+        loss_bbox=dict(type='AxisAlignedIoULoss', loss_weight=1.0),
+        n_classes=18,
+        n_channels=128,
+        n_reg_outs=6,
+        n_scales=3,
+        limit=27,
+        centerness_topk=18),
+    voxel_size=(.16, .16, .16),
+    n_voxels=(40, 40, 16))
+train_cfg = dict()
+test_cfg = dict(
+    nms_pre=1000,
+    iou_thr=.25,
+    score_thr=.01)
+img_norm_cfg = dict(mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
+
+dataset_type = 'ScanNetMultiViewDataset'
+data_root = 'data/scannet/'
+class_names = ('cabinet', 'bed', 'chair', 'sofa', 'table', 'door', 'window',
+               'bookshelf', 'picture', 'counter', 'desk', 'curtain',
+               'refrigerator', 'showercurtrain', 'toilet', 'sink', 'bathtub',
+               'garbagebin')
+
+train_pipeline = [
+    dict(type='LoadAnnotations3D'),
+    dict(
+        type='MultiViewPipeline',
+        n_images=20,
+        transforms=[
+            dict(type='LoadImageFromFile'),
+            dict(type='Resize', img_scale=(640, 480), keep_ratio=True),
+            dict(type='Normalize', **img_norm_cfg),
+            dict(type='Pad', size=(480, 640))
+        ]),
+    dict(type='RandomShiftOrigin', std=(.7, .7, .0)),
+    dict(type='DefaultFormatBundle3D', class_names=class_names),
+    dict(type='Collect3D', keys=['img', 'gt_bboxes_3d', 'gt_labels_3d'])
+]
+test_pipeline = [
+    dict(
+        type='MultiViewPipeline',
+        n_images=50,
+        transforms=[
+            dict(type='LoadImageFromFile'),
+            dict(type='Resize', img_scale=(640, 480), keep_ratio=True),
+            dict(type='Normalize', **img_norm_cfg),
+            dict(type='Pad', size=(480, 640))
+        ]),
+    dict(type='DefaultFormatBundle3D', class_names=class_names, with_label=False),
+    dict(type='Collect3D', keys=['img'])
+]
+data = dict(
+    samples_per_gpu=1,
+    workers_per_gpu=1,
+    train=dict(
+        type='RepeatDataset',
+        times=3,
+        dataset=dict(
+            type=dataset_type,
+            data_root=data_root,
+            ann_file=data_root + 'scannet_infos_train.pkl',
+            pipeline=train_pipeline,
+            classes=class_names,
+            filter_empty_gt=True,
+            box_type_3d='Depth')),
+    val=dict(
+        type=dataset_type,
+        data_root=data_root,
+        ann_file=data_root + 'scannet_infos_val.pkl',
+        pipeline=test_pipeline,
+        classes=class_names,
+        test_mode=True,
+        box_type_3d='Depth'),
+    test=dict(
+        type=dataset_type,
+        data_root=data_root,
+        ann_file=data_root + 'scannet_infos_val.pkl',
+        pipeline=test_pipeline,
+        classes=class_names,
+        test_mode=True,
+        box_type_3d='Depth')
+)
+
+optimizer = dict(
+    type='AdamW',
+    lr=0.0001,
+    weight_decay=0.0001,
+    paramwise_cfg=dict(
+        custom_keys={'backbone': dict(lr_mult=0.1, decay_mult=1.0)}))
+optimizer_config = dict(grad_clip=dict(max_norm=35., norm_type=2))
+lr_config = dict(policy='step', step=[8, 11])
+total_epochs = 12
+
+checkpoint_config = dict(interval=1, max_keep_ckpts=1)
+log_config = dict(
+    interval=50,
+    hooks=[
+        dict(type='TextLoggerHook'),
+        dict(type='TensorboardLoggerHook')
+    ])
+evaluation = dict(interval=1)
+dist_params = dict(backend='nccl')
+find_unused_parameters = True  # todo: fix number of FPN outputs
+log_level = 'INFO'
+load_from = None
+resume_from = None
+workflow = [('train', 1)]