Refactor detr 3.x conditional detr and group detr #9248

LYMDLUT · 2022-11-05T18:46:33Z

Implement the results in the paper "[Conditional DETR: Conditional DETR for Fast Training Convergence]" and "[Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment]"with mmdetection. The original code has released.

Remark

Besides, there are also brother PRs #9149, #8544, #8846, #8533, #8811 of three algorithms (DINO、old Condition DETR and DAB DETR) developed in parallel.

This is my first experience of contributing a PR to the open source community. I'm trying hard to make Conditional DETR and Group DETR available in mmdetection as soon as possible.

The mAP of Group DETR is still aligning，do not worry，it will release in very recent days.

* [Fix] Fix UT to be compatible with pytorch 1.6 (open-mmlab#8707) * Update * Update * force reinstall pycocotools * Fix build_cuda * docker install git * Update * comment other job to speedup process * update * uncomment * Update * Update * Add comments for --force-reinstall * [Refactor] Refactor anchor head and base head with boxlist (open-mmlab#8625) * Refactor anchor head * Update * Update * Update * Add a series of boxes tools * Fix box type to support n x box_dim boxes * revert box type changes * Add docstring * refactor retina_head * Update * Update * Fix comments * modify docstring of coder and ioucalculator * Replace with_boxlist with use_box_type * fix: fix config of detr-r18 * fix: modified import of MSDeformAttn in PixelDecoder of Mask2Former * feat: add TransformerDetector as the base detector of DETR-like detectors * refactor: refactor modules and configs of DETR * refactor: refactor DETR-related modules in transformer.py * refactor: refactor DETR-related modules in transformer.py * fix: add type comments in detr.py * correct trainloop in detr_r50 config * fix: modify the parent class of DETRHead to BaseModule * refactor: refactor modules and configs of Deformable DETR * fix: modify the usage of num_query * fix: modify the usage of num_query in configs * refactor: replace input_proj of detr with ChannelMapper neck * refactor: delete multi_apply in DETRHead.forward() * Update detr_r18_8xb2-500e_coco.py using channel mapper for r18 * change the name of detection_transfomer.py to base_detr.py * refactor: modify construct binary masks section of forward_pretransformer * refactor: utilize abstractmethod * update ABCmeta to make sure reload class TransformerDetector * some annotation * some annotation * some annotation * refactor: delete _init_transformer in detectors * refactor: modify args of deformable detr * refactor: modify about super().__init__() * Update detr_head.py Remove the multi feat lvl in function 'predict_by_feat' * Update detr.py update init_weights * some annotation for head * to make sure the head args the same as detector * to make sure the head args the same as detector * some bug * fix: fix bugs of num_pred in DeformableDETRHead * add kwargs to transformer * support MLP and sineembed position * detele positional encodeing * delete useless postnorm * Revert "add kwargs to transformer" This reverts commit a265c1a. * Update detr_head.py Update type and shape of args * Update detr_head.py fix args docstring in predict_by_feat * Update base_detr.py Update docstring for forward_pretransformer * Update deformable_detr.py Fix docstring * to support conditional detr with reload forward_transformer * fix: update config files of Two-stage and Box-refine * replace all bs with batch_size in detr-related files * update deformable.py and transformer.py * update docstring in base_detr * update docstring in base_detr, detr * doc refine * Revert "doc refine" This reverts commit b69da4f. * doc refine * doc refine * updabase_detr, detr, and le layers/transformdoc * fix doc in base_detr * add origin repo link * add origin repo link * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * refine doc * doc: add doc of the first edition of Deformable DETR * batch_size to bs * refine doc * refine doc * feat: add config comments of specific module * refactor: refactor base DETR class TransformerDetector * fix: fix wrong return typehint of forward_encoder in TransformerDetector * refactor: refactor DETR * refactor: refactor Deformable DETR * refactor: refactor forward_encoder and pre_decoder * fix: fix bugs of new edition * refactor: small modifications * fix: move get_reference_points to deformable_encoder * refactor: merge init_&inter_reference to references in Deformable DETR * modify docstring of get_valid_ratio in Deformable DETR * add some docstring * doc: add docstring of deformable_detr.py * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of deformable_detr_head.py * doc: modify docstring of deformable detr * doc: add docstring of base_detr.py * doc: refine docstring of base_detr.py * doc: refine docstring of base_detr.py * a little change of MLP * a little change of MLP * a little change of MLP * a little change of MLP * refine config * refine config * refine config * refine doc string for detr * little refine doc string for detr.py * tiny modification * doc: refine docstring of detr.py * tiny modifications to resolve the conversations * DETRHead.predict() draft * tiny modifications to resolve conversations * refactor: modify arg names and forward strategies of bbox_head * tiny modifications to resolve the conversations * support MLP * fix docsting of function pre_decoder * fix docsting of function pre_decoder * fix docstring * modifications for resolving conversations * refactor: eradicate key_padding_mask args * refactor: eradicate key_padding_mask args * fix: fix bug of deformable detr and resolve some conversations * refactor: rename base class with DetectionTransformer and other modifications * fix: fix config of detr * fix the bug of init * fix: fix init_weight of DETR and Deformable DETR * resolve conflict * fix auto-merge bug * fix pre-commit bug * refactor: move the position of encoder and decoder * delete Transformer in ci test * delete Transformer in ci test Co-authored-by: jbwang1997 <[email protected]> Co-authored-by: KeiChiTse <[email protected]> Co-authored-by: LYMDLUT <[email protected]> Co-authored-by: lym <[email protected]> Co-authored-by: Kei-Chi Tse <[email protected]>

…tr' into refactor-detr-3.x-rebase-groupdetr

Li-Qingyun

Good job my bro!

mmdet/models/detectors/conditioanl_detr.py

Li-Qingyun · 2022-11-06T01:40:08Z

configs/conditional_detr/conditional_detr_r50_8xb2-50e_coco.py

+_base_ = [
+    '../detr/detr_r50_8xb2-150e_coco.py'
+]
+model = dict(
+    type='ConditionalDETR',
+    num_query=300,
+    decoder=dict(
+        num_layers=6,
+        layer_cfg=dict(
+            self_attn_cfg=dict(embed_dims=256, num_heads=8, dropout=0.1, cross_attn=False),
+            cross_attn_cfg=dict(embed_dims=256, num_heads=8, dropout=0.1, cross_attn=True))),
+    bbox_head=dict(
+        type='ConditionalDETRHead',
+        loss_cls=dict(
+            _delete_=True,
+            type='FocalLoss',
+            use_sigmoid=True,
+            gamma=2.0,
+            alpha=0.25,
+            loss_weight=2.0)),
+    # training and testing settings
+    train_cfg=dict(
+        assigner=dict(
+            type='HungarianAssigner',
+            match_costs=[
+                dict(type='FocalLossCost', weight=2.0),
+                dict(type='BBoxL1Cost', weight=5.0, box_format='xywh'),
+                dict(type='IoUCost', iou_mode='giou', weight=2.0)
+            ])))
+
+# learning policy
+max_epochs = 50,
+
+param_scheduler = [dict(type='MultiStepLR', milestones=[40])]
+


Does the config of DINO require to inherit the Deformable DETR config? @jshilong

I prefer the inheritance approach, but is it supposed to have at least one config in an algorithm without too much inheritance?

configs/group_detr/group_detr_r50_8xb2-50e_coco.py

mmdet/models/dense_heads/conditional_detr_head.py

mmdet/models/layers/transformer/conditional_transformer.py

mmdet/models/dense_heads/conditional_detr_head.py

mmdet/models/layers/transformer/conditional_transformer.py

jshilong · 2022-11-27T13:47:28Z

mmdet/models/detectors/conditional_detr.py

+    """
+
+    def __init__(self, *arg, group_detr=1, **kwargs) -> None:
+        self.group_detr = group_detr


move all group detr related code to another pr

jshilong · 2022-11-27T13:47:52Z

mmdet/models/layers/transformer/conditional_detr_transformer.py

+        for layer_id in range(self.num_layers - 1):
+            self.layers[layer_id + 1].cross_attn.qpos_proj = None
+
+    def forward(self, query: Tensor, key: Tensor, value: Tensor,


jshilong · 2022-11-27T13:48:04Z

mmdet/models/layers/transformer/conditional_detr_transformer.py

+
+        nn.init.constant_(self.out_proj.bias, 0.)  # init out_proj
+
+    def forward_attn(self,


jshilong · 2022-11-27T13:48:18Z

mmdet/models/layers/transformer/conditional_detr_transformer.py

+        self._init_proj()
+        self.group_detr = group_detr
+
+    def _init_proj(self):


rename to init_layers

Li-Qingyun and others added 27 commits October 20, 2022 15:20

delete Transformer in ci test

9ac71ab

refine Transformer in ci test

436de72

add test_detr.py

db09116

add test_detr.py

8487eea

support deformable detr

2f66ad3

fix _forward bug

c64213b

delete test_deformable_detr_head.py

7b08916

delete test_maskformer.py

be1202e

todo

d024714

use relativte config unitest

5edb02b

use relativte config unitest

43a3af1

support conditional detr

b97ac76

add doc

72f679b

add gropdetr decoder self attention

05a6348

Add unitests for detr 3.x (open-mmlab#9089)

b257a29

add Tuple

2b2ac06

add test_detr.py

dae222c

add test_detr.py

29b1d42

support conditional detr

4f667f1

add doc

621da7c

add gropdetr decoder self attention

af115a6

add Tuple

cf19bb3

Merge remote-tracking branch 'origin/refactor-detr-3.x-rebase-groupde…

4477dba

…tr' into refactor-detr-3.x-rebase-groupdetr

move con decoder to layers

c520d85

add conditional config

01aadf5

support group detr

2916268

mm-assistant bot assigned chhluo Nov 5, 2022

Li-Qingyun reviewed Nov 6, 2022

View reviewed changes

mmdet/models/layers/transformer/conditional_transformer.py Outdated Show resolved Hide resolved

Li-Qingyun mentioned this pull request Nov 6, 2022

[Feat] Add DINO on MMDetection 3.x #9149

Merged

3 tasks

LYM-fire added 4 commits November 6, 2022 16:58

rename some variable in head

2bfb32f

move gen_sine_embed_for_ref to utils

cc26f32

rename the detector

8d01c79

refine util

57b8767

KeiChiTse mentioned this pull request Nov 6, 2022

Refactor DAB-DETR in MMDetection 3.x #9252

Merged

LYM-fire added 2 commits November 7, 2022 11:54

refine precommit

b9a6318

refine precommit

22ee0d2

ZwwWayne added this to the 3.0.0rc5 milestone Nov 21, 2022

fix bug of 50e stop

b3a002c

jshilong reviewed Nov 27, 2022

View reviewed changes

LYM-fire added 12 commits November 28, 2022 22:10

support 91class

36ff00b

condetr 91class config

2534694

change num_query to num_queries

ceb34df

refine dropout and attn_mask

8be12b5

add typehint

987fcbb

delete batch_first

41f1895

delete batch_first

d54cfe7

refine head doc

552239a

refine condetr doc

3d26369

refine condetr_transformer doc

4707ac2

fix the bug of class91

782fca0

fix the bug of class91

72544f8

LYMDLUT mentioned this pull request Nov 29, 2022

Refactor detr 3.x conditional detr #9405

Merged

KeiChiTse mentioned this pull request Dec 7, 2022

[Refactor] Change to BNC data flow for all DETRs #9460

Merged

jshilong force-pushed the refactor-detr branch from 07d7ae4 to 4c11117 Compare January 19, 2023 05:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor detr 3.x conditional detr and group detr #9248

Refactor detr 3.x conditional detr and group detr #9248

LYMDLUT commented Nov 5, 2022 •

edited

Loading

Li-Qingyun left a comment

Li-Qingyun Nov 6, 2022

Li-Qingyun Nov 6, 2022

jshilong Nov 27, 2022

jshilong Nov 27, 2022

jshilong Nov 27, 2022

jshilong Nov 27, 2022


		nn.init.constant_(self.out_proj.bias, 0.) # init out_proj

		def forward_attn(self,

Refactor detr 3.x conditional detr and group detr #9248

Are you sure you want to change the base?

Refactor detr 3.x conditional detr and group detr #9248

Conversation

LYMDLUT commented Nov 5, 2022 • edited Loading

Li-Qingyun left a comment

Choose a reason for hiding this comment

Li-Qingyun Nov 6, 2022

Choose a reason for hiding this comment

Li-Qingyun Nov 6, 2022

Choose a reason for hiding this comment

jshilong Nov 27, 2022

Choose a reason for hiding this comment

jshilong Nov 27, 2022

Choose a reason for hiding this comment

jshilong Nov 27, 2022

Choose a reason for hiding this comment

jshilong Nov 27, 2022

Choose a reason for hiding this comment

LYMDLUT commented Nov 5, 2022 •

edited

Loading