Add multi-class non-maximum suppression operator. #7953

qingqing01 · 2018-01-29T16:21:23Z

The clipping and scaling for output BBoxes will be done in next PR.
This PR only supports CPU. The GPU will be supported in the future.
If there are no detected BBoxes for all images, the output is -1 and the elements of output LoD are all zero.

… multiclass_nms_op

wanghaox · 2018-01-31T07:18:10Z

paddle/operators/multiclass_nms_op.cc

+    auto score_dims = ctx->GetInputDim("Scores");
+
+    PADDLE_ENFORCE_EQ(box_dims.size(), 2,
+                      "The rank of Input(Bboxes) must be 3.");


The rank of Input(Bboxes) must be 2.

wanghaox · 2018-01-31T07:19:08Z

paddle/operators/multiclass_nms_op.cc

+                      "The rank of Input(Bboxes) must be 3.");
+    PADDLE_ENFORCE_EQ(score_dims.size(), 3,
+                      "The rank of Input(Scores) must be 3.");
+    PADDLE_ENFORCE_EQ(box_dims[1], 4);


The shape of Input(Bboxes) must be [N, 4]

wanghaox · 2018-01-31T07:21:05Z

paddle/operators/multiclass_nms_op.cc

+    PADDLE_ENFORCE_EQ(score_dims.size(), 3,
+                      "The rank of Input(Scores) must be 3.");
+    PADDLE_ENFORCE_EQ(box_dims[1], 4);
+    PADDLE_ENFORCE_EQ(box_dims[0], score_dims[2]);


The predictions number of Input(Bboxes) and Input(Scores) must be the same.

Done. Thanks!

wanghaox · 2018-01-31T07:31:21Z

paddle/operators/multiclass_nms_op.cc

+T BBoxArea(const T* box, const bool normalized) {
+  if (box[2] < box[0] || box[3] < box[1]) {
+    // If bbox is invalid (e.g. xmax < xmin or ymax < ymin), return 0.
+    return T(0.);


use static_cast(0.) or brace initialization?

Done. Thanks!

wanghaox · 2018-01-31T07:58:23Z

paddle/operators/multiclass_nms_op.cc

+    for (const auto& it : selected_indices) {
+      int label = it.first;
+      const T* sdata = scores_data + label * predict_dim;
+      std::vector<int> indices = it.second;


std::vector& indices = it.second;

pkuyym · 2018-01-31T07:08:00Z

paddle/operators/multiclass_nms_op.cc

@@ -0,0 +1,375 @@
+/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.


Please fix the year.

pkuyym · 2018-01-31T07:25:36Z

paddle/operators/multiclass_nms_op.cc

+
+  void InferShape(framework::InferShapeContext* ctx) const override {
+    PADDLE_ENFORCE(ctx->HasInput("Bboxes"),
+                   "Input(Bboxes) of MulticlassNMS should not be null.");


Bboxes or BBoxes?

MulticlassNMS --> MultiClassNMSOp

Done. Change to BBoxes.

pkuyym · 2018-01-31T07:26:03Z

paddle/operators/multiclass_nms_op.cc

+    PADDLE_ENFORCE(ctx->HasInput("Bboxes"),
+                   "Input(Bboxes) of MulticlassNMS should not be null.");
+    PADDLE_ENFORCE(ctx->HasInput("Scores"),
+                   "Input(Scores) of MulticlassNMS should not be null.");


MulticlassNMS --> MulticlassNMSOp

pkuyym · 2018-01-31T07:52:20Z

paddle/operators/multiclass_nms_op.cc

+constexpr int64_t kOutputDim = 6;
+constexpr int64_t kBBoxSize = 4;
+
+class MulticlassNMSOp : public framework::OperatorWithKernel {


MulticlassNMSOp --> MultiClassNMSOp

pkuyym · 2018-01-31T07:52:47Z

paddle/operators/multiclass_nms_op.cc

+    auto score_dims = ctx->GetInputDim("Scores");
+
+    PADDLE_ENFORCE_EQ(box_dims.size(), 2,
+                      "The rank of Input(Bboxes) must be 3.");


must be 3 --> must be 2

pkuyym · 2018-01-31T08:41:50Z

paddle/operators/multiclass_nms_op.cc

+    AddInput("Bboxes",
+             "(Tensor) A 2-D Tensor with shape [M, 4] represents the location "
+             "predictions with M bboxes. 4 is the number of "
+             "each location coordinates.");


4 is the number of each location coordinates --> Each bounding box has four coordinate values and the layout is [xmin, ymin, xmax, ymax]

Done. Thanks!

pkuyym · 2018-01-31T08:44:10Z

paddle/operators/multiclass_nms_op.cc

+             "each location coordinates.");
+    AddInput("Scores",
+             "(Tensor) A 3-D Tensor with shape [N, C, M] represents the "
+             "confidence predictions. N is the batch size, C is the class "


confidence predictions --> predicted confidence scores.

Done. Thanks!

pkuyym · 2018-01-31T08:48:34Z

paddle/operators/multiclass_nms_op.cc

+    AddInput("Scores",
+             "(Tensor) A 3-D Tensor with shape [N, C, M] represents the "
+             "confidence predictions. N is the batch size, C is the class "
+             "number, M is number of predictions for each class, which is "


, M is number of predictions for each class --> and M is the number of bounding boxes. For each category there are total M scores which corresponding to M bounding boxes. Please note, M is equal to the 1st dimension of Bboxes.

Done. Thanks!

pkuyym · 2018-01-31T08:49:39Z

paddle/operators/multiclass_nms_op.cc

+    AddAttr<int>(
+        "background_label",
+        "(int64_t, defalut: 0) "
+        "The index of background label, the background label will be ignored.")


Please add If set to -1, then all categories will be considered.

pkuyym · 2018-01-31T09:03:17Z

paddle/operators/multiclass_nms_op.cc

+        .SetDefault(0);
+    AddAttr<float>("score_threshold",
+                   "(float) "
+                   "Only consider detections whose confidences are larger than "


Threshold to filter out bounding boxes with low confidence score.

qingqing01

@wanghaox @pkuyym

Thank you for your carefully review.

qingqing01 · 2018-02-01T13:04:15Z

paddle/operators/multiclass_nms_op.cc

+      : OpProtoAndCheckerMaker(proto, op_checker) {
+    AddInput("Bboxes",
+             "(Tensor) A 2-D Tensor with shape [M, 4] represents the location "
+             "predictions with M bboxes. 4 is the number of "


Done. Thank!

qingqing01 · 2018-02-01T13:04:23Z

paddle/operators/multiclass_nms_op.cc

+    AddInput("Bboxes",
+             "(Tensor) A 2-D Tensor with shape [M, 4] represents the location "
+             "predictions with M bboxes. 4 is the number of "
+             "each location coordinates.");


Done. Thanks!

qingqing01 · 2018-02-01T13:05:33Z

paddle/operators/multiclass_nms_op.cc

+             "each location coordinates.");
+    AddInput("Scores",
+             "(Tensor) A 3-D Tensor with shape [N, C, M] represents the "
+             "confidence predictions. N is the batch size, C is the class "


Done. Thanks!

qingqing01 · 2018-02-01T13:05:41Z

paddle/operators/multiclass_nms_op.cc

+    AddInput("Scores",
+             "(Tensor) A 3-D Tensor with shape [N, C, M] represents the "
+             "confidence predictions. N is the batch size, C is the class "
+             "number, M is number of predictions for each class, which is "


Done. Thanks!

qingqing01 · 2018-02-01T13:11:16Z

paddle/operators/multiclass_nms_op.cc

@@ -0,0 +1,375 @@
+/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.


qingqing01 · 2018-02-01T13:33:30Z

paddle/operators/multiclass_nms_op.cc

+        const T* bdata = bboxes_data + idx * kBBoxSize;
+        odata[count * kOutputDim] = label;           // label
+        odata[count * kOutputDim + 1] = sdata[idx];  // score
+        odata[count * kOutputDim + 2] = bdata[0];    // xmin


Done, use std::memcpy

qingqing01 · 2018-02-01T13:37:00Z

paddle/operators/multiclass_nms_op.cc

+    std::vector<size_t> batch_starts = {0};
+    for (int64_t i = 0; i < batch_size; ++i) {
+      Tensor ins_score = scores->Slice(i, i + 1);
+      ins_score.Resize({class_num, predict_dim});


The Slice in Tensor doesn't reduce dimension. The shape of ins_score is [1, C, M], resize to [C, M] here.

qingqing01 · 2018-02-01T13:53:54Z

paddle/operators/multiclass_nms_op.cc

+    if (normalized) {
+      return w * h;
+    } else {
+      // If bbox is not within range [0, 1].


qingqing01 · 2018-02-01T13:54:12Z

paddle/operators/multiclass_nms_op.cc

+    AddAttr<int>(
+        "background_label",
+        "(int64_t, defalut: 0) "
+        "The index of background label, the background label will be ignored.")


qingqing01 · 2018-02-01T13:54:17Z

paddle/operators/multiclass_nms_op.cc

+        .SetDefault(0);
+    AddAttr<float>("score_threshold",
+                   "(float) "
+                   "Only consider detections whose confidences are larger than "


… multiclass_nms_op

wanghaox · 2018-02-02T08:05:45Z

paddle/operators/multiclass_nms_op.cc

+limitations under the License. */
+
+#include "paddle/framework/op_registry.h"
+#include "paddle/operators/math/math_function.h"


Is this header necessary?

wanghaox · 2018-02-02T08:07:15Z

paddle/operators/multiclass_nms_op.cc

+}
+
+template <class T>
+T BBoxArea(const T* box, const bool normalized) {


add inline?

pkuyym

LGTM

pkuyym · 2018-02-02T08:22:01Z

paddle/operators/multiclass_nms_op.cc

+    std::vector<size_t> batch_starts = {0};
+    for (int64_t i = 0; i < batch_size; ++i) {
+      Tensor ins_score = scores->Slice(i, i + 1);
+      ins_score.Resize({class_num, predict_dim});


I see, thx.

qingqing01

@wanghaox Thanks!

qingqing01 · 2018-02-02T08:34:19Z

paddle/operators/multiclass_nms_op.cc

+limitations under the License. */
+
+#include "paddle/framework/op_registry.h"
+#include "paddle/operators/math/math_function.h"


qingqing01 · 2018-02-02T08:34:24Z

paddle/operators/multiclass_nms_op.cc

+}
+
+template <class T>
+T BBoxArea(const T* box, const bool normalized) {


qingqing01 added 3 commits January 29, 2018 23:49

Add multi-class non-maximum suppression operator.

912a4f2

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

73b6654

… multiclass_nms_op

Update doc for multiclass_nms_op.

2731fd9

qingqing01 force-pushed the multiclass_nms_op branch from 9ad769d to 2731fd9 Compare January 29, 2018 16:25

Fix bug in unit test.

35dec3d

qingqing01 requested review from wanghaox, pkuyym, wanghaoshuang and will-am January 30, 2018 10:02

Fix the output order and add more unit test cases.

5378864

wanghaox suggested changes Jan 31, 2018

View reviewed changes

pkuyym requested changes Jan 31, 2018

View reviewed changes

qingqing01 commented Feb 1, 2018

View reviewed changes

qingqing01 added 2 commits February 2, 2018 09:34

Follow comments.

f3415ec

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1d9a7e1

… multiclass_nms_op

qingqing01 force-pushed the multiclass_nms_op branch from 3b56ccd to 1d9a7e1 Compare February 2, 2018 01:35

qingqing01 mentioned this pull request Feb 2, 2018

The TODO lists for MobileNet-SSD model. #7488

Closed

25 tasks

wanghaox previously approved these changes Feb 2, 2018

View reviewed changes

pkuyym previously approved these changes Feb 2, 2018

View reviewed changes

Remove the redundant header file and make one function inlne.

a6f3846

qingqing01 dismissed stale reviews from pkuyym and wanghaox via a6f3846 February 2, 2018 08:33

qingqing01 commented Feb 2, 2018

View reviewed changes

wanghaox approved these changes Feb 2, 2018

View reviewed changes

qingqing01 merged commit c9ef69b into PaddlePaddle:develop Feb 2, 2018

qingqing01 deleted the multiclass_nms_op branch March 7, 2018 12:02

		@@ -0,0 +1,375 @@
		/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.

Add multi-class non-maximum suppression operator. #7953

Add multi-class non-maximum suppression operator. #7953

Conversation

qingqing01 commented Jan 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pkuyym Jan 31, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pkuyym left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qingqing01 commented Jan 29, 2018 •

edited

Loading

pkuyym Jan 31, 2018 •

edited

Loading