text_classification infer performance test #11080

tensor-tang · 2018-05-31T08:37:04Z

This enabled text_classification model inference test on both single thread and multi threads.

We can get performance result shown below, on CPU 2620 v2:

Threads Num	Latency per thread (ms)	Throughput (QPS=sample number/total time)
1	2.29	436
12	6.56	1808

tensor-tang · 2018-06-01T08:31:50Z

Last failure is:

[Step 1/1] In file included from /paddle/paddle/fluid/operators/tensorrt_engine_op.cc:19:0:
[08:12:28][Step 1/1] /paddle/paddle/fluid/inference/tensorrt/convert/op_converter.h: In member function 'void paddle::inference::tensorrt::OpConverter::ConvertOp(const paddle::framework::proto::OpDesc&, const std::unordered_set<std::__cxx11::basic_string<char> >&, const paddle::framework::Scope&, paddle::inference::tensorrt::TensorRTEngine*)':
[08:12:28][Step 1/1] /paddle/paddle/fluid/inference/tensorrt/convert/op_converter.h:43:51: error: no matching function for call to 'paddle::framework::OpDesc::OpDesc(const paddle::framework::proto::OpDesc&, std::nullptr_t, std::nullptr_t)'
[08:12:28][Step 1/1]      framework::OpDesc op_desc(op, nullptr, nullptr);
[08:12:28][Step 1/1]                                                    ^
[08:12:28][Step 1/1] In file included from /paddle/paddle/fluid/framework/block_desc.h:24:0,
[08:12:28][Step 1/1]                  from /paddle/paddle/fluid/framework/operator.h:26,
[08:12:28][Step 1/1]                  from /paddle/paddle/fluid/operators/tensorrt_engine_op.h:19,
[08:12:28][Step 1/1]                  from /paddle/paddle/fluid/operators/tensorrt_engine_op.cc:17:
[08:12:28][Step 1/1] /paddle/paddle/fluid/framework/op_desc.h:40:3: note: candidate: paddle::framework::OpDesc::OpDesc(const paddle::framework::OpDesc&, paddle::framework::BlockDesc*)
[08:12:28][Step 1/1]    OpDesc(const OpDesc &other, BlockDesc *block) {
[08:12:28][Step 1/1]    ^
[08:12:28][Step 1/1] /paddle/paddle/fluid/framework/op_desc.h:40:3: note:   candidate expects 2 arguments, 3 provided
[08:12:28][Step 1/1] paddle/fluid/operators/CMakeFiles/tensorrt_engine_op.dir/build.make:62: recipe for target 'paddle/fluid/operators/CMakeFiles/tensorrt_engine_op.dir/tensorrt_engine_op.cc.o' failed
[08:12:28][Step 1/1] /paddle/paddle/fluid/framework/op_desc.h:38:12: note: candidate: paddle::framework::OpDesc::OpDesc(paddle::framework::BlockDesc*)
[08:12:28][Step 1/1]    explicit OpDesc(BlockDesc *block) : block_(block) {}

https://paddleci.ngrok.io/viewLog.html?tab=buildLog&logTab=tree&filter=debug&expand=all&buildId=37591&_focus=4127

jacquesqiao · 2018-06-01T09:06:41Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  auto start_ms = GetCurrentMs();
+  for (size_t i = 0; i < inputs.size(); ++i) {
+    feed_targets[feed_target_names[0]] = inputs[i];
+    executor->Run(*copy_program, scope, &feed_targets, &fetch_targets, true,


create a subscope to run the executor

auto sub_scope = scope->NewScope(); executor->Run(sub_scope); scope->DeleteScope(sub_scope);

feed data into sub_scope if needed.

Thanks. I will update.

Superjomn · 2018-06-01T09:28:30Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  // 1. Define place, executor, scope
+  auto place = paddle::platform::CPUPlace();
+  auto executor = paddle::framework::Executor(place);
+  auto* scope = new paddle::framework::Scope();


std::unique_ptrpaddle::framework::Scope scope(new paddle::framework::Scope);

Superjomn · 2018-06-01T09:54:10Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  for (size_t i = 0; i < inputs.size(); ++i) {
+    feed_targets[feed_target_names[0]] = inputs[i];
+    executor->Run(*copy_program, &sub_scope, &feed_targets, &fetch_targets,
+                  true, true, feed_holder_name, fetch_holder_name);


add some comments to true such as

true/*use_gpu*/, true/* with_monitor*/

and it will be more clear.

OK, thanks~

Superjomn · 2018-06-01T09:55:54Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  return 1e+3 * time.tv_sec + 1e-3 * time.tv_usec;
+}
+
+// return size of total words


Add some comment to describe the function.

Superjomn · 2018-06-01T09:56:21Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  return sz;
+}
+
+void SplitData(


Add some comment to describe the function.

tensor-tang

will update

tensor-tang · 2018-06-01T10:00:39Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  return 1e+3 * time.tv_sec + 1e-3 * time.tv_usec;
+}
+
+// return size of total words


tensor-tang · 2018-06-01T10:00:53Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  return sz;
+}
+
+void SplitData(


tensor-tang · 2018-06-01T10:01:11Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  for (size_t i = 0; i < inputs.size(); ++i) {
+    feed_targets[feed_target_names[0]] = inputs[i];
+    executor->Run(*copy_program, &sub_scope, &feed_targets, &fetch_targets,
+                  true, true, feed_holder_name, fetch_holder_name);


OK, thanks~

tensor-tang · 2018-06-01T10:01:26Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+  // 1. Define place, executor, scope
+  auto place = paddle::platform::CPUPlace();
+  auto executor = paddle::framework::Executor(place);
+  auto* scope = new paddle::framework::Scope();


tensor-tang · 2018-06-01T12:48:14Z

Done @Superjomn

Superjomn · 2018-06-01T12:49:45Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+#include <omp.h>
+#endif
+
+DEFINE_string(modelpath, "", "Directory of the inference model.");


Sure, I used to follow dirname before.

Superjomn · 2018-06-01T12:49:54Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+#endif
+
+DEFINE_string(modelpath, "", "Directory of the inference model.");
+DEFINE_string(datafile, "", "File of input index data.");


Superjomn · 2018-06-01T12:51:12Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+size_t LoadData(std::vector<paddle::framework::LoDTensor>* out,
+                const std::string& filename) {
+  if (filename.empty()) {
+    return DummyData(out);


add a log, such as

LOG(WARN) << "pass empty filename, use some dummy data instead";

This WARN is added here: https://github.com/PaddlePaddle/Paddle/pull/11080/files#diff-1e4d3b9c8b2730e0dacdaf72e35d597fR149

panyx0718

Please address those small comments

panyx0718 · 2018-06-03T12:10:57Z

paddle/fluid/inference/tests/book/CMakeLists.txt

@@ -38,3 +38,10 @@ inference_test(recommender_system)
 #inference_test(rnn_encoder_decoder)
 #inference_test(understand_sentiment ARGS conv)
 inference_test(word2vec)
+
+# This is an unly work around to make this test run


Add a TODO to clean up?

panyx0718 · 2018-06-03T12:20:12Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+    while (getline(iss, field, ' ')) {
+      ids.push_back(stoi(field));
+    }
+    if (ids.size() >= 1024) {


After synced with NLP guys, they say they will ignore the inputs which is larger than 1024.

tensor-tang

Thanks for your inputs, I will update.

tensor-tang · 2018-06-04T02:49:12Z

paddle/fluid/inference/tests/book/CMakeLists.txt

@@ -38,3 +38,10 @@ inference_test(recommender_system)
 #inference_test(rnn_encoder_decoder)
 #inference_test(understand_sentiment ARGS conv)
 inference_test(word2vec)
+
+# This is an unly work around to make this test run


tensor-tang · 2018-06-04T02:52:16Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+#include <omp.h>
+#endif
+
+DEFINE_string(modelpath, "", "Directory of the inference model.");


Sure, I used to follow dirname before.

tensor-tang · 2018-06-04T02:52:26Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+#endif
+
+DEFINE_string(modelpath, "", "Directory of the inference model.");
+DEFINE_string(datafile, "", "File of input index data.");


tensor-tang · 2018-06-04T02:53:38Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+size_t LoadData(std::vector<paddle::framework::LoDTensor>* out,
+                const std::string& filename) {
+  if (filename.empty()) {
+    return DummyData(out);


This WARN is added here: https://github.com/PaddlePaddle/Paddle/pull/11080/files#diff-1e4d3b9c8b2730e0dacdaf72e35d597fR149

tensor-tang · 2018-06-04T02:55:19Z

paddle/fluid/inference/tests/book/test_inference_nlp.cc

+    while (getline(iss, field, ' ')) {
+      ids.push_back(stoi(field));
+    }
+    if (ids.size() >= 1024) {


After synced with NLP guys, they say they will ignore the inputs which is larger than 1024.

tensor-tang added 15 commits May 25, 2018 11:57

test infer nlp

98fb8e5

Merge remote-tracking branch 'ups/develop' into nlp

1b8b253

use the actual data

602e28b

enable more choices

ce20dfa

add threads test

400f5e7

enable multi-threads

c00843f

enable read dataset

7759941

retest single thread

4d11c8e

revert profiling

d13dd3b

add test

708bec2

remove the ugly test

733718c

add multi-thread test

5387562

add thread setting

a4822ed

refine code

4a24c23

Merge remote-tracking branch 'ups/develop' into nlp

06adccf

tensor-tang changed the title ~~[DO NOT MERGE] nlp performance test~~ nlp performance test Jun 1, 2018

tensor-tang changed the title ~~nlp performance test~~ text_classification infer performance test Jun 1, 2018

tensor-tang requested review from panyx0718 and jacquesqiao June 1, 2018 08:32

Superjomn mentioned this pull request Jun 1, 2018

fix compile error #11119

Merged

refine log and add QPS

3206bcd

tensor-tang force-pushed the nlp branch from 2a19a71 to 3206bcd Compare June 1, 2018 08:42

jacquesqiao reviewed Jun 1, 2018

View reviewed changes

Superjomn closed this in #11119 Jun 1, 2018

tensor-tang reopened this Jun 1, 2018

tensor-tang added 2 commits June 1, 2018 17:24

fix scope in thread

7e9f079

Merge remote-tracking branch 'ups/develop' into nlp

9c687a9

Superjomn reviewed Jun 1, 2018

View reviewed changes

tensor-tang commented Jun 1, 2018

View reviewed changes

tensor-tang added 2 commits June 1, 2018 19:35

add some comments

eaeb76c

work around with dummy test

38f8182

follow comment: refine where time started

99d00cc

Superjomn reviewed Jun 1, 2018

View reviewed changes

panyx0718 previously approved these changes Jun 3, 2018

View reviewed changes

tensor-tang commented Jun 4, 2018

View reviewed changes

follow comments

6ae7cbe

tensor-tang dismissed panyx0718’s stale review via 6ae7cbe June 4, 2018 03:09

panyx0718 approved these changes Jun 4, 2018

View reviewed changes

tensor-tang merged commit 07c48db into PaddlePaddle:develop Jun 4, 2018

tensor-tang deleted the nlp branch June 4, 2018 03:53

text_classification infer performance test #11080

text_classification infer performance test #11080

Conversation

tensor-tang commented May 31, 2018 • edited Loading

tensor-tang commented Jun 1, 2018

Choose a reason for hiding this comment

jacquesqiao Jun 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tensor-tang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tensor-tang commented Jun 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

panyx0718 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tensor-tang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tensor-tang commented May 31, 2018 •

edited

Loading

jacquesqiao Jun 1, 2018 •

edited

Loading