Fundamental Data Reading in C++ #8009

JiayiFeng · 2018-01-31T10:48:41Z

Until now, Paddle Fluid's data feeding still wholly dependents on Python code. To get rid of the Python environment and achieve the goal of "wrapping the whole training by a while loop op", a C++ data feeding mechanism is required.

In this PR we show a fundamental C++ data feeding process, which implements the data reading(simulated by random data generator), shuffling and batching. We can regard this PR as the foundation of further developing for #7646.

Following concepts are introduced in this PR:

`ReaderBase`(code)

ReaderBase is the abstract base class of all Readers.

`FileReader`(code) and `DecoratedReader`(code)

These two classes are derived from the ReaderBase and will further be derived by respective specific readers. That is to say, in our design, there are two kinds of readers: file readers and decorated readers. A file reader reads from a file of some specific format, and yield only one instance of data at a time. e.g. RecordIO reader, jpg reader, .... A decorated reader takes another reader(both file reader and decorated reader are OK) as its 'underlying reader'. It gets data from its underlying reader, does some process on them(shuffling, or batching), then yields processed data. The output data of a decorated reader can be a single instance or a batch. ShuffleReader(code) and BatchReader(code) are both decorated readers.

All the readers share exactly the same interfaces defined in ReaderBase. So they can be decorated for more than one time: We can shuffle a reader's outputs and then batch the shuffle outputs. The interface consistency also allows related ops use readers without knowing what they are exactly.

`ReaderHolder` (code)

Different readers belong to different class types. It leads to a problem: How can we drop them into Variables and fetch them out by a unified method? For example, if a Variable holds a BatchReader, we can not get it by the following code:

var->Get<ReaderBase>("batch_reader");

we have to write:

var->Get<BatchReader>("batch_reader");

This requires each time getting a reader from a variable we must know the reader's type exactly. It is nearly impossible.

To solve this problem, we introduce ReaderHolder as a wrapper. It acts as an empty decorator of ReaderBase, which erasing reader's type. With ReaderHolder we are able to fetch all types of readers by var->Get<ReaderHolder>("...") and regard the obtained object as a reader.

To create and invoke readers, some now ops are introduced:

`CreateReaderOp`

Each reader has its creating op. File readers' creating ops have no input and yield the created file reader as its output. Decorated readers' creating ops take the underlying readers as inputs and then yield new decorated readers.

`ReadOp`(code)

A reader is only a Variable. It cannot trigger the reading process by itself. So we add the ReadOp to execute it. A ReadOp takes a reader Variable as input. Each time it runs, it invokes the reader‘s ReadNext() function and gets a new batch of data(or only one instance of data, if we use file reader directly). The output data of a reader are in the form of std::vector<LoDTenosr>, so the ReadOp also needs to split the vector and move LoDTensors to their respective output Variables.

… dev_reader

reyoung · 2018-02-07T03:16:23Z

paddle/framework/reader.h

+// file readers
+
+template <typename T>
+class RandomReader : public FileReader {


This reader is used by unittests. Maybe we should use `RandomDataGenerator` instead of `RandomReader`, because 1. It does not read anything. 2. It is confusing between `ShuffleReader` and `RandomReader`.

reyoung · 2018-02-07T03:19:57Z

paddle/framework/reader.h

+  ReaderBase* Get() const { return reader_.get(); }
+
+  void ReadNext(std::vector<LoDTensor>* out) { reader_->ReadNext(out); }
+  bool HasNext() const { return reader_->HasNext(); }


For reader, maybe we should add a Reset() method to read from beginning?

Yes. And the further discussion is needed about what shall we exactly do when a reader finishes one pass reading.

And Reset has been used by ReaderHolder to make the API consistent with unique_ptr. So I will add readers a ReInit method instead.

reyoung · 2018-02-07T03:23:16Z

paddle/operators/create_reader_op.cc

+namespace paddle {
+namespace operators {
+
+std::vector<framework::DDim> RestoreShapes(const std::vector<int>& shape_concat,


this method should be static.

reyoung · 2018-02-07T03:28:17Z

paddle/framework/reader.cc

+        break;
+      }
+    }
+    std::random_shuffle(buffer_.begin(), buffer_.end());


This implementation could be very slow. However, we can optimize it later.

Thanks, I didn't know that before. I will add a TOOD here.

reyoung · 2018-02-07T03:29:55Z

paddle/framework/reader.cc

+      batch_shape[0] += ins_shape[0];
+    }
+
+    LoDTensor out_tensor;


Maybe we can invoke MergeTensorOp here.

The MergeTensorOp can only merge two LoDTensor(true branch out and false branch out). However, in BatchReader we need to merge far more than two LoDTensor.

reyoung · 2018-02-07T03:30:40Z

@JiayiFeng Maybe we can move the PR description to design doc?

JiayiFeng · 2018-02-07T06:08:48Z

Sure. I will move them to a new design doc in some later PR.

reyoung

Excellent

JiayiFeng added 2 commits January 30, 2018 20:27

init reader.h and reader.cc files

1acad21

draft of Reader classes

f32ca63

JiayiFeng changed the title ~~File Reader in C++~~ File Readers in C++ Jan 31, 2018

JiayiFeng changed the title ~~File Readers in C++~~ [WIP] File Readers in C++ Jan 31, 2018

JiayiFeng added 9 commits February 1, 2018 12:58

refine inheritance relationship

d8cc21d

Complete CreateRandomReaderOp

93cab64

Complete CreateShuffleReaderOp

1696cb0

Complete CreateBatchReaderOp

3dfd1da

refine code

53e697c

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

da8a56e

… dev_reader

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

6e6f5c7

… dev_reader

Add ReadOp

1010e39

refine code and add unit tests

0bb9c80

JiayiFeng requested review from reyoung and wangkuiyi February 6, 2018 17:34

JiayiFeng changed the title ~~[WIP] File Readers in C++~~ Fundamental Data Fedding in C++ Feb 6, 2018

JiayiFeng changed the title ~~Fundamental Data Fedding in C++~~ Fundamental Data Reading in C++ Feb 6, 2018

JiayiFeng requested a review from dzhwinter February 7, 2018 02:12

JiayiFeng added 2 commits February 7, 2018 10:17

fix a unit test

542bdef

refine code

b00cae6

reyoung reviewed Feb 7, 2018

View reviewed changes

fix compile errors

c1349d9

reyoung approved these changes Feb 7, 2018

View reviewed changes

JiayiFeng merged commit 812cf15 into PaddlePaddle:develop Feb 7, 2018

JiayiFeng deleted the dev_reader branch February 7, 2018 09:45

JiayiFeng mentioned this pull request Feb 7, 2018

add cpp_data_feeding.md #8237

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fundamental Data Reading in C++ #8009

Fundamental Data Reading in C++ #8009

JiayiFeng commented Jan 31, 2018 •

edited

Loading

reyoung Feb 7, 2018 •

edited

Loading

JiayiFeng Feb 7, 2018

reyoung Feb 7, 2018

JiayiFeng Feb 7, 2018 •

edited

Loading

reyoung Feb 7, 2018

JiayiFeng Feb 7, 2018

reyoung Feb 7, 2018

JiayiFeng Feb 7, 2018

reyoung Feb 7, 2018

JiayiFeng Feb 7, 2018

reyoung commented Feb 7, 2018 •

edited

Loading

JiayiFeng commented Feb 7, 2018

reyoung left a comment

Fundamental Data Reading in C++ #8009

Fundamental Data Reading in C++ #8009

Conversation

JiayiFeng commented Jan 31, 2018 • edited Loading

ReaderBase(code)

FileReader(code) and DecoratedReader(code)

ReaderHolder (code)

CreateReaderOp

ReadOp(code)

reyoung Feb 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiayiFeng Feb 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyoung commented Feb 7, 2018 • edited Loading

JiayiFeng commented Feb 7, 2018

reyoung left a comment

Choose a reason for hiding this comment

JiayiFeng commented Jan 31, 2018 •

edited

Loading

`ReaderBase`(code)

`FileReader`(code) and `DecoratedReader`(code)

`ReaderHolder` (code)

`CreateReaderOp`

`ReadOp`(code)

reyoung Feb 7, 2018 •

edited

Loading

JiayiFeng Feb 7, 2018 •

edited

Loading

reyoung commented Feb 7, 2018 •

edited

Loading