Design doc of compile time register gradient operators #4517
Conversation
reyoung
requested review from
wangkuiyi,
JiayiFeng,
QiJune and
tonyyang-svail
reyoung
force-pushed
the
feature/design_doc_of_new_grad_op
branch
2 times, most recently
from
52fec02 to
12713e9
Compare
doc/design/register_grad_op.md
Outdated
|
|
||
| ## Problem | ||
|
|
||
| Since we separate users program in two stages, compile time and runtime, we should record and look up the mapping relationship between an operator and its gradient operators when compile. However, we register this relationship in runtime by these `OpInfo` fields. |
There was a problem hiding this comment.
The execution of a neural network topology in PaddlePaddle is separated into two stages, complie-time and run-time.
At complie-time, a ProgramDesc will be generated. At run-time, the ProgramDesc will be executed on specific hardware. We can refer to the design of computation-graphs.
The Gradient Operator's OpDesc is also generated at compile-time. We have to find the mapping relationship between Operator's OpDesc and its GradientOp's OpDesc.
However, we make the mapping relationship between Operator and its GradientOp at run-time in OpInfo class currently.
doc/design/register_grad_op.md
Outdated
|
|
||
| ## Problem | ||
|
|
||
| Since we separate users program in two stages, compile time and runtime, we should record and look up the mapping relationship between an operator and its gradient operators when compile. However, we register this relationship in runtime by these `OpInfo` fields. |
There was a problem hiding this comment.
The Problem Posed
In our current operator registration mechanism, for each operator, the programmer should register a gradient operator creator function, which takes a C++ operator instance, and returns the corresponding gradient instance.
However, as we decided to separate the compilation and execution of DL models, we need to reshape the creator to take a protobuf OpDesc message, and returns a corresponding message.
More than that, the new registration mechanism need to support the fact that an operators' gradient computation might be a composition of operators.
Current Implementation
Proposed Solution
No description provided.