Does DAG includes the dependency Graph?

[Johnny-dai-git]

Hi, All

I have questions related to the DAG in MxNet.

We know that class Graph will create a computational Graph which is a DAG with DFS

However, we also have a dependency Graph which is created by the engines.

Which DAG is actually the "DAG" we are always talking about ?

Best Regards,
Johnny

[ptrendx]

Hard to say without knowing the context of who is talking 😄, but you are right that both the nnvm::Graph and the dependency graph created by the engine is a DAG of computation. The main differences between the 2 are:

• the nnvm::Graph is explicit data structure representation of the neural network graph, while the dependency graph is implicit - there is no explicit "dependency graph" class anywhere and it is "constructed" on the fly (there is no actual construction - the only thing that is kept is the set of dependencies required for given work item)
• the nnvm::Graph is used to hold the "static" portion of the computation - e.g. the neural network model (if you hybridize). The graph from dependency tracking is used to track every single computation in MXNet: the model (does not matter if hybridized or not), communication, weight update and all auxiliary imperative operations.

[Johnny-dai-git]

Thank you so much !!

I really understand this topic better !

However, will the dependency graph be static after several iterations?

Is there a way I can track the dependency graph ?

Best Regards,
Johnny

[ptrendx]

The dependency graph created by the engine is not static (or at least - it does not have to be, since it depends on what operations you push to it from your script). The graph stored in nnvm::Graph is static.

As for tracking the dependency graph from the engine - I don't think you can do that without custom changes to the engine code.

[Johnny-dai-git]

Thank you！ Przemyslaw Tredak ***@***.***>于2021年4月30日 周五下午4:49写道：

The dependency graph created by the engine is not static (or at least - it does not have to be, since it depends on what operations you push to it from your script). The graph stored in nnvm::Graph is static. As for tracking the dependency graph from the engine - I don't think you can do that without custom changes to the engine code. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19881 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDGLBZC53G57TT332FAPD3TLMJVRANCNFSM4XOCWJJA> .

-- Yuanjun Dai (he/him) P.hd Department of Computer and Data Sciences Case Western Reserve University Phone: (216)-235-8330 Office: Glennan 505

[Johnny-dai-git]

Hi, Sorry to continue this question. Actually, I am trying to find out how mxnet does the backward propagation in my project. I know there is a backward function in graph_executor.cc file. It seems that mxnet will push operators into the dependency engine for the BP process. However, my training program is using autograd api. And the autograd API is using RunGraph API for gradients calculation. In autograd, the bp process does not involve the dependency engine. All gradients will be calculated in one time. Why and how this happen ? When we should use the graph_executor.cc and when we should use the autograd ? Best Regards, Johnny

…

On Fri, Apr 30, 2021 at 4:49 PM Przemyslaw Tredak ***@***.***> wrote: The dependency graph created by the engine is not static (or at least - it does not have to be, since it depends on what operations you push to it from your script). The graph stored in nnvm::Graph is static. As for tracking the dependency graph from the engine - I don't think you can do that without custom changes to the engine code. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19881 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDGLBZC53G57TT332FAPD3TLMJVRANCNFSM4XOCWJJA> .

-- Yuanjun Dai (he/him) P.hd Department of Computer and Data Sciences Case Western Reserve University Phone: (216)-235-8330 Office: Glennan 505