feature/parallel_gpu

[tonyyang-svail]

A simple implementation parallel_do supporting multigpu

...
|
ParallelDo
    |		Split input
    |		Copy parameter to multiple GPUs
    -		Wait
    ||||	Forward on multiple GPUs
    -		Wait
    |		Merge output
    -		Wait
|
...
|
ParallelGradDo
    |		Split output@grad
    -		Wait
    ||||	Backward on multiple GPUs
    -		Wait
    |		AllReduce parameters
    -		Wait
|
...

TODO:

• Correctness
• init device context Fix/init #7345
• getPlaceOp add GetPlaces operator #6732

[chengduoZH]

...
|
ParallelDo
    |		Split input
    |		Copy parameter to multiple GPUs
    -		Wait
    ||||	Forward on multiple GPUs
    -		Wait

It seems that the first Wait is unnecessary. The current GPU begins forward calculation doesn't depend on whether another GPU has completed the transmission of parameters or not.

[tonyyang-svail]

@chengduoZH thanks for the suggestion. In the future implementation, we will have IO stream and computation stream. Then the waiting becomes necessary.

Let's target the correctness of this op first. :)

[reyoung]

Cool