-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding documentation for multi-device in Tensorflow #5412
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super awesome review! I am not sure if it is proper to put such a document in PaddlePaddle' official repo. Should we move it from this PR into a Wiki page? Thanks!
``` | ||
Here we are running the same operation(MatMul) on two different GPUs : '/gpu:2' and '/gpu:3', and on the CPU ('/cpu:0') the operation `tf.add_n` which adds all input tensors element-wise. This operation would collect the output of MatMulfrom the GPUs and aggregate them on the CPU. | ||
|
||
The [tutorial](https://www.TensorFlow.org/tutorials/deep_cnn) on CIFAR10 is a good example demonstrating how to do training with multiple GPUs on TensorFlow. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might need more information about Device Contexts and how they are used to build new operators/kernels?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good suggestion. Will dig up more on this and add a section.
Thank you for the review. I think Wiki page is a good idea, since it is meant for our educational purpose anyway. I will create a page and close this PR. |
Closing this PR and created the Wiki page: https://github.com/PaddlePaddle/Paddle/wiki/Understanding-Multi-device-training-in-TensorFlow as per @wangkuiyi 's review. |
Adding a document to explain how multi-device works in Tensorflow.