Tensorflow Timeline Shows The Gradients Average Is The Performance Bottleneck When Using Multiple Gpus

I use multiple (actually 2) GPUs to train a network. The network works well but I found the training speed fluctuates. This is the snipet I used for profiling: for i in range(

Solution 1:

I try to move the gradient average and gradient descend to GPU:0. Because my GPUs have peer2peer connections, the data move is fast and the computation in GPU is also fast. Place all these ops in the first GPU nearly solve my problem. It is welcomed if anyone has other comments :D