BatchNormsync with Adam Optimizer

[tarun005]

Is the bnsync code written specifically for SGD optimizer? The loss is not converging if I use and train the model with Adam optimizer.

[d-li14]

@tarun005 Have you tested with SGD optimizer? Does it drive the training process to convergence?

[tarun005]

Yes, the model converges with SGD, but same model does not if I replace SGD with adam.

[d-li14]

@tarun005 Although I suppose that BN should be irrelevant to the optimization method, when I used the syncbn by just adding the folder lib to $PATH, I met an error saying 'segmentation fault'. What's your usage?

[tarun005]

Agree that BN shouldn't be relevant to optimization method, but I have read somewhere that Adam requires global statistics at every iteration, so the implementation of BNsync given here could be an issue.

[jakubLangr]

Hi were either of you @tarun005 @d-li14 be able to upload the models to e.g. dropbox as they don't seem to be accessible on the Princeton site. That'd be awesome!