why it is so slow to use the caffe engine

[andeyeluguo]

it train the imagenet about ten thousand epoch during one day.I use the caffe engine,because when I use the cudnn engine ,it occors error.

[tidyzhang]

I encountered the same error, without cudnn engine, it can train successfully

[shicai]

The default caffe implementations of group conv is very slow, you should implement it by yourself for efficient training.

[Derekglk]

But why cudnn doesn't work in this case so that we have to use caffe engine? cudnn doesn't support channel-wise convolution, or it has some internal bugs?
The problem I met is that if I don't uncomment "#engine:CAFFE", my platform with 4G memory will soon run out of memory and caffe gets killed.
Thank you

[shicai]

cudnn allocates memory for each group.
since there are lots of groups in mobilenet, it is not enough even with 12GB memory.
you can take a look at this for speedup: BVLC/caffe#5665

[Derekglk]

Thank you, it's good to know the reasons.
Right now I used @yonghenglh6 his DepthwiseConvolution implementation, which runs much faster than caffe's group.
Ref: https://github.com/yonghenglh6/DepthwiseConvolution