AlexNet bacward shape missmatch + ReLu return a tuple

[Belegkarnil]

Hi,

I have implemented AlexNet in singa but I obtain an error during the backward_and_update instruction. I am using Singa 3.0.0.rc1 on cpu.

This is my AlexNet implementation:
`from singa import autograd
from singa import module
from singa import opt

all = ['AlexNet', 'alexnet']

class AlexNet(module.Module):
def init(self, num_classes=1000):
super(AlexNet, self).init()
# 12 sur GPU donc 6 & 6
self.features1 = [
autograd.Conv2d(3,64,kernel_size=11,stride=4,padding=2),
autograd.ReLU(),
autograd.MaxPool2d(kernel_size=3, stride=2),
autograd.Conv2d(64,192,kernel_size=5,padding=2),
autograd.ReLU(),
autograd.MaxPool2d(kernel_size=3, stride=2),
autograd.Conv2d(192,384,kernel_size=3,padding=1),
autograd.ReLU(),
autograd.Conv2d(384, 256,kernel_size=3,padding=1),
autograd.ReLU()
]
self.features2 = [
autograd.Conv2d(256, 256,kernel_size=3,padding=1),
autograd.ReLU(),
autograd.MaxPool2d(kernel_size=3, stride=2)
]
self.avgpool = autograd.AvgPool2d(6, stride=1)
self.flatten = autograd.Flatten()
self.classifier = [
autograd.Dropout(),
autograd.Linear(256 * 6 * 6, 4096),
autograd.ReLU(),
autograd.Dropout(),
autograd.Linear(4096, 4096),
autograd.ReLU(),
autograd.Linear(4096, num_classes)
]
self.optimizer = opt.SGD(lr=0.001, momentum=0.9)
def loss(self, out, ty):
return autograd.softmax_cross_entropy(out, ty)
def optim(self, loss, dist_option, spars):
if dist_option == 'fp32':
self.optimizer.backward_and_update(loss)
elif dist_option == 'fp16':
self.optimizer.backward_and_update_half(loss)
elif dist_option == 'partialUpdate':
self.optimizer.backward_and_partial_update(loss)
elif dist_option == 'sparseTopK':
self.optimizer.backward_and_sparse_update(loss, topK=True, spars=spars)
elif dist_option == 'sparseThreshold':
self.optimizer.backward_and_sparse_update(loss, topK=False, spars=spars)
def forward(self, x):
for (i,layers) in enumerate([self.features1, self.features2, [ self.avgpool,self.flatten ] , self.classifier]):
for (j,fn) in enumerate(layers):
x = fn(x)
if(type(x) is tuple):# FIXME I have to do that because of a bug in Singa? (ReLU)
x = x[0]
return x

def alexnet(**kwargs):
return AlexNet(**kwargs)
`
And I get : AssertionError: ('shape mismatch', (9216, 4096), (256, 4096))
Which is my first linear layer : 256 * 6 * 6, 4096

When I use my VGG16 implementation, I got a similar error :
AssertionError: ('shape mismatch', (25088, 4096), (512, 4096))

It seems that the backward operation does not map the correct shape to the corresponding layer.

Moreover, the ReLu class return a 1-tuple containing a Tensor. Is it intended or is it a bug?

[dcslin]

Hi, as pointed out by @chrishkchris , the convention is to use RELU as stateless layer.
usage:
https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40

For shape mismatch, you might need to check the shape of layers again. Let me know if further info is required.

[Belegkarnil]

Ok, I'll try but why to provide a statefull ReLU Layer? Is it for a specific purpose?

[Belegkarnil]

I compared my implementation to other frameworks and it is the same shapes.
Moreover the forward pass does not cause any issue, it is the backward pass.
This is why I suspect a bug. Is it possible?

[nudles]

Hi, as pointed out by @chrishkchris , the convention is to use RELU as stateless layer.
usage:
https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40

For shape mismatch, you might need to check the shape of layers again. Let me know if further info is required.

@dcslin Did you try to run the code pasted by @Belegkarnil ?
Can you reproduce the error?

[dcslin]

Hi, as pointed out by @chrishkchris , the convention is to use RELU as stateless layer.
usage:
https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40
For shape mismatch, you might need to check the shape of layers again. Let me know if further info is required.

@dcslin Did you try to run the code pasted by @Belegkarnil ?
Can you reproduce the error?

I am still checking the code

[dcslin]

Hi @Belegkarnil, you might need to change 256 * 6 * 6, 4096 to 256, 4096 to make it works.

Also you are recommended to use relu/dropout/flatten like this https://github.com/apache/singa/blob/master/examples/cnn/model/cnn.py#L40

[Belegkarnil]

Ok thanks a lot ! I assumed that it works like other frameworks but that the result of AvgPool has a different shape.