FCN for Semantic Segmentation via Keras

[rmkemker]

I have seen a few iterations of people trying to solve this problem, but I haven't seen a satisfactory solution. I am using the Tensorflow backend of Keras to recreate the work found here. I am new to this arena, so I am having troubles grasping what the authors did to train their network for VOC 2012. All of the training images are different sizes; and when I look at their paper/code, I am unable to see how they account for the varying input image size. Below is an attempt I made to build part of the desired network (inspired by the author's Caffe code):

inputData = Input(batch_shape=(None,3,None,None))

#First Layer
pad1 = ZeroPadding2D((100,100))(inputData)
conv1_1 = Convolution2D(64,3,3,activation='relu',border_mode='valid')(pad1)
conv1_2 = Convolution2D(64,3,3,activation='relu',border_mode='same')(conv1_1)
pool1 = MaxPooling2D((2,2), strides=(2,2))(conv1_2)

#Second Convolution
conv2_1 = Convolution2D(128,3,3,activation='relu',border_mode='same')(pool1)
conv2_2 = Convolution2D(128,3,3,activation='relu',border_mode='same')(conv2_1)
pool2 = MaxPooling2D((2,2), strides=(2,2))(conv2_2)

#Third Convolution
conv3_1 = Convolution2D(256,3,3,activation='relu',border_mode='same')(pool2)
conv3_2 = Convolution2D(256,3,3,activation='relu',border_mode='same')(conv3_1)
conv3_3 = Convolution2D(256,3,3,activation='relu',border_mode='same')(conv3_2)
pool3 = MaxPooling2D((2,2), strides=(2,2))(conv3_3)  

#Fourth Convolution
conv4_1 = Convolution2D(512,3,3,activation='relu',border_mode='same')(pool3)
conv4_2 = Convolution2D(512,3,3,activation='relu',border_mode='same')(conv4_1)
conv4_3 = Convolution2D(512,3,3,activation='relu',border_mode='same')(conv4_2)
pool4 = MaxPooling2D((2,2), strides=(2,2))(conv4_3)    

#Fifth Convolution
conv5_1 = Convolution2D(512,3,3,activation='relu',border_mode='same')(pool4)
conv5_2 = Convolution2D(512,3,3,activation='relu',border_mode='same')(conv5_1)
conv5_3 = Convolution2D(512,3,3,activation='relu',border_mode='same')(conv5_2)
pool5 = MaxPooling2D((2,2), strides=(2,2))(conv5_3)

#Fully Convolutional Layers 32s
fc6 = Convolution2D(4096,7,7,activation='relu',border_mode='valid')(pool5)
drop6 = Dropout(0.5)(fc6)
fc7 = Convolution2D(4096,1,1,activation='relu',border_mode='valid')(drop6)
drop7 = Dropout(0.5)(fc7)
score_fr = Convolution2D(21,1,1,border_mode='valid')(drop7)

#Deconv Layer
upscore  = Deconvolution2D(21,64,64,output_shape=(inputData.get_shape()),subsample=(32,32),bias=False)(score_fr)

flat = Flatten()(upscore)
pred32 = Dense(21,activation='softmax')(flat)
model = Model(input=inputData, output=[pred32])   

However, I get the "TypeError: Expected binary or unicode string, got (Dimension(None), Dimension(None), Dimension(None), Dimension(3))'" in the Deconvolution2D layer. Is there a better way for allowing various dimensions for training data? Should I just pad the training images to 500x500? The author says that they use the entire image vs passing patches of the image. Thank-you in advance.

EDIT: I found the new Deconvolution2D layer was recently added to Keras. I modified my code to incorporate this instead and got the new error listed above.

[McVilla]

Hi, @rmkemker what version of keras you were using? In keras 1.0.0 or above, I can't import Deconvolution2d.

[rmkemker]

Hey @McVilla, it is version 1.0.6...but I had to build it from source (not yet migrated to the PIP wheel yet I think). I rebuilt it this morning.

[ziky90]

@rmkemker Any progress here with dynamical shape inference? I'm getting to exactly same problems with a little bit different model.

[kielnino]

Same here. I tried the example from https://keras.io/layers/convolutional/#deconvolution2d and got also:
TypeError: Expected binary or unicode string, got (None, 3, 14, 14)
Keras and TF are both up to date...

[gravity1989]

I suppose you are using TF to solve this.

Instead of None, you need to use the exact batch size.

On Fri, Sep 16, 2016 at 1:57 PM, kielnino [email protected] wrote:

Same here. I tried the example from https://keras.io/layers/
convolutional/#deconvolution2d and got also:
TypeError: Expected binary or unicode string, got (None, 3, 14, 14)
Keras and TF are both up to date...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3354 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABkKwB9U9qtdWIBnGFYIJBgQX18EGfX-ks5qqlLygaJpZM4JZC9D
.

Regards,
R.V.

[rmkemker]

The answer is yes, but it involved writing my own layer. It worked, but I am working on some newer models that don't use Deconvolution2D. The models I am using now infer shape without any issue.

[gravity1989]

Are you using UpSampling followed by Convolution?

On Sun, Sep 18, 2016 at 12:54 AM, rmkemker [email protected] wrote:

The answer is yes, but it involved writing my own layer. It worked, but I
am working on some newer models that don't use Deconvolution2D. The models
I am using now infer shape without any issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3354 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABkKwM8lIfFH7qDtABq7gjYFR5aqS_n2ks5qrD5pgaJpZM4JZC9D
.

Regards,
R.V.

[rmkemker]

Yes I am. It is faster.

[shehzaadzd]

@rmkemker would it be possible to share your code? I am running into a similar problem while trying to implement the semantic segmentation paper.

[ahundt]

@shehzaadzd
These items & repositories are also relevant to FCN:

• official keras-contrib repository which is the upstream source for Keras, and has a DensenetFCN implementation
• additional datasets - coco + voc2012 keras-contrib#47 incorporating coco + voc 2012
• segmentation and FCN training examples + class keras-contrib#59 dealing with gaps in FCN training performance
• DenseNetFCN not training to expected performance keras-contrib#63 training on other FCN networks
• Keras-FCN, which I was planning to adapt for a merge into keras-contrib also has a SegDataGenerator implementation with some of the features you are looking for in your comments, plus additional models and experimental support for coco.
• https://github.com/0bserver07/One-Hundred-Layers-Tiramisu

[shehzaadzd]

Hey, thanks! 😃

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[mrgloom]

Deconvolution2D is now Conv2DTranspose?
https://keras.io/layers/convolutional/#conv2dtranspose

Also if you model is fully convolutional you can use image of any size, but in batch images should be same size and training with batch_size=1 can be slower.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[xiaomaxiao]

@rmkemker can you share you code , how to slove it