How to add a convolutional layer in the shortcut connection using functional API in Keras?

user_7575552 0
Viewed 575

Custom model

I am trying to implement a custom CNN model like the one shown in the attached figure using the functional API in Keras. I have written the code to implement the main branch but having issues in adding the 1x1 convolutions as a shortcut connection. The shortcut convolution is added for each pair of convolution blocks, just before the max-pooling layer. The code is given below:

input_shape = (256,256,3)
model_input = Input(shape=input_shape)
print(model_input) 

def custom_cnn(model_input):
    x = Conv2D(16, (3, 3), strides = (2,2), padding = 'same')(model_input)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(16, (3, 3), strides = (2,2), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(32, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(32, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(48, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(48, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(64, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(64, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(80, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(80, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
            
    x = GlobalAveragePooling2D()(x)
    x = Dense(num_classes, activation='softmax')(x)
    model = Model(inputs=model_input, outputs=x, name='custom_cnn')
    return model

#instantiate the model
custom_model = custom_cnn(model_input)

#display model summary
custom_model.summary()
conv-neural-network keras tensorflow deep-learning
2 Answers

here the implementation of the residual blocks inside your network following the schema:

enter image description here

num_classes = 3
input_shape = (256,256,3)
model_input = Input(shape=input_shape)

def custom_cnn(model_input):
    
    x = Conv2D(16, (3, 3), strides = (2,2), padding = 'same')(model_input)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(16, (3, 3), strides = (2,2), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    xx = Conv2D(16, (1,1), strides= (4,4), padding = 'same')(model_input)
    x = Add()([x,xx])
    xx = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(32, (3, 3), strides = (1,1), padding = 'same')(xx)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(32, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    xx = Conv2D(32, (1,1), strides= (1,1), padding = 'same')(xx)
    x = Add()([x,xx])
    xx = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(48, (3, 3), strides = (1,1), padding = 'same')(xx)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(48, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    xx = Conv2D(48, (1,1), strides= (1,1), padding = 'same')(xx)
    x = Add()([x,xx])
    xx = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(64, (3, 3), strides = (1,1), padding = 'same')(xx)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(64, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    xx = Conv2D(64, (1,1), strides= (1,1), padding = 'same')(xx)
    x = Add()([x,xx])
    xx = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
    
    x = Conv2D(80, (3, 3), strides = (1,1), padding = 'same')(xx)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(80, (3, 3), strides = (1,1), padding = 'same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    xx = Conv2D(80, (1,1), strides= (1,1), padding = 'same')(xx)
    x = Add()([x,xx])
    xx = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(x)
            
    x = GlobalAveragePooling2D()(xx)
    x = Dense(num_classes, activation='softmax')(x)
    model = Model(inputs=model_input, outputs=x, name='custom_cnn')
    return model

#instantiate the model
custom_model = custom_cnn(model_input)

#display model summary
custom_model.summary()
user_103750490

You can merge multiple layers with keras.layers.concatenate function so if you have 2 "parallel" layers x and y, you can do merge them into layer z like this:

z = layers.concatenate([x, y])

In your code, it would look something like this:

First, create the one "parallel" layer (note that both layers x and later y will be applied on the same layer starting with the model_input layer)

x = Conv2D(16, (3, 3), strides = (2,2), padding = 'same')(model_input)
x = BatchNormalization()(x)
x = Activation('relu')(x)
x = Conv2D(16, (3, 3), strides = (2,2), padding = 'same')(x)
x = BatchNormalization()(x)
x = Activation('relu')(x)

Then you would create the second "parallel" layer:

y = Conv2D(...)(model_input)

Merge them

z = layers.concatenate([x, y])

and apply max pooling on the z (on the layer that is the result of the concatenation)

z = MaxPooling2D(pool_size=(3, 3), strides=(2,2))(z)

And again, create two branches, both taking z as an input (where z is equivalent to the model_input from the previous iteration) and repeat for each of these blocks where you need to apply concatenation.

user_95188900
Related