How to pass several images through one SeparableConv2D?

user_11589183 0
Viewed 77

I have several images as input. I know I can use 3D convolutional layers, but I don't want to do it. Instead I want to find the patterns in the 2 dimentions of images.

I mean each image should be passed through SeparableConv2D, like that:

# this code raises an error ValueError: Input 0 of layer <name> is 
# incompatible with the layer: expected ndim=4, found ndim=5.
model = Sequential([
    Input(shape=(16, 128, 128, 1)),
    SeparableConv2D(32, 3),
    GlobalAvgPool3D(),
])

I know, I can use here Conv3D as Conv2D:

model = Sequential([
    Input(shape=(16, 128, 128, 1)),
    Conv3D(32, [1, 3, 3]),
    GlobalAvgPool3D(),
])

But I need exactly SeparableConv2D.

Maybe I can do it through the custom layer or something else? I just can't even imagine a solution.

P.S. Each input contains several images.

conv-neural-network keras tensorflow computer-vision python
2 Answers

Just to make sure your input shape should be 4D.As SeparableConv2D expects input of shape 4D tensor with shape: (batch_size, channels, rows, cols) if data_format='channels_first' or 4D tensor with shape: (batch_size, rows, cols, channels) if data_format='channels_last'.

Working example

import tensorflow as tf
input_shape = (16, 128, 128, 1)
x = tf.random.normal(input_shape)
y = tf.keras.layers.SeparableConv2D( 2, 3, activation='relu', input_shape=input_shape[1:])(x)
print(y.shape)

Output

(16, 126, 126, 2)
user_142902440

if you have images such as frames of video , and they are connected to each other in somehow you can loop over the conv2d and pass them one by one

check this example from my git repo for video action recognition

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        num_classes = 1
        dr_rate= 0.2
        pretrained = True
        rnn_hidden_size = 30
        rnn_num_layers = 2
        #get a pretrained vgg19 model ( taking only the cnn layers and fine tun them)
        baseModel = models.vgg19(pretrained=pretrained).features  
        i = 0
        for child in baseModel.children():
            if i < 28:
                for param in child.parameters():
                    param.requires_grad = False
            else:
                for param in child.parameters():
                    param.requires_grad = True
            i +=1

        num_features = 12800
        self.baseModel = baseModel
        self.dropout= nn.Dropout(dr_rate)
        self.rnn = nn.LSTM(num_features, rnn_hidden_size, rnn_num_layers , batch_first=True)
        self.fc2 = nn.Linear(30, 256)
        self.fc3 = nn.Linear(256, num_classes)
    def forward(self, x):
        batch_size, time_steps, C, H, W = x.size()
        # reshape input  to be (batch_size * timesteps, input_size)
        x = x.contiguous().view(batch_size * time_steps, C, H, W)
        x = self.baseModel(x)
        x = x.view(x.size(0), -1)
        #make output as  ( samples, timesteps, output_size)
        x = x.contiguous().view(batch_size , time_steps , x.size(-1))
        x , (hn, cn) = self.rnn(x)
        x = F.relu(self.fc2(x[:, -1, :])) # get output of the last  lstm not full sequence
        x = self.dropout(x)
        x = self.fc3(x)
        return x

the main idea is in this block we distribute each frame or image to the conv net and than we reshape or even feed its to new network

# reshape input  to be (batch_size * timesteps, input_size)
 x = x.contiguous().view(batch_size * time_steps, C, H, W)
 # feed to the pre-trained conv model
 x = self.baseModel(x)
 # flatten the output
 x = x.view(x.size(0), -1)
 # make the new correct shape (batch_size , timesteps , output_size)
 x = x.contiguous().view(batch_size , time_steps , x.size(-1))  # this x is now ready to be entred or feed into lstm layer
user_25808310
Related