Concatenate CNN to compare two images

Question

I am training an CNN to compare two images. The CNN then can say if new images i put in the CNN are equal or not. Therefor i concatenate the images via cv2 so that i have images of shape (330,530,6) (in rgb) or like i would do it in grayscale (330,530,2). This works fine for me, but i would like to know how the CNN works if i concatenate two models after flattening both of them. I'm working with the keras sequential modells. Is there a way to concatenate these two models without changing everything? Because i give the Data in the fit method i m not sure how i hand over both data to each model.

Here is the Modle i work with:

CNN = Sequential()
CNN.add(layers.Conv2D(32,(3,3),activation='relu',kernel_regularizer=regularizers.l2(l2Reg),input_shape=(330,530,2)))
CNN.add(layers.MaxPool2D(pool_size=(2, 2)))
CNN.add(layers.Conv2D(32,(3,3),activation='relu',kernel_regularizer=regularizers.l2(l2Reg)))
CNN.add(layers.MaxPool2D(pool_size=(3, 3)))
CNN.add(layers.Conv2D(64,(3,3),activation='relu',kernel_regularizer=regularizers.l2(l2Reg)))
CNN.add(layers.MaxPool2D(pool_size=(3, 3)))
CNN.add(layers.Conv2D(64,(3,3),activation='relu',kernel_regularizer=regularizers.l2(l2Reg)))
CNN.add(layers.MaxPool2D(pool_size=(3, 3)))
CNN.add(layers.Flatten())
CNN.add(layers.Dropout(0.5))
CNN.add(layers.Dense(128,activation='relu',kernel_regularizer=regularizers.l2(l2Reg)))
CNN.add(layers.Dense(2,activation='softmax'))
CNN.summary()
CNN.compile(optimizer=optimizers.RMSprop(lr=1e-4),loss='categorical_crossentropy',metrics=['accuracy'])
history=CNN.fit(XTrain,YTrain,epochs=40,batch_size=32,validation_split=0.15)
scores = CNN.evaluate(XTest,YTest,batch_size=32)
print("Accuracy: %.2f%%" % (scores[1]*100))
CNN.save("AnodenerkennungDreiV")

user_13191441 · Answer

You can create a list of inputs, in your case with 2 items, let's say 'image1' and 'image2'. You can then create a branch of convolutional and pooling layers for each image that end with a Flatten layer.

Example with 330x530 images in grayscale:

import numpy as np
from keras.utils import plot_model
from keras.models import Model
from keras.layers import Conv2D, MaxPool2D, Flatten, Input, concatenate, Dense

inputs = [Input(shape=(330,530,1), name='image1'), Input(shape=(330,530,1), name='image2')]

flattened_layers = []
for input in inputs:
    conv_layer = Conv2D(32,(3,3),activation='relu')(input)
    conv_layer = MaxPool2D(pool_size=(2,2))(conv_layer) 
    # note that previous layer is used as input for creating the next layer,
    # you'll need to do this for every layer.

    # add more layers here
    flattened_layers.append(Flatten()(conv_layer))

output = concatenate(flattened_layers, axis=0) #you have to check which axis you want to use here
#add more layers here
output = Dense(2,activation='softmax')(output)

model = Model(inputs=inputs, outputs=output)
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])
plot_model(model, "C:/help_me_pls/example_model.png", show_shapes=True)

To feed data into this model you will need a dictionary as X values. X['image1'] should contain all first images and X['image2'] all second images. You should still be able to use the same y values.

Below an image showing the architecture of the example model:

I hope I've understood your question correctly and this is what you are looking for.

Concatenate CNN to compare two images

1 Answers