How does the multi-input deep learning work in Keras?

user_10594993 0
Viewed 464

I have a multi-input convolutional neural network model that inputs 2 images from 2 datasets to give one output which is the class of the two inputs. The two datasets have the same classes. I used 2 vgg16 models and concatenate them to classify the two images.

vgg16_model = keras.applications.vgg16.VGG16()
input_layer1= vgg16_model .input
last_layer1 = vgg16_model.get_layer('fc2').output


vgg16_model2 = keras.applications.vgg16.VGG16()
input_layer2= vgg16_model .input
last_layer2 = vgg16_model.get_layer('fc2').output


con = concatenate([last_layer1, last_layer2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(con)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2], outputs=[output_layer])

My questions are:

1- Which case from the following represents how the images enter to the model?

One to One

database1-img1 + database2-img1

database1-img2 + database2-img2

database1-img3 + database2-img3

database1-img4 + database2-img4

.........

Many to many

database1-img1 + database2-img1

database1-img1 + database2-img2

database1-img1 + database2-img3

database1-img1 + database2-img4

database1-img2 + database2-img1

database1-img2 + database2-img2

database1-img2 + database2-img3

database1-img2 + database2-img4

.........

2- In general in deep learning, Does the images enter from the two datasets to the model at the same time have the same class (labels) or not?

keras tensorflow deep-learning machine-learning neural-network
1 Answers
  1. It is a 1:1 mapping, the same should be with multiple outputs as well.

When you have a model such as Model(inputs=[input_layer1, input_layer2], outputs=[output_layer]) or even Model(inputs=[input_layer1, input_layer2], outputs=[output_layer1, output_layer2]) , You must feed it with inputs / output of the same shape.
Assume the other case - You will need to have ds1.shape[0] * ds2.shape[0] different labels, for each possible mix of the 2 datasets, and will need to have them ordered at a certain way. That is not really feasible, at least not simply.

2. Its not as if the same images have the same label, but the Pair of both images have a single label.

user_44669620
Related