Multi-input Convolutional Neural Network for Images Classification

user_10594993 0
Viewed 2518

I am beginner in deep learning and I want to create a multi-input Convolutional Neural Network (CNN) model in Keras for Images Classification.

I am working to create a CNN model that takes two images and gives one output which is the class of the two images.

I have two datasets: type1 and type2, and each dataset contains the same classes, but the number of images in each class in the dataset type1 is higher than the number of images in each class in the dataset type2. The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to one class (ClassA or ClassB or------).

The following is the structure of the datasets.

Type1 dataset
|Train
              |ClassA
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassB
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassC
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassD
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

Type2 dataset
|Train
              |ClassA
                             |image1
                             |image2
                            -----
              |ClassB
                             |image1
                             |image2
                            -----
              |ClassC
                             |image1
                             |image2
                            -----
              |ClassD
                             |image1
                             |image2
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

The model is very similar to the model in this images, but it has more layers before flatten layer.

Multi-input

I created a custom generator that inputs two images (from type 1 and 2), and each image from type1 be paired with every images from type2 as long as these images belong to the same class (label).

The problem is when executing fit_generator I get infinite loop as the following:

  Found *** images belonging to 100 classes.

    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes. ......
.................................................................

Here is my custom generator code:

input_imgen = ImageDataGenerator( 
                                  rotation_range=10,
                                  shear_range=0.2,
                                  zoom_range=0.1,
                                  width_shift_range=0.1,
                                  height_shift_range=0.1
                                  )



test_imgen = ImageDataGenerator()



def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):


    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    while True:
      X2i = genX2.next() 
      Type1 = []
      Type2 = []
      image1 = []
      image2 = []

      while True:
        X1i = genX1.next() 
        for i in range(len(X2i[1])): #Type2
          for j in range(len(X1i[1])): #Type1
            if all(X2i[1][i]) == all(X1i[1][j]): # have same label
              image1.append(X1i[0][j]) # add image
              image1.append(X1i[1][j]) # add label
              image2.append(X2i[0][i]) # add image
              image2.append(X2i[1][i]) # add label
      Type1.append(image1)
      Type2.append(image2)
      yield [Type1 [0], Type2 [0]], Type2 [1]  #Yield both images and their mutual label


inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)       

validgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=32,
                                          img_height=224,
                                          img_width=224) 

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=test_face_data,
                                          dir2=test_face_data,
                                          batch_size=32,
                                          img_height=224,
                                          img_width=224)


    # compile the model
    multi_model.compile(
            loss='categorical_crossentropy',
            optimizer=Adam(lr=0.0001),
            metrics=['accuracy']
        )


# train the model and save the history
history = multi_model.fit_generator(
inputgenerator,
steps_per_epoch=len(train_data) // batch_size,
epochs=10,
verbose=1,
validation_data=validgenerator,
validation_steps=len(valid_data) // batch_size,
use_multiprocessing=True,
shuffle=False
)

Could you please help me to solve this problem and create the custom generator??

keras tensorflow deep-learning computer-vision machine-learning
1 Answers
class MultipleInputGenerator(Sequence):
    def __init__(self, dir_train_01, dir_train_02, batch_size):
        # Keras generator
        self.generator = ImageDataGenerator(rotation_range=15, 
                                            width_shift_range=0.2,
                                            height_shift_range=0.2,
                                            shear_range=0.2,
                                            zoom_range=0.2,
                                            horizontal_flip=True, 
                                            fill_mode='nearest')

        # Real time multiple input data augmentation
        self.genX1 = self.generator.flow_from_directory(
            dir_train_01,
            target_size = IMG_SIZE,
            batch_size  = BATCH_SIZE,
            class_mode  = 'categorical',
            color_mode  = 'rgb',          # convert to rgb
            seed        = SEED 
        )
        self.genX2 = self.generator.flow_from_directory(
            dir_train_02,
            target_size = IMG_SIZE,
            batch_size  = BATCH_SIZE,
            class_mode  = 'categorical',
            color_mode  = 'rgb',          # convert to rgb
            seed        = SEED 
        )

    def __len__(self):
        """It is mandatory to implement it on Keras Sequence"""
        return self.genX1.__len__()

    def __getitem__(self, index):
        """Getting items from the 2 generators and packing them"""
        X1_batch, Y_batch = self.genX1.__getitem__(index)
        X2_batch, Y_batch = self.genX2.__getitem__(index)

        X_batch = [X1_batch, X2_batch]

        return X_batch, Y_batch
user_53363500
Related