Cascaded Neural Network architecture and input using TensorFlow

Question

I am currently replicating a machine learning architecture found in this paper: https://downloads.hindawi.com/journals/acisc/2018/1439312.pdf

Specifically on page 4 of the paper:

Any ideas on how to implement this on TensorFlow? My current model looks like this:

BATCH_SIZE = 32

INPUT_SIZE1 = (15, 30, 3)
INPUT_SIZE2 = (30, 60, 3)
INPUT_SIZE3 = (40 ,80, 3)

LEARNING_RATE = 0.001
EPOCHS = 10

CNN_CR1_INPUT = keras.Input(shape = INPUT_SIZE1)
CNN_CR1 = Conv2D(64, (5, 5), strides=2, padding='same', activation='relu')(CNN_CR1_INPUT)
CNN_CR1 = MaxPooling2D(3,3)(CNN_CR1)
CNN_CR1 = Conv2D(64, (3,3), strides=1, padding='same', activation='relu')(CNN_CR1)
CNN_CR1 = Conv2D(64, (3,3), strides=2, padding='same', activation='relu')(CNN_CR1)
CNN_CR1 = Flatten()(CNN_CR1)
CNN_CR1_OUTPUT = Dense(1)(CNN_CR1)

CNN_CR2_INPUT = keras.Input(shape = INPUT_SIZE2)
CNN_CR2 = Conv2D(64, (5,5), strides=2, padding='same', activation='relu')(CNN_CR2_INPUT)
CNN_CR2 = MaxPooling2D(3, 3)(CNN_CR2)
CNN_CR2 = Conv2D(64, (3,3), strides=1, padding='same', activation='relu')(CNN_CR2)
CNN_CR2 = Conv2D(64, (3,3), strides=2, padding='same', activation='relu')(CNN_CR2)
CNN_CR2 = Flatten()(CNN_CR2)
CNN_CR2_OUTPUT = Dense(1)(CNN_CR2)

CNN_CR3_INPUT = keras.Input(shape = INPUT_SIZE3)
CNN_CR3 = Conv2D(64, (5,5), strides=2, padding='same', activation='relu')(CNN_CR3_INPUT)
CNN_CR3 = MaxPooling2D(3, 3)(CNN_CR3)
CNN_CR3 = Conv2D(64, (3,3), strides=1, padding='same', activation='relu')(CNN_CR3)
CNN_CR3 = Conv2D(64, (3,3), strides=2, padding='same', activation='relu')(CNN_CR3)
CNN_CR3 = Flatten()(CNN_CR3)
CNN_CR3_OUTPUT = Dense(1)(CNN_CR3)

# SUGGESTION: This kinda weird. If this works, we only need 1 valitadation datagen? Not sure how it all connect together.
CNN_MAX = Maximum()([CNN_CR1_OUTPUT, CNN_CR2_OUTPUT, CNN_CR3_OUTPUT])

CNN_MODEL = keras.Model(inputs=[CNN_CR1_INPUT, CNN_CR2_INPUT, CNN_CR3_INPUT], outputs=[CNN_MAX])

I am not sure if the model I make is correct or not and I need some assistance. Also, how do you create the input pipeline for a cascading neural network like this one? I already tried this:

TRAIN_DATAGEN1 = ImageDataGenerator(
    # SUGGESTION: Not sure if this is needed??
    rescale = 1/255.0
)

TRAIN_GENERATOR1 = TRAIN_DATAGEN1.flow_from_directory(
    os.path.join(WORKING_DATASETS['GI4E']['train']['images'], '0'),
    target_size = (15, 30),
    class_mode ='binary',
    batch_size = BATCH_SIZE
)

TEST_DATAGEN1 = ImageDataGenerator(
    rescale = 1/255.0,
)

TEST_GENERATOR1 = TEST_DATAGEN1.flow_from_directory(
    os.path.join(WORKING_DATASETS['GI4E']['test']['images'], '0'),
    target_size = (15, 30),
    class_mode ='binary',
    batch_size = BATCH_SIZE
)

# CNN 2
TRAIN_DATAGEN2 = ImageDataGenerator(
    rescale = 1/255.0
)

TRAIN_GENERATOR2 = TRAIN_DATAGEN2.flow_from_directory(
    os.path.join(WORKING_DATASETS['GI4E']['train']['images'], '1'),
    target_size = (30, 60),
    class_mode ='binary',
    batch_size = BATCH_SIZE
)

TEST_DATAGEN2 = ImageDataGenerator(
    rescale = 1/255.0,
)

TEST_GENERATOR2 = TEST_DATAGEN2.flow_from_directory(
    os.path.join(WORKING_DATASETS['GI4E']['test']['images'], '1'),
    target_size = (30, 60),
    class_mode ='binary',
    batch_size = BATCH_SIZE
)

# CNN 3
TRAIN_DATAGEN3 = ImageDataGenerator(
    rescale = 1/255.0
)

TRAIN_GENERATOR3 = TRAIN_DATAGEN3.flow_from_directory(
    os.path.join(WORKING_DATASETS['GI4E']['train']['images'], '2'),
    target_size = (40 ,80),
    class_mode = 'binary',
    batch_size = BATCH_SIZE
)

TEST_DATAGEN3 = ImageDataGenerator(
    rescale = 1/255.0,
)

TEST_GENERATOR3 = TEST_DATAGEN3.flow_from_directory(
    os.path.join(WORKING_DATASETS['GI4E']['test']['images'], '2'),
    target_size = (40 ,80),
    class_mode = 'binary',
    batch_size = BATCH_SIZE
)

But it spits out an error when I try to fit it.

ValueError: Failed to find data adapter that can handle input: ( containing values of types {""}),

After digging through the documentation, I am aware that TensorFlow cannot handle a bunch of ImageDataGenerator together in a list. But I'm not sure how to feed images to the model without ImageDataGenerator.

So, to summarize:

How to recreate said model in TensorFlow? Is the model that I create correct already?
How to create the input pipeline for the model? Any alternative beside ImageDataGenerator?

user_20051519 · Answer

You could try building the model with tf.data - load the datas with keras.utils , for data augmentation you can add layers on to model architecture (tf.keras.layers.Rescaling,tf.keras.layers.RandomRotation etc). There are other augmentation methods that are given in the docs below which are more efficient in utilizing the GPU that ImageDataGenerator

https://www.tensorflow.org/tutorials/images/data_augmentation#data_augmentation_2 https://www.tensorflow.org/api_docs/python/tf/keras/utils https://www.tensorflow.org/guide/keras/preprocessing_layers

Cascaded Neural Network architecture and input using TensorFlow

1 Answers