I have this network where x_tr2 has a shape of (120k,4,3,3,1) and x_tr1 has shape of (120k, 120,140,3). For each entry in x_tr1 there's a corresponding temporal information stacked in the shape of (4,3,3,1) in x_tr2, which is passed in LSTM network whenever a new entry comes in x_tr1. I tried using time distributed layer for this but it moves with (time step=1), I rather want something which can move with (time step=4), Here's my code: Basically I want x_tr2[0], x_tr2[1], x_tr2[2], x_tr2[3] in time distributed stack whenever x_tr1[0] enters model_vgg19. Similarly x_tr2[4], x_tr2[5], x_tr2[6], x_tr2[7] in time distributed stack whenever x_tr1[1] enters model_vgg19.
temp_inp = Input(shape = (4, 3,3, 1))
lstm1 = TimeDistributed(Convolution2D(32, (3,3), strides = (1, 1), activation = 'relu'))(temp_inp)
lstm2 = TimeDistributed(Flatten())(lstm1)
lstm3 = TimeDistributed(BatchNormalization())(lstm2)
decoder = Bidirectional(LSTM(25,return_sequences=False))(lstm3)
img_inp = Input(shape = (120, 144, 3))
model_vgg19 = InceptionV3(weights = 'imagenet', include_top = False)(img_inp)
pool = GlobalAveragePooling2D()(model_vgg19)
b = Dropout(0.2)(pool)
c=Dense(10,activation='relu')(b)
cnn_out = Dense(10, activation = 'relu')(c)
merged = keras.layers.concatenate([cnn_out, decoder], axis=1)
layer2=Dense(1, activation = 'sigmoid')(merged)
merged_model=Model(inputs=[img_inp,temp_inp],outputs= layer2)
merged_model.summary()
merged_model.compile(loss = 'binary_crossentropy', optimizer = 'SGD', metrics = ['accuracy', tf.keras.metrics.AUC()])
training = merged_model.fit([x_tr1,x_tr2], y_tr,validation_data=([x_te1,x_te2],y_te), epochs = 10, shuffle = False, batch_size = 64)