Parallel Convolutions using Keras

Question

What i am trying to do is create a text classification model which combines CNNS and word embeddings.The basic idea is that we have an Embedding layer at the start of the network and then 2 parallel convolutional networks for find 2,3 - grams.
Each of these convolution layers takes the output of the embedding layer as input.
After the outputs of the two cnn layers are concatenated,flattend and feeded to a Dense.
My input is tokenized,numerical sentences of length 27(shape = (None,27)) and i have 1244 of these sentences.
I've managed to create a sequential model wit ha single cnn layer but struggle wit hthe above My code so far :

input_shape = Embedding(voc, 100,weights=[embedding_matrix], input_length=X.shape[1])

tower_1 = Conv1D(filters=100, kernel_size=2, activation='relu')(input_shape)
tower_1 = MaxPooling1D(pool_size=2)(tower_1)

tower_2 =  Conv1D(filters=100, kernel_size=3, activation='relu')(input_shape)
tower_2 = MaxPooling1D(pool_size=2)(tower_2)


merged = keras.layers.concatenate([tower_1, tower_2,], axis=1)
merged = Flatten()(merged)

out = Dense(3, activation='softmax')(merged)
model = Model(input_shape, out)

This produces this error:

TypeError: Inputs to a layer should be tensors. Got:

i have also trid replacing

input_shape = Embedding(voc, 100,weights=[embedding_matrix], input_length=X.shape[1])

with:

input_tensor = Input(shape=(1244,27))
input_shape = Embedding(voc, 100,weights=[embedding_matrix], input_length=X.shape[1])(input_tensor)

which gives me this error:

ValueError: Input 0 of layer "max_pooling1d_23" is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: (None, 1244, 26, 100)

user_9657861 · Answer

You should define your Input layer without the number of samples. Just the sentence length:

import tensorflow as tf

inputs = tf.keras.layers.Input((27,))
embedded = tf.keras.layers.Embedding(50, 100, input_length=27)(inputs)

tower_1 = tf.keras.layers.Conv1D(filters=100, kernel_size=2, activation='relu')(embedded)
tower_1 = tf.keras.layers.MaxPooling1D(pool_size=2)(tower_1)

tower_2 = tf.keras.layers.Conv1D(filters=100, kernel_size=3, activation='relu')(embedded)
tower_2 = tf.keras.layers.MaxPooling1D(pool_size=2)(tower_2)

merged = tf.keras.layers.concatenate([tower_1, tower_2,], axis=1)
merged = tf.keras.layers.Flatten()(merged)

out = tf.keras.layers.Dense(3, activation='softmax')(merged)
model = tf.keras.Model(inputs, out)
print(model.summary())

Usage:

samples = 5
random_input = tf.random.uniform((samples, 27), maxval=50, dtype=tf.int32)
print(model(random_input))

tf.Tensor(
[[0.31525075 0.33163014 0.3531191 ]
 [0.3266019  0.3295619  0.34383622]
 [0.32351935 0.32669052 0.34979013]
 [0.32954428 0.33178467 0.33867106]
 [0.32966062 0.3283257  0.34201372]], shape=(5, 3), dtype=float32)

Parallel Convolutions using Keras

1 Answers