Embedding and Concatenate - Tensorflow Keras

user_13236659 0
Viewed 1855

I´m trying to replicate a Neural Network. The Neural network´s architecture is LSTM Model. The first input is a hashed word as binary vector of size 2^18 which is embedded in a trainable 500 dimensional distributed representation using an embedding layer.

The number of words differs from each batch element. After the words were embedding and dropout is applied, i need to concatenate with the features vector, which has 24 features for each word.

The problem is that first input after embbeding has different dimensions of feature vectors. The words embbedings has a dimension (None, None, 18, 500) and features vetors has a dimension (None, None, 24). The first None elem is the batch size, the second None is the num of words for each batch elem.

How can i concatenate the embbeding word with feature vector?

Below is my code:

        inputs = Input(shape=(None, 18,), dtype=np.int16, name="Inp1")
        embbed_input = Embedding(input_dim=1, output_dim=500, input_length=18)
        aux = embbed_input(inputs)
        aux = Dropout(rate=self.dropout_rate)(aux)
        inputs_feat = Input(shape=(None, 24,), dtype=np.float32, name="Inp2")
        aux = concatenate([aux, inputs_feat], axis=2) #ValueError here 
        aux = Dense(units=600, activation="relu")(aux)
        aux = Dense(units=600, activation="relu")(aux)
        aux = Bidirectional(LSTM(units=400, return_sequences=True))(aux)
        aux = Dropout(rate=self.dropout_rate)(aux)
        aux = Dense(units=600, activation="relu")(aux)
        aux = Dense(units=600, activation="relu")(aux)
        aux = Dense(units=29, activation="sigmoid")(aux)


        ValueError: A 'Concatenate' layer require inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, None, 18, 500), (None, None, 24)]
keras tensorflow embedding concatenation python
1 Answers

Since a single step input contains 18 inputs for embedding layer, we might need to reshape the embedding layer output to flatten the final two dimensions.

    # import Reshape layer
    from tf.keras.layers import Reshape

    inputs = Input(shape=(None, 18,), dtype=np.int16, name="Inp1")
    embbed_input = Embedding(input_dim=1, output_dim=500, input_length=18)
    aux = embbed_input(inputs)
    # Note: it is generally not a great idea to add dropout just after the embedding layer
    aux = Dropout(rate=self.dropout_rate)(aux)
    # before the concatenate layer, reshape it to (None, None, 500*18)
    aux = Reshape(target_shape=(-1,-1,500*18))(aux)
    inputs_feat = Input(shape=(None, 24,), dtype=np.float32, name="Inp2")
    aux = concatenate([aux, inputs_feat], axis=2) # no need to specify axis when it is the final dimension
    aux = Dense(units=600, activation="relu")(aux)
    aux = Dense(units=600, activation="relu")(aux)
    aux = Bidirectional(LSTM(units=400, return_sequences=True))(aux)
    aux = Dropout(rate=self.dropout_rate)(aux)
    aux = Dense(units=600, activation="relu")(aux)
    aux = Dense(units=600, activation="relu")(aux)
    aux = Dense(units=29, activation="sigmoid")(aux)
user_117746840
Related