How to change input dimensions for LSTM layer

user_15236214 0
Viewed 830

I need to make a model that has 2 dropout layers and two LSTM layers. Unfortunately I have a problem with input shape that goes to my second LSTM layer. After searching for the problem I found out I need to change the input dimensions but I don't know how to do that. I found an option that requires using Lambda layer but I can't import it to my environmet (it's a coursera environment). Have you got any suggestions how to deal with my error?

model = Sequential()
Layer1 = model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
Layer2 = model.add(Bidirectional(LSTM(20)))
Layer3 = model.add(Dropout(.03))
Layer4 = model.add(LSTM(20))
Layer5 = model.add(Dense(total_words, 
    kernel_regularizer=regularizers.l1_l2(l1=1e-5, l2=1e-4),
    bias_regularizer=regularizers.l2(1e-4),
    activity_regularizer=regularizers.l2(1e-5)))
          # A Dense Layer including regularizers
Layer6 = model.add(Dense(total_words, activation = 'softmax'))
          
# Pick an optimizer
          
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.summary()

Error:

ValueError: Input 0 of layer lstm_20 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 40]
recurrent-neural-network keras tensorflow machine-learning python
1 Answers

Thank you @Marco Cerliani and @Joanna Kastelik for the update.

For the benefit of community providing solution here using sample data as shown below

import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM, Dropout,  Dense
from tensorflow.keras.regularizers import l1_l2, l2

total_words = 478
max_sequence_len = 90

model = Sequential()
Layer1 = model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
Layer2 = model.add(Bidirectional(LSTM(20)))
Layer3 = model.add(Dropout(.03))
Layer4 = model.add(LSTM(20))
Layer5 = model.add(Dense(total_words, 
    kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4),
    bias_regularizer=l2(1e-4),
    activity_regularizer=l2(1e-5)))
          # A Dense Layer including regularizers
Layer6 = model.add(Dense(total_words, activation = 'softmax'))
          
# Pick an optimizer
          
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.summary()

Output:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-8ce04225c92d> in <module>()
     12 Layer2 = model.add(Bidirectional(LSTM(20)))
     13 Layer3 = model.add(Dropout(.03))
---> 14 Layer4 = model.add(LSTM(20))
     15 Layer5 = model.add(Dense(total_words, 
     16     kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4),

8 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/input_spec.py in assert_input_compatibility(input_spec, inputs, layer_name)
    221                          'expected ndim=' + str(spec.ndim) + ', found ndim=' +
    222                          str(ndim) + '. Full shape received: ' +
--> 223                          str(tuple(shape)))
    224     if spec.max_ndim is not None:
    225       ndim = x.shape.rank

ValueError: Input 0 of layer lstm_1 is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: (None, 40)

Fixed code:

Your issue can be resolved once you add return_sequences=True to the LSTM (i.e, Layer2) layer.

model = Sequential()
Layer1 = model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
Layer2 = model.add(Bidirectional(LSTM(20, return_sequences=True)))
Layer3 = model.add(Dropout(.03))
Layer4 = model.add(LSTM(20))
Layer5 = model.add(Dense(total_words, 
    kernel_regularizer=l1_l2(l1=1e-5, l2=1e-4),
    bias_regularizer=l2(1e-4),
    activity_regularizer=l2(1e-5)))
          # A Dense Layer including regularizers
Layer6 = model.add(Dense(total_words, activation = 'softmax'))
          
# Pick an optimizer
          
model.compile(loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.summary()

Output:

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 89, 64)            30592     
_________________________________________________________________
bidirectional_1 (Bidirection (None, 89, 40)            13600     
_________________________________________________________________
dropout_1 (Dropout)          (None, 89, 40)            0         
_________________________________________________________________
lstm_3 (LSTM)                (None, 20)                4880      
_________________________________________________________________
dense (Dense)                (None, 478)               10038     
_________________________________________________________________
dense_1 (Dense)              (None, 478)               228962    
=================================================================
Total params: 288,072
Trainable params: 288,072
Non-trainable params: 0
_________________________________________________________________
user_142906810
Related