How can I do return_sequences for a stacked LSTM model with PyTorch?

user_239879 0
Viewed 746

I have a Tensorflow / Keras model with:


        self.model.add(Bidirectional(LSTM(lstm1_size, input_shape=(
            seq_length, feature_dim), return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Bidirectional(
            LSTM(lstm2_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        # BOTTLENECK HERE

        self.model.add(Bidirectional(
            LSTM(lstm3_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Bidirectional(
            LSTM(lstm4_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Bidirectional(
            LSTM(lstm5_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Dense(feature_dim, activation='linear'))

How do I create a stacked PyTorch model with the return_sequences? My understanding of return_sequences is that it returns the "output" of each layer of LSTM's which are then fed into the next layer.

How would I accomplish this with PyToch?

pytorch keras tensorflow python
1 Answers

PyTorch always returns sequences.

https://pytorch.org/docs/stable/nn.html#lstm

enter image description here

Example:

import torch as t

batch_size = 2
time_steps = 10
features = 2
data = t.empty(batch_size, time_steps, features).normal_()

lstm = t.nn.LSTM(input_size=2, hidden_size=3, bidirectional=True, batch_first=True)

output, (h_n, c_n) = lstm(data)
[output.shape, h_n.shape, c_n.shape]

[torch.Size([2, 10, 6]), torch.Size([2, 2, 3]), torch.Size([2, 2, 3])]

class Net(t.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.lstm_1 = t.nn.LSTM(input_size=2, hidden_size=3, bidirectional=True, batch_first=True)
        self.lstm_2 = t.nn.LSTM(input_size=2*3, hidden_size=4, bidirectional=True, batch_first=True)

    def forward(self, input):
        output, (h_n, c_n) = self.lstm_1(input)
        output, (h_n, c_n) = self.lstm_2(output)
        return output

net = Net()

net(data).shape

torch.Size([2, 10, 8])

user_56389040
Related