I have a data having 100 features: Partial Preview of Data
I want to implement Attention to extract 32 features from these 100 and feed those to LSTM as done similarly in this paper: https://arxiv.org/abs/1902.11074
I want to implement the same architecture as given in the paper but am not able to formulate the code for doing it.
X = data.iloc[:, 1:101].values
y = data.iloc[:, 0].values
# splitting the data
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.2,
random_state=2)
X_train = X_train.reshape(-1, 1, 100)
X_test = X_test.reshape(-1, 1, 100)
print("Training & Testing Data Shape: ", X_train.shape, X_test.shape, y_train.shape, y_test.shape)
# Model
model = tf.keras.Sequential()
model.add(tf.keras.layers.LSTM(128, return_sequences=True, input_shape=(1, 100)))
model.add(tf.keras.layers.Dropout(0.3))
model.add(tf.keras.layers.LSTM(32, return_sequences=False))
model.add(tf.keras.layers.Dropout(0.3))
model.add(tf.keras.layers.Dense(1, activation = 'linear'))
# Compile Model
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01,
beta_1=0.9,
beta_2=0.999,
epsilon=1e-7)
model.compile(loss='mean_absolute_error',
optimizer=optimizer)
# Fitting the model
history = model.fit(X_train, y_train,
epochs=30,
batch_size=64,
verbose=1,
validation_split=0.2,
shuffle=True)
Some more details:
Training & Testing Data Shape: (1890, 1, 100) (473, 1, 100) (1890,) (473,)
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
lstm (LSTM) (None, 1, 128) 117248
_________________________________________________________________
dropout (Dropout) (None, 1, 128) 0
_________________________________________________________________
lstm_1 (LSTM) (None, 32) 20608
_________________________________________________________________
dropout_1 (Dropout) (None, 32) 0
_________________________________________________________________
dense (Dense) (None, 1) 33
=================================================================
Total params: 137,889
Trainable params: 137,889
Non-trainable params: 0
_________________________________________________________________