I am rather new to Python and the Deep Learning topic. I using a Model with two LSTM layers in order to classify twitter sentiment. My training Dataset contains 1.6 Mio tweets, for efficiency purposes I am using only 200k tweets for tuning my model right now.
D = 75
M = 30
model = Sequential()
model.add(tf.keras.Input(shape=(T,)))
model.add(Embedding(V+1, D))
model.add(LSTM(M, dropout = 0.2, return_sequences=True,
kernel_regularizer=regularizers.L2(0.02), recurrent_dropout = 0.2))
model.add(LSTM(M, dropout = 0.2, return_sequences=True,
kernel_regularizer=regularizers.L2(0.02), recurrent_dropout = 0.2))
model.add(GlobalMaxPooling1D())
model.add(Dense(1, activation="sigmoid"))
Training and optimzer settings:
epochs = 8
learning_rate = 0.4
momentum = 0.0
opt = tf.keras.optimizers.SGD(learning_rate = learning_rate, momentum =momentum,
nesterov = False, name ="SGD")
Compiling & Fitting
model.compile(optimizer = opt, loss = "binary_crossentropy", metrics = ["accuracy"])
enter code here
r = model.fit(pad_train, y_train, validation_data=(pad_test, y_test), epochs = epochs,
batch_size = 16)
The training output for this configuartion is the best I was able to achieve when playing with the learning rate, momentum, decay, batchsize etc.
Epoch 1/8 8750/8750 - 150s 17ms/step - loss: 0.5595 - accuracy: 0.6972 - val_loss: 0.4955 - val_accuracy: 0.7594
Epoch 2/8 8750/8750 - 152s 17ms/step - loss: 0.4728 - accuracy: 0.7752 - val_loss: 0.4576 - val_accuracy: 0.7859
Epoch 3/8 8750/8750 - 154s 18ms/step - loss: 0.4569 - accuracy: 0.7849 - val_loss: 0.4537 - val_accuracy: 0.7875
Epoch 4/8 8750/8750 - 160s 18ms/step - loss: 0.4473 - accuracy: 0.7891 - val_loss: 0.4464 - val_accuracy: 0.7901
Epoch 5/8 8750/8750 - 158s 18ms/step - loss: 0.4393 - accuracy: 0.7944 - val_loss: 0.4616 - val_accuracy: 0.7847
Epoch 6/8 8750/8750 - 158s 18ms/step - loss: 0.4332 - accuracy: 0.7976 - val_loss: 0.4403 - val_accuracy: 0.7942
Epoch 7/8 8750/8750 - 156s 18ms/step - loss: 0.4267 - accuracy: 0.8014 - val_loss: 0.4404 - val_accuracy: 0.7946
Epoch 8/8 8750/8750 - 155s 18ms/step - loss: 0.4219 - accuracy: 0.8041 - val_loss: 0.4408 - val_accuracy: 0.7935
Question 1: While I am actually happy with the accuracy I am kinda suspicious with the loss I got. Maybe someone with some experience can evaluate the results I got? (In the sentiment classification context -> labelling positive or negative)
Question 2: Do you have some tips as how to improve my model further in tuning the hyperparameters or the model itself (additional layers etc.)?
Question 3: I am training my model using my CPU (9900KF). I tried it once with my RTX 3070 but it was about 5 times slower. Is there any reason for this? I thought training on GPU should be way faster?
Thanks for your help!