Python Keras LSTM learning converges too fast on high loss

user_7078744 0
Viewed 6644

This is more of a deep learning conceptual problem, and if this is not the right platform I'll take it elsewhere.

I'm trying to use a Keras LSTM sequential model to learn sequences of text and map them to a numeric value (a regression problem).

The thing is, the learning always converges too fast on high loss (both training and testing). I've tried all possible hyperparameters, and I have a feeling it's a local minima issue that causes the model's high bias.

My questions are basically :

  1. How to initialize weights and bias given this problem?
  2. Which optimizer to use?
  3. How deep I should extend the network (I'm afraid that if I use a very deep network, the training time will be unbearable and the model variance will grow)
  4. Should I add more training data?

Input and output are normalized with minmax.

I am using SGD with momentum, currently 3 LSTM layers (126,256,128) and 2 dense layers (200 and 1 output neuron)

I have printed the weights after few epochs and noticed that many weights are zero and the rest are basically have the value of 1 (or very close to it).

Here are some plots from tensorboard :enter image description here

lstm keras tensorflow deep-learning python
2 Answers

Consider reducing your batch_size. With large batch_size, it could be that your gradient at some point couldn't find any more variation in your data's stochasticity and for that reason it convergences earlier.

user_54685360
Related