Very Low Accuracy With LSTM

user_10371314 0
Viewed 631

I don't have any idea about low accuracy in my configuration (always 0.1508). Data shape : (1476,1000,1)

scaler = MinMaxScaler(feature_range=(0,1))
scaled_X = scaler.fit_transform(train_Data)

....

 myModel = Sequential()

 myModel.add(LSTM(128,input_shape=(myData.shape[1:]),activation='relu',return_sequences=True))
 myModel.add(Dropout(0.2))
 myModel.add(BatchNormalization())

 myModel.add(LSTM(128,activation='relu',return_sequences=True))
 myModel.add(Dropout(0.2))
 myModel.add(BatchNormalization())

 myModel.add(LSTM(64,activation='relu',return_sequences=True))
 myModel.add(Dropout(0.2))
 myModel.add(BatchNormalization())

 myModel.add(LSTM(32,activation='relu'))
 myModel.add(Dropout(0.2))
 myModel.add(BatchNormalization())

 myModel.add(Dense(16,activation='relu'))
 myModel.add(Dropout(0.2))

 myModel.add(Dense(8,activation='softmax'))
 #myModel.add(Dropout(0.2))

 opt = tf.keras.optimizers.SGD(lr=0.001,decay=1e-6)
 ls  = tf.keras.losses.categorical_crossentropy

Also sometimes following warnings:

W1014 21:02:57.125363  6600 ag_logging.py:146] Entity <function Function._initialize_uninitialized_variables.<locals>.initialize_variables at 0x00000188C58C3E18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: 
WARNING: Entity <function Function._initialize_uninitialized_variables.<locals>.initialize_variables at 0x00000188C58C3E18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause:
lstm keras tensorflow
1 Answers

Two main culprits are: Dropout layers, data preprocessing. In detail & others:

  • Dropout on stacked LSTMs is known to yield poor performance, as it introduces too much noise for stable time-dependency feature extraction. Fix: use recurrent_dropout
  • If you are working with signal data, or otherwise data with (1) outliers; (2) phase information; (3) frequency information - MinMaxScaler will corrupt the latter two plus amplitude information per (1). Fix: use StandardScaler, or QuantileTransformer
  • Consider using Nadam optimizer over SGD; it proved vastly dominant in my LSTM applications, and is generally more hyperparameter-robust than SGD
  • Consider using CuDNNLSTM; it can run 10x faster
  • Ensure your data's shaped properly for LSTM: (batch_size, timesteps, features) - or equivalently, (samples, timesteps, channels)

Note of warning: if you do use recurrent_dropout, use activation='tanh', as 'relu' is unstable.

UPDATE: True culprit: insufficient data. Details here

user_101337970
Related