input/output/recurrent dropout layers in BiLSTM_Classifier and how they affect the model and prediction

user_12498420 0
Viewed 187

I would like to have some understanding/information on the input/output/recurrent dropout layers in BiLSTM_Classifier and how they affect the model and prediction.

# Output drop out
model_out_dp = Sequential()
model_out_dp.add(Embedding(vocab_size, embedding_dim, input_length=maxlen,weights=[embedding_matrix],trainable=False))
model_out_dp.add(Bidirectional(LSTM(64)))
model_out_dp.add(Dropout(0.5))
model_out_dp.add(Dense(8, activation='softmax'))

# input drop out
model_input_dp = Sequential()
model_input_dp.add(Embedding(vocab_size, embedding_dim, input_length=maxlen,weights=[embedding_matrix],trainable=False))
model_input_dp.add(Bidirectional(LSTM(64,dropout=0.5)))
model_input_dp.add(Dense(8, activation='softmax'))

# recurrent drop out
model_rec_dp = Sequential()
model_rec_dp.add(Embedding(vocab_size, embedding_dim, input_length=maxlen,weights=[embedding_matrix],trainable=False))
model_rec_dp.add(Bidirectional(LSTM(64,recurrent_dropout=0.5)))
model_rec_dp.add(Dense(8, activation='softmax'))
dropout lstm tensorflow nlp python
1 Answers

First we split 'S's and 'A's into groups per the rule -- we assign a unique `group' to each S followed by any number (including none) of As. We also number elements in each group in a sequence

df['group'] = (df['First']=='S').cumsum()
df['el'] = df.groupby('group').cumcount()

Looks like this:

    First    Second                                               group    el
--  -------  -------------------------------------------------  -------  ----
 0  S        Keeping the Secret of Genetic Testing                    1     0
 1  S        What is genetic risk ?                                   2     0
 2  S        Genetic risk refers more to your chance of inh...        3     0
 3  A        3 4|||Rloc-||||||REQUIRED|||-NONE-|||0                   3     1
 4  S        People get certain disease because of genetic ...        4     0
 5  A        1 2|||Wci|||develop|||REQUIRED|||-NONE-|||0              4     1
 6  A        3 4|||Nn|||diseases|||REQUIRED|||-NONE-|||0              4     2
 7  S        How much a genetic change tells us about your ...        5     0
 8  S        If your genetic results indicate that you have...        6     0
 9  A        8 8|||ArtOrDet|||the|||REQUIRED|||-NONE-|||0             6     1

Now we set the multi-index to 'group' and 'el' and then unstack 'el' into headers

df.set_index(['group','el'])['Second'].unstack(level=1)

so it looks like

  group  0                                                  1                                             2
-------  -------------------------------------------------  --------------------------------------------  -------------------------------------------
      1  Keeping the Secret of Genetic Testing              nan                                           nan
      2  What is genetic risk ?                             nan                                           nan
      3  Genetic risk refers more to your chance of inh...  3 4|||Rloc-||||||REQUIRED|||-NONE-|||0        nan
      4  People get certain disease because of genetic ...  1 2|||Wci|||develop|||REQUIRED|||-NONE-|||0   3 4|||Nn|||diseases|||REQUIRED|||-NONE-|||0
      5  How much a genetic change tells us about your ...  nan                                           nan
      6  If your genetic results indicate that you have...  8 8|||ArtOrDet|||the|||REQUIRED|||-NONE-|||0  nan

This looks like pretty much what you want, except the names of the columns that you can change with .rename(columns = {...}) if you need to, and .fillna(0) if you want to replace NaNs with 0s

user_145514260
Related