How to use the ground truth answers from the previous time step for the decoders

Viewed 64

I found this on an article here:

Decoder takes the hidden state of the last Encoder RNN cell as the initial state of its first RNN cell along with the token as the initial input to produce an output sequence. We use Teacher Forcing for faster and efficient training of the decoder. Teacher forcing is a method for quickly and efficiently training recurrent neural network models that use the ground truth from a prior time step as input. In this method, the right answer is given as the beginning of training so that the model will train quickly and efficiently.

enter image description here

Now, I am unable to use the Teacher Forcing Method for the decoder. Can someone help me out?

What I have done so far

Now, I am taking many English sentences and tokenizing them, and then keeping it in a variable X. The shape of X is [7000, 5], which means that there are 7000 English sentences, each which has 5 words in it. I am doing the same with German Sentences and keeping it in a variable Y, which has shape of [7000, 10, 1], which means that there are 7000 total German Sentences, each having 10 words in it (I am not hot-encoding it, as I am using sparse categorical cross-entropy as the loss function). Then, I defined my model which is this:

    Model = Sequential([
          Embedding(english_vocab_size, 256, input_length=english_max_len, mask_zero=True),
          LSTM(256, activation='relu'),
          RepeatVector(german_max_len),
          LSTM(256, activation='relu', return_sequences=True),
          Dense(german_vocab_size, activation='softmax')
    ])

The english_vocab_size and german_vocab_size is the total number of English words and German words present in the English and German vocabulary. The english_max_len and german_max_len are total number of words each English and German sentence has in it.

Now, from here what should I do to use the Teacher Forcing Technique?

0 Answers
Related