What are the inputs to the transformer encoder and decoder in BERT?

user_3253481 0
Viewed 3616

I was reading the BERT paper and was not clear regarding the inputs to the transformer encoder and decoder.

For learning masked language model (Cloze task), the paper says that 15% of the tokens are masked and the network is trained to predict the masked tokens. Since this is the case, what are the inputs to the transformer encoder and decoder?

BERT input representation (from the paper)

Is the input to the transformer encoder this input representation (see image above). If so, what is the decoder input?

Further, how is the output loss computed? Is it a softmax for only the masked locations? For this, the same linear layer is used for all masked tokens?

huggingface-transformers deep-learning nlp python
1 Answers

Ah, but you see, BERT does not include a Transformer decoder. It is only the encoder part, with a classifier added on top.

For masked word prediction, the classifier acts as a decoder of sorts, trying to reconstruct the true identities of the masked words. Classifying Non-masked is not included in the classification task and does not effect loss.

BERT is also trained on predicting whether a pair of sentences really does precedes one another or not.

I do not remember how the two losses are weighted.

I hope this draws a clearer picture.

user_21828570
Related