I'm starting to wrap my head around the transformer architecture, but there are some things that I am not yet able to grasp.
In decoder-free transformers, such as BERT, the tokenizer includes always the tokens CLS and SEP before and after a sentence. I understand that CLS acts both as BOS and as a single hidden output that gives the classification information, but I am a bit lost about why does it need SEP for the masked language modeling part.
I'll explain a bit more about the utility I expect to get. In my case, I want to train a transformer to act as an autoencoder, so target = input. There would be no decoder, since my idea is to reduce the dimensionality of the original vocabulary into less embedding dimensions, and then study (not sure how yet, but will get there) the reduced space in order to extract useful information.
Therefore, an example would be:
string_input = "The cat is black"
tokens_input = [1,2,3,4]
string_target = "The cat is black"
tokens_output = [1,2,3,4]
Now when tokenizing, assuming that we tokenize in the basis of word by word, what would be the advantage of adding BOS and EOS?
I think these are only useful when you are using the self-attention decoder, right? so, since in that case, for the decoder the outputs would have to enter right-shifted, the vectors would be:
input_string = "The cat is black EOS"
input_tokens = [1,2,3,4,5]
shifted_output_string = "BOS The cat is black"
shifted_output_tokens = [6,1,2,3,4]
output_string = "The cat is black EOS"
output_token = [1,2,3,4,5]
However, BERT does not have a self-attention decoder, but a simple feedforward layer. That is why I'm not sure of understanding the purpose of these special tokens.
In summary, the questions would be:
- Do you always need BOS and EOS tokens, even if you don't have a transformer decoder?
- Why does BERT, that does not have a transformer decoder, require the SEP token for the masked language model part?