Understanding BERT vocab [unusedxxx] tokens:

user_12769533 0
Viewed 3938

I am trying to understand BERT vocab here. It has 1000 [unusedxxx] tokens. I don't follow the usage of these tokens. I understand other special tokens like [SEP], [CLS], but what is [unused] used for?

Thanks!

huggingface-transformers
1 Answers

A quick search reveals the use of this, specifically in the discussion of the original BERT implementation, and this HuggingFace thread.

Unused tokens are helpful if you want to introduce specific words to your fine-tuning or further pre-training procedure; they allow you to treat words that are relevant only in your context just like you want, and avoid subword splitting that would occur with the original vocabulary of BERT. To quote from the first discussion:

Just replace the "[unusedX]" tokens with your vocabulary. Since these were not used they are effectively randomly initialized.

user_36072030
Related