I'm using the mT5 model from HuggingFace Transformers to perform some conditional generation and apply it to text classification.
I noticed that whatever the T5 model generates (and before it's trained sufficiently) it usually generates special tokens that look like <extra_id0> or <extra_id1>. These tokens aren't contained in T5Tokenizer.special_tokens either and I'm wondering what they mean.
I looked at the HuggingFace page for T5 and it seems like these are special tokens that are used during MLM training? Is that correct?
I couldn't find any details about these tokens in the original paper or anywhere else online strangely enough. Does anybody know what the meaning of these tokens are? Thanks.