What do the `<extra_idx>` tokens in T5 mean?

Viewed 17

I'm using the mT5 model from HuggingFace Transformers to perform some conditional generation and apply it to text classification.

I noticed that whatever the T5 model generates (and before it's trained sufficiently) it usually generates special tokens that look like <extra_id0> or <extra_id1>. These tokens aren't contained in T5Tokenizer.special_tokens either and I'm wondering what they mean.

I looked at the HuggingFace page for T5 and it seems like these are special tokens that are used during MLM training? Is that correct?

I couldn't find any details about these tokens in the original paper or anywhere else online strangely enough. Does anybody know what the meaning of these tokens are? Thanks.

0 Answers
Related