I am working on modifying the tokenization code for GPT2BPEEncoder in PyTorch's text module. PyTorch GitHub file. The code uses Google's RE2 for performing pre-tokenization step (code reference). The input text is first tokenized using the following pattern:
const Regex kGPT2Regex(
"(\\'s|\\'t|\\'re|\\'ve|\\'m|\\'ll|\\'d| ?\\pL+|"
" ?\\pN+| ?[^\\s\\v\\pL\\pN]+|[\\s\\v]+)");
...
while (kGPT2Regex.FindAndConsume(&inp, &token)) {
...
For my feature, the code needs to skip tokenizing for special tokens such as [UNK], [SEP], and [CLS] (square brackets are considered a part of the token). To do this, I tried adding the desired tokens to the pattern like this:
const Regex kGPT2Regex(
"(\[UNK\]|\[SEP\]|\[CLS\]|\\'s|\\'t|\\'re|\\'ve|\\'m|\\'ll|\\'d| ?\\pL+|"
" ?\\pN+| ?[^\\s\\v\\pL\\pN]+|[\\s\\v]+)");
But it doesn't seem to help separate the special tokens. Regex 101 attempt: https://regex101.com/r/zaZd29/1
What's the correct way to use this pattern to separate special tokens to be handled separately?