how to match a string sequence including square brackets using Google RE2

Viewed 13

I am working on modifying the tokenization code for GPT2BPEEncoder in PyTorch's text module. PyTorch GitHub file. The code uses Google's RE2 for performing pre-tokenization step (code reference). The input text is first tokenized using the following pattern:

const Regex kGPT2Regex(
    "(\\'s|\\'t|\\'re|\\'ve|\\'m|\\'ll|\\'d| ?\\pL+|"
    " ?\\pN+| ?[^\\s\\v\\pL\\pN]+|[\\s\\v]+)");
...
while (kGPT2Regex.FindAndConsume(&inp, &token)) {
...

For my feature, the code needs to skip tokenizing for special tokens such as [UNK], [SEP], and [CLS] (square brackets are considered a part of the token). To do this, I tried adding the desired tokens to the pattern like this:

const Regex kGPT2Regex(
    "(\[UNK\]|\[SEP\]|\[CLS\]|\\'s|\\'t|\\'re|\\'ve|\\'m|\\'ll|\\'d| ?\\pL+|"
    " ?\\pN+| ?[^\\s\\v\\pL\\pN]+|[\\s\\v]+)");

But it doesn't seem to help separate the special tokens. Regex 101 attempt: https://regex101.com/r/zaZd29/1

What's the correct way to use this pattern to separate special tokens to be handled separately?

0 Answers
Related