I've read many questions on SO about how to use the regex token \b for Unicode text (since \b only detects word boundaries in ASCII text reliably). All the answers (if I'm not mistaken) end up suggesting using some form of lookahead (and/or lookbehind) to find word boundaries in Unicode text.
However, lookaheads are not exactly the same as \b in one important way. The re2 python library requires that you don't use lookaheads in order to compile your regex. The reason for this is that there is no way to compile the regex to a finite-state automata with a lookahead without backtracking.
I've found that a large regex (thousands of characters long) is run against thousands of paragraphs the speed difference is drastic. It takes about 20 minutes with a lookahead and 2 minutes with \b (on ASCII text). The issue is I need to run this regex on Unicode.
My question is, is there an equivalent to \b (at least in Python) that maintains the zero-width property? The zero-width property allows you to avoid using lookaheads since it doesn't consume the space around the match.
Edit:
To explain the use case a bit more, I am doing Named Entity Recognition (per this definition of NER) in Unicode text. For simplicity, I'll give an English example:
John Bob and Sam went to the store
I have a regex with a list of names. The code to find these 3 names would be
word_boundary = "\b"
reg = re2.compile(fr"{word_boundary}(John|Bob|Sam){word_boundary}")
Since John and Bob are next to each other in the text, we require either
- a zero-width
word_boundaryso the space is not consumed between John and Bob. Currently I only know of\bbut this will not work for my Unicode text. - a lookahead so the space is not consumed between John and Bob. I don't want to do this because of the speed cost.
Using \P{L} as word_boundary would not work here.