How do I tokenize to separate apostrophes?

zen145 0
Viewed 32

I have a function made to tokenize the words given; however, the problem ensures in the fact that it does separate 'isn't' and 'o'brian' and similar words. This is the code:

from typing import List
import re

def tokenize(text: str) -> List[str]:
    return re.sub(r'(\W+)', r' \1 ', text.lower()).split()

When I put in something like

"He said 'Isn't O'Brian the best?'"

it'll come as

['he', 'said', "'", "isn't", "o'brian", 'the', 'best', '?', "'"],

not

['he', 'said', "'", 'isn', "'", 't', 'o', "'", 'brian', 'the', 'best', "?'"]

I'm really lost because I've tried to do more than re.sub to separate the words and tried to split them, but it seemingly is not working.

tokenize python-3.8 python
2 Answers

You need a better definition of "word", for example:

word_re = r"\b[A-Za-z]+(?:'[A-Za-z]+)?\b"

that is, a word boundary, then some letters, then optionally an apostrophe, followed by letters. Once you've got it, use findall to extract words:

words = [w.lower() for w in re.findall(word_re, text)]
gog0

if the apostrophes arent important to the meaning of the word, ie "obrien" and "o'brien" are the same for your purposes you could do some basic text pre-processing to remove all the apostrophes using text.replace("'","") see here

if they do matter, you could replace them with Unicode (the unicode for ' is U+0027) and translate them out of Unicode after you've tokenized them, or just text.replace("'","U+0027") before you tokenize, and text.replace("U+0027","'") after

hope that helps, goodluck!

JoshAsh0
Related