Why is & tokenized as "&" in Python NLTK

user_10630288 0
Viewed 153

When trying to use the Toktok word tokenizer from NLTK in Python3

string='&& Test & and L&R '
from nltk.tokenize.toktok import ToktokTokenizer
ToktokTokenizer().tokenize(string)

I obtain the following output:

['&&', 'Test', '&', 'and', 'L&R']

Looks like it escapes the & in a strange way. I'm using NLTK version 3.3 and Python 3.6.4.

Any guess why this happens and an efficient way of solving it? I know I can go through the answer with

[tok.replace("&","&") for tok in tokenized_sentence]

but it seems a dirty hack. I would like to know if there is a way of not producing this effect in the first way.

tokenize nltk python
1 Answers

As mentioned by @snakecharmerb for the & the source states: 

# Replace problematic character with numeric character reference.

One approach to solve the issue is to override the fields on the ToktokTokenizer instance, for example:

import re

from nltk.tokenize.toktok import ToktokTokenizer

string = '&& Test & and L&R '

tokenizer = ToktokTokenizer()
tokenizer.AMPERCENT = re.compile('& '), '& '
tokenizer.TOKTOK_REGEXES = [(regex, sub) if sub != '& ' else (re.compile('& '), '& ') for (regex, sub) in
                            ToktokTokenizer.TOKTOK_REGEXES]

result = tokenizer.tokenize(string)
print(result)

Output

['&&', 'Test', '&', 'and', 'L&R']
user_40015920
Related