How to modify spacy tokenizer to split URLs into individual words

user_839957 0
Viewed 237

I want to modify the default tokenizer to split URL's into individual words. Here's what I currently have


import spacy
nlp = spacy.blank('en')
infixes = nlp.Defaults.infixes + [r'\.']
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer
print(list(nlp('www.internet.com'))) 
# ['www.internet.com']
# want it to be ['www', '.', 'internet', '.', 'com']

I'm looking at the usage examples and source code for the tokenizer, but I can't work out this particular case.

spacy nlp python
1 Answers

You're not seeing results you want because url get caught by URL_MATCH first (it has higher precedence):

import spacy
nlp = spacy.blank('en')
txt = 'Check this out www.internet.com'
doc = nlp(txt)
nlp.tokenizer.explain(txt)

[('TOKEN', 'Check'),
 ('TOKEN', 'this'),
 ('TOKEN', 'out'),
 ('URL_MATCH', 'www.internet.com')]

One of the possible solutions:

nlp.tokenizer.url_match = None
infixes = nlp.Defaults.infixes + [r'\.']
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer
doc = nlp(txt)
list(doc)

[Check, this, out, www, ., internet, ., com]
user_43170580
Related