How can I modify the Spacy English tokenizer so that it will split on, and split apart, specific pairs of punctuation:
import spacy
nlp = spacy.load('en_core_web_md')
doc = nlp("running.(together")
# desired outcome
assert( [t.text for t in doc] == ["running", ".", "(", "together"])
what is currently get, is just one token, "running.(together".
By modify, I mean: do all of the current English tokenization, but also split this run-on case too.
Personally, I would solve this problem by creating a custom tokenizer. Here's the code for that according to the test case that you shared.
import re
import spacy
from spacy.tokenizer import Tokenizer
infix_re = re.compile(r'''[\(]|[.]''') # it would split either on ( or .
def custom_tokenizer(nlp):
return Tokenizer(nlp.vocab, infix_finditer=infix_re.finditer)
nlp = spacy.load("en_core_web_md")
nlp.tokenizer = custom_tokenizer(nlp)
doc = nlp("running.(together")
assert( [t.text for t in doc] == ["running", ".", "(", "together"]) # passes
UPDATE Yes, It will retain the capabilities of the default tokenizer plus your run-on case.
If you want to learn more about this, please check out the official docs