Custom tokenization rule spacy

user_14364181 0
Viewed 614

How do I add a custom tokenization rule to spacy for the case of wanting a number and a symbol or word to be tokenized together. E.g. the following sentence:

"I 100% like apples. I like 500g of apples"

is tokenized as follows:

['I', '100', '%', 'like', 'apples', '.', 'I', 'like', '500', 'g', 'of', 'apples']

It would be preferable if it was tokenized like this:

['I', '100%', 'like', 'apples', '.', 'I', 'like', '500g', 'of', 'apples']

The following code was used to generate this:

import spacy
nlp = spacy.load("en_core_web_sm")
text = "I 100% like apples. I like 500g of apples"
print([token.text for token in nlp(text)])
spacy-3 spacy nlp python
2 Answers

So normally you can modify the tokenizer by adding special rules or something, but in this particular case it's trickier than that. spaCy actually has a lot of code to make sure that suffixes like those in your example become separate tokens. So what you have to do is remove the relevant rules.

In this example code I just look for the set of rules that contain '%' and remove it; it just so happens that rule also contains unit suffixes like "g". So this does what you want:

import spacy

nlp = spacy.blank("en")
text = "I 100% like apples. I like 500g of apples"

# remove the entry with units and %
suffixes = [ss for ss in nlp.Defaults.suffixes if '%' not in ss]
suffix_regex = spacy.util.compile_suffix_regex(suffixes)
nlp.tokenizer.suffix_search = suffix_regex.search

print(list(nlp(text)))

You can see the list of rule definitions here.

user_3557150

I understand you mean to give a simple example but there are a couple of things here that are of concern.

  • Typically, stopwords and punctuation are removed first as, particularly with topic modeling, they take up quite a bit of processing power but add very little.
  • If you read through the documentation, you'll see Parts of Speech analysis is a fairly central feature. While you may not be intending to use that, you should understand that you're going against the grain here in that you're looking to conjoin things (eg. a QUANTMOD or Quantifier phrase modifier with the NUM or number it modifies) rather than tease out concepts from term (SpaCy example is 'Gimme' --> 'gim' (or give) and 'me')

But if you're really bent on going down this path, SpaCy documentation will get you there.

user_60398300
Related