I'm trying to split words like 'olive-oil','high-fat','all-purpose' which are tokenized into one chunk.
The desired tokenization has to be
['olive','-','oil','high','-','fat','all','-','purpose']
I looked into retokenizer and the usage was like below.
doc = nlp("I live in NewYork")
with doc.retokenize() as retokenizer:
heads = [(doc[3], 1), doc[2]]
attrs = {"POS": ["PROPN", "PROPN"],
"DEP": ["pobj", "compound"]}
retokenizer.split(doc[3], ["New", "York"], heads=heads, attrs=attrs)
As you could see in the last line, to retokenize a chunk into pieces, I have provide what the result would be. I don't think this is an efficient way of processing words because if I have to provide all the result, it means I just manually type all the possibilities, which I don't think feasible plan.
Given that I know all the cases and provide the ending result one by one it might be way more efficient that just find the words to be replaced and replace them into what I want manually.
I believe there must be a way to generalize them.
If anyone knows to the way to tokenize the words I put at the top, can you help me?
Thank you