NLP reverse tokenizing (going from tokens to nicely formatted sentence)

user_5453184 0
Viewed 7787

Python's Spacy package has a statistical tokenizer that intelligently splits a sentence into tokens. My question is, is there a package that allows me to go backwards, i.e. from list of tokens to a nicely formatted sentence? Essentially, I want a function that lets me do the following:

>>> toks = ['hello', ',', 'i', 'ca', "n't", 'feel', 'my', 'feet', '!']
>>> some_function(toks)
"Hello, I can't feel my feet!"

It probably needs some sort of statistical/rules-based procedure to know how spacing, capitalization or contractions should work in a proper sentence.

spacy nlp python
3 Answers

You can use nltk to some extent for detokenization like this. You'll need to do some post processing or modify the regexes, but here is a sample idea:

import re
from nltk.tokenize.treebank import TreebankWordDetokenizer as Detok
detokenizer = Detok()
text = detokenizer.detokenize(tokens)
text = re.sub('\s*,\s*', ', ', text)
text = re.sub('\s*\.\s*', '. ', text)
text = re.sub('\s*\?\s*', '? ', text)

There are more edge cases with punctuations, but this is pretty simple and slightly better than ' '.join

user_25348760

I've described my approach here.

And it does create a nice sentence out of tokenized words even if you dont have spacy doc content.

user_67305260
Related