Keywords inside a spaceless string (using NLP?)

Viewed 25

I am trying to find the relevant set of keywords inside a spaceless string. An example would be:

freelancemarketingconsultant

By reading it, you can distinguish the following keywords:

freelance marketing consultant

You can see the task is not trivial, as a common confusion would be to distinguish between "free" and "lance".

Are there known (potentially NLP) techniques to extract keywords from such strings?

1 Answers

You can use the Viterbi algorithm to find the most likely (best) way to split the string. There's a library called wordsegment that does this in Python, and you can read more about the technique at Peter Norvig's page.

There's also a recent research project called Hashtag Master which uses neural methods to tokenize hashtags.

This isn't a common problem in English, but it's standard in languages were spaces don't split words, like Japanese. There are a variety of methods, and research continues, but Viterbi based methods generally have the best balance of speed and accuracy.

Related