python word decomposition into subwords: e.g. motorbike -> motor, bike

domagaja 0
Viewed 57

I have a list of words like [bike, motorbike, copyright]. Now I want to check if th word consists of subwords which are also stand alone words. That means the ouput of my algorithm should be something like: [bike, motor, motorbike, copy, right, copyright].

I already now how to check if a word is a english word:

import enchant
english_words = []
arr = [bike, motorbike, copyright, apfel]
d_brit = enchant.Dict("en_GB")
for word in arr:
    if d_brit.check(word):
        english_words.append(word)

I also found an algorithm which decomposes the word in all possible ways: Splitting a word into all possible 'subwords' - All possible combinations

Unfortunately, splitting the word like this and then check if it is an english word takes simply to long, because my dataset is way to huge.

Can anyone help?

python
2 Answers

The nested for loops used in the code are extremely slow in Python. As performance seems to be the main issue, I would recommend to look for available Python packages to do parts of the job, or to build your own extension module, e.g. using Cython, or to not use Python at all.

BernieD0

Some alternatives to splitting the word like this:

  • searching for words that start with the first characters of str. If found word is start of str, check if rest is a word in dataset
  • split the str in two portions that make sense when looking at the length distribution of the dataset i.e. what are the most common word lengths? Then searching for matches with basic comparison. (just a wild idea)

These are a few quick ideas for faster algorithms i can think of. But if these are not quick enough, then BernieD is right.

Jon_Kle0
Related