Stemmer function that takes a string and returns the stems of each word in a list

usernL1e1 0
Viewed 15

I am trying to create this function which takes a string as input and returns a list containing the stem of each word in the string. The problem is, that using a nested for loop, the words in the string are appended multiple times in the list. Is there a way to avoid this?

def stemmer(text):
    
    stemmed_string = []
    res = text.split()
    suffixes = ('ed', 'ly', 'ing')
    
    for word in res:
            for i in range(len(suffixes)):
                if word.endswith(suffixes[i]):
                    stemmed_string.append(word[:-len(suffixes[i])])
                elif len(word) > 8:
                    stemmed_string.append(word[:8])
                else:
                    stemmed_string.append(word)
    
    return stemmed_string

If I call the function on this text ('I have a dog is barking') this is the output:

['I',
 'I',
 'I',
 'have',
 'have',
 'have',
 'a',
 'a',
 'a',
 'dog',
 'dog',
 'dog',
 'that',
 'that',
 'that',
 'is',
 'is',
 'is',
 'barking',
 'barking',
 'bark']
nested-for-loop python
1 Answers

You are appending something in each round of the loop over suffixes. To avoid the problem, don't do that.

It's not clear if you want to add the shortest possible string out of a set of candidates, or how to handle stacked suffixes. Here's a version which always strips as much as possible.

def stemmer(text):
    stemmed_string = []
    suffixes = ('ed', 'ly', 'ing')
    
    for word in text.split():
        for suffix in suffixes:
            if word.endswith(suffix):
                word = word[:-len(suffix)]
        stemmed_string.append(word)
    
    return stemmed_string

Notice the fixed syntax for looping over a list, too.

This will reduce "sparingly" to "spar", etc. Like every naïve stemmer, this will also do stupid things with words like "sly" and "thing".

Demo: https://ideone.com/a7FqBp

tripleee0
Related