How to lemmatize strings in pandas dataframes?

Viewed 4011

I have a Python Pandas dataframe, where I need to lemmatize the words in two of the columns. I am using using spacy for this.

import spacy
nlp = spacy.load("en")

I am trying to use lemmatization based on this example (which works perfectly fine):

doc3 = nlp(u"this is spacy lemmatize testing. programming books are more better than others")
for token in doc3: 
    print (token, token.lemma, token.lemma_)

I have rewritten this to loop through each row of one of the columns in my dataframe:

for row in example['col1']:
    for token in row:
        print(token.lemma_)

This works, however, I have not been able to figure out how to replace the words in col1 with the lemmatized words.

I have tried this, which does not return an error, but also does not replace any words. Any idea what is going wrong?

for row in example['col1']:
    for token in row:
        token = token.lemma_
1 Answers
Related