Using spacy to lemmatize a column of parsed html text in a Pandas Dataframe

user_6322082 0
Viewed 1655

I want to do something quite trivial but struggeled to write the function to do it. For a NLP Multiclass Classification task I have to preprocess a pandas DataFrame. The column of interest is parsed html text (column: "tweet"). I normalize my data (lowercase, remove punctuation, stopwords, ...) and then I want to lemmatize it using spacy and write it back as a column. However, I can't get the function together. I found a couple of examples on SO, but they all use lists and I cannot translate that to a DF. Because I have a DataFrame which is quite large (10GB) I wanted to use a function which is not tooooo slow. Any help or suggestions would be appreciated. Thank you :)

# My real text is in german, but since Englisch is more frequent I use "en_core_web_sm" here
import spacy
en_core = spacy.load('en_core_web_sm')

# Create DataFrame
pos_tweets = [('I love this car', 'positive'), ('This view is amazing', 'positive'), ('I feel great this morning', 'positive'), ('I am so excited about the concert', 'positive'), ('He is my best friend', 'positive')]
df = pd.DataFrame(pos_tweets)
df.columns = ["tweet","class"]

# Normalization
df['tweet'] = [entry.lower() for entry in df['tweet']]
# Tokenization
df["tokenized"] = [w.split() for w in df["tweet"]]

# Lemmatization
# This is where I struggle. I can't get together the English Model en_core, lemma_ and stuff :(
df["lemmatized"] = df['tokenized'].apply(lambda x: [en_core(y.lemma_) for y in x])
pandas spacy apply lemmatization python
1 Answers

You need to run it on the text, not tokens.

df["lemmatized"] = df['tweet'].apply(lambda x: " ".join([y.lemma_ for y in en_core(x)]))

Here, x will be a sentence/text in the tweet column, en_core(x) will create a document out of it, and y will represent each token, with y.lemma_ yielding the word lemma. " ".join(...) will concat all the lemms found into a single space-separated string.

user_38329700
Related