Spacy : apply pipeline to each row of a dataframe

user_15500816 0
Viewed 121

I have a big dataframe (over 20 000 rows) and I want to apply Spacy (v3) to it. I need all the components of the spacy pipeline. I already tried with apply (row-wise) but it takes forever:

df = pd.read_csv(f, sep='\t', encoding='utf-8')
df['Text_spacy'] = df['Text_initial'].apply(lambda x: nlp(x))

display(df)

The column Text initial contains something like :

Text_initial
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Ut enim ad minim veniam.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Sunt in culpa qui officia deserunt mollit anim id est laborum.
Pharetra magna ac placerat vestibulum lectus.
Nec feugiat nisl pretium fusce id velit ut.
Amet justo donec enim diam vulputate ut pharetra.
Nibh venenatis cras sed felis eget velit aliquet sagittis id.

Notice that in every row, the phrases are already separated by a \n. The segmentation will be made based on the \n. Thus, my question is:
Is there a faster way to apply spacy to each row of the df in a loop ?
Iterrows() is even slower.
Would a batch (e.g. the first 100 rows, then the next 100 ... till the end) be faster ?

spacy-3 spacy dataframe nlp
1 Answers

nlp.pipe() expects an iterable of strings, so try this:

df = pd.read_csv(f, sep='\t', encoding='utf-8')
df['Text_spacy'] = [d for d in nlp.pipe(df['Text_initial'])]

Although the question is what do you want to get from SpaCy (Tokenization, Lemmatization, POS etc.). Because applying it like this will just put the Doc objects into your DataFrame (which are then represented as the tokens string tuples for printing).

user_181896220
Related