Tokenization with spaCy

user_19755006 0
Viewed 100

I'm a rookie in Python and I hope to find here help for my text analyse project. I'm trying to tokenize several text elements which I imported from an excel in python spyder. Each text element is in a seperate row.

I tried this code but it doesen't work.

import spacy

df = txtdf.head()
df2 = df['Textelements']

from spacy.lang.de import German

nlp = German()
doc = nlp(df2)
tokens = [token.text for token in doc]
print(tokens)

And I get this error in the console:

ValueError: [E1041] Expected a string, Doc, or bytes as input, but got: <class 'pandas.core.series.Series'>

spacy text-mining dataframe nlp string
1 Answers

nlp(df2) expects a single string/document to process, but it receives a pandas Series of strings. Try to batch process the column:

docs = nlp.pipe(df2)
for doc in docs:
    tokens = [token.text for token in doc]
    print(tokens)

Or (not recommended because its slower) iterate over the column and call the processing for each row:

for row in df2:
    doc = nlp(row)
    tokens = [token.text for token in doc]
    print(tokens)

If you really do want to process the entire column as one big blob of text you have to implement a way to join it all into one text beforehand.

user_181896220
Related