Remove stopwords using spaCy from list dataframe

user_11812455 0
Viewed 363

I want to remove stopwords using spaCy after tokenize. But, given me an error and the error is AttributeError: 'str' object has no attribute 'is_stop' The data I want to do is the data after the tokenizing process which is in column named 'tokenizing' How to fix it?

import pandas as pd
from spacy.lang.id import Indonesian

nlp = Indonesian()

data = [
    'Aku suka sekali beradai di wilayah yang dingin',
    'Kembali jika terjadi sesuatu di sana',
    'Sampai berapa lama kamu akan pergi dari sini',
]
df = pd.DataFrame({'text': data})
df['text'] = df['text'].str.lower()
df.head()

#Tokenizing
def tokenize(word):
  return [token.text for token in nlp(word)]

df['tokenizing'] = df['text'].apply(tokenize)
df.head()

#Remove stopwords
def stopwords_remover(words):
  return [stopwords for stopwords in words if not stopwords.is_stop]

df['stopwords'] = df['tokenizing'].apply(stopwords_remover)
df
spacy nlp python
1 Answers

You are processing a list of strings, and a string is not a SpaCy token, thus, it has no is_stop attribute.

You need to keep a list of SpaCy tokens in the tokenizing column, change def tokenize(word) to:

def tokenize(word):
    return [token for token in nlp(word)]

Output:

enter image description here

If you need to keep tokenizing column filled with token texts and make stopwords from scratch, use

def stopwords_remover(words):
    return [stopwords for stopwords in nlp(words) if not stopwords.is_stop]

df['stopwords'] = df['text'].apply(stopwords_remover)
user_38329700
Related