I´m trying to remove self-defined stopwords with Tfidf, but although using different approaches, the stopwords I defined are not removed multiple times - it seems as they are removed only once.
stop_words = ["er","sie","es","sehr", "geehrte","geehrter","herr","frau","ihre","ihrem","ihren","der","die", "das","viele", "gruesse","gruessen","mit", "von", "auf", "unter","ab", "fuer", "von", "gmbh", "und", "oder","email", "am", "ist","nicht", "wir", "hiermit", "unser", "unsere", "unseren","ohne", "bitten", "uns", "bis", "zur","am","bei", "des", "dessen", "deren", "dem", "nach","zu", "eines", "einen", "einer", "einem", "dies", "des", "den", "dank", "wurde", "wird", "war", "sein","in", "als", "gerne", "gerne", "wieder","welcher", "welche", "welchem","welchen","welches", "hat","hatte","freundlich", "freundliche", "freundlichen", "freundliches", "wenn", "wuerden", "durch"]
vectorizer = TfidfVectorizer(ngram_range=[1,1], stop_words=stop_words)
X_text_set = vectorizer.fit_transform(X_text_set)
These are the results without using Stopwords:
y_train_text size:(1872,)
y_val_text size:(401,)
y_test_text size:(402,)
X_train_text size:(1872, 35941)
X_val_text size:(401, 35941)
X_test_text size:(402, 35941)
These are the results after using stopwords:
y_train_text size:(1872,)
y_val_text size:(401,)
y_test_text size:(402,)
X_train_text size:(1872, 35867)
X_val_text size:(401, 35867)
X_test_text size:(402, 35867)
As you can see, every word is only removed once. Since these are common words, I would expect rather hundreds of occurrences to be removed.
Could anyone please help me?