How to prioritize certain features with max_features parameter in countvectorizer

user_16895885 0
Viewed 147

I have a working program but I realized that some important n-grams in the test data were not a part of the 6500 max_features I had allowed in the training data. Is it possible to add a feature like "liar" or "pathetic" as a feature that I would train with my training data?

This is what I currently have for making the vectorizer:

vectorizer = CountVectorizer(ngram_range=(1, 2)
                            ,max_features=6500)
X = vectorizer.fit_transform(train['text'])
feature_names = vectorizer.get_feature_names()
countvectorizer scikit-learn python
1 Answers

This is hacky, and you probably cannot count on it working in the future, but CountVectorizer primarily relies on the learned attribute vocabulary_, which is a dictionary with tokens as keys and "feature index" as values. You can add to that dictionary and everything appears to work as intended; borrowing from the example in the docs:

from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?',
]
vectorizer2 = CountVectorizer(analyzer='word', ngram_range=(2, 2))
X2 = vectorizer2.fit_transform(corpus)
print(X2.toarray())

## Output:
# [[0 0 1 1 0 0 1 0 0 0 0 1 0]
#  [0 1 0 1 0 1 0 1 0 0 1 0 0]
#  [1 0 0 1 0 0 0 0 1 1 0 1 0]
#  [0 0 1 0 1 0 1 0 0 0 0 0 1]]

# Now we tweak:
vocab_len = len(vectorizer2.vocabulary_)
vectorizer2.vocabulary_['new token'] = vocab_len  # append to end
print(vectorizer2.transform(["And this document has a new token"]).toarray())

## Output
# [[1 0 0 0 0 0 0 0 0 0 1 0 0 1]]
user_104958930
Related