What is the difference between Tfidftransformer and Tfidfvectorizer?

user_12541639 0
Viewed 1243

I am bit confused about the use of Tfidftransformer & Tfidfvectorizer as they both look similar. One uses words to convert matrix (Tfidfvectorizer) and the other used already converted text (using CountVectorizer) to matrix.

Can any one explain the difference between these two?

tfidfvectorizer scikit-learn pandas tf-idf python
1 Answers

CountVectorizer + TfidfTransformer = TfidfVectorizer this is the simplicity pragmatic way to understand. TfidfVectorizer performs CountVectorizer and TfidfTransformer in one step.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline

# transformer a
a = Pipeline(steps =[
        ('count_verctorizer',  CountVectorizer()),  
        ('tfidf', TfidfTransformer()),
])

# transformer b
b = TfidfVectorizer()

a and b transformers will do the same transformation.

If the preprocess includes of only TFIDF before feeding features to the model, then b will be best choice. But there times where we would like to split the preprocesses. For examples we want to keep only best terms before doing the Inverses Document Frequency. In such times, we would opt for a. Because we can perform CountVectorizer then do additional preprocessing before IDF. E.g.


from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_selection import chi2
from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import Pipeline
from sklearn.linear_model import  LogisticRegressionCV

# do counter terms and allow max 150k terms with 1-2 Ngrams
# select the best 10K (reducing the size of our features)
# do the IDF and the pass to our model

hisia = Pipeline(steps =[
        ('count_verctorizer',  CountVectorizer(ngram_range=(1, 2), 
                                 max_features=150000,

                                )
        ),
        ('feature_selector', SelectKBest(chi2, k=10000)),
        ('tfidf', TfidfTransformer(sublinear_tf=True)),
        ('logistic_regression', LogisticRegressionCV(cv=5,
                                                    solver='saga',
                                                    scoring='accuracy',
                                                    max_iter=200,
                                                    n_jobs=-1,
                                                    random_state=42, 
                                                    verbose=0))
])

In the example, we performed a feature selection of terms before passing them toIDF. This is possible because we can split the TFIDF by first doing CountVectorizer and the TfidfTransformer

user_68582440
Related