Scikit-learn tfidf vectorizer in minibatches?

user_3523464 0
Viewed 351

I've been trying to perform tf-idf heuristic on a large corpus.

Can I iteratively read the documents, and call the 

vectorizer.fit()

In each iteration? Does this take into account only the current iteration, or does it remember the previous ones?

Thanks!

scikit-learn tf-idf
1 Answers

The solution to your problem will depend on your particular application. You could consider gensim's tfidf implementation which is more efficient and does not need to keep the entire corpus in memory as this post explains.

user_21718570
Related