How do I get word/term frequencies in a corpus using Scikit Learn?

user_4570472 0
Viewed 394

I have a corpus of documents and I'd like to extract the word frequencies in each document. I could use CountVectorizer() to get term counts per document, and I could use TfidfVectorizer() to get term frequency-inverse document frequency, but neither seems to give me term frequencies alone. How do I get term frequencies?

This related question seems to ask my question, but the question and answers there concern term counts, not term frequencies. Maybe I'm the one misunderstanding these terms, but my understanding is that term counts are the integer number of times each term appears in the document whereas term frequencies are the term counts divided by the document length.

scikit-learn python
1 Answers

There is the TfidfTransformer for this purpose. From the docs:

Transform a count matrix to a normalized tf or tf-idf representation

Since it only transforms a count matrix, you would need to use it in conjunction with an already vectorized matrix or use CountVectorizer before:

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer


X_count = CountVectorizer().fit_transform(X_train)  # use first if X_train is not vectorized
X_tf = TfidfTransformer(use_idf=False).fit_transform(X_count)

Note that by setting use_idf=False you will get the term-frequency only.

user_131828850
Related