Is this correct tfidf?

Viewed 1195

I am trying to get tfidf from a document. But I dont think it is giving me correct values or I may be doing some thing wrong. Please suggest. Code and output below:

from sklearn.feature_extraction.text import TfidfVectorizer
books = ["Hello there this is first book to be read by wordcount script.", "This is second book to be read by wordcount script. It has some additionl information.", "just third book."]
vectorizer = TfidfVectorizer()
response = vectorizer.fit_transform(books)
feature_names = vectorizer.get_feature_names()
for col in response.nonzero()[1]:
   print feature_names[col], '-', response[0, col]

Update 1: (As suggested by juanpa.arrivillaga)

vectorizer = TfidfVectorizer(smooth_idf=False)

Output:

script - 0.269290317245
wordcount - 0.269290317245
by - 0.269290317245
read - 0.269290317245
be - 0.269290317245
to - 0.269290317245
book - 0.209127954024
first - 0.354084405732
is - 0.269290317245
this - 0.269290317245
there - 0.354084405732
hello - 0.354084405732
information - 0.0
...

Output after update 1:

script - 0.256536760895
wordcount - 0.256536760895
by - 0.256536760895
read - 0.256536760895
be - 0.256536760895
to - 0.256536760895
book - 0.182528018244
first - 0.383055542114
is - 0.256536760895
this - 0.256536760895
there - 0.383055542114
hello - 0.383055542114
information - 0.0
...

As per my understanding tfidf is = tf * idf. And the way I manually calculate it as example:

document 1: "Hello there this is first book to be read by wordcount script." document 2: "This is second book to be read by wordcount script. It has some additionl information." document 3: "just third book."

Tfidf for Hello:

tf= 1/12(total terms in document 1)= 0.08333333333
idf= log(3(total documents)/1(no. of document with term in it))= 0.47712125472
0.08333333333*0.47712125472= 0.03976008865 

which is different from below (hello - 0.354084405732).

Manual calculation after update 1:

tf = 1
idf= log(nd/df) +1 = log (3/1) +1= 0.47712125472 + 1= 1.47712 
tfidf = tf*idf = 1* 1.47712= 1.47712

(not same as code output "hello - 0.383055542114" after idf smoothing)

Any help to understand whats going on is highly appreciated..

1 Answers
Related