calculate IDF from scratch

user_8500043 0
Viewed 72

I am trying to calculate the IDF values from scratch so I can understand how it works behind the scene but still couldn't get the expected results.

I have a set of documents like this

docs = {'d1' :'t3 t3 t3 t6 t6', 
        'd2' :'t1 t2 t3 t3 t6', 
        'd3' :'t3 t3 t4 t5', 
        'd4' :'t4 t5 t6 t6', 
        'd5' :'t1 t2 t3 t5'}

and I am supposed to calculate a value called query_sum_idf that represents the sum value of IDF for the tokens sent in request which is in this case ['t5', 't2']

The expected value in this case is 1.4271 ± 1.0e-04, but when developing the calculus, here is what I did and I got different results : math.log10(5 / float(3)) + math.log10(5 / float(2)) = 0.6197, where 5 is the total number of documents and 3 is the count of t5in all the documents and last 2is the count of t2 in all the documents.

As you can see the results are different, so I'm asking if I'm missing something around.

tf-idf nlp python-3.x python
1 Answers

Interesting problem you have there.

I have tried to do this task on my own, and looked up common definitions of IDF on wikipedia. I calculate IDF like this:

import math
def idf(docs, term):
    count = 0
    for key, doc in docs.items():
        if term in doc: count += 1
    return math.log(len(docs)/count)

The issue is, you are using log10 instead of natural logarithm. Even on wiki, it is not defined, which log to use. I hope I helped.

user_12531210
Related