I'm wondering why my Pandas implementation of TF-IDF shows slightly different results than the sklearn implementation.
Here's my implementation:
text = ["aa bb cc dd ee", "bb cc dd dd"]
terms = [Counter(t.split(' ')) for t in text]
tf = pd.DataFrame(terms)
tf = tf.fillna(0)
num_docs = len(text)
idf = np.log(num_docs / tf[tf >= 1].count()) + 1
tf_idf = tf * idf
norm = np.sqrt((tf_idf ** 2).sum(axis=1))
norm_tf_idf = tf_idf.div(norm, axis=0)
>>> norm_tf_idf
aa bb cc dd ee
0 0.572929 0.338381 0.338381 0.338381 0.572929
1 0.000000 0.408248 0.408248 0.816497 0.000000
However if I use sklearn:
tf = TfidfVectorizer(smooth_idf=False, stop_words=None, sublinear_tf=True)
x = tf.fit_transform(text)
sk = pd.DataFrame(x.toarray())
sk.columns = tf.get_feature_names()
sk
>>> sk
aa bb cc dd ee
0 0.572929 0.338381 0.338381 0.338381 0.572929
1 0.000000 0.453295 0.453295 0.767495 0.000000
Or if we subtract them:
>>> norm_tf_idf - sk
aa bb cc dd ee
0 0.0 0.000000 0.000000 0.000000 0.0
1 0.0 -0.045046 -0.045046 0.049002 0.0
Edit:
I found out the sklearn idf is not exactly the same as my idf, but we can attribute this to floating point precision I think:
sklearn idf: [1.69314718 1. 1. 1. 1.69314718]
my idf: [1.693147 1.000000 1.000000 1.000000 1.693147]
And even if I use the sklearn idf, I still get different results.
Furthermore, if I don't normalize and use the sklearn idf values only the TF-IDF for dd of the second document differs:
sk_tfv = TfidfVectorizer(smooth_idf=False, stop_words=None, token_pattern=r"(?u)\b\w+\b", sublinear_tf=True, norm=None)
x = sk_tf.fit_transform(text)
sk_tf_idf = pd.DataFrame(x.toarray())
...
idf = sk_tfv.idf_
tf_idf = tf * idf
>>> tf_idf - sk_tf_idf
aa bb cc dd ee
0 0.0 0.0 0.0 0.000000 0.0
1 0.0 0.0 0.0 0.306853 0.0
This means either two things:
1. The problem is my TF. However this is easily checkable and doesn't seems to be the case. Or,
2. sklearn doesn't just do TF * IDF, but does something more, which I have to look into.