Why is vectorizer.fit_transform(x).astype('bool') different from vectorizer.set_params(binary=True).fit_transform(x)?

user_4400686 0
Viewed 921

Here is a minimal example of what I'm talking about: 

import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer

data = fetch_20newsgroups()
x = data.data

vec = TfidfVectorizer(min_df=0.01, max_df=0.5)
mat = vec.fit_transform(x).astype('bool')

vec.set_params(binary=True)
print(np.array_equal(mat, vec.fit_transform(x)))

This prints False. What is the underlying difference between setting binary=True and setting all nonzero values to True?

EDIT: As answered by @juanpa.arrivillaga, TfidfVectorizer(binary=True) still does the inverse document frequency calculation. However, I also noticed CountVectorizer(binary=True) doesn't produce the same output as .astype('bool') either. Below is an example:

In [1]: import numpy as np
   ...: from sklearn.datasets import fetch_20newsgroups
   ...: from sklearn.feature_extraction.text import CountVectorizer
   ...:
   ...: data = fetch_20newsgroups()
   ...: x = data.data
   ...:
   ...: vec = CountVectorizer(min_df=0.01, max_df=0.5)
   ...: a = vec.fit_transform(x).astype('bool')
   ...:
   ...: vec.set_params(binary=True)
   ...: b = vec.fit_transform(x).astype('bool')
   ...: print(np.array_equal(a, b))
   ...:
False

In [2]: a
Out[2]:
<11314x2141 sparse matrix of type '<class 'numpy.bool_'>'
        with 950068 stored elements in Compressed Sparse Row format>

In [3]: b
Out[3]:
<11314x2141 sparse matrix of type '<class 'numpy.bool_'>'
        with 950068 stored elements in Compressed Sparse Row format>

The dimensions and dtype are the same, which leads me to believe the content of those matrices are different. Just by eyeballing the output of print(a) and print(b), they look the same.

scikit-learn python
1 Answers

You are fundamentally confusing two things.

One is conversion to the boolean numpy datatype, which is the equivalent to the python data type which accepts two values, True and False, except it is represented as a single byte in the underlying primitive array.

Passing the binary argument to the TfidfVectorizer changes the way the data is modeled. In short, if you use binary=True, the total counts will be binary, i.e. either seen or not seen. Then you do the usual tf-id transformation. From the docs:

If True, all non-zero term counts are set to 1. This does not mean outputs will have only 0/1 values, only that the tf term in tf-idf is binary. (Set idf and normalization to False to get 0/1 outputs.)

So you don't even get a boolean output.

So consider:

In [10]: import numpy as np
    ...: from sklearn.feature_extraction.text import TfidfVectorizer
    ...:

In [11]: data = [
    ...:     'The quick brown fox jumped over the lazy dog',
    ...:     'how much wood could a woodchuck chuck if a woodchuck could chuck wood'
    ...: ]

In [12]: TfidfVectorizer().fit_transform(data).todense()
Out[12]:
matrix([[ 0.30151134,  0.        ,  0.        ,  0.30151134,  0.30151134,
          0.        ,  0.        ,  0.30151134,  0.30151134,  0.        ,
          0.30151134,  0.30151134,  0.60302269,  0.        ,  0.        ],
        [ 0.        ,  0.45883147,  0.45883147,  0.        ,  0.        ,
          0.22941573,  0.22941573,  0.        ,  0.        ,  0.22941573,
          0.        ,  0.        ,  0.        ,  0.45883147,  0.45883147]])

In [13]: TfidfVectorizer().fit_transform(data).todense().astype('bool')
Out[13]:
matrix([[ True, False, False,  True,  True, False, False,  True,  True,
         False,  True,  True,  True, False, False],
        [False,  True,  True, False, False,  True,  True, False, False,
          True, False, False, False,  True,  True]], dtype=bool)

And now notice that usin binary will still return a floating-point type:

In [14]: TfidfVectorizer(binary=True).fit_transform(data).todense()
Out[14]:
matrix([[ 0.35355339,  0.        ,  0.        ,  0.35355339,  0.35355339,
          0.        ,  0.        ,  0.35355339,  0.35355339,  0.        ,
          0.35355339,  0.35355339,  0.35355339,  0.        ,  0.        ],
        [ 0.        ,  0.37796447,  0.37796447,  0.        ,  0.        ,
          0.37796447,  0.37796447,  0.        ,  0.        ,  0.37796447,
          0.        ,  0.        ,  0.        ,  0.37796447,  0.37796447]])

It just changes the results.

user_50144550
Related