What does the default sklearn TfidfVectorizer preprocessor do?

user_5363885 0
Viewed 1818

I am looking at sklearn's TfidfVectorizer, specifically at the preprocessor input parameter which has the following documentation:

"Override the preprocessing (string transformation) stage while preserving the tokenizing and n-grams generation steps."

I am trying to figure out exactly what (if anything) does the preprocessing stage do when I don't override it?

I have an experiment where I am looking at the number of stored elements in the resulting sparse matrix using the following code:

vectorizer = TfidfVectorizer(stop_words=words, preprocessor=process, ngram_range=(1,1), strip_accents='unicode')
vect = vectorizer.fit_transform(twenty_train.data)
items_stored = vect.nnz
  • When I do not override the preprocessor, the resulting matrix stores 1278323 elements.
  • When I override the preprocessor with an empty method, the resulting matrix stores 1441372 elements.
  • When I override the preprocessor with a method including s = re.sub("[^a-zA-Z]", " ", s), the resulting matrix stores 1331597 elements.
  • I have not been able to influence the sparse matrix's size (or accuracy when used in classification) with any other processing steps.

Clearly there are differences from the deafult sklearn result, no preprocessing and my attempt at replicating the preprocessing step. I am struggling to find documentation on what specifically the preprocessor does by default.

I have also checked the source code for the TfidfVectorizer - however I wasn't able to figure out what preprocessor was doing from here either.

Does anyone happen to know what code is executed or what preprocessing steps are taken by sklearn's default preprocessor?

scikit-learn machine-learning python
1 Answers

Are you looking for this?

def build_preprocessor(self):
    """Return a function to preprocess the text before tokenization"""
    if self.preprocessor is not None:
        return self.preprocessor

    # unfortunately python functools package does not have an efficient
    # `compose` function that would have allowed us to chain a dynamic
    # number of functions. However the cost of a lambda call is a few
    # hundreds of nanoseconds which is negligible when compared to the
    # cost of tokenizing a string of 1000 chars for instance.
    noop = lambda x: x

    # accent stripping
    if not self.strip_accents:
        strip_accents = noop
    elif callable(self.strip_accents):
        strip_accents = self.strip_accents
    elif self.strip_accents == 'ascii':
        strip_accents = strip_accents_ascii
    elif self.strip_accents == 'unicode':
        strip_accents = strip_accents_unicode
    else:
        raise ValueError('Invalid value for "strip_accents": %s' %
                         self.strip_accents)

    if self.lowercase:
        return lambda x: strip_accents(x.lower())
    else:
        return strip_accents

From here: https://github.com/scikit-learn/scikit-learn/blob/bac89c2/sklearn/feature_extraction/text.py#L230

user_83806380
Related