How to save TF IDF vectorizer with custom tokenizer? python

Viewed 34

I am trying to save my tf idf vectorizer with pickle, but unfortunatelly i keep getting an error

Can't pickle local object 'preprocess.<locals>.my_tokenizer'

This is the vectorizer where my tokenizer is my custom tokenizer :

tfidf_vectorizer = TfidfVectorizer(tokenizer= my_tokenizer, 
  ngram_range=(1,3), min_df=0.005, max_df=0.2) 

This is the pickle

import pickle
with open('vectorizer.game', 'wb') as fin:
      pickle.dump(tfidf_vectorizer, fin) 

I do not know how to make it global

0 Answers
Related