Result too large when using scikit linear_kernel for TF-IDF computation

user_13017532 0
Viewed 1892

I am new in python and currently trying to develop a content based recommender system. I am reading a csv file with about 60,000 rows and using TfidfVectorizer.fit_transform which makes a matrix size (63098, 9081) and then apply linear_kernel that then crashes and returns the error RuntimeError: nnz of the result is too large.

tf = TfidfVectorizer(analyzer='word', ngram_range=(1, 3), min_df=100, stop_words='english')
tfidf_matrix = tf.fit_transform(ds['description'])
result = linear_kernel(tfidf_matrix, tfidf_matrix)

Logs shows the supposed result would be 20gb in size. Is there a way to chunk the data and how to apply that in my current structure? Thanks in advance.

recommendation-engine data-science scikit-learn tf-idf python
1 Answers

This is because you are using all unique words in your data to create the TfidfVectorizer The shape of your matrix will be No. of documents * No. of unique words, your system's current memory might not support this size, to avoid the memory error you can limit the matrix size using the max_features parameter as mentioned in the docs

A size of 5000 or 10000 is normally used, but you can try different values and see what suits you.

user_113678370
Related