Below is agglomerative clustering from scikit-learn. The problem is, to use the .fit method I need to convert the sparse matrix representation to an array. Since I have 80k documents with around 400k terms, the created array will have 80k x 400k elements. This will be a huge array and naturally, my computer has no memory and raised an Exception.
from sklearn.cluster import AgglomerativeClustering
model = AgglomerativeClustering(n_clusters = n_clusters,
linkage = linkage,
affinity = affinity,
distance_threshold = distance_threshold)
model = model.fit(vectorspace.toarray())
With KMeans, I don't need to transform the sparse matrix representation to an array. Notice that I don't need to call the .toarray() inside the model.fit(). Therefore, my computer does not raise an Exception, since it does not use that much memory.
model=KMeans(n_clusters=n_clusters)
model = model.fit(vectorspace)
Does anyone know of any other python module or a workaround for this? I want to make Agglomerative Clustering exclusively with sparse matrixes, so I can do this with a large document collection, such as mine. Notice that my computer raises an Error and I can't even compute the clusters.