Say, I have 200 documents 200 pages each. A lot of words, I would say. And I want to cluster them into different groups. I have created a fairly simple text clustering algorithm. It currently works like this:
Iterate through each document, create a giant string and append it to array. So, at the end there will be 200 giant strings.
Morph each string, that means remove stop words, lemmatize every word, remove noise and other stuff.
I'm currently using this sample to get the result.
def fit_transform(self, morphed_documents): self.matrix = self.vectorizer.fit_transform(morphed_documents) return self.matrix def fit(self, number_of_clusters, matrix): model = KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=10, n_clusters=number_of_clusters, n_init=30, random_state=3454, tol=0.0001, verbose=0) model.fit(matrix) return model
It kinda works, it takes some human analysis (I believe K-means isn't perfect) but the main issue is that my current approach is really slow. Basically, it takes more than 1 hour to go through every book.
My question is, are there faster and simpler algorithms that can be used? Can I speed up my current algorithm or I'm doing complete nonsense from the beginning? Basically, any help will be highly appreciated!