I know that we can find the suitable number of clusters for the KMean algorithm to be used to create the clustering model of machine learning by plotting the Elbow graph. My question is there any formula that I can use to calculate the n-clusters by using information from the dataset like number of features or other information?
1 Answers
The method is call the ELbow method
K-means clustering is an unsupervised learning algorithm which aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest centroid. The algorithm aims to minimize the squared Euclidean distances between the observation and the centroid of cluster to which it belongs.
If you fit your data within a range of clusters (or k) you could see that at a certain point the sum of Euclidean distances of the points to the center is not decreasing (or not decreasing enough) by adding more clusters to the model. That point would be optimal number of clusters.
you should do something like this, considering your data is is named df_grouped
from sklearn.cluster import KMeans
Sum_of_squared_distances = []
for i in range(1,19):
kmeans = KMeans(n_clusters=i, init='k-means++', random_state=0)
kmeans.fit(df_grouped.dropna())
Sum_of_squared_distances.append(kmeans.inertia_)
plt.plot(range(1,19), Sum_of_squared_distances, 'bx-')
plt.xlabel('k')
plt.ylabel('Sum_of_squared_distances')
plt.title('Elbow Method For Optimal k')
plt.show()
When we plot the graph of ‘value of k’ on x-axis and ‘value of Epsilon’ on y-axis, there is an elbow formation at the optimum value of ‘k’. In the image above would be 3 (or maybe 4) where the "elbow" is formed, since adding more cluster does not significantly reduce the sum of squared Euclidean. The decision between 3 and 4 can be done related to your data and what ever makes more sense to the project you might be working on.
Related