K-means, knee locator, not quite correct

user_13957738 0
Viewed 1208

Using some codes I found here and in google, I have made a kmeans clustering algorithem based on Knee. locator:

def kmeans_clustering(data_set: pd.DataFrame, column: str, clusters_count: int):
    '''K means clustering on dataframe, choosing specific columns and cluster count'''
    data_set = data_set[column].to_numpy()
    data_values = data_set.reshape(-1, 1)
    kmeans = KMeans(n_clusters=clusters_count).fit(data_values)
    y = kmeans.fit_predict(data_values)  # prediction of k
    frame = pd.DataFrame(y, index=data_set.index)
    frame.index = frame.index[frame.index.isin(data_set.index)]
    return frame.join(data_set)

wcss =[]
    for i in range(1, 11):
        kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, n_init=10, random_state=0)
        kmeans.fit(RTN)
        wcss.append(kmeans.inertia_)
    kn = KneeLocator(
                        range(1, 11), wcss,  # put range in variable
                        curve='convex',
                        direction='decreasing', interp_method='interp1d')

............

for i in RTN:
    frame = kmeans_clustering(data_set=RTN, column= val, clusters_count= kn.knee)

............

My dataframe is "RTN". Columns represnted by "val" (has around thousand). After running the code I get the following warning:

ConvergenceWarning: Number of distinct clusters (3) found smaller than n_clusters (4). Possibly due to duplicate points in X.
  kmeans = KMeans(n_clusters=clusters_count).fit(data_values)
C:\Users\AppData\Local\Programs\Python\Python38-32\lib\site-packages\sklearn\cluster\_kmeans.py:1122: ConvergenceWarning: Number of distinct clusters (3) found smaller than n_clusters (4). Possibly due to duplicate points in X.
  return self.fit(X, sample_weight=sample_weight).labels_

I understand that I possibly have done a "double operation", but where did it go wrong?

k-means cluster-analysis machine-learning python
1 Answers

I think you haven't gone wrong. It just means, that KMeans did not find enough clusters. You know, KMeans chooses a number of data points in your set to build clusters. Then it assigns the rest of the points to these clusters and calculates the "center" of these clusters, then it assgins again all points to the nearest center and so on until there is no change anymore.

During this process it might happen, that the data points assigned to one cluster all get "captured" by other clusters nearby, so that the number of clusters gets smaller. This strongly depends on the initial choice of your data points with which clustering is started.

So you can try to set the random_state parameter to see, if you can get the desired number of clusters. If you use that parameter, you could also do a small number of runs with the same cluster size and take the best clustering you get for each cluster_size (the smallest inertia_). You might also want to play around with the init parameter.

Here an example:
Example data set

Immagine, you want to have the points clustered in four clusters. If you are unlucky the random selection used the four red points. With this setup in the intial step it is almost certain, that the 3 single points in the middle are added to one cluster while the three outer structures will get their own cluster. So after the first clustering step the inner cluster (with the three inner points) will be assigned to a point in the middle of these 3 points, but at the same time the middle point of the outer clusters will move towards the three points as well and it might end up that the middle points are then closer to the middle of the outer clusters as to their own cluster and so will be "captured" by the outer clusters, which results in the inner cluster to contain no more points and disappear.

user_69251850
Related