Supervised Clustering - Metric to Evaluate Each Ground Truth Label?

user_11085743 0
Viewed 885

I am performing K-means clustering on a dataset but I have ground truth labels available. I have used them during clustering to find the V-Measure and Adjusted Rand scores to get the best K.

To assess my best model, I would like a metric for each known label that describes how well it was clustered - almost like the Purity score but for a label spread across multiple clusters.

For example, label 0 has 5 data points, therefore we have the following:
true_labels = [0,0,0,0,0]
cluster_numbers = [1,1,1,1,1] (i.e. all label 0 points are in the same cluster)
--> should return a perfect score of 1.0

And if the points of the labels are spread across multiple clusters like this
cluster_numbers = [0,0,0,1,1]
--> return score of 0.6

Does anyone know of a metric that can be used to evaluate each ground truth label in clustering? This does not have to act the same way as the examples I gave above.

cluster-analysis machine-learning statistics algorithm
2 Answers

I think you can try evaluating a confusion matrix and checking the majority vote. For example we use the iris dataset:

from sklearn.datasets import load_iris
from sklearn.cluster import SpectralClustering
from sklearn.metrics import confusion_matrix
import pandas as pd
    
data = load_iris()
df = pd.DataFrame(data.data, 
columns=['sepal.length','sepal.width','petal.length','petal.width'])
labels = data.target

And we perform spectral clustering:

clustering = SpectralClustering(n_clusters=3,
assign_labels="discretize",random_state=0).fit(df)
    
M = confusion_matrix(labels,clustering.labels_)
    
M
array([[ 0, 50,  0],
      [ 2,  0, 48],
      [37,  0, 13]])

For every row, you have the prediction for each label. The purity as you have above is:

np.max(M,axis=1)/np.sum(M,axis=1)
array([1.  , 0.96, 0.74])

If you would like an overall agreement, you can use:

from sklearn.metrics import adjusted_rand_score
adjusted_rand_score(labels,clustering.labels_)
0.7436826319432357
user_122584590

Why not use the adjusted rand score for this purpose also? You can calculate it for each cluster and average across clusters if you want. It ignores permutations which is an advantage for your application

user_18818780
Related