Clustering evaluation with a manually labeled ground-truth

Question

I am working with document clusterization with different methods (Kmeans, HAC, and LDA).

Now I have made some manual clusterization of my documents to check if, and how many, of the documents that I have stated that have to go together, actually go together.

The output from clustering (for all the methods) is standard and is a CSV with the format:

Clustering Output

id    label
 0        1
 1        4
 2        0
     ...
 N        1

Where 'id' is the ID of a document in my repository and label refers to the cluster assigned. Notice that the number of clusters could not be always the same across the different methods.

The other document that I have prepared is loaded into a pandas dataframe in the form:

GroundTruth

id    label1    label2    label3    [...]    labelM
 0         0         1         0                  1
 1         1         1         0                  0
 2         0         0         1                  0

Is a sort of 'one-hot-encoding', be careful that during the manual label assignations from the user, are allowed multiple labels (since documents can belong to different topics)

Now I am stuck with the logic behind the comparison. As the first try I have done this:

Group documents in Clustering Output by cluster label
Loop over documents in each group
Loop over GroundTruth dataframe columns and extract rows where value == 1 (label assigned)

Computed how many items from (3) belongs to (2) in percentage.

for label, df_group in clusters_groups:
docs_in_cluster = df_group['id'].values.tolist()
# here we have a dataframe for each cluster
for (columnName, columnData) in ground_truth.iteritems():
    overlapping = 0
    count = 0
    if columnName != 'id':
        togethers = ground_truth.loc[ground_truth[columnName] == 1, 'id']
        for doc in togethers.values.tolist():
            if int(doc) in docs_in_cluster:
                count += 1
        overlapping = count/len(togethers.values.tolist()) * 100
print("For label:", label, " the overlapping is: {:.2f}%".format(overlapping))

The code works, but I am not really convinced about this method.

There is a better way to evaluate the overlapping of the sets?

user_13730780 · Answer

You may want to compute enrichment of your detected clusterlabel with the groundtruth instead of percentage overlap.

For this, you can use the hnet library which computes a P-value based on the Hypergeometric test for each of the GroundTruth with the input label.

Documentation can be found here: https://erdogant.github.io/hnet. The following section may be of interested for your case: https://erdogant.github.io/hnet/pages/html/Use%20Cases.html#cluster-enrichment

pip install hnet

# Import library
import hnet

# Run cluster enrichment using HNet
results = hnet.enrichment(GroundTruth, label)

# Examine the results
print(results)

The result is in the form of a dataframe. The first column is the category_label, the GroundTruth labels in your case. The second column: P stands for P-value, which is the computed significance of the category_label with the target variable label. The category_labels with P is smaller then alpha, lets say 0.05 are significantly enriched with your cluster-label.

Clustering evaluation with a manually labeled ground-truth

1 Answers