Visualize overlapping categories in one-hot encoding Pandas DataFrame

ignacioct 0
Viewed 28

I have a pandas DataFrame that looks like this:

enter image description here

One instance can have a value of 1 in more than one category. I was wondering how to visualize the categories that overlap the most with each other. I thought of maybe Venn's diagrams, but I don't know ow to generate them from my position.

Thanks in advance :)

data-science deep-learning pandas python
1 Answers

Maybe could you compute the overlapping between categories pair-wise, then plot the resulting matrix with a heat map for visualization purpose, with plt.matshow ?

With A the numpy array of the one hot labels, of shape (N, d), N the number of examples, d the number of categories, I would do:

common_occurences = A.T@A

This does the cross product of each category with each other, thus every value of the resulting matrix is the count of how often has a category occured with another in the examples.

Then, you can have a global visualization.

import matplotlib.pyplot as plt
plt.matshow(common_occurences)

You might want to normalize your matrix if some categories are overall more present than other in the dataset, because their pair co-occurences will be much higher than others, eventhough they don't happen together more frequently than not.

PlainRavioli0
Related