Hierarchical clustering for categorical data in python

Viewed 8643

I have a categorical attributes that contains string values. three of them contains dayname(mon---sun) monthname and time interval(morning afternoon evening), the other two as i mentioned before has district and street names. followed by gender ,role, comments(it is a predefined fixed field that have values as good, bad strong agree etc)surname and first name.my intention is to cluster them and visualize it. I applied k-mean clustering using this WEKA bur it did not work. Now I wish to apply hierarchical clustering on it. I found this code:

import scipy
import scipy.cluster.hierarchy as sch
X = scipy.randn(100, 2)     # 100 2-dimensional observations
d = sch.distance.pdist(X)   # vector of (100 choose 2) pairwise distances
L = sch.linkage(d, method='complete')
ind = sch.fcluster(L, 0.5*d.max(), 'distance')

However, X in above code is numeric; I have categorical data. Is there some way that I can use a numarray of categorical data to find the distance? In other words can I use categorical data of string values to find the distance? I would then use that distance in sch.linkage(d, method='complete')

2 Answers

Another possibility is the use of the Hamming distance.

Y = pdist(X, 'hamming')

Computes the normalized Hamming distance, or the proportion of those vector elements between two n-vectors u and v which disagree. To save memory, the matrix X can be of type boolean.

If your categorical data is represented by a single character e.g.: "m"/"f" it could be what you are looking for.

https://en.wikipedia.org/wiki/Hamming_distance

https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html#scipy.spatial.distance.pdist

Related