python clustering package that can cluster based in distance matrix but also predict new row (without new clustering/distance matrix)

user_283538 0
Viewed 79

I am aware of various (sklearn) clustering algorithm that work with distance matrices - e.g. produced via a proximity matrix coming from a random forest (some clumsy reproducible code below). Is there any clustering algorithm (working with distance matrix), where the fitted cluster model (e.g. cluster_model below) can produce the cluster membership of a new data row?

from sklearn.metrics.pairwise import pairwise_distances
from sklearn.cluster import AgglomerativeClustering
import pandas as pd
import numpy as np
import os
from sklearn.ensemble import RandomForestRegressor
from sklearn import datasets

def distanceMatrix(model, X, normalize=True):

    terminals = model.apply(X)
    nTrees = terminals.shape[1]

    a = terminals[:,0]
    proxMat = 1 * np.equal.outer(a, a)

    for i in range(1, nTrees):
        a = terminals[:,i]
        proxMat += 1*np.equal.outer(a, a)

    if normalize:
        proxMat = proxMat / nTrees

    return 1 - proxMat  

# use iris data to make example reproducible and fast
iris = datasets.load_iris()
df = pd.DataFrame(iris['data'], columns = iris['feature_names'])
df['target'] = pd.Series(iris['target'], name = 'target_values')
df['target_name'] = df['target'].replace([0,1,2], ['iris-' + species for species in iris['target_names'].tolist()])

# simple one hot
df['iris_setosa'] = (df['target_name'] == 'iris-setosa').astype(int)
df['iris_versicolor'] = (df['target_name'] == 'iris-versicolor').astype(int)
df['iris_virginica'] = (df['target_name'] == 'iris-virginica').astype(int)

# the new regression model "target"
y = df['petal width (cm)']

X = df.drop([
    'target'
    ,'target_name'
    ,'petal width (cm)'
], axis = 1)

# fit random forest just for the purpose of getting proximity matrix
# open question does it matter which target is picked and/or whether regresion or classification?
# this is just to produce a toy dataset with mixed data
overfitted_model = RandomForestRegressor(n_estimators=250, min_samples_leaf=10)
overfitted_model.fit(X, y)

distance_matrix = distanceMatrix(overfitted_model, X, normalize=True)

cluster_model = AgglomerativeClustering(n_clusters=3, affinity='precomputed', linkage='average')
cluster_model.fit(distance_matrix)

df['label'] = cluster_model.labels_

PS:

  • Readers may find this interesting in this context.
hierarchical-clustering scikit-learn random-forest python
1 Answers

For Agglomerative Clustering, adding additional data points requires a recompute of the clusters because of how this type of clustering works. Agglomerative Clustering iteratively builds clusters based on started points, and then merges according to a linkage measure, so adding new data points can and will modify the final clusters.

...
cluster_model.fit_predict(distance_matrix)

df['label'] = cluster_model.labels_

Check out fit_predict, which is more relevant for unsupervised or transductive estimators.

output:

     sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  target     target_name  iris_setosa  iris_versicolor  iris_virginica  label
0                  5.1               3.5                1.4               0.2       0     iris-setosa            1                0               0      1
1                  4.9               3.0                1.4               0.2       0     iris-setosa            1                0               0      1
2                  4.7               3.2                1.3               0.2       0     iris-setosa            1                0               0      1
3                  4.6               3.1                1.5               0.2       0     iris-setosa            1                0               0      1
4                  5.0               3.6                1.4               0.2       0     iris-setosa            1                0               0      1
..                 ...               ...                ...               ...     ...             ...          ...              ...             ...    ...
145                6.7               3.0                5.2               2.3       2  iris-virginica            0                0               1      2
146                6.3               2.5                5.0               1.9       2  iris-virginica            0                0               1      2
147                6.5               3.0                5.2               2.0       2  iris-virginica            0                0               1      2
148                6.2               3.4                5.4               2.3       2  iris-virginica            0                0               1      2
149                5.9               3.0                5.1               1.8       2  iris-virginica            0                0               1      2
user_84682640
Related