DBSCAN eps and min_samples

Question

I have been trying to use DBSCAN in order to detect outliers, from my understanding DBSCAN outputs -1 as outlier and 1 as inliner, but after I ran the code, I'm getting numbers that are not -1 or 1, can someone please explain why? Also is it normal to find the best value of eps using trial and error, because I couldn't figure out a way to find the best possible eps value.

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

from sklearn.cluster import DBSCAN



df = pd.read_csv('Final After Simple Filtering.csv',index_col=None,low_memory=True)


# Dropping columns with low feature importance
del df['AmbTemp_DegC']
del df['NacelleOrientation_Deg']
del df['MeasuredYawError']



#applying DBSCAN


DBSCAN = DBSCAN(eps = 1.8, min_samples =10,n_jobs=-1)

df['anomaly'] = DBSCAN.fit_predict(df)


np.unique(df['anomaly'],return_counts=True)

(array([  -1,    0,    1, ..., 8462, 8463, 8464]),
array([1737565, 3539278, 4455734, ...,      13,       8,       8]))

Thank you.

user_11065347 · Answer

Well, you did not actually get the real idea of DBSCAN.

This is a copy from wikipedia:

A point p is a core point if at least minPts points are within distance ε of it (including p).

A point q is directly reachable from p if point q is within distance ε from core point p. Points are only said to be directly reachable from core points.

A point q is reachable from p if there is a path p1, ..., pn with p1 = p and pn = q, where each pi+1 is directly reachable from pi. Note that this implies that all points on the path must be core points, with the possible exception of q.

All points not reachable from any other point are outliers or noise points.

So saying in easier words, The idea is that:

Any sample who has min_samples neigbours by the distance of epsilon is a core sample.
Any data sample which is not core, but has at least one core neighbor (with a distance less than eps), is a directly reachable sample and can be added to the cluster.
Any data sample which is not directly reachable nor a core, but has at least one directly reachable neighbor (with a distance less than eps) is a reachable sample and will be added to the cluster.
Any other examples are considered to be noise, outlier or whatever you want to name it.( and those will be labeled by -1)

Depending on the parameters of the clustering (eps and min_samples) , you are very likely to have more than two clusters. You see, that is the reason you are seeing other values than 0 and -1 in the result of your clustering.

To answer your second question

Also is it normal to find the best value of eps using trial and error,

If you mean doing cross-validation( over a set where you know the cluster labels or you can approximate the correct clustering), yes I think that is the normal way to do it

PS: The paper is very good and comprehensive. I highly suggest you have a look. Good luck.

user_2825570 · Answer

sklearn.cluster.DBSCAN gives -1 for noise, which is an outlier, all the other values other than -1 is the cluster number or cluster group. To see the total number of clusters you can use the command DBSCAN.labels_

What is eps or Epsilon value used in DBScan?

Epsilon is the local radius for expanding clusters. Think of it as a step size - DBSCAN never takes a step larger than this, but by doing multiple steps DBSCAN clusters can become much larger than eps.

How to find the best eps value?

use any hyperparameter tuning method / package like GridSearchCV or Hyperopt. You can use any of the following indices mentioned here.

user_5212614 · Answer

I have found this to be a really good example of getting to understand how DBSCAN works.

import numpy as np

from sklearn.cluster import DBSCAN
from sklearn import metrics
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler


# #############################################################################
# Generate sample data
centers = [[1, 1], [-1, -1], [1, -1]]
X, labels_true = make_blobs(n_samples=750, centers=centers, cluster_std=0.4,
                            random_state=0)

X = StandardScaler().fit_transform(X)

# #############################################################################
# Compute DBSCAN
db = DBSCAN(eps=0.3, min_samples=10).fit(X)
core_samples_mask = np.zeros_like(db.labels_, dtype=bool)
core_samples_mask[db.core_sample_indices_] = True
labels = db.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
n_noise_ = list(labels).count(-1)

print('Estimated number of clusters: %d' % n_clusters_)
print('Estimated number of noise points: %d' % n_noise_)
print("Homogeneity: %0.3f" % metrics.homogeneity_score(labels_true, labels))
print("Completeness: %0.3f" % metrics.completeness_score(labels_true, labels))
print("V-measure: %0.3f" % metrics.v_measure_score(labels_true, labels))
print("Adjusted Rand Index: %0.3f"
      % metrics.adjusted_rand_score(labels_true, labels))
print("Adjusted Mutual Information: %0.3f"
      % metrics.adjusted_mutual_info_score(labels_true, labels))
print("Silhouette Coefficient: %0.3f"
      % metrics.silhouette_score(X, labels))

# #############################################################################
# Plot result
import matplotlib.pyplot as plt

# Black removed and is used for noise instead.
unique_labels = set(labels)
colors = [plt.cm.Spectral(each)
          for each in np.linspace(0, 1, len(unique_labels))]
for k, col in zip(unique_labels, colors):
    if k == -1:
        # Black used for noise.
        col = [0, 0, 0, 1]

    class_member_mask = (labels == k)

    xy = X[class_member_mask & core_samples_mask]
    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
             markeredgecolor='k', markersize=14)

    xy = X[class_member_mask & ~core_samples_mask]
    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
             markeredgecolor='k', markersize=6)

plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()

a = np.array(labels)
a

Result:

array([ 0,  1,  0,  2,  0,  1,  1,  2,  0,  0,  1,  1,  1,  2,  1,  0, -1,
        1,  1,  2,  2,  2,  2,  2,  1,  1,  2,  0,  0,  2,  0,  1,  1,  0,
        1,  0,  2,  0,  0,  2,  2,  1,  1,  1,  1,  1,  0,  2,  0,  1,  2,
        2,  1,  1,  2,  2,  1,  0,  2,  1,  2,  2,  2,  2,  2,  0,  2,  2,
        0,  0,  0,  2,  0,  0,  2,  1, -1,  1,  0,  2,  1,  1,  0,  0,  0,
        0,  1,  2,  1,  2,  2,  0,  1,  0,  1, -1,  1,  1,  0,  0,  2,  1,
        2,  0,  2,  2,  2,  2, -1,  0, -1,  1,  1,  1,  1,  0,  0,  1,  0,
        1,  2,  1,  0,  0,  1,  2,  1,  0,  0,  2,  0,  2,  2,  2,  0, -1,
        2,  2,  0,  1,  0,  2,  0,  0,  2,  2, -1,  2,  1, -1,  2,  1,  1,
        2,  2,  2,  0,  1,  0,  1,  0,  1,  0,  2,  2, -1,  1,  2,  2,  1,
        0,  1,  2,  2,  2,  1,  1,  2,  2,  0,  1,  2,  0,  0,  2,  0,  0,
        1,  0,  1,  0,  1,  1,  2,  2,  0,  0,  1,  1,  2,  1,  2,  2,  2,
        2,  0,  2,  0,  2,  2,  0,  2,  2,  2,  0,  0,  1,  1,  1,  2,  2,
        2,  2,  1,  2,  2,  0,  0,  2,  0,  0,  0,  1,  0,  1,  1,  1,  2,
        1,  1,  0,  1,  2,  2,  1,  2,  2,  1,  0,  0,  1,  1,  1,  0,  1,
        0,  2,  0,  2,  2,  2,  2,  2,  1,  1,  0,  0,  1,  1,  0,  0,  2,
        1, -1,  2,  1,  1,  2,  1,  2,  0,  2,  2,  0,  1,  2,  2,  0,  2,
        2,  0,  0,  2,  0,  2,  0,  2,  1,  0,  0,  0,  1,  2,  1,  2,  2,
        0,  2,  2,  0,  0,  2,  1,  1,  1,  1,  1,  0,  1,  1,  1,  1,  0,
        0,  1,  1,  1,  0,  2,  0,  1,  2,  2,  0,  0,  2,  0,  2,  1,  0,
        2,  0,  2,  0,  2,  2,  0,  1,  0,  1,  0,  2,  2,  1,  1,  1,  2,
        0,  2,  0,  2,  1,  2,  2,  0,  1,  0,  1,  0,  0,  0,  0,  2,  0,
        2,  0,  1,  0,  1,  2,  1,  1,  1,  0,  1,  1,  0,  2,  1,  0,  2,
        2,  1,  1,  2,  2,  2,  1,  2,  1,  2,  0,  2,  1,  2,  1,  0,  1,
        0,  1,  1,  0,  1,  2, -1,  1,  0,  0,  2,  1,  2,  2,  2,  2,  1,
        0,  0,  0,  0,  1,  0,  2,  1,  0,  1,  2,  0,  0,  1,  0,  1,  1,
        0, -1,  0,  2,  2,  2,  1,  1,  2,  0,  1,  0,  0,  1,  0,  1,  1,
        2,  2, -1,  0,  1,  2,  2,  1,  1,  1,  1,  0,  0,  0,  2,  2,  1,
        2,  1,  0,  0,  1,  2,  1,  0,  0,  2,  0,  1,  0,  2,  1,  0,  2,
        2,  1,  0,  0,  0,  2,  1,  1,  0,  2,  0,  0,  1,  1,  1,  1,  0,
        1,  0,  1,  0,  0,  2,  0,  1,  1,  2,  1,  1,  0,  1,  0,  2,  1,
        0,  0,  1,  0,  1,  1,  2,  2,  1,  2,  2,  1,  2,  1,  1,  1,  1,
        2,  0,  0,  0,  1,  2,  2,  0,  2,  0,  2,  1,  0,  1,  1,  0,  0,
        1,  2,  1,  2,  2,  0,  2,  1,  1,  1,  2,  0,  0,  2,  0,  2,  2,
        0,  2,  0,  1,  1,  1,  1,  0,  0,  0,  2,  1,  1,  1,  1,  2,  2,
        2,  0,  2,  1,  1,  0,  0,  1,  0,  2,  1,  2,  1,  0,  2,  2,  0,
        0,  1,  0,  0,  2,  0,  0,  0,  2,  0,  2,  0,  0,  1,  1,  0,  0,
        1,  2,  2,  0,  0,  0,  0,  2, -1,  1,  1,  2,  1,  0,  0,  2,  2,
        0,  1,  2,  0,  1,  2,  2,  1,  0,  0, -1, -1,  2,  0,  0,  0,  2,
       -1,  2,  0,  1,  1,  1,  1,  1,  0,  0,  2,  1,  2,  0,  1,  1,  1,
        0,  2,  1,  1, -1,  2,  1,  2,  0,  2,  2,  1,  0,  0,  0,  1,  1,
        2,  0,  0,  2,  2,  1,  2,  2,  2,  0,  2,  1,  2,  1,  1,  1,  2,
        0,  2,  0,  2,  2,  0,  0,  2,  1,  2,  0,  2,  0,  0,  0,  1,  0,
        2,  1,  2,  0,  1,  0,  0,  2,  0,  2,  1,  1,  2,  1,  0,  1,  2,
        1,  2], dtype=int64)

Those -1 data points are outliers. Let's count the number of outliers and see if it matches what we see in the image above.

list(a)
b = a.tolist()
count = b.count(-1)
count

Result:

We got 18! Perfect!!

Relevant Link:

https://scikit-learn.org/stable/auto_examples/cluster/plot_dbscan.html#sphx-glr-auto-examples-cluster-plot-dbscan-py

DBSCAN eps and min_samples

3 Answers