How can I search for anomalies in each column in a Dataframe?

user_12458214 0
Viewed 1646

I have a Dataframe and my goal is to find anomalies for each different column. So I am looking for univariate anomalies.

Let's assume this is my Dataframe:

df=pd.DataFrame(np.random.rand(100, 6) * 1, columns=['A','B','C','D','E','F'])

I am faced with two questions:

  1. Which algorithms are adequate for this goal? E.g. Isolation Forest?
  2. How could I run an algorithm (E.g. Isolation Forest) over all columns, rather than doing it column per column? Can I use a for loop?

Thanks for your help!

anomaly-detection dataframe for-loop python
1 Answers

Q2:eg.

df = pd.DataFrame({"bytes":[1,2,3,4,5], "flow":[1,2,3,4,5], "userid":[1,2,3,4,5]}).set_index("userid")

def get_anomaly(arr):
    # your algorithm
    if arr.bytes < 3 and arr.flow < 3:
        return -1
    elif arr.bytes > 3 and arr.flow > 3:
        return 1
    else:
        return 0

df['is_anomaly'] = df.apply(get_anomaly, axis=1)

>>> df
   bytes  flow  userid  is_anomaly
0      1     1       1       -1
1      2     2       2       -1
2      3     3       3        0
3      4     4       4        1
4      5     5       5        1

We can talk a little bit about Q1.

Level 0: Linear relationships or other experiences

Box-plot: min outlier < Q1-1.5ΔQ <= normal data <= Q3+1.5ΔQ < max outlier

Scott rule: Δb=3.5σn1/3 .Split the data and do distribution statistics

Other data status: avg. mean std and so on.

Level 1: Statistical algorithm

Great algo: 
CMP
https://www.sciencedirect.com/science/article/abs/pii/S1389128616301633

Beehive
https://nds2.ccs.neu.edu/papers/Beehive.pdf

CBLOF
https://www.goldiges.de/publications/Anomaly_Detection_Algorithms_for_RapidMiner.pdf

And some AR MA ARMA algo, I don't know much.

Level 2: Unsupervised learning

Kmeans and so on...(This is actually quite a lot)

Level 3: Supervised learning

from elasticsearch (doc)

EWMA  
s2=α*x2+(1-α)*s1

Holt-Linear  
s2=α*x2+(1-α)*(s1+t1)
t2=ß*(s2-s1)+(1-ß)*t1

Holt-Winters
si=α(xi-pi-k)+(1-α)(si-1+ti-1)
ti=ß(si-si-1)+(1-ß)ti-1
pi=γ(xi-si)+(1-γ)pi-k

from ML
CNN RNN LSTM Prefixspan AutoML Bayes and so on.(There are a few scenarios you can use.)

There are too many left unlisted, too many algorithms to use, too many appropriate, too many details to write down. UEBA's thinking is important when analyzing anomalies.

user_115079100
Related