How to find outliers in a series, vectorized?

Viewed 906

I have a pandas.Series of positive numbers. I need to find the indexes of "outliers", whose values depart by 3 or more from the previous "norm".

How to vectorize this function:

def baseline(s):
    values = []
    indexes = []
    last_valid = s.iloc[0]
    for idx, val in s.iteritems():
        if abs(val - last_valid) >= 3:
            values.append(val)
            indexes.append(idx)
        else:
            last_valid = val
    return pd.Series(values, index=indexes)

For example, if the input is:

import pandas as pd
s = pd.Series([7,8,9,10,14,10,10,14,100,14,10])
print baseline(s)

the desired output is:

4     14
7     14
8    100
9     14

Note that the 10 values after the 14s are not returned because they are "back to normal" values.

Edits:

  • Added abs() to the code. The numbers are positive.
  • The purpose here is to speed up the code.
  • An answer that doesn't exactly imitate the code may be acceptable.
  • Changed the example to include another edge case, where the values slowly change by 3.
2 Answers
Related