I am trying to create and implement a function that identifies outliers in datasets in Python using the numpy module, keep getting 'ValueError'

user_14471931 0
Viewed 108

I am attempting to create an 'Outlier' function that detects outliers in data sets. I am then trying to call the function into a for loop but it keeps giving me a ValueError. I have a brief understanding of why the error occurs. It's because numpy doesn't let you set arrays as Booleans (Please correct me if I'm wrong). I was just wondering if there was a way around this, and how would I implement the a.any(), a.all() suggestions the error is giving me.

Code:

import numpy as np

def Outlier(a, IQR, Q1, Q3):
    if a < Q1 - 1.5 * IQR or a > Q3 + 1.5 * IQR:    
        outlier = True
    else:
        outlier = False        
    return(outlier)


data_clean = []

Q1 = np.percentile(data, 25)
Q3 = np.percentile(data, 75)

print("Q1 = {:,.2f}".format(Q1)) 
print("Q3 = {:,.2f}".format(Q3)) 

n= len(data)

for i in range(n): 
    outlier[i] = Outlier(data, IQR, Q1, Q3) # Error

    if outlier[i] == False : # Error
        data_clean.append(data[i])
    else:
        print("value removed (outlier) = {:,.2f}".format(data[i]))

data_clean = np.asarray(data_clean)
n = len(data_clean) 
print("n = {:.0f}".format(n)) 

print("data_clean = {}".format(data_clean))

Full error:

ValueError                                Traceback (most recent call last)
<ipython-input-30-f686bd0a0718> in <module>

---> 19     outlier[i] = Outlier(data, IQR, Q1, Q3)
---> 20     if outlier[i] == False :            #check for outlier

<ipython-input-29-1f034e2a09b6> in Outlier(a, IQR, Q1, Q3)
3 def Outlier(a, IQR, Q1, Q3):
4
---> 5     if a < Q1 - 1.5 * IQR or a > Q3 + 1.5 * IQR:
6
7              outlier = True
ValueError: The truth value of an array with more than one element is ambiguous.
Use a.any() or a.all()

Thanks in advance. Just to clarify the code above is meant to check for outliers in a dataset and then add the non-outliers to a data_clean list, leaving the dataset with no outliers.

dataset statistics numpy python
1 Answers

I think that, rather than iterating, you could do everything inside the function without looping:

import numpy as np

def detect_outliers(arr, pct_bounds=(75, 25), mulitplier=1.5):
    upper, lower = np.percentile(arr, pct_bounds)
    spread = upper - lower
    # Create a mask where the data are more than or less than 1.5 * IQR from the median
    mask = (arr < (np.median(arr) - (multiplier * spread))) | (arr > (np.median(arr) + (multiplier * spread)))
    outliers = arr[mask]
    # Invert mask
    new_array = arr[~mask]
    return outliers, new_array

test_data = np.random.normal(size=100)
outliers, data = detect_outliers(test_data)
user_66998750
Related