Python Pandas - Find and Group Outliers

user_12821675 0
Viewed 146

I have a pd dataframe with multiple columns like so (simplified for ease of read) - each row consists of an id (uuid), index, and one or more features:

uuid         index           Atrium       Ventricle 
di-abc           0            20.73           26.21
di-abc           1            18.92           25.14
di-efg           7            19.02            0.30 
di-efg           9             1.23            0.51
di-efg           6            21.24           26.02
di-hjk           3            22.10           25.16
di-hjk           6            19.16           25.57

I would like to:

  1. Find the outliers for each feature (i.e. columns 'Atrium' and 'Ventricle')
  2. Export the outliers in the following format:
outliers = {
    'Atrium' : [
         {'uuid' : 'di-efg', 'index' : 9, 'value' : 1.23},
     ],
     'Ventricle' : [
         {'uuid' : 'di-efg', 'index' : 7, 'value' : 0.30},
         {'uuid' : 'di-efg', 'index' : 9, 'value' : 0.53},
    ]
}

Caveats (bonus points for handing this):

  1. The number of features (and hence columns) is dynamic
  2. A single row can contain zero, one, two, or more outliers

I am having difficulty with both steps outside of a double for loop. Is there an efficient way to compute the outliers from this dataframe?

Here is a working, though not efficient, method of capturing what I am trying to accomplish:

# initialize variables:
outliers = {}
features = ['Atrium', 'Ventricle']

# iterate over each feature:
for feature in features:

    # set feature on outlier to empty list:
    outliers[feature] = []
            
    # create a dataframe of outliers for that specific feature:
    outlier_df = df[df[feature] > (df[feature].mean() + df[feature].std())] # can mess with this if needed
    outlier_df = outlier_df[['dicom', 'frame', 'index', feature]]
            
    # iterate through the data frame and find the uuid, index, and feature:
    for index, row in outlier_df.iterrows():

        # append each outlier to the outlier dictionary:
        outliers[feature].append({
             'uuid' : row['uuid'],
             'index' : row['index'],
             'value' : row[feature],
         })
pandas python
1 Answers

Here is one way to approach the problem by defining a function which takes the input argument as column name and returns the all the outliers in the current column in the desired format:

def detect_outliers(col):
    # Define your outlier detection condition here
    mask = (df[col] - df[col].mean()).abs() > df[col].std()
    return df.loc[mask, ['uuid', 'index', col]]\
             .rename(columns={col: 'value'}).to_dict('records')

outliers = {col: detect_outliers(col) for col in features}

Alternate approach a bit more involved with pandas operations like stacking, grouping and aggregation:

# Select only feature columns
feature_df = df.set_index(['uuid', 'index'])[features]

# Define your outlier detection condition
mask = (feature_df - feature_df.mean()).abs() > feature_df.std()

# Prepare outlier dataframe
outlier_df = feature_df[mask].stack().reset_index(level=[0, 1], name='value')
outlier_df['records'] = outlier_df.to_dict('r')

# Get the outliers in the desired format
outliers = outlier_df.groupby(level=0).agg(list)['records'].to_dict()

>>> outliers

{
    'Atrium': [
        {'uuid': 'di-efg', 'index': 9, 'value': 1.23}
    ],
    'Ventricle': [
        {'uuid': 'di-efg', 'index': 7, 'value': 0.3},
        {'uuid': 'di-efg', 'index': 9, 'value': 0.51}
    ]
}
user_128331660
Related