How to remove 99th percentile outliers in R

user_7503634 0
Viewed 982

I simply want to replace outlier values (those above the 99th percentile) by NA for each group individualy. I dont know how to do that without creating an entirely new dataframe which is redundant. Any ideas ?

group <- c('A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B', 'B')
var1 <- c(1, 2, 3, 4, 5, 6, 7, 8, 9, 50, 1, 2, 3, 4, 5, 6, 7, 8, 9, 50)
df = data.frame(group, var1)

unique_groups = unique(df$group)

df2 = data.frame()

for(g in 1:length(unique_groups)) {
  
  subset_df <- df[grep(unique_groups[1], df$group), ]

  quantiles <- quantile(subset_df$var1, c(.01, .99), na.rm = TRUE)
  
  subset_df$var1[subset_df$var1 > quantiles[2]] <- NA
  
  df2 <- rbind(df2, subset_df)
}

Thank you,

r
3 Answers

Another approach using dplyr

library(dplyr)
df %>% 
  group_by(group) %>% 
  mutate(var1 = ifelse(var1 > quantile(var1, .99), NA, var1))

You can also filter for keeping values meeting a conditon

df %>% 
  group_by(group) %>% 
  filter(var1 <= quantile(var1, .99))
user_13157670

Using ave.

df$var1[with(df, !!ave(var1, group, FUN=function(x) x > quantile(x, .99)))] <- NA
df
#    group var1
# 1      A    1
# 2      A    2
# 3      A    3
# 4      A    4
# 5      A    5
# 6      A    6
# 7      A    7
# 8      A    8
# 9      A    9
# 10     A   NA
# 11     B    1
# 12     B    2
# 13     B    3
# 14     B    4
# 15     B    5
# 16     B    6
# 17     B    7
# 18     B    8
# 19     B    9
# 20     B   NA
user_65740380

You could also use the replace() creating your index with quantile():

df$var1 = replace(df$var1, df$var1 >= quantile(df$var1, .99), NA)
user_101122850
Related