Remove outliers from the wage distribution

user_12378480 0
Viewed 37

Consider the following column of (random generated) wages

mydata<-runif(2500, min =-500, max = 100000)
mydata<-as.data.frame(mydata)
colnames(mydata)[1]<-"wage"

#How to drop the bottom and top 5%?

(The wages can be negative because of taxes payable). Now I want to drop the 5% lowest wages and the 5% highest wages as I consider these wages as outliers. Is there a way to effectively do this quickly? I looked at the website, but I could not find a link that tells me how to do this.

I appreciate any help.

r
4 Answers

Here is a solution using the dplyr package. The quantile can calculate the percentile. In your case, top and bottom 5% would be the 95 and 5 percentile.

library(dplyr)

mydata2 <- mydata %>%
  filter(wage < quantile(wage, 0.95) & wage > quantile(wage, 0.05))
user_76698090 
rm(list = ls())
mydata<-runif(2500, min =-500, max = 100000)
mydata<-as.data.frame(mydata)
colnames(mydata)[1]<-"wage"

n <- length(mydata$wage)

wage <- sort(mydata$wage)
wage <- wage[(126):(n-125)]
user_115721730

Here is an alternative using base R:

lower <- quantile(mydata[["wage"]],0.05)
upper <- quantile(mydata[["wage"]],0.95)
mydata[ which( mydata[["wage"]] > lower & mydata[["wage"]] < upper ) , , drop=FALSE]

The argument drop=FALSE is required to retain the data.frame structure.

user_151577680

You can use quantile and findInterval to subset those within 5% and 95%:

mydata[findInterval(mydata$wage, quantile(mydata$wage, c(0.05, 0.95))) == 1,,
  drop=FALSE]
user_104885040
Related