What is sigma clipping? How do you know when to apply it?

user_4643092 0
Viewed 9924

I'm reading a book on Data Science for Python and the author applies 'sigma-clipping operation' to remove outliers due to typos. However the process isn't explained at all.

What is sigma clipping? Is it only applicable for certain data (eg. in the book it's used towards birth rates in US)?

As per the text:

quartiles = np.percentile(births['births'], [25, 50, 75]) #so we find the 25th, 50th, and 75th percentiles
mu = quartiles[1] #we set mu = 50th percentile
sig = 0.74 * (quartiles[2] - quartiles[0]) #???

This final line is a robust estimate of the sample mean, where the 0.74 comes 
from the interquartile range of a Gaussian distribution.

Why 0.74? Is there a proof for this?

data-science pandas statistics numpy python
3 Answers

I think there is a small typo to the sentence that "this final line is a strong estimate of the sample average". From the previous proof, I think the final line is a solid estimate of 1 Sigma for births if the normal distribution is followed.

user_134094010
Related