Different values of Numpy.var() and Pandas.var()

user_10342778 0
Viewed 418

I am learning a bit about Standard Scaler in datasets. I am noticing a strange behavior which I think might be a syntax or logical error in my code, but can anyone correct me.

As we know that when we do StandardScaler, we have a std of 1 and mean of 0 as Var = Stdev^2.

Now when I used this in my code with sklearn.preprocessing.StandardScaler, after using fit_transform, the variance was 1. But when I changed the value in original data frame, It was 1.5 for both rows. Can I know why is it and how to make it 1 using StandardScaler. Thanks.

Here is my dataset

    Country GDP Area    Continent
0   India   2.79    3.287   Asia
1   USA    20.54    9.840   North America
2   China   13.61   9.590   Asia

and here is my code

from sklearn.preprocessing import StandardScaler

ss = StandardScaler()

var =  ss.fit_transform(catDf.iloc[:,1:-1]).var() #This variance is equal to 1

catDf.iloc[:,1:-1] = ss.fit_transform(catDf.iloc[:,1:-1]) 

print("Variance in Numpy array", var) # Approx 1
print("Variance in Data Frame", catDf.var())# 1.5 for both numerical columns

Hopefully, you got what I want to say.

scikit-learn pandas statistics numpy python
1 Answers

The problem is due to using different degrees of freedom. The scikit-learn docs state that they use the biased estimator or sample variance:

We use a biased estimator for the standard deviation, equivalent to numpy.std(x, ddof=0). Note that the choice of ddof is unlikely to affect model performance.

On the other hand, DataFrame.var is by default the unbiased estimator

Return unbiased variance over requested axis. Normalized by N-1 by default. This can be changed using the ddof argument

With only 3 points, the difference between dividing by 3 or 2 would lead to a 1.5x difference (exactly what you see). Remedy this by using ddof=0 for DataFrame.var()

print(catDf.var(ddof=0))
#GDP     1.0
#Area    1.0
user_43333590
Related