numpy variance vs homegrown variance (different result)

Question

I am reading calculating variance and see a formula variance = sum( (vals - mean(vals))**2 ) implemented in Python like:

def variance(values, mean):
    return sum([(x-mean)**2 for x in values])

I used this implementation and also walked through it step by step in a notebook and got the same answer. However, when I used .var(), I got a different result. Is the numpy implementation different? Is the function stated earlier incorrect?

With a randomly generated 4-element array nums like:

array([[0.10193815],
       [0.99458125],
       [0.57937386],
       [0.27331835]])

nums.var() = 0.11502594279263058

variance(nums, nums.mean()) = 0.4601037711705223

user_4044696 · Answer

There is well-known problem with NumPy std deviation (and variance) - there is no Bessel correction.

E.g. Python 3.8, Win10 x64

import numpy as np
import pandas as pd

a = np.asarray([1., 2., 3., 4., 5.])

print(np.std(a))
print(pd.Series(a).std())

will print

1.4142135623730951
1.5811388300841898

You have to do correction manually.

print(np.std(a)*np.sqrt((l := len(a))/(l-1.0)))

will print

1.5811388300841898

in full agreement with Pandas and pretty much all other software. For example, in R

x <- c(1, 2, 3, 4, 5)
sd(x)

will print

[1] 1.581139

UPDATE

IT is worth mentioning that NumPy now provides additional parameter to fix for bessel correction (and compute unbiased estimator) - ddof. Default is 0, but when you set it to 1, right result will be printed

print(np.std([1., 2., 3., 4., 5.], ddof=1))

prints

1.5811388300841898

user_4621513 · Answer

https://numpy.org/doc/stable/reference/generated/numpy.var.html says:

The variance is the average of the squared deviations from the mean, i.e., var = mean(abs(x - x.mean())**2).

i.e., where you are using sum, NumPy uses mean.

If you divide your result by 4 (the length of nums) you should get the same value.

numpy variance vs homegrown variance (different result)

2 Answers