Why is the function imported from a module significantly slower than a local function in Python?

user_15572468 0
Viewed 82

I'm trying to calculate the standard deviation for a large set of data, at first I used the statistics module to import the function.

from statistics import pstdev

But the result is very slow, so I decided to write a local helper method which does the exactly same thing,

def get_std_dev(ls):
    n = len(ls)
    mean = sum(ls) / n
    var = sum((x - mean)**2 for x in ls) / n
    std_dev = var ** 0.5
    return std_dev

This runs significantly faster! Here are the runtime comparison

Runtime with my written function:  0:00:00.532228
Runtime with imported module function:  0:00:17.605583

I am very confused why the imported function is so slow compared to my local written function. Does it have to do with memory location?

The only difference between the two functions are the these pieces of codes

    stdev = get_std_dev(close_price_list) # my written one
    stdev = pstdev(close_price_list) # the imported function
module oop python
1 Answers

This is not an answer

This is just to explain some of my comments more clearly. With x = np.random.random(10_000_000) I get these timings:

In [3]: %timeit statistics.pstdev(x)
39.4 s ± 428 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit get_std_dev(x)
4.53 s ± 132 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit x.std(ddof=0)
52.6 ms ± 2.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
user_62987120
Related