Using dask.bag vs normal python list?

user_8072983 0
Viewed 266

When I run this parallel dask.bag code below, I seem to get much slower computation than the sequential Python code. Any insights into why?

import dask.bag as db

def is_even(x):
    return not x % 2

Dask code:

%%timeit
b = db.from_sequence(range(2000000))
c = b.filter(is_even).map(lambda x: x ** 2)
c.compute() 

>>> 12.8 s ± 1.15 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

# With n = 8000000
>>> 50.7 s ± 2.76 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

Python code:

%%timeit
b = list(range(2000000))
b = list(filter(is_even, b))
b = list(map(lambda x: x ** 2, b))

>>> 547 ms ± 8.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# With n = 8000000
>>> 2.25 s ± 102 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
dask parallel-processing python
1 Answers

Thanks to @abarnert for the suggestion to look at overhead through longer task length.

It seems like the length of each task was too short, and the overhead made Dask slower. I changed the exponent from 2 to 10000 to make each task longer. This example produces what I was expecting:

Python code:

%%timeit
b = list(range(50000))
b = list(filter(is_even, b))
b = list(map(lambda x: x ** 10000, b))

>>> 34.8 s ± 2.19 s per loop (mean ± std. dev. of 7 runs, 1 loop each)

Dask code: 

%%timeit
b = db.from_sequence(range(50000))
c = b.filter(is_even).map(lambda x: x ** 10000)
c.compute()

>>> 26.4 s ± 409 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
user_80729830
Related