pyspark performance vs pure python doing simple sum

user_5092926 0
Viewed 602

On a Ubuntu 16.04 virtual machine with 4 CPUs, I did a simple comparison on the performance of pyspark vs pure python. I run spark as local installation on the virtual machine with 4 cpus.

#!/home/python3/venv/bin/python3
import pyspark
from pyspark.sql import SparkSession
from operator import add
from datetime import datetime

spark = SparkSession.builder.appName('ai_project').getOrCreate()
len = 1000000000
for i in range(1):
    start = datetime.now()
    print("start:", start)
    t=spark.sparkContext.parallelize(range(len))
    a = t.fold(0, add)
    print(a)
    end= datetime.now()
    print("end for spark sum:", end, end-start)  

s = 0
start = datetime.now()
print("start for loop sum:", start)
for i in range(len):
    s = s + i
print("sum=", s)
end= datetime.now()
print("end for loop sum:", end, end-start)

Here is the output:

(venv) cju@ubuntu-16:~/cloudmap3-ai-datasets/examples$ ./t1.py 
20/05/15 10:22:34 WARN Utils: Your hostname, ubuntu-16 resolves to a loopback address: 127.0.1.1; using 192.168.2.113 instead (on interface enp0s3)
20/05/15 10:22:34 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
20/05/15 10:22:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
start: 2020-05-15 10:22:36.771454
49999999995000000000                                                            
end for spark sum: 2020-05-15 10:36:34.811598 0:13:58.040144
start for loop sum: 2020-05-15 10:36:34.811630
sum= 49999999995000000000
end for loop sum: 2020-05-15 10:47:53.770664 0:11:18.959034

The questions are:

 1. pyspark code took 13:58 minutes and the pure python code took 11:18
    minutes. when pyspark code is running, 4 CPUs is at 100%
    utilization. when pure python code is running, only 1 CPU is at 100%
    utilization. I would expect the pyspark code take much less time.
 2. Another question is about the add operator. Is the add operation
    done in a python process and the result is communicated back to JVM?
    Or the add operation is done in the JVM of the worker process?
 3. add is not a udf. It should be running in JVM rather than in a 
    python process. Could someone explain more?

Changing fold to reduce gives the same result. I also tried using dataframe to do the sum. But it is even worse. When the dataframe sum is running, only one CPU is at 100%. Also dataframe is using much more memory and much more slow. Here is the comparison:

--------------------------------------------------    
|len   |pyspark rdd|pyspark dataframe|pure python|
--------------------------------------------------
|10^7  |1.39sec    |16.36sec         |0.67sec    |
--------------------------------------------------
|10^8  |7.02sec    | out of memory   |6.94sec    |
--------------------------------------------------
|10^9  |13:58min   | out of memory   |11:18min   |
--------------------------------------------------
pyspark python
2 Answers

IF you are comparing spark performance with python or pandas for small operations, then pandas will always outperform spark.

Spark is a distributed processing engine which should be used when data size is very large in 10's of GB as spark distributes the data to all the cores and process it individually.

Now for your sum operation python executes faster as it process it on single core and it doesn't includes any overhead while spark first distributes it then process it then reduces the result to single node which in itself is a lot of overhead for such small operations.

If you want more performance gain for this type of operations I would suggest use Numpy as it's C based and performs vectorized calculations.

user_53527480

It could be the overhead of running spark. But...I'm not sure fold should be used in this scenario. You may want .reduce(add).

Another thought is converting the rdd to a spark dataframe and then use pyspark.sql.functions.sum. That is likely to be much more performant since it's kept in the jvm.

The add function is done in python. The serialization to/from jvm to python to apply the add function will result in a huge performance hit. This is why it's often recommended to stay away from python functions/udf's when using pyspark. Opt for pandas udf only when absolutely necessary.

user_135461780
Related