In Pyspark, I can create a RDD from a list and decide how many partitions to have:
sc = SparkContext()
sc.parallelize(xrange(0, 10), 4)
How does the number of partitions I decide to partition my RDD in influence the performance? And how does this depend on the number of core my machine has?
Number of partition have high impact on spark's code performance.
Ideally the spark partition implies how much data you want to shuffle. Normally you should set this parameter on your shuffle size(shuffle read/write) and then you can set the number of partition as 128 to 256 MB per partition to gain maximum performance.
You can set partition in your spark sql code by setting the property as:
spark.sql.shuffle.partitions
or while using any dataframe you can set this by below:
df.repartition(numOfPartitions)