How many partitions will pyspark-sql create while reading a .csv?
My understanding of this is that
number of partitions = math.ceil(file_size/spark.conf.get('spark.sql.files.maxPartitionBytes'))
On my machine:
spark.conf.get('spark.sql.files.maxPartitionBytes')
output:
'134217728b' #128MBs
However, I am not observing this behaviour. I create a file that is taking up 96 MB on disk. I run spark in local mode. I have an 8 core laptop. I think it should have read in 1 partition. However, the file is getting read in 8 partitions. Below is the codebase I have used:
import pandas as pd
import numpy as np
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
#creating a small DataFrame. This will occupy 96 MBs on disk
pd.DataFrame({'id':np.arange(10000000),'b':np.random.choice(['a','b','c','d'],size=(10000000,),p=[0.25,0.25,0.25,0.25])}).to_csv('df_s.csv',index=None)
sd=spark.read.csv('df_s.csv',schema="id int, b string")
sd.rdd.getNumPartitions()
output: 8
Can you please help me understand why am I seeing 8 partitions irrespective of the file size?