pyspark partitioning data using partitionby

Viewed 23560

I understand that partitionBy function partitions my data. If I use rdd.partitionBy(100) it will partition my data by key into 100 parts. i.e. data associated with similar keys will be grouped together

  1. Is my understanding correct?
  2. Is it advisable to have number of partitions equal to number of available cores? Does that make processing more efficient?
  3. what if my data is not in key,value format. Can i still use this function?
  4. lets say my data is serial_number_of_student,student_name. In this case can i partition my data by student_name instead of the serial_number?
2 Answers
Related