Spark DataFrame RangePartitioner

user_2832655 0
Viewed 1791

[New to Spark] Language - Scala

As per docs, RangePartitioner sorts and divides the elements into chunks and distributes the chunks to different machines. How would it work for below example.

Let's say we have a dataframe with 2 columns and one column (say 'A') has continuous values from 1 to 1000. There is another dataframe with same schema but the corresponding column has only 4 values 30, 250, 500, 900. (These could be any values, randomly selected from 1 to 1000)

If I partition both using RangePartitioner,

df_a.partitionByRange($"A")
df_b.partitionByRange($"A")

how will the data from both the dataframes be distributed across nodes ?

Assuming that the number of partitions is 5.

Also, if I know that second DataFrame has less number of values then will reducing number of partitions for it make any difference ?

What I am struggling to understand is that how Spark maps one partition of df_a to a partition of df_b and how it sends (if it does) both those partitions to same machine for processing.

apache-spark-dataset apache-spark-sql apache-spark
1 Answers

A very detailed explanation of how RangePartitioner works internally is described here

Specific to your question, RangePartitioner samples the RDD at runtime, collects the statistics, and only then are the ranges (limits) evaluated. Note that there are 2 parameters here - ranges (logical), and partitions (physical). The number of partitions can be affected by many factors - number of input files, inherited number from parent RDD, 'spark.sql.shuffle.partitions' in case of shuffling, etc. The ranges evaluated according to the sampling. In any case, RangePartitioner ensures every range is contained in single partition.

how will the data from both the dataframes be distributed across nodes ? how Spark maps one partition of df_a to a partition of df_b

I assume you implicitly mean joining 'A' and 'B', otherwise the question does not make any sense. In that case, Spark would make sure to match partitions with ranges on both DataFrames, according to their statistics.

user_46491430
Related