(Spark) Is there any possible way to optimize two large rdd join when both of them is too large for memory(means cannot use broadcast)?

Viewed 210

As title.

Is there any possible way to optimize two large rdd join when both of them is too large for memory? In this case I suppose we cannot use broadcast for map side join.

If I have to join this two rdd, and both of them is too large to fit in memory:

country_rdd: 
(id, country)

income_rdd:
(id, (income, month, year))

joined_rdd = income_rdd.join(country_rdd)

Is there any possible way to reduce the shuffling here? Or anything I can do to tuning the join performance?

Besides, the joined_rdd will be further calculated and reduced only by country and time, not relevant to id anymore. Eg: my final result = income for different country in different years. What's the best practice to do that?

I used to consider do some pre-partition, but seems if I only need to do join once that won't help much?

1 Answers
Related