As title.
Is there any possible way to optimize two large rdd join when both of them is too large for memory? In this case I suppose we cannot use broadcast for map side join.
If I have to join this two rdd, and both of them is too large to fit in memory:
country_rdd:
(id, country)
income_rdd:
(id, (income, month, year))
joined_rdd = income_rdd.join(country_rdd)
Is there any possible way to reduce the shuffling here? Or anything I can do to tuning the join performance?
Besides, the joined_rdd will be further calculated and reduced only by country and time, not relevant to id anymore. Eg: my final result = income for different country in different years. What's the best practice to do that?
I used to consider do some pre-partition, but seems if I only need to do join once that won't help much?