I have data on mongodb. I want to reach that data from spark and make aggregations on executor not in driver.
But the thing that I couldnt understand is I m not sure that it is possible .
For example I have data on mongodb like this:
key value
---- -----
1 5
1 7
1 8
lets say I have 3 executor and when I select data in spark from mongo , each executor get 1 row from db.
So I want to get max value for each key. How is it possible without getting it to the driver?
As I know each executor has own value in itself and I cant get max value for a key. (Maybe I can write my own shuffle system and get same key in same executor but i m not sure is it efficient way)
my scala code is like this:
.....
df.coalesce(8).foreachPartition { iter =>
if (iter.hasNext) {
val con = ConnectionPool.getConnectionPool(dbName, url, user, pass, poolSize).getConnection
val batchSelectValue= con.prepareStatement(sql)
while (iter.hasNext) {
/*my logic*/
}
}
/* I want some aggregations in here*/
}
.....