How to aggregate data which come from MongoDB on Spark executor not in driver?

Viewed 21

I have data on mongodb. I want to reach that data from spark and make aggregations on executor not in driver.

But the thing that I couldnt understand is I m not sure that it is possible .

For example I have data on mongodb like this:

key     value
----    -----
1         5
1         7
1         8

lets say I have 3 executor and when I select data in spark from mongo , each executor get 1 row from db.

So I want to get max value for each key. How is it possible without getting it to the driver?

As I know each executor has own value in itself and I cant get max value for a key. (Maybe I can write my own shuffle system and get same key in same executor but i m not sure is it efficient way)

my scala code is like this:

.....

 df.coalesce(8).foreachPartition { iter =>

      if (iter.hasNext) {

        val con = ConnectionPool.getConnectionPool(dbName, url, user, pass, poolSize).getConnection

        val batchSelectValue= con.prepareStatement(sql)

        while (iter.hasNext) {
                /*my logic*/              
          }
 
        }
             /* I want some aggregations in here*/ 
    }
.....
0 Answers
Related