Is there an efficient way to return Array[Int] from a spark Dataframe without using collect()

Viewed 16

I have a dataframe something like this.

root
 |-- key1: string (nullable = true)
 |-- value1: string (nullable = true)

+----+------+
|key1|value1|
+----+------+
|  E1|     1|
|  E3|     0|
|  E4|     1|
|  E2|     0|
...
+----+------+

And i convert "value1" column to array[Int] by using collect() function as below. But this is not efficient solution, it takes 10-15 seconds. Because there are lots of data in the dataframe and in each spark streaming cycle, data is collected to the driver.

val data = Seq(("E1","1"),
      ("E3","0"),
      ("E4","1"),
      ("E2","0")
)

val columns = Seq("key1", "value1")
import spark.implicits._
val df = data.toDF(columns:_*)
val ordered_df = df.orderBy("key1").select("value1").collect().map(_(0)).toList
ordered_df.foreach(print)
Output :
1001

So, what is the efficient way to return Array of Int from the above dataframe without using Collect() function ?

Thanks,

0 Answers
Related