I have a dataframe something like this.
root
|-- key1: string (nullable = true)
|-- value1: string (nullable = true)
+----+------+
|key1|value1|
+----+------+
| E1| 1|
| E3| 0|
| E4| 1|
| E2| 0|
...
+----+------+
And i convert "value1" column to array[Int] by using collect() function as below. But this is not efficient solution, it takes 10-15 seconds. Because there are lots of data in the dataframe and in each spark streaming cycle, data is collected to the driver.
val data = Seq(("E1","1"),
("E3","0"),
("E4","1"),
("E2","0")
)
val columns = Seq("key1", "value1")
import spark.implicits._
val df = data.toDF(columns:_*)
val ordered_df = df.orderBy("key1").select("value1").collect().map(_(0)).toList
ordered_df.foreach(print)
Output : 1001
So, what is the efficient way to return Array of Int from the above dataframe without using Collect() function ?
Thanks,