Convert a DataFrame to RDD and Split the RDD into the same number of Columns as DataFrame Dynamically

user_16568640 0
Viewed 78

I am trying to Convert a DataFrame into RDD and Splitting them into Specific number of Columns based on Number of Columns in DataFrame Dynamically and Elegantly

i.e This is a sample data from a table in hive employee

Id  Name    Age State   City
123 Bob 34  Texas   Dallas
456 Stan    26  Florida Tampa

val temp_df = spark.sql("Select * from employee")
val temp2_rdd = temp_df.rdd.map(x => (x(0),x(1),x(2),x(3))

I am looking to generate the tem2_rdd dynamically based on the number of columns from the table. It should not be hard coded as i did.

As the maximum size of tuple is 22 in scala, any other collection that can hold the rdd efficiently.

Coding Language : Spark Scala

Please advise.

rdd apache-spark dataframe scala
1 Answers

Instead of extracting and transforming each element using index you can use toSeq method of Row object.

  val temp_df = spark.sql("Select * from employee")
  // RDD[List[Any]]
  val temp2_rdd = temp_df.rdd.map(_.toSeq.toList)
  // RDD[List[String]]
  val temp3_rdd = temp_df.rdd.map(_.toSeq.map(_.toString).toList)
user_87733090
Related