I have two columns in a row which have range values. eg: from_series: 100 and to_series: 110. Now I need to convert these to multiple rows.
name value from_series to_series
aaa 32 100 110
I need to convert as shown below:
name value series
aaa 32 100
aaa 32 101
aaa 32 102 until 110
I think this code snippet should work where df is your dataFrame.
df.rdd.flatMap{row=>
val name = row.getAs[String]("name")
val value = row.getAs[Int]("value")
val fromSeries = row.getAs[Int]("from_series")
val toSeries = row.getAs[Int]("to_series")
(fromSeries to toSeries).map(seriesValue=>(name,value,seriesValue))
}