select efficiently specific columns from multiple dataframes join

Viewed 26

I'm joining multiple dataframe and selecting specific columns to create desired output

For example

    df1.join(df2, condition).join(df3, some condition).join(df4, some condition)
    .map {x =>
       
       val someData_1 = x._1._1._1
       val someData_2 = x._1._1._2
       val someData_3 = x._1._2
       val someData_4 = x._2

       Output(
       someData_1.getAs("colName1").asInstanceOf[String],
       .
       .
       .
       someData_2.getAs("colName2").asInstanceOf[String],
       .
       .
       .
       someData_3.getAs("colName3").asInstanceOf[String],
       .
       .
       .
       someData_4.getAs("colName4").asInstanceOf[String],
       .
       .
       .
)
}

As of now that's how I'm selecting specific columns from output of multiple DatFrames join and using asInstanceOf[String] as well to get String type.

Is there any other efficient way of selecting specific cols from multiple joins in DataFrame ? And secondly, how can I avoid using asInstanceOf[String] while creating new Instance Output using different columns ?

Please suggest ideas.

Thanks

0 Answers
Related