Read a path from a dataframe column and add another column from the dataframe

user_6432418 0
Viewed 75

I have a dataframe as below

id, path
id1, path1
id2, path2
id3, path3

I want to read parquet files in the paths mentioned above and add id column after reading the data to its respective outputs and in the end union all the results

Code:

case class cls_lyr(id: String, path: String)
val selColDf = df.select("id", "path").dropDuplicates
val newdf = selColDf.as[cls_lyr].take(selColDf.count.toInt).foreach(t => {
                      var id = t.id
                      var path= t.path
                      val lkpDf = spark.read.parquet(path)
                      val finalDf = lkpDf.withColumn("portf_id", lit(id))
                      }
                  )

How would I union the data from the 3 paths? Is there any other better way this can be done?

apache-spark scala
1 Answers

You can use map instead of foreach, so that you will get an array of dataframes, which you can reduce to a single dataframe using unionAll:

selColDf.as[cls_lyr]
    .collect
    .map(t => spark.read.parquet(t.path).withColumn("portf_id", lit(t.id)))
    .reduce(_ unionAll _)
user_141657300
Related