I need to import a CSV file that contains several fields, I must later loop on some fields that interest us to recover the data contained in it.
In the file there is a field named query that contains SQL queries that must be executed and store in another CSV file that will contain the fields to retrieve as well as the results of each query.
Below is my code so far:
// step1:read the file
val table_requete = spark.read.format("com.databricks.spark.csv").option("header","true").option("delimiter", ";").load("/user/swychowski/ClientAnlytics_Controle/00_Params/filtre.csv")
req.registerTempTable("req")
// step2:read the file
However, I dont know how to loop and store on another file at the same time.