I have a dataframe format of
id|name|1|[aaa]|2|[bbb]
Need to write csv file in format
id,name
1,""[""aaa""]""
2,""[""bbb""]""
Not getting correct format. Could you please help. Thanks in advance.
My code like
data=[{"id":1,"name":["aaa"]},{"id":2,"name":["bbb"]}]
df_t=spark.createDataFrame(data)
df_t=df_t.withColumn("name",concat(lit('"'), col('name').cast(StringType()),lit('"')))
df_t.show()
members_df1.coalesce(1).write.mode("overwrite").options(header="True",
delimiter=",",escape="\"" ).csv("s3://bucket/folder")