How to write an encoding specificed csv file in spark?

user_1649585 0
Viewed 190

I want to write a GBK encoding csv file from a DataFrame. I find the option "encoding" in CSVOptions and it works well when I read a GBK encoding file with option("encoding", "GBK") in local environment. But when I use the write function like below

df.repartition(1).write.option("header", value = true).option("encoding", "GBK").csv("F:\\project\\test1")

The csv file I got is always in UTF8 encoding. I have tried to change the encoding option value to ISO-8859-1 or another encoding format, it doesn't change the output csv file encoding.

apache-spark-sql apache-spark
1 Answers

Even I'm having the issue. I have a csv file which is generated using .net C#, where encoding is mentioned as ISO-8559-1. When I open the file in winmerge, default encoding is showing as Windows-1252. But the csv file which is generated using

pdata.coalesce(1).write().format("csv")
.option("header","true").option("delimiter",";").option("encoding","ISO-8859-1")
.save("target/temp/csv/");

generates a csv file and when I open the file in winmerge it shows UTF-8 encoding

user_28223620
Related