How to parse CSV file with UTF-8 encoding?

user_936786 0
Viewed 49482

I use Spark 2.1.

input csv file contains unicode characters like shown below

unicode-input-csv

While parsing this csv file, the output is shown like below

unicode-output-csv

I use MS Excel 2010 to view files.

The Java code used is

@Test
public void TestCSV() throws IOException {
    String inputPath = "/user/jpattnaik/1945/unicode.csv";
    String outputPath = "file:\\C:\\Users\\jpattnaik\\ubuntu-bkp\\backup\\bug-fixing\\1945\\output-csv";
    getSparkSession()
      .read()
      .option("inferSchema", "true")
      .option("header", "true")
      .option("encoding", "UTF-8")
      .csv(inputPath)
      .write()
      .option("header", "true")
      .option("encoding", "UTF-8")
      .mode(SaveMode.Overwrite)
      .csv(outputPath);
}

How can I get the output same as input?

apache-spark csv unicode
4 Answers

I was able to read ISO-8859-1 using spark but when I store the same data to S3/hdfs back and read it, the format is converting to UTF-8.

ex: é to é

val df = spark.read.format("csv").option("delimiter", ",").option("ESCAPE quote", '"'). option("header",true).option("encoding", "ISO-8859-1").load("s3://bucket/folder")
user_88264220

.option('encoding', 'ISO-8859-1') worked for me. Acute, caret, cedilla accents among others appeared correctly.

user_137277290

Same problem, I solved encoding with "UTF-8"

  input_df = (spark.read
              .option("sep", sep)
              .option("header", "true")
              .option("encoding", "UTF-8")
              .csv(my_path)
              )
user_38442700
Related