Spark-Scala Malformed Line Issue

Question

I have a control-A delimited file which I am trying to convert to parquet format. However in the file there is a String field with a single " in it.

Reading the data like below:

val dataframe = sparkSession.sqlContext.read
                .format("com.databricks.spark.csv")
                .option("delimiter", datasetDelimiter)
                .option("header", "false")
                .option("mode","FAILFAST")
                //.option("mode", "DROPMALFORMED")
                .option("treatEmptyValuesAsNulls","true")
                .option("nullValue"," ")
                .option("ignoreLeadingWhiteSpace", "true")
                .option("ignoreTrailingWhiteSpace", "true")
                .schema(schema)
                .load(fileLocation)
                dataframe

As you can see there is just an open double quote in the data and no closed double quote. This is resulting in Malformed Line exception. While reading I have explicitly mention the delimiter as U0001. Is there any way to convert such data to parquet without losing any data

Spark-Scala Malformed Line Issue

1 Answers