pyspark only escapes the first quote after a backslash from csv

user_4255302 0
Viewed 146

I have a csv data in which I've escaped existing backslash characters using another backslash:

year,content
2021,"\\"foo\\",bar"

I'd like to read it using spark and display the data. The expected data on the dataframe:

+----------+--------------+
| year     | content      |
+----------+--------------+
| 2021     | \"foo\",bar  |
+----------+--------------+

But when I ran this:

schema = StructType(
        [
            StructField("year", IntegerType(), False),
            StructField("content", StringType(), False),
        ]
   )

    df = (
        spark.read.csv(
            f"s3://path/to/csv",
            schema=schema,
            header=True
        )
    )

    df.show(20,False)

I'm getting this:

+----------+--------------+
| year     | content      |
+----------+--------------+
| 2021     | "\"foo\\"    |
+----------+--------------+

Any idea how to handle this properly?

pyspark apache-spark
1 Answers

If you can save your data like this (single escape instead of double escape)

year,content
2021,"\"foo\", bar"

then you can read it as you're doing and the output would be

# +----+----------+
# |year|content   |
# +----+----------+
# |2021|"foo", bar|
# +----+----------+
user_34415100
Related