How to handle csv file with multiple delimiters in PySpark and Scala Spark?

user_12771012 0
Viewed 715

I have the following scenario to handle in PySpark.

I have csv data in the following format where delimiter is @|# and the data in name column is enclosed in double quotes. Header is also separated by same delimiter @|#

name@|#age@|#gender
"Name1"@|#34@|#Male
"Name2"@|#60@|#Female

And I need to clean the data and get a dataframe like below

+----------+-----+------------+
| Name     | Age |  Gender    |
+----------+------------------+
| Name1    | 34  |  Male      |
| Name2    | 60  |  Female    |
+----------+-----+------------+

Also, in Scala Spark how can we achieve the same?

pyspark apache-spark
1 Answers

You can read about which options you can set for reading from csv here(python doc) or here(scala doc)

For your case, you need "sep" and "header":

python:


spark.read.options({"sep":"@|#", "header":"true"}).csv("path_to_file")

or scala:


spark.read.options(Map("sep" -> "@|#", "header" -> "true")).csv("file.csv")
user_77798650
Related