Transform the header and the column data separated by "|" into separate columns with certain header name and data rows in pyspark

user_14651153 0
Viewed 148

I have read a txt file in pyspark used for a homework with operator "|" separating 3 columns of data. How can I separate these 3 columns in a proper array? This is what I have achieved so far:

+---------------------+
|bid|gender|department|
+---------------------+
|      1|M|Informatics|
|              2|M|Low|
+---------------------+

As you see the header is created but not separated as columns followed by the data.

pyspark apache-spark
1 Answers

Try by specifying delimiter option as | and read as .csv file in pyspark

Example:

df=spark.read.option("header","true").option("delimiter","|").csv("<path_to_csv_file>")
df.show()
#+---+------+-----------+
#|bid|gender| department|
#+---+------+-----------+
#|  1|     M|Informatics|
#|  2|     M|        Low|
#+---+------+-----------+

UPDATE:

#Save dataframe as file
df.write.mode("either overwrite/append ").option("header","true").save("location")

#writing to hive table
df.write.saveAsTable("<db>.<table_name>")
user_76326950
Related