Is it possible in Pyspark to get the csv representation of a dataframe as a string?

user_15691448 0
Viewed 53

I'm trying to get the same result as a pandas to_csv called without a path argument. Currently I'm saving the dataframe as a csv to then read it and I'd like to avoid this step.

path_or_buf: str or file handle, default None File path or object, if None is provided the result is returned as a string. If a non-binary file object is passed, it should be opened with newline=’’, disabling universal newlines. If a binary file object is passed, mode might need to contain a ‘b’.

Having a big dataset, the toPandas function doesn't work. Does someone know if it's possible in pyspark or know a work around ?

pyspark apache-spark
2 Answers

You can use to_csv:

csv_string = df.agg(F.concat_ws('\n', F.collect_list(F.to_csv(F.struct(df.columns))))).head()[0]
user_141657300

You can just use to_csv to convert list of columns to csv as below

from pyspark.sql import functions as f

df.select(f.to_csv(f.struct(df.columns))).show(truncate=False)
user_65514260
Related