how to find size of pyspark dataframe

Viewed 192

I have a two pyspark dataframe like below

df1 :-

 +---+----------+------+
 |age|       dob|is_fan|
 +---+----------+------+
 | 30|1990-05-04|  true|
 | 26|1994-09-24| false|
 +---+----------+------+

df2 :-

  +---+----------+------+
  |age|       dob|is_fan|
  +---+----------+------+
  | 29|1990-05-03|  true|
  | 25|1994-09-23| false|
  +---+----------+------+

I am doing union operation on this and creating the new dataframe df3.

+---+----------+------+
|age|       dob|is_fan|
+---+----------+------+
| 29|1990-05-03|  true|
| 25|1994-09-23| false|
| 30|1990-05-04|  true|
| 26|1994-09-24| false|
+---+----------+------+

I want to find the size of the df3 dataframe in MB. For single datafrme df1 i have tried below code and look it into Statistics part to find it. But after union there are multiple Statistics parameter.

dd3.createOrReplaceTempView('test')
spark.sql('explain cost select * from test').show(truncate=False)

Is there any other way to find the size of dataframe after union operation?

0 Answers
Related