I have a two pyspark dataframe like below
df1 :-
+---+----------+------+
|age| dob|is_fan|
+---+----------+------+
| 30|1990-05-04| true|
| 26|1994-09-24| false|
+---+----------+------+
df2 :-
+---+----------+------+
|age| dob|is_fan|
+---+----------+------+
| 29|1990-05-03| true|
| 25|1994-09-23| false|
+---+----------+------+
I am doing union operation on this and creating the new dataframe df3.
+---+----------+------+
|age| dob|is_fan|
+---+----------+------+
| 29|1990-05-03| true|
| 25|1994-09-23| false|
| 30|1990-05-04| true|
| 26|1994-09-24| false|
+---+----------+------+
I want to find the size of the df3 dataframe in MB. For single datafrme df1 i have tried below code and look it into Statistics part to find it. But after union there are multiple Statistics parameter.
dd3.createOrReplaceTempView('test')
spark.sql('explain cost select * from test').show(truncate=False)
Is there any other way to find the size of dataframe after union operation?