How to Create Empty Spark DataFrame in PySpark and Append Data?

user_11303509 0
Viewed 5238

I have a task of combining multiple Spark DataFrames generated from a for loop together. So I thought to create an empty DataFrame before running the for loop and then combine them by UnionAll. result is the name of data frames generated from for loop.

Below is the code:

empty = sqlContext.createDataFrame(sc.emptyRDD(), StructType([]))
empty = empty.unionAll(result)

Below is the error:

first table has 0 columns and the second table has 25 columns

Looks like I have to specify specific schema when creating the empty Spark DataFrame. I am wondering if there is a way to make it work without doing so or simply combine Spark DataFrames. I do have many columns to specify otherwise.

Thanks in advance!

pyspark apache-spark-sql python
1 Answers

As your empty dataframe doesn't has any columns so when we do unionAll we need to have same number of columns.

Try by creating empty dataframe with result dataframe schema then do unionAll.

Example:

result=spark.createDataFrame([(1,2,3,4)],['id','a','b','c'])
empty = sqlContext.createDataFrame(sc.emptyRDD(), result.schema)
empty.unionAll(result).show()
#+---+---+---+---+
#| id|  a|  b|  c|
#+---+---+---+---+
#|  1|  2|  3|  4|
#+---+---+---+---+
user_76326950
Related