How can I read headers with .createDataFrame()?

user_5491428 0
Viewed 208

I am reading some data from a Google API. The data being returned looks like so:

print(data)
[['col1', 'col2', 'col3'], ['543', '15197', '49877729'], ['9288', '285985', '29762969'], ['13709', '28146', '94968440'], ['1', '98480', '154381985'], ['586', '137709', '74962864'] ...]

The first row is the header names. I then convert the above to a Spark df like so:

spark_df = (
  spark
  .createDataFrame(data, header=True)
)

Which gives this error:

TypeError: createDataFrame() got an unexpected keyword argument 'header'

How can I solve this? I looked at the accepted answer here How to Set Pyspark Dataframe Headers to another Row? but surely a loop can be avoided for this? The second answer suggests the header argument but it errors for me.

pyspark apache-spark-sql apache-spark python
1 Answers

You can try to use the first list item as the schema (second argument), and use the rest of the list as the data:

df = spark.createDataFrame(data[1:], data[0])
user_141657300
Related