ValueError: DataFrame constructor not properly called (Databricks/Python)

Viewed 1129

I'm trying to setup a Pandas Dataframe for processing my data in Databricks. My data is imported from a file on my local machine as seen below: Snip of the data

# Import packages
import pandas as pd
import numpy as np

ownr = spark.read.format("csv").load("dbfs:/FileStore/shared_uploads/directory/carsownr.csv")

# View the shape and data types
#print(ownr.shape)
print(ownr.dtypes);

#Setup Dataframes
df1 = pd.DataFrame(data=ownr);

And when I run this, I get the following error message:

ValueError: DataFrame constructor not properly called!
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<command-3703442649830271> in <module>
      1 #Setup Dataframes
----> 2 df1 = pd.DataFrame(data=ownr);
      3 df2 = pd.DataFrame(data=curr);

/databricks/python/lib/python3.7/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    507                 )
    508             else:
--> 509                 raise ValueError("DataFrame constructor not properly called!")?
    510 
    511         NDFrame.__init__(self, mgr, fastpath=True)

ValueError: DataFrame constructor not properly called!

I thought it may be because of my data, so I checked and all the fields are strings. Will I need to convert each column/field to the appropriate datatype for it to successfully be added in the dataframe?

I've searched the forums for similar error messages, but couldn't find a relevant one. I'd appreciate any help possible, thanks!

1 Answers

If you want to get Pandas Dataframe from the Spark Dataframe, then you need to use toPandas function instead:

df1 = ownr.toPandas()

Please also note that size of the Spark Dataframe should be small enough to fit into driver's memory.

Also, if you're looking for Pandas API for Spark, then look onto Koalas library (Spark < 3.2, DBR < 10.0) that is also included into upcoming Spark 3.2 under name Pandas API for Spark (DBR 10+) - in this case you'll get benefits of both Spark & Pandas

Related