I'm trying to setup a Pandas Dataframe for processing my data in Databricks. My data is imported from a file on my local machine as seen below: Snip of the data
# Import packages
import pandas as pd
import numpy as np
ownr = spark.read.format("csv").load("dbfs:/FileStore/shared_uploads/directory/carsownr.csv")
# View the shape and data types
#print(ownr.shape)
print(ownr.dtypes);
#Setup Dataframes
df1 = pd.DataFrame(data=ownr);
And when I run this, I get the following error message:
ValueError: DataFrame constructor not properly called!
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<command-3703442649830271> in <module>
1 #Setup Dataframes
----> 2 df1 = pd.DataFrame(data=ownr);
3 df2 = pd.DataFrame(data=curr);
/databricks/python/lib/python3.7/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
507 )
508 else:
--> 509 raise ValueError("DataFrame constructor not properly called!")?
510
511 NDFrame.__init__(self, mgr, fastpath=True)
ValueError: DataFrame constructor not properly called!
I thought it may be because of my data, so I checked and all the fields are strings. Will I need to convert each column/field to the appropriate datatype for it to successfully be added in the dataframe?
I've searched the forums for similar error messages, but couldn't find a relevant one. I'd appreciate any help possible, thanks!