I have a dataframe which I read in using pyspark with:
df1 = spark.read.csv("/user/me/data/*").toPandas()
Unfortunately, pyspark leaves all the types as Object, even numerical values. I need to merge this with another dataframe I read in with df2 = pd.read_csv("file.csv") so I need the types in df1 to be inferred exactly as pandas would have done it.
How can you infer types of an existing pandas dataframe?