I am looking for advice to resolve a TypeError when operating on the entire PySpark DataFrame.
I have a PySpark DataFrame with the following schema and I want to apply further operations like count() or show() to the DataFrame then convert the Spark DataFrame to a Pandas DataFrame. As you can see below, the error being returned when performing .count() on the PySpark DataFrame is as follows
TypeError: element in array element in array element in array field prediction: ArrayType(FloatType,true) can not accept object -1.2425838708877563 in type <class 'float'>
I am able to successfully display the first row from the Pandas DataFrame, using .head(), after converting the PySpark DataFrame to Pandas DataFrame.
