How can I resolve the "need struct type but got struct"

user_12125142 0
Viewed 5103

enter image description here

As you can see on my picture, I have a column named probability and I want to create a new column from the probability column. I want to extract values from the probability column which is an array. But while trying to do so, I receive an error:

"Can't extract value from probability#52427: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>"

Here is my extraction code:

preds_test = preds.withColumn("newCol", col("probability").getItem(3))

Can someone please tell me what I did wrong?

pyspark-dataframes pyspark apache-spark scala
1 Answers

I figured it out. I used a lambda function. This is my code:

preds_subset = preds.select('CustomerID','prediction', probs_churn('probability')).orderBy(asc("probability"))```
user_121251420
Related