Converting a vector column in a dataframe back into an array column

user_3387134 0
Viewed 13460

I have a dataframe with two columns one of which (called dist) is a dense vector. How can I convert it back into an array column of integers.

+---+-----+
| id| dist| 
+---+-----+
|1.0|[2.0]|
|2.0|[4.0]|
|3.0|[6.0]|
|4.0|[8.0]|
+---+-----+

I tried using several variants of the following udf but it returns a type mismatch error

val toInt4 = udf[Int, Vector]({ (a) => (a)})  

val result = df.withColumn("dist", toDf4(df("dist"))).select("dist")
apache-spark-mllib apache-spark scala
3 Answers

I struggled for a while to get the answer from @ThomasLuechtefeld working. But was running into this very frustrating error:

org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(features_scaled)' due to data type mismatch: argument 1 requires vector type, however, '`features_scaled`' is of vector type.

Turns out I needed to import DenseVector from the ml package instead of the mllib package.

So this worked for me:

import org.apache.spark.ml.linalg.DenseVector
import org.apache.spark.sql.functions._

val vectorToColumn = udf{ (x:DenseVector, index: Int) => x(index) }
myDataframe.withColumn("clusters_scaled",vectorToColumn(col("features_scaled"),lit(0)))

Yes, the only difference is that first line. This should absolutely be a comment, but I don't have the reputation. Sorry!

user_49960210
Related