How to update Spark Column using an array of values in java

vs777 0
Viewed 43

I have a Dataset loaded in the memory and for every row I need to updated the value of the particular column.

I Iterate thru every row and make an API call( passing row values as parameters ) that returns me different value for every single row. I collect these values into the ArrayList.

I tried to use :

List<Object> newValues = new ArrayList<>();
// populate the list with values
dataframe.withColumn("c1",functions.lit(newValues));

but I am getting

SparkRunTimeException : feature is not supported.

In essence what I need is to create a new Column that will contain all the values returned by my API call and then replace an existing column.

dataset java apache-spark
1 Answers

Lit internally map the provided object type to its equivalent type. Since the input has type Array[Object] with component type as Object. Spark might not be able to map the vague object to any specific spark type. Please convert the Arraylist of Object to arraylist of String or some specific type as below-

List<String> newValues = new ArrayList<>();
// populate the list with values
Dataset<Row> d1 = dataframe.withColumn("c1", lit(newValues.toArray(new String[0])));
d1.show();
userP0Yj0
Related