How can you update values in a dataset?

user_5728737 0
Viewed 8917

So as far as I know Apache Spark doesn't has a functionality that imitates the update SQL command. Like, I can change a single value in a column given a certain condition. The only way around that is to use the following command I was instructed to use (here in Stackoverflow): withColumn(columnName, where('condition', value));

However, the condition should be of column type, meaning I have to use the built in column filtering functions apache has (equalTo, isin, lt, gt, etc). Is there a way I can instead use an SQL statement instead of those built in functions?

The problem is I'm given a text file with SQL statements, like WHERE ID > 5 or WHERE AGE != 50, etc. Then I have to label values based on those conditions, and I thought of following the withColumn() approach but I can't plug-in an SQL statement in that function. Any idea of how I can go around this?

apache-spark-sql apache-spark
4 Answers

If you are using java dataset you can update dataset by below. here is the code

Dataset ratesFinal1 = ratesFinal.filter(" on_behalf_of_comp_id != 'COMM_DERIVS' ");

ratesFinal1 = ratesFinal1.filter(" status != 'Hit/Lift' ");

Dataset ratesFinalSwap = ratesFinal1.filter (" on_behalf_of_comp_id in ('SAPPHIRE','BOND') and cash_derivative != 'cash'");

ratesFinalSwap = ratesFinalSwap.withColumn("ins_type_str",functions.lit("SWAP"));

adding new column with value from existing column

ratesFinalSTW = ratesFinalSTW.withColumn("action", ratesFinalSTW.col("status"));

user_71968980
Related