Apply java funciton URLDecoder.decode to whole column in Spark 3

user_1195001 0
Viewed 1056

I have a dataframe column containing url encoded string such as:

I would like to do something like that:

someDF.withColumn('newcol', URLDecoder.decode( col("mystring"), "utf-8" ))
someDF.show()
|         mystring         |         newcol      |
--------------------------------------------------
| ThisIs%201rstString      | ThisIs 1rstString   |        
| This%20is%3Ethisone      | This is>thisone     |
| and%20so%20one           | and so one          |

How should I do such thing I guess map function is around the corner but can't firgure out how to us it.

Note: this is a sample and it is not an option to create multiple replace statement as there is many other encoded characters and list may vary, I'd like to use a simple reliable method to do so.

apache-spark scala java
2 Answers

You can try the SparkSQL builtin function reflect:

reflect(class, method[, arg1[, arg2 ..]]) - Calls a method with reflection.

df = spark.createDataFrame([(e,) for e in ["ThisIs%201rstString", "This%20is%3Ethisone", "and%20so%20one"]], ["mystring"])

df.selectExpr("*", "reflect('java.net.URLDecoder','decode', mystring, 'utf-8') as newcol").show()

+-------------------+-----------------+
|           mystring|           newcol|
+-------------------+-----------------+
|ThisIs%201rstString|ThisIs 1rstString|
|This%20is%3Ethisone|  This is>thisone|
|     and%20so%20one|       and so one|
+-------------------+-----------------+

Note: the above is Python code, you should be able to do the same with Scala.

user_95107290

Create a UDF that performs the work

import java.net.URLDecoder
def decode(in:String) =  URLDecoder.decode(in, "utf-8")
val decode_udf = udf(decode(_))
df.withColumn("newcol", decode_udf('mystring)).show()

prints the expected result.

user_21298010
Related