How to Replace empty string with N/A in Scala Spark?

user_5536733 0
Viewed 344

I'm trying out an age old problem of replacing empty strings in a certain column in a Spark Scala dataframe with N/A, but to no avail.

Original Dataframe:

+----------+--------------+
|Testing ID|Test this Code|
+----------+--------------+
|    545242|            ""|
|    643533|          994A|
|    856563|            ""|
+----------+--------------+

First code I tried:

val a = sssd.withColumn("Test this Code", when($"Test this Code" === "", lit("N/A")).otherwise($"Test this Code"))

But nothing happens, zero changes observed. Hence I tried another way by using regexp_replace, code:

import org.apache.spark.sql.functions._
val a = sssd.withColumn("Test this Code", regexp_replace(col("Test this Code"), "", "N/A"))

But then, the output is strange enough, its the following:

+----------+------------------------+
|Testing ID|          Test this Code|
+----------+------------------------+
|    545242|             N/A"N/A"N/A|
|    643533|                    994A|
|    856563|             N/A"N/A"N/A|
+----------+------------------------+

I went through other SO answers, but to no avail, any help?

apache-spark-sql apache-spark scala regex
1 Answers

Try this. I suspect it's not an empty string, but actually a string of two quotes.

val a = sssd.withColumn("Test this Code", when($"Test this Code" === "\"\"", lit("N/A")).otherwise($"Test this Code"))
user_141657300
Related