split column and get last element spark dataset

user_9845366 0
Viewed 305

I wrote the following code:

def extractGenderFromName(ds: Dataset[Toto])
                          (implicit sparkSession: SparkSession): Dataset[Toto] = {
    if (ds("gender") == null)
      ds.withColumn("gender", split(col(identity), "/")(1))
}

Here I want to get the last part of the split of col identity, I put 1 but it can be 2 or 3. How to dynamically take the last part of col identity?

apache-spark-dataset apache-spark-sql apache-spark dataframe scala
1 Answers

Use element_at function with negative index -1 for last element of an array column. Also, you need to use when expression to check whether the column gender is null, not Scala if-statement:

ds.withColumn(
  "gender",
  when(
    col("gender").isNull, 
    element_at(split(col("identity"), "/"), -1)
  ).otherwise(col("gender"))
)

Note that you can actually use substring_index function without having to split the column identity like this:

substring_index(col("identity"), "/", -1)
user_13865510
Related