How to apply a function to every string in a dataframe

user_4805561 0
Viewed 53
    {
    "cars": {
        "Nissan": {
            "Sentra": {"doors":4, "transmission":"automatic"},
            "Maxima": {"doors":4, "transmission":"automatic","colors":["b#lack","pin###k"]}
        },
        "Ford": {
            "Taurus": {"doors":4, "transmission":"automatic"},
            "Escort": {"doors":4, "transmission":"auto#matic"}
        }
    }
}

I have this JSON that I have read, and I want to remove every # symbol in every string that may exist. My problem is doing this function generic, so it could work on every schema that I may encounter and not only this schema as used in JSON above.

apache-spark-sql apache-spark scala
1 Answers

You could do something like this: Get all the fields from the schema, use fold with the DataFrame itself as an accumulator and, apply the function that you want

  def replaceSymbol(df: DataFrame): DataFrame =
    df.schema.fieldNames.foldLeft(df)((df, field) => df.withColumn(field, regexp_replace(col(field), "#", "")))

You might need to check if the column is String or not.

user_70418710
Related