How can I replace multiple characters from all columns of a spark dataframe?

Question

I have a dataframe containing multiple columns.

>>> df.take(1)
[Row(A=u'{dt:dt=string, content=Prod}', B=u'{dt:dt=string, content=Staging}')]

I want to remove both curly braces '{' and '}' from values of column A and B of df. I know we can use:

df.withColumn('A',regexp_replace('A','//{',''))
df.withColumn('A',regexp_replace('A','//}',''))
df.withColumn('B',regexp_replace('B','//}',''))

How do I replace characters dynamically for all columns of Spark dataframe? (Pandas version is shown below)

df = df.replace({'{':'','}':''},regex=True)

user_8960822 · Answer

Just use proper regular expression:

df.withColumn("A", regexp_replace("A", "[{}]", ""))

user_3005989 · Answer

To modify a dataframe df and apply regexp_replace to multiple columns given by listOfColumns you could use foldLeft like so:

val newDf = listOfColumns.foldLeft(df)((acc, x) => acc.withColumn(x, regexp_replace(col(x), ..., ...)))