Spark Scala How to replace spacial character from beginning of column name

user_6508163 0
Viewed 45

I have a data frame df

df = source_df.select("data.*").distinct()

df.show(20,false)

---------------+----------------+--------------------------+
id             |_name           |_address  |area           |
--------------------------+----------------+---------------+
222222222222444|null            |india     |Default Value  |
000000000055555|null            |usa       |Default Value  |
+--------------+----------------+--------------------------+

I want to replace spacial character(_) from beginning of column name , and the desire output will be

---------------+----------------+--------------------------+
id             |name            |address   |area           |
--------------------------+----------------+---------------+
222222222222444|null            |india     |Default Value  |
000000000055555|null            |usa       |Default Value  |
+--------------+----------------+--------------------------+

i have tried with below statement but count not found desire output

df=df.columns.foldLeft(df){(newdf, colname) =>newdf.withColumnRenamed(colname, colname.replace("^_", ""))}

apache-spark scala
1 Answers

You can just do a rename via a select

val newCols = df.columns.map(c => col(c) as (if (c.startsWith("_")) c.drop(1) else c))
val newDf = df.select(newCols:_*)

This will rename columns starting with _ and leave columns that don't. Column order will be maintained as well.

Alternative syntax for the if condition is:

val newCols = df.columns.map(c => if (c.startsWith("_")) col(c) as c.drop(1) else col(c))
val newDf = df.select(newCols:_*)

Just a matter of taste there.

user_20159040
Related