Column( header) sub-string replace in pyspark

user_11358042 0
Viewed 88

  • I want to rename column name by replacing a sub string.

    In below example i have to remove "xy" from each column. Is there any predefined function or do i need to create UDF ?

xy.col1   xy.col2
      2         3
      3         5
      4         4
      5         8
pyspark
1 Answers

You can do this by making a dictionary of original column names and the cleaned column names (here I am splitting on the dot and selecting the right character after split) , also note the backtick when doing so since the columns have a dot in them "`" 

import pyspark.sql.functions as F

d = dict(zip(df.columns,[i.split('.')[-1] for i in df.columns]))
df.select(*[F.col(f"`{k}`").alias(v) for k,v in d.items()]).show()

+----+----+
|col1|col2|
+----+----+
|   2|   3|
|   3|   5|
|   4|   4|
|   5|   8|
+----+----+
user_98406370
Related