If a dataframe is looks like this df = df.withColumn('NUM_COL', lit('Hey$$$ Hey$ T$$')) and when I need to make this string as
Hey$ Hey$ T$. I couldn't find any proper solutions for this. For an instance in IBM datastage there is a way to do this by using Trim(mylink.mystring,".") for remove redundant chars. What would be the best solution for this in PySpark?
remove multiple occurred chars from a string except one char in pyspark
Viewed 46
1 Answers
regexp_replace should do the job:
df = df.withColumn('replaced', F.regexp_replace('value', r'(.)\1+', '$1'))
The idea is here to use the (first) group from the pattern as replacement.
Related