how to remove one single letter from pyspark df column?

Viewed 74

how I can remove all single letter from a column of pyspark dataframe efficiently ?

col1

"a k summer"
"h jkdf"
"jkh polo"

output:

"summer"
"jkdf"
"jkh polo"
2 Answers

Alternatively, you can use regex to replace.

Assuming your dataframe looks like

+------------+
|        text|
+------------+
|an  k summer|
|      h jkdf|
| jkh m  polo|
+------------+
df = df.withColumn('result', F.trim(F.regexp_replace(F.col('text'), r'\s*\b\w{1}\b\s*', ' ')))

Regex explanation

\s*   => 0 or more spaces
\b    => Word boundary 
\w{1} => Any single character

\s*\b\w{1}\b\s*  => Match a single character and surrounding 0 or more spaces

Result

+------------+---------+
|        text|   result|
+------------+---------+
|an  k summer|an summer|
|      h jkdf|     jkdf|
| jkh m  polo| jkh polo|
+------------+---------+

You can split your column by white space, then remove letters and concatenate them back

import pyspark.sql.functions as f


df = spark.createDataFrame([
  ["a k summer"],
  ["h jkdf"],
  ["jkh polo"],
  ["hello world f o ba  "]
], "col1 string")

df = df.withColumn("col2", f.concat_ws(" ", f.expr("FILTER(SPLIT(col1, ' '), value -> LENGTH(value) > 1)")))
df.show()

Output

+--------------------+--------------+
|                col1|          col2|
+--------------------+--------------+
|          a k summer|        summer|
|              h jkdf|          jkdf|
|            jkh polo|      jkh polo|
|hello world f o ba  |hello world ba|
+--------------------+--------------+
Related