how I can remove all single letter from a column of pyspark dataframe efficiently ?
col1
"a k summer"
"h jkdf"
"jkh polo"
output:
"summer"
"jkdf"
"jkh polo"
how I can remove all single letter from a column of pyspark dataframe efficiently ?
col1
"a k summer"
"h jkdf"
"jkh polo"
output:
"summer"
"jkdf"
"jkh polo"
Alternatively, you can use regex to replace.
Assuming your dataframe looks like
+------------+
| text|
+------------+
|an k summer|
| h jkdf|
| jkh m polo|
+------------+
df = df.withColumn('result', F.trim(F.regexp_replace(F.col('text'), r'\s*\b\w{1}\b\s*', ' ')))
Regex explanation
\s* => 0 or more spaces
\b => Word boundary
\w{1} => Any single character
\s*\b\w{1}\b\s* => Match a single character and surrounding 0 or more spaces
Result
+------------+---------+
| text| result|
+------------+---------+
|an k summer|an summer|
| h jkdf| jkdf|
| jkh m polo| jkh polo|
+------------+---------+
You can split your column by white space, then remove letters and concatenate them back
import pyspark.sql.functions as f
df = spark.createDataFrame([
["a k summer"],
["h jkdf"],
["jkh polo"],
["hello world f o ba "]
], "col1 string")
df = df.withColumn("col2", f.concat_ws(" ", f.expr("FILTER(SPLIT(col1, ' '), value -> LENGTH(value) > 1)")))
df.show()
Output
+--------------------+--------------+
| col1| col2|
+--------------------+--------------+
| a k summer| summer|
| h jkdf| jkdf|
| jkh polo| jkh polo|
|hello world f o ba |hello world ba|
+--------------------+--------------+