I have data contains column A
A
107/108
105
103
103/104
Output should be like:-
105
103
I have tried lot with filter function in pyspark and also in pysql even but code doesn't work
remove rows which contain special character like '/' in pyspark or pysql
Viewed 1062
1 Answers
You can use either rlike,like,contains functions with negation (~)
df=spark.createDataFrame([('107/108',),('105',),('103',),('103/104',)],['A'])
df.show()
#+-------+
#| A|
#+-------+
#|107/108|
#| 105|
#| 103|
#|103/104|
#+-------+
from pyspark.sql.functions import *
#using rlike function
df.filter(~col("A").rlike("\/")).show()
#using like function
df.filter(~col("A").like("%/%")).show()
#using contains function
df.filter(~col("A").contains("/")).show()
#+---+
#| A|
#+---+
#|105|
#|103|
#+---+
UPDATE:
df=spark.createDataFrame([('107/108',),('105',),('103',),('103/104',),('',)],['A'])
df.show()
#+-------+
#| A|
#+-------+
#|107/108|
#| 105|
#| 103|
#|103/104|
#| |
#+-------+
df.filter(~col("A").rlike("\/")).show()
df.filter(~col("A").like("%/%")).show()
df.filter(~col("A").contains("/")).show()
#+---+
#| A|
#+---+
#|105|
#|103|
#| |
#+---+
Related