Remove strange character from tokenization array

user_16788296 0
Viewed 56

I have a very dirty Pyspark dataframe, i.e. full of weird characters like:
  • ɴɪᴄᴇ ᴏɴᴇ ᴀᴩᴩ
  • பரமசிவம்
  • and many others

I'm doing the data processing and cleaning (tokenization, stopword removal, ...) and this is my dataframe:

content score label classWeigth words filtered terms_stemmed
absolutely love d... 5 1 0.48 [absolutely, love... [absolutely, love... [absolut, love, d...
absolutely love t... 5 1 0.48 [absolutely, love... [absolutely, love... [absolut, love, g...
absolutely phenom... 5 1 0.48 [absolutely, phen... [absolutely, phen... [absolut, phenome...
absolutely shocki... 1 0 0.52 [absolutely, shoc... [absolutely, shoc... [absolut, shock, ...
accept the phone ... 1 0 0.52 [accept, the, pho... [accept, phone, n... [accept, phone, n...

How can I access the word column and remove all weird characters, like the ones mentioned above?

pyspark data-cleaning machine-learning python
1 Answers

Try this UDF.

>>> @udf('array<string>')
... def filter_udf(a):
...     from builtins import filter
...     return list(filter(lambda s: s.isascii(), a))
... 

>>> df = spark.createDataFrame([(['pyspark','பரமசிவம்'],)])
>>> df.select(filter_udf('_1')).show()
+--------------+
|filter_udf(_1)|
+--------------+
|     [pyspark]|
+--------------+
user_87733090
Related