Run a Regex on a dataframe and store the results in a new dataframe

user_3279893 0
Viewed 67

I have the following dataframe

+----------------------------------
|______value______________________|
| I am going to school         |
| why are you crying         |
| You are not very good my friend |

And I want to filter the rows with emojis and put into a new dataframe. I wrote the following code to convert the dataframe into a list and then iterate through the list to identify sentences with emojis. But I don't know how to apply those regex within dataframe.

Existing code

def convertDataFrameToList(combinedDataFrame : DataFrame) : List[Any] = {
    val myList=   combinedDataFrame.select("value").rdd.map(r => r(0)).collect.toList
    myList
  }
val listOutput = convertDataFrameToList(myDaframe)
for(element<- listOutput) {
 val emojiValues =  raw"\p{block=Emoticons}".r.findAllIn(element).toSeq
         val   y =    raw"\p{block=Miscellaneous Symbols and Pictographs}".r.findAllIn(element).toSeq
         val p =  emojiValues ++ y

//process further
}

UPDATE

I tried the following regex

 val emoticonResult = myKafkaDataFrame.filter(regexp_extract(col("value"), raw"([\p{block=Emoticons},\p{block=Miscellaneous Symbols and Pictographs},\uuD83E\uDD00-\uD83E\uDDFF])", 1) =!= "")

And as a result rows with emojis as well as rows without any emoji is also returned. Can I know whats wrong in my code?

emoji apache-spark-sql apache-spark dataframe scala
1 Answers

You can use regexp_extract with your regex:

val emojis = df.filter(regexp_extract($"value", raw"(\p{block=Emoticons})", 1) =!= "")
val no_emojis = df.filter(regexp_extract($"value", raw"(\p{block=Emoticons})", 1) === "")

emojis.show(false)
+--------------------------+
|value                     |
+--------------------------+
|I am going to school    |
|why are you crying    |
+--------------------------+

no_emojis.show(false)
+-------------------------------+
|value                          |
+-------------------------------+
|You are not very good my friend|
+-------------------------------+
user_141657300
Related