How to implement String contains with equals ignore case to filter content of a dataframe?

user_7119501 0
Viewed 127

I have a json file in the below format:

{Continent: Asia, Description: Biggest continent in the WORLD},
{Continent: Europe, Description: Coldest countries in the world}
{Continent: Africa, Description: Second continent in the WorLD}
{Continent: Australia, Description: The only country & continent in the world}

I am trying to filter the rows of the dataframe that contains the string world in it. I wrote the following code for it.

val continents = spark.read.json("path/to/input.json")

To filter the rows containing the word world in the column Description

continents.filter($"Description".contains("world"))

The above line only filters the rows only if world is completely in lower case.

Is there any way I can apply a filter where case is ignored ?

apache-spark scala
2 Answers

For comparison, you can convert the whole description to upper or lower case, in the following case I have converted the description to lower case for comparison.

import org.apache.spark.sql.functions.lower


continents.filter(lower($"Description").contains("world"))
user_18793880

Use with column to change the description to lowercase and add more complex functions. Filter out on new column

from pyspark.sql.functions import lower, col

continents.withColumn('lower_desc', lower(col('Description'))
continents.filter($"lower_desc".contains("world"))
user_78823670
Related