Case sensitive join in Spark

user_2819400 0
Viewed 305

I am dealing with a scenario in which I need to write a case sensitive join condition. For that, I found there is a spark config property spark.sql.caseSensitive that can be altered. However, there is no impact on the final result set if I set this property to True or False. In both ways, I am not getting results for language=java from the below sample PySpark code. Can anyone please help with how to handle this scenario?

spark.conf.set("spark.sql.caseSensitive", False)

columns1 = ["language","users_count"]
data1 = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]

columns2 = ["language","note"]
data2 = [("java", "JVM based"), ("Python", "Indentation is imp"), ("Scala", "Derived from Java")]

df1 = spark.createDataFrame(data1, columns1)
df2 = spark.createDataFrame(data2, columns2)

#df1.createOrReplaceTempView("df1")
#df2.createOrReplaceTempView("df2")

df = df1.join(df2, on="language", how="inner")
display(df)
pyspark databricks apache-spark
1 Answers

  1. My understanding of spark.sql.caseSensitive is that it affects SQL, not the data.

  2. As for your join itself, if you do not want to lowercase or uppercase your data, which I can understand why, you can create a key column, which is the lowercase version of the value you want to join on. If you are having more complex situation, your key column could even become a md5() of one/more columns. Make sure everything stays lowercase/uppercase though to make the comparison works.

user_31972440
Related