spark dataframe overwrite creates duplicates

user_17014228 0
Viewed 283

I have a groupby function in df as follows:

val df2 = df1.groupBy($"word").agg(sum($"word_num") as "cnt")

df2.write.format("csv")
  .mode(SaveMode.Overwrite)
  .option("header", "true").option("delimiter", "\t")
  .save(outputPath)

After save the df2, when i reload it from hdfs:

val pFeatureDF = spark.read.format("csv")
  .option("header","true")
  .option("delimiter","\t")
  .load(outputPath)

There exists duplicate key word in df2 like this:

word cnt
word1 10990
word1 11
word2 20980
word2 10
apache-spark dataframe duplicates scala
1 Answers

Since groupBy can't return duplicates for the aggregated keys by definition, it's probably due to spaces in the values of the column word.

You can remove the spaces from both sides of the value by using the function trim:

import org.apache.spark.sql.functions.{sum, trim}

val df2 = df1.groupBy(trim($"word")).agg(sum($"word_num") as "cnt")
user_124002140
Related