Better Alternatives to EXCEPT Spark Scala

user_7712680 0
Viewed 369

I have been told that EXCEPT is a very costly operation and one should always try to avoid using EXCEPT. My Use Case -

val myFilter = "rollNo='11' AND class='10'"
val rawDataDf = spark.table(<table_name>)
val myFilteredDataframe = rawDataDf.where(myFilter)
val allOthersDataframe = rawDataDf.except(myFilteredDataframe)

But I am confused, in such use case , what are my alternatives ?

apache-spark scala
1 Answers

Use left anti join as below-

 val df = spark.range(2).withColumn("name", lit("foo"))
    df.show(false)
    df.printSchema()
    /**
      * +---+----+
      * |id |name|
      * +---+----+
      * |0  |foo |
      * |1  |foo |
      * +---+----+
      *
      * root
      * |-- id: long (nullable = false)
      * |-- name: string (nullable = false)
      */
    val df2 = df.filter("id=0")
    df.join(df2, df.columns.toSeq, "leftanti")
      .show(false)

    /**
      * +---+----+
      * |id |name|
      * +---+----+
      * |1  |foo |
      * +---+----+
      */
user_47588230
Related