Suppose I have two DataFrames and do something like
df1.except(df2). If one of the columns in the DataFrame is a float/double, what tolerance (if any) will be used for equality testing by default?
My observation is that if I do something like
val df1 = spark.sql("select cast(1.0001 as float) as foo")
val df2 = spark.sql("select cast(1.0002 - 0.0001 as float) as foo")
val df3 = spark.sql("select cast(1.0002 as float) as foo")
then df1.except(df2) and df2.except(df3) act as might be naively expected.
Note that I'm not asking for how to fix or avoid any problems this might create; I'm only asking for what will happen, if you do not take steps to acknowledge the issues with floating point by nature being inexact.