So I'm trying to run an SQL query in databricks. I ran a similar one earlier today that worked with no issues. I took that query, negated one of the lines, and I keep getting this error after about 4 mins of run time: "Error in SQL statement: SparkException: Exception thrown in Future.get"
Here's the line: sgrp_cd not in (7520)
If I change this to sgrp_cd in (7520), it runs. When I say not in, I get the error. I'm baffled. I've run similar queries with not in in the past and it was fine so truly not sure what I'm missing.
I thought maybe it was the syntax so I tried <> and ne and those don't work either (<> gives same error, ne just doesn't work at all)
I can't share any more but it has to be this line because it runs fine without it. Any idea? I can't find much info about this error online.
Thanks.
UPDATE: I had a range join optimization at beginning (that worked fine with the first query!), but in desperation I took that out and now it runs... not as critical but still curious if anybody knows why this weird thing is happening.