While working with spark and snowflake connctivity I see the following line in my log:
SnowflakeStrategy: Pushdown failed: SQL compilation error
I looked at the generated SQL statement and it shows group by clause in it. Whereas my actual SQL statement in code does not show any group by statement. Additionally, my code was not failed at this line.
Why this message and is there really some impact on output? I don't see my output affected!
The Spark connector attempts to take any Spark SQL that you've written and translate it into SQL so that the logic can be pushed into Snowflake and executed closer to the data. But it has a fall-back mechanism: if it can't do the pushdown because of some unsupported functionality, or if Snowflake gives an error on the original query, it will fall-back to a simpler—and possibly more expensive—plan.
The error you're seeing is part of that fall-back: it probably got an error from Snowflake and fell back to a simpler plan. Your code isn't failing because the query is being issued and answered, just not with the pushdown the connector initially tried.
With the Spark SQL, the query history in Snowflake, and your software version information, we can probably tell you more.
