Catch exceptions for executed sql in spark

user_1733834 0
Viewed 1848

I have an SQL built dynamically and it has some corrupted strings in it like wrong column name or used sm instead of sum.

When I execute the sparksession.sql() on this generated string , the entire spark job fails as expected and I can see all the logs as in the resource manager logs.

Is there a way for me to catch this exception in a try catch block or so and handle this exception and move forward with my logic.

apache-spark-sql apache-spark hadoop java
1 Answers

For now this worked for me , but I dont think this is the ideal solution.

        try {
            return spark.sql(query);
        }catch (Exception e) {
      e.getMessage();
}

This threw me a org.apache.spark.sql.AnalysisException exception for all sql syntax , wrong column or tablename queries. But I couldnt catch this exception in particular though.

user_17338340
Related