trying to write a unit test that validates spark sql. I'm on python. I tried this:
spark = SparkSession.builder
.master('local[1]')
.appName('local-spark-unittest')
.config("spark.sql.legacy.createHiveTableByDefault", "false")
.getOrCreate()
try:
parser = spark._jsparkSession.sessionState().sqlParser()
parser.parseExpression("""WITH t AS (SELECT 1) SELECT * FROM t""")
print('Validation successful')
except ParseException as e:
print('Validation failed')
I get as a result:
except py4j.protocol.Py4JJavaError as e:
converted = convert_exception(e.java_exception)
if not isinstance(converted, UnknownException):
# Hide where the exception came from that shows a non-Pythonic
# JVM exception message.
> raise converted from None
E pyspark.sql.utils.ParseException:
E mismatched input 'AS' expecting {<EOF>, '-'}(line 2, pos 12)
If I run the same with just parser.parseExpression("SELECT 1") Then it passes with no issues.