pyspark sql parseExpression with cte results with mismatched input 'AS' expecting {<EOF>, '-'}

Viewed 36

trying to write a unit test that validates spark sql. I'm on python. I tried this:

    spark = SparkSession.builder
            .master('local[1]')
            .appName('local-spark-unittest')
            .config("spark.sql.legacy.createHiveTableByDefault", "false")
            .getOrCreate()
    try:
        parser = spark._jsparkSession.sessionState().sqlParser()
        parser.parseExpression("""WITH t AS (SELECT 1) SELECT * FROM t""")
        print('Validation successful')
    except ParseException as e:
        print('Validation failed')

I get as a result:

        except py4j.protocol.Py4JJavaError as e:
            converted = convert_exception(e.java_exception)
            if not isinstance(converted, UnknownException):
                # Hide where the exception came from that shows a non-Pythonic
                # JVM exception message.
>               raise converted from None
E               pyspark.sql.utils.ParseException: 
E               mismatched input 'AS' expecting {<EOF>, '-'}(line 2, pos 12)

If I run the same with just parser.parseExpression("SELECT 1") Then it passes with no issues.

1 Answers

Adding brackets around the query will make it work:

parser.parseExpression("""(WITH t AS (SELECT 1) SELECT * FROM t)""")
Related