A third-party application connects to a databricks general-purpose cluster and fires some SQL queries. All worked fine on databricks runtime 7.3LTS, but when we upgraded the cluster to runtime 9.1LTS, the where clause in the query suddenly did not contain quotes around the string value.
This is the incoming query with the 9.1LTS runtime:
22/01/26 08:16:52 INFO SparkExecuteStatementOperation: Submitting query 'SELECT * FROM (SELECT `TimeColumn`,`ValueColumn` FROM `database`.`table1` WHERE `TimeColumn` >= { ts '2022-12-26 09:14:55' } AND `TimeColumn` <= { ts '2022-01-26 09:14:55' } AND **`CounterName`=28STO0004** ORDER BY `TagTimeStamp`) LIMIT_ZERO LIMIT 0'
This is the incoming query with the 7.3LTS runtime:
22/01/26 08:28:48 INFO SparkExecuteStatementOperation: Submitting query 'SELECT * FROM (SELECT C_79736572615f79736572615f74616774696d6576616c7565.`TimeColumn` AS C_0, C_79736572615f79736572615f74616774696d6576616c7565.`ValueColumn` AS C_43 FROM `database`.`table1` C_79736572615f79736572615f74616774696d6576616c7565 WHERE (**(C_79736572615f79736572615f74616774696d6576616c7565.`CounterName` = '28STO0004')** AND (C_79736572615f79736572615f74616774696d6576616c7565.`TimeColumn` >= TIMESTAMP '2022-12-26 09:14:55') AND (C_79736572615f79736572615f74616774696d6576616c7565.`TimeColumn` <= TIMESTAMP '2022-01-26 09:14:55')) ORDER BY C_0 ASC ) LIMIT_ZERO LIMIT 0'
This image is the error we receive in the third-party application.
No changes were made to the third-party application. We also have no configuration options on the application, nor query diagnostics.
When we set up a databricks SQL-endpoint, we received the same error as the databricks runtime 9.1LTS cluster.