We are building an app on top of Azure HDInsight Spark 3.0 cluster and Delta Lake 0.8.0. I have no problems with creating and accessing delta tables in Jupyter Notebook. However, when I try to connect to the delta tables via Spark SQL using BI tools (Tableau, PowerBI) I get the following error:
An error occurred while communicating with Spark SQL
Unable to connect to the server. Check that the server is running and that you have access privileges to the requested database.
Error Code: B19090E0
[Simba][Hardy] (35) Error from server: error code: '0' error message: 'Error running query: java.lang.ClassNotFoundException: delta.DefaultSource'.
The table "[default].[accounts]" does not exist.`
I can see the tables within the BI tool interface but it fails to load the data. I assume those delta libraries provided for SparkSession are not available when connectors make requests to Spark SQL. Do I need to somehow configure Spark or Spark SQL to work with the delta tables in BI tools?