Apache Spark Connector for SQL Server and Azure SQL

user_6267265 0
Viewed 959

I'm trying to write data from Azure Databricks to Azure SQL using this connector - com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0, but getting the below error message -

Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, 10.139.64.4, executor 0): java.lang.NoClassDefFoundError: com/microsoft/sqlserver/jdbc/ISQLServerBulkData

Is this spark connector working with Azure Databricks to Azure SQL? have any one tested this?

azure-databricks apache-spark azure
1 Answers

For some reason, Microsoft publishes only jar for connector itself, and it doesn't include JDBC driver that is required for connector. If you build code yourself, build system produces the assembly artifact, but that could be used, but it's isn't published. You can workaround the problem by adding the JDBC driver explicitly (the com.microsoft.sqlserver:mssql-jdbc:8.4.1.jre8 coordinate), like this (for spark-submit/spark-shell/pyspark, etc.):

bin/spark-shell --packages \
  com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0-alpha,com.microsoft.sqlserver:mssql-jdbc:8.4.1.jre8

or adding two libraries via UI

Alex_Ott_186270
Related