Error running Scala Bulkcopy on Azure Databricks Runtime 7.3 LTS

user_18627728 0
Viewed 107

We have recently upgraded our Azure Databricks runtime from 6.4 to 7.3LTS. And the issue is that my Scala bulk insert for SQL Server MI is not working anymore. The error i am getting is as below

%scala
val bulkCopyConfig = Config(Map(
"url" -> jdbchostURL,
"databaseName" -> jdbcDatabase,
"user" -> jdbcUsername,
"password" -> jdbcPassword,
"dbTable" -> jdbcTableName,
"bulkCopyBatchSize" -> "200000",
"bulkCopyTableLock" -> "true",
"bulkCopyTimeout" -> "600"
))

The error that i am getting is

java.lang.NoClassDefFoundError: scala/Product$class

I have checked GitHub and other sites, no specific answer to the problem is available. Please help

Tried multiple solutions but to no avail, expecting a solution that can fix the above error and enable to run scala 2.11 code on scala 2.12 easily.

databricks scala
1 Answers

this kind of error is usually caused by incompatible connector - make sure that your SQL Server connector is compiled for Scala 2.12 (com.microsoft.azure:spark-mssql-connector_2.12:1.1.0 for 7.3/Spark 3.0 - see compatibility table on Github) - it's impossible to use connector compiled for Scala 2.11 on DBR 7.3.

P.S. It makes sense to go to DBR 10.4 already, as 7.3 will be end of life in ~half year

Alex_Ott_186270
Related