Exception "com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor" in scala Application

Question

I made a scala Application for data movement process from Mysql DB to GCS using shadow jar.

Gradle Dependencies used:

 compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.0'
 compile group: 'mysql', name: 'mysql-connector-java', version: '8.0.15'
 compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.0'
 compile group: 'com.google.cloud', name: 'google-cloud-storage', version: '1.110.0'
 compile group: 'com.google.oauth-client', name: 'google-oauth-client', version: '1.30.6'
 compile group: 'com.typesafe', name: 'config', version: '1.0.2'
 compile group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.116.3'
 compile group: 'com.google.api', name: 'gax', version: '0.2.0'
compile group: 'com.microsoft.sqlserver', name: 'mssql-jdbc', version: '6.1.0.jre7'

Java version used: 1.8

Error 1:

Unable to create blob in GCS to move parquet files there with below exception:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;

Resolution1: On research on this error I concluded that it is due to dependencies are getting missed or some dependencies version conflict as I am using shadow jar, and to resolve that I added

--conf spark.driver.userClassPathFirst=true --conf spark.executor.userClassPathFirst=true

in my spark submit cmd.

Error2: Now I was able to create blob in my GCS and error1 was resolved but spark session stopped working and was unable to fetch data from DB and threw below exception:

Exception in thread "main" java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.hive.orc.OrcFileFormat not a subtype

So how can I execute my shadow jar without any errors.

Code snippet for GCS error(Error1):

val Credentials: GoogleCredentials = ServiceAccountCredentials.fromPkcs8(client_id, client_email, private_key, private_key_id, null)
val Storage= StorageOptions.newBuilder().setCredentials(Credentials).setProjectId(strProjectId).build().getService

val blobId: BlobId = BlobId.of(strGcsBucket, s"$Dir")
val blobInfo: BlobInfo = BlobInfo.newBuilder(blobId).build()
Storage.create(blobInfo, Files.readAllBytes(fPath))

Code snippet for Spark session error(Error2):

val Spark: SparkSession = SparkSession.builder()
    .master("local")
    .appName("Migration App")
    .getOrCreate()

user_14013569 · Answer

I got this error resolved by using relocate() in my build.gradle. For reference please find below:

shadowJar {
    zip64 true
    mergeServiceFiles()
    relocate ('com.google.common', 'shaded.com.google.common')
}

This made clear to my fatjar that it has give preferences to the versions of dependencies mentioned shadowjar and not to local platform dependencies.

Exception "com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor" in scala Application

1 Answers