Exception while using google-cloud library to execute BigQuery queries

user_1511294 0
Viewed 237

I am trying to run a truncate statement from a spark code(from dataproc cluster) using the google-cloud-bigquery library .

1)This is the code

import com.google.cloud.bigquery.{BigQueryException, BigQueryOptions, QueryJobConfiguration}
import com.google.cloud.http.BaseHttpServiceException
object BigQueryClient {

  def truncateBigQueryTable(tableName: String): Unit = {
    val bigquery = BigQueryOptions.getDefaultInstance.getService
    try {
      val truncQuery = s"TRUNCATE TABLE  `$tableName`"
      val truncQueryConfig = QueryJobConfiguration.newBuilder(truncQuery).build
      bigquery.query(truncQueryConfig)
    }
    catch {
      case e: BigQueryException =>println("Table truncation failed \n" + e.toString)
    }
  }

}

2)In build.gradle , I have added the dependency

implementation group: 'com.google.cloud', name: 'google-cloud-bigquery', version: '1.133.0'

3)In the Spark Job I have added the google-cloud-bigquery jar as a dependency as well.

--jars gs://bucket_name/google-cloud-bigquery-1.133.0.jar

But when I run the spark job , it fails with the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/google/cloud/http/BaseHttpServiceException My question is what is it that I am missing ? Because I am already adding the jar as an argument in the spark job.

google-cloud-dataproc apache-spark google-bigquery java
2 Answers

use --packages com.google.cloud:google-cloud-bigquery:1.133.1. It will make spark download all the transitive dependencies as well

user_828040

It's running in my case

sparkSession
    .conf()
    .set(
        "spark.jars.packages",
        "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.18.1");

Please use the version as per the scala version you have in the cluster.

user_78646840
Related