Dataproc notebook cannot import or export to BigQuery : Class Not Found Exception

user_8715275 0
Viewed 245

Here is the spark session I am making. I include the latest jar for spark big query connector for Dataproc 1.5 .

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("GCS to BigQuery - Dev") \
.config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar').getOrCreate() 

df = spark.read \
  .format('bigquery') \
  .option('table', 'publicdata.samples.shakespeare') \
  .load()

Here's the exception :

Py4JJavaError: An error occurred while calling o228.load. : java.lang.ClassNotFoundException: Failed to find data source: bigquery

Edit 1: I have created the cluster from UI and not shell. I have seen solutions where when people add the jar file while creating the cluster from shell , this seems to work. But I am very curious as to why, when doing it from UI , it wont work.

Edit 2: I am using jupyter notebook to run the code above.

google-cloud-dataproc apache-spark google-bigquery
2 Answers

I think the description in SPARK-21752 is relevant - by this time the application is already launched and you cannot change its classpath. Please try to run with pyspark --jars gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar (and then you can skip the .config() part.

user_828040

Everything works fine If you just regress to the Dataproc image 1.4 and use the connector jar gs://spark-lib/bigquery/spark-bigquery-latest.jar

user_87152750
Related