Spark initialisation failing in dataproc - java.util.ServiceConfigurationError

user_12285881 0
Viewed 167

Trying to run a spark job on dataproc. But the job is failing to even initialise spark context.

When spark context was created with yarn-client as master, I got below error:

        SparkSession sparkSession = SparkSession.builder()
                .appName("Sample App")
                .master("yarn-client")
                .getOrCreate();

ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.util.ServiceConfigurationError: org.apache.spark.scheduler.ExternalClusterManager: Provider org.apache.spark.scheduler.cluster.k8s.KubernetesClusterManager not a subtype
    at java.util.ServiceLoader.fail(ServiceLoader.java:239)
    at java.util.ServiceLoader.access$300(ServiceLoader.java:185)

Or

And when I don't set the master or pass local instead, I get below error when I submit the job:

ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)

From my understanding, this error should not come up as Google cloud storage connector is automatically installed/configured in dataproc, so why i am seeing this error.

Any guidance on how I can resolve either of these errors and what they mean.

google-cloud-dataproc apache-spark google-cloud-platform java
1 Answers

You don't need .master("yarn-client"), just remove that line. Spark on Dataproc has been configured to use YARN as the cluster manager by default. These are the related properties:

spark.master=yarn
spark.submit.deployMode=client

If you do want to set master explicitly, the correct value is "yarn" instead of "yarn-client".

See this doc for more details.

user_4042640
Related