I'm running PySpark application on Dataproc cluster.
The dependency jars are in a cloud storage bucket.
While creating the cluster I've set the "spark.jars" property to point to the "gs://" path of the jars in cloud storage. In SparkUI for the application, I could see the dependency jars under ClassPath entries. But the Spark application fails with "ClassNotFoundException".
Alternatively I've tried to submit the job using "gcloud dataproc" cli command and passed the jars using "--jars" flag. Still I get the same error.
What am I missing here?