adding jar files in dataproc cluster Initialization script for Jupyter usage

userrwdC 0
Viewed 29

I have a requirement to read xlsx files and create Avro files, but the GCP Cluster does not have these jars.

  1. My Pyspark process is using sources excel and Avro. I would like to include the jars for these in the cluster creation Terraform script by calling a cluster_startup.sh What should be the contents of the cluster_startup.sh for excel and Avro jars
initialization_action
{
      script      = "gs://bucket_name/cluster_startup.sh"

      timeout_sec = 500    
}

If any additional information is needed in this script like maven dependencies for Avro, please provide that details.

  1. I would like to invoke the jars like below in pyspark Jupyter

Please note we do not have access to terminal/shell/external internet for git, so this has to be invoked in the Pyspark jupyter only.

Where will these jars be saved during the cluster creation initialization process?

excel_jar=f"gs://{bucket_name}/com.crealytics:spark-excel_2.11:0.13.0"

avro_jar=f"gs://{bucket_name}/spark-avro-assembly-3.1.0-SNAPSHOT.jar"

spark = SparkSession.builder \
.master("local") \
.appName("Word Count") \
.config("spark.jars",excel_jar,avro_jar) \
.getOrCreate()
df = spark.read.format("com.crealytics.spark.excel") \
.option("useHeader", "true") \
.option("inferSchema", "true") \
.option("dataAddress", "'NameOfYourExcelSheet'!A1") \
.load("your_file"))

df.write.format("avro").save("/tmp/output/test.avro")

Please provide the required details.

google-cloud-dataproc pyspark
1 Answers

There are several options:

  1. When creating the cluster, use an init action to download the jars to /usr/lib/spark/jars. Say your jars are in gs://my-bucket/jars/, then the init action would be gsutil cp gs://my-bucket/jars/*.jar /usr/lib/spark/jars/.

  2. Or add spark.jars with jar URIs (e.g., gs://{bucket_name}/spark-avro-assembly-3.1.0-SNAPSHOT.ja) or spark.jars.packages with package names (e.g., com.crealytics:spark-excel_2.11:0.13.0). I noticed in your code, seems you are mixing URI and package name. And You need to join excel_jar and avro_jar with , as a single string. The properties can be used as command line flag when submitting Spark/PySpark jobs with gcloud or spark-submit, or as code in Jupyter notebooks.

Dagang0
Related