Passing parameters into dataproc pyspark job

user_8785546 0
Viewed 7028

How do you pass parameters into the python script being called in a dataproc pyspark job submit? Here is a cmd I've been mucking with:

 gcloud dataproc jobs submit pyspark --cluster my-dataproc \ 
 file:///usr/test-pyspark.py \
 --properties=^:^p1="7day":p2="2017-10-01"

This is the output returned:

 Job [vvvvvvv-vvvv-vvvv-vvvv-0vvvvvv] submitted. Waiting for job output...
 Warning: Ignoring non-spark config property: p2=2017-10-01 
 Warning: Ignoring non-spark config property: p1=7day 
 Found script=/usr/test-pyspark.py 
 Traceback (most recent call last):   
   File "/usr/test-pyspark.py", line 52, in <module>
     print(sys.argv[1]) 
 IndexError: list index out of range`

Clearly doesn't recognize the 2 params I'm trying to pass in. I also tried:

me@my-dataproc-m:~$  gcloud dataproc jobs submit pyspark --cluster=my-dataproc test-pyspark.py 7day 2017-11-01

But that returned with:

ERROR: (gcloud.dataproc.jobs.submit.pyspark) unrecognized arguments:
  7day
  2017-11-01

The pattern I use to pass params with the hive jobs doesn't work for pyspark.

Any help appreciated!

Thanks, Melissa

google-cloud-dataproc
2 Answers

This worked for me for passing multiple arguments -

gcloud dataproc jobs submit pyspark --cluster <cluster_name> --region europe-west1 --properties spark.master=yarn --properties spark.submit.deployMode=client --properties spark.sql.adaptive.enabled=true --properties spark.executor.memoryOverhead=8192 --properties spark.driver.memoryOverhead=4096 .py -- --arg1=value1 --arg2=value2

or simply saying -

gcloud dataproc jobs submit pyspark --cluster <cluster_name> <name-of-the-script>.py -- --arg1=value1 --arg2=value2

All the Best !

user_99152310
Related