How do I specify multiple shell scripts as initialization actions for Dataproc cluster creation?

user_1736407 0
Viewed 435

Google's documentation says that --initialization-actions takes a list of GCS URLs. If I specify one:

--initialization-actions 'gs://my-project/myscript.sh'

This works fine.

--initialization-actions 'gs://my-project/myscript.sh', 'gs://my-project/myscript2.sh'

Gives the following error:

INVALID_ARGUMENT: Google Cloud Storage object does not exist 'gs://my-project/myscript.sh gs://my-project/myscript2.sh'

Same without quotes, and with or without a space after the comma.

I tried encapsulating in square brackets:

--initialization-actions ['gs://my-project/myscript.sh', 'gs://my-project/myscript2.sh']

And the error this time is:

Executable '['gs://my-project/myscript.sh', 'gs://my-project/myscript2.sh']' URI must begin with 'gs://'

I can confirm one million percent that the paths I am using are valid, and that both objects are valid shell scripts. Is there something obvious I am missing?

google-cloud-storage google-cloud-dataproc google-cloud-platform
2 Answers

You should remove the space between the scripts:

--initialization-actions gs://my-project/myscript.sh,gs://my-project/myscript2.sh
user_828040

Just figured it out, the format needs to be:

--initialization-actions 'gs://my-project/myscript.sh, gs://my-project/myscript2.sh'

ie both scripts in a single set of quotes, with comma separating.

user_17364070
Related