Dataproc how to run a initialization-actions script only on master node and skip running on worker nodes

user_12387333 0
Viewed 250

I have a use-case in which I want to utilize initialization-actions in dataproc to run a script that will install the necessary items. I have used a couple of initialization actions scripts on dataproc which work well, and does install on both master and worker nodes. But here, I would like to run it on master node only.

Specifically I m installing on top of yarn on master node ,So that it will automatically install on other nodes (worker nodes).

Is there any flag in Dataproc Initialization actions which will skip installing on worker nodes?

hadoop-yarn google-cloud-dataproc apache-spark google-cloud-platform initialization
1 Answers

Init actions cannot be skipped, but they can include node role specific logic:

ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [[ "${ROLE}" == 'Master' ]]; then
  #... master specific actions ...
else
  #... worker specific actions ...
fi

See this doc for more details.

user_4042640
Related