Unable to create Dataproc cluster on Google Cloud Platform "NodeInitializationAction must specify executable"

Viewed 379

Facing below error while creating Dataproc Cluster on Google Cloud Platform. We are using Mercury plugin for Airflow. Just want to understand like what is the issue. I tried many options but till now I could not reach any conclusion.

[2020-03-26 06:04:14,612] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:14,612] {models.py:1352} INFO - Executing <Task(GoogleCloudCreateDataprocCluster): create_medax_cluster> on 2020-03-26 06:03:40.963562
[2020-03-26 06:04:14,649] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:14,649] {gcp_api_base_hook.py:73} INFO - Getting connection using `gcloud auth` user, since no key file is defined for hook.
[2020-03-26 06:04:14,655] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:14,655] {discovery.py:267} INFO - URL being requested: GET https://www.googleapis.com/discovery/v1/apis/dataproc/v1beta2/rest
[2020-03-26 06:04:14,655] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:14,655] {transport.py:157} INFO - Attempting refresh to obtain initial access_token
[2020-03-26 06:04:14,732] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:14,731] {discovery.py:866} INFO - URL being requested: GET https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json
[2020-03-26 06:04:15,455] {base_task_runner.py:95} INFO - Subtask: File gs://xxxxxxxx-xxxxxxxx-dpl-artif/dataproc/dataproc-init.sh will not be executed on dataproc startup.
[2020-03-26 06:04:15,455] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:15,455] {discovery.py:866} INFO - URL being requested: POST https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json
[2020-03-26 06:04:20,490] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,490] {discovery.py:866} INFO - URL being requested: GET https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json
[2020-03-26 06:04:20,534] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,533] {models.py:1427} ERROR - <HttpError 400 when requesting https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json returned "Multiple validation errors:
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask:  - NodeInitializationAction must specify executable
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask:  - Object URI '' is not a valid GCS URI">
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1384, in run
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/airflow/plugins/mercury_plugins.py", line 1516, in execute
[2020-03-26 06:04:20,535] {base_task_runner.py:95} INFO - Subtask:     raise e
[2020-03-26 06:04:20,536] {base_task_runner.py:95} INFO - Subtask: HttpError: <HttpError 400 when requesting https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json returned "Multiple validation errors:
[2020-03-26 06:04:20,536] {base_task_runner.py:95} INFO - Subtask:  - NodeInitializationAction must specify executable
[2020-03-26 06:04:20,536] {base_task_runner.py:95} INFO - Subtask:  - Object URI '' is not a valid GCS URI">
[2020-03-26 06:04:20,536] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,534] {models.py:1451} INFO - Marking task as FAILED.
[2020-03-26 06:04:20,537] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,537] {configuration.py:609} WARNING - section/key [smtp/smtp_user] not found in config
[2020-03-26 06:04:20,538] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,538] {models.py:1466} ERROR - Failed at executing callback
[2020-03-26 06:04:20,539] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,538] {models.py:1467} ERROR - [Errno 99] Cannot assign requested address
[2020-03-26 06:04:20,539] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
[2020-03-26 06:04:20,539] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1464, in handle_failure
[2020-03-26 06:04:20,539] {base_task_runner.py:95} INFO - Subtask:     task.on_failure_callback(context)
[2020-03-26 06:04:20,539] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/airflow/dags/notification.py", line 35, in on_failure_callback
[2020-03-26 06:04:20,539] {base_task_runner.py:95} INFO - Subtask:     return operator.execute(context=context)
[2020-03-26 06:04:20,540] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/operators/email_operator.py", line 64, in execute
[2020-03-26 06:04:20,540] {base_task_runner.py:95} INFO - Subtask:     send_email(self.to, self.subject, self.html_content, files=self.files, cc=self.cc, bcc=self.bcc, mime_subtype=self.mime_subtype)
[2020-03-26 06:04:20,540] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/email.py", line 44, in send_email
[2020-03-26 06:04:20,540] {base_task_runner.py:95} INFO - Subtask:     return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype)
[2020-03-26 06:04:20,540] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/email.py", line 86, in send_email_smtp
[2020-03-26 06:04:20,541] {base_task_runner.py:95} INFO - Subtask:     send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun)
[2020-03-26 06:04:20,541] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/email.py", line 104, in send_MIME_email
[2020-03-26 06:04:20,541] {base_task_runner.py:95} INFO - Subtask:     s = smtplib.SMTP_SSL(SMTP_HOST, SMTP_PORT) if SMTP_SSL else smtplib.SMTP(SMTP_HOST, SMTP_PORT)
[2020-03-26 06:04:20,541] {base_task_runner.py:95} INFO - Subtask:   File "/usr/lib/python2.7/smtplib.py", line 256, in __init__
[2020-03-26 06:04:20,542] {base_task_runner.py:95} INFO - Subtask:     (code, msg) = self.connect(host, port)
[2020-03-26 06:04:20,542] {base_task_runner.py:95} INFO - Subtask:   File "/usr/lib/python2.7/smtplib.py", line 316, in connect
[2020-03-26 06:04:20,542] {base_task_runner.py:95} INFO - Subtask:     self.sock = self._get_socket(host, port, self.timeout)
[2020-03-26 06:04:20,542] {base_task_runner.py:95} INFO - Subtask:   File "/usr/lib/python2.7/smtplib.py", line 291, in _get_socket
[2020-03-26 06:04:20,542] {base_task_runner.py:95} INFO - Subtask:     return socket.create_connection((host, port), timeout)
[2020-03-26 06:04:20,543] {base_task_runner.py:95} INFO - Subtask:   File "/usr/lib/python2.7/socket.py", line 575, in create_connection
[2020-03-26 06:04:20,543] {base_task_runner.py:95} INFO - Subtask:     raise err
[2020-03-26 06:04:20,543] {base_task_runner.py:95} INFO - Subtask: error: [Errno 99] Cannot assign requested address
[2020-03-26 06:04:20,553] {base_task_runner.py:95} INFO - Subtask: [2020-03-26 06:04:20,553] {models.py:1472} ERROR - <HttpError 400 when requesting https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json returned "Multiple validation errors:
[2020-03-26 06:04:20,553] {base_task_runner.py:95} INFO - Subtask:  - NodeInitializationAction must specify executable
[2020-03-26 06:04:20,554] {base_task_runner.py:95} INFO - Subtask:  - Object URI '' is not a valid GCS URI">
[2020-03-26 06:04:20,554] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last):
[2020-03-26 06:04:20,554] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/bin/airflow", line 28, in <module>
[2020-03-26 06:04:20,554] {base_task_runner.py:95} INFO - Subtask:     args.func(args)
[2020-03-26 06:04:20,554] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 422, in run
[2020-03-26 06:04:20,554] {base_task_runner.py:95} INFO - Subtask:     pool=args.pool,
[2020-03-26 06:04:20,555] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 53, in wrapper
[2020-03-26 06:04:20,555] {base_task_runner.py:95} INFO - Subtask:     result = func(*args, **kwargs)
[2020-03-26 06:04:20,555] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1384, in run
[2020-03-26 06:04:20,555] {base_task_runner.py:95} INFO - Subtask:     result = task_copy.execute(context=context)
[2020-03-26 06:04:20,555] {base_task_runner.py:95} INFO - Subtask:   File "/usr/local/airflow/plugins/mercury_plugins.py", line 1516, in execute
[2020-03-26 06:04:20,555] {base_task_runner.py:95} INFO - Subtask:     raise e
[2020-03-26 06:04:20,556] {base_task_runner.py:95} INFO - Subtask: googleapiclient.errors.HttpError: <HttpError 400 when requesting https://dataproc.googleapis.com/v1beta2/projects/dh-xxxxxxxx-xxxxxxxx-72410/regions/xxxxxxxx/clusters?alt=json returned "Multiple validation errors:
[2020-03-26 06:04:20,556] {base_task_runner.py:95} INFO - Subtask:  - NodeInitializationAction must specify executable
[2020-03-26 06:04:20,556] {base_task_runner.py:95} INFO - Subtask:  - Object URI '' is not a valid GCS URI">
[2020-03-26 06:04:22,646] {jobs.py:2107} INFO - Task exited with return code 1

I tried changing name of cluster. But that did not work.

1 Answers

The issues is that you specified an empty path for initialization action in Airflow workflow:

INFO - Subtask:  - Object URI '' is not a valid GCS URI">

Here is an example how you should do it using DataprocClusterCreateOperator in Airflow:

DataprocClusterCreateOperator(
    # ...
    init_actions_uris = ['gs://<BUCKET>/path/to/init/action.sh'],
    # ...
)

If you are not using initialization actions, then you should not set init_actions_uris parameter at all, or you should set it to None.

Related