SSHOperator with ComputeEngineSSHHook

user_7479376 0
Viewed 477

I am trying to run a command using ssh in a GCP VM in airflow via the SSHOperator as described here:

ssh_to_vm_task = SSHOperator(
    task_id="ssh_to_vm_task",
    ssh_hook=ComputeEngineSSHHook(
        instance_name=<MYINSTANCE>,
        project_id=<MYPROJECT>,
        zone=<MYZONE>,
        use_oslogin=False,
        use_iap_tunnel=True,
        use_internal_ip=False
        ),
    command="echo test_message",
    dag=dag
)

However, I get a airflow.exceptions.AirflowException: SSH operator error: [Errno 2] No such file or directory: 'gcloud' error.

Docker is installed via docker-compose following these instructions.

Other Airflow GCP operators (such as BigQueryCheckOperator) work correctly. So at first sight it does not seem like a configuration problem.

Could you please help me? Is this a bug?

google-compute-engine airflow google-cloud-platform virtual-machine ssh
2 Answers

It seems the issue is that gcloud was not installed in the docker container by default. This has been solved by following instructions in here: it is necessary to add

RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg  add - && apt-get update -y && apt-get install google-cloud-sdk -y

to the dockerfile that is used to install airflow / install dependencies.

user_74793760

Check if the TCP port 22 is allowed through the firewall on your GCP VM instance, and make sure that the VM instance also allows SSH access and is properly configured in that VM instance. Furthermore, be sure that the IP from which you are trying to SSH at the VM instance is whitelisted through the firewall.

You can use the following command in GCP to check the ingress firewall rule for the network that contains the destination VM instance. Additionally, you can consult this [link]for more information.

This is an example of what you have to do. ´´´

gcloud compute firewall-rules list --filter network=[NETWORK-NAME] \
    --filter INGRESS \
    --sort-by priority \
    --format="table(
        name,
        network,
        direction,
        priority,
        sourceRanges.list():label=SRC_RANGES,
        destinationRanges.list():label=DEST_RANGES,
        allowed[].map().firewall_rule().list():label=ALLOW,
        denied[].map().firewall_rule().list():label=DENY,
        sourceTags.list():label=SRC_TAGS,
        sourceServiceAccounts.list():label=SRC_SVC_ACCT,
        targetTags.list():label=TARGET_TAGS,
        targetServiceAccounts.list():label=TARGET_SVC_ACCT
        )"

´´´
user_178941040
Related