AWS managed airflow to Databricks SSL error

user_9878967 0
Viewed 141

I am trying to use databricks inbuilt operator provided by airflow such as DatabricksSubmitRunOperator or DatabricksRunNowOperator but i am not able to use it, it gives below error:

Attempt 1 API Request to Databricks failed with reason: HTTPSConnectionPool(host='usdev.databaricks.xyz.com', port=443): Max retries exceeded with url: /api/2.0/jobs/run-now (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))

if i were to use custom python code then i can supply verify = False in requests module of python but since i am using this inbuilt operator i don't see any option to disable the ssl validation from airflow to databricks.

sample code being used:

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.providers.databricks.operators.databricks import (
    DatabricksRunNowOperator,
)
from datetime import datetime, timedelta

"""
SUCCESS SCENARIO.
- It will run the job_id=14, that is already created in databricks.
- Basically, it will trigger what that job is supposed to do (that is, run a notebook)
"""


# Define params for Run Now Operator
notebook_params = {"Variable": 5}


with DAG(
    "databricks_dag_run_now",
    start_date=datetime(2021, 1, 1),
    schedule_interval="@daily",
    catchup=False,
    default_args={
        "email_on_failure": False,
        "email_on_retry": False,
        "retry_delay": timedelta(minutes=2),
    },
) as dag:

    t0 = DummyOperator(
        task_id='start'
    )

    opr_run_now = DatabricksRunNowOperator(
        task_id="run_now",
        databricks_conn_id="custom_databricks_conn",
        job_id=12345,
        notebook_params=notebook_params,
    )


    t0 >> opr_run_now
mwaa airflow databricks amazon-web-services python
1 Answers

From the host name I can suppose that you're using load balancer or something like proxy before actual Databricks workspace, and this load balancer/proxy uses the self-signed certificate (actual certificates of Datarbicks workspaces are signed with well-known CA certificate).

To solve your problem you need to extract a CA certificate that was used to sign certificate of your load balancer/proxy (usdev.databaricks.xyz.com), for example, using the code from this answer or using web browser, and add it to the certificate chain of certifi package that is used by requests package with something like this (shell script, certificate should be in the PEM format):

CERTIFI_HOME="$(python -m certifi 2>/dev/null)"
cat path_to_certificate_file >> $CERTIFI_HOME
Alex_Ott_186270
Related