Airflow - Unable to import Spark provider - package: name 'client' is not defined

user_13953116 0
Viewed 1006

I have installed the Apache Spark provider on top of my exiting Airflow 2.0.0 installation with:

pip install apache-airflow-providers-apache-spark

When I start the webserver it is unable to import the provider:

[2021-01-19 18:49:46,871] {providers_manager.py:279} WARNING - Exception when importing 'airflow.providers.apache.spark.hooks.spark_jdbc.SparkJDBCHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined
[2021-01-19 18:49:46,873] {providers_manager.py:279} WARNING - Exception when importing 'airflow.providers.apache.spark.hooks.spark_submit.SparkSubmitHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined
[2021-01-19 18:49:46,941] {providers_manager.py:279} WARNING - Exception when importing 'airflow.providers.apache.spark.hooks.spark_jdbc.SparkJDBCHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined
[2021-01-19 18:49:46,942] {providers_manager.py:279} WARNING - Exception when importing 'airflow.providers.apache.spark.hooks.spark_submit.SparkSubmitHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined

Any idea how to overcome this. For the record I have Python 3.8.5 and pip 20.0.2 on Ubuntu 20.04.

Thanks!

airflow apache-spark pip python
1 Answers

Solution

I solved it by installing the kubernetes client for python.

Possible cause

It seems there is something wrong with catching exceptions in the spark_submit module. In airflow/providers/apache/spark/hooks/spark_submit.py there is an except for the ImportError that would be thrown if the kubernetes client was not currently installed.

try:
    from airflow.kubernetes import kube_client
except ImportError:
    pass

Nonetheless, the exception that causes the error is a NameError because the client variable is not defined. Found in the airflow scheduler logs:

Traceback (most recent call last):
airflow-scheduler           |   File "/usr/local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 302, in _load_modules_from_file
airflow-scheduler           |     loader.exec_module(new_module)
airflow-scheduler           |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
airflow-scheduler           |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
airflow-scheduler           |   File "/opt/airflow/dags/spark_dag.py", line 4, in <module>
airflow-scheduler           |     from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
airflow-scheduler           |   File "/usr/local/lib/python3.8/site-packages/airflow/providers/apache/spark/operators/spark_submit.py", line 22, in <module>
airflow-scheduler           |     from airflow.providers.apache.spark.hooks.spark_submit import SparkSubmitHook
airflow-scheduler           |   File "/usr/local/lib/python3.8/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py", line 32, in <module>
airflow-scheduler           |     from airflow.kubernetes import kube_client
airflow-scheduler           |   File "/usr/local/lib/python3.8/site-packages/airflow/kubernetes/kube_client.py", line 101, in <module>
airflow-scheduler           |     ) -> client.CoreV1Api:
airflow-scheduler           | NameError: name 'client' is not defined
user_151971940
Related