How can I run a spark-job from Airflow when both Spark and Airflow are installed in the same virtual environment?

Viewed 781

I am trying to run from Airflow a test ETL pyspark job on my laptop. I have installed in the same virtual environment both Spark and Airflow. Is important to mention that I can run the job from virtual environment successfully but it fails from Airflow. From the Airflow logs I can see that the ETL task fails because it looks for the dag file in a temporary directory when trying to execute it. Here is the error: "python: can't open file '/tmp/airflowtmpd9uwr02_/spark_etl_1.py'" (entire log below).

> AIRFLOW_CTX_DAG_OWNER=florian
AIRFLOW_CTX_DAG_ID=Example_Airflow_and_Spark
AIRFLOW_CTX_TASK_ID=spark_task_etl
AIRFLOW_CTX_EXECUTION_DATE=2021-02-24T14:28:23.580164+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-02-24T14:28:23.580164+00:00
[2021-02-24 16:28:27,498] {bash.py:135} INFO - Tmp dir root location: 
 /tmp
[2021-02-24 16:28:27,499] {bash.py:158} INFO - Running command: /home/florian/HDD2/WORK/PROJECTS/SPARK/spark_env/bin/spark-submit --master local spark_etl_1.py
[2021-02-24 16:28:27,503] {bash.py:169} INFO - Output:
[2021-02-24 16:28:28,273] {bash.py:173} INFO - 21/02/24 16:28:28 WARN Utils: Your hostname, florian-TUXEDO resolves to a loopback address: 127.0.1.1; using 192.168.0.107 instead (on interface wlp2s0)
[2021-02-24 16:28:28,273] {bash.py:173} INFO - 21/02/24 16:28:28 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
[2021-02-24 16:28:28,620] {bash.py:173} INFO - 21/02/24 16:28:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[2021-02-24 16:28:28,833] {bash.py:173} INFO - **python: can't open file '/tmp/airflowtmpd9uwr02_/spark_etl_1.py': [Errno 2] No such file or directory**
[2021-02-24 16:28:28,837] {bash.py:173} INFO - log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
[2021-02-24 16:28:28,837] {bash.py:173} INFO - log4j:WARN Please initialize the log4j system properly.
[2021-02-24 16:28:28,837] {bash.py:173} INFO - log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[2021-02-24 16:28:28,854] {bash.py:177} INFO - Command exited with return code 2
[2021-02-24 16:28:28,857] {taskinstance.py:1455} ERROR - Bash command failed. The command returned a non-zero exit code.
Traceback (most recent call last):

And this is the dag code:

from datetime import timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago

srcDir = "/home/florian/HDD2/WORK/PROJECTS/Personale/SPARK/"
sparkSubmit = "/home/florian/HDD2/WORK/PROJECTS/Personale/SPARK/spark_env/bin/spark-submit/"


args = {
    'owner': 'florian',
}

dag = DAG(
    dag_id='Example_Airflow_and_Spark',
    default_args=args,
    description='Test Spark ETL job from Airflow',
    start_date=days_ago(2),
    dagrun_timeout=timedelta(minutes=60),

)


change_dir = BashOperator(
    task_id='change_working_directory',
    bash_command="cd " + srcDir,
    dag=dag,
)

### [activate virtual environment]
activate_env = BashOperator(
    task_id='activate_virtual_env',
    bash_command="source /home/florian/HDD2/WORK/PROJECTS/SPARK/spark_env/bin/activate",
    dag=dag,
)


### [spark_job_operator_bash]
spark_submit_cmd="/home/florian/HDD2/WORK/PROJECTS/SPARK/spark_env/bin/spark-submit --master local spark_etl_1.py"

spark_job = BashOperator(
    task_id='spark_task_etl',
    bash_command=spark_submit_cmd,
    dag=dag,
)


### [deactivate virtual environment]
deactivate_env = BashOperator(
    task_id='deactivate_virtual_env',
    bash_command="deactivate",
    dag=dag,
)

change_dir >> activate_env >> spark_job >> deactivate_env
1 Answers

Your DAG resides in AIRFLOW_HOME which I presume is your /tmp folder. When you execute Spark submit, Airflow looks for the Spark script in the same folder, therefore

/home/florian/HDD2/WORK/PROJECTS/SPARK/spark_env/bin/spark-submit --master local spark_etl_1.py

is equivalent to

/home/florian/HDD2/WORK/PROJECTS/SPARK/spark_env/bin/spark-submit --master local ${AIRFLOW_HOME}/spark_etl_1.py

You might want to consider submitting the absolute path of the Spark script instead of just providing the file name spark_etl_1.py, e.g.

/home/florian/HDD2/WORK/PROJECTS/SPARK/spark_env/bin/spark-submit --master local /path/from/root/spark_etl_1.py
Related