I’ve seen airflow used in some projects recently. I’ve noticed that sometimes, people choose to use the PythonOperator with a callback, and other times the BashOperator with something like «python mytask.py». What’s the rationale for spawning a separate bash process just to run a python function/script? What are the pros and cons of each approach?
When to use PythonOperator with a callback vs. BashOperator with "python mytask.py"
pletnes
0
Viewed 38
1 Answers
Probably the script is calling a main function or something similar. In that case and from my point of view, I think you should always run your python functions from a python operator.
The same as is doesn't make much sense to run a bash command from python operator using for example os.system(command) as callable inside the python operator, it doesn't make much sense to execute a script containing a python function from a bash operator.
If you want to execute a python function define somewhere else outside the defined dags folder parsed by airflow, you can simply import it as a module as long as it's accessible in your PYTHONPATH. And you can always add that path where your script with the function is found using sys.path.append('my_path')
For more info about pythonoperator: https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html
Related