Right now I have the following DAG (omitted some non-relevant syntax)
from airflow.operator.python import PythonVirtualenvOperator
t1 = PythonVirtualenvOperator(requirements = req1)
t2 = PythonVirtualenvOperator(requirements = req2)
t3 = PythonVirtualenvOperator(requirements = req3)
t4 = PythonVirtualenvOperator(requirements = req1) #Yes, its the same as t1
t1>>t2>>t3>>t4
I'm a big fan of venvs, and since we are multiple people deploying DAGs on the same server, we can then keep the versions seperated. The issue is that above we have to create a venv for each of the tasks (which takes some time, not much, but some), and a lot of the packages in the requirements are the same.
Isn't there a way to create a virtual-environment for a specific DAG thus being able to use PythonOperator instead of PythonVirtualenvOperator? Or is the best way then to create a Docker container for each DAG?