Airflow with varying conda environments

user_360826 0
Viewed 477

I have an ecosystem where I have multiple git repositories. Each repository has its own conda environment that is very specific to that repo and does stuff.

In investigating airflow, it appears to be a great fit for problems I face.

My stack is likely familiar to a lot of airflow users, namely:

I am curious to know how people deal with this in the wild.

Do most [DAGs][2] call [bash operators][3] vs [python operators][4]?

Is the idealized vision for airflow that all python operator tasks are actually calling web services via [requests][5] or something similarly abstracted?

I am definitely planning on using airflow to solve some messy problems but want to be sure I build in the right direction and did not find a clearly organized doc on this exact question.

It would seem 100% suboptimal, and not in the vision of the project, to install airflow in each environment and have each airflow instance own various tasks (this may even cause airflow issues I am unfamiliar with). If I am wrong on this point, I should course-correct immediately.

Thank you to anyone in the community who can provide guidance on this.

[2]: https://airflow.apache.org/docs/apache-airflow/1.10.12/concepts.html#:~:text=In%20Airflow%2C%20a%20DAG%20%E2%80%93%20or,and%20their%20dependencies)%20as%20code. [3]: https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html [4]: https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html [5]: https://docs.python-requests.org/en/latest/

airflow
1 Answers

For your case you can use:

  • PythonVirtualeEnvOperator - this one will build new virtualenv every time it needs one so might be a little brittle
  • KubernetesPodOperator - where you can have different variant of the images with different environments and choose the one you want for each task (requires Kubernetes)
  • DockerOperator - same as KubernetesPodOperator, but requires just Docker engine

However the best thing for that is availale as of Airflow 2.2.

In the new Airflow 2.2 (released literally few days ago) you also have the new @task.docker decorator which allows you to combine the best of both worlds:

  1. separate images to keep your envs isolated and manageable
  2. Python code which is written as regular decorated function in your DAG.

More info here:

https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments

user_5167010
Related