Is there a way to run BashOperator tasks in a virtual environment in airflow?

Viewed 16

I am aware of how task decorators are used to decorate python callables to create virtual environments for them to run in. However I have a requirement where I need to run a BashOperator based task in a virtual environment.

with DAG(
    "test_dag_venv",
    default_args=default_args,
    description='Dag to test venv',
    schedule_interval="@once",
    start_date=datetime(2022, 1, 6, 10, 45),
    tags=['testing'],
    concurrency=1,
    is_paused_upon_creation=True,
    catchup=False  # dont run previous and backfill; run only latest
) as dag:
    @task.virtualenv(task_id="print_test", requirements=['numpy'], system_site_packages=False)
    def print_test():
        import numpy as np
        print(np.__version__)
    t1 = print_test()
    t1

The above works as expected.Is there anything close to run BashOperator tasks as well. Something like.

  with DAG(
    "test_dag_venv",
    default_args=default_args,
    description='Dag to test venv',
    schedule_interval="@once",
    start_date=datetime(2022, 1, 6, 10, 45),
    tags=['testing'],
    concurrency=1,
    is_paused_upon_creation=True,
    catchup=False  # dont run previous and backfill; run only latest
) as dag:
    t1 = BashOperator(task_id='t1',
    bash_command="python scripts/my_script.py",
    requirements=['numpy']
    dag=dag)
    t1




0 Answers
Related