Airflow - run sequentially couple of operators

Viewed 25

I'm trying to execute a couple of airflow's operator e.g.

PythonOperator

BashOperator

sequentially based on a parameter but using dynamic tasks to execute N tasks in parallel.

So, I got parameters params1, params2, ..., paramsN and I need to execute

( PythonOperator(param1) -> BashOperator(param1)), (PythonOperator(param2) -> BashOperator(param2)) ...

and I want these operators to run in parallel.

What I tried is to subclass Pythonoperator to inherit the partial method:

class PythonBashOperator(PythonOperator):
    def __init__(self, **kwargs):
       super().__init__(param=none, **kwargs)
       self.param = param

and to override execute

def execute(self, context):
    task_1= PythonOperator(**kwargs, command=<something with param>)
    task_2= BashOperator(.., bash_command<something with param>)
    task1 >> task2

I don't understand how to override the expand method, i'm trying with--->

PythonBashOperator.partial(...).expand(param=[1,2,3])

but expand does not recognizes param

Anyone has some ideas?

0 Answers
Related