I'm trying to execute a couple of airflow's operator e.g.
PythonOperator
BashOperator
sequentially based on a parameter but using dynamic tasks to execute N tasks in parallel.
So, I got parameters params1, params2, ..., paramsN and I need to execute
( PythonOperator(param1) -> BashOperator(param1)), (PythonOperator(param2) -> BashOperator(param2)) ...
and I want these operators to run in parallel.
What I tried is to subclass Pythonoperator to inherit the partial method:
class PythonBashOperator(PythonOperator):
def __init__(self, **kwargs):
super().__init__(param=none, **kwargs)
self.param = param
and to override execute
def execute(self, context):
task_1= PythonOperator(**kwargs, command=<something with param>)
task_2= BashOperator(.., bash_command<something with param>)
task1 >> task2
I don't understand how to override the expand method, i'm trying with--->
PythonBashOperator.partial(...).expand(param=[1,2,3])
but expand does not recognizes param
Anyone has some ideas?