Is it possible to break lines between a bitfshift operator when setting Task dependencies in a DAG?
My DAG has 10 Tasks, and they all must be executed in sequence, with no parallelization.
I'd like to chain them in my code vertically, like below:
task_1 \
>> task_2 \
>> task_3 \
>> task_n \
Instead of horizontally:
task_1 >> task_2 >> task_3 >> task_n
I tried the \ character but I had no success with it.
Thank you
There is (as far as I know) two ways to do this in Python. The first being the backslash, which you already tried, the second being the parentheses. In your case it would look like this:
foo = (task_1 >> task_2 >> task_3 >> task_n)
PEP8 on maximum line length says:
The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. Long lines can be broken over multiple lines by wrapping expressions in parentheses. These should be used in preference to using a backslash for line continuation.
When breaking your line keep in mind that your binary operator (>> in this case) always comes on the new line in front of the statment. This makes it a lot more readable e.g.
foo = variable1 +
variableThatIsVeryLong -
var *
varAverage
vs.
foo = variable1
+ variableThatIsVeryLong
- var
* varAverage
This way it is immediately clear to the reader what operator is being used.
One possible reason the backslash is not working might be because of a space right after it.
You can use chain function from airflow.utils.helpers to chain linear dependencies.
Example:
from airflow.utils.helpers import chain
tasks = [op1, op2, op3, op4, op5]
chain(*tasks)
is equivalent to:
op1 >> op2 >> op3 >> op4 >> op5
Details: https://airflow.apache.org/docs/stable/concepts.html#relationship-builders