In airflow, can I shorten these task dependency lines?
a >> b >> c >> f >> G
a >> b >> d >> f >> G
To
a >> b
b >> c
b >> d
c >> f
d >> f
f >> g
These are equivalent? Any difference in timings or efficiencies? Reason for asking that if you are using a code formatter line length becomes and issue with the first style and longer variable task names.
Any recommendations or suggestions?
There aren't any timing or efficiency pros/cons to either of those variations but this is more of a "what is more readable" question I think.
The first is more readable. However, there is some duplication and it can be shortened by using a list to have tasks execute in parallel:
a >> b >> [c, d] >> f >> G
For really long dependency chains, I like using the convenient chain() method. It's a little more flexible (i.e. you cannot set dependencies between two lists of tasks using the bitshift operators but you can with chain()) and might be considered more approachable since it's a function. Oddly enough, this does help with line-length constraints in code formatting too in my experience and gets reformatted nicely with those formatters (like black for example). Using this method, the example dependency could be rewritten as:
from airflow.models.baseoperator import chain
chain(a, b, [c, d], f, G)