Does an Airflow operator need to return anything in order to function correctly?

Viewed 1049

In Airflow, if I create a custom operator class, and use it in a dag, must it actually return anything?

Couple of examples, let's say I create sensor operator (inherits from base_sensor_operator) which checks every 5 min for the existence of a file somewhere. Similar to check_for_wildcard_key() and how it returns True or False. Does any method I create have to actually have a return statement with this True/False value in order to function? Will the scheduler know to retry this task without a return statement?

Another example is if I create a custom operator (non sensor) that executes SQL code in a database that isn't as common, so I need to create my own operator to connect to that database. So let's say I am using the method execute_SQL() inside my custom operator class. I need not have a return value, right? Since the operator object itself inside of a dag file will simply just "run", I don't need to return anything?

I understand sometimes we have a use for xcom_push and xcom_pull, so it might be good to have return values in case we need to send/receive results between tasks. Is it good practice to return something even if it might not be used? Will custom operator classes work without return statements?

1 Answers

No, an Operator class does not need to return anything. Actually the main function of an operator is the execute method, and most of them does not return anything. Examples: BranchPythonOperator, TriggerDagRunOperator, DatabricksRunNowOperator, etc. They just do the job.

On the other hand, the poke method of the sensors need to return True or False since the execute method of the BaseSensorOperator calls self.poke method, and takes action based on the return value.

Related