How to log inside Python function in PythonOperator in Airflow

user_1549741 0
Viewed 3941

I am using a PythonOperator in my Airflow DAG and I need to print something inside the operator's Python function. I tried to print but apparently it didn't work out. Not so sure that's going to work. Next I tried to pass self.log in PythonOperator but I am not sure how to pass that reference.

task = PythonOperator(
    task_id='task1',
    python_callable=my_func,
    params={ ... },
    provide_context=True,
    dag=dag
)

...

def my_func(**context):
    ...
    print(some_message) # this didn't work.
airflow
1 Answers

The way you should be logging in Airflow (and in Python in general) is through the logging module, that is,

import logging

on top of the DAG definition and

logging.info(some_message)

in place of the print statement in your my_func function. Other functions you can use besides info() (with different logging level/criticality) can be found in the Python docs: https://docs.python.org/3/howto/logging.html#logging-basic-tutorial

user_8251900
Related