Airflow Logging with Custom Filenames?

user_3023615 0
Viewed 686

I am using Apache Airflow 1.10.9 (based on puckel/docker-airflow docker image) to run several Python scripts in a DAG via the BashOperator. The logs are currently written to /usr/local/airflow/logs.

Is it possible to configure Airflow to

  1. also write the logs to another directory like /home/foo/logs
  2. The logs should only contain the stdout from the python scripts

  3. The logs should be stored in the following directory/filename format:

    /home/foo/logs/[execution-date]-[dag-id]-[task-id].log

  4. Retries should be appended to the same .log file, if possible. Otherwise, we can have the naming convention:

    /home/foo/logs/[execution-date]-[dag-id]-[task-id]-[retry-number].log

Thanks everyone!

Example DAG

from airflow import DAG
from airflow.operators.bash_operator import BashOperator

default_args = { ... }

dag = DAG(
    'mydag',
    default_args=default_args,
    schedule_interval='*/10 * * * *',
)

# Log to /home/foo/logs/2020-05-12-mydag-hello_world.log
t1 = BashOperator(
    task_id='hello_world',
    bash_command='/path/to/env/bin/python /path/to/scripts/hello_world.py',
    dag=dag,
)


# Log to /home/foo/logs/2020-05-12-mydag-hey_there.log
t2 = BashOperator(
    task_id='hey_there',
    bash_command='/path/to/env/bin/python /path/to/scripts/hey_there.py',
    dag=dag,
)

t1 >> t2
airflow python-3.x logging python
1 Answers

https://bcb.github.io/airflow/run-dag-and-watch-logs

This link has an answer.

Set the FILENAME_TEMPLATE setting.

export AIRFLOW__CORE__LOG_FILENAME_TEMPLATE="{{ ti.dag_id }}.log"

or

you can edit the airflow.cfg file log_filename_template variable

add any airflow related variables there.

user_13017530
Related