Airflow 2 on k8s S3 logging is not working

user_12987852 0
Viewed 393

I'm using the latest helm chart to install Airflow 2.1.1 on k8s. I have a problem with s3 logging - I'm keep getting the error message:

*** Falling back to local log
*** Log file does not exist: /opt/airflow/logs/test_connection/send_slack_message/2021-07-16T08:48:27.337421+00:00/2.log
*** Fetching from: http://airflow2-worker-1.airflow2-worker.airflow2.svc.cluster.local:8793/log/test_connection/send_slack_message/2021-07-16T08:48:27.337421+00:00/2.log

in the task logs.

this is the relevant part from the chart values:

AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_logs"
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://.../temp/airflow_logs/stg"

The s3_logs connection is defined like this: enter image description here

What am I missing?

Technical details:

  • chart - airflow-8.4.0

  • app version - 2.1.1

  • eks version - 1.17

airflow amazon-s3
1 Answers

So it seems that the S3 target folder should exist before writing the first log and that solves the issue. I hope that it will help someone in the future!

user_129878520
Related