Logs from Airflow DAGs not written to GCP Cloud Logging

user_13054038 0
Viewed 78

In our Airflow DAG code, we are trying to write the airflow logs directly into stack-driver using the sample code present in: https://cloud.google.com/logging/docs/reference/libraries#log-entries

However, as per default setting the logs get written to the logs directory in the Airflow GCE VM. We changed the airflow.cfg file to include the below entries(https://airflow.apache.org/docs/apache-airflow-providers-google/stable/logging/stackdriver.html):

remote_logging = True
remote_base_log_folder = stackdriver://airflow-task-logs
google_key_path = <path in the vm to the json file>

But still we are not able to see the logs in the cloud logging/stackdriver when we filter by the log name 'airflow-task-logs'.

What extra configuration is required to make it work?

google-cloud-logging google-cloud-stackdriver airflow google-cloud-platform
1 Answers

Given the nature of how the value for remote_base_log_folder is parsed, you need to provide it as an absolute path.

from urllib.parse import urlparse

urlparse("stackdriver://airflow-task-logs").path[1:]
# evaluates to  ''

urlparse("stackdriver:///airflow-task-logs").path[1:]
# evaluates to airflow-task-logs

Set remote_base_log_folder

remote_base_log_folder = 'stackdriver:///airflow-task-logs'
user_51898110
Related