AWS managed Ariflow doesn't fetch log

user_3079886 0
Viewed 3028

I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch.

I am getting the following error in the log of Air flow:

*** Reading remote log from Cloudwatch log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
Could not read remote logs from log_group: airflow-Task log_stream: 2021-08-26T08_08_24+00_00/1.log.
mwaa airflow amazon-web-services logging
3 Answers

AWS has a troubleshooting guide for this specific case.

*** Reading remote log from Cloudwatch log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.Could not read remote logs from log_group: airflow-{environmentName}-Task log_stream: {DAG_ID}/{TASK_ID}/{time}/{n}.log.

https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs

I would check the following:

  1. Verify that you enabled task logs at the INFO level for your environment.
  2. Check if the MWAA Execution role has CloudWatch read access policy attached.
  3. Eventually try adding apache-airflow[amazon] provider to the MWAA Requirements file.
user_30698710

It was an issue related to role and policies.

MWAA Execution role didn't have the required CloudWatch access policy attached to it.

user_30798860

In my case the issue was because of encryption. Check if your MWAA uses the same KMS key as your CloudWatch Log Groups.

If MWAA is created with custom KMS key this key is also used for CloudWatch Log Groups encryption. When you delete and re-create MWAA (with new KMS key) Log Groups are not touched and remain encrypted with old key. And they are not accessible for MWAA any more.

Two ways to fix it:

  1. Delete Log Groups, create new Log Groups with the same name and using KMS key which is used by MWAA.
  2. Don't use KMS key when you create MWAA. In this case, AWS managed key aws/airflow will be used automatically. It should help to avoid dependency on particular KMS key.
user_42589160
Related