Cloud Composer - DAG Task Log File is not Found

user_12386620 0
Viewed 1179

Since a few days ago some tasks throw an error at the start of every DAG run. It seems the log file is not found to retrieve the logging from the task.

*** 404 GET https://storage.googleapis.com/download/storage/v1/b/europe-west1-ventis-brand-f-65ab79d1-bucket/o/logs%2Fimport_ebay_snapshot_feeds_ES%2Fstart%2F2021-11-30T08%3A00%3A00%2B00%3A00%2F3.log?alt=media: No such object: europe-west1-ventis-brand-f-65ab79d1-bucket/logs/import_ebay_snapshot_feeds_ES/start/2021-11-30T08:00:00+00:00/3.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

I've upgraded to the latest version of Cloud Composer and the tasks run on Python3.

There is the environment configuration:

Resources:

Workloads configuration:

  • Scheduler
  • 1 vCPU, 2 GB memory, 2 GB storage

Number of schedulers: 2

Web server:

  • 1 vCPU, 2 GB memory, 2 GB storage

Worker 1 vCPU, 2 GB memory, 1 GB storage

Number of workers:

  • Autoscaling between 1 and 4 workers

Core infrastructure: Environment size: Small

GKE cluster projects/*********************************

There are no related issues regarding this error in Cloud Composer changelog. How could this being fixed?

google-cloud-composer airflow google-cloud-platform
1 Answers

Its a bug in the Cloud composer environment and has already been reported. You can track this conversation: Re: Log not visible via Airflow web log and other similar forums. For fixing the issue, its recommended to update you composer environment or use an stable version.

There is some workaround suggested (You can try in this order, they are independent from each other):

  1. Remove the logs from the /logs folder in Composer GCS bucket and archive it in some other place (outside of /logs folder).

or

  1. Manually update the web server configuration to read logs directly from a new bucket in your project. You would first need to grant viewer roles (like roles/storage.legacyBucketReader and roles/storage.legacyObjectReader) on the bucket to the service account running the web server.
  • edit /home/airflow/gcs/airflow.cfg → remote_base_log_folder = <newbucket> with proper permissions as described above.

or

  1. If you don't have DRS (Domain restricted sharing) enabled which I believe you don't. You can create a new Composer environment, this time through v1 Composer API or without Beta features enabled in Cloud Console. This way Composer will create an environment without the DRS-compliant setup, so without the bucket-to-bucket synchronization. The problem is that you would need to migrate your DAGs and data to the new environment.
user_132487380
Related