Airflow (LocalExecutor) - Docker :: Job is failing with Log file does not exist

user_10628775 0
Viewed 236

Airflow version: 1.10.9

Executor : LocalExecutor

Docket Setup

when job runs sometime we are getting following error. I have searched in web, many people faced this issue in celeryExecutor but we are using LocalExecutor(Docker setup). How can I resolve this problem?

*** Log file does not exist: /home/ubuntu/airflow/airflow/logs/es_update_relevance_score/es_update_relevance_score/2020-05-14T16:26:06.062416+00:00/1.log
*** Fetching from: http://:8793/log/es_update_relevance_score/es_update_relevance_score/2020-05-14T16:26:06.062416+00:00/1.log
*** Failed to fetch log file from worker. Invalid URL 'http://:8793/log/es_update_relevance_score/es_update_relevance_score/2020-05-14T16:26:06.062416+00:00/1.log': No host supplied
airflow
1 Answers

Here is one approach I've seen when running the scheduler and webserver in their own containers and using LocalExecutor:

  1. Mount a host log directory as a volume into both the scheduler and webserver containers:
    volumes:
        - /location/on/host/airflow/logs:/opt/airflow/logs
  1. Make sure the user within the airflow containers (usually airflow) has permissions to read and write that directory. If the permissions are wrong you will see an error like the one in your post.

This probably won't scale beyond LocalExecutor usage though.

user_106821640
Related