Airflow Log file does not exist:

I had a similar issue, and I figured that in my case the worker node (I was using Celery Executor) was exhausted and therefore unavailable to execute any dags on it, can you check the CPU and memory utilized by the worker node (or its alternative if you are not using celery executor).

You can try to increase the CPU and memory for that applicable node and try.

Happened to me as well using LocalExecutor and an Airflow setup on Docker Compose. Eventually, I figured that the webserver would fail to fetch old logs whenever I recreated my Docker containers. Digging deeper, I realized that the webserver was failing to fetch the logs because it didn't have access to the filesystem of the scheduler (where the logs live).

The fix was to ensure that both the scheduler and the webserver services in docker-compose.yml share a volume with the logs, i.e.:

# docker-compose.yml

version: "3.9"
services:
  scheduler:
    image: ...
    volumes:
      - airflow_logs:/airflow/logs
    ...
  webserver:
    image: ...
    volumes:
      - airflow_logs:/airflow/logs
    ...
volumes:
  airflow_logs: