Access Forbidden while accessing log in airflow with CeleryExecutor

Question

I'm trying to deploy Airflow on kubernetes (on Azure Kubernetes Service) with the celery Executor. However, once a task is done, I get the following error while trying to access its logs:

*** Log file does not exist: /opt/airflow/logs/maintenance/clean_events/2021-08-23T14:46:18.953030+00:00/1.log
*** Fetching from: http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/maintenance/clean_events/2021-08-23T14:46:18.953030+00:00/1.log
*** Failed to fetch log file from worker. 403 Client Error: FORBIDDEN for url: http://airflow-worker-0.airflow-worker.airflow.svc.cluster.local:8793/log/maintenance/clean_events/2021-08-23T14:46:18.953030+00:00/1.log
For more information check: https://httpstatuses.com/403

my charts.yaml is pretty simple

---
airflow:
   image:
      repository: myrepo.azurecr.io/maintenance-scripts
      tag: latest
      pullPolicy: Always
      pullSecret: "secret"
   executor: CeleryExecutor
   config:
      AIRFLOW__CORE__LOAD_EXAMPLES: "True"
      AIRFLOW__KUBERNETES__DELETE_WORKER_PODS: "False"
   users:
    - username: admin
      password: password
      role: Admin
      email: admin@example.com
      firstName: admin
      lastName: admin
rbac:
   create: true
serviceAccount:
    create: true
#postgresql:
#    enabled: true
workers:
    enabled: true
redis:
    enabled: true
flower:
    enabled: false
global:
  postgresql: {
      storageClass: managed
  }
persistence:
  fixPermissions: true
  storageClassName: managed

I have not been able to fix this, and it seems to be the most basic conf you can use on airflow, anyone knows where this could come from ?

Thanks a lot

user_516701 · Answer

You need to have the same webserver secret configured for both webserver and workers: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key

It's been recently fixed as potential security vulnerability - now you need to know the secret key to be able to retrieve logs (it was unauthenticated before).

user_19816518 · Answer

0481aeef5916   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   25 minutes ago   Up 25 minutes (healthy)   0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp   deploy_test_flower_1 bdf02b964712   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   25 minutes ago   Up 3 minutes (healthy)    8080/tcp                                              deploy_test_airflow-scheduler_1
fba0574dbdd5   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   25 minutes ago   Up 13 minutes (healthy)   8080/tcp                                              deploy_test_airflow-worker_1
d074bd03b8cd   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   25 minutes ago   Up 18 minutes (healthy)   8080/tcp                                              deploy_test_airflow-triggerer_1

hello，every docker components need same secret_key ? I tried to keep these components the same airflow.cfg,I also get the following error while trying to access its logs.

Access Forbidden while accessing log in airflow with CeleryExecutor

2 Answers