Airflow Kubernetes DAG creates pod but stop running on queued

Viewed 552

I have an Airflow 1.10.15 (a.k.a. Bridge Release) in my AWS Kubernetes cluster. It uses KubernetesExecutor.

I have a Hello World KubernetesExecutor DAG which should print Hello World. When triggering the DAG, it creates a pod but it never prints the Hello World.

Here are all the logs after the pod has been completed running: enter image description here

Describing the pod will give me logs which has no errors or failures: enter image description here

1 Answers

You should check Task logs, not Kubernetes logs. Kubernetes logs keep information about "atempting to run" the task (and looks that it's all ok here).

Now, when you log anything in the running tasks, it does not go to the k8S logs - it goes to task logs. By default, when you configure Airflow, the logs for tasks are stored separately - basically every task has its own log. This is in order so that you can pull the logs and see them in Airflow UI when you click on "logs" for this particular task execution.

Just check it in the UI or in the "${AIRFLOW_HOME}/logs" folder.

Related