I've recently started a proof-of-concept to extend our Airflow to use the KubernetesPodOperator to spin up a pod in our kubernetes environment, that also hosts our airflow. This all works; however I've noticed that the logs that we get have the Info for running the task instance and the task instance success; however the stdout from the container is not captured in the log files.
I can access this information if I set the KubernetesPodOperator to leave the pod and then I can do a kubectl logs from the container and get the stdout information.
Example Log Output:
[2020-11-17 03:09:16,604] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: alex_kube_test.passing-task 2020-11-17T02:50:00+00:00 [queued]>
[2020-11-17 03:09:16,632] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: alex_kube_test.passing-task 2020-11-17T02:50:00+00:00 [queued]>
[2020-11-17 03:09:16,632] {{taskinstance.py:880}} INFO -
--------------------------------------------------------------------------------
[2020-11-17 03:09:16,632] {{taskinstance.py:881}} INFO - Starting attempt 2 of 3
[2020-11-17 03:09:16,632] {{taskinstance.py:882}} INFO -
--------------------------------------------------------------------------------
[2020-11-17 03:09:16,650] {{taskinstance.py:901}} INFO - Executing <Task(KubernetesPodOperator): passing-task> on 2020-11-17T02:50:00+00:00
[2020-11-17 03:09:16,652] {{standard_task_runner.py:54}} INFO - Started process 1380 to run task
[2020-11-17 03:09:16,669] {{standard_task_runner.py:77}} INFO - Running: ['airflow', 'run', 'alex_kube_test', 'passing-task', '2020-11-17T02:50:00+00:00', '--job_id', '113975', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/alex_kube_test.py', '--cfg_path', '/tmp/tmpmgyu498h']
[2020-11-17 03:09:16,670] {{standard_task_runner.py:78}} INFO - Job 113975: Subtask passing-task
[2020-11-17 03:09:16,745] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: alex_kube_test.passing-task 2020-11-17T02:50:00+00:00 [running]> airflow-worker-686849bf86-bpq4w
[2020-11-17 03:09:16,839] {{logging_mixin.py:112}} WARNING - /usr/local/lib/python3.6/site-packages/urllib3/connection.py:395: SubjectAltNameWarning: Certificate for us-east-1-services-kubernetes-private.vevodev.com has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
SubjectAltNameWarning,
[2020-11-17 03:09:16,851] {{logging_mixin.py:112}} WARNING - /usr/local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:330: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod`.
security_context=_extract_security_context(pod.spec.security_context)
[2020-11-17 03:09:16,851] {{logging_mixin.py:112}} WARNING - /usr/local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:77: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod` instead.
pod = self._mutate_pod_backcompat(pod)
[2020-11-17 03:09:18,960] {{taskinstance.py:1070}} INFO - Marking task as SUCCESS.dag_id=alex_kube_test, task_id=passing-task, execution_date=20201117T025000, start_date=20201117T030916, end_date=20201117T030918
What the KubeCtl Logs output returns:
uptime from procps-ng 3.3.10
Shouldn't this stdout be in the log if I have get_logs=True? How do I make sure that the logs capture the stdout of the container?
