Airflow KubernetePodOperator fails

Viewed 26

I'm trying to setup a KubernetesPodOperator in my dag so I can run some task in my EKS cluster. The operator points to an image I published in ECR. So I used the following command to create the ecr secret:

kubectl create secret docker-registry ecr --docker-server=xxxx.dkr.ecr.$region.amazonaws.com --docker-username=AWS --docker-password=$(shell aws ecr get-login-password --region $region)

My DAG code goes below:

import os
from airflow import DAG
from airflow.kubernetes.secret import Secret
from datetime import datetime, timedelta
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)

with DAG(
    dag_id="eks_poc",....
) as dag:
    eks_k8s = KubernetesPodOperator(
        namespace='eks-poc-namespace',
        cluster_context="arn:aws:eks:$region:xxxx:cluster/eks-poc",
        image='xxxx.dkr.ecr.$region.amazonaws.com/python-app',
        image_pull_secrets=[k8s.V1LocalObjectReference('ecr')],
        cmds=["python3", "/opt/app/app.py"],
        labels={"FOO": "Val"},
        name="eks-poc-pod",
        is_delete_operator_pod=True,
        in_cluster=False,
        task_id="task-two",
        get_logs=True,
        do_xcom_push=True
    )

The task failed with error below:

[2022-09-14 05:03:40,124] {refresh_config.py:71} ERROR - [Errno 2] No such file or directory: 'aws': 'aws'
[2022-09-14 05:03:40,758] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 350, in execute
    pod_list = client.list_namespaced_pod(self.namespace, label_selector=label_selector)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
    (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 12905, in list_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1f2b7b45-4d1d-4178-9c8c-8709c16fbe32', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '0fb27af0-e840-4042-9fc4-f4d4fe05c643', 'X-Kubernetes-Pf-Prioritylevel-Uid': '89864181-64fb-47b2-8394-fcb2f0cbb89e', 'Date': 'Wed, 14 Sep 2022 05:03:40 GMT', 'Content-Length': '279'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"eks-poc-namespace\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

It appears that airflow doesn't know how to authenticate to EKS/ECR. My local kuerbernete configure is at default location ~/.kube/config and commands like kubectl get svc works fine.

Could anyone advise, please?

0 Answers
Related