I'm trying to setup a KubernetesPodOperator in my dag so I can run some task in my EKS cluster. The operator points to an image I published in ECR. So I used the following command to create the ecr secret:
kubectl create secret docker-registry ecr --docker-server=xxxx.dkr.ecr.$region.amazonaws.com --docker-username=AWS --docker-password=$(shell aws ecr get-login-password --region $region)
My DAG code goes below:
import os
from airflow import DAG
from airflow.kubernetes.secret import Secret
from datetime import datetime, timedelta
from kubernetes.client import models as k8s
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
with DAG(
dag_id="eks_poc",....
) as dag:
eks_k8s = KubernetesPodOperator(
namespace='eks-poc-namespace',
cluster_context="arn:aws:eks:$region:xxxx:cluster/eks-poc",
image='xxxx.dkr.ecr.$region.amazonaws.com/python-app',
image_pull_secrets=[k8s.V1LocalObjectReference('ecr')],
cmds=["python3", "/opt/app/app.py"],
labels={"FOO": "Val"},
name="eks-poc-pod",
is_delete_operator_pod=True,
in_cluster=False,
task_id="task-two",
get_logs=True,
do_xcom_push=True
)
The task failed with error below:
[2022-09-14 05:03:40,124] {refresh_config.py:71} ERROR - [Errno 2] No such file or directory: 'aws': 'aws'
[2022-09-14 05:03:40,758] {taskinstance.py:1482} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 350, in execute
pod_list = client.list_namespaced_pod(self.namespace, label_selector=label_selector)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
(data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 12905, in list_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 345, in call_api
_preload_content, _request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
_request_timeout=_request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 366, in request
headers=headers)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 241, in GET
query_params=query_params)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '1f2b7b45-4d1d-4178-9c8c-8709c16fbe32', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '0fb27af0-e840-4042-9fc4-f4d4fe05c643', 'X-Kubernetes-Pf-Prioritylevel-Uid': '89864181-64fb-47b2-8394-fcb2f0cbb89e', 'Date': 'Wed, 14 Sep 2022 05:03:40 GMT', 'Content-Length': '279'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"eks-poc-namespace\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
It appears that airflow doesn't know how to authenticate to EKS/ECR. My local kuerbernete configure is at default location ~/.kube/config and commands like kubectl get svc works fine.
Could anyone advise, please?