How to deploy airflow via Helm on Kubernetes while using a custom image?

user_11271048 0
Viewed 1302

My objective is to be able to deploy airflow on Kubernetes using a custom image (placed in ECR. The reason I want to use this custom Image is because I want to deploy another tool (dbt) with airflow in the same container (also open for other suggestions there)

What actually worked: I have managed to use this Helm chart (which uses the following image as default) to deploy

What I tried to do and did not work: I wanted to now exchange the default image with my custom image in ECR, so I created values.yaml, that contains:

airflow:
  image:
    repository: 000000000.dkr.ecr.eu-central-1.amazonaws.com/foo/meow
    tag: latest

And then ran:

helm upgrade airflow-pod airflow-stable/airflow --version "7.14.0" --values values.yaml

Which I expected to override the default yaml and pull the image from ECR instead. I then ran describe pod airflow-pod, and found the following log of the error (snippet):

  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned 000000000.dkr.ecr.eu-central-1.amazonaws.com/foo/meow to ip-10-0-0-0eu-central-1.compute.internal
  Normal   Pulling    12m                   kubelet            Pulling image "000000000.dkr.ecr.eu-central-1.amazonaws.com/foo/meow:latest"
  Normal   Pulled     11m                   kubelet            Successfully pulled image "000000000.dkr.ecr.eu-central-1.amazonaws.com/foo/meow:latest"
  Normal   Created    9m39s (x5 over 11m)   kubelet            Created container airflow-web
  Normal   Pulled     9m39s (x4 over 11m)   kubelet            Container image "000000000.dkr.ecr.eu-central-1.amazonaws.com/foo/meow:latest" already present on machine
  Warning  Failed     9m38s (x5 over 11m)   kubelet            Error: failed to start container "airflow-web": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/usr/bin/dumb-init": stat /usr/bin/dumb-init: no such file or directory: unknown

What I have tried to confirm/fix the issue

First, I tried to see if it is an ECR issue. So I put the same original image in ECR (instead of my own image with dbt), and found that the same error above persists.

Second, I digged and found the following question that makes me think I can't use that airflow helm chart from an ECR repo (non-official one).

One last approach I took as an alternative path: I tried to use the chart on the Apache airflow repo like:

helm install airflow . --namespace airflow-deploy --set executor=CeleryExecutor --set workers.keda.enabled=true --set workers.persistence.enabled=false

But I got the error:

Error: failed post-install: timed out waiting for the condition
airflow kubernetes
1 Answers

The original image seems to have "dumb-init" binary in, so it should work. However, if you use "imagePullPolicy: IfNotPresent" then Kubernetes might cache the image and even if you re-upload a new image to ECR it might not be pulled (though I believe for latest it should be, unless some custom configuration of the Kubernetes is in place.

See https://kubernetes.io/docs/concepts/containers/images/#updating-images

You can always run the image and check it's content locally. The official Docker image of Airflow has support for bash command:

docker run -it apache/airflow:1.10.12-python3.6 bash
airflow@18278a339579:/opt/airflow$ /usr/bin/dumb-init --help
dumb-init v1.2.2
Usage: /usr/bin/dumb-init [option] command [[arg] ...]
dumb-init is a simple process supervisor that forwards signals to children.
It is designed to run as PID1 in minimal container environments.
Optional arguments:
   -c, --single-child   Run in single-child mode.
                        In this mode, signals are only proxied to the
                        direct child and not any of its descendants.
   -r, --rewrite s:r    Rewrite received signal s to new signal r before proxying.
                        To ignore (not proxy) a signal, rewrite it to 0.
                        This option can be specified multiple times.
   -v, --verbose        Print debugging information to stderr.
   -h, --help           Print this help message and exit.
   -V, --version        Print the current version and exit.
Full help is available online at https://github.com/Yelp/dumb-init
user_5167010
Related