Airflow KubernetesPodOperator image pull policy pull if different SHA digest

user_6077549 0
Viewed 693

Currently using KubernetesPodOperator with default_image_policy (IfNotPresent). Will be using static tag IDs for different environments. For example, in dev env, the tag will be dev, in qa env, the tag will be qa, and so on. The issue is if there is actually a new version (different sha digest) of the image but same tag ID. I can change the image policy to Always but then it will download all the time. The Airflow DAG contain several tasks that uses KubernetesPodOperator of the same image and I don't want an image downloaded always for each of the task runs.

Is there an image policy that checks the sha digest(instead of tag ID) if exists and downloads it if not?

airflow kubernetes
1 Answers

In the containers image definition you can specify the sha to be sure it pulls/uses the correct one. For example:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu@sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]

or for Airflow KubernetesPodOperator you can use:

example = KubernetesPodOperator(
    image=ubuntu@sha256:bc2f7250f69267c9c6b66d7b6a81a54d3878bb85f1ebb5f951c896d13e6ba537,
    task_id="example_task",
    ...
)
user_148088790
Related