Add registry prefix to all images with kustomize image transformer

user_398670 0
Viewed 334

A common requirement when deploying Kubernetes manifests to a cluster is to prefix the container names with a trusted registry prefix that mirrors the allowed images. Usually used along with an admission controller.

Is there a sensible way to do this using Kustomize without having to list every single image by name in the kustomization.yaml images: transformer stanza?

Given this kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - "https://github.com/prometheus-operator/kube-prometheus"

if I want to prefix all the images it references with mytrusted.registry/ I need to append this to my kustomization.yaml:

images:
- name: grafana/grafana
  newName: mytrusted.registry/grafana/grafana
- name: jimmidyson/configmap-reload
  newName: mytrusted.registry/jimmidyson/configmap-reload
- name: k8s.gcr.io/kube-state-metrics/kube-state-metrics
  newName: mytrusted.registry/k8s.gcr.io/kube-state-metrics/kube-state-metrics
- name: k8s.gcr.io/prometheus-adapter/prometheus-adapter
  newName: mytrusted.registry/k8s.gcr.io/prometheus-adapter/prometheus-adapter
- name: quay.io/brancz/kube-rbac-proxy
  newName: mytrusted.registry/quay.io/brancz/kube-rbac-proxy
- name: quay.io/prometheus/alertmanager
  newName: mytrusted.registry/quay.io/prometheus/alertmanager
- name: quay.io/prometheus/blackbox-exporter
  newName: mytrusted.registry/quay.io/prometheus/blackbox-exporter
- name: quay.io/prometheus/node-exporter
  newName: mytrusted.registry/quay.io/prometheus/node-exporter
- name: quay.io/prometheus-operator/prometheus-operator
  newName: mytrusted.registry/quay.io/prometheus-operator/prometheus-operator
- name: quay.io/prometheus/prometheus
  newName: mytrusted.registry/quay.io/prometheus/prometheus

which I generated with this putrid, fragile monstrosity (which WILL break if your containers are specified by hash, or you have a port in your registry prefix):

kustomize build | \
  grep 'image:' | \
  awk '$2 != "" { print $2}' | \
  sort -u | \
  cut -d : -f 1 | \
  jq --raw-input '{ name: ., newName: ("mytrusted.registry/" + .) }' | \
  faq -s -fjson -oyaml '{ images: .}'

(Note that the above will also NOT WORK completely, because Kustomize doesn't recognise images outside PodTemplates, such as those in the kind: Alertmanager spec.image or the kind: Prometheus spec.image; it'd still be better than the current situation).

What I want instead is to able to express this in the image transformer without generating and maintaining lists of images, with something like the imaginary, does not work example:

images:
  - name: "(*)"
    newName: "mytrusted.registry/$1"

i.e. use a capture group. Or something functionally similar, like an image transformer "prependName" option or similar.

This must be such a common problem to have, but I can't for the life of me find a well established way this is done by convention in the k8s world. Just lots of DIY fragile hacks.

kustomize kubernetes
1 Answers

This answer is probably too late to help the original asker, but maybe it will help others who stumble upon this question through Google, like I did.

Kustomize has a built-in PrefixTransformer that can add a prefix to all your images, or indeed to any arbitrary field in your specs.

Create a file named image-prefix.yaml with the following contents:

apiVersion: builtin
kind: PrefixTransformer
metadata:
  name: image-prefix

prefix: mytrusted.registry/

fieldSpecs:
- path: spec/template/spec/containers/image
- path: spec/template/spec/initContainers/image
- path: spec/image  # for kind Prometheus and Alertmanager

Then add this transformer to your kustomization.yaml as follows:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- "https://github.com/prometheus-operator/kube-prometheus"

transformers:
- image-prefix.yaml

That should do it.

When you build this, you should see your prefix automatically added to all the images:

$ kubectl kustomize | grep image:
...
        image: mytrusted.registry/quay.io/prometheus/blackbox-exporter:v0.22.0
        image: mytrusted.registry/jimmidyson/configmap-reload:v0.5.0
        image: mytrusted.registry/quay.io/brancz/kube-rbac-proxy:v0.13.0
        image: mytrusted.registry/grafana/grafana:my-tag
        image: mytrusted.registry/k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.6.0
...

I tested this with kubectl 1.25 and the version of Kustomize that comes bundled with it:

$ kubectl version --short --client
...
Client Version: v1.25.0
Kustomize Version: v4.5.7

You can further restrict the PrefixTransformer by using GVK (group/version/kind) triplets. For example, if for some reason you wanted to apply your image prefix only to Deployments, but not to DaemonSets, StatefulSets, or others, you would put something like this in your image-prefix.yaml file:

fieldSpecs:
- kind: Deployment
  path: spec/template/spec/containers/image
- kind: Deployment
  path: spec/template/spec/initContainers/image

Also note that the ImageTransformer runs before the PrefixTransformer, so if you wanted to override the tag of a particular image in your kustomization.yaml, you should use the original image name without the prefix:

images:
- name: grafana/grafana
  newTag: my-tag

Unfortunately there is no clear documentation for PrefixTransformer that I could find, or I would have linked it here. I discovered all this by digging through Kustomize source code.

There are quite a few other built-in transformers that might be of interest, you can glean their usage by looking at the *_test.go files in each of the subfolders here:

https://github.com/kubernetes-sigs/kustomize/tree/master/plugin/builtin

user_20581840
Related