InitContainer not idempotent, how to prevent it from running twice?

Viewed 911

My Kubernetes deployment contains an init container:

apiVersion: apps/v1
kind: Deployment
spec:
    spec:
      initContainers:
      - name: init-stuff
        image: myregistry.azurecr.io/myinitstuff:latest
      containers:
      - name: my-actual-app
        image: myregistry.azurecr.io/myactualapp:latest
        ports:
        - containerPort: 80

The init container basically runs a few setup steps in a database (different pod).

As documentation explicitly calls out,

Because init containers can be restarted, retried, or re-executed, init container code should be idempotent. In particular, code that writes to files on EmptyDirs should be prepared for the possibility that an output file already exists.

Unfortunately, the code running in the init container is currently not idempotent. Hence, when the pod gets restarted (which can happen for a number of reasons), the init container runs again, resulting in things like duplicate rows in db tables etc.

I understand that the proper way to fix this would be to refactor the code that's running in the initContainer, making it idempotent. For example, before inserting data, check if it already exists.

Unfortunately, in my case, this is a non-trivial problem. The code running in the init container does all sorts of complex things and just wasn't ever designed to be idempotent.

So, I'm hoping to come up with a less expensive solution which doesn't require the init container code to be idempotent, but simply prevents it from running a second time should the pod get restarted.

One approach I was thinking off here involves adding a second init container which records the fact that the first init container already ran, and add some conditional logic to the first initContainer to check for that. Something like:

    spec:
      initContainers:
      - name: init-stuff
        image: myregistry.azurecr.io/myinitstuff:latest
        command: ["run-but-only-if-no-previous-completion-recorded.sh"]
      - name: record-init-stuff-completion
        image: myregistry.azurecr.io/record-init-stuff-completion:latest

This strikes me as overly complicated though. Is there really no easy way to tell Kubernetes that an init container isn't idempotent and therefore should only run once? Any other suggestions how to approach my problem?

0 Answers
Related