How can I keep a Pod from crashing so that I can debug it?

Redefine the command

In development only, a temporary hack to keep a Kubernetes pod from crashing is to redefine it and specify the container's command (corresponding to a Docker ENTRYPOINT) and args to be a command that will not crash. For instance:

  containers:
  - name: something
    image: some-image
    # `shell -c` evaluates a string as shell input
    command: [ "sh", "-c"]
    # loop forever, outputting "yo" every 5 seconds
    args: ["while true; do echo 'yo' && sleep 5; done;"]

This allows the container to run and gives you a chance to shell into it, like kubectl exec -it pod/some-pod -- sh, and investigate what may be wrong.

This needs to be undone after debugging so that the container will run the command it's actually meant to run.

Adapted from this blog post.

There are also other methods used for debugging pods that are worth noting in your use case scenario:

• If your container has previously crashed, you can access the previous container's crash log with: kubectl logs --previous ${POD_NAME} ${CONTAINER_NAME}

• Debugging with an ephemeral debug container: Ephemeral containers are useful for interactive troubleshooting when kubectl exec is insufficient because a container has crashed or a container image doesn't include debugging utilities, such as with distroless images. kubectl has an alpha command that can create ephemeral containers for debugging beginning with version v1.18. An example for this method can be found here.

in my case I did build using mac m1/silicon. In this case pod crashes and there is no explicit message about this.

The problem was that I a also debugged using docker on the same m1 so could not really see what is wrong.

I would need to build image using docker build --platform linux/amd64.