Create a pod from a failing k8s CronJob

user_5922350 0
Viewed 138

I have a CronJob which runs every 10 minutes and executes some command.

It's failing for some reason and I want to exec into it to investigate and fix it (There's a 1 time command I need to run to fix a shared volume*).

The issue is when I try to run exec I get this error, which is expected:

error: cannot exec into a container in a completed pod; current phase is Failed

I would like to create a new pod from the job definition and run a custom command on that (e.g. tail -f) so that it runs without crashing and I can exec into it to investigate and fix the issue.

I've been struggling to do this and have only found 2 solutions which both seem a bit hacky (I've used both and they do work, but since I'm still developing the feature I've had to reset a few times)

  1. I change the command on the k8s YAML file to tail -f then update the Helm repo and exec on the new container. Fix the issue and revert back.
  2. Copy the job to a new Pod YAML file in a directory outside of the Helm repo, with tail -f. Create it with the kubectl apply -f command. Then I can exec on it, do what I need and delete the pod.

The issue with the first is that I change the Helm repo. The second requires some duplication and adaptation of code, but it's not too bad.

What I would like is a kubectl command I can run to do this. Kind of like how you can create a job from a CronJob:

kubectl create job --from=cronjob/myjob myjob-manual

If I could do this to create a pod, or to create a job with a command which never finishes (like tail -f) it would solve my problem.

*The command I need to run is to pass in some TOTP credentials as a 1 time task to login to a service. The cookies to stay logged in will then exist on the shared volume so I won't have to do this again. I don't want to pass in the TOTP master key as a secret and add logic to interpret it either. So the most simple solution is to set up this service and once in a while I exec into the pod and login using the TOTP value again.

One more note. This is for a personal project and a tool I use for my own use. It's not a critical service I am offering to someone else so I don't mind if something goes wrong once in a while and I need to intervene.

kubernetes-helm kubernetes
1 Answers

Looked into this question more, your option 2 is the most viable solution.

Adding a sidecar container - it's the same as option 1, but even more difficult/time consuming.

As mentioned in comments, there are no options for direct imperative pod creation from job/cronjob. Available options can be checked for kubectl:

Also tried out of the interest (logic is to run the command from cronjob and then continue with specified command), but did not work out:

$ kubectl create job --from=cronjob/test-cronjob manual-job -- tail -f
error: cannot specify --from and command
user_155372010
Related