How do I run delayed_jobs in Kubernetes?

Question

I can log into console from one of the pods (on kubernetes) and run this command:

RAILS_ENV=production bin/delayed_job start

The jobs are run correctly doing that. However when the pods are deleted or restarted, the jobs stop running.

I also tried adding the command above in an initializer file (eg config/initializers/delayed_jobs_runner.rb), but I get a recursive loop when starting the app.

Another thing I tried to do is create a new file called my-jobs.yaml with this

apiVersion: batch/v1
kind: Job
metadata:
  name: job
spec:
  template:
    spec:
      containers:
      - name: job
        image: gcr.io/test-app-123/somename:latest
        command: ["/bin/bash", "-l", "-c"]
        args: ["RAILS_ENV=production bundle exec rake jobs:work"]
      restartPolicy: Never
  backoffLimit: 4

I then do kubectl apply -f my-jobs.yaml, but the jobs aren't running.

Any idea how to run delayed_jobs correctly in kubernetes?

EDIT: Here's my Dockerfile:

FROM gcr.io/google_appengine/ruby

# Install 2.5.1 if not already preinstalled by the base image
RUN cd /rbenv/plugins/ruby-build && \
    git pull && \
    rbenv install -s 2.5.1 && \
    rbenv global 2.5.1 && \
    gem install -q --no-rdoc --no-ri bundler 
    # --version 1.11.2

ENV RBENV_VERSION 2.5.1

# Copy the application files.
COPY . /app/

# Install required gems.
RUN bundle install --deployment && rbenv rehash

# Set environment variables.
ENV RACK_ENV=production \
    RAILS_ENV=production \
    RAILS_SERVE_STATIC_FILES=true

# Run asset pipeline.
RUN bundle exec rake assets:precompile


CMD ["setup.sh"]


# Reset entrypoint to override base image.
ENTRYPOINT ["/bin/bash"]




################### setup.sh ############################
cd /app && RAILS_ENV=production bundle exec script/delayed_job -n 2 start
bundle exec foreman start --formation "$FORMATION"
#########################################################

user_8993347 · Answer

Running multiple processes in one docker container is problematic as you cannot easily observe lifetime of particular process - every container need one process which is "main" and when it exit, container also exit.

As looking on Github (https://github.com/collectiveidea/delayed_job#user-content-running-jobs) I would strongly suggest to change a little your starting command to run it in foreground because now when you are starting Kubernetes job with daemons - job is ending immediately as docker container lifetime is directly related to "main" foreground process lifetime so when you run only background process your main process exit immediately and your container too.

Change your command to:

RAILS_ENV=production script/delayed_job run

What start worker in foreground so your Kubernetes Job won't exit. Please note also that Kubernetes Jobs are not intended to such infinitive tasks (job should has start and end) so I would suggest to use ReplicaSet for that

user_1899716 · Answer

Now I am doing this:

this_pid=$$
(while [[ $(ps -ef | grep delayed_job | grep -v -e grep -e tail | head -c1 | wc -c) -ne 0 ]]; do sleep 10; done; kill -- -$this_pid) &

after starting multiple workers. And after this I tail -f the logs so that those go to the standard output of the container. I am quite crazy, so I am also running logrotate to keep the logs in check. The rails environment is pretty big anyway, so the container needs to be pretty big, and we need to be able to run many jobs and I don't want many pods running to do so. This seems to be efficient and will stop and restart if the workers die for some reason.

How do I run delayed_jobs in Kubernetes?

2 Answers