Celery unavailable after dockerization in a Django app

user_16050071 0
Viewed 41

I have a Django app that I've been trying to dockerize. I've successfully dockerized Django with gunicorn and nginx so the main part of the app is running.

However, I also have tasks that need to be run using Celery. I dockerized rabbitmq, and I think that was successful, as before installing it I had Connexion refused and I don't anymore.

I only lack Celery to be dockerized so, as my tasks are not executed nor stored in the database, my Celery configuration is probably wrong, however I couldn't find where I'm going wrong with it.

Here is my docker-compose.yml:

version: '3.8'

services:
  django_gunicorn:
    volumes:
      - static:/app/static
      - media:/media
    env_file:
      - env
    build:
      context: .
    ports: 
      - "8000:8000"

  nginx:
    build: ./nginx
    volumes:
      - static:/static
      - media:/media
    ports:
      - "80:80"
    depends_on:
      - django_gunicorn

  rabbitmq3:
    image: rabbitmq:3-alpine
    ports:
      - 5672:5672

  celery:
    restart: always
    build:
      context: .
    command: celery -A main worker -l info
    env_file:
      - env
    depends_on:
      - rabbitmq3
      - django_gunicorn


volumes:
  static:
  media:

Dockerfile:

FROM python:3.10.5-alpine

RUN pip install --upgrade pip

RUN wget https://upload.wikimedia.org/wikipedia/commons/b/b9/First-google-logo.gif -O media/media.gif

COPY ./requirements.txt .
RUN pip install -r requirements.txt
COPY ./src /app

WORKDIR /app

COPY ./entrypoint.sh /
ENTRYPOINT ["sh", "/entrypoint.sh"]

entrypoint.sh:

#!/bin/sh

python manage.py migrate
python manage.py collectstatic --no-input

gunicorn main.wsgi:application --bind 0.0.0.0:8000

Celery container logs

C:\Users\stephane.bernardelli\Documents\TestApp>docker logs 60b3767e9845
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, django_celery_beat, django_celery_results, scripts, sessions
Running migrations:
  No migrations to apply.

9771 static files copied to '/app/static'.
[2022-07-22 08:32:33 +0000] [9] [INFO] Starting gunicorn 20.1.0
[2022-07-22 08:32:33 +0000] [9] [INFO] Listening at: http://0.0.0.0:8000 (9)
[2022-07-22 08:32:33 +0000] [9] [INFO] Using worker: sync
[2022-07-22 08:32:33 +0000] [10] [INFO] Booting worker with pid: 10
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, django_celery_beat, django_celery_results, scripts, sessions
Running migrations:
  No migrations to apply.

0 static files copied to '/app/static', 9771 unmodified.
[2022-07-22 08:55:07 +0000] [9] [INFO] Starting gunicorn 20.1.0
[2022-07-22 08:55:07 +0000] [9] [INFO] Listening at: http://0.0.0.0:8000 (9)
[2022-07-22 08:55:07 +0000] [9] [INFO] Using worker: sync
[2022-07-22 08:55:07 +0000] [10] [INFO] Booting worker with pid: 10
docker-compose docker celery django python
1 Answers

I think the reason is because your containers are unable to talk to each other. By default all the containers are in bridge network and this issue arises. You can try grouping all the containers in a network as shown below.

version: '3.8'

services:
  django_gunicorn:
    volumes:
      - static:/app/static
      - media:/media
    env_file:
      - env
    build:
      context: .
    ports: 
      - "8000:8000"
    networks:
      - dev_network

  nginx:
    build: ./nginx
    volumes:
      - static:/static
      - media:/media
    ports:
      - "80:80"
    depends_on:
      - django_gunicorn
    networks:
      - dev_network

  rabbitmq3:
    image: rabbitmq:3-alpine
    ports:
      - 5672:5672
    networks:
      - dev_network

  celery:
    restart: always
    build:
      context: .
    command: celery -A main worker -l info
    env_file:
      - env
    depends_on:
      - rabbitmq3
      - django_gunicorn
    networks:
      - dev_network

networks:
  dev_network:
    external: false

volumes:
  static:
  media:

And in your celery configuration for broker URL, you need to specify it in this format.

amqp://username:password@rabbitmq3:5672/

in your entrypoint.sh add the below line at the end so that docker-compose commands also gets executed.

exec "$@"

or you can remove the Entrypoint from DockerFile and do as below.

for gunicorn container command in docker-compose.yml

command:
      [sh, -c, "python manage.py migrate && gunicorn main.wsgi:application --bind 0.0.0.0:8000 "]

Note : collect the static file before hand or you can append it in the above command

for celery container command in docker-compose.yml remains same.

Note: using the second method is not an issue in this case. But if you have some complex validation or task to perform before container spin up, go with Entrypoint script only

user_150727040
Related