Celery tasks don't run in docker container

Viewed 3042

In Django, I want to perform a Celery task (let's say add 2 numbers) when a user uploads a new file in /media. What I've done is to use signals so when the associated Upload object is saved the celery task will be fired.

Here's my code and Docker configuration:

signals.py

from django.db.models.signals import post_save
from django.dispatch import receiver
from core.models import Upload
from core.tasks import add_me

def upload_save(sender, instance, signal, *args, **kwargs):
    print("IN UPLOAD SIGNAL") # <----- LOGS PRINT UP TO HERE, IN CONTAINERS
    add_me.delay(10)    

post_save.connect(upload_save, sender=Upload) # My post save signal

tasks.py

from celery import shared_task

@shared_task(ignore_result=True, max_retries=3)    
def add_me(upload_id):
    print('In celery') # <----- This is not printed when in Docker!
    return upload_id + 20

views.py

class UploadView(mixins.CreateModelMixin, generics.GenericAPIView):

    serializer_class = UploadSerializer

    def post(self, request, *args, **kwargs):
        serializer = UploadSerializer(data=request.data)
        print("SECOND AFTER")
        print(request.data) <------ I can see my file name here
        if serializer.is_valid():
            print("THIRD AFTER") <------ This is printer OK in all cases
            serializer.save()
            print("FOURTH AFTER") <----- But this is not printed when in Docker!
            return response.Response(
                {"Message": "Your file was  uploaded"},
                status=status.HTTP_201_CREATED,
            )

        return response.Response(
            {"Message": "Failure", "Errors": serializer.errors},
            status=status.HTTP_403_FORBIDDEN,
        )

docker-compose.yml

version: "3.8"
services:
  db:
    # build: ./database_docker/
    image: postgres
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: test_db
      POSTGRES_USER: test_user
      POSTGRES_PASSWORD: test_pass
    # volumes:
    #   - media:/code/media

  web:
    build: ./docker/
    command: bash -c "python manage.py migrate --noinput && python manage.py runserver 0.0.0.0:8000"
    volumes:
      - .:/code
      - media:/code/media
    ports:
      - "8000:8000"
    depends_on:
      - db

  rabbitmq:
    image: rabbitmq:3.6.10
    volumes:
      - media:/code/media

  worker:
    build: ./docker/
    command: celery -A example_worker worker --loglevel=debug -n worker1.%h
    volumes:
      - .:/code
      - media:/code/media
    depends_on:
      - db
      - rabbitmq
volumes:
  media: 

Dockerfile

FROM python:latest
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN pip3 install -r requirements.txt
COPY . /code/
WORKDIR /code

Everything works OK when not in Docker.

The problem is that when I'm deploying the above in Docker and try to upload a file, the request never finishes even-though the file is uploaded in the media folder (confirmed it by accessing its contents in both the web and worker containers).

More specifically it seems that the Celery task is not executed (finished?) and the code after the serializer.save() is never reached.

When I remove the signal (thus no Celery task is fired) everything is OK. Can someone please help me?

1 Answers

I just figured it out. Turns out that I need to add the following in the __init__.py of my application.

from .celery import app as celery_app

__all__ = ("celery_app",)

Don't know why everything is running smoothly without this piece of code when I'm not using containers...

Related