celery for separate django docker microservices

updated

I have 2 django microservices. main_ms microservice with config project and Users app.

Oms1_ms microservice with config project and Oms1 app.

redis==4.3.4
celery==5.2.7
celery[redis]==5.2.7

both have celery.py. main_ms's is at Users app and is like this

from celery import Celery
app = Celery ('Users', broker='amqp://admin:mypass@rabbit:5672', backend='redis://redis:6379')

and for Oms1_ms, is at Oms1 app. instead of Users I have Oms1

so celery instances are named Users and Oms1.

in both __init__.pys I have

from .celery import app as celery_app
__all__ = ('celery_app')

in main_ms\Users\views.py which calls the other celery

from .celery import app
from rest_framework import views
from rest_framework import status
from rest_framework.response import Response
class Oms1_Oms1ListView(views.APIView):
    def get(self, request, *args, **kwargs):
        remoteResponse = app.send_task('Oms1.fun')
        print(remoteResponse)
        return Response({}, status=status.HTTP_200_OK)

in Oms1_ms\Oms1\tasks.py I have

from .celery import app
@app.task()
def fun(x):
    print('in fun')
    return {'sd':'as always'}

thanks to @def soudani now I run docker-compose then in main_ms container I run celery -A Users.celery worker --loglevel=info

and in Oms1_ms container I run celery -A Oms1.celery worker --loglevel=info

then message rates in rabbitmq panel become like the picture below.

so they both get connected and there is no error or waring.

but when I make an http request to view(Oms1_Oms1ListView), the other function fun is not called. in fun is not printed and in rabbitmq panel I see: there is no messages sent in queue messages.

and message rates are no different and as before.

question

so how to set celery with django docker microservices correctly? where I have gone wrong?

I also know one of celery uses is primarily to process workloads within a service. but finding an example of celery for microservices with conditions below has been a cumbersome task to me.

1. all django projects are separate, each on their own container and they don't share a parent folder, so when calling tasks we cant import the function handling it unlike the @shared_task of celery.
2. it has depicted how to run celery worker command.
3. how to setup and attach celery instances to rabbit/redis or mongodb.

is it possible to do these with celery? if not what options are available to achieve these?

docker-compose

I find it extravagant to put docker-compose but anyway.

version: "3.9"
services:
  # Redis
  redis:
    image: redis:7.0.4-alpine
    container_name: redis
    ports:
      - "6379:6379"
  # rabbit
  rabbit:
    hostname: rabbit
    image: rabbitmq:3.10.7-management-alpine
    container_name: rabbit_container
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS=mypass
    ports:
      - 15672:15672
      - 5672:5672
    depends_on:
      - redis
  # Main Database Postgres
  main_postgres_ser:
    image: postgres:14.4-alpine
    container_name: main_postgres_container
    environment:
      - POSTGRES_DB=${main_DB_NAME}
      - POSTGRES_USER=${main_DB_USER}
      - POSTGRES_PASSWORD=${main_DB_PASSWORD}
    volumes:
      - ./data/main_ms:/var/lib/postgresql/data
    restart: always
    ports:
      - 5432:5432 # DB port
    depends_on:
      - rabbit
  # Main Django Application
  main_django_ser:
    container_name: main_django_container
    build:
      context: ./main_ms
      dockerfile: Dockerfile.main
    restart: on-failure
    command:
      sh -c "
      python manage.py runserver 0.0.0.0:8000"
    environment:
      - PYTHONUNBUFFERED=1
    ports:
      - 8000:8000
    volumes:
      - ./main_ms:/main_ms
    depends_on:
      - main_postgres_ser
  oms1_postgres_ser:
    image: postgres:14.4-alpine
    container_name: Oms1_postgres_container
    environment:
      - POSTGRES_DB=${Oms1_DB_NAME}
      - POSTGRES_USER=${Oms1_DB_USER}
      - POSTGRES_PASSWORD=${Oms1_DB_PASSWORD}
    volumes:
      - ./data/Oms1:/var/lib/postgresql/data
    restart: always
    ports:
      - 5433:5432 # DB port
    depends_on:
      - rabbit
  # Oms1 Django Application
  oms1_django_ser:
    container_name: Oms1_django_container
    build:
      context: ./Oms1_ms
      dockerfile: Dockerfile.Oms1
    restart: on-failure
    command:
      sh -c "
      python manage.py runserver 0.0.0.0:8000"
    environment:
      - PYTHONUNBUFFERED=1
    ports:
      - 8001:8000
    volumes:
      - ./Oms1_ms:/Oms1_ms
    depends_on:
      - oms1_postgres_ser
volumes:
  main_postgres_ser:
    driver: local
  oms1_postgres_ser:
    driver: local