How to have dedicated instances for celery containers when deploying django project using docker-compose on elastic beanstalk?

Viewed 57

I have a web-app that uses celery for heavy calculations. It was deployed on AWS with elastic beanstalk, using docker-compose. The broker is Amazon SQS, while redis is used as result backend.

Here is the docker-compose file to understand the structure:

version: '3.8'

volumes:
  data:
    driver_opts:
      type: "nfs"
      o: "addr=XXXXXXXXXXXXXXXXXXXXXXX,rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
      device: ":/"
  static:
services:
    web:
        build: ./web
        command: >
            bash -c "python3 manage.py migrate
            && python3 manage.py collectstatic  --noinput
            && gunicorn APPNAME.asgi:application --access-logfile '-' --error-logfile '-' --capture-output --enable-stdio-inheritance -k uvicorn.workers.UvicornWorker -b 0.0.0.0:8000"
        volumes:
          - data:/home/webserver/data/
          - static:/home/webserver/static/
        ports:
          - "8000:8000"
        depends_on:
          - redis
    celery:
        build: ./web
        command: celery -A APPNAME worker -Ofair -l INFO
        volumes:
          - data:/home/webserver/data/
        depends_on:
          - web
    redis:
        image: redis:6.2.6-alpine
        ports:
          - "6379:6379"
    nginx:
        build : ./nginx
        ports:
          - "80:80"
        depends_on:
          - web
        volumes:
      - static:/home/webserver/static/

It works well, including the celery tasks. Now, I would like to separate the web/redis/nginx services from the celery containers that would have their own dedicated instances, with the ability to scale. Is there a proper way to do that (using CLI)? Should I consider separate beanstalk applications?

0 Answers
Related