I have a web-app that uses celery for heavy calculations. It was deployed on AWS with elastic beanstalk, using docker-compose. The broker is Amazon SQS, while redis is used as result backend.
Here is the docker-compose file to understand the structure:
version: '3.8'
volumes:
data:
driver_opts:
type: "nfs"
o: "addr=XXXXXXXXXXXXXXXXXXXXXXX,rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
device: ":/"
static:
services:
web:
build: ./web
command: >
bash -c "python3 manage.py migrate
&& python3 manage.py collectstatic --noinput
&& gunicorn APPNAME.asgi:application --access-logfile '-' --error-logfile '-' --capture-output --enable-stdio-inheritance -k uvicorn.workers.UvicornWorker -b 0.0.0.0:8000"
volumes:
- data:/home/webserver/data/
- static:/home/webserver/static/
ports:
- "8000:8000"
depends_on:
- redis
celery:
build: ./web
command: celery -A APPNAME worker -Ofair -l INFO
volumes:
- data:/home/webserver/data/
depends_on:
- web
redis:
image: redis:6.2.6-alpine
ports:
- "6379:6379"
nginx:
build : ./nginx
ports:
- "80:80"
depends_on:
- web
volumes:
- static:/home/webserver/static/
It works well, including the celery tasks. Now, I would like to separate the web/redis/nginx services from the celery containers that would have their own dedicated instances, with the ability to scale. Is there a proper way to do that (using CLI)? Should I consider separate beanstalk applications?