I have 3 docker containers - my_django_app, rabbitmq, and celery_worker. I have implemented it on my local system using docker-compose.yml which is as follows:
version: '3'
services:
web: &my_django_app
build: .
command: python3 manage.py runserver 0.0.0.0:8000
ports:
- "80:8000"
depends_on:
- rabbitmq
rabbitmq:
image: rabbitmq:latest
celery_worker:
<<: *my_django_app
command: celery -A MyDjangoApp worker --autoscale=10,1 --loglevel=info
ports: []
depends_on:
- rabbitmq
When I run this on my local system, it works perfectly fine. And then I deployed these images to AWS Elastic Beanstalk (Multi container environment) using Dockerrun.aws.json which is as follows:
{
"AWSEBDockerrunVersion": 2,
"Authentication": {
"Bucket": "cred-keeper",
"Key": "index.docker.io/.dockercfg"
},
"containerDefinitions": [{
"Authentication": {
"Bucket": "cred-keeper",
"Key": "index.docker.io/.dockercfg"
},
"command": [
"celery",
"-A",
"MyDjangoApp",
"worker",
"--autoscale=10,1",
"--loglevel=info"
],
"essential": true,
"image": "myName/my_django_app:latest",
"name": "celery_worker",
"memory": 150
},
{
"essential": true,
"image": "rabbitmq:latest",
"name": "rabbitmq",
"memory": 256,
},
{
"Authentication": {
"Bucket": "cred-keeper",
"Key": "index.docker.io/.dockercfg"
},
"command": [
"python3",
"manage.py",
"runserver",
"0.0.0.0:8000"
],
"essential": true,
"image": "myName/my_django_app:latest",
"memory": 256,
"name": "web",
"portMappings": [{
"containerPort": 8000,
"hostPort": 80
}]
}
],
"family": "",
"volumes": []
}
I saw the logs for the 3 containers by downloading the logs from AWS Elastic Beanstalk, and the containers web as well as rabbitmq are working just fine, but celery_worker shows logs like:
[2020-06-30 20:17:22,885: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq:5672//: failed to resolve broker hostname.
Trying again in 2.00 seconds... (1/100)
[2020-06-30 20:17:24,898: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq:5672//: failed to resolve broker hostname.
Trying again in 4.00 seconds... (2/100)
[2020-06-30 20:17:28,914: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbitmq:5672//: failed to resolve broker hostname.
Trying again in 6.00 seconds... (3/100)
.
.
.
[2020-06-30 20:16:45,662: CRITICAL/MainProcess] Unrecoverable error: OperationalError('failed to resolve broker hostname')
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 137, in _connect
host, port, family, socket.SOCK_STREAM, SOL_TCP)
File "/usr/local/lib/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
yield
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 430, in ensure_connection
callback, timeout=timeout)
File "/usr/local/lib/python3.7/site-packages/kombu/utils/functional.py", line 344, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 283, in connect
return self.connection
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 839, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 794, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.7/site-packages/amqp/connection.py", line 311, in connect
self.transport.connect()
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 77, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 148, in _connect
"failed to resolve broker hostname"))
OSError: failed to resolve broker hostname
My CELERY_BROKER_URL in my Django's settings is "amqp://rabbitmq". Also, my celery.py is as:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Speeve.settings')
app = Celery('MyDjangoApp')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
@app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
What do I need to do in order for my celery container to work properly on AWS Elastic Beanstack?
Please help!