How to use Amazon SQS as Celery broker, without creating / listing queues?

user_9255304 0
Viewed 1337

For a project, my organisation is planning on shifting the Celery broker to SQS from redis. Can somebody please guide me as to how to tweak the Celery settings so that I can use a predefined SQS queue, without celery trying to create / list queues (since I do not have those permissions).

I have tried the settings below:

CELERY_BROKER_URL = 'sqs://'
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'predefined_queues':{
        'MyQueue' : {
            'url' : '<SQS Queue URL>',
        }
    }
}
CELERY_TASK_DEFAULT_QUEUE = 'MyQueue'
CELERY_ROUTES = {
    'tasks.*':{
        'queue' : 'MyQueue'
    }
}

After applying these settings I still get the following error whenever I tried to send a message to SQS queue through celery: An error occurred (AccessDenied) when calling the CreateQueue operation: Access to the resource https://queue.amazonaws.com/ is denied.

Why is celery still attempting to create a queue even when I have passed the predefined_queues setting: https://docs.celeryproject.org/en/stable/getting-started/brokers/sqs.html#predefined-queues ?

Thanks in advance!

amazon-sqs celery python-3.x
1 Answers

Celery worker(s) need to be associated to an IAM role which has CreateQueue action allowed. If your Celery workers run on EC2 instances then the simplest thing to do is to use instance-profile, and let the instance role be able to execute CreateQueue actions.

Even though my company is a heavy user of AWS (literally everything is on AWS) I suggest you think twice when you decide to use an AWS service you can't have on your workstation, and SQS is one such service.

user_8764970
Related