How to securely connect to Azure redis from celery-django project?

Question

I am trying to convert from a local Redis container to a managed service in Azure.

I know I have the correct server name and key because

redis-cli -h .redis.cache.windows.net -p 6379 -a

connects.

Locally my connection before in celery_manager.py was

app = Celery(broker=redis://redis:6379)

I updated broker successfully with the non-ssl port enabled.

broker=redis://:@.redis.cache.windows.net:6379 per [this question]

I tried updating the broker to:

broker=redis://:@.redis.cache.windows.net:6380

I got this warning in Celery log:

[2020-12-03 20:54:00,491: WARNING/celery] Secure redis scheme specified (rediss) with no ssl options, defaulting to insecure SSL behaviour.

And this exception in django-tasks:

020-12-03 20:54:31,223 - INFO - runworker - Using single-threaded worker.
2020-12-03 20:54:31,224 - INFO - runworker - Running worker against channel layer default (asgi_redis.core.RedisChannelLayer)
2020-12-03 20:54:31,225 - INFO - worker - Listening on channels http.request, websocket.connect, websocket.disconnect, websocket.receive
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 185, in _read_from_socket
raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/src/manage.py", line 22, in 
    execute_from_command_line(sys.argv)
  File "/root/miniconda3/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
  File "/root/miniconda3/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/miniconda3/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
  File "/root/miniconda3/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
  File "/root/miniconda3/lib/python3.6/site-packages/channels/management/commands/runworker.py", line 82, in handle
worker.run()
  File "/root/miniconda3/lib/python3.6/site-packages/channels/worker.py", line 87, in run
channel, content = self.channel_layer.receive_many(channels, block=True)
  File "/root/miniconda3/lib/python3.6/site-packages/asgi_redis/core.py", line 128, in receive_many
result = connection.blpop(list_names, timeout=self.blpop_timeout)
  File "/root/miniconda3/lib/python3.6/site-packages/redis/client.py", line 1550, in blpop
return self.execute_command('BLPOP', *keys)
  File "/root/miniconda3/lib/python3.6/site-packages/redis/client.py", line 772, in execute_command
connection = pool.get_connection(command_name, **options)
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 994, in get_connection
connection.connect()
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 502, in connect
self.on_connect()
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 570, in on_connect
if nativestr(self.read_response()) != 'OK':
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 637, in read_response
response = self._parser.read_response()
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 290, in read_response
response = self._buffer.readline()
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 224, in readline
self._read_from_socket()
  File "/root/miniconda3/lib/python3.6/site-packages/redis/connection.py", line 199, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

I saw this question and added the broker_use_ssl, but I am not sure how to actually use the configuration.

This did not change anything:

app = Celery(broker=redis_url, broker_use_ssl={'cert_reqs':'ssl.CERT_REQUIRED'})

I corrected the port above and tried @Stanley Gong suggestion, including removing the quotes from the ssl.CERT_REQUIRED. But no change in the results.

import ssl 

app = Celery(broker=redis_url, broker_use_ssl={'cert_reqs':ssl.CERT_REQUIRED})

user_7361760 · Answer

My Working solution. in my case redis host was on Azure cache for Redis.

# settings.py
import ssl
_broker_url = f'rediss://:{REDIS_PASS}@{REDIS_HOST_PORT_URL}'
BROKER_URL = _broker_url
CELERY_RESULT_BACKEND = _broker_url
BROKER_USE_SSL={'ssl_cert_reqs': ssl.CERT_REQUIRED}
CELERY_REDIS_BACKEND_USE_SSL={'ssl_cert_reqs': ssl.CERT_REQUIRED}

Notice extra s in rediss this is for ssl connection.

For celery.py file everything will be as it is except

# celery.py 
app = Celery('appname')

We can also do the configuration according to your exact needs.

change

BROKER_USE_SSL to broker_use_ssl and

CELERY_REDIS_BACKEND_USE_SSL to redis_backend_use_ssl.

Then you can do like this

app = Celery('appname', broker_url=url, broker_use_ssl={'ssl_cert_reqs': ssl.CERT_REQUIRED})
#if needed please add configuration for result_backend same as this.

How to securely connect to Azure redis from celery-django project?

1 Answers