How to share redis connection pool across celery tasks?

Viewed 547

I have the following setup, where each task has to hit the Redis database every 10 seconds. I would like to share the connection pool across them, to achieve a greater efficiency.

Celery.py

from celery import Celery

app = Celery("notify", include=["notify.tasks"])
app.config_from_object("notify.celery_config")


app.conf.beat_schedule = {
    "run-push-every-10-seconds": {"task": "notify.tasks.push", "schedule": 10.0}
}

Tasks.py

import os
import redis    
from notify.celery import app
from config import get_settings

settings = get_settings()
redis_pool = None


def init():
    global redis_pool
    print("PID %d: initializing redis pool..." % os.getpid())
    redis_pool = redis.ConnectionPool(
        host=settings.REDIS_IP,
        password=settings.REDIS_PASSWORD,
        port=settings.REDIS_PORT,
        db=settings.REDIS_DB,
    )


@app.task
def push():
    r = redis.Redis(
        connection_pool=redis_pool,
        decode_responses=True,
    )
    print(r.keys("*"))

I did some research and it was suggested to create the ConnectionPool globally and injecting it in when establishing a connection with Redis as I did above in init().

I suppose I have to call once init() when the process or worker has been initialised. But it's not clear to me how to achieve this. Thanks

1 Answers
Related