I have the following setup, where each task has to hit the Redis database every 10 seconds. I would like to share the connection pool across them, to achieve a greater efficiency.
Celery.py
from celery import Celery
app = Celery("notify", include=["notify.tasks"])
app.config_from_object("notify.celery_config")
app.conf.beat_schedule = {
"run-push-every-10-seconds": {"task": "notify.tasks.push", "schedule": 10.0}
}
Tasks.py
import os
import redis
from notify.celery import app
from config import get_settings
settings = get_settings()
redis_pool = None
def init():
global redis_pool
print("PID %d: initializing redis pool..." % os.getpid())
redis_pool = redis.ConnectionPool(
host=settings.REDIS_IP,
password=settings.REDIS_PASSWORD,
port=settings.REDIS_PORT,
db=settings.REDIS_DB,
)
@app.task
def push():
r = redis.Redis(
connection_pool=redis_pool,
decode_responses=True,
)
print(r.keys("*"))
I did some research and it was suggested to create the ConnectionPool globally and injecting it in when establishing a connection with Redis as I did above in init().
I suppose I have to call once init() when the process or worker has been initialised. But it's not clear to me how to achieve this. Thanks