I am trying to setup an airflow cluster. I am planning to use redis as celery backend. I have seen people using sentinel redis successfully. I wanted to know if it is possible to use redis cluster instead? If not then why not?
Is it possible to use redis cluster instead of sentinel as celery backend for airflow cluster
Viewed 2466
1 Answers
Celery doesn't have support for using Redis cluster as broker. It can use Redis highly available setup as broker (with Sentinels), but has no support for Redis cluster to be used as broker.
Reference:
Airflow CROSSSLOT Keys in request don't hash to the same slot error using AWS ElastiCache
How to use more than 2 redis nodes in django celery
To make Redis cluster to work we need to change the celery backend! not a feasible solution.
Related