We are using celery to make third party http calls. We have around 100+ of tasks which simply calls the third party HTTP API calls. Some tasks call the API's in bulk, for example half a million requests at 4 AM in morning, while some are continuous stream of API calls receiving requests almost once or twice per second.
Most of API call response time is between 500 - 800 ms.
We are seeing very slow delivery rates with celery. For most of the above tasks, the max delivery rate is around 100/s (max) to almost 1/s (min). I believe this is very poor and something is definitely wrong, but I am not able to figure out what it is.
We started with cluster of 3 servers and incrementally made it a cluster of 7 servers, but with no improvement. We have tried with different concurrency settings from autoscale to fixed 10, 20, 50, 100 workers. There is no result backend and our broker is RabbitMQ.
Since our task execution time is very small, less than a second for most, we have also tried making prefetch count unlimited to various values.
--time-limit=1800 --maxtasksperchild=1000 -Ofair -c 64 --config=celeryconfig_production
Servers are 64 G RAM, Centos 6.6.
Can you give me idea on what could be wrong or pointers on how to solve it?
Should we go with gevents? Though I have little of idea of what it is.