My main celery app is running in AWS in an EC2 instance ("main"), and the tasks it generates interact with an RDS database in the same AZ ("db"). The workload generates up to thousands of tasks every minute and I need to execute them in parallel as quickly as possible. I have workers consuming tasks in two physical locations. One from a separate EC2 instance ("worker EC2") from main but in same AZ as it and db, and one from a physical machine in our office's private data center ("worker local").
Both worker EC2 and local were running prefork event pooling with autoscale==70,4 and were working fine (tasks completing in 2-3s), but CPU and memory usage was high and I need even more parallelism if possible. So I've been experimenting with eventlet and gevent with concurrency=100. I am stuck on the following issues:
- Eventlet on worker EC2 task completion time (1s) is faster than prefork (2-3s). However, on worker local, task completion time is much slower (17-25s) and most of the time it is stuck performing simple queries to the db and I/O with a local Redis cache. CPU and memory are not heavily taxed by the process. The same issues on worker local occur with Gevent, but more severe. Tasks complete in 200+ seconds.
- Eventlet on worker EC2 task completion time is faster, but it doesn't seem to be using full concurrency. Though each task is completing within 1-1.5 seconds I never see more than 10-15 tasks completed within any given one second window. So it seems full concurrency is not being utilized. CPU is 60-80% utilized and memory less than 50%, so not being heavily taxed.
- Local worker running both Gevent and Eventlet frequently goes idle even with tasks in the queue. Once I restart the worker it starts consuming tasks again for some time, then goes idle again. I notice that in these scenarios the worker gets
BrokenPipeerror. NOTE this does NOT happen with prefork, so I don't think it's the connection/network but rather something to do with the pooling package. This behavior is worse for Gevent than Eventlet, but both are bad
Questions:
Why would Eventlet and Gevent be performing differently between the local and EC2 workers? Only difference between these two workers is that one is physically closer to the main celery app and the DB, would that affect how green threads execute?
How to get Eventlet to actually utilize the full concurrency setting?
Why would Gevent take so long to access in-memory cache?
How can I get Eventlet/Gevent on remote worker to consistently consume tasks?
Note NONE of these issues occur with the exact same setup with same workers using prefork. The only issue with prefork is that the worker machines' CPU and memory are heavily taxed with even 70 concurrency, and I would like it to be closer to 700.