Which requests should be handled by the webserver and which by a task queue worker?

Viewed 40

I am working on a Python web app that uses Celery to schedule and execute user job requests.

Most of the time the requests submitted by a user can't be resolved immediately and thus it makes sense to me to schedule them in a queue.

However, now that I have the whole queuing architecture in place, I'm confused about whether I should delegate all the request processing logic to the queue/workers or if I should leave some of the work to the webserver itself.

For example, apart from the job scheduling, there are times where a user only needs to perform a simple database query, or retrieve a static JSON file. Should I also delegate these "synchronous" requests to the queue/workers?

Right now, my webserver controllers don't do anything except validating incoming JSON request schemas and forwarding them to the queue. What are the pros and cons of having a dumb webserver like this?

1 Answers

I believe the way you have it right now plus giving the workers the small jobs now is good. That way the workers would be overloaded first in the event of an attack or huge request influx. :)

Related