I am trying to understand how FastAPI/Gunicorn handles and queues requests.
Client contract: Once the client dispatches a request, it needs to receive a response in 5 seconds. If the client does not receive a response within 5 seconds, the request is no longer useful and the client stops listening for the response.
Scenario: Consider the following scenario:
- Let's say we have 4 workers (using Gunicorn as process manager with UvicornWorkers; using tiangolo's docker image)
- Client sends 100 concurrent requests
- Each request takes 5 seconds to complete
- We process the first 4 requests in 5 seconds
- But by that time, the remaining 96 requests accepted in the queue (I am not sure if queueing happens or how it works, would appreciate it if you can explain this) have already spent 5 seconds there and breached the client contract. Processing these 96 requests is a waste of resources as the client is no longer listening for them.
Question:
- How would I discard these 96 requests i.e. if each request has been waiting for more than 5 seconds in the queue, drop them?