We have a single node elasticsearch for an ecommerce website with ~500 query per second. This image shows our elasticsearch node metrics for a period of 3 hours:
In the image, when queue count reaches 1000 (queue size), query rate decreased significantly. It seems elasticsearch panics when both thread pool and queue is full and starts to rejecting most of queries. The intended behavior should be like responding queries as much as possible and only rejecting those that are more than real capacity. My question is that is this behavior natural or should we change our configs?
thread pool and queues in elasticsearch:
a node holds several thread pools in order to improve how threads memory consumption are managed within a node. Many of these pools also have queues associated with them, which allow pending requests to be held instead of discarded.