I'm trying to diagnose an issue where our ElasticSearch search queue seemingly randomly fills up.
The behavior we observe in our monitor is that on one node of our cluster the search queue growth (just one) and after the search thread pool is used up we start getting timeouts of course. There seems to be one query that is blocking the while thing. The only way for us to resolve the problem at the moment is to restart the node.
You can see below the relevant behavior in charts: First the queue size, then the pending cluster tasks (to show that no other operations are blocking or queing up, e.g. index operations or so) and finally the active threads for the search thread pool. The spike at 11 o'clock is the restart of the node.
The log files on all nodes show no entries during an hour before or after the issue until we restarted the node. Only garbage collection events of around 200 -600ms and only one on the relevant node but that is around 20 minutes before the event.
My questions: - how can I debug this as there is no information logged anywhere on a failing or timing out query? - what are possible reasons for this? We don't have dynamic queries or anything similar - can I set a query timeout or clear / reset active searches when this happens to prevent a node restart?
Some more details that don't apply, based on questions so far:
- exactly same hardware (16 cores, 60GB mem)
- same config, no special nodes
- no swap enabled
- nothing noticeable on other metrics like IO or CPU
- not a master node
- no special shards, three shards per node each node, pertty standard queries, all queries getting send to ES for 10 minutes before are queries that typically finish within 5-10ms, all the ones we get a timeout on are the same, no increase in query rate or anything else
- we have 5 nodes for this deployment, all accessed round robin
- we have a slow log of 2 seconds on info level, no entries
The hot threads after 1 minute of queue build up are at https://gist.github.com/elm-/5ed398054ea6b46522c0, several snapshots of some dumps over a few moments.