For troubleshooting a performance issue on Elasticsearch cluster, I started looking into the slow logs for some indices and noticed below strange thing:
types[estype], stats[], search_type[QUERY_THEN_FETCH], total_shards[6],,
Slow logs show that Elasticsearch query was hitting six shards, which is just not possible as my index has just two primary shards with two replicas** and according to Elasticsearch documentation, for a search query ES will send a request to either a primary shard or replica shard but not to both.
So According to the above logic, Elasticsearch must query only two shards, then how come in my slow logs it's happening as six(6) shards, and this is causing doubts in my mind for the slowness of these queries.
Note:- If I grab the ES query from slow-log and directly hit using curl, in response, it shows correct two shards only.