Elasticsearch 5 stuck reading from disk

user_2495331 0
Viewed 535

I have a cluster of 6 nodes with ES 5.4 with 4B small documents yet indexed.
Documents are organized in ~9K indexes, for a total of 2TB. The indexes' occupancy varies from few KB to hundreds of GB and they are sharded in order to keep each shard under 20GB.

Cluster health query responds with:

{
    cluster_name: "##########",
    status: "green",
    timed_out: false,
    number_of_nodes: 6,
    number_of_data_nodes: 6,
    active_primary_shards: 9014,
    active_shards: 9034,
    relocating_shards: 0,
    initializing_shards: 0,
    unassigned_shards: 0,
    delayed_unassigned_shards: 0,
    number_of_pending_tasks: 0,
    number_of_in_flight_fetch: 0,
    task_max_waiting_in_queue_millis: 0,
    active_shards_percent_as_number: 100
}

Before sending any query to the cluster, it is stable and it gets a bulk index query every second with 10 or some thousand of documents with no problem.

Everything is fine until I redirect some traffic to this cluster. As soon as it starts to respond the majority of the servers start reading from disk at 250 MB/s making the cluster unresponsive: enter image description here

What it is strange is that I cloned this ES configuration on AWS (same hardware, same Linux kernel, but different Linux version) and there I have no problem: enter image description here NB: note that 40MB/s of disk read is what I always had on servers that are serving traffic.

Relevant Elasticsearch 5 configurations are:

  • Xms12g -Xmx12g in jvm.options

I also tested it with the following configurations, but without succeeded:

  • bootstrap.memory_lock:true
  • MAX_OPEN_FILES=1000000

Each server has 16CPU and 32GB of RAM; some have Linux Jessie 8.7, other Jessie 8.6; all have kernel 3.16.0-4-amd64.

I checked that cache on each node with localhost:9200/_nodes/stats/indices/query_cache?pretty&human and all the servers have similar statistics: cache size, cache hit, miss and eviction.

It doesn't seem a warm up operation, since on AWS cloned cluster I never see this behavior and also because it never ends.
I can't find useful information under /var/log/elasticsearch/*.

Am I doing anything wrong?
What should I change in order to solve this problem?

Thanks!

elasticsearch-5 elasticsearch java
3 Answers

The max content of an HTTP request. Defaults to 100mb

servers start reading from disk at 250 MB/s making the cluster unresponsive - The max content of an HTTP request. Defaults to 100mb. . If set to greater than Integer.MAX_VALUE, it will be reset to 100mb.

This will become unresponsive and you might see the logs related this. Check with the max read size of the indices.

Check with Elasticsearch HTTP

user_43488240

a few things;

  1. 5.x has been EOL for years now, please upgrade as a matter of urgency
  2. you are heavily oversharded

for point 2 - you either need to;

  1. upgrade to handle that amount of shards, the memory management in 7.X is far superior
  2. reduce your shard count by reindexing
  3. add more nodes to deal with the load
user_156281200
Related