Elasticsearch cluster dropping messages

Boaz Lowenstein 0
Viewed 17

In my cluster trying to insert message (from filebeat) I get

(status=400): {"type":"illegal_argument_exception","reason":"Validation Failed: 1: this action would add [2] shards, but this cluster currently has [2999]/[3000] maximum normal shards open;"}, dropping event!

looking at the cluster health its looks like it has available shards

"cluster_name" : "elastic", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 4, "number_of_data_nodes" : 1, "active_primary_shards" : 1511, "active_shards" : 1511, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1488, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 1159, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 57267, "active_shards_percent_as_number" : 50.3834611537179 any ideas ?

elasticsearch
1 Answers

Tldr;

It seems like you only have one data node. As you can see you have 1511 allocated shards, and 1488 unassigned shards

And guess what happens if we add them up ^^. We get the 2999.

Elasticsearch does not allow to have the primary shards and its replicas on the same node.

But since you have only one data node all those unassigned shards are most likely replicas.

Solution

short term

Create your new index with 0 replica.

POST /my-index-000001
{ 
  "settings": {
    "index" : {
      "number_of_replicas" : 0
    }
  }
}

You could also set all other index with no replica via:

PUT /*/_settings
{
  "index" : {
    "number_of_replicas" : 0
  }
}

Long term

You either need to change all you indices to only have one replica or add another node. So that those replicas get allocated. At the moment you are risking data loss of the node crash.

Then as per the documentation you can change the value of cluster.routing.allocation.total_shards_per_node which must be set a 3000

userxL1j0
Related