Elasticsearch on AWS RED and reroute not allowed

user_1594749 0
Viewed 1623

I have created an ES domain to search for the vpcglow logs and cloud trail log with daily indexing. Right now, the status is RED :

{
   "cluster_name": "678628912247:test",
   "status": "red",
   "timed_out": false,
   "number_of_nodes": 17,
   "number_of_data_nodes": 17,
   "active_primary_shards": 687,
   "active_shards": 1374,
   "relocating_shards": 0,
   "initializing_shards": 0,
   "unassigned_shards": 8,
   "number_of_pending_tasks": 0
}

Further investigating I found, one index is RED:

red   open cwl-2016.02.19 5 1    381700  102899 335.8mb 167.9mb

Looking into the shards:

cwl-2016.02.19  2   p   UNASSIGNED                  
cwl-2016.02.19  2   r   UNASSIGNED                  
cwl-2016.02.19  0   p   UNASSIGNED                  
cwl-2016.02.19  0   r   UNASSIGNED                  
cwl-2016.02.19  3   p   STARTED 381700  167.9mb x.x.x.x Elektra Natchios
cwl-2016.02.19  3   r   STARTED 381700  167.9mb x.x.x.x Chronos 
cwl-2016.02.19  1   p   UNASSIGNED                  
cwl-2016.02.19  1   r   UNASSIGNED                  
cwl-2016.02.19  4   p   UNASSIGNED                  
cwl-2016.02.19  4   r   UNASSIGNED

I tried to reroute the shards to less used nodes, but it gives me:

{"Message":"Your request: '/_cluster/reroute' is not allowed."}

Any advice please what I should do now.

Thanks & Regards.

elasticsearch amazon-web-services
3 Answers

A red cluster status means that at least one primary shard and its replicas are not allocated to a node.

Since you have already found the red index best option is to delete it. If deletion is not possible then you restore the snapshot (Please note AWS automatically takes snapshots)

As last resort you can contact AWS support and they can restore it for you.

Its important to fix red cluster since once cluster is red AWS stop taking automatic snapshots.

user_113664650

RED cluster means one or more primary shards are not available and its means a data-loss and its very serious issue which requires an immediate fix.

  1. If you have snapshot then try to recover the index from it.
  2. In future try to increase replica so that you don't loose the primary shards and it can be easily recover from replica shards.
  3. Check the ES cluster logs and try to find out the reason of missing primary shards.
  4. See if reroute API can be useful, it will if you have shard available on disk but there is no data-node where ES can allocate it, see if you can data-node or create a configuration which can recover the primary shards.
  5. Regarding the Error when trying to run reroute API, it seems its a permission issue which you can solve by having proper access.
user_40394310

Elasticsearch Allocation API

Allocation API will help you understand the cluster allocation issues.

curl -XGET "location:9200/_cluster/allocation/explain"

Resolve the issues or reasons explained by allocation API and reinitiate the allocation with the following

curl -X POST http://127.0.0.1:9200/_cluster/reroute?retry_failed=true
user_43488240
Related