Increase shard count in elastic search Kubernetes

user_3789406 0
Viewed 3520

Deployed an Elasticsearch on GKE. ECK has 3 master and 2 data nodes. When creating an index with 1 shard and 1 replica getting 400 error response when the shard count reached 1000.

Is it possible to increase the shard count? Please suggest a way to increase the shards count.

Error details

Invalid NEST response built from a unsuccessful (400) low level call on PUT: /t-2af14520-e76d-4091-a51d-1c89cf990e97
Audit trail of this API call:
[1] BadResponse: Node: http://eck.com/ Took: 00:00:00.2833118
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: PUT /t-2af14520-e76d-4091-a51d-1c89cf990e97. ServerError: Type: illegal_argument_exception Reason: "Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2000]/[2000] maximum shards open;"
Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
google-kubernetes-engine elasticsearch kubernetes
2 Answers

You can change the shard number as follows. Refer Cluster update settings API for more details. I recommend that you apply it after test.

PUT /_cluster/settings
{
    "persistent" : {
        "cluster.max_shards_per_node" : 3000
    }
}
user_88917380

Daein park answer is correct. you can increase the limit of cluster.max_shards_per_nodes. but you should consider that there was some reasons that they put this limit on cluster. you will encounter some problems about Heap usage, low search/index performance and cluster instability.

so I recommend that try to decrease number of indices and shards in cluster. you can use shrink API or reindex API for that purpose. https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

user_97921040
Related