Shards and nodes for large static data in Elasticsearch

user_2543020 0
Viewed 84

I have a dataset of around 1.5T - 2T that I'd like to store in Elasticsearch. The fields are mostly flat and the document sizes are quite small. It is only updated once every 3 months so fairly static.

The main priority is search performance.

If the upload and updates take a week it doesn't matter. I realise a lot depends on how I structure my index and mappings and I'm confident I have that covered. The searches are infrequent but must be fast. What I'm not sure about, even after reading all the docs I can find, is the number of shards, number of nodes etc, given the static nature of the data. Basically the design of the cluster for static data.

Should I still stick with no more than 30-50G per shard? Do I need to consider cpu cores per shard? Thanks in advance for your guidance.

elasticsearch sharding
1 Answers

Some suggestions, not every one of them is appropriate to the actual situation.

Part 1

 1.The data is distributed across at least three nodes (hdd about ~5, ssd about ~3)
 2.And 100G+ per shard (Consider your cluster and disk performance)
 3.Increases the refresh time of the segments

Part 2

 1.The key is mapping, optimize the mapping for each field
 2.Make sure DOC is dense
 3.Set the route key if one exists or useful
 4.Try to use the keyword instead of something like long or int. Term queries are always better than range queries
 5.Take advantage of the 4 Levels cache (RequestCache,LRUQueryCache,MMAPCache,FileSystemCache), use index-sorting to optimize queries
 6.Auto generate _id
 7.Turn off doc values that do not require fields
 8.Filter can be used to turn off the scoring function if not required
 9.index.merge.scheduler.max_thread_count: 1 if use hdd

See this

{
    "mappings": {
        "data": {
            "dynamic": "false",
            "_source": {
                "includes": ["XXX"]  # only searchable filed in _source
            },
            "properties": {
                "state": {
                    "type": "keyword",  # state: int, but use keyword
                    "doc_values": false  
                },
                "b": {
                    "type": "long"    # range search
                }
            }
        }
    },
   "settings": {......}
}
user_115079100
Related