Replica and shard settings not applied in elasticsearch template

user_2287200 0
Viewed 209

I've added a template like this:

curl -X PUT "e.f.g.h:9200/_template/impression-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["impression-%{+YYYY.MM.dd}"],
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 2
  },
  "mappings": {
    "_doc": {
      "_source": {
        "enabled": false
      },
      "dynamic": false,
      "properties": {
        "message": {
          "type": "object",
          "properties": {
...

And I've logstash instance that read events from kafka on write them to ES. Here is my logstash config:

input {
  kafka {
    topics => ["impression"]
    bootstrap_servers => "a.b.c.d:9092"
  }
}
filter {
  json {
    source => "message"
    target => "message"
  }
}
output {
    elasticsearch {
        hosts => ["e.f.g.h:9200"]
        index => "impression-%{+YYYY.MM.dd}"
        template_name => "impression-template"
    }
}

But each day I get index with 5 shard and 1 replica (which is default config of ES). How I could fix that so I could get 2 replica and 2 shard?

logstash-configuration logstash elasticsearch
1 Answers

Not sure you can add index_pattern as my_index-%{+YYYY.MM.dd}, because when you create it and PUT my_index-2019.03.10 it will have empty mapping because it's not recognized. I had same issue, and workaround for this was to set index_pattern as my_index-* and add year suffix to indices which should look like my_index-2017, my_index-2018...

{
  "my_index_template" : {
    "order" : 0,
    "index_patterns" : [
      "my_index-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "number_of_replicas" : "1"
      }
    },...

I took year part from timestamp field (YYYY-MM-dd) to generate year and add it to the end of index name by logstash

grok {
      match => [
         "timestamp", "(?<index_year>%{YEAR})"
      ]
    }

    mutate {
        add_field => {
            "[@metadata][index_year]" => "%{index_year}"
        }
    }

    mutate {
        remove_field => [ "index_year", "@version" ]
    }
}
output{
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "my_index-%{[@metadata][index_year]}"
        document_id => "%{some_field}"
    }
}

After logstash was completed, I've managed to get my_index-2017, my_index-2018 and my_index-2019 indices with 5 shards, and 1 replica and correct mapping as I predefined in my template.

user_83821180
Related