Kibana reports a field is conflicting, how can I resolve it?

user_802755 0
Viewed 7870

In Kibana I've noticed after I did an Index Pattern refresh that my one field shows up as conflicted.

Example:

Conflicting field in Kibana

So I understand that this is because Elastic Search found values in that field that are of different types, how can I determine that? It is causing my Visuals to break as they can't work with conflicting fields. How can I get around this problem for the existing data?

kibana elasticsearch
1 Answers

After many hours of playing around and going through the Elastic documentation I have finally found an answer to my problem.

In Elastic Search 5.1 (the version I used) you can re-index those specific Indexes that are "problematic".

You can find this in Kibana by clicking on Management > Index Patterns and looking for the field that shows up as conflicted. Then click on the corresponding pencil icon to look at the field's details. In there is will show the Indexes under the different field types.

I wrote a script in Power-Shell that automated this for me by specifying the "problematic indexes" and then it does the following (let's assume your problematic index is called: log-20170101):

  • Create a mapping for log-20170101-1
  • Re-index log-20170101 to log-20170101-1
  • Delete log-20170101
  • Create a mapping for log-20170101
  • Re-index log-20170101-1 to log-20170101
  • Delete log-20170101-1

Now when you Refresh your Index Patter in Kibana you will notice that the field is no longer conflicted.

You can read up on: Mappings and Re-Indexing

Make sure that when you specify your new mapping below, that you use the appropriate mapping data-types that you are looking for.

You can get an existing mapping by querying the Elastic API with:

GET /_mapping/<your mapping name>

Here is a skeleton (sample) script I did in Power-Shell, it is very basic but I think it can help.

$index_list = @( 
    "log-20170101"
)

$index_list  | % {
    $index_name = $_

    $mapping_body = "
    {
        ""mappings"": {
            ""logevent"": {
                ""properties"": {
                    ""@timestamp"": {
                        ""type"": ""date""
                    },
                    ""correlationId"": {
                        ""type"": ""text"",
                        ""fields"": {
                            ""keyword"": {
                                ""type"": ""keyword"",
                                ""ignore_above"": 256
                            }
                        }
                    },
                    ""duration"": {
                        ""properties"": {
                            ""TotalMilliseconds"": {
                                ""type"": ""float""
                            }
                        }
                    }
                }
            }
        }
    }"

    $reindex_body = "{
        ""source"": {
            ""index"": ""$index_name""
        },
        ""dest"": {
            ""index"": ""$index_name-1""
        }
    }"

    $reindex_body_reverse = "{
        ""source"": {
            ""index"": ""$index_name-1""
        },
        ""dest"": {
            ""index"": ""$index_name""
        }
    }"

    Invoke-WebRequest -Uri http://elasticserver:9200/$index_name-1 -Method Put -Body $mapping_body
    Invoke-WebRequest -Uri http://elasticserver:9200/_reindex -Method Post -Body $reindex_body
    Invoke-WebRequest -Uri http://elasticserver:9200/$index_name -Method Delete
    Invoke-WebRequest -Uri http://elasticserver:9200/$index_name -Method Put -Body $mapping_body
    Invoke-WebRequest -Uri http://elasticserver:9200/_reindex -Method Post -Body $reindex_body_reverse
    Invoke-WebRequest -Uri http://elasticserver:9200/$index_name-1 -Method Delete
}

EDIT

See this post for how to setup default mappings going forward to try and prevent this problem from happening again.

user_8027550
Related