Kibana fails to pick up date from elasticsearch when I include the hour and minute

user_1738522 0
Viewed 195

I'm really stuggling to get this specific time format into elasticsearch so I can graph it in Kibana. I cannot change this format. My elasticsearch data and mapping is in this format:

STEP 1: Setup Mapping

PUT http://<>.com:5101/myindex6/_doc/1

{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "HH:mm yyyy-MM-dd"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}

Step 2: Add Data

PUT http://<>.com:5101/myindex6

{
  "test" : [ {
    "data" : "119050300",
    "date" : "10:00 2019-06-03"
  } ]
}

In Kibana it wont find this as a date and wont allow me to map it as one. However, if I remove the time aspect and use the date, and do this instead, it works fine:

Data

{
  "test" : [ {
    "data" : "119050300",
    "date" : "2019-06-03"
  } ]
}

Map

{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "yyyy-MM-dd"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}

Can someone please tell me how to include the time and not have it break, so I can filter on time in kibana.

kibana elasticsearch
5 Answers

There is the difference between the mapping and the structure document that you are indexing. Also the endpoints you are using seems be swapped. Follow the steps below:

1. Create index 
PUT myindex6
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "HH:mm yyyy-MM-dd"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}
2. Index document 
POST myindex6/_doc/1
{
  "data": "119050300",
  "date": "10:00 2019-06-03"
}

Notice the endpoints used to create index and then to index a document. Also notice the structure of document is in line with the mapping. In your case you are indexing a document with a test field which is an array of object with fields data and date. This structure doesn't match to the mapping created in step 1.

user_11363710

As already pointed in the above answer, the data which you are indexing doesn't match with the mapping created. What you need to do is update your mapping with test field as nested.

{
  "mappings": {
    "properties": {
        "test": {
            "type": "nested",
            "properties": {
                "date": {
                    "type": "date",
                    "format": "HH:mm yyyy-MM-dd"
                },
                "data": {
                    "type": "integer"
                }
            }
        }
    }
}}

And then re-index your data. The date field will appear in Kibana after that.

user_44027860

As everyone has already pointed out, your index mapping differs from your data, so you need to update the mapping. I would highly suggest revising Elasticsearch's index mapping guide to get a good understanding of mappings and their effect on the data indexed.

On another note, I saw you are still using mapping types. If you are just getting started with your ES use cases, it will be good to avoid using them as they will be deprecated and eventually removed from ES [source].

user_53552720

You've got it all right, except one single tiny thing. In your mapping you specify the date format as being epoch_millis but in your document the datetime is specified in epoch_second.

So you have two choices:

  1. Either you keep the mapping as it is and multiply all your date times by a factor of 1000
  2. Or you change your mapping with "format": "epoch_second" and keep your data as it is.

In any case, you need to reload your index pattern in Kibana afterwards and it will work. Note that I tested this on Kibana 4.2.

user_111520320

This is how I think it will work.

PUT http://<>.com:5101/myindex6/_doc/1

{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "yyyy-MM-dd HH:mm:ss"
      },
        "data": {
        "type":   "integer"
      }
    }
  }
}

PUT http://<>.com:5101/myindex6

{
  "test" : [ {
    "data" : "119050300",
    "date" : "2019-06-03T10:00:00Z"
  } ]
}
user_67293970
Related