Get data from only the latest Elastic index in Grafana

user_662345 0
Viewed 90

I have a series of indexes in Elastic, myindex-YYYY.MM.DD. In a Grafana panel, I want to read data only from the latest such index each time. I have created a datasource [myindex-]YYYY.MM.DD with pattern Daily, but this reads from all indexes. I can't find out whether limiting to the latest index should be done in the data source or in the panel options.

An alternative could be to filter the documents so that I get only those whose @timestamp equals the max @timestamp, but I can't figure out this either. I can get the max @timestamp with this:

GET /myindex-*/_search
{
  "size": 0,
  "aggs": {
    "max_timestamp": { "max": { "field": "@timestamp" } }
  }
}

I’d need to save the result in a variable and use it in another query, but I can’t find a way to do this in Grafana.

grafana elasticsearch
1 Answers

My conclusion (from reading whatever I could find and from the absence of answers to this question) is that what I want is not possible to do directly. I ended up creating a myindex-latest alias to the latest of the myindex-YYYY.MM.DD series. I did this by running a script similar to the following (in my case it's being run by Logstash after creation of myindex-YYYY.MM.DD finishes):

#!/bin/bash
#
# This script creates elastic alias myindex-latest for the index
# myindex-YYYY.MM.DD, where YYYY.MM.DD is the current date.

curdate=`date +%Y.%m.%d`

read -r -d '' JSON <<EOF1
    {
        "actions": [
            {
                "remove": {
                    "index": "*",
                    "alias": "myindex-latest"
                }
            },
            {
                "add": {
                    "index": "myindex-$curdate",
                    "alias": "myindex-latest"
                }
            }
        ]
    }
EOF1

curl -X POST \
    -H "Content-Type: application/json" \
    "http://es01:9200/_aliases" \
    -d "$JSON"
user_6623450
Related