How to remove documents from elasticsearch by date period or by size?

user_7916116 0
Viewed 2314

I want to remove data from elasticsearch with configuring the data date period, for example, data older that 30 days, or maybe by size, like if index size is bigger than 100Mb then remove old data. I use logstash to move logs to one index in elasticsearch. How can i do that?

logstash elasticsearch
2 Answers

If you want to explicitly delete logs based on ur criteria you can use delete_by_query

POST /my_logs/_delete_by_query
{
    "query": {
        "range": {
            "date": {
                "lte": <your_target_date>
            }
        }
    }
}
user_93311340

Deleting older data from the current index is not a easy way to do it. You can configure logstash to create new index daily. Then you can access all your data through index patterns or alias.

Then you will be able to delete older indexes without much issue based on the date.

You can automate these using curator - https://www.elastic.co/guide/en/elasticsearch/client/curator/5.8/index.html

See this post on configuring logstash to create indexes daily Create a new index per day for Elasticsearch in Logstash configuration

user_42552530
Related