How to change log retention policy in elk stack?

user_13845422 0
Viewed 731

I am using elk stack to retain and monitor the nginx-ingress logs of my k8s cluster. Instead of kibana Im using Grafana and instead of fluentd Im using fuent-bit. I found one documentation saying elasticsearch retain logs for 7 days and I also found an article where it said it retains logs life long.

All I want is the logs for last 6 months and any logs beyond that is not needed.

I have checked values.yaml file of elasticsearch to see if I can find the configuration option to change logs retention time but to no avail.

Has anyone worked with similar stack and knows how to change logs retention time??? Your time will be highly appreciated.

elasticsearch
1 Answers

For retaining data, You need to configure Index Lifecycle Policy. Currently, if you have not configured ILM policy, then Elastic will retain log data for a lifetime and it will not automatically deleted. You can create policy from Kibana as well but as you are mentioning you are not using Kibana, you can follow below command.

To create a lifecycle policy from Kibana, open the menu and go to Stack Management > Index Lifecycle Policies. Click Create policy.

You can configured ILM Policy using below API:

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {} 
        }
      }
    }
  }
}

The above policy will rolls the index over if it was created at least 7 days ago and Delete the index 30 days after rollover.

You can assign created policy to your index using below command:

PUT logs-my_app-default/_settings
{
  "index": {
    "lifecycle": {
      "name": "my_policy"
    }
  }
}

Update

You can use Explain lifecycle API to validate if ILM is working properly or not.

GET my-index-000001/_ilm/explain
user_54892760
Related