Hello to everyone out there,
we currently have a problem with one of our pods in an AWS EKS cluster. The pod logs pretty heavily and gets evicted regularly, because it exceeds the ephemeral-storage limit. Unluckily, we are running an old version of EKS with docker container engine, but we aim to migrate to a newer version soon. Currently, we also do not have any log rotation configured.
While thinking about a way to avoid that behavior, I came up with 2 possibilities:
- Remove the ephemeral-storage limit currently set and use a log rotation tool (e.g. fluentd) to rotate the logs on node-level.
- Have a sidecar container which somehow clears/rotates the logs inside the container (is it possible with logs written to stdout?)
Ideally, I´d like to have a hybrid solution which allows for having ephemeral-storage limit, if wanted, but avoids filling it up with logs. I´m wondering what happens in situation 1 after some time. Will the memory consumption increase? This seems to be a pretty standard problem, but I was not able to find a satisfying answer to this during my research. How do you usually deal with log-intensive pods in Kubernetes?
Thanks for any hints and answers in advance