how to limit the kubelet log size or rotate in kubernetes

user_2628868 0
Viewed 164

Today when I using duc command to check the kubernetes(v1.21) cluster host disk usage like this:

[root@k8smasterone log]# duc ls -Fg /var/log/
  5.4G messages-20220515             [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++]
  3.9G messages                      [++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                                   ]
571.1M messages-20220508             [++++++++++++++++++

the kubelet output too large files, what should I do to limit the kubelet log size? I have read this issue to talk about the log should handle by kubelet: https://github.com/containerd/containerd/issues/4830. but I did not found any possible solution to solve the kubelet log rotate problem. PS: I am using containerd not docker right now. This is my journald config:

[root@k8smasterone log]# cat /etc/systemd/journald.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=50M
#SystemKeepFree=
SystemMaxFileSize=20M
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
MaxRetentionSec=1week
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
kubernetes
1 Answers

You can tweak the following two parameters in the kubelet config:

containerLogMaxSize string A quantity defines the maximum size of the container log file before it is rotated. For example "5Mi" or "256Ki". Dynamic Kubelet Config (beta): If dynamically updating this field, consider that it may trigger log rotation. Default: "10Mi"

containerLogMaxFiles int32 Maximum number of container log files that can be present for a container. Dynamic Kubelet Config (beta): If dynamically updating this field, consider that lowering it may cause log files to be deleted. Default: 5

Example:

sudo vi /etc/kubernetes/kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
nodeStatusUpdateFrequency: "10s"
failSwapOn: True
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: True
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.crt
authorization:
  mode: Webhook
staticPodPath: /etc/kubernetes/manifests
cgroupDriver: systemd
containerLogMaxFiles: 5    # Maximum number of container logs to retain. 
containerLogMaxSize: 1Mi   # Change the size of /var/log/containers/<pod-name>/log files size to 1M max.
maxPods: 110
address: 192.168.22.5
readOnlyPort: 0
healthzPort: 10248
healthzBindAddress: 127.0.0.1
kubeletCgroups: /systemd/system.slice
clusterDomain: cluster.local
protectKernelDefaults: true
rotateCertificates: true
clusterDNS:
- 8.8.8.8
kubeReserved:
  cpu: 200m
  memory: 512Mi
resolvConf: "/run/systemd/resolve/resolv.conf"
eventRecordQPS: 5
shutdownGracePeriod: 60s
shutdownGracePeriodCriticalPods: 20s

Reload the kubelet config:

sudo systemctl daemon-reload && sudo systemctl restart kubelet
user_63096010
Related