I've set up Logback for my business applications to keep the last 50MB of archived logs for each application, 10MB for each file, using size and time based rolling policy. Logs are getting consumed by Filebeat and are expected to be deleted soon afterwards.
Works like a charm - old archives are being deleted...until %i index value reaches 996. Then, old logs are rolled over but are no longer deleted, which results in disk space getting filled very quickly. Policy configuration:
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_HOME}business-log-${APPLICATION_NAME}-${ROLLING_PATTERN}-%i.json</fileNamePattern>
<maxFileSize>${BUSINESS_APPENDER_MAX_FILE_SIZE:-10MB}</maxFileSize>
<totalSizeCap>${BUSINESS_APPENDER_TOTAL_SIZE_CAP:-50MB}</totalSizeCap>
<maxHistory>${BUSINESS_APPENDER_MAX_HISTORY:-5}</maxHistory>
<cleanHistoryOnStart>${BUSINESS_APPENDER_CLEAN_HISTORY_ON_START:-true}</cleanHistoryOnStart>
</rollingPolicy>
Directory containing log files:
...
-rw-r--r-- 1 root root 10545515 Aug 26 04:54 log-example-application-20210826-1216.json
-rw-r--r-- 1 root root 10578986 Aug 26 04:55 log-example-application-20210826-1217.json
-rw-r--r-- 1 root root 10551848 Aug 26 04:55 log-example-application-20210826-1218.json
-rw-r--r-- 1 root root 10551008 Aug 26 04:55 log-example-application-20210826-1219.json
-rw-r--r-- 1 root root 10506275 Aug 26 04:56 log-example-application-20210826-1220.json
-rw-r--r-- 1 root root 10534379 Aug 26 04:56 log-example-application-20210826-1221.json
-rw-r--r-- 1 root root 10513144 Aug 26 04:56 log-example-application-20210826-1222.json
-rw-r--r-- 1 root root 10588197 Aug 26 04:57 log-example-application-20210826-1223.json
-rw-r--r-- 1 root root 10532073 Aug 26 04:57 log-example-application-20210826-1224.json
-rw-r--r-- 1 root root 10528993 Aug 26 04:57 log-example-application-20210826-1225.json
-rw-r--r-- 1 root root 10543370 Aug 26 04:57 log-example-application-20210826-1226.json
-rw-r--r-- 1 root root 10558424 Aug 26 04:58 log-example-application-20210826-1227.json
-rw-r--r-- 1 root root 10567860 Aug 26 04:58 log-example-application-20210826-1228.json
-rw-r--r-- 1 root root 10537785 Aug 26 04:58 log-example-application-20210826-1229.json
-rw-r--r-- 1 root root 10573044 Aug 26 04:59 log-example-application-20210826-1230.json
-rw-r--r-- 1 root root 10542241 Aug 26 03:40 log-example-application-20210826-996.json
-rw-r--r-- 1 root root 10534389 Aug 26 03:40 log-example-application-20210826-997.json
-rw-r--r-- 1 root root 10536618 Aug 26 03:41 log-example-application-20210826-998.json
-rw-r--r-- 1 root root 10565976 Aug 26 03:41 log-example-application-20210826-999.json
This problem occured many times for different applications, usually when experiencing huge load of incoming logs, e.g. due to issues with connecting to Kafka broker. Has anyone experienced such issue? What could be the solution?