What's the most standard/simple way to configure filebeat, ELK for plain text java logs with logback format, without modifying existing applications?

Viewed 172

I need to send to elasticsearch or logstash Spring-boot plain text logs files, having the logback format, by the mean of filebeat.
You know hat these logs often have multi-lines depicting exceptions when they occur, like many Java ones.

2021-02-23 08:25:55.988  INFO 27940 --- [cast-exchange-0] o.a.s.sql.execution.FileSourceScanExec   : Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
2021-02-23 08:25:56.051  INFO 27940 --- [           main] o.a.s.s.c.e.codegen.CodeGenerator        : Code generated in 19.159775 ms
2021-02-23 08:25:56.078  INFO 27940 --- [           main] o.a.s.s.c.e.codegen.CodeGenerator        : Code generated in 20.223781 ms
2021-02-23 08:25:56.096  INFO 27940 --- [           main] o.a.s.s.c.e.codegen.CodeGenerator        : Code generated in 12.228878 ms
2021-02-23 08:25:56.179  INFO 27940 --- [cast-exchange-0] org.apache.spark.SparkContext            : Starting job: run at ThreadPoolExecutor.java:1128
2021-02-23 08:25:56.197  INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler  : Got job 0 (run at ThreadPoolExecutor.java:1128) with 1 output partitions
2021-02-23 08:25:56.197  INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler  : Final stage: ResultStage 0 (run at ThreadPoolExecutor.java:1128)
2021-02-23 08:25:56.198  INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler  : Parents of final stage: List()
2021-02-23 08:25:56.199  INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler  : Missing parents: List()
2021-02-23 08:25:56.202  INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler  : Submitting ResultStage 0 (MapPartitionsRDD[3] at run at ThreadPoolExecutor.java:1128), which has no missing parents
2021-02-23 08:25:56.218  INFO 27940 --- [uler-event-loop] o.a.spark.storage.memory.MemoryStore     : Block broadcast_1 stored as values in memory (estimated size 14.3 KB, free 2.8 GB)
2021-02-23 08:25:56.221  INFO 27940 --- [uler-event-loop] o.a.spark.storage.memory.MemoryStore     : Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.8 KB, free 2.8 GB)
2021-02-23 08:25:56.222  INFO 27940 --- [er-event-loop-2] o.apache.spark.storage.BlockManagerInfo  : Added broadcast_1_piece0 in memory on 192.168.0.12:42221 (size: 7.8 KB, free: 2.8 GB)

Over the Internet, I've found posts of attempts :

  1. to create a specific grok handling them. Some searching for special character @ to detect the apparition of an exception.

  2. to generate a json log file instead of a plain text file, to avoid the problem.
    But mine will always be in plain text. Some are already existing and come from the past.

  3. then, also, they are modifying their existing Spring-boot application, changing at least its pom.xml.
    I'm not allowed to do that neither. I can intervene in elasticsearch, logstash, kibana and filebeat configurations, but that's all. I can't enter the source code of all existing applications that are producing logs.

I've found on elastic or grok website example for setting nginx, tomcat logs parsers with sometimes only a single command to install and start them under bash. Or, something really simple to do.

What is the most simple way to start the parsing by elk & filebeat of existing logs files having logback format, without changing them (keeping them plain text like they are), with little configuration, and without modifying Spring-boot applications themselves ?

0 Answers
Related