I need to send to elasticsearch or logstash Spring-boot plain text logs files, having the logback format, by the mean of filebeat.
You know hat these logs often have multi-lines depicting exceptions when they occur, like many Java ones.
2021-02-23 08:25:55.988 INFO 27940 --- [cast-exchange-0] o.a.s.sql.execution.FileSourceScanExec : Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
2021-02-23 08:25:56.051 INFO 27940 --- [ main] o.a.s.s.c.e.codegen.CodeGenerator : Code generated in 19.159775 ms
2021-02-23 08:25:56.078 INFO 27940 --- [ main] o.a.s.s.c.e.codegen.CodeGenerator : Code generated in 20.223781 ms
2021-02-23 08:25:56.096 INFO 27940 --- [ main] o.a.s.s.c.e.codegen.CodeGenerator : Code generated in 12.228878 ms
2021-02-23 08:25:56.179 INFO 27940 --- [cast-exchange-0] org.apache.spark.SparkContext : Starting job: run at ThreadPoolExecutor.java:1128
2021-02-23 08:25:56.197 INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler : Got job 0 (run at ThreadPoolExecutor.java:1128) with 1 output partitions
2021-02-23 08:25:56.197 INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler : Final stage: ResultStage 0 (run at ThreadPoolExecutor.java:1128)
2021-02-23 08:25:56.198 INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler : Parents of final stage: List()
2021-02-23 08:25:56.199 INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler : Missing parents: List()
2021-02-23 08:25:56.202 INFO 27940 --- [uler-event-loop] org.apache.spark.scheduler.DAGScheduler : Submitting ResultStage 0 (MapPartitionsRDD[3] at run at ThreadPoolExecutor.java:1128), which has no missing parents
2021-02-23 08:25:56.218 INFO 27940 --- [uler-event-loop] o.a.spark.storage.memory.MemoryStore : Block broadcast_1 stored as values in memory (estimated size 14.3 KB, free 2.8 GB)
2021-02-23 08:25:56.221 INFO 27940 --- [uler-event-loop] o.a.spark.storage.memory.MemoryStore : Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.8 KB, free 2.8 GB)
2021-02-23 08:25:56.222 INFO 27940 --- [er-event-loop-2] o.apache.spark.storage.BlockManagerInfo : Added broadcast_1_piece0 in memory on 192.168.0.12:42221 (size: 7.8 KB, free: 2.8 GB)
Over the Internet, I've found posts of attempts :
to create a specific
grokhandling them. Some searching for special character@to detect the apparition of an exception.to generate a
jsonlog file instead of a plain text file, to avoid the problem.
But mine will always be in plain text. Some are already existing and come from the past.then, also, they are modifying their existing Spring-boot application, changing at least its
pom.xml.
I'm not allowed to do that neither. I can intervene in elasticsearch, logstash, kibana and filebeat configurations, but that's all. I can't enter the source code of all existing applications that are producing logs.
I've found on elastic or grok website example for setting nginx, tomcat logs parsers with sometimes only a single command to install and start them under bash. Or, something really simple to do.
What is the most simple way to start the parsing by elk & filebeat of existing logs files having logback format, without changing them (keeping them plain text like they are), with little configuration, and without modifying Spring-boot applications themselves ?