Regex process logfiles with multiline and ungreedy flags

user_3594622 0
Viewed 106

The logfile I am trying to match contains different severities with different outputs. Some logs are oneline and some are multilines with a lot of debug information included. The regex I have finds all the entries, but for multilines stops matching at the newline character. The environment is Ungreedy.

^\[(?<type>\w+):(?<severity>\w+),(?<timestamp>\d+-\d+-\d+T\d+:\d+:\d+.\d+Z),(?<node_name>[^:]*):([^:]+):(?<source>[^\]]+)\](?<message>.*)$

One log starts either at [error_logger...] or at [ns_server...]. The last group of the log is the "message" group and that should finish until the next occurrence of another log (error_logger OR ns_server). In this example are 3 different logs where the one-lines are matched correctly, while the one with multiline (which contains INFO REPORT or JSON Payload) is not.

[ns_server:debug,2022-06-30T23:52:17.409Z,ns_1@vm-1.c.server_name.internal:compaction_daemon<0.476.0>:compaction_scheduler:schedule_next:60]Finished compaction for compact_views too soon. Next run will be in 30s
[error_logger:info,2022-06-30T23:52:17.765Z,ns_1@vm-1.c.server_name.internal:error_logger<0.32.0>:ale_error_logger_handler:do_log:203]
=========================INFO REPORT=========================
{net_kernel,{connect,normal,
                     'ns_1@vm-2.c.server_name.internal'}}
[ns_server:info,2022-06-30T23:52:18.405Z,ns_1@vm-1.c.server_name.internal:<0.821.0>:ns_orchestrator:handle_info:523]Skipping janitor in state janitor_running

Here is the test with some examples of the logs I need to process: https://regex101.com/r/HyiWIy/1 It already has the flags global, multiline and ungreedy set.

Background: The logs are read from Google Cloud OPS Agent and pushed to the Cloud Logging in GCP. Cloud OPS Agent is using fluent bit for logs and metrics. Out of all different combinations I have tried, the agent matches exactly the entries that are matched in the provided example. Using a modifier like this: (?-U) crashes the fluent-bit service, so that is not the way I'd like to go.

In the screenshot you can see that the first line is processed fine until a newline. After that, every line of the unmatched block is processed separately and counts as a separate log line (although it's not)

Log Output

regex
2 Answers

You don't have to make the environment ungreedy, and in the message group repeat matching all lines until the next line that starts with for example [ word chars and :

^\[(?<type>\w+):(?<severity>\w+),(?<timestamp>\d+-\d+-\d+T\d+:\d+:\d+.\d+Z),(?<node_name>[^:]*):([^:]+):(?<source>[^\]]+)\](?<message>.*(?:\n(?!\[\w+:).*)*)$

Regex demo

user_54249880

What you need is a non-greedy message-capturing group that can cross linefeeds in a multi-line regex (. does not match \n in multi-line regex) and a positive look-ahead to identify the end of the message.

This regex should do what needs to be done:

^\[(?<type>\w+):(?<severity>\w+),(?<timestamp>\d+-\d+-\d+T\d+:\d+:\d+.\d+Z),(?<node_name>[^:]*):([^:]+):(?<source>[^\]]+)\](?<message>[\s\S]*?)(?=\n\[ns_server:|\n\[error_logger:|\Z)

The trick here is the [\s\S]*? to capture the message. It is a non-greedy term that can even in multi-line-mode go across newline characters. Non-greedy is necessary for it to not run over the beginning of the next log-entry that is found by the last parenthesised term: a positive look-ahead that identifies either the start of a new log-entry or the end of the file (\Z). Since it is a look-ahead, it does not consume the next log-entry's start, so it can still be used for the next match.

Look here https://regex101.com/r/qsmEQO/1 to see the regex in action.

user_195401110
Related