Regex to tokenize log line

user_839733 0
Viewed 136

I've a log line as follows:

[2021-03-10 00:13:32.901] [DefaultDispatcher-worker-2 @coroutine#3] [DEBUG] [4231c006d9083a302fce59d5f0957226] [42c5ac3c0acfc68d] [GreeterImpl] Hello John

It's 6 blocks of text within [] and then the rest. I'm looking for a regex to extract the text within [], and also at the end. A text block within [] can be empty.

I tried (?:\[([^\[\]]*)\])+([^\[\]]+) but it only matches the first block in []. I've also tried (?:(?<=\[)[^\[\]]*(?=\]))+([^\[\]]+) but that doesn't match anything.

FWIW, the regex will be implemented in Java.

regex-lookarounds regex-group regex java
2 Answers

Short edit: This slightly simpler regular expression works too:

(?:(?<=\[)[^\[\]]*)|(?:(?<=\])[^\[\]]*$)

I have taken it from your own comment.

Original answer follows.

TL;DR

(?:(?<=^\[| \[)[^\[\]]*)|(?:(?<=\] )[^\[\]]*$)

Explanation: There are two parts separated by |, “or”.

  1. The first part, (?:(?<=^\[| \[)[^\[\]]*) matches what is inside square brackets. [^\[\]]* near the end matches the longest possible run of characters that are neither [ nor ]. (?<=^\[| \[) requires it to be preceded either by the beginning of the string and a [ or by [. Finally I have put the whole thing into a non-capturing group to make sure that the lookbehind has precedence over the |.
  2. The second part, (?:(?<=\] )[^\[\]]*$), matches what is outside square brackets at the end of the log line (Hello John in the example). This time the run of non-brackets must be preceded by ] and followed by the end of the line.

See it in action:

  1. On regex101 where I built it

  2. In Java:

    String logLine = "[2021-03-10 00:13:32.901]"
        + " [DefaultDispatcher-worker-2 @coroutine#3] [DEBUG]"
        + " [4231c006d9083a302fce59d5f0957226] [42c5ac3c0acfc68d]"
        + " [GreeterImpl] Hello John";

Matcher m = Pattern
        .compile("(?:(?<=^\\[| \\[)[^\\[\\]]*)|(?:(?<=\\] )[^\\[\\]]*$)")
        .matcher(logLine);
while (m.find()) {
    System.out.println(m.group());
}

Output is:

2021-03-10 00:13:32.901
DefaultDispatcher-worker-2 @coroutine#3
DEBUG
4231c006d9083a302fce59d5f0957226
42c5ac3c0acfc68d
GreeterImpl
Hello John

A different idea: String.split()

    String[] tokens = logLine.split("\\] \\[|\\] (?!\\[)");
    assert tokens[0].startsWith("[") : logLine;
    tokens[0] = tokens[0].substring(1);

    for (String token : tokens) {
        System.out.println(token);
    }

Output is the same as before.

I am splitting at either ] [ or ] not followed by [ (for the last split). It leaves the first [ intact, so I have to remove that separately, which is not so nice. Otherwise I find it simpler to understand than the other solutions.

user_57728820

Another option is to make use of the \G anchor to match the [...] parts at the start, and optionally the rest in group 2.

That way you can differentiate which parts are between the square brackets and which part is the rest.

\G\[([^][]*)]\h+([^][]+$)?

The pattern matches:

  • \G Assert the position at the end of the previous match or at the start of the string for the first match
  • \[ Match [
  • ( Capture group 1
    • [^][]* Match 0+ occurrences or any char except [ or ]
  • ) Close group 1
  • ]\h+ Match the closing ] and 1 or more horizontal whitespace chars
  • ( Capture group 2
    • [^][]+$ Match 1+ occurrences of any char except [ or ]
  • )? Close group 2 and make it optional

Regex demo | Java demo

In Java with the doubled backslashes

String regex = "\\G\\[([^\\]\\[]*)]\\h+([^\\]\\[]+$)?";
user_54249880
Related