Regexp JSON filtering in LogQL

Question

I'd like to translate Kibana query like to following to LogQL:

host:("test1-myservice-*") AND level:ERROR 
AND NOT logger_name:"com.example.ExampleClass" 
AND _exists_:stack_trace 
AND NOT stack_trace:(
    "interrupted"
    OR "Read timed out"
    OR "java.lang.InterruptedException"
)

I have tried the following in Grafana Explore but it does not return any records for our JSON log messages:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace =~ ".*InterruptedException.*"

While using != instead of =~ it returns all records:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ ".*InterruptedException.*"

If I'm right the following applies from the documentations to the stack_trace field of the JSON log line:

String type work exactly like Prometheus label matchers use in log stream selector. This means you can use the same operations (=,!=,=~,!~).

Source: Label filter expression

The following seems to work but it seems awkward:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
!~ ".*InterruptedException.*|.*Read timed out.*"
| json

Furthermore, if I'm right, it searches for InterruptedException and Read timed out substrings in the complete JSON string instead of its stack_trace field only.

Is there a more LogQL-ish way to translate the Kibana query above to LogQL? Should the !~ operator work in this case?

Environment: Grafana 7.5.4 / 8.2.3, Loki: 2.4.1

user_716092 · Answer

Not sure how your log lines look exactly, but I think you don't need to extract the labels out (by using | json

This is a pretty useful article on how to write queries. how-to-create-fast-queries-with-lokis-logql-to-filter-terabytes-of-logs-in-seconds You can also make use of the new Pattern parser instead of the regex if you want to make the query more readable.

So without really knowing how your log lines look, I think this should work well:

{host=~"test1-myservice-.*"}
!= "com.example.ExampleClass" 
!~ ".*InterruptedException.*|.*Read timed out.*"

Based on your needs you can also make use of the Pattern parser I've mentioned before.

user_843804 · Answer

This works:

{host=~"test1-myservice-.*"} | json 
| logger_name != "com.example.ExampleClass" 
| stack_trace !="" 
| stack_trace !~ "(?s).*InterruptedException.*"

Note the (?s) which enables matching new lines for the regex . character. (The stack_trace field of the JSON log message usually contain multiple lines.)

This is also mentioned in the Log stream selector part of the documentation:

Note: The =~ regex operator is fully anchored, meaning regex must match against the entire string, including newlines. The regex . character does not match newlines by default. If you want the regex dot character to match newlines you can use the single-line flag, like so: (?s)search_term.+ matches search_term.

Regexp JSON filtering in LogQL

2 Answers