Is there a way to use inferred values in a regex based LOKI query?

Viewed 686

I do have the following log lines in our Loki data source:

Trace_MSG_IN: 933a1425-5e62-545d-9947-274c5b870607
Trace_MSG_IN: 933a1425-5e62-545d-9947-274c5b870607
Trace_MSG_CR: 933a1425-5e62-545d-9947-274c5b870607
Trace_MSG_IN: a37da313-512a-50e6-9050-fd917b30c632
Trace_MSG_CR: a37da313-512a-50e6-9050-fd917b30c632

The desired output, to create the appropriate graph, would be the count of how often the message identifier was found together with the type of the trace-point:

msg_id=933a1425-5e62-545d-9947-274c5b870607, type=IN => 2
msg_id=933a1425-5e62-545d-9947-274c5b870607, type=CR => 1
msg_id=a37da313-512a-50e6-9050-fd917b30c632, type=IN => 1
msg_id=a37da313-512a-50e6-9050-fd917b30c632, type=CR => 1

The following LogQL query does the grouping, but I cannot see a way how I can integrate the counting functionality.

{build="runtimeidentifier"} |= "Trace_MSG_" | regexp "(?P<type>(CR|IN)): (?P<msg_id>[a-z0-9.-]{36})"

My try to integrate the counting functionality, based on https://grafana.com/blog/2021/01/11/how-to-use-logql-range-aggregations-in-loki/#a-word-on-grouping, was the following query:

count by(msg_id) ({build="runtimeidentifier"} |= "Trace_MSG_" | regexp "(?P<type>(CR|IN)): (?P<msg_id>[a-z0-9.-]{36})" | type = "CR" )

But this results in a syntax error

parse error at line 1, col 155: syntax error: unexpected )

As well as other variants of this query:

count_over_time({build="runtimeidentifier"} |= "Trace_MSG_" | regexp "(?P<type>(CR|IN)): (?P<msg_id>[a-z0-9.-]{36})" [10m]) (same error)

Is there even a way to use inferred values that are not an explicit part of the logline in queries?

1 Answers

How about something like this:

sum by (type, msg_id) (count_over_time({build="runtimeidentifier"} |= "Trace_MSG_" | pattern `Trace_MSG_<type>: <msg_id>` [$__interval]))

Not sure if I got it completely right with your pattern but you can see an example here which is quite similar to what you would like to accomplish:

Related