Prometheus PromQL Outer Join

Question

I have two Prometheus metric gauges that I want to join and keep certain types of labels at the end.

-Gauge 1 metric sample:

{attnid="dfw002", hhmm="00:001",status="200", device="android_mobile", network="tnt"}

-Gauge 2 metric sample:

{attnid="dfw002", hhmm="00:001", adr_code="24107",description="No profile found"}

After joining I want the metric labels to display something like this in the legend:

{adr_code="24107", attnid="dfw002", description="No profile found",
   device="android_mobile", network="tnt", status="200"}

(so everything but hhmm field)

If there is no match in Gauge 2 metric I just want to show Gauge 1 metrics then so:

 {attnid="dfw002", device="android_mobile", network="tnt",
   status="200"}

with no hhmm

I want to use the fields attnid and hhmm to match/join but I do not want to sum/count by hhmm at the end.

As for one-to-many relationship, there could be multiple matches in Gauge 2 for 1 metric in Gauge 1.

Right now I am using this query:

 sum((max_over_time(adr_general_gauge{attnid=~"$AttnID", hhmm=~".*",
   device=~"$Device", network=~"$Network", status=~"$Status"}[5m])) + on
   (hhmm, attnid) group_right(network, device, status) 0*
   (max_over_time(adr_code_gauge{attnid=~"$AttnID", hhmm=~".*",
   adr_code=~"$ADR_Code", description=~".*", device=~"$Device",
   network=~"$Network"}[5m])) or on (attnid, hhmm)
   (max_over_time(adr_general_gauge{attnid=~"$AttnID", hhmm=~".*",
   device=~"$Device", network=~"$Network", status=~"$Status"}[5m]))) by
   (attnid, status, device, network, adr_code, description)

but the result is a bit redundant because both of them are showing up

{adr_code="24107", attnid="dfw002", description="No profile found",
   device="android_mobile", network="tnt", status="200"}

AND

{attnid="dfw002", device="android_mobile", network="tnt",
   status="200"} around the same time period but I really just want
   {adr_code="24107", attnid="dfw002", description="No profile found",
   device="android_mobile", network="tnt", status="200"}

The problem is the timestamp is slightly different for the metrics I am joining due to one log comes in later than the other (so that's why I am using max_over_time [5m] to be able to join.

What I really want is more like a if statement that if it matches then show the combined value but if not just show Gauge 1 metric.

Like if:

sum((max_over_time(adr_general_gauge{attnid=~"$AttnID", hhmm=~".*",
   device=~"$Device", network=~"$Network", status=~"$Status"}[5m])) + on
   (hhmm, attnid) group_right(network, device, status) 0*
   (max_over_time(adr_code_gauge{attnid=~"$AttnID", hhmm=~".*",
   adr_code=~"$ADR_Code", description=~".*", device=~"$Device",
   network=~"$Network"}[5m]))) by (attnid, status, device, network, adr_code, description) == 0 
then 

 sum(max_over_time(adr_general_gauge{attnid=~"$AttnID", hhmm=~".*",   
   device=~"$Device", network=~"$Network", status=~"$Status"}[5m]))) by 
   (attnid, status, device, description)

Prometheus PromQL Outer Join

0 Answers