I am designing a metric with prometheus that should be able to measure the operational time of my program which runs on a microservice architecture with the aim of doing a performance test. I used the Timer object from the micrometer library so i ended up with a metric_name_sum counter and a metric_name_count counter. The metric is run on each of the microservice of my program and it has a tag that shows the object name and one for the service in which it is running.
The counters are correctly saved on prometheus but I encountered a couple of problems.
The Promql formula I am using to determine the time it took an object to be processed by a service is the following : avg by ( objectName,application)(rate(metric_name_sum[1m])) / avg by ( objectName,application)(rate(metric_name_count[1m]))
The first problem I am encountering is that the services are really quick, therefore it's really common that counters get created during the scape interval even when dealing with 100 object with the same name and in this way the division of the two vectors will return NaN. I know this is the expected behavior from prometheus, but is there a way to change this so that it returns a positive integer from result? Afaik this is because rate works only when you have 2+ values in the specified range, so is there a way to initialize the counter before the actual recording of data so that it doesn't happen?
The second problem I have is that, assuming the metric is working correctly, I want to then sum the result of the aforementioned expression to obtain the total time that an object took to go through all the services , so I used this : sum by ( objectName) ( expression)
What I noticed is that it works only if the different vector i am summing together share the same time series e.g it works if two of them have data at 12.00 but not if one has data at 11.59 and the other at 12.01. Is there a way to use the sum operator disregarding this restraint? Is there maybe another operator or function that achieves what I want?
Thanks in advance for the cooperation, sorry for the long winded question, it's my first time writing here and I wanted to be as precise as possible.