I am running Prometheus to monitor my system and am currently building in application-level metrics.
The issue I am having is with long running processes. I want to know how many are running at any time, and I can't seem to find a good solution for this with Prometheus.
The processes are running async from multiple redundant applications that both receive traffic at the same time. The processes run between 10 minutes and a few hours so my initial naive approach was to count a counter up whenever a process was started and increment another counter when the processes stopped.
The issue I am having here is that I see the uptick when a process starts in Grafana with the rate operator, but I can't monitor how many are running at any time.
Since two applications are managing the same pool of processes I can't really use a gauge to report the current number of running processes because a process might be started on instance A and then stop on instance B (or any other of the running application instances). (And the number of processes would be requested from a shared database so it would be inflated).
Now an approach I tried was to subtract started - finished counters to get to the currently running instance. But this would run out of sync quite quickly if the application reporting a process as finishing at some point would be restarted or die before the metric was scraped by prometheus (so I would get into a state where 0 is no longer the baseline).
Any suggestions on how to handle this?