My team just started using Prometheus for monitoring our Django web application (django-prometheus). Like any common setup, metrics are published to a "/metrics" endpoint, which is scraped by Prometheus.
Our problem is the fact that we deploy several times a day. On every deployment, the metrics are reset (all counters, gauges, etc. go back to zero) so we never have more than a few hours of metrics to review.
I'd imagine this is a common problem with Prometheus. Is more than a few hours worth of aggregated metrics okay? Is there something I should do to persist metrics at any point in the pipeline?