whenever we are restarting the Prometheus service it causes data loss of 2-3 hours. This was not happening previously, but when we started adding more data/integration into it, we are experiencing this issue.
Tried to google over but did not find any answer to it, is anyone facing the same issue?
Prometheus version: 2.13.1
Any leads will help, TIA.
Prometheus stores up to 2 hours of recently scraped metrics in write-ahead log aka WAL. Every two hours it runs a compaction job, which converts the data from WAL into a persistent block. Prometheus converts data from WAL into a persistent block on graceful shutdown (e.g. when SIGINT or SIGTERM signal is sent to it and the controlling process waits until Prometheus process successfully finishes).
It is likely in your case Prometheus is shut down via SIGKILL signal or via out of memory event (aka OOM). In these cases the WAL data with recently collected samples (up to 2 hours) isn't stored in a persistent block, so it becomes lost.
Prometheus tries replaying WAL (e.g. recovering data from WAL) on the next start, but it may fail to do so if WAL data is corrupted due to the previous unclean shutdown of the Prometheus. See this article for details.
P.S. Try another Prometheus-like monitoring system I work on - VictoriaMetrics. It doesn't use WAL - instead it stores data to persistent blocks every second. So it may lose only data collected during the last second before unclean shutdown.
