Prometheus target metrics scrape record time not match the configured interval time

user_17208804 0
Viewed 118

Prometheus config

- job_name: monitor/monitor-kubelet/1
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  ...

actual state

cpu usage metrics of a container

value            time            interval 
564010.259718631 @1634796181.961 35.13
564038.239253667 @1634796207.239 25.28
564061.535707756 @1634796234.983 27.74
564089.671915667 @1634796259.893 24.91
564120.98571483  @1634796292.126 32.23
564151.00582766  @1634796325.768 33.64
564180.591742732 @1634796358.304 32.54
564222.871858907 @1634796394.18 35.88
564250.313854137 @1634796424.652 30.47
564265.597564892 @1634796440.941 16.29
564288.788884994 @1634796471.012 30.07
564314.519065257 @1634796500.054 29.04
564340.769640412 @1634796533.434 33.38
564375.470987326 @1634796568.181 34.75
564392.951227567 @1634796587.909 19.73
564428.292375175 @1634796623.551 35.64
564465.554481447 @1634796660.313 36.76
564498.230676101 @1634796687.866 27.55
564533.118751601 @1634796720.387 32.52

This will cause breakpoints in the graph when irate(metrics).

How to optimize this situation？

prometheus
1 Answers

The scrape_interval is not something that is absolute. It just means that you'll have a scrape approximately every 30 seconds in your case.
So this is perfectly normal to have your scrape happening in a 25s to 35s window. I agree, the lines with 16s and 19s are "surprising" but it's not a bug.

Now for your graph, you should not experience any break assuming that you are using a correct range for your range query.
What's a correct range ? It should be at least four time your scrape interval, in your case it'll be 2min.
Why is that ? To avoid breakpoint due to missing scrape, slow scrape, slow ingestion, etc.

You can find more info on this here

user_115063910
Related