Is there a way to monitor kube cron jobs using prometheus

I'm using these rules with kube-state-metrics:

groups:
- name: job.rules
  rules:
  - alert: CronJobRunning
    expr: time() -kube_cronjob_next_schedule_time > 3600
    for: 1h
    labels:
      severity: warning
    annotations:
      description: CronJob {{$labels.namespaces}}/{{$labels.cronjob}} is taking more than 1h to complete
      summary: CronJob didn't finish after 1h

  - alert: JobCompletion
    expr: kube_job_spec_completions - kube_job_status_succeeded  > 0
    for: 1h
    labels:
      severity: warning
    annotations:
      description: Job completion is taking more than 1h to complete
        cronjob {{$labels.namespaces}}/{{$labels.job}}
      summary: Job {{$labels.job}} didn't finish to complete after 1h

  - alert: JobFailed
    expr: kube_job_status_failed  > 0
    for: 1h
    labels:
      severity: warning
    annotations:
      description: Job {{$labels.namespaces}}/{{$labels.job}} failed to complete
      summary: Job failed

The way to monitoring cronjobs with Prometheus is to have them push a metric indicating the last time they succeeded to the pushgateway. You can then alert on if the cronjob hasn't succeeded recently enough.

All answers so far are unaware of namespaces, and are dependent on custom labeling in the Job.

The latter can be fixed as kube-state-metrics version 1.6.0 introduced a new metric kube_job_owner which solves the problem matching Jobs and CronJobs.

NOTE: In kube-state-metrics 1.4.0 the job label was renamed to job_name to avoid a label collision with Prometheus.

clamp_max(
  max by (namespace, owner_name, job_name) (
    max by (namespace, owner_name, job_name) (
      kube_job_status_start_time
      *
      on (job_name) group_left(owner_name) max by (namespace, owner_name, job_name) (kube_job_owner{owner_kind="CronJob"})
    )
    ==
    on (namespace, owner_name) group_left max by (namespace, owner_name) (
      kube_job_status_start_time
      *
      on (job_name) group_left(owner_name) max by (namespace, owner_name, job_name) (kube_job_owner{owner_kind="CronJob"})
    )
  ),
  1
)
*
on (namespace, job_name) group_left kube_job_status_failed

The output can be further improved by renaming the owner_name label to cronjob by surrounding the expression with

max without (owner_name) (
  label_replace(
    <expression from above>
  ,
  "cronjob", "$1", "owner_name", "(.+)"
  )
)

(the label_replace() function adds a new cronjob label, while max() drops the owner_name label)

The kube-state-metrics exporter also includes various CronJob related metrics: https://github.com/kubernetes/kube-state-metrics/blob/master/Documentation/cronjob-metrics.md, but unfortunately doesn't seem to include success CronJob success/failure.

I was able to simplify this medium post (label_replace was not working for me for some reason) https://medium.com/@tristan_96324/prometheus-k8s-cronjob-alerts-94bee7b90511

My cron query looks like this (we have "component" labels on all cronjobs instead of "cronjob", but you can use your favorite label)

clamp_max(max(
    kube_job_status_start_time
    * ON(job) GROUP_RIGHT()
    kube_job_labels{label_component!=""}
  ) BY (job, label_component)
  == ON(label_component) GROUP_LEFT()
  max(
    kube_job_status_start_time
    * ON(job) GROUP_RIGHT()
    kube_job_labels{label_component!=""}
) BY (label_component), 1) * ON(job) GROUP_LEFT() 
kube_job_status_failed

Plug this into the prometheus expression dashboard to make sure you get results (1 means the cron failed the last time, 0 means it succeeded or hasn't run yet).

For alerting, add != 0, and the query will return with ANY cronjob that failed.