Prometheus alert rules for cluster autoscaler metrics

user_3217396 0
Viewed 604

I want to create a Prometheus alert rules for below scenarios,

Max capacity reached for the cluster

Unusual Scaling activity

I think "Max capacity reached for the cluster" can be obtained with combination of following metrics,

1.cluster_autoscaler_unscheulable_pods_count >0

2. sum(cluster_autoscaler_unneeded_nodes_count)==0

And,"Unusual Scaling activity" can be obtained from sum(cluster_autoscaler_scaled_up_nodes_total)

I have enabled metrics for Cluster autoscaler.However I am not sure how to create prometheus rule expressions with these metrics.Should I create any Service monitors? how to combine these metrics for the scenarios mentioned above? Do you already have examples of Prometheus rules for the Cluster autoscaler metrics?

prometheus-operator kubernetes
1 Answers

@Rad4 @Deepak Deore

Service monitor can be create here https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/values.yaml#L330-L332

For rules below are not exactly what you request, but I have found some openshift reference here: https://github.com/openshift/cluster-autoscaler-operator/blob/master/pkg/controller/clusterautoscaler/monitoring.go

I convert them into helm values.yaml format and you probably can reference on how I wrote them

    additionalPrometheusRulesMap:

      autoscaler.rules:
        groups:
        - name: autoscaler.prometheus.rules
          rules:
          - alert: ClusterAutoscalerUnschedulablePods
            expr: cluster_autoscaler_unschedulable_pods_count{service="aws-system"} > 0
            for: 20m
            labels:
              severity: info
            annotations:
              summary: Cluster Autoscaler has {{ $value }} unschedulable pods
              description: The cluster autoscaler is unable to scale up and is alerting that there are unschedulable pods because of this condition. This may be caused by the cluster autoscaler reaching its resources limits, or by Kubernetes waiting for new nodes to become ready.

          - alert: ClusterAutoscalerNotSafeToScale
            expr: cluster_autoscaler_cluster_safe_to_autoscale{service="aws-system"} !=1
            for: 15m
            labels:
              severity: warning
            annotations:
              summary: Cluster Autoscaler is reporting that the cluster is not ready for scaling
              description: The cluster autoscaler has detected that the number of unready nodes is too high and it is not safe to continute scaling operations. It makes this determination by checking that the number of ready nodes is greater than the minimum ready count (default of 3) and the ratio of unready to ready nodes is less than the maximum unready node percentage (default of 45%). If either of those conditions are not true then the cluster autoscaler will enter an unsafe to scale state until the conditions change.

          - alert: ClusterAutoscalerUnableToScaleCPULimitReached
            expr: cluster_autoscaler_cluster_cpu_current_cores >= cluster_autoscaler_cpu_limits_cores{direction="maximum"}
            for: 15m
            labels:
              severity: info
            annotations:
              summary: Cluster Autoscaler has reached its CPU core limit and is unable to scale out
              description: The number of total cores in the cluster has exceeded the maximum number set on the cluster autoscaler. This is calculated by summing the cpu capacity for all nodes in the cluster and comparing that number against the maximum cores value set for the cluster autoscaler (default 320000 cores). 

          - alert: ClusterAutoscalerUnableToScaleMemoryLimitReached
            expr: cluster_autoscaler_cluster_memory_current_bytes >= cluster_autoscaler_memory_limits_bytes{direction="maximum"}
            for: 15m
            labels:
              severity: info
            annotations:
              summary: Cluster Autoscaler has reached its Memory bytes limit and is unable to scale out
              description: The number of total bytes of RAM in the cluster has exceeded the maximum number set on the cluster autoscaler. This is calculated by summing the memory capacity for all nodes in the cluster and comparing that number against the maximum memory bytes value set for the cluster autoscaler (default 6400000 gigabytes).
user_66648160
Related